From robert at bign.nl Sun May 1 16:27:04 2016 From: robert at bign.nl (Robert van Geel) Date: Sun, 1 May 2016 22:27:04 +0200 Subject: [Python-ideas] Object grabbing Message-ID: <57266698.7020602@bign.nl> First of all I'm new to this. I tried figuring out if some inquiry like mine below already was posted but I couldn't find it, frankly partly because I don't know what to look for, I'm not sure if there's a name for this idea. I'm not convinced my idea below is solid so apologies if it's na?ve but I figured to post it anyway. It has to do with the possibility to fold typical opcodes pairs by introducing a language construct. The idea is to be able to write this code: myobject.a myobject.b myobject.c() myobject.d = 1 like this: using myobject: .a .b .c() .d = 1 The first version would lead to these instructions: LOAD_FAST 0 (self) LOAD_ATTR 0 (a) POP_TOP LOAD_FAST 0 (self) LOAD_ATTR 1 (b) POP_TOP LOAD_FAST 0 (self) LOAD_ATTR 2 (c) CALL_FUNCTION 0 POP_TOP LOAD_CONST 1 (1) LOAD_FAST 0 (self) STORE_ATTR 3 (d) LOAD_CONST 0 (None) the using keyword would grab an internal handle to the object, leading to the ability to reduce the number of opcodes by introducing folded ones: PUSH_HANDLE 0 (myobject) LOAD_HATTR 0 (a) POP_TOP LOAD_HATTR 1 (b) POP_TOP LOAD_HATTR 1 (b) CALL_FUNCTION 0 (c) POP_TOP LOAD_CONST 1 (1) STORE_HATTR 3 (d) POP_HANDLE LOAD_CONST 0 (None) The rationale behind is that both typographically for the programmer this is more elegant than typing the variable name over and again. For the internals of Python it would reduce the number of handled opcodes at the cost of a number of new opcodes: PUSH_HANDLE => pushes object to 'direct handle stack' LOAD_HATTR = LOAD_FAST + LOAD_ATTR STORE_HATTR = LOAD_FAST + STORE_ATTR POP_HANDLE => pops object to 'direct handle stack' Since these pairs are quite numerous this could lead to a speed gain for code that uses a lot of "self." or "object." invocations. Shoot me. Robert From greg.ewing at canterbury.ac.nz Sun May 1 19:56:00 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 02 May 2016 11:56:00 +1200 Subject: [Python-ideas] Object grabbing In-Reply-To: <57266698.7020602@bign.nl> References: <57266698.7020602@bign.nl> Message-ID: <57269790.4080401@canterbury.ac.nz> Robert van Geel wrote: > using myobject: > .a > .b > .c() > .d = 1 > The rationale behind is that both typographically for the programmer this is > more elegant than typing the variable name over and again. Suggestions like this have been made before, and the conclusion has always been that very little would be gained over using a short intermediate name, e.g. m = myobject m.a() m.b() m.c() m.d = 1 That's just as readable and almost as easy to type. It also has the advantage that you're not restricted to just one "easy acess" object at a time. The only new thing in your proposal is the change to the bytecode, and that could be achieved by treating it as an optimisation. A sufficiently smart code generator could notice that you were repeatedly operating on the same object and produce the bytecode you suggest. -- Greg From rosuav at gmail.com Sun May 1 20:03:31 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 May 2016 10:03:31 +1000 Subject: [Python-ideas] Object grabbing In-Reply-To: <57269790.4080401@canterbury.ac.nz> References: <57266698.7020602@bign.nl> <57269790.4080401@canterbury.ac.nz> Message-ID: On Mon, May 2, 2016 at 9:56 AM, Greg Ewing wrote: > The only new thing in your proposal is the change to the > bytecode, and that could be achieved by treating it as > an optimisation. A sufficiently smart code generator > could notice that you were repeatedly operating on the same > object and produce the bytecode you suggest. "self" was mentioned. I wonder whether there could be some additional optimization magic applied to a function's first argument, same as the no-arg super() call? Since it's a semantic-free optimization, any Python implementation would be free to mess with this. Have opcodes for "LOAD_ARG1_ATTR" and "STORE_ARG1_ATTR", semantically equivalent to LOAD_FAST 0 followed by LOAD_ATTR or STORE_ATTR. ChrisA From steve at pearwood.info Sun May 1 21:02:11 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 2 May 2016 11:02:11 +1000 Subject: [Python-ideas] Object grabbing In-Reply-To: <57266698.7020602@bign.nl> References: <57266698.7020602@bign.nl> Message-ID: <20160502010211.GF13497@ando.pearwood.info> Hello Robert, and welcome, On Sun, May 01, 2016 at 10:27:04PM +0200, Robert van Geel wrote: [...] > The idea is to be able to write this code: > > myobject.a > myobject.b > myobject.c() > myobject.d = 1 > > like this: > > using myobject: > .a > .b > .c() > .d = 1 I think this is closely related to the Pascal "with" statement, which has a FAQ: https://docs.python.org/2/faq/design.html#why-doesn-t-python-have-a-with-statement-for-attribute-assignments You require a leading dot to access attribute names, which avoids the ambiguity between attributes and variables, but it also fails the "new syntax shouldn't look like grit on Tim's monitor" test. using myobject: .method(.eggs + (.spam or ham) - tomato - .cheese) All of these proposals raise the question, what do you do with nested blocks? using myobject: print(.attr) using yourobject: print(.attr) Obviously in the outer block, ".attr" refers to myobject. My guess is that inside the inner block, it refers to yourobject. What happens if you want to refer to both? Would we have to write this? using myobject: print(.attr) # myobject using yourobject: print(.attr) # yourobject print(myobject.attr) # be explicit Now imagine that you are a beginner, trying to understand this code block. What would you think it does? Might you not be surprised that the two references to ".attr" refer to different variables? And of course there is always the objection that the barrier to adding a new keyword is quite high. Somewhere out there, somebody is using "using" as a variable name (perhaps a decorator?) and making this a keyword will break her code. Is it worth it? Despite these objections, I'm cautiously interested in this. I don't think the objections are insurmountable, and if there is a significant gain in readability and performance, it may be worth while. A cautious and tentative +1. It may be worth you doing a survey of other languages and seeing if they have anything similar. -- Steve From random832 at fastmail.com Sun May 1 21:25:17 2016 From: random832 at fastmail.com (Random832) Date: Sun, 01 May 2016 21:25:17 -0400 Subject: [Python-ideas] Object grabbing In-Reply-To: <20160502010211.GF13497@ando.pearwood.info> References: <57266698.7020602@bign.nl> <20160502010211.GF13497@ando.pearwood.info> Message-ID: <1462152317.1889465.595152593.161A5618@webmail.messagingengine.com> On Sun, May 1, 2016, at 21:02, Steven D'Aprano wrote: > Now imagine that you are a beginner, trying to understand this code > block. What would you think it does? Might you not be surprised that > the two references to ".attr" refer to different variables? > > And of course there is always the objection that the barrier to adding a > new keyword is quite high. Somewhere out there, somebody is using > "using" as a variable name (perhaps a decorator?) and making this a > keyword will break her code. Is it worth it? Or you could make it context-sensitive, with it being an identifier unless it's used in this construction. Not saying it's a good idea, just that it's technically an option. > Despite these objections, I'm cautiously interested in this. I don't > think the objections are insurmountable, and if there is a significant > gain in readability and performance, it may be worth while. > > A cautious and tentative +1. > > It may be worth you doing a survey of other languages and seeing if they > have anything similar. Javascript's deprecated "with" statement is superficially similar, but it doesn't use the dot, instead it basically turns the scope for non-local variables into a chained map. Visual Basic has something (also called "with") that is much more similar to what's being proposed. Is there any reason not to simply implement a performance optimization for the normal syntax? Is the behavior of code which reaches into the parent frame and modifies its local variables guaranteed? From steve at pearwood.info Sun May 1 21:29:16 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 2 May 2016 11:29:16 +1000 Subject: [Python-ideas] Object grabbing In-Reply-To: <57269790.4080401@canterbury.ac.nz> References: <57266698.7020602@bign.nl> <57269790.4080401@canterbury.ac.nz> Message-ID: <20160502012916.GG13497@ando.pearwood.info> On Mon, May 02, 2016 at 11:56:00AM +1200, Greg Ewing wrote: > Suggestions like this have been made before, and the conclusion > has always been that very little would be gained over using > a short intermediate name, e.g. > > m = myobject > m.a() > m.b() > m.c() > m.d = 1 > > That's just as readable and almost as easy to type. It also > has the advantage that you're not restricted to just one > "easy acess" object at a time. Multiple short, simple statements like the above are not the best demonstration of the "one letter variable name" technique, nor of the advantage of the "Pascal with" (using) keyword. We should be thinking of larger expressions, with multiple references to the same name: print(myobject.method(myobject.eggs + (myobject.spam or ham)) - myobject.tomato - cheese) which becomes either: m = myobject print(m.method(m.eggs + (m.spam or ham)) - m.tomato - cheese) del m versus: using myobject: print(.method(.eggs + (.spam or ham)) - .tomato - cheese) I don't think there's a lot between them, but if the second can be significantly more efficient, that might push it over the line. Why do I `del m` at the end? Because this might be in the global scope, not inside a function, and you might not want an extraneous variable floating around polluting the namespace. > The only new thing in your proposal is the change to the > bytecode, and that could be achieved by treating it as > an optimisation. A sufficiently smart code generator > could notice that you were repeatedly operating on the same > object and produce the bytecode you suggest. No, I don't think it can be. Or rather, the semantics are not the same, and the compiler shouldn't choose to optimize the code. Consider: myobject.a() myobject.b() You cannot assume that it is the same myobject in both statements! Method a might have rebound the name to something else. Perhaps Victor's FAT Python will be smart enough to tell whether or not a global name myobject has been changed, but how about: myobject[1].lookup['key'].property.a() myobject[1].lookup['key'].property.b() I don't think any Python compiler is going to be able to look deep inside an arbitrarily complex reference and tell whether or not it is safe to optimize. But the programmer may be able to explicitly choose the optimization. Of course, in this case, they can just as easily factor out the constant part and assign it to a temporary short name: m = myobject[1].lookup['key'] m.property.a() m.property.b() # versus using myobject[1].lookup['key']: .property.a() .property.b() so it becomes a matter of two things: taste and performance. In my opinion, the two idioms are roughly the same in readability, give or take a few concerns about temp variables versus grit on Tim's monitor. But the first has to do two name lookups of "m", which the second can optimize. Can the second beat that by a significant amount? What if there were ten name lookups? Thirty? -- Steve From joshua.morton13 at gmail.com Sun May 1 21:32:50 2016 From: joshua.morton13 at gmail.com (Joshua Morton) Date: Mon, 02 May 2016 01:32:50 +0000 Subject: [Python-ideas] Object grabbing In-Reply-To: <20160502010211.GF13497@ando.pearwood.info> References: <57266698.7020602@bign.nl> <20160502010211.GF13497@ando.pearwood.info> Message-ID: To be fair the second problem you mention already exists with nested functions: def example(x): print(x) def inner(x): print(x) suffers from exactly the same issues as the proposed example. Admittedly, the case you give is much, much less explicit since the attributes are given in the function call example but not with the attribute access one. My thoughts on this syntax would be to add a special object to the standard library, that uses the existing `with` syntax. Take this as an example: import math with Namespace(math): print(pi**2) import math with Namespace(m=math): # alternatively `Namespace(math) as m:` print(m.pi**2) import foo, bar with Namespace(f=foo, b=bar): print(f.spam) print(b.spam) import math, pprint with Namespace(math, pprint): pprint(pi**2) With all of these modifying locals within the context manager. I assume that currently this could be done with some AST rewriting, but a clean standard implementation would be perhaps nice. There are some obvious problems: name conflicts could through an error, but that then leads to really ugly nested code or kludgy workarounds. I'm not sure of good solutions to those, but I'm a +1 if that matters, assuming those can be solved. --Josh On Sun, May 1, 2016 at 9:07 PM Steven D'Aprano wrote: > Hello Robert, and welcome, > > On Sun, May 01, 2016 at 10:27:04PM +0200, Robert van Geel wrote: > [...] > > The idea is to be able to write this code: > > > > myobject.a > > myobject.b > > myobject.c() > > myobject.d = 1 > > > > like this: > > > > using myobject: > > .a > > .b > > .c() > > .d = 1 > > I think this is closely related to the Pascal "with" statement, which has > a FAQ: > > > https://docs.python.org/2/faq/design.html#why-doesn-t-python-have-a-with-statement-for-attribute-assignments > > You require a leading dot to access attribute names, which avoids the > ambiguity between attributes and variables, but it also fails the "new > syntax shouldn't look like grit on Tim's monitor" test. > > using myobject: > .method(.eggs + (.spam or ham) - tomato - .cheese) > > > All of these proposals raise the question, what do you do with nested > blocks? > > using myobject: > print(.attr) > using yourobject: > print(.attr) > > > Obviously in the outer block, ".attr" refers to myobject. My guess is > that inside the inner block, it refers to yourobject. What happens if > you want to refer to both? Would we have to write this? > > using myobject: > print(.attr) # myobject > using yourobject: > print(.attr) # yourobject > print(myobject.attr) # be explicit > > > Now imagine that you are a beginner, trying to understand this code > block. What would you think it does? Might you not be surprised that > the two references to ".attr" refer to different variables? > > And of course there is always the objection that the barrier to adding a > new keyword is quite high. Somewhere out there, somebody is using > "using" as a variable name (perhaps a decorator?) and making this a > keyword will break her code. Is it worth it? > > > Despite these objections, I'm cautiously interested in this. I don't > think the objections are insurmountable, and if there is a significant > gain in readability and performance, it may be worth while. > > A cautious and tentative +1. > > It may be worth you doing a survey of other languages and seeing if they > have anything similar. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun May 1 21:36:34 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 2 May 2016 11:36:34 +1000 Subject: [Python-ideas] Object grabbing In-Reply-To: <1462152317.1889465.595152593.161A5618@webmail.messagingengine.com> References: <57266698.7020602@bign.nl> <20160502010211.GF13497@ando.pearwood.info> <1462152317.1889465.595152593.161A5618@webmail.messagingengine.com> Message-ID: <20160502013634.GH13497@ando.pearwood.info> On Sun, May 01, 2016 at 09:25:17PM -0400, Random832 wrote: > Is there any reason not to simply implement a performance optimization > for the normal syntax? Yes. You can't assume that myobject is the same object in both statements. A silly toy example: class MyObject: def deactivate(self): global myobject myobject = None def fire(self): launch_missiles() myobject.deactivate() myobject.fire() A more realistic case would be if you have a mutable object: myobject[1].lookup['key'].property.a() myobject[1].lookup['key'].property.b() The compiler cannot assume that (myobject[1].lookup['key'].property) will refer to the same thing in the two calls. But the programmer can explicitly do so, using either a temporary variable, or the proposed "using" statement. -- Steve From random832 at fastmail.com Sun May 1 21:41:10 2016 From: random832 at fastmail.com (Random832) Date: Sun, 01 May 2016 21:41:10 -0400 Subject: [Python-ideas] Object grabbing In-Reply-To: <20160502013634.GH13497@ando.pearwood.info> References: <57266698.7020602@bign.nl> <20160502010211.GF13497@ando.pearwood.info> <1462152317.1889465.595152593.161A5618@webmail.messagingengine.com> <20160502013634.GH13497@ando.pearwood.info> Message-ID: <1462153270.1892197.595166921.4FD1A847@webmail.messagingengine.com> On Sun, May 1, 2016, at 21:36, Steven D'Aprano wrote: > On Sun, May 01, 2016 at 09:25:17PM -0400, Random832 wrote: > > Is there any reason not to simply implement a performance optimization > > for the normal syntax? > > Yes. You can't assume that myobject is the same object in both > statements. A silly toy example: [snip] > A more realistic case would be if you have a mutable object: [snip] Well, by "normal" I meant assigning it to a temporary variable... and a local one, at that. Too bad there's no easy way to get a quick local scope without making a function, for stuff at the module level... From rosuav at gmail.com Sun May 1 22:05:13 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 May 2016 12:05:13 +1000 Subject: [Python-ideas] Object grabbing In-Reply-To: <20160502013634.GH13497@ando.pearwood.info> References: <57266698.7020602@bign.nl> <20160502010211.GF13497@ando.pearwood.info> <1462152317.1889465.595152593.161A5618@webmail.messagingengine.com> <20160502013634.GH13497@ando.pearwood.info> Message-ID: On Mon, May 2, 2016 at 11:36 AM, Steven D'Aprano wrote: > On Sun, May 01, 2016 at 09:25:17PM -0400, Random832 wrote: > >> Is there any reason not to simply implement a performance optimization >> for the normal syntax? > > Yes. You can't assume that myobject is the same object in both > statements. A silly toy example: > > class MyObject: > def deactivate(self): > global myobject > myobject = None > def fire(self): > launch_missiles() > > myobject.deactivate() > myobject.fire() > > A more realistic case would be if you have a mutable object: > > myobject[1].lookup['key'].property.a() > myobject[1].lookup['key'].property.b() > > The compiler cannot assume that (myobject[1].lookup['key'].property) > will refer to the same thing in the two calls. But the programmer can > explicitly do so, using either a temporary variable, or the proposed > "using" statement. If you profile a big program, I expect you'll find that module-level code amounts to a tiny proportion of run-time. Restricting this optimization to function-local names (no nonlocals, no globals - the LOAD_FAST opcode in current CPython) wouldn't materially harm it, and it would make the code a lot easier to reason about. The only possible issue that I can see is with an inner function messing with the outer function: def f(): obj = MyObject() def switch_obj(): nonlocal obj obj = MyObject() return 5 obj.attr1 = switch_obj() obj.attr2 = 42 return obj But a quick check with dis.dis(f) shows that this, too, uses LOAD_DEREF, so that should be safe - a byte-code transformation should safely be able to look for the LOAD_FAST and depend on it not changing (short of insane shenanigans - if you mutate sys._getframe().f_locals to tamper with another function's locals, you deserve all you get... and I'm not sure if it'd even work). This would be restricted to lookups on simple names. Frankly, I wouldn't expect anything else. If you want to call "myobject[1].lookup['key'].property.a" and "etc etc .b", you should probably be giving that a name *for the benefit of humans*, never mind about the interpreter! And if you want it optimized, definitely assign that to a local name. ChrisA From steve at pearwood.info Sun May 1 23:25:56 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 2 May 2016 13:25:56 +1000 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <20160502010211.GF13497@ando.pearwood.info> <1462152317.1889465.595152593.161A5618@webmail.messagingengine.com> <20160502013634.GH13497@ando.pearwood.info> Message-ID: <20160502032556.GI13497@ando.pearwood.info> On Mon, May 02, 2016 at 12:05:13PM +1000, Chris Angelico wrote: > This would be restricted to lookups on simple names. Frankly, I > wouldn't expect anything else. And I completely disagree. If this proposal is for simple names alone, I don't want it -- it would be one more syntactic feature to learn for virtually no benefit to the reader or the writer. (If the name is so long as to be a burden to type, copy and paste it.) > If you want to call > "myobject[1].lookup['key'].property.a" and "etc etc .b", you should > probably be giving that a name *for the benefit of humans* That goes completely against the advice "use a temporary one-letter name". Using, let's say, "x" or "m" for the variable name is not really for the benefit of the reader, otherwise it would be given a meaningful name like current_record_address. But that obviously goes against the idea of reducing the amount needed to type/read. *Whatever* solution you use, whether it is the one-letter temp name, or some variation on Pascal-with/using, you are compromising explicitness for convenience and/or efficiency. If this is worth doing, it has to be worth doing for complex references not just simple names. Otherwise, what's the point? Just use a shorter name. > never mind > about the interpreter! And if you want it optimized, definitely assign > that to a local name. If you have to create a one-off function just to get access to a local variable, the cost of creating a function will surely blow away any realistic gains you might get from using a local name. That shouldn't apply to the "using" block, which can use the same reference without needing to create a function object. If I'm reading you correctly, you're suggesting that (1) "using" should be limited to simple names, and (2) any (hypothetical) optimization should be only applied if the name is a function local, not inside a class definition or global scope. With these restrictions, the value of this feature keeps going down and down and down, to the point that it is not worth adding it. There is cost to new syntactic features, and once you strip away most of the benefits, you're left with something that is no longer worth the effort of adding and learning the new syntax. So a very strong -1 on the crippled/restricted version of "using", and a tentative +1 for the unrestricted version if it can be shown that there are significant performance gains to be made over the temp name version. (I'm still not quite 100% convinced that the benefit outways the "grit on Tim's monitor rule", but I'm leaning that way.) -- Steve From guido at python.org Sun May 1 23:27:40 2016 From: guido at python.org (Guido van Rossum) Date: Sun, 1 May 2016 20:27:40 -0700 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <20160502010211.GF13497@ando.pearwood.info> <1462152317.1889465.595152593.161A5618@webmail.messagingengine.com> <20160502013634.GH13497@ando.pearwood.info> Message-ID: On Sun, May 1, 2016 at 7:05 PM, Chris Angelico wrote: > On Mon, May 2, 2016 at 11:36 AM, Steven D'Aprano > wrote: > > On Sun, May 01, 2016 at 09:25:17PM -0400, Random832 wrote: > > > >> Is there any reason not to simply implement a performance optimization > >> for the normal syntax? > If the variable is already a local, there's not much of a performance win to be had, since replacing a LOAD_FAST with a DUP_TOP doesn't make much of a difference (and if it is anything else the optimization isn't safe, as you already explained). HOWEVER... I doubt that this is what the OP is after. I also don't think they're really after having to write less. I think what they want is to express the IDEA of doing various things to/using attributes of a specific object. (OK, I will now stop using all caps. :-) Like Steven, I think it's a reasonable idea. The idea of using a leading dot is good. There are some edge cases around nesting but I think they can be dealt with. I have also seen plenty of code that would benefit from this idiom, e.g. code that repeats something like `team.users[i]` several times in one call -- I imagine the author just didn't want to interrupt their flow by putting it in a local variable. But unlike Steven, I'm still lukewarm at most and would currently vote -0. Does this feature find a lot of use in other languages? If so, is the code using it typically clearer than the alternative? If a coder currently writes team.evict_user(team.users[i].uid, team.users[i].email, "Bye bye") because they can't be bothered to write user = team.users[i] team.evict_user(user.uid, user.email, "Bye bye") then would they bother writing this instead? using team.users[i]: team.evict_user(.uid, .email, "Bye bye") And is that really clearer? Even if it's used often enough that people will recognize it, the dot is an awfully small character and easily missed (on my screen there are smudges larger than a dot in the standard font :-). Plus there's the cost of the extra indent. Sure, omitting the user variable probably makes the line shorter, but what if there are other lines in the same block that don't use it? I've seen plenty of code doing something less readable to avoid an extra indent (which might require breaking lines that would otherwise just fit). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From leewangzhong+python at gmail.com Sun May 1 23:28:44 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Sun, 1 May 2016 23:28:44 -0400 Subject: [Python-ideas] Object grabbing In-Reply-To: <57266698.7020602@bign.nl> References: <57266698.7020602@bign.nl> Message-ID: On Sun, May 1, 2016 at 4:27 PM, Robert van Geel wrote: > First of all I'm new to this. I tried figuring out if some inquiry like mine > below already was posted but I couldn't find it, frankly partly because I > don't know what to look for, I'm not sure if there's a name for this idea. > I'm not convinced my idea below is solid so apologies if it's na?ve but I > figured to post it anyway. > It has to do with the possibility to fold typical opcodes pairs by > introducing a language construct. > > The idea is to be able to write this code: > > myobject.a > myobject.b > myobject.c() > myobject.d = 1 > > like this: > > using myobject: > .a > .b > .c() > .d = 1 Would the following solve your usecase? Explicit naming: with myobject import a,b,c,d: a b c() d = 1 Alternatively, putting the object at the end (like in gen expressions): with a,b,c,d from myobject: a b c() d = 1 Questions: * Should this add additional scope? * Descriptors and `__getattr__`-only attributes: Do you get the attribute at the start of the block, or do you call `__getattr__` every time? * That `d = 1` is Pythonically odd if it works as in the original example. From steve at pearwood.info Sun May 1 23:36:12 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 2 May 2016 13:36:12 +1000 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <20160502010211.GF13497@ando.pearwood.info> <1462152317.1889465.595152593.161A5618@webmail.messagingengine.com> <20160502013634.GH13497@ando.pearwood.info> Message-ID: <20160502033612.GJ13497@ando.pearwood.info> On Sun, May 01, 2016 at 08:27:40PM -0700, Guido van Rossum wrote: > Like Steven, I think it's a reasonable idea. [...] > But unlike Steven, I'm still lukewarm at most and would currently vote -0. That's why I say it's a tentative +1 :-) I think we need to see that the performance gains, if any, are real (not just hypothetical), and that the proposal is readable enough to overcome the fact that the leading dot is awfully small. And no, I wouldn't like to see a second attribute-access syntax like $attr just for the sake of this proposal. If the OP still wants to argue in favour of this, perhaps he can pull out some real-life examples of code that would benefit? -- Steve From me+python at ixokai.io Mon May 2 01:07:54 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Sun, 01 May 2016 22:07:54 -0700 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <20160502010211.GF13497@ando.pearwood.info> <1462152317.1889465.595152593.161A5618@webmail.messagingengine.com> <20160502013634.GH13497@ando.pearwood.info> Message-ID: <1462165674.1166221.595270801.3FE086D8@webmail.messagingengine.com> On Sun, May 1, 2016, at 08:27 PM, Guido van Rossum wrote: > If a coder currently writes > > team.evict_user(team.users[i].uid, team.users[i].email, "Bye bye") > > because they can't be bothered to write > user = team.users[i] > team.evict_user(user.uid, user.email, "Bye bye") > then would they bother writing this instead? > using team.users[i]: > team.evict_user(.uid, .email, "Bye bye") I seriously don't clean my screen enough to find this an OK syntax. "self.x" is perfectly readable, but not because the dot is particularly visible, but because the dot is sufficient to combine those two parts into a single unit to my eye. It makes the two parts blur together more then the dot is, itself, strongly visible. Every use case I can think of for this feature is better (as defined as more readable?) spelled as a local variable, even if its a short variable I'd normally argue against, like: u = team.users[i] team.evict_user(u.uid, u.email, "Bye bye") Normally I don't like variables like 'u', but if they're very local in functions that aren't long, they're fine. In those cases where they aren't fine, the proposed using's dot syntax would remove from vision what you're currently operating on and would therefore not be good either. To me this feature sacrifices readability for writability, and readability is more important. -- Stephen Hansen m e @ i x o k a i? . i o -------------- next part -------------- An HTML attachment was scrubbed... URL: From ian.g.kelly at gmail.com Mon May 2 01:37:49 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Sun, 1 May 2016 23:37:49 -0600 Subject: [Python-ideas] Object grabbing In-Reply-To: <20160502010211.GF13497@ando.pearwood.info> References: <57266698.7020602@bign.nl> <20160502010211.GF13497@ando.pearwood.info> Message-ID: On Sun, May 1, 2016 at 7:02 PM, Steven D'Aprano wrote: > And of course there is always the objection that the barrier to adding a > new keyword is quite high. Somewhere out there, somebody is using > "using" as a variable name (perhaps a decorator?) and making this a > keyword will break her code. Is it worth it? In particular, it would break Django: https://docs.djangoproject.com/en/1.9/ref/models/querysets/#using From greg.ewing at canterbury.ac.nz Mon May 2 01:55:13 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 02 May 2016 17:55:13 +1200 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <20160502010211.GF13497@ando.pearwood.info> Message-ID: <5726EBC1.6060202@canterbury.ac.nz> Joshua Morton wrote: > import math > with Namespace(math): > print(pi**2) That can't work if pi isn't assigned to as a local somewhere else in the function, because the compiler won't have allocated it a slot in locals for Namespace to modify. > with Namespace(m=math):` > print(m.pi**2) I don't see how that's any better than m = math print(m.pi**2) -- Greg From greg.ewing at canterbury.ac.nz Mon May 2 01:58:06 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 02 May 2016 17:58:06 +1200 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> Message-ID: <5726EC6E.1000804@canterbury.ac.nz> Franklin? Lee wrote: > with a,b,c,d from myobject: > a > b > c() > d = 1 This example is a bit weird -- if a, b, etc. are locals then it's equivalent in every way to just with a,b,c,d from myobject: c() d = 1 -- Greg From pavol.lisy at gmail.com Mon May 2 02:50:02 2016 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Mon, 2 May 2016 08:50:02 +0200 Subject: [Python-ideas] Object grabbing In-Reply-To: <57266698.7020602@bign.nl> References: <57266698.7020602@bign.nl> Message-ID: I just tried to extrapolate this idea a little bit. I searched for sources with many "self" in one line and found this function in mpmath/calculus/extrapolation.py: def factor_sidi(self, i): return (self.theta + self.n - 1) * (self.theta + self.n - 2) / self.ctx.mpf((self.theta + 2 * self.n - i - 2) * (self.theta + 2 * self.n - i - 3)) def factor_sidi(using self, i): return (.theta + .n - 1) * (.theta + .n - 2) / .ctx.mpf((.theta + 2 * .n - i - 2) * (.theta + 2 * .n - i - 3)) What dou you think? From robert at bign.nl Mon May 2 03:38:17 2016 From: robert at bign.nl (Robert van Geel) Date: Mon, 2 May 2016 09:38:17 +0200 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> Message-ID: <572703E9.1080907@bign.nl> On 5/2/2016 8:50 AM, Pavol Lisy wrote: > I just tried to extrapolate this idea a little bit. > > I searched for sources with many "self" in one line and found this > function in mpmath/calculus/extrapolation.py: > > def factor_sidi(self, i): > return (self.theta + self.n - 1) * (self.theta + self.n - 2) / > self.ctx.mpf((self.theta + 2 * self.n - i - 2) * (self.theta + 2 * > self.n - i - 3)) > > def factor_sidi(using self, i): > return (.theta + .n - 1) * (.theta + .n - 2) / > .ctx.mpf((.theta + 2 * .n - i - 2) * (.theta + 2 * .n - i - 3)) > > What dou you think? As stated in another post, I'm personally used to this syntax because I used to work in a language that has this (Visual Foxpro) so for me it feels logic and natural hence 'pythonic'. I can imagine that for the python eye this currently looks ... off... but the whole idea is that the construct is new so it would look a bit unusual for a while. From robert at bign.nl Mon May 2 03:34:31 2016 From: robert at bign.nl (Robert van Geel) Date: Mon, 2 May 2016 09:34:31 +0200 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> Message-ID: <57270307.1040809@bign.nl> On 5/2/2016 5:28 AM, Franklin? Lee wrote: > On Sun, May 1, 2016 at 4:27 PM, Robert van Geel wrote: >> First of all I'm new to this. I tried figuring out if some inquiry like mine >> below already was posted but I couldn't find it, frankly partly because I >> don't know what to look for, I'm not sure if there's a name for this idea. >> I'm not convinced my idea below is solid so apologies if it's na?ve but I >> figured to post it anyway. >> It has to do with the possibility to fold typical opcodes pairs by >> introducing a language construct. >> >> The idea is to be able to write this code: >> >> myobject.a >> myobject.b >> myobject.c() >> myobject.d = 1 >> >> like this: >> >> using myobject: >> .a >> .b >> .c() >> .d = 1 > Would the following solve your usecase? Explicit naming: > > with myobject import a,b,c,d: > a > b > c() > d = 1 > > > Alternatively, putting the object at the end (like in gen expressions): > > with a,b,c,d from myobject: > a > b > c() > d = 1 > > > Questions: > * Should this add additional scope? > * Descriptors and `__getattr__`-only attributes: Do you get the > attribute at the start of the block, or do you call `__getattr__` > every time? > * That `d = 1` is Pythonically odd if it works as in the original example. Sidenote: the idea I posed is actually borrowed from a language i used to use, Visual Foxpro, where it was implemented using the 'with' keyword. Your proposal has the need to duplicate all variables, so instead of multiple times typing "self." you have to type them double in the line using "with" so it's shorter but not optimally short. Also if you want the accompanying faster opcodes that means you have to put 4 variables on the stack straight away for fast access, instead of one. To answer you questions: - No it would not add scope, it's just a combination of syntactical sugar combined with faster opcodes. - The attributes are not grabbed at the beginning of the block, instead internally in memory there's a 'grabbed object' stack that can be accessed directly without the LOAD_FAST opcode - Your fourth remarks refers to your non-dotted code. I think the ".d = 1" statement feels pythonic but again maybe because I was so used to the syntax in the VFP language From jmcs at jsantos.eu Mon May 2 07:48:48 2016 From: jmcs at jsantos.eu (=?UTF-8?B?Sm/Do28gU2FudG9z?=) Date: Mon, 2 May 2016 13:48:48 +0200 Subject: [Python-ideas] Object grabbing In-Reply-To: <57270307.1040809@bign.nl> References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> Message-ID: I think the ".d = 1" statement feels like a bug waiting to happen. It's very easy to miss a dot. On 2 May 2016 at 09:34, Robert van Geel wrote: > On 5/2/2016 5:28 AM, Franklin? Lee wrote: > >> On Sun, May 1, 2016 at 4:27 PM, Robert van Geel wrote: >> >>> First of all I'm new to this. I tried figuring out if some inquiry like >>> mine >>> below already was posted but I couldn't find it, frankly partly because I >>> don't know what to look for, I'm not sure if there's a name for this >>> idea. >>> I'm not convinced my idea below is solid so apologies if it's na?ve but I >>> figured to post it anyway. >>> It has to do with the possibility to fold typical opcodes pairs by >>> introducing a language construct. >>> >>> The idea is to be able to write this code: >>> >>> myobject.a >>> myobject.b >>> myobject.c() >>> myobject.d = 1 >>> >>> like this: >>> >>> using myobject: >>> .a >>> .b >>> .c() >>> .d = 1 >>> >> Would the following solve your usecase? Explicit naming: >> >> with myobject import a,b,c,d: >> a >> b >> c() >> d = 1 >> >> >> Alternatively, putting the object at the end (like in gen expressions): >> >> with a,b,c,d from myobject: >> a >> b >> c() >> d = 1 >> >> >> Questions: >> * Should this add additional scope? >> * Descriptors and `__getattr__`-only attributes: Do you get the >> attribute at the start of the block, or do you call `__getattr__` >> every time? >> * That `d = 1` is Pythonically odd if it works as in the original >> example. >> > > Sidenote: the idea I posed is actually borrowed from a language i used to > use, Visual Foxpro, where it was implemented using the 'with' keyword. > > Your proposal has the need to duplicate all variables, so instead of > multiple times typing "self." you have to type them double in the line > using "with" so it's shorter but not optimally short. Also if you want the > accompanying faster opcodes that means you have to put 4 variables on the > stack straight away for fast access, instead of one. > > To answer you questions: > - No it would not add scope, it's just a combination of syntactical sugar > combined with faster opcodes. > - The attributes are not grabbed at the beginning of the block, instead > internally in memory there's a 'grabbed object' stack that can be accessed > directly without the LOAD_FAST opcode > - Your fourth remarks refers to your non-dotted code. I think the ".d = 1" > statement feels pythonic but again maybe because I was so used to the > syntax in the VFP language > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmcs at jsantos.eu Mon May 2 08:12:28 2016 From: jmcs at jsantos.eu (=?UTF-8?B?Sm/Do28gU2FudG9z?=) Date: Mon, 2 May 2016 14:12:28 +0200 Subject: [Python-ideas] Object grabbing In-Reply-To: <5727423A.3080807@bign.nl> References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <5727423A.3080807@bign.nl> Message-ID: That is matter of experience, I already lost too many hours of my life looking for weird bugs caused by misspellings and missing commas, and I don't even work with people new to python. On 2 May 2016 14:04, "Robert van Geel" wrote: > That's a matter of perception. As said this syntax was implemented and > used for a long time in another (now obsolete) language (visual foxpro) and > there it never led to confusion or errors nor have I ever heard such > complaints from any other user in that community. For Python developers, > since this is a yet unfamiliar syntax, this might look unfamiliar and hence > rejectable. > > On 5/2/2016 1:48 PM, Jo?o Santos wrote: > > I think the ".d = 1" statement feels like a bug waiting to happen. It's > very easy to miss a dot. > > On 2 May 2016 at 09:34, Robert van Geel wrote: > >> On 5/2/2016 5:28 AM, Franklin? Lee wrote: >> >>> On Sun, May 1, 2016 at 4:27 PM, Robert van Geel wrote: >>> >>>> First of all I'm new to this. I tried figuring out if some inquiry like >>>> mine >>>> below already was posted but I couldn't find it, frankly partly because >>>> I >>>> don't know what to look for, I'm not sure if there's a name for this >>>> idea. >>>> I'm not convinced my idea below is solid so apologies if it's na?ve but >>>> I >>>> figured to post it anyway. >>>> It has to do with the possibility to fold typical opcodes pairs by >>>> introducing a language construct. >>>> >>>> The idea is to be able to write this code: >>>> >>>> myobject.a >>>> myobject.b >>>> myobject.c() >>>> myobject.d = 1 >>>> >>>> like this: >>>> >>>> using myobject: >>>> .a >>>> .b >>>> .c() >>>> .d = 1 >>>> >>> Would the following solve your usecase? Explicit naming: >>> >>> with myobject import a,b,c,d: >>> a >>> b >>> c() >>> d = 1 >>> >>> >>> Alternatively, putting the object at the end (like in gen expressions): >>> >>> with a,b,c,d from myobject: >>> a >>> b >>> c() >>> d = 1 >>> >>> >>> Questions: >>> * Should this add additional scope? >>> * Descriptors and `__getattr__`-only attributes: Do you get the >>> attribute at the start of the block, or do you call `__getattr__` >>> every time? >>> * That `d = 1` is Pythonically odd if it works as in the original >>> example. >>> >> >> Sidenote: the idea I posed is actually borrowed from a language i used to >> use, Visual Foxpro, where it was implemented using the 'with' keyword. >> >> Your proposal has the need to duplicate all variables, so instead of >> multiple times typing "self." you have to type them double in the line >> using "with" so it's shorter but not optimally short. Also if you want the >> accompanying faster opcodes that means you have to put 4 variables on the >> stack straight away for fast access, instead of one. >> >> To answer you questions: >> - No it would not add scope, it's just a combination of syntactical sugar >> combined with faster opcodes. >> - The attributes are not grabbed at the beginning of the block, instead >> internally in memory there's a 'grabbed object' stack that can be accessed >> directly without the LOAD_FAST opcode >> - Your fourth remarks refers to your non-dotted code. I think the ".d = >> 1" statement feels pythonic but again maybe because I was so used to the >> syntax in the VFP language >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Mon May 2 08:20:14 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Mon, 2 May 2016 15:20:14 +0300 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> Message-ID: On Mon, May 2, 2016 at 2:48 PM, Jo?o Santos wrote: > I think the ".d = 1" statement feels like a bug waiting to happen. It's very > easy to miss a dot. > I suppose that's a valid concern, especially regarding assignments, because the code typically would still run. But in the beginning of the line, it is usually quite easy to see if there's a dot or not, assuming a fixed-width font and proper indenting. Anyway, it seems to me this would make sense in with statements: with open(filename): #do stuff .write(stuff) #do more stuff .write(more_stuff) -- Koos From tritium-list at sdamon.com Mon May 2 08:44:18 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Mon, 2 May 2016 08:44:18 -0400 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> Message-ID: <57274BA2.3030907@sdamon.com> with open(foo), open(bar): .write(baz) # does what? On 5/2/2016 08:20, Koos Zevenhoven wrote: > On Mon, May 2, 2016 at 2:48 PM, Jo?o Santos wrote: >> I think the ".d = 1" statement feels like a bug waiting to happen. It's very >> easy to miss a dot. >> > I suppose that's a valid concern, especially regarding assignments, > because the code typically would still run. But in the beginning of > the line, it is usually quite easy to see if there's a dot or not, > assuming a fixed-width font and proper indenting. > > Anyway, it seems to me this would make sense in with statements: > > with open(filename): > #do stuff > .write(stuff) > #do more stuff > .write(more_stuff) > > -- Koos > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From walker_s at hotmail.co.uk Mon May 2 08:48:17 2016 From: walker_s at hotmail.co.uk (SW) Date: Mon, 2 May 2016 13:48:17 +0100 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> Message-ID: On 02/05/16 13:20, Koos Zevenhoven wrote: > On Mon, May 2, 2016 at 2:48 PM, Jo?o Santos wrote: >> I think the ".d = 1" statement feels like a bug waiting to happen. It's very >> easy to miss a dot. >> > I suppose that's a valid concern, especially regarding assignments, > because the code typically would still run. But in the beginning of > the line, it is usually quite easy to see if there's a dot or not, > assuming a fixed-width font and proper indenting. It may be easy to see if there's a dot or not, but it may not be easy to tell whether there /should/ be a dot when there isn't. e.g. .configuration = {'yes': 'no'} vs configuration = {'yes': 'no'} When you might have later in the code something that assigns to a variable with the same name, and/or operates on that variable I think it'd become more difficult to determine. Thanks, S From rob.cliffe at btinternet.com Mon May 2 09:27:13 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 2 May 2016 14:27:13 +0100 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> Message-ID: <572755B1.2040603@btinternet.com> On 02/05/2016 07:50, Pavol Lisy wrote: > I just tried to extrapolate this idea a little bit. > > I searched for sources with many "self" in one line and found this > function in mpmath/calculus/extrapolation.py: > > def factor_sidi(self, i): > return (self.theta + self.n - 1) * (self.theta + self.n - 2) / > self.ctx.mpf((self.theta + 2 * self.n - i - 2) * (self.theta + 2 * > self.n - i - 3)) > > def factor_sidi(using self, i): > return (.theta + .n - 1) * (.theta + .n - 2) / > .ctx.mpf((.theta + 2 * .n - i - 2) * (.theta + 2 * .n - i - 3)) > > What dou you think? What is wrong with def factor_sidi(self, i): theta = self.theta n = self.n return (theta + n - 1) * (theta + n - 2) / self.ctx.mpf((theta + 2 * n - i - 2) * (theta + 2 * n - i - 3)) Arguably more readable, and does 6 fewer attribute accesses. From k7hoven at gmail.com Mon May 2 09:57:02 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Mon, 2 May 2016 16:57:02 +0300 Subject: [Python-ideas] Object grabbing In-Reply-To: <57274BA2.3030907@sdamon.com> References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> Message-ID: On Mon, May 2, 2016 at 3:44 PM, Alexander Walters wrote: > with open(foo), open(bar): > .write(baz) # does what? > Hopefully it would raise an exception, even though, in the spirit of May 1st, it would pick one of the two files at random ;-). -- Koos From robert at bign.nl Mon May 2 08:04:10 2016 From: robert at bign.nl (Robert van Geel) Date: Mon, 2 May 2016 14:04:10 +0200 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> Message-ID: <5727423A.3080807@bign.nl> That's a matter of perception. As said this syntax was implemented and used for a long time in another (now obsolete) language (visual foxpro) and there it never led to confusion or errors nor have I ever heard such complaints from any other user in that community. For Python developers, since this is a yet unfamiliar syntax, this might look unfamiliar and hence rejectable. On 5/2/2016 1:48 PM, Jo?o Santos wrote: > I think the ".d = 1" statement feels like a bug waiting to happen. > It's very easy to miss a dot. > > On 2 May 2016 at 09:34, Robert van Geel > wrote: > > On 5/2/2016 5:28 AM, Franklin? Lee wrote: > > On Sun, May 1, 2016 at 4:27 PM, Robert van Geel > > wrote: > > First of all I'm new to this. I tried figuring out if some > inquiry like mine > below already was posted but I couldn't find it, frankly > partly because I > don't know what to look for, I'm not sure if there's a > name for this idea. > I'm not convinced my idea below is solid so apologies if > it's na?ve but I > figured to post it anyway. > It has to do with the possibility to fold typical opcodes > pairs by > introducing a language construct. > > The idea is to be able to write this code: > > myobject.a > myobject.b > myobject.c() > myobject.d = 1 > > like this: > > using myobject: > .a > .b > .c() > .d = 1 > > Would the following solve your usecase? Explicit naming: > > with myobject import a,b,c,d: > a > b > c() > d = 1 > > > Alternatively, putting the object at the end (like in gen > expressions): > > with a,b,c,d from myobject: > a > b > c() > d = 1 > > > Questions: > * Should this add additional scope? > * Descriptors and `__getattr__`-only attributes: Do you > get the > attribute at the start of the block, or do you call `__getattr__` > every time? > * That `d = 1` is Pythonically odd if it works as in the > original example. > > > Sidenote: the idea I posed is actually borrowed from a language i > used to use, Visual Foxpro, where it was implemented using the > 'with' keyword. > > Your proposal has the need to duplicate all variables, so instead > of multiple times typing "self." you have to type them double in > the line using "with" so it's shorter but not optimally short. > Also if you want the accompanying faster opcodes that means you > have to put 4 variables on the stack straight away for fast > access, instead of one. > > To answer you questions: > - No it would not add scope, it's just a combination of > syntactical sugar combined with faster opcodes. > - The attributes are not grabbed at the beginning of the block, > instead internally in memory there's a 'grabbed object' stack that > can be accessed directly without the LOAD_FAST opcode > - Your fourth remarks refers to your non-dotted code. I think the > ".d = 1" statement feels pythonic but again maybe because I was so > used to the syntax in the VFP language > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Mon May 2 10:23:14 2016 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Mon, 2 May 2016 16:23:14 +0200 Subject: [Python-ideas] Specify an alternative exception for "assert" Message-ID: assert statement gives the possibility to display the text which goes along with the AssertionError exception. Most of the times though, what would be more appropriate is to raise a different exception (e.g. ValueError). My proposal is to be able to specify an exception as a replacement for AssertionError as in: >>> assert callable(fun), ValueError("object is not a callable") ValueError: object is not a callable Specifically, this would be useful at the top of a function or method, where argument types or values are usually checked: def retry(times=3, timeout=0.1, callback=None): assert times >= 1, ValueError("times must be >= 1") assert isinstance(timeout, (int, float)), ValueError("invalid timeout") assert callable(callback), ValueError("callback is not a callable") ...as opposed to: def retry(times=3, timeout=0.1, callback=None): if not times >= 1: raise ValueError("times must be >= 1") if not isinstance(timeout, (int, float)): raise ValueError("invalid timeout") if not callable(callback): raise ValueError("callback is not a callable") Other than saving 1 line for each type/value check, this has the advantage that the assertion logic (e.g. "times >= 1") is shown in the traceback message itself, because it's on the same line, enriching the context and giving more information in case of error. Thoughts? -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruce at leban.us Mon May 2 10:35:38 2016 From: bruce at leban.us (Bruce Leban) Date: Mon, 2 May 2016 07:35:38 -0700 Subject: [Python-ideas] Object grabbing In-Reply-To: <57266698.7020602@bign.nl> References: <57266698.7020602@bign.nl> Message-ID: Using a single letter (or short) variable name works well most of the time but has one problem: the variable can leak. It's easy to forget to write the del statement. Imagine a block statement that had the effect of deleting any variables initialized in the block. That is (using the with keyword in this example but there are other choices): x = foo() with: y = x + 1 x = y + 2 is equivalent to: x = foo() y = x + 1 x = y + 2 del y or after optimization: x = (foo() + 1) + 2 [Obviously more useful with real code but this is sufficiently illustrative. Also I imagine someone might say it's just foo() + 3 but I don't know the type of foo().] --- Bruce -- --- Bruce Check out my puzzle book and get it free here: http://J.mp/ingToConclusionsFree (available on iOS) -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert at bign.nl Mon May 2 10:25:10 2016 From: robert at bign.nl (Robert van Geel) Date: Mon, 2 May 2016 16:25:10 +0200 Subject: [Python-ideas] Object grabbing (was: Re: Python-ideas Digest, Vol 114, Issue 5) In-Reply-To: References: Message-ID: <57276346.3030902@bign.nl> I'm receiving digests and seem to not have each individual mail, so here's a digested response: One side of this that's not discussed is the possible optimization. I can imagine if you've got a lot of objectproperty write and -reads this could actually make certain use cases a lot faster such as this one, saving 6 opcodes: def __init__(self, left, top, width, height, content): self.left = left self.top = top self.width = width self.height = height self.content = content.upper() self.decideColors() self.draw() versus: def __init__(self, left, top, width, height, content): with self: .left = left .top = top .width = width .height = height .content = content.upper() .decideColors() .draw() The suggestion that you could accomplish this with a (peephole) optimizer does not seem quite correct to me: x = myobject.b() ... z = myobject.c() does not necessarily have a consistent pointer to myobject although it would require some acrobatics to change them in a way that can not be seen by an optimizer. You can think about exec() or even another thread intervening into a generator function, not something I would do but who knows. The advantage of the construct is that the .dotted variable is guaranteed to point to the same physical object in memory with an increased reference count during the statement and a decrease happening at dedent. Django would break when the 'using' keyword would be used for that, but the choice of keyword is arbitrary. Maybe an extended use of the 'with' word would be elegant, when the object would not have an __enter__ and/or __exit__ function it could still run for the purpose of this mechanism. The disadvantage is that you could not use the with construct for this purpose only without also triggering the __enter__ function. On 5/2/2016 2:49 PM, python-ideas-request at python.org wrote: > [cut] > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 2 May 2016 14:12:28 +0200 > From: Jo?o Santos > To: Robert van Geel > Cc: python-ideas at python.org, "Franklin? Lee" > > Subject: Re: [Python-ideas] Object grabbing > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > That is matter of experience, I already lost too many hours of my life > looking for weird bugs caused by misspellings and missing commas, and I > don't even work with people new to python. > On 2 May 2016 14:04, "Robert van Geel" wrote: > > ------------------------------ Message: 4 Date: Mon, 2 May 2016 > 13:48:17 +0100 From: SW To: > python-ideas at python.org Subject: Re: [Python-ideas] Object grabbing > Message-ID: > Content-Type: text/plain; charset="utf-8" On 02/05/16 13:20, Koos > Zevenhoven wrote: >> On Mon, May 2, 2016 at 2:48 PM, Jo?o Santos wrote: >>> I think the ".d = 1" statement feels like a bug waiting to happen. It's very >>> easy to miss a dot. >>> >> I suppose that's a valid concern, especially regarding assignments, >> because the code typically would still run. But in the beginning of >> the line, it is usually quite easy to see if there's a dot or not, >> assuming a fixed-width font and proper indenting. > It may be easy to see if there's a dot or not, but it may not be easy to > tell whether there /should/ be a dot when there isn't. > > e.g. > > .configuration = {'yes': 'no'} > vs > > configuration = {'yes': 'no'} > > When you might have later in the code something that assigns to a > variable with the same name, and/or operates on that variable I think > it'd become more difficult to determine. > > Thanks, > S > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > > ------------------------------ > > End of Python-ideas Digest, Vol 114, Issue 5 > ******************************************** From rymg19 at gmail.com Mon May 2 11:01:04 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 2 May 2016 10:01:04 -0500 Subject: [Python-ideas] Specify an alternative exception for "assert" In-Reply-To: References: Message-ID: Other than the fact that this would completely fail when run with -O... I believe I brought this up a while back (~1-2 years), and that was the reply. -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ On May 2, 2016 9:23 AM, "Giampaolo Rodola'" wrote: > assert statement gives the possibility to display the text which goes > along with the AssertionError exception. Most of the times though, what > would be more appropriate is to raise a different exception (e.g. > ValueError). My proposal is to be able to specify an exception as a > replacement for AssertionError as in: > > >>> assert callable(fun), ValueError("object is not a callable") > ValueError: object is not a callable > > Specifically, this would be useful at the top of a function or method, > where argument types or values are usually checked: > > def retry(times=3, timeout=0.1, callback=None): > assert times >= 1, ValueError("times must be >= 1") > assert isinstance(timeout, (int, float)), ValueError("invalid timeout") > assert callable(callback), ValueError("callback is not a callable") > > ...as opposed to: > > def retry(times=3, timeout=0.1, callback=None): > if not times >= 1: > raise ValueError("times must be >= 1") > if not isinstance(timeout, (int, float)): > raise ValueError("invalid timeout") > if not callable(callback): > raise ValueError("callback is not a callable") > > Other than saving 1 line for each type/value check, this has the advantage > that the assertion logic (e.g. "times >= 1") is shown in the traceback > message itself, because it's on the same line, enriching the context and > giving more information in case of error. > > Thoughts? > > -- > Giampaolo - http://grodola.blogspot.com > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon May 2 11:14:19 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 2 May 2016 08:14:19 -0700 Subject: [Python-ideas] Specify an alternative exception for "assert" In-Reply-To: References: Message-ID: On Mon, May 2, 2016 at 8:01 AM, Ryan Gonzalez wrote: > Other than the fact that this would completely fail when run with -O... > But maybe that's fine, or intended. I can see a fair number of uses for this, including subclasses of AssertionError. +0.5 -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Mon May 2 11:17:49 2016 From: random832 at fastmail.com (Random832) Date: Mon, 02 May 2016 11:17:49 -0400 Subject: [Python-ideas] Specify an alternative exception for "assert" In-Reply-To: References: Message-ID: <1462202269.1077984.595681529.06779B74@webmail.messagingengine.com> On Mon, May 2, 2016, at 11:14, Guido van Rossum wrote: > On Mon, May 2, 2016 at 8:01 AM, Ryan Gonzalez wrote: > > > Other than the fact that this would completely fail when run with -O... > > > But maybe that's fine, or intended. > > I can see a fair number of uses for this, including subclasses of > AssertionError. I think this would be an attractive nuisance, and if it's implemented at all it should forbid types that are *not* subclasses of AssertionError in order to mitigate that. From random832 at fastmail.com Mon May 2 11:21:31 2016 From: random832 at fastmail.com (Random832) Date: Mon, 02 May 2016 11:21:31 -0400 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> Message-ID: <1462202491.1078913.595683249.3A398DD7@webmail.messagingengine.com> On Mon, May 2, 2016, at 10:35, Bruce Leban wrote: > Using a single letter (or short) variable name works well most of the > time > but has one problem: the variable can leak. It's easy to forget to write > the del statement. Imagine a block statement that had the effect of > deleting any variables initialized in the block. Should it delete the variables, or should it be a new scope? They're subtly different. If the former, how do you differentiate a variable initialized in the block from a variable assigned in the block? What if you really *do* want to keep one of them? (The answer in both cases if it's a new scope would be to use "nonlocal") From g.rodola at gmail.com Mon May 2 11:28:44 2016 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Mon, 2 May 2016 17:28:44 +0200 Subject: [Python-ideas] Specify an alternative exception for "assert" In-Reply-To: <1462202269.1077984.595681529.06779B74@webmail.messagingengine.com> References: <1462202269.1077984.595681529.06779B74@webmail.messagingengine.com> Message-ID: On Mon, May 2, 2016 at 5:17 PM, Random832 wrote: > On Mon, May 2, 2016, at 11:14, Guido van Rossum wrote: > > On Mon, May 2, 2016 at 8:01 AM, Ryan Gonzalez wrote: > > > > > Other than the fact that this would completely fail when run with -O... > > > > > But maybe that's fine, or intended. > > > > I can see a fair number of uses for this, including subclasses of > > AssertionError. > > I think this would be an attractive nuisance, and if it's implemented at > all it should forbid types that are *not* subclasses of AssertionError > in order to mitigate that. Why such a constraint? I think most of the times the desired exception would be either ValueError or TypeError, both of which do not inherit from AssertionError. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon May 2 11:31:52 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 2 May 2016 08:31:52 -0700 Subject: [Python-ideas] Object grabbing (was: Re: Python-ideas Digest, Vol 114, Issue 5) In-Reply-To: <57276346.3030902@bign.nl> References: <57276346.3030902@bign.nl> Message-ID: On Mon, May 2, 2016 at 7:25 AM, Robert van Geel wrote: > I'm receiving digests and seem to not have each individual mail, so here's > a digested response: > > One side of this that's not discussed is the possible optimization. I can > imagine if you've got a lot of objectproperty write and -reads this could > actually make certain use cases a lot faster such as this one, saving 6 > opcodes: > The continued suggestion that this would be a considerable performance boost worries me. Compared to caching the expression into a local variable the speed-up will be immeasurable. Staring at generated byte code and counting instructions is not a good way to think about Python performance; unlike the model you may have of the hardware underneath, Python instructions, being object-oriented, are not roughly equivalent. The savings on local variable operations of the proposal is minuscule. > def __init__(self, left, top, width, height, content): > self.left = left > self.top = top > self.width = width > self.height = height > self.content = content.upper() > self.decideColors() > self.draw() > > versus: > > def __init__(self, left, top, width, height, content): > with self: > .left = left > .top = top > .width = width > .height = height > .content = content.upper() > .decideColors() > .draw() > > The suggestion that you could accomplish this with a (peephole) optimizer > does not seem quite correct to me: > > x = myobject.b() > ... > z = myobject.c() > > does not necessarily have a consistent pointer to myobject although it > would require some acrobatics to change them in a way that can not be seen > by an optimizer. > You can think about exec() or even another thread intervening into a > generator function, not something I would do but who knows. > If myobject is a local variable it's actually very simple to know just from looking at the bytecodes in between. > The advantage of the construct is that the .dotted variable is guaranteed > to point to the same physical object in memory with an increased reference > count during the statement and a decrease happening at dedent. > I think you're barking up the wrong tree. If you want this construct you have to explain how it makes code clearer, more readable, more writable, less error-prone. > Django would break when the 'using' keyword would be used for that, but > the choice of keyword is arbitrary. Maybe an extended use of the 'with' > word would be elegant, when the object would not have an __enter__ and/or > __exit__ function it could still run for the purpose of this mechanism. The > disadvantage is that you could not use the with construct for this purpose > only without also triggering the __enter__ function. > Mixing this into 'with' would be terrible, because the two constructs have nothing in common. The choice if keyword is not entirely arbitrary; if we can't come up with a decent keyword the feature is dead. But the introduction would have to include a `from __future__ import ` statement anyway for at least one, maybe two release cycles (we did this with the `with` statement itself, around 2.4/2.5). So Django will have plenty of time to change. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon May 2 11:33:00 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 2 May 2016 08:33:00 -0700 Subject: [Python-ideas] Specify an alternative exception for "assert" In-Reply-To: <1462202269.1077984.595681529.06779B74@webmail.messagingengine.com> References: <1462202269.1077984.595681529.06779B74@webmail.messagingengine.com> Message-ID: On Mon, May 2, 2016 at 8:17 AM, Random832 wrote: > On Mon, May 2, 2016, at 11:14, Guido van Rossum wrote: > > On Mon, May 2, 2016 at 8:01 AM, Ryan Gonzalez wrote: > > > > > Other than the fact that this would completely fail when run with -O... > > > > > But maybe that's fine, or intended. > > > > I can see a fair number of uses for this, including subclasses of > > AssertionError. > > I think this would be an attractive nuisance, and if it's implemented at > all it should forbid types that are *not* subclasses of AssertionError > in order to mitigate that. > You can't make allegations of being an attractive nuisance without explaining your reasoning. Why do you think this increases bad behavior? What kind of bad behavior are you thinking of? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Mon May 2 11:36:07 2016 From: random832 at fastmail.com (Random832) Date: Mon, 02 May 2016 11:36:07 -0400 Subject: [Python-ideas] Specify an alternative exception for "assert" In-Reply-To: References: <1462202269.1077984.595681529.06779B74@webmail.messagingengine.com> Message-ID: <1462203367.1082172.595697481.4DB81FCD@webmail.messagingengine.com> On Mon, May 2, 2016, at 11:28, Giampaolo Rodola' wrote: > Why such a constraint? I think most of the times the desired exception > would be either ValueError or TypeError, both of which do not inherit > from > AssertionError. The point is that it's not appropriate to use assert for checks that are part of the semantics of the function. If all that you gain is having the check logic on the same source line as the raise for the stack trace, why not simply do "if value < 1: raise ValueError('value must be >= 1')"? if/raise is only two more characters than assert. "if not/raise" is a *bit* more, but not enough to worry about. You could trim off a character (and have a better style if we're squeamish about single-line suites) by adding a "raise ... if ..." syntax. From njs at pobox.com Mon May 2 11:36:52 2016 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 2 May 2016 08:36:52 -0700 Subject: [Python-ideas] Object grabbing (was: Re: Python-ideas Digest, Vol 114, Issue 5) In-Reply-To: <57276346.3030902@bign.nl> References: <57276346.3030902@bign.nl> Message-ID: On May 2, 2016 7:55 AM, "Robert van Geel" wrote: > > I'm receiving digests and seem to not have each individual mail, so here's a digested response: > > One side of this that's not discussed is the possible optimization. I can imagine if you've got a lot of objectproperty write and -reads this could actually make certain use cases a lot faster such as this one, saving 6 opcodes: > > def __init__(self, left, top, width, height, content): > self.left = left > self.top = top > self.width = width > self.height = height > self.content = content.upper() > self.decideColors() > self.draw() > > versus: > > def __init__(self, left, top, width, height, content): > with self: > .left = left > .top = top > .width = width > .height = height > .content = content.upper() > .decideColors() > .draw() > > The suggestion that you could accomplish this with a (peephole) optimizer does not seem quite correct to me: > > x = myobject.b() > ... > z = myobject.c() > > does not necessarily have a consistent pointer to myobject although it would require some acrobatics to change them in a way that can not be seen by an optimizer. > You can think about exec() or even another thread intervening into a generator function, not something I would do but who knows. It's actually part of CPython's current semantics that local variables cannot be modified via stack introspection exactly because it allows certain optimizations. You can read them, it might look like you can touch them (you can get access to what's allegedly the real dict of locals), but this is an illusion: modifications to that dict are not reflected back to the running code. See PyFrame_LocalsToFast and vice versa. So teaching the peephole optimizer to do this would be legal afaict. I'd be surprised and interested to see an example where those 6 opcodes made any non-trivial difference to a real program though -- thanks to the above-mentioned optimizations, accessing a local variable is already one of CPython's fastest operations. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon May 2 11:38:57 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 2 May 2016 10:38:57 -0500 Subject: [Python-ideas] Specify an alternative exception for "assert" In-Reply-To: References: <1462202269.1077984.595681529.06779B74@webmail.messagingengine.com> Message-ID: I feel like you just brought up the exact issue. Assertions should *never* be used for things like this. Because, one day, some stupid idiot is going to use assertions to perform some sort of data validation, someone else will use that library with -O, and the world will explode. It's just far too attractive to use but far too easy to misuse. And, if you really want: if my_cond: raise MyError('abc') Compare that to: assert my_cond, MyError('abc') Also, RPython uses assertions for error handling. Trust me, dealing with that is *not* fun. -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ On May 2, 2016 10:29 AM, "Giampaolo Rodola'" wrote: > > > On Mon, May 2, 2016 at 5:17 PM, Random832 wrote: > >> On Mon, May 2, 2016, at 11:14, Guido van Rossum wrote: >> > On Mon, May 2, 2016 at 8:01 AM, Ryan Gonzalez wrote: >> > >> > > Other than the fact that this would completely fail when run with >> -O... >> > > >> > But maybe that's fine, or intended. >> > >> > I can see a fair number of uses for this, including subclasses of >> > AssertionError. >> >> I think this would be an attractive nuisance, and if it's implemented at >> all it should forbid types that are *not* subclasses of AssertionError >> in order to mitigate that. > > > Why such a constraint? I think most of the times the desired exception > would be either ValueError or TypeError, both of which do not inherit from > AssertionError. > > -- > Giampaolo - http://grodola.blogspot.com > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Mon May 2 11:39:58 2016 From: random832 at fastmail.com (Random832) Date: Mon, 02 May 2016 11:39:58 -0400 Subject: [Python-ideas] Specify an alternative exception for "assert" In-Reply-To: References: <1462202269.1077984.595681529.06779B74@webmail.messagingengine.com> Message-ID: <1462203598.1082910.595702889.25D8AF9E@webmail.messagingengine.com> On Mon, May 2, 2016, at 11:33, Guido van Rossum wrote: > You can't make allegations of being an attractive nuisance without > explaining your reasoning. Why do you think this increases bad behavior? > What kind of bad behavior are you thinking of? Using assert for checks that are intended to be permanent rather than for debug purposes (i.e. the *reason* it would fail when run with -O); I thought that was obvious. From guido at python.org Mon May 2 11:54:44 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 2 May 2016 08:54:44 -0700 Subject: [Python-ideas] Specify an alternative exception for "assert" In-Reply-To: References: <1462202269.1077984.595681529.06779B74@webmail.messagingengine.com> Message-ID: On Mon, May 2, 2016 at 8:38 AM, Ryan Gonzalez wrote: > I feel like you just brought up the exact issue. Assertions should *never* > be used for things like this. Because, one day, some stupid idiot is going > to use assertions to perform some sort of data validation, someone else > will use that library with -O, and the world will explode. > > That's a very strong opinion, and a bit of a doomsday scenario. I hear worries about this a lot, but I've never heard a story from someone to whom this actually happened. > It's just far too attractive to use but far too easy to misuse. > The thing is, assert exists and is not going away. People are writing these asserts today (there are still many in asyncio). In many cases the assert is not guarding against bad data or user input, but simply against a confused caller. The assumption is that once the program is debugged enough to go to production (where you use -O) the need for the asserts has gone down enough that it doesn't matter the checks are gone -- the confused call site has long been found during tests and fixed. > And, if you really want: > > if my_cond: raise MyError('abc') > > Compare that to: > > assert my_cond, MyError('abc') > There's a bug in your example, and that bug is a good argument for using assert: the equivalent of that assert should be if not my_cond: raise MyError('abc') and that extra inversion often makes the code a little less straightforward. Compare if times < 1: raise ValueError() to assert times >= 1: raise ValueError() The latter states positively what we're expecting. This is much more helpful for the reader. Also, RPython uses assertions for error handling. Trust me, dealing with > that is *not* fun. > RPython is not Python; isn't that a feature? :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon May 2 12:32:44 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 2 May 2016 11:32:44 -0500 Subject: [Python-ideas] Specify an alternative exception for "assert" In-Reply-To: References: <1462202269.1077984.595681529.06779B74@webmail.messagingengine.com> Message-ID: On May 2, 2016 10:55 AM, "Guido van Rossum" wrote: > > On Mon, May 2, 2016 at 8:38 AM, Ryan Gonzalez wrote: >> >> I feel like you just brought up the exact issue. Assertions should *never* be used for things like this. Because, one day, some stupid idiot is going to use assertions to perform some sort of data validation, someone else will use that library with -O, and the world will explode. > > That's a very strong opinion, and a bit of a doomsday scenario. I hear worries about this a lot, but I've never heard a story from someone to whom this actually happened. >> >> It's just far too attractive to use but far too easy to misuse. > > The thing is, assert exists and is not going away. People are writing these asserts today (there are still many in asyncio). In many cases the assert is not guarding against bad data or user input, but simply against a confused caller. The assumption is that once the program is debugged enough to go to production (where you use -O) the need for the asserts has gone down enough that it doesn't matter the checks are gone -- the confused call site has long been found during tests and fixed. I would still prefer the proposal that requires the errors to derive from AssertionError, since there *are* people that will use this for some sort of data validation. It's more likely because of Python's "easier to ask forgiveness than permission" idea; imagine someone "asserts" with a custom exception that some sort of input data is correct/safe. Then someone else uses the library in their website. With -O. If it's restricted to subclasses of AssertionError, then people will be less likely to use it for random error checking that *should* run in release mode. >> >> And, if you really want: >> >> if my_cond: raise MyError('abc') >> >> Compare that to: >> >> assert my_cond, MyError('abc') > > There's a bug in your example, and that bug is a good argument for using assert: the equivalent of that assert should be > > if not my_cond: raise MyError('abc') > > and that extra inversion often makes the code a little less straightforward. Compare > > if times < 1: raise ValueError() > > to > > assert times >= 1: raise ValueError() > > The latter states positively what we're expecting. This is much more helpful for the reader. > Well, I was also typing from a phone while walking through the mall, so that may have something to do with the mistake. :) >> Also, RPython uses assertions for error handling. Trust me, dealing with that is *not* fun. > > > RPython is not Python; isn't that a feature? :-) > It is! :) > -- > --Guido van Rossum (python.org/~guido) -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon May 2 12:37:40 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 2 May 2016 09:37:40 -0700 Subject: [Python-ideas] Specify an alternative exception for "assert" In-Reply-To: References: <1462202269.1077984.595681529.06779B74@webmail.messagingengine.com> Message-ID: On Mon, May 2, 2016 at 9:32 AM, Ryan Gonzalez wrote: > I would still prefer the proposal that requires the errors to derive from > AssertionError, since there *are* people that will use this for some sort > of data validation. It's more likely because of Python's "easier to ask > forgiveness than permission" idea; imagine someone "asserts" with a custom > exception that some sort of input data is correct/safe. Then someone else > uses the library in their website. With -O. If it's restricted to > subclasses of AssertionError, then people will be less likely to use it for > random error checking that *should* run in release mode. > I don't believe your logic is correct. And the OP most assuredly does *not* want to raise a subclass of AssertionError. He's written enough Python code to know what he's talking about, so I take him serious -- whereas your argument so far is mere FUD. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Mon May 2 13:04:44 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 2 May 2016 19:04:44 +0200 Subject: [Python-ideas] Object grabbing In-Reply-To: <1462165674.1166221.595270801.3FE086D8@webmail.messagingengine.com> References: <57266698.7020602@bign.nl> <20160502010211.GF13497@ando.pearwood.info> <1462152317.1889465.595152593.161A5618@webmail.messagingengine.com> <20160502013634.GH13497@ando.pearwood.info> <1462165674.1166221.595270801.3FE086D8@webmail.messagingengine.com> Message-ID: <572788AC.7050306@mail.de> On 02.05.2016 07:07, Stephen Hansen wrote: > I seriously don't clean my screen enough to find this an OK syntax. > "self.x" is perfectly readable, but not because the dot is > particularly visible, but because the dot is sufficient to combine > those two parts into a single unit to my eye. It makes the two parts > blur together more then the dot is, itself, strongly visible. The same for me. -1 for this feature. > > Every use case I can think of for this feature is better (as defined > as more readable?) spelled as a local variable, even if its a short > variable I'd normally argue against, like: > > u = team.users[i] > team.evict_user(u.uid, u.email, "Bye bye") > > Normally I don't like variables like 'u', but if they're very local in > functions that aren't long, they're fine. In those cases where they > aren't fine, the proposed using's dot syntax would remove from vision > what you're currently operating on and would therefore not be good either. In this case, I could even live with "user". > To me this feature sacrifices readability for writability, and > readability is more important. Yep. Especially, when adding yet another way for "performance reasons". Python can optimize the presented cases without introducing a new syntax. Best, Sven From g.rodola at gmail.com Mon May 2 13:45:15 2016 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Mon, 2 May 2016 19:45:15 +0200 Subject: [Python-ideas] Specify an alternative exception for "assert" In-Reply-To: References: <1462202269.1077984.595681529.06779B74@webmail.messagingengine.com> Message-ID: On Mon, May 2, 2016 at 6:32 PM, Ryan Gonzalez wrote: > > On May 2, 2016 10:55 AM, "Guido van Rossum" wrote: > > > > On Mon, May 2, 2016 at 8:38 AM, Ryan Gonzalez wrote: > >> > >> I feel like you just brought up the exact issue. Assertions should > *never* be used for things like this. Because, one day, some stupid idiot > is going to use assertions to perform some sort of data validation, someone > else will use that library with -O, and the world will explode. > > > > That's a very strong opinion, and a bit of a doomsday scenario. I hear > worries about this a lot, but I've never heard a story from someone to whom > this actually happened. > >> > >> It's just far too attractive to use but far too easy to misuse. > > > > The thing is, assert exists and is not going away. People are writing > these asserts today (there are still many in asyncio). In many cases the > assert is not guarding against bad data or user input, but simply against a > confused caller. The assumption is that once the program is debugged enough > to go to production (where you use -O) the need for the asserts has gone > down enough that it doesn't matter the checks are gone -- the confused call > site has long been found during tests and fixed. > > I would still prefer the proposal that requires the errors to derive from > AssertionError > The proposal I'm making is the exact opposite (allow a different exception than AssertionError) so I'm not sure I understand your objection. > , since there *are* people that will use this for some sort of data > validation. > Personally I never used asserts for data validation. Usually they are used: #1 in unit tests, if you don't like using the verbose unittest's self.assert* APIs #2 in application code as an extra safety to make sure, e.g., that the function's caller is passing the "right things", that an attribute has a certain value at some point, etc. Note that #2 is meant to help the developer(s), not the final user. It's something which is there as a guard against possible mistakes but not much more, and to my understanding this is how most people feel about "assert"s. Could allowing to raise a specific and "more correct" exception type as a one-liner be an incentive to use "assert" to validate user input? I don't know, but my impression is that: #1 most people are "educated" enough to know that assert is usually not meant to validate user input #2 very few people use -O (e.g. I believe I never used it myself) #3 I expect whoever uses -O is supposed to be aware of what are the implications of it -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon May 2 14:07:13 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 3 May 2016 04:07:13 +1000 Subject: [Python-ideas] Specify an alternative exception for "assert" In-Reply-To: References: Message-ID: <20160502180712.GK13497@ando.pearwood.info> On Mon, May 02, 2016 at 04:23:14PM +0200, Giampaolo Rodola' wrote: > assert statement gives the possibility to display the text which goes along > with the AssertionError exception. Most of the times though, what would be > more appropriate is to raise a different exception (e.g. ValueError). My > proposal is to be able to specify an exception as a replacement for > AssertionError as in: > > >>> assert callable(fun), ValueError("object is not a callable") > ValueError: object is not a callable My apologies for the length of this post. For the benefit of those in a hurry, here is the executive summary, or TL;DR version: Strong -1 for this proposal. Assertions have different semantics to explicit "if condition: raise" checks, and it is good and useful that they behave differently. In particular, a failed assert should always be a bug, and never an expected exception which the user may wish to catch. We should not blur the difference between an assert and an explict if...raise just for the sake of saving a line of code. If you find yourself wanting assert to raise a different exception type (like TypeError), that is a warning sign that you shouldn't be using assert: you're probably abusing assert for code code that needs the if...raise semantics, not the assert semantics. Longer, and hopefully more articulate explanation (and hopefully not too rambling) follows: I work with some people who insist on always using "assert" for all their error checking. ALL of it, whether of public or private functions, including end-user data validation. Because "it saves a line" and "it makes it easy to remember what exception to catch".[1] So their code is riddled with "try...except AssertionError", and heaven help us if the end user runs their code with -O. "It doesn't matter, nobody will do that." (Fortunately, so far the end users have not been savvy enough to know about -O.) Assertions have a number of uses, such as checked comments, testing invariants, contract checking[2], etc. Some time ago, I wrote this to try to explain what I believe are good and bad uses of assert: http://import-that.dreamwidth.org/676.html I believe that assert (as opposed to an explicit test and raise) strongly communicates the intent of the programmer: assert condition, "error" says that this is an internal check which (in a bug-free program) is safe to disable. The assertion can be thought of as a form of "trust, but verify". There are a few different interpretations of assert, but they all agree that *assertions are safe to remove* (provided the program is bug-free). Once you are confident that the assertions will never trigger, you can safely disable them, and your program should still work. Whereas an explicit test and raise: if not condition: raise Exception("error") strongly says that this is checking *external* input from a source that cannot be trusted (say, user supplied input, or external code that doesn't obey your code's internal contracts). Rather than "trust but verify" it is more "don't trust", and even in a bug-free program, you cannot do without this check. You can never disable these checks. (By trust, I don't necessarily mean that the user or code is actively hostile and trying to subvert your function. I just mean that you cannot trust that it will provide valid input to your function. You MUST perform the check, even in a bug-free program.) I think that the difference between those two sorts of checks is important, and I do not wish to see the distinction weakened or removed. I think that this proposal will weaken that distinction by allowing assert to raise non-AssertionError and encouraging people to treat it as a lazy shortcut for if...raise. I think it will make it harder to distinguish between program bugs and expected errors that can be caught. More on this below. (At the risk of weakening my own argument, I acknowledge that there are grey areas where there may be legitimate difference of opinion whether a particular check is better as an assert or an if...raise. Sometimes it comes down to the programmer's opinion. But I think it is still important to keep the distinction, and not blur the two cases just for the sake of those grey areas, and especially not just to save a line and a few keystrokes.) AssertionError strongly indicates an internal program bug (a logic error, a failed checked comment, a contract violation etc). Consequently, I should never need to catch AssertionError directly. AssertionError is always a failure of "trust, but verify" and therefore an internal bug. And importantly, no other exceptions should count as this sort of failure. (I'm giving an ideal, of course, and in real life people vary in how rigourously they apply the ideals I describe. Some people seemingly choose exceptions arbitrarily. No doubt we all have to deal with badly chosen exceptions. But just because people will misuse exceptions no matter what we do, doesn't mean we should add syntax to make it easier to misuse them.) For example, contract violations shouldn't raise ValueError, because then the caller might treat that contract violation (a bug) as an expected error and catch the exception. Same goes for invariants and checked comments: process(thelist) # if we get here, the list has at least two items assert len(thelist) >= 2, 'error' The intent here is that AssertionError is *not* part of the function API, it should not be treated as an expected error which the caller can catch and deal with. AssertionError signals strongly "don't think about catching this, it's a bug that must be fixed, not an expected error". If the failure *is* expected, then I ought to communicate that clearly by using an explicit if...raise with a different exception: process(thelist) # if we get here and the list has less than two items, that's an error if len(thelist) < 2, ValueError('error') But with the proposed change, the code can send mixed messages: assert len(thelist) >= 2, ValueError('error') The assert says that this can be safely discarded once the program is bug-free, i.e. that the exception should never be raised. But the ValueError says that the exception is expected and the test shouldn't be removed. If ValueError does get raised, is that a failed invariant, i.e. a bug? Or an expected error that the caller can and should deal with? Who can tell? Is it safe to disable that assertion? The intention of the programmer is harder to tell, and there are more ways to get it wrong. [1] I admit it: sometimes I'm lazy and use assert this way too. We're all human. But never in production code. [2] As in Design By Contract. -- Steve From barry at python.org Mon May 2 14:45:27 2016 From: barry at python.org (Barry Warsaw) Date: Mon, 2 May 2016 14:45:27 -0400 Subject: [Python-ideas] Object grabbing (was: Re: Python-ideas Digest, Vol 114, Issue 5) References: <57276346.3030902@bign.nl> Message-ID: <20160502144527.30e3c03a@anarchist.wooz.org> On May 02, 2016, at 04:25 PM, Robert van Geel wrote: >Django would break when the 'using' keyword would be used for that Not that it's super popular, but I'd hate to lose 'using' to a keyword too. http://flufli18n.readthedocs.io/en/latest/docs/using.html Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From njs at pobox.com Mon May 2 14:51:46 2016 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 2 May 2016 11:51:46 -0700 Subject: [Python-ideas] Object grabbing (was: Re: Python-ideas Digest, Vol 114, Issue 5) In-Reply-To: <20160502144527.30e3c03a@anarchist.wooz.org> References: <57276346.3030902@bign.nl> <20160502144527.30e3c03a@anarchist.wooz.org> Message-ID: On May 2, 2016 11:46 AM, "Barry Warsaw" wrote: > > On May 02, 2016, at 04:25 PM, Robert van Geel wrote: > > >Django would break when the 'using' keyword would be used for that > > Not that it's super popular, but I'd hate to lose 'using' to a keyword too. > > http://flufli18n.readthedocs.io/en/latest/docs/using.html I guess the obvious alternative would be in namespace: ... The pandas folks would love to have something like this actually, if it let you write in some_table: total_height = height_feet * 12 + height_inches instead of the current some_table["total_height"] = some_table["height_feet"] * 12 + some_table["height_inches"] Of course this requires that the namespace be accessed via some kind of hookable interface, not just a built in object or dict. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon May 2 15:34:20 2016 From: barry at python.org (Barry Warsaw) Date: Mon, 2 May 2016 15:34:20 -0400 Subject: [Python-ideas] Object grabbing (was: Re: Python-ideas Digest, Vol 114, Issue 5) In-Reply-To: References: <57276346.3030902@bign.nl> <20160502144527.30e3c03a@anarchist.wooz.org> Message-ID: <20160502153420.58df5e4e@anarchist.wooz.org> On May 02, 2016, at 11:51 AM, Nathaniel Smith wrote: >I guess the obvious alternative would be > >in namespace: > ... Reusing an existing keyword would be better, yes. >The pandas folks would love to have something like this actually, if it let >you write > >in some_table: > total_height = height_feet * 12 + height_inches > >instead of the current > >some_table["total_height"] = some_table["height_feet"] * 12 + >some_table["height_inches"] My first impression was yuk! because it wouldn't be clear to me where to find those names upon visual inspection of the code. But after thinking about it, it seems kind of interesting. I think what you're proposing is a layering of additional namespaces stack on (under?) the standard ones. >Of course this requires that the namespace be accessed via some kind of >hookable interface, not just a built in object or dict. Something like: class Kevinspacey: def __namespace__(self): return self.__dict__ Then an instance of Kevinspacey could be used in an in-namespace clause: hopper = Kevinspacey() in hopper: ants = 1 hopper.ants -> 1 Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From sjoerdjob at sjec.nl Mon May 2 15:37:24 2016 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Mon, 2 May 2016 21:37:24 +0200 Subject: [Python-ideas] Specify an alternative exception for "assert" In-Reply-To: <20160502180712.GK13497@ando.pearwood.info> References: <20160502180712.GK13497@ando.pearwood.info> Message-ID: <20160502193724.GA17057@sjoerdjob.com> My sentiments almost exactly. The only 'however' I have is with respect to the 'Design By Contract' part, as I do not think of `assert` as suitable for checking the pre-condition. The way I use `assert` is that I only use it for the cases where the `assert` can not be triggered except for when other programmers circumvent the API. I use it for: * Checking loop invariants (when especially tricky). * Checking class invariants. * Checking function arguments to protected/private functions and methods. But do not use it for: * Checking function arguments for public methods. * Checking public attributes of a class. I am not sure if by 'Design by Contract' you also mean the function arguments of public functions. Ideally, when I have written a class, none of the assertions get triggered unless *I* have made a bug in the implementation of said class. In particular, bugs (or mistakes in reasoning) in other classes should not be able to trigger the assertions. To me, `AssertionError` means: The implementor of said functionality made a mistake. It should never mean: "The developer called the library incorrectly", but always: "The library is broken.". However, even though I'm also -1 on the proposal as it stands, it also makes me recognise there is something missing. To be precise: a clearly defined way of doing expensive argument checking only when no '-O' is supplied. To me, `assert foo, bar` is (really desireable sugar) for if __debug__ and not (foo): raise AssertionError(bar) And I could likewise envision wanting sugar for if __debug__ and not (foo): raise bar Where `foo` is a really expensive check (or in a tight loop!). In cases where the source is external to the library/class/function, an `AssertionError` is not the right case. def bisect(values, value): # does not make sense, as `values` comes from outside. assert is_sorted(values) # does make sense, but is 'more expensive'. if __debug__ and not is_sorted(values): raise ValueError("Expected values to be sorted") # business-logic! Now it is tempting to re-use `assert` for that, as it also has the __debug__ sugarring. But using `assert` is not correct, as `assert` is not supposed to be used for checking across boundaries. On the other hand, it might also be a cue to look into solutions for supporting pre/post-condition checking in an elegant manner. As I am writing this, it does however deem to me that it is a matter of purism vs pragmatism. What is desired is some 'checking' that is to be done only during development. So from that point, I see 3 ways forward: 1. Add new syntax for the `raise bar` case above. However, I can not really see a good name for that. Maybe `check` or `verify`, but changes are quite high that would introduce a new keyword. 2. Add support for not only strings, but also exception-subclasses as the second argument to an `assert`. 3. Do nothing. Really tempting. The first approach is most pure, but will in essence boil down to the third approach because it is most expensive. The second approach seems like an impure, pragmatic and viable approach. I may not like it, but I think it stands more of a chance than the more pure (1), and delivers more value than (3). Kind regards, Sjoerd Job On Tue, May 03, 2016 at 04:07:13AM +1000, Steven D'Aprano wrote: > On Mon, May 02, 2016 at 04:23:14PM +0200, Giampaolo Rodola' wrote: > > assert statement gives the possibility to display the text which goes along > > with the AssertionError exception. Most of the times though, what would be > > more appropriate is to raise a different exception (e.g. ValueError). My > > proposal is to be able to specify an exception as a replacement for > > AssertionError as in: > > > > >>> assert callable(fun), ValueError("object is not a callable") > > ValueError: object is not a callable > > My apologies for the length of this post. For the benefit of those in a > hurry, here is the executive summary, or TL;DR version: > > Strong -1 for this proposal. > > Assertions have different semantics to explicit "if condition: raise" > checks, and it is good and useful that they behave differently. In > particular, a failed assert should always be a bug, and never an > expected exception which the user may wish to catch. We should not blur > the difference between an assert and an explict if...raise just for the > sake of saving a line of code. If you find yourself wanting assert to > raise a different exception type (like TypeError), that is a warning > sign that you shouldn't be using assert: you're probably abusing > assert for code code that needs the if...raise semantics, not the assert > semantics. > > Longer, and hopefully more articulate explanation (and hopefully not too > rambling) follows: > > > > I work with some people who insist on always using "assert" for all > their error checking. ALL of it, whether of public or private functions, > including end-user data validation. Because "it saves a line" and "it > makes it easy to remember what exception to catch".[1] So their code is > riddled with "try...except AssertionError", and heaven help us if the > end user runs their code with -O. "It doesn't matter, nobody will do > that." (Fortunately, so far the end users have not been savvy enough to > know about -O.) > > Assertions have a number of uses, such as checked comments, testing > invariants, contract checking[2], etc. Some time ago, I wrote this to > try to explain what I believe are good and bad uses of assert: > > http://import-that.dreamwidth.org/676.html > > I believe that assert (as opposed to an explicit test and raise) > strongly communicates the intent of the programmer: > > assert condition, "error" > > says that this is an internal check which (in a bug-free program) is > safe to disable. The assertion can be thought of as a form of "trust, > but verify". There are a few different interpretations of assert, but > they all agree that *assertions are safe to remove* (provided the > program is bug-free). Once you are confident that the assertions will > never trigger, you can safely disable them, and your program should > still work. > > Whereas an explicit test and raise: > > if not condition: raise Exception("error") > > strongly says that this is checking *external* input from a source that > cannot be trusted (say, user supplied input, or external code that > doesn't obey your code's internal contracts). Rather than "trust but > verify" it is more "don't trust", and even in a bug-free program, you > cannot do without this check. You can never disable these checks. > > (By trust, I don't necessarily mean that the user or code is actively > hostile and trying to subvert your function. I just mean that you cannot > trust that it will provide valid input to your function. You MUST > perform the check, even in a bug-free program.) > > I think that the difference between those two sorts of checks is > important, and I do not wish to see the distinction weakened or removed. > I think that this proposal will weaken that distinction by allowing > assert to raise non-AssertionError and encouraging people to treat it as > a lazy shortcut for if...raise. I think it will make it harder to > distinguish between program bugs and expected errors that can be caught. > More on this below. > > (At the risk of weakening my own argument, I acknowledge that there are > grey areas where there may be legitimate difference of opinion whether a > particular check is better as an assert or an if...raise. Sometimes it > comes down to the programmer's opinion. But I think it is still > important to keep the distinction, and not blur the two cases just for > the sake of those grey areas, and especially not just to save a line and > a few keystrokes.) > > > AssertionError strongly indicates an internal program bug (a logic > error, a failed checked comment, a contract violation etc). > Consequently, I should never need to catch AssertionError directly. > AssertionError is always a failure of "trust, but verify" and therefore > an internal bug. And importantly, no other exceptions should count as > this sort of failure. > > (I'm giving an ideal, of course, and in real life people vary in how > rigourously they apply the ideals I describe. Some people seemingly > choose exceptions arbitrarily. No doubt we all have to deal with badly > chosen exceptions. But just because people will misuse exceptions no > matter what we do, doesn't mean we should add syntax to make it easier > to misuse them.) > > For example, contract violations shouldn't raise ValueError, because > then the caller might treat that contract violation (a bug) as an > expected error and catch the exception. Same goes for invariants and > checked comments: > > process(thelist) > # if we get here, the list has at least two items > assert len(thelist) >= 2, 'error' > > > The intent here is that AssertionError is *not* part of the function > API, it should not be treated as an expected error which the caller can > catch and deal with. AssertionError signals strongly "don't think about > catching this, it's a bug that must be fixed, not an expected error". > > If the failure *is* expected, then I ought to communicate that clearly > by using an explicit if...raise with a different exception: > > process(thelist) > # if we get here and the list has less than two items, that's an error > if len(thelist) < 2, ValueError('error') > > > But with the proposed change, the code can send mixed messages: > > assert len(thelist) >= 2, ValueError('error') > > > The assert says that this can be safely discarded once the program is > bug-free, i.e. that the exception should never be raised. But the > ValueError says that the exception is expected and the test shouldn't be > removed. If ValueError does get raised, is that a failed invariant, i.e. > a bug? Or an expected error that the caller can and should deal with? > Who can tell? Is it safe to disable that assertion? The intention of the > programmer is harder to tell, and there are more ways to get it wrong. > > > > > > [1] I admit it: sometimes I'm lazy and use assert this way too. We're > all human. But never in production code. > > [2] As in Design By Contract. > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From python at lucidity.plus.com Mon May 2 16:25:01 2016 From: python at lucidity.plus.com (Erik) Date: Mon, 2 May 2016 21:25:01 +0100 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> Message-ID: <5727B79D.8060404@lucidity.plus.com> On 02/05/16 14:57, Koos Zevenhoven wrote: > On Mon, May 2, 2016 at 3:44 PM, Alexander Walters > wrote: >> with open(foo), open(bar): >> .write(baz) # does what? >> > > Hopefully it would raise an exception, Then you are suggesting that an attribute is looked up at runtime on _all_ objects declared in all live 'with' scopes every time one is referenced in order to determine if it's an ambiguous reference (at the time of that reference). O(n). I'm pretty sure that will never happen. E. From python at mrabarnett.plus.com Mon May 2 16:41:52 2016 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 2 May 2016 21:41:52 +0100 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> Message-ID: On 2016-05-02 14:57, Koos Zevenhoven wrote: > On Mon, May 2, 2016 at 3:44 PM, Alexander Walters > wrote: >> with open(foo), open(bar): >> .write(baz) # does what? >> > > Hopefully it would raise an exception, even though, in the spirit of > May 1st, it would pick one of the two files at random ;-). > This: with open(foo), open(bar): .write(baz) # does what? is basically just a shorter way to write this: with open(foo): with open(bar): .write(baz) # does what? so that suggests that it would pick the last one. Anyway, the exception (or warning?) could be raised at compile-time if the leading-dot form was used in a 'with' that had more than one expression. From python at lucidity.plus.com Mon May 2 17:03:25 2016 From: python at lucidity.plus.com (Erik) Date: Mon, 2 May 2016 22:03:25 +0100 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> Message-ID: <5727C09D.5070802@lucidity.plus.com> On 02/05/16 21:41, MRAB wrote: > Anyway, the exception (or warning?) could be raised at compile-time if > the leading-dot form was used in a 'with' that had more than one > expression. How will the compiler know what attributes will be available on the objects when the code is executed? Rgds, E. From k7hoven at gmail.com Mon May 2 17:34:15 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 3 May 2016 00:34:15 +0300 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> Message-ID: On Mon, May 2, 2016 at 11:41 PM, MRAB wrote: > On 2016-05-02 14:57, Koos Zevenhoven wrote: >> >> On Mon, May 2, 2016 at 3:44 PM, Alexander Walters wrote: >>> >>> with open(foo), open(bar): >>> .write(baz) # does what? >>> >> >> Hopefully it would raise an exception, even though, in the spirit of >> May 1st, it would pick one of the two files at random ;-). >> [...] > > Anyway, the exception (or warning?) could be raised at compile-time if the > leading-dot form was used in a 'with' that had more than one expression. > Exactly. A SyntaxError perhaps? So, if a single-expression with-statement would allow dot-attribute syntax for referring to attributes of the value, like in my previous example: with open(filename, 'w'): #do stuff .write(stuff) #more stuff .write(stuff) Then, if one introduces something roughly as follows: import contextlib @contextlib.contextmanager def implicit(obj): yield obj It would also allow one to do things like: with implicit(food): meal = .spam + .eggs + .cheese # meal = food.spam + food.eggs + food.cheese with implicit(self): .foo = foo # self.foo = foo .bar = bar # self.bar = bar This would be very explicit about what is implicit :). This way, the required addition to the with statement would indeed not change the meaning of 'with' (or mix it with different meanings). The only(?) addition would be .attr access to attributes of the object "yielded" by the context manager. -- Koos From luigi at semenzato.com Mon May 2 17:36:35 2016 From: luigi at semenzato.com (Luigi Semenzato) Date: Mon, 2 May 2016 14:36:35 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys Message-ID: Hello and sorry if this is an old and over-discussed topic. I could not find such discussions here. This was discussed and rejected at https://bugs.python.org/issue16385, but I am not convinced that the discussion presented all arguments properly. I tried reopening the bug at http://bugs.python.org/issue26910, but was told I should discuss it here instead. The original problem description: lives_in = { 'lion': ['Africa', 'America'], 'parrot': ['Europe'], #... 100+ more rows here 'lion': ['Europe'], #... 100+ more rows here } The above constructor overwrites the first 'lion' entry silently, often causing unexpected behavior. These are the arguments presented in favor of the rejection, followed by my rebuttals. 1. "An error is out of the question for compatibility reasons". No real rebuttal here, except I wonder if exceptions are ever made and under what circumstances. I should point out that a warning may also create incompatibilities. 2. "There are ways to rewrite this as a loop on a list". Yes of course, but the entire point of the dictionary literal is to offer a concise and convenient notation for entering a dictionary as program text. 3. "Being able to re-write keys is fundamental to Python dicts and why they can be used for Python's mutable namespaces". This is fine but it applies to the data structure, not the literal constructor. 4. "A code generator could depend on being able to write duplicate keys without having to go back and erase previous output". Yes, but it seems wrong to facilitate automated code generation at the expense of human code writing. For hand-written code, I claim that in most (all?) cases it would be preferable to have the compiler detect key duplications. It is easier for an automated code generator to check for duplications than it is for a human. 5. "There is already pylint". True, but it forces an additional step. For context, someone ran into this problem in my team at Google (we fixed it using pylint). I haven't seen any valid reason (in the bug or elsewhere) in favor of these constructor semantics. From the discussions I have seen, it seems just an oversight in the implementation/specification of dictionary literals. I'd be happy to hear stronger reasoning in favor of the status quo. (additional search keywords: dict constructor, dict literal) From python at lucidity.plus.com Mon May 2 17:37:57 2016 From: python at lucidity.plus.com (Erik) Date: Mon, 2 May 2016 22:37:57 +0100 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> Message-ID: <5727C8B5.8040205@lucidity.plus.com> On 02/05/16 22:34, Koos Zevenhoven wrote: > On Mon, May 2, 2016 at 11:41 PM, MRAB wrote: >> On 2016-05-02 14:57, Koos Zevenhoven wrote: >> Anyway, the exception (or warning?) could be raised at compile-time if the >> leading-dot form was used in a 'with' that had more than one expression. >> > > Exactly. A SyntaxError perhaps? Now you're suggesting a SyntaxError for valid syntax ;) E. From joshua.morton13 at gmail.com Mon May 2 18:00:08 2016 From: joshua.morton13 at gmail.com (Joshua Morton) Date: Mon, 02 May 2016 22:00:08 +0000 Subject: [Python-ideas] Object grabbing In-Reply-To: <5727C8B5.8040205@lucidity.plus.com> References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> <5727C8B5.8040205@lucidity.plus.com> Message-ID: It should be a name error, perhaps 'Accessed object's name cannot be inferred', or a similar error to UnboundLocalException. On Mon, May 2, 2016, 17:41 Erik wrote: > On 02/05/16 22:34, Koos Zevenhoven wrote: > > On Mon, May 2, 2016 at 11:41 PM, MRAB > wrote: > >> On 2016-05-02 14:57, Koos Zevenhoven wrote: > >> Anyway, the exception (or warning?) could be raised at compile-time if > the > >> leading-dot form was used in a 'with' that had more than one expression. > >> > > > > Exactly. A SyntaxError perhaps? > > Now you're suggesting a SyntaxError for valid syntax ;) > > E. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at lucidity.plus.com Mon May 2 18:06:04 2016 From: python at lucidity.plus.com (Erik) Date: Mon, 2 May 2016 23:06:04 +0100 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> <5727C8B5.8040205@lucidity.plus.com> Message-ID: <5727CF4C.5000201@lucidity.plus.com> On 02/05/16 23:00, Joshua Morton wrote: > It should be a name error, perhaps 'Accessed object's name cannot be > inferred', or a similar error to UnboundLocalException. You are missing the point. It is not reasonable to expect an error to be generated at compile time: because the compiler has no way of knowing what attributes will exist on the objects at runtime. It is not reasonable to expect an error to be generated at runtime: because each and every object would have to be queried as to whether they have the attribute to know if there is ambiguity on each and every reference to an attribute. The question of which error or warning or exception should be generated is moot ;) E. From python at lucidity.plus.com Mon May 2 18:11:17 2016 From: python at lucidity.plus.com (Erik) Date: Mon, 2 May 2016 23:11:17 +0100 Subject: [Python-ideas] Object grabbing In-Reply-To: <5727CF4C.5000201@lucidity.plus.com> References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> <5727C8B5.8040205@lucidity.plus.com> <5727CF4C.5000201@lucidity.plus.com> Message-ID: <5727D085.8000904@lucidity.plus.com> On 02/05/16 23:06, Erik wrote: > The question of which error or warning or exception should be generated > is moot ;) Before someone jumps on me, I meant 'irrelevant'. E. From python at mrabarnett.plus.com Mon May 2 18:19:36 2016 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 2 May 2016 23:19:36 +0100 Subject: [Python-ideas] Object grabbing In-Reply-To: <5727CF4C.5000201@lucidity.plus.com> References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> <5727C8B5.8040205@lucidity.plus.com> <5727CF4C.5000201@lucidity.plus.com> Message-ID: On 2016-05-02 23:06, Erik wrote: > On 02/05/16 23:00, Joshua Morton wrote: >> It should be a name error, perhaps 'Accessed object's name cannot be >> inferred', or a similar error to UnboundLocalException. > > You are missing the point. > > It is not reasonable to expect an error to be generated at compile time: > because the compiler has no way of knowing what attributes will exist on > the objects at runtime. > > It is not reasonable to expect an error to be generated at runtime: > because each and every object would have to be queried as to whether > they have the attribute to know if there is ambiguity on each and every > reference to an attribute. > > The question of which error or warning or exception should be generated > is moot ;) > You _don't_ have to know attributes are available. It's as simple(?) as: when it sees the leading-dot notation, it checks how many expressions the enclosing 'with' was followed by. For example: with something: .foo() is OK (the 'with' has 1 expression), but: with first_thing, second_else: .foo() might not be OK (the 'with' has 2 expressions). From joshua.morton13 at gmail.com Mon May 2 18:28:00 2016 From: joshua.morton13 at gmail.com (Joshua Morton) Date: Mon, 02 May 2016 22:28:00 +0000 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> <5727C8B5.8040205@lucidity.plus.com> <5727CF4C.5000201@lucidity.plus.com> Message-ID: > It is not reasonable to expect an error to be generated at runtime: > because each and every object would have to be queried as to whether > they have the attribute to know if there is ambiguity on each and every > reference to an attribute. It absolutely is: if you're using an "inferred" syntax like in object: .value but there are multiple such objects to look at, Python immediately throws an error, before doing any kind of lookup. Hence the error doesn't say "I couldn't find the attribute x on any object" but instead "There's ambiguity so I didn't even try". On Mon, May 2, 2016 at 6:19 PM MRAB wrote: > On 2016-05-02 23:06, Erik wrote: > > On 02/05/16 23:00, Joshua Morton wrote: > >> It should be a name error, perhaps 'Accessed object's name cannot be > >> inferred', or a similar error to UnboundLocalException. > > > > You are missing the point. > > > > It is not reasonable to expect an error to be generated at compile time: > > because the compiler has no way of knowing what attributes will exist on > > the objects at runtime. > > > > It is not reasonable to expect an error to be generated at runtime: > > because each and every object would have to be queried as to whether > > they have the attribute to know if there is ambiguity on each and every > > reference to an attribute. > > > > The question of which error or warning or exception should be generated > > is moot ;) > > > You _don't_ have to know attributes are available. It's as simple(?) as: > when it sees the leading-dot notation, it checks how many expressions > the enclosing 'with' was followed by. > > For example: > > with something: > .foo() > > is OK (the 'with' has 1 expression), but: > > with first_thing, second_else: > .foo() > > might not be OK (the 'with' has 2 expressions). > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Mon May 2 18:34:58 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 3 May 2016 01:34:58 +0300 Subject: [Python-ideas] Object grabbing In-Reply-To: <5727C8B5.8040205@lucidity.plus.com> References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> <5727C8B5.8040205@lucidity.plus.com> Message-ID: On Tue, May 3, 2016 at 12:37 AM, Erik wrote: > On 02/05/16 22:34, Koos Zevenhoven wrote: >> >> On Mon, May 2, 2016 at 11:41 PM, MRAB wrote: >>> >>> On 2016-05-02 14:57, Koos Zevenhoven wrote: >>> Anyway, the exception (or warning?) could be raised at compile-time if >>> the >>> leading-dot form was used in a 'with' that had more than one expression. >>> >> >> Exactly. A SyntaxError perhaps? > > > Now you're suggesting a SyntaxError for valid syntax ;) > No. It seems to me the signal-to-noise ratio of this thread is getting worse. D?j? vu (or French for 'already seen', if unicode fails you). What I'm suggesting (in addition to the actual suggestion) is not to allow plain .attr attribute access inside a with statement "with multiple expressions". Kind of the same way as, for instance, break and continue are not allowed outside loops: "SyntaxError: 'continue' not properly in loop". Or the same way as plain .attr syntax is currently not allowed at all: "SyntaxError: invalid syntax". -- Koos From python at lucidity.plus.com Mon May 2 18:59:05 2016 From: python at lucidity.plus.com (Erik) Date: Mon, 2 May 2016 23:59:05 +0100 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> <5727C8B5.8040205@lucidity.plus.com> Message-ID: <5727DBB9.1070104@lucidity.plus.com> On 02/05/16 23:34, Koos Zevenhoven wrote: > It seems to me the signal-to-noise ratio of this thread is getting > worse. I tend to agree. > D?j? vu (or French for 'already seen', if unicode fails you). What has translating one human language phrase to another human language got to do with Unicode? > What I'm suggesting (in addition to the actual suggestion) is not to > allow plain .attr attribute access inside a with statement "with > multiple expressions". Kind of the same way as, for instance, break > and continue are not allowed outside loops: You're now suggesting that the lexer/parser changes behaviour dynamically depending on what has been seen before. > Kind of the same way as, for instance, break > and continue are not allowed outside loops: Not the same thing at all. Those are absolute syntax rules. What you are suggesting is conditional based on context. E. From rosuav at gmail.com Mon May 2 19:30:01 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 3 May 2016 09:30:01 +1000 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> <5727C8B5.8040205@lucidity.plus.com> <5727CF4C.5000201@lucidity.plus.com> Message-ID: On Tue, May 3, 2016 at 8:19 AM, MRAB wrote: > You _don't_ have to know attributes are available. It's as simple(?) as: > when it sees the leading-dot notation, it checks how many expressions the > enclosing 'with' was followed by. > > For example: > > with something: > .foo() > > is OK (the 'with' has 1 expression), but: > > with first_thing, second_else: > .foo() > > might not be OK (the 'with' has 2 expressions). Replace "might" with "will" and I'd agree with you. Keep it simple: no ambiguity. ChrisA From greg.ewing at canterbury.ac.nz Mon May 2 19:32:33 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 03 May 2016 11:32:33 +1200 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <20160502010211.GF13497@ando.pearwood.info> <5726EBC1.6060202@canterbury.ac.nz> Message-ID: <5727E391.8060706@canterbury.ac.nz> Joshua Morton wrote: > would something like replacing all attribute access with a > > try: > x > except NameError: > namespace.x # or something like this > > work, or are you saying that since 'namespace' wouldn't be in slots > either, this would fail? I meant that Namespace can't be an ordinary Python object that works without cooperation from the compiler. If the compiler is allowed to recognise the use of Namespace and generate different code, anything is possible. -- Greg From python at mrabarnett.plus.com Mon May 2 19:46:25 2016 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 3 May 2016 00:46:25 +0100 Subject: [Python-ideas] Object grabbing In-Reply-To: <5727DBB9.1070104@lucidity.plus.com> References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> <5727C8B5.8040205@lucidity.plus.com> <5727DBB9.1070104@lucidity.plus.com> Message-ID: <6e2a14e1-bf1b-72ed-bf9b-0883dd963adf@mrabarnett.plus.com> On 2016-05-02 23:59, Erik wrote: > On 02/05/16 23:34, Koos Zevenhoven wrote: >> It seems to me the signal-to-noise ratio of this thread is getting >> worse. > > I tend to agree. > >> D?j? vu (or French for 'already seen', if unicode fails you). > > What has translating one human language phrase to another human language > got to do with Unicode? > The French phrase has accents. >> What I'm suggesting (in addition to the actual suggestion) is not to >> allow plain .attr attribute access inside a with statement "with >> multiple expressions". Kind of the same way as, for instance, break >> and continue are not allowed outside loops: > > You're now suggesting that the lexer/parser changes behaviour > dynamically depending on what has been seen before. > > > Kind of the same way as, for instance, break > > and continue are not allowed outside loops: > > Not the same thing at all. Those are absolute syntax rules. What you are > suggesting is conditional based on context. > The use of 'break' and 'continue' are also conditional on context; they can be used only within a loop. Another example: an empty 'raise' statement can be used only within an 'except' suite. This suggestion is that a leading .attr can be used only within a 'with' statement that has a single expression. From ericsnowcurrently at gmail.com Mon May 2 20:06:13 2016 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 2 May 2016 18:06:13 -0600 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> Message-ID: On Mon, May 2, 2016 at 8:35 AM, Bruce Leban wrote: > Using a single letter (or short) variable name works well most of the time > but has one problem: the variable can leak. It's easy to forget to write the > del statement. Imagine a block statement that had the effect of deleting any > variables initialized in the block. That is (using the with keyword in this > example but there are other choices): > > x = foo() > with: > y = x + 1 > x = y + 2 > > is equivalent to: > > x = foo() > y = x + 1 > x = y + 2 > del y > > or after optimization: > > x = (foo() + 1) + 2 > > [Obviously more useful with real code but this is sufficiently illustrative. > Also I imagine someone might say it's just foo() + 3 but I don't know the > type of foo().] Hmm. That reminds me of the "given" syntax (PEP 3150: https://www.python.org/dev/peps/pep-3150/). This is like allowing blocks of code to be treated as first-order, a la Ruby. Then again, functions give us just about all we need, and the "given" syntax basically gives us multi-line lambdas. :) Perhaps it's time to dust off that proposal. -eric From joshua.morton13 at gmail.com Mon May 2 21:41:08 2016 From: joshua.morton13 at gmail.com (Joshua Morton) Date: Tue, 03 May 2016 01:41:08 +0000 Subject: [Python-ideas] Object grabbing In-Reply-To: <5727E391.8060706@canterbury.ac.nz> References: <57266698.7020602@bign.nl> <20160502010211.GF13497@ando.pearwood.info> <5726EBC1.6060202@canterbury.ac.nz> <5727E391.8060706@canterbury.ac.nz> Message-ID: Right that makes sense. Is there any precedent for non-keywords being special cased by the compiler (I'm thinking specifically of Exceptions)? There's obviously from __future__ imports, but I think going in that direction would be simultaneously elegant, explicit, backwards compatible, and an awful idea. --Josh On Mon, May 2, 2016 at 7:32 PM Greg Ewing wrote: > Joshua Morton wrote: > > would something like replacing all attribute access with a > > > > try: > > x > > except NameError: > > namespace.x # or something like this > > > > work, or are you saying that since 'namespace' wouldn't be in slots > > either, this would fail? > > I meant that Namespace can't be an ordinary Python object > that works without cooperation from the compiler. If the > compiler is allowed to recognise the use of Namespace > and generate different code, anything is possible. > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon May 2 21:40:56 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 3 May 2016 11:40:56 +1000 Subject: [Python-ideas] Object grabbing In-Reply-To: <5727B79D.8060404@lucidity.plus.com> References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> <5727B79D.8060404@lucidity.plus.com> Message-ID: <20160503014055.GN13497@ando.pearwood.info> On Mon, May 02, 2016 at 09:25:01PM +0100, Erik wrote: > On 02/05/16 14:57, Koos Zevenhoven wrote: > >On Mon, May 2, 2016 at 3:44 PM, Alexander Walters > > wrote: > >>with open(foo), open(bar): > >> .write(baz) # does what? open(bar) could shadow open(foo), just like in this: with open(foo): .write(baz) # refers to foo with open(bar): .write(baz) # refers to bar, shadowing foo There's no ambiguity here: last one seen wins, the same rule used all over Python: import math as m, string as m, urllib as m, decimal as m x, x, x, x = 1, 2, 3, 4 etc. So I don't think this is an insurmountable problem. > >Hopefully it would raise an exception, > > Then you are suggesting that an attribute is looked up at runtime on > _all_ objects declared in all live 'with' scopes every time one is > referenced in order to determine if it's an ambiguous reference (at the > time of that reference). > > O(n). I'm pretty sure that will never happen. But for very small n. I mean, if you somehow manage to nest five or six hundred with statements, you probably deserve whatever happens to you... *wink* -- Steve From joshua.morton13 at gmail.com Mon May 2 21:43:01 2016 From: joshua.morton13 at gmail.com (Joshua Morton) Date: Tue, 03 May 2016 01:43:01 +0000 Subject: [Python-ideas] Object grabbing In-Reply-To: <5727E241.4020007@lucidity.plus.com> References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> <5727C8B5.8040205@lucidity.plus.com> <5727CF4C.5000201@lucidity.plus.com> <5727E241.4020007@lucidity.plus.com> Message-ID: The latter, the former isn't currently syntax (as in `with [a, b, c]"` will throw an error). This was, however, mainly if we were using a variant of the with syntax, since having But take the following example: def func(ctx): with ns: # namespace with ctx: # only known at runtime .attr You can't have that be a compile time error. Which actually makes the specific syntax of a bare `with` a strong -1 now, since mixing it with normal context managers is incredibly ambiguous. This would either need to be `in` or `given` as mentioned elsewhere, or something like with `Magic_Namespace_Object(ns):` --Josh On Mon, May 2, 2016 at 7:26 PM Erik wrote: > On 02/05/16 23:28, Joshua Morton wrote: > > > It is not reasonable to expect an error to be generated at runtime: > > > because each and every object would have to be queried as to whether > > > they have the attribute to know if there is ambiguity on each and > every > > > reference to an attribute. > > > > It absolutely is: > > It almost certainly is not ;) > > > if you're using an "inferred" syntax like > > > > in object: > > .value > > > > but there are multiple such objects to look at, Python immediately > > throws an error, > > It is not clear what you mean by "multiple such objects". Do you mean > that 'object' can be some sort of sequence, or do you mean that the > syntax used for multiple objects is "in object0, object1:" or similar? > > If the latter, then given the choice between the two, I would agree with > some sort of compile time error over a runtime exception. Why wait until > runtime? > > If the former (which might mean a runtime error is appropriate), then > you need to explain what you are suggesting in a lot more detail. > > Regards, E. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon May 2 21:47:00 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 2 May 2016 18:47:00 -0700 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> <5727C8B5.8040205@lucidity.plus.com> <5727CF4C.5000201@lucidity.plus.com> <5727E241.4020007@lucidity.plus.com> Message-ID: On Mon, May 2, 2016 at 6:43 PM, Joshua Morton wrote: > Which actually makes the specific syntax of a bare `with` a strong -1 now, > since mixing it with normal context managers is incredibly ambiguous. > I'm glad you eventually reached this conclusion. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon May 2 21:49:21 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 3 May 2016 11:49:21 +1000 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> <20160502010211.GF13497@ando.pearwood.info> <5726EBC1.6060202@canterbury.ac.nz> <5727E391.8060706@canterbury.ac.nz> Message-ID: <20160503014920.GO13497@ando.pearwood.info> On Tue, May 03, 2016 at 01:41:08AM +0000, Joshua Morton wrote: > Right that makes sense. Is there any precedent for non-keywords being > special cased by the compiler (I'm thinking specifically of Exceptions)? For a while "as" was context sensitive, being treated as a keyword where necessary but still allowing you to use it as a variable name. In Python 2.5: py> import math as as py> as :1: Warning: 'as' will become a reserved keyword in Python 2.6 I believe that the core devs have sworn the most terrible and bloody oaths to never allow abominations like this again... *wink* -- Steve From jcgoble3 at gmail.com Mon May 2 22:12:39 2016 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Mon, 2 May 2016 22:12:39 -0400 Subject: [Python-ideas] Object grabbing In-Reply-To: <20160503014920.GO13497@ando.pearwood.info> References: <57266698.7020602@bign.nl> <20160502010211.GF13497@ando.pearwood.info> <5726EBC1.6060202@canterbury.ac.nz> <5727E391.8060706@canterbury.ac.nz> <20160503014920.GO13497@ando.pearwood.info> Message-ID: On Mon, May 2, 2016 at 9:49 PM, Steven D'Aprano wrote: > On Tue, May 03, 2016 at 01:41:08AM +0000, Joshua Morton wrote: >> Right that makes sense. Is there any precedent for non-keywords being >> special cased by the compiler (I'm thinking specifically of Exceptions)? > > For a while "as" was context sensitive, being treated as a keyword where > necessary but still allowing you to use it as a variable name. In Python > 2.5: > > py> import math as as > py> as > :1: Warning: 'as' will become a reserved keyword in Python 2.6 > > > I believe that the core devs have sworn the most terrible and bloody > oaths to never allow abominations like this again... > > *wink* async and await. Special-cased in certain contexts in 3.5 and 3.6 and will become reserved keywords in 3.7. [1][2] (On a side note, I just ran Python 3.5.1 on Windows with -Wd, and neither async nor await print a DeprecationWarning or PendingDeprecationWarning when used as a name. Should they?) [1] https://www.python.org/dev/peps/pep-0492/#deprecation-plans [2] https://docs.python.org/3/whatsnew/3.5.html#new-keywords From bruce at leban.us Mon May 2 22:49:54 2016 From: bruce at leban.us (Bruce Leban) Date: Mon, 2 May 2016 19:49:54 -0700 Subject: [Python-ideas] Object grabbing In-Reply-To: References: <57266698.7020602@bign.nl> Message-ID: On Mon, May 2, 2016 at 8:21 AM, Random832 wrote: > On Mon, May 2, 2016, at 10:35, Bruce Leban wrote: > > Using a single letter (or short) variable name works well most of the > > time > > but has one problem: the variable can leak. It's easy to forget to write > > the del statement. Imagine a block statement that had the effect of > > deleting any variables initialized in the block. > > > Should it delete the variables, or should it be a new scope? They're > subtly different. > > If the former, how do you differentiate a variable initialized in the > block from a variable assigned in the block? What if you really *do* > want to keep one of them? (The answer in both cases if it's a new scope > would be to use "nonlocal") Fair question. It's not a new scope because it's too clumsy to have to declare every variable outside the block nonlocal. If a variable X is not local to the enclosing scope (or declared global), then it's local to the block. I don't think you can do this at runtime; I think it has to be determined at compile time. Consider: def foo(i): if i: x = 1 with: x = 2 y = 3 This should not del x regardless of the value of i. In the following case the variable in the block affects the outer scope. Consider: def foo(i): with: y = 3 return y Treating this as def foo(i): y = 3 del y return y has the right result (unbound error). Without the block return y would be a reference to a global variable. On Mon, May 2, 2016 at 5:06 PM, Eric Snow wrote: > On Mon, May 2, 2016 at 8:35 AM, Bruce Leban wrote: > > Using a single letter (or short) variable name works well most of the > time > > > > > x = foo() > > with: > > y = x + 1 > > x = y + 2 > > > > Hmm. That reminds me of the "given" syntax (PEP 3150: > https://www.python.org/dev/peps/pep-3150/). This is like allowing > blocks of code to be treated as first-order, a la Ruby. Then again, > functions give us just about all we need, and the "given" syntax > basically gives us multi-line lambdas. :) Perhaps it's time to dust > off that proposal. > There is a difference. "given" or whatever it's called requires the variables to be declared and initialized at the top, while this would automatically delete any references created in the block. On Mon, May 2, 2016 at 8:31 AM, Guido van Rossum wrote: > The choice of keyword is not entirely arbitrary; if we can't come up with > a decent keyword the feature is dead. But the introduction would have to > include a `from __future__ import ` statement anyway for at > least one, maybe two release cycles (we did this with the `with` statement > itself, around 2.4/2.5). So Django will have plenty of time to change. > > Note that the "auto-deleting block" (for want of a better descriptive name) could be introduced with a contextual keyword. Currently : is not valid syntax so any identifier could be used for these blocks without requiring it to be recognized as a keyword in other contexts. --- Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Tue May 3 03:05:42 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 3 May 2016 09:05:42 +0200 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: Message-ID: <57284DC6.9000705@gmail.com> With Python 3.5, you can unpack in dictionaries: d = {**foo, **bar} It's now the defacto way of merging 2 dicts, and it requires that you can use twice the same key for ovious reasons. So even if you have good points, it's not possible to do this. Le 02/05/2016 23:36, Luigi Semenzato a ?crit : > Hello and sorry if this is an old and over-discussed topic. I could > not find such discussions here. > > This was discussed and rejected at https://bugs.python.org/issue16385, > but I am not convinced that the discussion presented all arguments > properly. I tried reopening the bug at > http://bugs.python.org/issue26910, but was told I should discuss it > here instead. > > The original problem description: > > lives_in = { 'lion': ['Africa', 'America'], > 'parrot': ['Europe'], > #... 100+ more rows here > 'lion': ['Europe'], > #... 100+ more rows here > } > > The above constructor overwrites the first 'lion' entry silently, > often causing unexpected behavior. > > These are the arguments presented in favor of the rejection, followed > by my rebuttals. > > 1. "An error is out of the question for compatibility reasons". No > real rebuttal here, except I wonder if exceptions are ever made and > under what circumstances. I should point out that a warning may also > create incompatibilities. > > 2. "There are ways to rewrite this as a loop on a list". Yes of > course, but the entire point of the dictionary literal is to offer a > concise and convenient notation for entering a dictionary as program > text. > > 3. "Being able to re-write keys is fundamental to Python dicts and why > they can be used for Python's mutable namespaces". This is fine but > it applies to the data structure, not the literal constructor. > > 4. "A code generator could depend on being able to write duplicate > keys without having to go back and erase previous output". Yes, but > it seems wrong to facilitate automated code generation at the expense > of human code writing. For hand-written code, I claim that in most > (all?) cases it would be preferable to have the compiler detect key > duplications. It is easier for an automated code generator to check > for duplications than it is for a human. > > 5. "There is already pylint". True, but it forces an additional step. > > For context, someone ran into this problem in my team at Google (we > fixed it using pylint). I haven't seen any valid reason (in the bug > or elsewhere) in favor of these constructor semantics. From the > discussions I have seen, it seems just an oversight in the > implementation/specification of dictionary literals. I'd be happy to > hear stronger reasoning in favor of the status quo. > > (additional search keywords: dict constructor, dict literal) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From desmoulinmichel at gmail.com Tue May 3 03:05:42 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 3 May 2016 09:05:42 +0200 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: Message-ID: <57284DC6.4020007@gmail.com> With Python 3.5, you can unpack in dictionaries: d = {**foo, **bar} It's now the defacto way of merging 2 dicts, and it requires that you can use twice the same key for ovious reasons. So even if you have good points, it's not possible to do this. Le 02/05/2016 23:36, Luigi Semenzato a ?crit : > Hello and sorry if this is an old and over-discussed topic. I could > not find such discussions here. > > This was discussed and rejected at https://bugs.python.org/issue16385, > but I am not convinced that the discussion presented all arguments > properly. I tried reopening the bug at > http://bugs.python.org/issue26910, but was told I should discuss it > here instead. > > The original problem description: > > lives_in = { 'lion': ['Africa', 'America'], > 'parrot': ['Europe'], > #... 100+ more rows here > 'lion': ['Europe'], > #... 100+ more rows here > } > > The above constructor overwrites the first 'lion' entry silently, > often causing unexpected behavior. > > These are the arguments presented in favor of the rejection, followed > by my rebuttals. > > 1. "An error is out of the question for compatibility reasons". No > real rebuttal here, except I wonder if exceptions are ever made and > under what circumstances. I should point out that a warning may also > create incompatibilities. > > 2. "There are ways to rewrite this as a loop on a list". Yes of > course, but the entire point of the dictionary literal is to offer a > concise and convenient notation for entering a dictionary as program > text. > > 3. "Being able to re-write keys is fundamental to Python dicts and why > they can be used for Python's mutable namespaces". This is fine but > it applies to the data structure, not the literal constructor. > > 4. "A code generator could depend on being able to write duplicate > keys without having to go back and erase previous output". Yes, but > it seems wrong to facilitate automated code generation at the expense > of human code writing. For hand-written code, I claim that in most > (all?) cases it would be preferable to have the compiler detect key > duplications. It is easier for an automated code generator to check > for duplications than it is for a human. > > 5. "There is already pylint". True, but it forces an additional step. > > For context, someone ran into this problem in my team at Google (we > fixed it using pylint). I haven't seen any valid reason (in the bug > or elsewhere) in favor of these constructor semantics. From the > discussions I have seen, it seems just an oversight in the > implementation/specification of dictionary literals. I'd be happy to > hear stronger reasoning in favor of the status quo. > > (additional search keywords: dict constructor, dict literal) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From greg at krypto.org Tue May 3 03:08:38 2016 From: greg at krypto.org (Gregory P. Smith) Date: Tue, 03 May 2016 07:08:38 +0000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <57284DC6.9000705@gmail.com> References: <57284DC6.9000705@gmail.com> Message-ID: It'd still be possible to do it for literal listed keys only if someone wanted to figure out how to do it at parsing time rather than runtime. On Tue, May 3, 2016 at 12:06 AM Michel Desmoulin wrote: > With Python 3.5, you can unpack in dictionaries: > > d = {**foo, **bar} > > It's now the defacto way of merging 2 dicts, and it requires that you > can use twice the same key for ovious reasons. > > So even if you have good points, it's not possible to do this. > > Le 02/05/2016 23:36, Luigi Semenzato a ?crit : > > Hello and sorry if this is an old and over-discussed topic. I could > > not find such discussions here. > > > > This was discussed and rejected at https://bugs.python.org/issue16385, > > but I am not convinced that the discussion presented all arguments > > properly. I tried reopening the bug at > > http://bugs.python.org/issue26910, but was told I should discuss it > > here instead. > > > > The original problem description: > > > > lives_in = { 'lion': ['Africa', 'America'], > > 'parrot': ['Europe'], > > #... 100+ more rows here > > 'lion': ['Europe'], > > #... 100+ more rows here > > } > > > > The above constructor overwrites the first 'lion' entry silently, > > often causing unexpected behavior. > > > > These are the arguments presented in favor of the rejection, followed > > by my rebuttals. > > > > 1. "An error is out of the question for compatibility reasons". No > > real rebuttal here, except I wonder if exceptions are ever made and > > under what circumstances. I should point out that a warning may also > > create incompatibilities. > > > > 2. "There are ways to rewrite this as a loop on a list". Yes of > > course, but the entire point of the dictionary literal is to offer a > > concise and convenient notation for entering a dictionary as program > > text. > > > > 3. "Being able to re-write keys is fundamental to Python dicts and why > > they can be used for Python's mutable namespaces". This is fine but > > it applies to the data structure, not the literal constructor. > > > > 4. "A code generator could depend on being able to write duplicate > > keys without having to go back and erase previous output". Yes, but > > it seems wrong to facilitate automated code generation at the expense > > of human code writing. For hand-written code, I claim that in most > > (all?) cases it would be preferable to have the compiler detect key > > duplications. It is easier for an automated code generator to check > > for duplications than it is for a human. > > > > 5. "There is already pylint". True, but it forces an additional step. > > > > For context, someone ran into this problem in my team at Google (we > > fixed it using pylint). I haven't seen any valid reason (in the bug > > or elsewhere) in favor of these constructor semantics. From the > > discussions I have seen, it seems just an oversight in the > > implementation/specification of dictionary literals. I'd be happy to > > hear stronger reasoning in favor of the status quo. > > > > (additional search keywords: dict constructor, dict literal) > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Tue May 3 03:22:35 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 3 May 2016 09:22:35 +0200 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57284DC6.9000705@gmail.com> Message-ID: <572851BB.70705@gmail.com> That would introduce a very weird special case. All those would work: foo = 1 bar = "1" {1: True, 2 - 1: True, foo: True, int(bar): True, int("1"): True, **{1: True}} But not this: {1: True, 1: True} It would be very confusing. Le 03/05/2016 09:08, Gregory P. Smith a ?crit : > It'd still be possible to do it for literal listed keys only if someone > wanted to figure out how to do it at parsing time rather than runtime. > > On Tue, May 3, 2016 at 12:06 AM Michel Desmoulin > > wrote: > > With Python 3.5, you can unpack in dictionaries: > > d = {**foo, **bar} > > It's now the defacto way of merging 2 dicts, and it requires that you > can use twice the same key for ovious reasons. > > So even if you have good points, it's not possible to do this. > > Le 02/05/2016 23:36, Luigi Semenzato a ?crit : > > Hello and sorry if this is an old and over-discussed topic. I could > > not find such discussions here. > > > > This was discussed and rejected at https://bugs.python.org/issue16385, > > but I am not convinced that the discussion presented all arguments > > properly. I tried reopening the bug at > > http://bugs.python.org/issue26910, but was told I should discuss it > > here instead. > > > > The original problem description: > > > > lives_in = { 'lion': ['Africa', 'America'], > > 'parrot': ['Europe'], > > #... 100+ more rows here > > 'lion': ['Europe'], > > #... 100+ more rows here > > } > > > > The above constructor overwrites the first 'lion' entry silently, > > often causing unexpected behavior. > > > > These are the arguments presented in favor of the rejection, followed > > by my rebuttals. > > > > 1. "An error is out of the question for compatibility reasons". No > > real rebuttal here, except I wonder if exceptions are ever made and > > under what circumstances. I should point out that a warning may also > > create incompatibilities. > > > > 2. "There are ways to rewrite this as a loop on a list". Yes of > > course, but the entire point of the dictionary literal is to offer a > > concise and convenient notation for entering a dictionary as program > > text. > > > > 3. "Being able to re-write keys is fundamental to Python dicts and why > > they can be used for Python's mutable namespaces". This is fine but > > it applies to the data structure, not the literal constructor. > > > > 4. "A code generator could depend on being able to write duplicate > > keys without having to go back and erase previous output". Yes, but > > it seems wrong to facilitate automated code generation at the expense > > of human code writing. For hand-written code, I claim that in most > > (all?) cases it would be preferable to have the compiler detect key > > duplications. It is easier for an automated code generator to check > > for duplications than it is for a human. > > > > 5. "There is already pylint". True, but it forces an additional step. > > > > For context, someone ran into this problem in my team at Google (we > > fixed it using pylint). I haven't seen any valid reason (in the bug > > or elsewhere) in favor of these constructor semantics. From the > > discussions I have seen, it seems just an oversight in the > > implementation/specification of dictionary literals. I'd be happy to > > hear stronger reasoning in favor of the status quo. > > > > (additional search keywords: dict constructor, dict literal) > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From desmoulinmichel at gmail.com Tue May 3 03:22:35 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 3 May 2016 09:22:35 +0200 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57284DC6.9000705@gmail.com> Message-ID: <572851BB.8020801@gmail.com> That would introduce a very weird special case. All those would work: foo = 1 bar = "1" {1: True, 2 - 1: True, foo: True, int(bar): True, int("1"): True, **{1: True}} But not this: {1: True, 1: True} It would be very confusing. Le 03/05/2016 09:08, Gregory P. Smith a ?crit : > It'd still be possible to do it for literal listed keys only if someone > wanted to figure out how to do it at parsing time rather than runtime. > > On Tue, May 3, 2016 at 12:06 AM Michel Desmoulin > > wrote: > > With Python 3.5, you can unpack in dictionaries: > > d = {**foo, **bar} > > It's now the defacto way of merging 2 dicts, and it requires that you > can use twice the same key for ovious reasons. > > So even if you have good points, it's not possible to do this. > > Le 02/05/2016 23:36, Luigi Semenzato a ?crit : > > Hello and sorry if this is an old and over-discussed topic. I could > > not find such discussions here. > > > > This was discussed and rejected at https://bugs.python.org/issue16385, > > but I am not convinced that the discussion presented all arguments > > properly. I tried reopening the bug at > > http://bugs.python.org/issue26910, but was told I should discuss it > > here instead. > > > > The original problem description: > > > > lives_in = { 'lion': ['Africa', 'America'], > > 'parrot': ['Europe'], > > #... 100+ more rows here > > 'lion': ['Europe'], > > #... 100+ more rows here > > } > > > > The above constructor overwrites the first 'lion' entry silently, > > often causing unexpected behavior. > > > > These are the arguments presented in favor of the rejection, followed > > by my rebuttals. > > > > 1. "An error is out of the question for compatibility reasons". No > > real rebuttal here, except I wonder if exceptions are ever made and > > under what circumstances. I should point out that a warning may also > > create incompatibilities. > > > > 2. "There are ways to rewrite this as a loop on a list". Yes of > > course, but the entire point of the dictionary literal is to offer a > > concise and convenient notation for entering a dictionary as program > > text. > > > > 3. "Being able to re-write keys is fundamental to Python dicts and why > > they can be used for Python's mutable namespaces". This is fine but > > it applies to the data structure, not the literal constructor. > > > > 4. "A code generator could depend on being able to write duplicate > > keys without having to go back and erase previous output". Yes, but > > it seems wrong to facilitate automated code generation at the expense > > of human code writing. For hand-written code, I claim that in most > > (all?) cases it would be preferable to have the compiler detect key > > duplications. It is easier for an automated code generator to check > > for duplications than it is for a human. > > > > 5. "There is already pylint". True, but it forces an additional step. > > > > For context, someone ran into this problem in my team at Google (we > > fixed it using pylint). I haven't seen any valid reason (in the bug > > or elsewhere) in favor of these constructor semantics. From the > > discussions I have seen, it seems just an oversight in the > > implementation/specification of dictionary literals. I'd be happy to > > hear stronger reasoning in favor of the status quo. > > > > (additional search keywords: dict constructor, dict literal) > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From robert at bign.nl Tue May 3 03:21:24 2016 From: robert at bign.nl (Robert van Geel) Date: Tue, 3 May 2016 09:21:24 +0200 Subject: [Python-ideas] Object grabbing In-Reply-To: References: Message-ID: <57285174.2000407@bign.nl> summarizing: 1- the with statement should not be used for this 2- I overrated the code optimization of this change, optimization is not sufficiently large and nothing that can't be done without this syntax 3- picking a keyword would be non-trivial, 'using' has drawbacks, 'in' might be a candidate (frankly I like it) 4- the benefits of .dotted shorthand syntax are disputed, some think it's pythonic and readable, others don't think it is 5 - Pascal style has been discussed in the past and has its own faq explaining rejection. So anyway a dot would be needed, this would also take away possible ambiguity between locals and object properties About 4, the argument that full variable writing is more explicit, hence Pythonic: I think clinging to 20 same-indentation level references to 'obj.' is explicit to the point of sillyness. We also dont use locals.obj.x because the localness of a variable is also present without explicity so a bit of pragmatism could have a voice. People who think .dotted syntax is less readable could still use the familiar syntax. That would lead to two ways to do the same thing but on the same level as the type of decision as to whether to use a lambda or a full function. In the end implementation of this proposal would offer syntactical sugar without added functionality, especially given point 2 above. That would make the feature as desirable as the number of happy eyebrows when looking at code like this: in disussion: .arguments = [a, b, c] .argue() .summarize() x = .eastheticfactor .decision = .calculate(x) On 5/3/2016 4:50 AM, python-ideas-request at python.org wrote: > Send Python-ideas mailing list submissions to > python-ideas at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/python-ideas > or, via email, send a message with subject or body 'help' to > python-ideas-request at python.org > > You can reach the person managing the list at > python-ideas-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-ideas digest..." > > > Today's Topics: > > 1. Re: Object grabbing (Guido van Rossum) > 2. Re: Object grabbing (Steven D'Aprano) > 3. Re: Object grabbing (Jonathan Goble) > 4. Re: Object grabbing (Bruce Leban) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 2 May 2016 18:47:00 -0700 > From: Guido van Rossum > To: Joshua Morton > Cc: "python-ideas at python.org" > Subject: Re: [Python-ideas] Object grabbing > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > On Mon, May 2, 2016 at 6:43 PM, Joshua Morton > wrote: > >> Which actually makes the specific syntax of a bare `with` a strong -1 now, >> since mixing it with normal context managers is incredibly ambiguous. >> > I'm glad you eventually reached this conclusion. > From me+python at ixokai.io Tue May 3 04:11:09 2016 From: me+python at ixokai.io (Stephen Hansen) Date: Tue, 03 May 2016 01:11:09 -0700 Subject: [Python-ideas] Object grabbing In-Reply-To: <57285174.2000407@bign.nl> References: <57285174.2000407@bign.nl> Message-ID: <1462263069.1506620.596457369.38FC4EB0@webmail.messagingengine.com> On Tue, May 3, 2016, at 12:21 AM, Robert van Geel wrote: > summarizing: >??? 1- the with statement should not be used for this with means something now. It shouldn't have two confusing meanings. All this discussion keeps using with when I hope that's a nonstarter. If people can get behind 'using' or 'in', I don't know, but 'with' should be absolutely out. With has a meaning in Python, a meaning which doesn't map readily into this space (where 'import x as y' and 'except Ex as y' do map readily, as its about renaming/aliasing) >??? 4- the benefits of .dotted shorthand syntax are disputed, some think > it's pythonic and readable, others don't think it is To call it disputed undersells the claim. I'd call it horribly unreadable. A dot is just shy of invisible on its own, a leading dot evaporates besides the tabbed whitespace. No, no, no. >??? 5 - Pascal style has been discussed in the past and has its own faq > explaining rejection. So anyway a dot would be needed, this would also > take away possible ambiguity between locals and object properties > > About 4, the argument that full variable writing is more explicit, hence > Pythonic: I think clinging to 20 same-indentation level references to > 'obj.' is explicit to the point of sillyness. We also dont use > locals.obj.x because the localness of a variable is also present without > explicity so a bit of pragmatism could have a voice. People who think > .dotted syntax is less readable could still use the familiar syntax. Again: no, no, no. Readability is more important then writability. "People who think .dotted syntax is less readable" need to be answered with a *serious* gain to override their objections. What does: using thing["this"]: ??? .that = "okay" ??? .other = "right" gain over t = thing["this"] t.that = "okay" t.other = "right" This feature is pure syntax sugar over local variables which are just fine as they are. -- Stephen Hansen ? m e @ i x o k a i? . i o From rob.cliffe at btinternet.com Tue May 3 04:00:56 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Tue, 3 May 2016 09:00:56 +0100 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <572851BB.8020801@gmail.com> References: <57284DC6.9000705@gmail.com> <572851BB.8020801@gmail.com> Message-ID: <57285AB8.1040300@btinternet.com> On 03/05/2016 08:22, Michel Desmoulin wrote: > That would introduce a very weird special case. All those would work: > > foo = 1 > bar = "1" > > {1: True, 2 - 1: True, foo: True, int(bar): True, int("1"): True, **{1: > True}} > > But not this: > > {1: True, 1: True} Perfectly true. > > It would be very confusing. It could conceivably confuse someone, but the example quoted by the OP with all literal keys is realistic and yours is contrived. There is practical benefit in spotting duplicate literal keys, if it can be done with either an error or a warning (whether at compile time or run time). There is even a kind of analogy: dict(lion=1, lion=2) SyntaxError: keyword argument repeated Is that also confusing? From guido at python.org Tue May 3 10:44:37 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 3 May 2016 07:44:37 -0700 Subject: [Python-ideas] Object grabbing In-Reply-To: <1462263069.1506620.596457369.38FC4EB0@webmail.messagingengine.com> References: <57285174.2000407@bign.nl> <1462263069.1506620.596457369.38FC4EB0@webmail.messagingengine.com> Message-ID: Thanks Robert for a crisp summary of the most reasonable proposal and its ins and outs; I think the 'in' keyword and requiring leading dots are the best version of all versions considered. Thanks Stephen for concisely formulating the strongest argument against: it gains too little over the alternative using existing syntax, and in fact the alternative may be considered more readable (because more explicit). I would now like to close the discussion with a rejection. This email can be framed to celebrate that. --Guido On Tue, May 3, 2016 at 1:11 AM, Stephen Hansen wrote: > > On Tue, May 3, 2016, at 12:21 AM, Robert van Geel wrote: > > summarizing: > > 1- the with statement should not be used for this > > with means something now. It shouldn't have two confusing meanings. All > this discussion keeps using with when I hope that's a nonstarter. > > If people can get behind 'using' or 'in', I don't know, but 'with' > should be absolutely out. With has a meaning in Python, a meaning which > doesn't map readily into this space (where 'import x as y' and 'except > Ex as y' do map readily, as its about renaming/aliasing) > > > 4- the benefits of .dotted shorthand syntax are disputed, some think > > it's pythonic and readable, others don't think it is > > To call it disputed undersells the claim. I'd call it horribly > unreadable. A dot is just shy of invisible on its own, a leading dot > evaporates besides the tabbed whitespace. No, no, no. > > > 5 - Pascal style has been discussed in the past and has its own faq > > explaining rejection. So anyway a dot would be needed, this would also > > take away possible ambiguity between locals and object properties > > > > About 4, the argument that full variable writing is more explicit, hence > > Pythonic: I think clinging to 20 same-indentation level references to > > 'obj.' is explicit to the point of sillyness. We also dont use > > locals.obj.x because the localness of a variable is also present without > > explicity so a bit of pragmatism could have a voice. People who think > > .dotted syntax is less readable could still use the familiar syntax. > > Again: no, no, no. Readability is more important then writability. > "People who think .dotted syntax is less readable" need to be answered > with a *serious* gain to override their objections. What does: > > using thing["this"]: > .that = "okay" > .other = "right" > > gain over > > t = thing["this"] > t.that = "okay" > t.other = "right" > > This feature is pure syntax sugar over local variables which are just > fine as they are. > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From leewangzhong+python at gmail.com Tue May 3 11:36:34 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Tue, 3 May 2016 11:36:34 -0400 Subject: [Python-ideas] Specify an alternative exception for "assert" In-Reply-To: <1462202269.1077984.595681529.06779B74@webmail.messagingengine.com> References: <1462202269.1077984.595681529.06779B74@webmail.messagingengine.com> Message-ID: On May 2, 2016 11:18 AM, "Random832" wrote: > > On Mon, May 2, 2016, at 11:14, Guido van Rossum wrote: > > On Mon, May 2, 2016 at 8:01 AM, Ryan Gonzalez wrote: > > > > > Other than the fact that this would completely fail when run with -O... > > > > > But maybe that's fine, or intended. > > > > I can see a fair number of uses for this, including subclasses of > > AssertionError. > > I think this would be an attractive nuisance, and if it's implemented at > all it should forbid types that are *not* subclasses of AssertionError > in order to mitigate that. What about wrapping the error in an AssertionError? The error being raised is really an assertion error, since it came from an assert. A wrapped ValueError isn't going to be caught by an `except ValueError:` clause. That might be a good thing: failed asserts usually shouldn't be caught, since they indicate a bug. Even if they are caught (e.g. for logging), they shouldn't be caught by something expecting a non-AssertionError. -------------- next part -------------- An HTML attachment was scrubbed... URL: From luigi at semenzato.com Tue May 3 12:47:07 2016 From: luigi at semenzato.com (Luigi Semenzato) Date: Tue, 3 May 2016 09:47:07 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <57285AB8.1040300@btinternet.com> References: <57284DC6.9000705@gmail.com> <572851BB.8020801@gmail.com> <57285AB8.1040300@btinternet.com> Message-ID: I am still not seeing good objections to not adding at least a warning. >> On Tue, May 3, 2016 at 12:06 AM Michel Desmoulin >> > wrote: >> >> With Python 3.5, you can unpack in dictionaries: >> >> d = {**foo, **bar} and similarly: > foo = 1 > bar = "1" > > {1: True, 2 - 1: True, foo: True, int(bar): True, int("1"): True, **{1: > True}} > > But not this: > > {1: True, 1: True} Probably I didn't make it sufficiently clear in my OP, but this is only for duplicate literal keys in the same constructor, which would cover the vast majority of user errors. A well-known precedent is the divide-by-zero warning in C. The GNU compiler produces a warning when, after constant folding, an expression contains a division by zero. For instance: test.c:9:12: warning: division by zero [-Wdiv-by-zero] int z = 1/0; and this: test.c:9:12: warning: division by zero [-Wdiv-by-zero] int z = 1/(1 - 1); but this code doesn't produce a warning: const int x = 0; int z = 1/x; For the dict constructor, I don't think that the constant folding is even particularly useful, although it wouldn't hurt. Implementation-wise, checking for duplicate literals while parsing is just a matter of maintaining a hash table for that scope. This is already done for identifiers, and probably heavily optimized, so duplicate detection wouldn't add much code or be particularly expensive. From michael.selik at gmail.com Tue May 3 15:21:01 2016 From: michael.selik at gmail.com (Michael Selik) Date: Tue, 03 May 2016 19:21:01 +0000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: Message-ID: On Mon, May 2, 2016 at 5:36 PM Luigi Semenzato wrote: > For context, someone ran into this problem in my team at Google (we > fixed it using pylint). I haven't seen any valid reason (in the bug > or elsewhere) in favor of these constructor semantics. From the > discussions I have seen, it seems just an oversight in the > implementation/specification of dictionary literals. I'd be happy to > hear stronger reasoning in favor of the status quo. > Do you feel that "prefer status quo" is not strong reasoning? http://www.curiousefficiency.org/posts/2011/02/status-quo-wins-stalemate.html Your word choice of "[no] valid reason" is interesting. In the bug tracker discussion, folks argue that the problem is easily solved with code style standards and static analyzers. I thought that was valid. Rob Cliffe mentioned the fact that repeated keyword arguments causes a syntax error. I see the parallel here, except that keyword arguments must be valid identifiers. Literal dicts syntactically may have any expression as the key. Creating a special case for the parser to enforce uniqueness of number and string literals as keys seems more trouble than its worth. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue May 3 15:59:36 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 03 May 2016 12:59:36 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: Message-ID: <57290328.8020408@stoneleaf.us> On 05/03/2016 12:21 PM, Michael Selik wrote: > On Mon, May 2, 2016 at 5:36 PM Luigi Semenzato wrote: >> >> For context, someone ran into this problem in my team at Google (we >> fixed it using pylint). I haven't seen any valid reason (in the bug >> or elsewhere) in favor of these constructor semantics. From the >> discussions I have seen, it seems just an oversight in the >> implementation/specification of dictionary literals. I'd be happy to >> hear stronger reasoning in favor of the status quo. > > Do you feel that "prefer status quo" is not strong reasoning? > http://www.curiousefficiency.org/posts/2011/02/status-quo-wins-stalemate.html It is not strong enough to prevent every good idea, or Python would be a static language (as in, no more changes except maybe bug fixes). > Rob Cliffe mentioned the fact that repeated keyword arguments causes a > syntax error. I see the parallel here, except that keyword arguments > must be valid identifiers. Literal dicts syntactically may have any > expression as the key. Which seems irrelevant to your argument: a duplicate key is a duplicate key whether it's 123 or 'xyz'. > Creating a special case for the parser to enforce > uniqueness of number and string literals as keys I'm pretty sure the OP would be happy with uniqueness of keys, whether those keys were string literals, numeric literals, or function objects. > seems more trouble than its worth. Maybe. That is what we are discussing. -- ~Ethan~ From pavol.lisy at gmail.com Tue May 3 16:28:11 2016 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Tue, 3 May 2016 22:28:11 +0200 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: Message-ID: 2016-05-03 21:21 GMT+02:00, Michael Selik : [...] > Rob Cliffe mentioned the fact that repeated keyword arguments causes a > syntax error. I see the parallel here, except that keyword arguments must > be valid identifiers. Literal dicts syntactically may have any expression > as the key. Creating a special case for the parser to enforce uniqueness of > number and string literals as keys seems more trouble than its worth. Just for curiosity: keys could be valid identifier, could be different (at least in one POV) and still cause Syntax error as keyword argument repeated. {'ij':1, '?':2} # this is dict with 2 keys and 2 different values dict(ij=1, ?=2) # SyntaxError: keyword argument repeated It is because parser is doing NFKC normalization for non-ascii identifiers - see https://www.python.org/dev/peps/pep-3131/ ('\u0069\u006a' == 'ij' != '?' == '\u0133') From michael.selik at gmail.com Tue May 3 16:43:53 2016 From: michael.selik at gmail.com (Michael Selik) Date: Tue, 03 May 2016 20:43:53 +0000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <57290328.8020408@stoneleaf.us> References: <57290328.8020408@stoneleaf.us> Message-ID: On Tue, May 3, 2016 at 4:00 PM Ethan Furman wrote: > On 05/03/2016 12:21 PM, Michael Selik wrote: > > On Mon, May 2, 2016 at 5:36 PM Luigi Semenzato wrote: > >> > >> For context, someone ran into this problem in my team at Google (we > >> fixed it using pylint). I haven't seen any valid reason (in the bug > >> or elsewhere) in favor of these constructor semantics. From the > >> discussions I have seen, it seems just an oversight in the > >> implementation/specification of dictionary literals. I'd be happy to > >> hear stronger reasoning in favor of the status quo. > > > > Do you feel that "prefer status quo" is not strong reasoning? > > > http://www.curiousefficiency.org/posts/2011/02/status-quo-wins-stalemate.html > > It is not strong enough to prevent every good idea, or Python would be a > static language (as in, no more changes except maybe bug fixes). > Of course. I'm no more saying "stop change" than you are saying "all change is good". Luigi, as with many who have ideas for new features, appears to be trying to shift the burden of proof. I think it's well established that the burden of proof (proving it's a good idea) rests on the one advocating change. > > Rob Cliffe mentioned the fact that repeated keyword arguments causes a > > syntax error. I see the parallel here, except that keyword arguments > > must be valid identifiers. Literal dicts syntactically may have any > > expression as the key. > > Which seems irrelevant to your argument: a duplicate key is a duplicate > key whether it's 123 or 'xyz'. > If an expression includes an impure function, then the duplication of assignment to that key may have a desirable side-effect. > > Creating a special case for the parser to enforce > > uniqueness of number and string literals as keys > > I'm pretty sure the OP would be happy with uniqueness of keys, whether > those keys were string literals, numeric literals, or function objects. > How would you handle an expression that evaluates differently for each call? For example: {random(): 0, random(): 1} > > seems more trouble than its worth. > > Maybe. That is what we are discussing. > Indeed. Let me flip the original request: I'd like to hear stronger arguments for change, please. I'd be particularly interested in hearing how often Pylint has caught this mistake. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue May 3 17:27:16 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 03 May 2016 14:27:16 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57290328.8020408@stoneleaf.us> Message-ID: <572917B4.9000007@stoneleaf.us> On 05/03/2016 01:43 PM, Michael Selik wrote: > On Tue, May 3, 2016 at 4:00 PM Ethan Furman wrote: >> Which seems irrelevant to your argument: a duplicate key is a duplicate >> key whether it's 123 or 'xyz'. > > If an expression includes an impure function, then the duplication of > assignment to that key may have a desirable side-effect. I'm willing to say that should be done with an existing dict, not in a literal. >> I'm pretty sure the OP would be happy with uniqueness of keys, whether >> those keys were string literals, numeric literals, or function objects. > > How would you handle an expression that evaluates differently for each > call? For example: > > {random(): 0, random(): 1} Easy: Don't Do That. ;) > Let me flip the original request: I'd like to hear stronger arguments for > change, please. I'd be particularly interested in hearing how often Pylint > has caught this mistake. Well, when this happened to me I spent a loooonnnnnngggg time figuring out what the problem is. One just doesn't expect duplicate keys to not raise: --> dict(one=1, one='uno') File "", line 1 SyntaxError: keyword argument repeated So I guess the current advice is: Don't use dict literals, use dict() instead. And that just seems silly. ;) -- ~Ethan~ From luigi at semenzato.com Tue May 3 17:29:10 2016 From: luigi at semenzato.com (Luigi Semenzato) Date: Tue, 3 May 2016 14:29:10 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57290328.8020408@stoneleaf.us> Message-ID: On Tue, May 3, 2016 at 1:43 PM, Michael Selik wrote: > On Tue, May 3, 2016 at 4:00 PM Ethan Furman wrote: >> >> On 05/03/2016 12:21 PM, Michael Selik wrote: >> > On Mon, May 2, 2016 at 5:36 PM Luigi Semenzato wrote: >> >> >> >> For context, someone ran into this problem in my team at Google (we >> >> fixed it using pylint). I haven't seen any valid reason (in the bug >> >> or elsewhere) in favor of these constructor semantics. From the >> >> discussions I have seen, it seems just an oversight in the >> >> implementation/specification of dictionary literals. I'd be happy to >> >> hear stronger reasoning in favor of the status quo. >> > >> > Do you feel that "prefer status quo" is not strong reasoning? Correct. In this particular case I think that the change is better. It is simple, helpful, and doesn't seem to have huge implications (if any) in terms of documentation. >> > http://www.curiousefficiency.org/posts/2011/02/status-quo-wins-stalemate.html >> >> It is not strong enough to prevent every good idea, or Python would be a >> static language (as in, no more changes except maybe bug fixes). Correct. > Of course. I'm no more saying "stop change" than you are saying "all change > is good". Luigi, as with many who have ideas for new features, appears to be > trying to shift the burden of proof. I think it's well established that the > burden of proof (proving it's a good idea) rests on the one advocating > change. Which gives me one more opportunity to present this case: A dictionary literal is a convenient input format for structured data, and is used as such in many cases, for instance instead of equivalent JSON, which requires an additional module, an additional parsing step, and learning an additional syntax (a lot of Python programmers don't know JSON). The convenience, however, is hampered by the duplicate key issue. Some of the strong points of Python are: 1. easy to learn; 2. does the right thing in most cases. It just seems to me that the right thing here would be to signal duplicate literal keys. >> > Rob Cliffe mentioned the fact that repeated keyword arguments causes a >> > syntax error. I see the parallel here, except that keyword arguments >> > must be valid identifiers. Literal dicts syntactically may have any >> > expression as the key. >> >> Which seems irrelevant to your argument: a duplicate key is a duplicate >> key whether it's 123 or 'xyz'. > > If an expression includes an impure function, then the duplication of > assignment to that key may have a desirable side-effect. Yes, this is absolutely true, but probably very rare, and arguably not a recommendable coding practice, therefore IMO it offers very little support for the status quo. >> > Creating a special case for the parser to enforce >> > uniqueness of number and string literals as keys >> >> I'm pretty sure the OP would be happy with uniqueness of keys, whether >> those keys were string literals, numeric literals, or function objects. > > How would you handle an expression that evaluates differently for each call? > For example: > > {random(): 0, random(): 1} We discussed this already. My request only applies to keys that are literals. (He's talking about function objects, not function calls.) >> >> > seems more trouble than its worth. >> >> Maybe. That is what we are discussing. > > Indeed. Let me flip the original request: I'd like to hear stronger > arguments for change, please. I'd be particularly interested in hearing how > often Pylint has caught this mistake. Pylint catches this mistake fine. But many Python programmers don't know about pylint either. For instance, a bunch of hardware engineers that wrote some python as a driver for a circuit simulator. Anyway, thank you for making these points. From steve at pearwood.info Tue May 3 19:51:21 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 4 May 2016 09:51:21 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: Message-ID: <20160503235120.GA12028@ando.pearwood.info> Hi Luigi, On Mon, May 02, 2016 at 02:36:35PM -0700, Luigi Semenzato wrote: [...] > lives_in = { 'lion': ['Africa', 'America'], > 'parrot': ['Europe'], > #... 100+ more rows here > 'lion': ['Europe'], > #... 100+ more rows here > } > > The above constructor overwrites the first 'lion' entry silently, > often causing unexpected behavior. [...] > For context, someone ran into this problem in my team at Google (we > fixed it using pylint). I haven't seen any valid reason (in the bug > or elsewhere) in favor of these constructor semantics. From the > discussions I have seen, it seems just an oversight in the > implementation/specification of dictionary literals. I'd be happy to > hear stronger reasoning in favor of the status quo. As far as I can see, you haven't specified in *detail* what change you wish to propose. It is difficult for me to judge your proposal when I don't know precisely what you are proposing. Should duplicate keys be a SyntaxError at compile time, or a TypeError at runtime? Or something else? What counts as "duplicate keys"? I presume that you mean that two keys count as duplicate if they hash to the same value, and are equal. But you keep mentioning "literals" -- does this mean you care more about whether they look the same rather than are the same? # duplicate literals, forbidden d = {100: 1, 100: 2} # duplicate non-literals, allowed d = {100: 1, len("ab")*50: 2} You keep mentioning "dictionary literal", but there actually is no such thing in Python. I think you mean a dict display. (Don't worry, I make the same mistake.) But the point is, a "dict literal" (display) can contain keys which are not themselves literals, as above. Those keys can have arbitrarily complex semantics, including side-effects. What do you expect to happen? -- Steve From leewangzhong+python at gmail.com Tue May 3 20:25:02 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Tue, 3 May 2016 20:25:02 -0400 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: Message-ID: On May 3, 2016 3:21 PM, "Michael Selik" wrote: > > On Mon, May 2, 2016 at 5:36 PM Luigi Semenzato wrote: >> >> For context, someone ran into this problem in my team at Google (we >> fixed it using pylint). I haven't seen any valid reason (in the bug >> or elsewhere) in favor of these constructor semantics. From the >> discussions I have seen, it seems just an oversight in the >> implementation/specification of dictionary literals. I'd be happy to >> hear stronger reasoning in favor of the status quo. > > > Do you feel that "prefer status quo" is not strong reasoning? > http://www.curiousefficiency.org/posts/2011/02/status-quo-wins-stalemate.html > > Your word choice of "[no] valid reason" is interesting. In the bug tracker discussion, folks argue that the problem is easily solved with code style standards and static analyzers. I thought that was valid. "X is detectable", whether by tools, or by magic imps yelling at you when you do it, is not an argument for X. When asking for a useful purpose for the current semantics, "prefer the status quo" is not strong enough. "Errors should never pass silently," and there is an argument on the table that duplicate spelled-out keys are probably written in error. He isn't shifting the burden, he's putting the ball in your court. > Literal dicts syntactically may have any expression as the key. Creating a special case for the parser to enforce uniqueness of number and string literals as keys seems more trouble than its worth. That is an argument. Literal keys, or a single name repeated in the key list, might be an error, but you can't say the same about expressions. If you only error on literal keys, then there is an inconsistency between literals and expressions. By the way, unless [[the name is guaranteed not to be rebound in a closure called within the dict display]], multiple items with the same variable name as key isn't necessarily an error: def f(): x=1 def g(): nonlocal x x += 1 return x return {x: g(), x: g(), x: g()} print(f()) # {2: 2, 3: 3, 4: 4} "Don't do that" doesn't say what the compiler behavior should be: should it analyze for nonlocal closures, or should it only check literals? -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue May 3 20:23:00 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 4 May 2016 10:23:00 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <572917B4.9000007@stoneleaf.us> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> Message-ID: <20160504002300.GB12028@ando.pearwood.info> On Tue, May 03, 2016 at 02:27:16PM -0700, Ethan Furman wrote: > On 05/03/2016 01:43 PM, Michael Selik wrote: > >On Tue, May 3, 2016 at 4:00 PM Ethan Furman wrote: > > >>Which seems irrelevant to your argument: a duplicate key is a duplicate > >>key whether it's 123 or 'xyz'. > > > >If an expression includes an impure function, then the duplication of > >assignment to that key may have a desirable side-effect. > > I'm willing to say that should be done with an existing dict, not in a > literal. But are you willing to say that the compiler should enforce that stylistic judgement? > >How would you handle an expression that evaluates differently for each > >call? For example: > > > > {random(): 0, random(): 1} > > Easy: Don't Do That. ;) I see your wink, so I presume you're not actually suggesting that the compiler (1) prohibit all function calls in dict displays, or (2) hard-code the function name "random" in a black list. So what are you actually suggesting? Michael is raising a good point here. If you don't like random as an example, how about: d = {spam(a): 'a', spam(b): 'BB', spam(c): 'Ccc'} I'm intentionally not giving you the values of a, b or c, or telling you what spam() returns. Now you have the same information available to you as the compiler has at compile time. What do you intend to do? It's one thing to say "duplicate keys should be prohibited", and another to come up with a detailed explanation of what precisely should happen. > >Let me flip the original request: I'd like to hear stronger arguments for > > change, please. I'd be particularly interested in hearing how often > > Pylint has caught this mistake. > > Well, when this happened to me I spent a loooonnnnnngggg time figuring > out what the problem is. Okay, but how often does this happen? Daily? Weekly? Once per career? What's a loooonnnnnngggg time? Twenty minutes? Twenty man-weeks? > One just doesn't expect duplicate keys to not raise: > > --> dict(one=1, one='uno') > File "", line 1 > SyntaxError: keyword argument repeated Huh, I had completely forgotten that duplicate keyword arguments raise a syntax error. I thought you got a runtime TypeError. There's a difference though. Keyword argument *names* must be identifiers, not expressions, and duplicates can be recognised by the compiler at compile-time: py> def spam(**kw): pass ... py> spam(eggs=print("side-effect")) side-effect py> spam(eggs=print("side-effect"), eggs=print('another side-effect')) File "", line 1 SyntaxError: keyword argument repeated Duplicate keys are not, and in general can't be: py> d = {'eggs': print("side-effect"), ... 'eggs': print('another side-effect')} side-effect another side-effect py> d {'eggs': None} If you think of the dict constructor as something like: for key,item in initial_values: self[key] = item then the current behaviour is perfectly valid, and the advice is, if you don't want duplicate keys, don't use them in the first place. If you think of it as: for key,item in initial_values: if key in self: raise TypeError('duplicate key') else: self[key] = item then you have to deal with the fact that you might only notice a duplicate after mutating your dict, which may include side-effects. -- Steve From luigi at semenzato.com Tue May 3 20:46:33 2016 From: luigi at semenzato.com (Luigi Semenzato) Date: Tue, 3 May 2016 17:46:33 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <20160503235120.GA12028@ando.pearwood.info> References: <20160503235120.GA12028@ando.pearwood.info> Message-ID: Hi, On Tue, May 3, 2016 at 4:51 PM, Steven D'Aprano wrote: > Hi Luigi, > > On Mon, May 02, 2016 at 02:36:35PM -0700, Luigi Semenzato wrote: > > [...] >> lives_in = { 'lion': ['Africa', 'America'], >> 'parrot': ['Europe'], >> #... 100+ more rows here >> 'lion': ['Europe'], >> #... 100+ more rows here >> } >> >> The above constructor overwrites the first 'lion' entry silently, >> often causing unexpected behavior. > [...] >> For context, someone ran into this problem in my team at Google (we >> fixed it using pylint). I haven't seen any valid reason (in the bug >> or elsewhere) in favor of these constructor semantics. From the >> discussions I have seen, it seems just an oversight in the >> implementation/specification of dictionary literals. I'd be happy to >> hear stronger reasoning in favor of the status quo. > > As far as I can see, you haven't specified in *detail* what change you > wish to propose. It is difficult for me to judge your proposal when I > don't know precisely what you are proposing. Yes, and I apologize, because on further reflection I don't even know what is possible within the model of the Python interpreter. > Should duplicate keys be a SyntaxError at compile time, or a TypeError > at runtime? Or something else? Is there such a thing as a SyntaxWarning? From my perspective it would be fine to make it a SyntaxError, but I am not sure it would be overall a good choice for legacy code (i.e. as an error it might break old code, and I don't know how many other things a new language specification is going to break). It could also be a run-time error, but it might be nicer to detect it earlier. Maybe both. > What counts as "duplicate keys"? I presume that you mean that two keys > count as duplicate if they hash to the same value, and are equal. But > you keep mentioning "literals" -- does this mean you care more about > whether they look the same rather than are the same? Correct. The errors that I am guessing matter the most are those for which folks copy-paste a key-value pair, where the key is a literal string, intending to change the key, and then forget to change it. > # duplicate literals, forbidden > d = {100: 1, 100: 2} Correct. Possibly caught during parsing. > # duplicate non-literals, allowed > d = {100: 1, len("ab")*50: 2} For sure allowed during parsing, possibly caught at run-time. > You keep mentioning "dictionary literal", but there actually is no > such thing in Python. I think you mean a dict display. Yes sorry, bad use of the term. I meant the curly-brace constructor---and only that. > (Don't worry, I > make the same mistake.) But the point is, a "dict literal" (display) can > contain keys which are not themselves literals, as above. Those keys can > have arbitrarily complex semantics, including side-effects. What do you > expect to happen? I'd mainly like to catch the "obvious" duplicates, just like the GNU C compiler catches the "obvious" divisions by zero. And by that I mean occurrences of duplicate string literals as keys. For instance: At parse time: {"foo": 333, "foo": 444} # forbidden---must catch {"foo": 333, "f" + "oo": 444} # OK to miss but also OK to catch {"foo": 333, function_returning_foo(): 444} # not caught (i.e. no miracles expected) At run time, I still slightly suspect that it may be more useful to prohibit duplicate keys in the same constructor than it is to allow them, but I don't have a clear opinion, because of both legacy code and other possible semantic issues. Thanks! > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group. > To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/yPH6ukUAQjc/unsubscribe. > To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. From steve at pearwood.info Tue May 3 21:09:20 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 4 May 2016 11:09:20 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: Message-ID: <20160504010920.GC12028@ando.pearwood.info> On Mon, May 02, 2016 at 02:36:35PM -0700, Luigi Semenzato wrote: > The original problem description: > > lives_in = { 'lion': ['Africa', 'America'], > 'parrot': ['Europe'], > #... 100+ more rows here > 'lion': ['Europe'], > #... 100+ more rows here > } > > The above constructor overwrites the first 'lion' entry silently, > often causing unexpected behavior. Did your colleague really have 200+ items in the dict? No matter, I suppose. The same principle applies. When you have significant amount of data in a dict (or any other data structure, such as a list, tree, whatever), the programmer has to take responsibility for the data validation. Not the compiler. Out of all the possible errors, why is "duplicate key" so special? Your colleague could have caused unexpected behaviour in many ways: lives_in = { # missed value 'lion': ['Africa', 'America'], # misspelled value 'parrot': ['Eruope'], # misspelled key 'kangeroo': ['Australia'], # invalid key 'kettle': ['Arctic'], # invalid value 'aardvark': 'South Africa', # missed key # oops, forgot 'tiger' altogether } Where was your colleague's data validation? I'm sorry that your colleague lost a lot of time debugging this failure, but you might have had exactly the same result from any of the above errors. Unless somebody can demonstrate that "duplicate keys" is a systematic and common failure among Python programmers, I think that it is perfectly reasonable to put the onus on detecting duplicates on the programmer, just like all those other data errors. The data validation need not be a big burden. In my own code, unless the dict is so small that I can easily see that it is correct with my own eyes, I always follow it with an assertion: assert len(lives_in) == 250 which is a cheap test for at least some duplicate, missed or extra keys. But depending on your use-case, it may be that dict is the wrong data structure to use, and you need something that will validate items as they are entered. Unless your dict is small enough that you can see it is correct by eye, you need some sort of check that your data is valid, that you haven't forgotten keys, or misspelled them. The dict constructor won't and can't do that for you, so you need to do it youself. Once you're doing that, then it is no extra effort to check for duplicates. So unless you have a good answer to the question "Why are duplicate keys so special that the dict constructor has to guard against them, when it doesn't guard against all the other failures we have to check for?", I think the status quo should stand. There is one obvious answer: Duplicate keys are special because, unlike the other errors, the dict constructor CAN guard against them. That would be a reasonable answer. But I'm not sure if it is special *enough* to justify violating the Zen of Python. That may be a matter of opinion and taste. -- Steve From ethan at stoneleaf.us Tue May 3 21:40:32 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 03 May 2016 18:40:32 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <20160504002300.GB12028@ando.pearwood.info> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> Message-ID: <57295310.7080809@stoneleaf.us> On 05/03/2016 05:23 PM, Steven D'Aprano wrote: > On Tue, May 03, 2016 at 02:27:16PM -0700, Ethan Furman wrote: >> On 05/03/2016 01:43 PM, Michael Selik wrote: >>> On Tue, May 3, 2016 at 4:00 PM Ethan Furman wrote: >>>> Which seems irrelevant to your argument: a duplicate key is a duplicate >>>> key whether it's 123 or 'xyz'. >>> >>> If an expression includes an impure function, then the duplication of >>> assignment to that key may have a desirable side-effect. >> >> I'm willing to say that should be done with an existing dict, not in a >> literal. > > But are you willing to say that the compiler should enforce that > stylistic judgement? > > >>> How would you handle an expression that evaluates differently for each >>> call? For example: >>> >>> {random(): 0, random(): 1} >> >> Easy: Don't Do That. ;) > > I see your wink, so I presume you're not actually suggesting that the > compiler (1) prohibit all function calls in dict displays, or (2) > hard-code the function name "random" in a black list. Indeed. > So what are you actually suggesting? Michael is raising a good point > here. If you don't like random as an example, how about: > > d = {spam(a): 'a', spam(b): 'BB', spam(c): 'Ccc'} > > I'm intentionally not giving you the values of a, b or c, or telling > you what spam() returns. Now you have the same information available > to you as the compiler has at compile time. What do you intend to do? Since the dict created by that dict display happens at run time, I am suggesting that during the process of creating that dict that any keys, however generated or retrieved, that are duplicates of keys already in the dict, cause an appropriate error (to be determined). Also, any dict display that is able to be used for dict creation at compile time (thereby avoiding the run-time logic) should have the same checks and raise the same error if duplicate keys are found (I imagine both run-time and compile-time dict creation from dict displays would use the same underlying function). > It's one thing to say "duplicate keys should be prohibited", and another > to come up with a detailed explanation of what precisely should happen. Hopefully my explanation is detail enough. >>> Let me flip the original request: I'd like to hear stronger arguments for >>> change, please. I'd be particularly interested in hearing how often >>> Pylint has caught this mistake. >> >> Well, when this happened to me I spent a loooonnnnnngggg time figuring >> out what the problem is. > > Okay, but how often does this happen? Daily? Weekly? Once per career? Once a year, maybe. The experience is painful enough to remember when the subject arises, but not painful enough to remember to be careful. ;) > What's a loooonnnnnngggg time? Twenty minutes? Twenty man-weeks? Longer than an hour, shorter than a day. >> One just doesn't expect duplicate keys to not raise: >> >> --> dict(one=1, one='uno') >> File "", line 1 >> SyntaxError: keyword argument repeated > > There's a difference though. Keyword argument *names* must be > identifiers, not expressions, and duplicates can be recognised by the > compiler at compile-time: I am largely unconcerned by the run-time/compile-time distinction in this case -- whenever it happens, check that the incoming key doesn't already exist. > If you think of it as: > > for key,item in initial_values: > if key in self: > raise TypeError('duplicate key') > else: > self[key] = item > > > then you have to deal with the fact that you might only notice a > duplicate after mutating your dict, which may include side-effects. Not sure what you mean by "mutating your dict" -- we're still talking about initial "dict display" to "dict" conversion, yes? -- ~Ethan~ From luigi at semenzato.com Tue May 3 21:58:43 2016 From: luigi at semenzato.com (Luigi Semenzato) Date: Tue, 3 May 2016 18:58:43 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <57295310.7080809@stoneleaf.us> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> Message-ID: On Tue, May 3, 2016 at 6:40 PM, Ethan Furman wrote: > Since the dict created by that dict display happens at run time, I am > suggesting that during the process of creating that dict that any keys, > however generated or retrieved, that are duplicates of keys already in the > dict, cause an appropriate error (to be determined). > > Also, any dict display that is able to be used for dict creation at compile > time (thereby avoiding the run-time logic) should have the same checks and > raise the same error if duplicate keys are found (I imagine both run-time > and compile-time dict creation from dict displays would use the same > underlying function). I agree with Ethan that this is the correct way of looking at this problem, rather than my original, confusing presentation. Duplicate keys in a dict display should be an error, period. In some cases the error can be detected at parse time, which is nicer. In other cases it can be detected only at run time, but it should always be an error. Also, Steven D'Aprano wrote: > Duplicate keys are special because, unlike the other errors, the dict > constructor CAN guard against them. > > That would be a reasonable answer. But I'm not sure if it is special > *enough* to justify violating the Zen of Python. That may be a matter of > opinion and taste. Thank you, I will gladly take the "reasonable answer" qualification offered here. And I agree that often these issues build down to aesthetics. But it's hard for me to see how the current semantics support the Zen of Python. I wouldn't presume that all original design decisions were infallible, and I still haven't heard a single good argument in favor of *allowing* duplicate keys. From rob.cliffe at btinternet.com Tue May 3 22:00:22 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 4 May 2016 03:00:22 +0100 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <57295310.7080809@stoneleaf.us> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> Message-ID: <572957B6.90809@btinternet.com> On 04/05/2016 02:40, Ethan Furman wrote: > On 05/03/2016 05:23 PM, Steven D'Aprano wrote: >> On Tue, May 03, 2016 at 02:27:16PM -0700, Ethan Furman wrote: >>> On 05/03/2016 01:43 PM, Michael Selik wrote: >>>> On Tue, May 3, 2016 at 4:00 PM Ethan Furman wrote: >>>>> Which seems irrelevant to your argument: a duplicate key is a >>>>> duplicate >>>>> key whether it's 123 or 'xyz'. >>>> >>>> If an expression includes an impure function, then the duplication of >>>> assignment to that key may have a desirable side-effect. >>> >>> I'm willing to say that should be done with an existing dict, not in a >>> literal. >> >> But are you willing to say that the compiler should enforce that >> stylistic judgement? >> >> >>>> How would you handle an expression that evaluates differently for each >>>> call? For example: >>>> >>>> {random(): 0, random(): 1} >>> >>> Easy: Don't Do That. ;) >> >> I see your wink, so I presume you're not actually suggesting that the >> compiler (1) prohibit all function calls in dict displays, or (2) >> hard-code the function name "random" in a black list. > > Indeed. > >> So what are you actually suggesting? Michael is raising a good point >> here. If you don't like random as an example, how about: >> >> d = {spam(a): 'a', spam(b): 'BB', spam(c): 'Ccc'} > > > > I'm intentionally not giving you the values of a, b or c, or telling > > you what spam() returns. Now you have the same information available > > to you as the compiler has at compile time. What do you intend to do? > > Since the dict created by that dict display happens at run time, I am > suggesting that during the process of creating that dict that any > keys, however generated or retrieved, that are duplicates of keys > already in the dict, cause an appropriate error (to be determined). > Sorry, I favour raising an error for duplicate *literal constant* keys (presumably at compile time). But I think making the above example cause a runtime error if two keys happened to be equal would break too much existing code. Rob From rob.cliffe at btinternet.com Tue May 3 21:54:39 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 4 May 2016 02:54:39 +0100 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <20160503235120.GA12028@ando.pearwood.info> References: <20160503235120.GA12028@ando.pearwood.info> Message-ID: <5729565F.7070306@btinternet.com> On 04/05/2016 00:51, Steven D'Aprano wrote: > Hi Luigi, > > On Mon, May 02, 2016 at 02:36:35PM -0700, Luigi Semenzato wrote: > > [...] >> lives_in = { 'lion': ['Africa', 'America'], >> 'parrot': ['Europe'], >> #... 100+ more rows here >> 'lion': ['Europe'], >> #... 100+ more rows here >> } >> >> The above constructor overwrites the first 'lion' entry silently, >> often causing unexpected behavior. > [...] >> For context, someone ran into this problem in my team at Google (we >> fixed it using pylint). I haven't seen any valid reason (in the bug >> or elsewhere) in favor of these constructor semantics. From the >> discussions I have seen, it seems just an oversight in the >> implementation/specification of dictionary literals. I'd be happy to >> hear stronger reasoning in favor of the status quo. > As far as I can see, you haven't specified in *detail* what change you > wish to propose. It is difficult for me to judge your proposal when I > don't know precisely what you are proposing. > > Should duplicate keys be a SyntaxError at compile time, or a TypeError > at runtime? Or something else? > > What counts as "duplicate keys"? I presume that you mean that two keys > count as duplicate if they hash to the same value, and are equal. But > you keep mentioning "literals" -- does this mean you care more about > whether they look the same rather than are the same? > > # duplicate literals, forbidden > d = {100: 1, 100: 2} > > # duplicate non-literals, allowed > d = {100: 1, len("ab")*50: 2} > > > You keep mentioning "dictionary literal", but there actually is no > such thing in Python. I think you mean a dict display. (Don't worry, I > make the same mistake.) But the point is, a "dict literal" (display) can > contain keys which are not themselves literals, as above. Those keys can > have arbitrarily complex semantics, including side-effects. What do you > expect to happen? > My initial reaction to the OP was negative, as in most contexts where keys are added to dictionaries, repeated keys silently overwrite, and consistency is a virtue. However, the original use case (a long dict literal - (sorry, transcribe that into 'dict display' if you wish; to me 'dict literal' is an evocative and comprehensible term and I will continue to use it, not meaning to offend)) is a completely plausible one, and it now seems to me that detecting duplicates is a case where 'practicality beats purity'. I agree with everything in Luigi's post of 30-May 2016 22:29 (my local time, sorry). I would just add that as someone who has used a linter (pylint) occasionally - I am trying to discipline myself to use it regularly - even if you are aware of it, there is still a barrier to its use: linters tend to produce a huge amount of mostly useless guff which you have to search to find the few nuggets that you really need. Steven, I am sure that this is not your intention, but it feels as if your requests for clarification are nitpicking and attempts to throw dust in our eyes, and avoiding a serious attempt to address the OP. But let me (presuming,if he will forgive me, to speak for the OP) attempt to answer by making a concrete proposal, as a starting point for discussion: *In a dictionary literal, it should be a syntax error to have two keys which are literal int or literal basestring values and which are equal. *(I say basestring, meaning that { 'lion' : 'x', u'lion' : 'y' } would be an error. So of course would { 18L : 'x', 0x12 : 'y' } etc.) From this starting point, bikeshedding can begin: (1) Is there any other kind of literal (i.e. other than int or string) which should be included? (Float and complex? Decimal?? What have I missed?) (2) Should constant int/string expressions which are folded to constants at compile time also be included i.e. would { 2 : 'x', 1+1 : 'y } also be an error? (Suggestion: This is an implementation detail, undefined. It's irrelevant to the practical case.) (3) Should a runtime error be raised instead of a SyntaxError ? (I can't imagine why, but if it turns out to be easier to implement, fine.) (4) Should a warning be raised instead of an error? (5) Should the action be different if not just the keys, but the values also are constant and equal (i.e. a no-effect repeat), e.g. a warning instead of an error? ( I suggest no. It's still likely to be unintentional, i.e. a mistake that the programmer would want to know about.) (6) Are there any other changes to the proposal which would simplify implementation, while still addressing the original use case? (7) If the proposal is accepted, should it appear complete in the next Python release, or should there be a more gradual adoption process? (8) Can/should anything be done to mitigate breaking automatic code generators which do generate duplicate literal keys? Later: While I was composing the above, 1 more post arrived from Luigi and from Steven. I must reply to two points from Steven: (1) "When you have significant amount of data in a dict (or any other data structure, such as a list, tree, whatever), the programmer has to take responsibility for the data validation. Not the compiler. Out of all the possible errors, why is "duplicate key" so special? Your colleague could have caused unexpected behaviour in many ways" No, no, no. Computers detect our errors and we would be helpless if they didn't. If this argument were taken to its logical conclusion, you would say that it is the programmer's responsibility to ensure that an entire (10000-line missile guidance?) program is syntactically correct, and all the compiler need do is either say "OK" or "Syntax Error" (or worse, try to run a syntactically incorrect program). Yes, Luigi's colleague could have made many mistakes, but surely it is a good thing for as many of them as is reasonably possible to be detected by the computer? Then we just have to decide what "reasonably possible" is. (2) "So unless you have a good answer to the question "Why are duplicate keys so special that the dict constructor has to guard against them, when it doesn't guard against all the other failures we have to check for?", I think the status quo should stand. There is one obvious answer: Duplicate keys are special because, unlike the other errors, the dict constructor CAN guard against them. That would be a reasonable answer. But I'm not sure if it is special *enough* to justify violating the Zen of Python. That may be a matter of opinion and taste." Sure, it is a matter of opinion. I ask: In the OP, what is the probability that (1) the programmer deliberately included a duplicate literal key (2) the programmer made a mistake? I concede that the answer may be different for automatic code generators (although if I wrote one that generated duplicate literal keys, I wouldn't be proud of it). Best wishes Rob Cliffe -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue May 3 22:20:42 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 3 May 2016 22:20:42 -0400 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <20160504002300.GB12028@ando.pearwood.info> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> Message-ID: On 5/3/2016 8:23 PM, Steven D'Aprano wrote: >> One just doesn't expect duplicate keys to not raise: >> >> --> dict(one=1, one='uno') >> File "", line 1 >> SyntaxError: keyword argument repeated > > Huh, I had completely forgotten that duplicate keyword arguments raise > a syntax error. I thought you got a runtime TypeError. In a sense, it should be. >>> def f(x): pass >>> f(1, x=2) Traceback (most recent call last): File "", line 1, in f(1, x=2) TypeError: f() got multiple values for argument 'x' But the case of multiple values via duplicate keywords can be and is caught when parsing, and parse errors are always (except maybe for ...) SyntaxErrors. -- Terry Jan Reedy From rosuav at gmail.com Tue May 3 22:22:58 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 May 2016 12:22:58 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <5729565F.7070306@btinternet.com> References: <20160503235120.GA12028@ando.pearwood.info> <5729565F.7070306@btinternet.com> Message-ID: On Wed, May 4, 2016 at 11:54 AM, Rob Cliffe wrote: > In a dictionary literal, it should be a syntax error to have two keys which > are literal int or literal basestring values and which are equal. > > (I say basestring, meaning that { 'lion' : 'x', u'lion' : 'y' } would > be an error. So of course would { 18L : 'x', 0x12 : 'y' } etc.) > > (7) If the proposal is accepted, should it appear complete in the next > Python release, or should there be a more gradual adoption process? This is the very soonest it can possibly appear, which means that notions of "long integer" and "basestring" aren't really applicable - they're Python 2 concepts. Text strings and byte strings in Python 3 are completely different: >>> {'lion': 'x', b'lion': 'y'} {'lion': 'x', b'lion': 'y'} and there aren't any short/long integers. So here's my rewording of your proposal: *** Identical string or numeric literals used as keys in the same dictionary display MAY cause a SyntaxError. *** Whether constants get folded before this check, and whether {1:'x', 1.0:'y'} causes an error, can be implementation-defined. Implementations are welcome to not check for this at all if they wish. Valid types for this check would be str, bytes, int, float, and complex - there's no way that the same literal could result in unequal values, so these are all safe. I'm still not sure whether I'm in favour of the proposal or not, but that's how I'd word it. ChrisA From random832 at fastmail.com Tue May 3 22:32:46 2016 From: random832 at fastmail.com (Random832) Date: Tue, 03 May 2016 22:32:46 -0400 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <20160504002300.GB12028@ando.pearwood.info> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> Message-ID: <1462329166.2320481.597438369.69F33F10@webmail.messagingengine.com> On Tue, May 3, 2016, at 20:23, Steven D'Aprano wrote: > On Tue, May 03, 2016 at 02:27:16PM -0700, Ethan Furman wrote: > > On 05/03/2016 01:43 PM, Michael Selik wrote: > > >How would you handle an expression that evaluates differently for each > > >call? For example: > > > > > > {random(): 0, random(): 1} > > > > Easy: Don't Do That. ;) > > I see your wink, so I presume you're not actually suggesting that the > compiler (1) prohibit all function calls in dict displays, or (2) > hard-code the function name "random" in a black list. How about prohibit it anyway (on the grounds of the two key expressions being identical - no comment on spam(a) vs spam(b)), and if the user *really* wants it they'll do {random(): x for x in (0, 1)} [or {other: stuff, **{that}}] From random832 at fastmail.com Tue May 3 22:44:08 2016 From: random832 at fastmail.com (Random832) Date: Tue, 03 May 2016 22:44:08 -0400 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <20160503235120.GA12028@ando.pearwood.info> <5729565F.7070306@btinternet.com> Message-ID: <1462329848.2323164.597442769.2339C543@webmail.messagingengine.com> On Tue, May 3, 2016, at 22:22, Chris Angelico wrote: > Whether constants get folded before this check, and whether {1:'x', > 1.0:'y'} causes an error, can be implementation-defined. For *general* constant folding, maybe, but what's the point of having it if it doesn't apply to negative numbers and nonimaginary complex numbers? How about tuple displays consisting of only numeric and string literals, empty tuple displays, and tuple displays which themselves satisfy this definition? From rosuav at gmail.com Tue May 3 23:04:50 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 May 2016 13:04:50 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <1462329848.2323164.597442769.2339C543@webmail.messagingengine.com> References: <20160503235120.GA12028@ando.pearwood.info> <5729565F.7070306@btinternet.com> <1462329848.2323164.597442769.2339C543@webmail.messagingengine.com> Message-ID: On Wed, May 4, 2016 at 12:44 PM, Random832 wrote: > On Tue, May 3, 2016, at 22:22, Chris Angelico wrote: >> Whether constants get folded before this check, and whether {1:'x', >> 1.0:'y'} causes an error, can be implementation-defined. > > For *general* constant folding, maybe, but what's the point of having it > if it doesn't apply to negative numbers and nonimaginary complex > numbers? How about tuple displays consisting of only numeric and string > literals, empty tuple displays, and tuple displays which themselves > satisfy this definition? That's why I say implementation-defined. I would generally expect -5 to count as a literal, but if (2+3) or ("a", "b") doesn't, so be it. Of course, I would expect 'a' and "a" to be seen as identical. Likewise 100 and 0x64. Beyond that, it's basically up to whatever the implementation finds easy; as long as the keys *would be* folded, and are literals, it's permissible to raise the error. ChrisA From tjreedy at udel.edu Tue May 3 23:23:39 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 3 May 2016 23:23:39 -0400 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <20160503235120.GA12028@ando.pearwood.info> Message-ID: On 5/3/2016 8:46 PM, Luigi Semenzato wrote: > On Tue, May 3, 2016 at 4:51 PM, Steven D'Aprano wrote: The thread seems to partly be based on a confusion between literals and displays, which led to dict displays being called dict literals. A literal defines an int or string at the initial lexing or tokenizing phase of the parser. A display, like comprehensions, abbreviates runtime code for creating a composite collection object. >> What counts as "duplicate keys"? I presume that you mean that two keys >> count as duplicate if they hash to the same value, and are equal. But >> you keep mentioning "literals" -- does this mean you care more about >> whether they look the same rather than are the same? > > Correct. The errors that I am guessing matter the most are those for > which folks copy-paste a key-value pair, where the key is a literal > string, intending to change the key, and then forget to change it. > >> # duplicate literals, forbidden >> d = {100: 1, 100: 2} > > Correct. Possibly caught during parsing. The above is equivalent to and an abbreviation of d = {}; d[100] = 1; d[100] = 2 A bit silly, but quite legal. Every value replacement necessarily involves a 'duplicate key'. Both of the above are also equivalent to d = {a:b for a,b in ((100,1), (100,2))} This is also equivalent to d = {} for a, b in ((100,1), (100,2)): d[a] = b which is close to the python version of how the comprehension would implemented in python. Here is a 5th version, which I expect is close to the actual CPython code for a dict display. tem = [100, 1, 100, 2] # as in the top of the virtual machine stack n = 4 # number of items in display = 2 * number of pairs d = {} for i in range(-n, 0, 2): d[tem[i]] = tem[i+1] We have 5 equivalent snippets, each of which could have the same bug. Why single out just one to raise an exception? My guess is that it is partly from mistakenly thinking of the one as a like a lexer literal rather than as runtime code. -- Terry Jan Reedy From rosuav at gmail.com Tue May 3 23:31:33 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 May 2016 13:31:33 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <20160503235120.GA12028@ando.pearwood.info> Message-ID: On Wed, May 4, 2016 at 1:23 PM, Terry Reedy wrote: > Why single out just one to raise an exception? My guess is that it is > partly from mistakenly thinking of the one as a like a lexer literal rather > than as runtime code. But the lexer's definition of "literal" is extremely narrow, and doesn't reflect the way people think about them. Most people's idea of a literal is closer to what ast.literal_eval can accept: Help on function literal_eval in module ast: literal_eval(node_or_string) Safely evaluate an expression node or a string containing a Python expression. The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None. >>> ast.literal_eval("[1,-2,(3+4j)]") [1, -2, (3+4j)] Neither -2 nor (3+4j) is technically a literal, and certainly list display isn't a literal, but "literal_eval" is happy to parse them. And that's how people think about them. Duplicate keys in dict display is more similar to duplicate keyword arguments in a function call than to reassignment. Can you show any good code that reuses a key in a single dict display? Wouldn't it at best be code smell? That doesn't necessarily mean that the compiler should stop you from doing it; but I do think that there's a fundamental difference in expectation here. And if the check happens, it wants to be a compile-time check, not a run-time one. ChrisA From ncoghlan at gmail.com Wed May 4 00:03:59 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 May 2016 14:03:59 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <57295310.7080809@stoneleaf.us> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> Message-ID: On 4 May 2016 at 11:40, Ethan Furman wrote: > On 05/03/2016 05:23 PM, Steven D'Aprano wrote: >> I'm intentionally not giving you the values of a, b or c, or telling >> you what spam() returns. Now you have the same information available >> to you as the compiler has at compile time. What do you intend to do? > > Since the dict created by that dict display happens at run time, I am > suggesting that during the process of creating that dict that any keys, > however generated or retrieved, that are duplicates of keys already in the > dict, cause an appropriate error (to be determined). I was curious as to whether or not it was technically feasible to implement this "duplicate keys" check solely for dict displays in CPython without impacting other dict use cases, and it turns out it should be. The key point is that BUILD_MAP already has its own PyDict_SetItem() loop in ceval.c (iterating over stack entries), and hence doesn't rely on the "dict_common_update" code shared by dict.update and the dict constructor(s). As such, the technical change being requested here (at least for CPython), is that BUILD_MAP be updated to do a PyDict_Contains() check prior to each key insertion, and: 1. Report a DeprecationWarning in 3.6 2. Report a [RuntimeError? ValueError?] in 3.7+ For code generators emitting dict displays, running their content through OrderedDict (or an equivalent if the code generator is written in something other than Python) prior to writing it out will be sufficient to eliminate duplicates. Since PyDict_Contains() is O(1), this shouldn't introduce a major slowdown in dict display evaluation (although the impact should still be measued prior to making any changes). All-in-all, the concept gets a +0 from me, although to allow for implementations that don't share CPython's distinction between dict displays and the normal dict constructor, any such behaviour (if introduced) should likely be flagged as an implementation detail that may vary across interpreters (as I would have been -1 if BUILD_MAP's implementation wasn't already independent of dict_common_update). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From random832 at fastmail.com Wed May 4 00:18:59 2016 From: random832 at fastmail.com (Random832) Date: Wed, 04 May 2016 00:18:59 -0400 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <20160503235120.GA12028@ando.pearwood.info> Message-ID: <1462335539.2350378.597495545.720CC407@webmail.messagingengine.com> On Tue, May 3, 2016, at 23:31, Chris Angelico wrote: > Duplicate keys in dict display > is more similar to duplicate keyword arguments in a function call than > to reassignment. To play devil's advocate for a minute here... Why *don't* we allow duplicate keyword arguments in a function call, anyway? CALL_FUNCTION doesn't seem to actually care, if I create this situation by monkey-patching the calling function's consts to get duplicate keywords. All the arguments that have been raised here for *allowing* duplicate keyword arguments seem to apply just as well to f(**{'a': 1}, **{'a': 2}) [where 'a' may not be the only key present and they may not be literal^Wdisplays], yet that is prohibited. Why not remove that check, too? If we can blithely pass the same constant key twice to BUILD_MAP, why *shouldn't* we be able to do the same to CALL_FUNCTION? Any argument against the latter applies equally to the former, as far as I can see. From michael.selik at gmail.com Wed May 4 00:20:16 2016 From: michael.selik at gmail.com (Michael Selik) Date: Wed, 04 May 2016 04:20:16 +0000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> Message-ID: On Wed, May 4, 2016 at 12:04 AM Nick Coghlan wrote: > As such, the technical change being requested here (at least for > CPython), is that BUILD_MAP be updated to do a PyDict_Contains() > check prior to each key insertion, and: > 1. Report a DeprecationWarning in 3.6 > 2. Report a [RuntimeError? ValueError?] in 3.7+ > To be clear, this implementation would potentially raise an error in the random example -- ``{random(): 0, random(): 1}`` -- correct? If so, then code smell or not, I think that's too much breakage. We could restrict the PyDict_Contains() call to only enforcing uniqueness of string or numeric literals, but then we'd still have some oddball situations: def a(): return 'a' {a(): 0, 'a': 1} -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.selik at gmail.com Wed May 4 00:40:16 2016 From: michael.selik at gmail.com (Michael Selik) Date: Wed, 04 May 2016 04:40:16 +0000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <1462335539.2350378.597495545.720CC407@webmail.messagingengine.com> References: <20160503235120.GA12028@ando.pearwood.info> <1462335539.2350378.597495545.720CC407@webmail.messagingengine.com> Message-ID: On Wed, May 4, 2016 at 12:19 AM Random832 wrote: > On Tue, May 3, 2016, at 23:31, Chris Angelico wrote: > > Duplicate keys in dict display > > is more similar to duplicate keyword arguments in a function call than > > to reassignment. > > If we can blithely pass the same constant key twice to BUILD_MAP, why > *shouldn't* we be able to do the same to CALL_FUNCTION? Because keyword arguments can only be valid identifiers, rather than expressions. There's absolutely no reason to use the same identifier as keyword twice, but there could be some oddball reasons to use the same expression as dict key twice. [I know I've said that before, but it's a long thread. My apologies if you'd already read that.] -------------- next part -------------- An HTML attachment was scrubbed... URL: From leewangzhong+python at gmail.com Wed May 4 02:04:03 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Wed, 4 May 2016 02:04:03 -0400 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <20160503235120.GA12028@ando.pearwood.info> <1462335539.2350378.597495545.720CC407@webmail.messagingengine.com> Message-ID: On Wed, May 4, 2016 at 12:40 AM, Michael Selik wrote: > > On Wed, May 4, 2016 at 12:19 AM Random832 wrote: >> >> On Tue, May 3, 2016, at 23:31, Chris Angelico wrote: >> > Duplicate keys in dict display >> > is more similar to duplicate keyword arguments in a function call than >> > to reassignment. >> >> If we can blithely pass the same constant key twice to BUILD_MAP, why >> *shouldn't* we be able to do the same to CALL_FUNCTION? > > Because keyword arguments can only be valid identifiers, rather than > expressions. There's absolutely no reason to use the same identifier as > keyword twice, but there could be some oddball reasons to use the same > expression as dict key twice. The uniqueness rule applies to splatted args, too. def printn(*args, **kwargs): print(*args, **kwargs, sep='\n') printn('hello', 'world', sep='\n') # => TypeError: print() got multiple values for keyword argument 'sep' Not a very useful example, but if you want to override or ignore incoming kwargs... ...... For semantics, I see a few possibilities (either error or warning): 0. Detect duplicate literal keys at compile-time. 1. Detect duplicate literals and hereditarily-literal displays and expressions at compile-time. (This includes keys which are tuple displays containing only hereditarily-literal elements, expressions of literals, and, er, implicit string concatenation like `'hello' " " 'world'`. As keys, these things are guaranteed to be immutable.) 2. Also statically analyze for duplicate variables as keys. (I think if it's a local variable that isn't captured by a closure using `nonlocal`, it is safe to assume that it's constant for that dict display.) 3. Constant-folding. (This leads to the dark path of static analysis shipping with the language.) float('inf'). Analyze at runtime. 1j. Declare that duplicate keys in a dict display causes underdefined behavior, and let interpreters decide how to handle it. -1. Identical expressions are warned against as duplicate keys. (By the way, anything that happens for dicts should also happen for sets.) From ethan at stoneleaf.us Wed May 4 02:32:04 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 03 May 2016 23:32:04 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <1462335539.2350378.597495545.720CC407@webmail.messagingengine.com> References: <20160503235120.GA12028@ando.pearwood.info> <1462335539.2350378.597495545.720CC407@webmail.messagingengine.com> Message-ID: <57299764.6080103@stoneleaf.us> On 05/03/2016 09:18 PM, Random832 wrote: > To play devil's advocate for a minute here... > > Why *don't* we allow duplicate keyword arguments in a function call, > anyway? CALL_FUNCTION doesn't seem to actually care, if I create this > situation by monkey-patching the calling function's consts to get > duplicate keywords. All the arguments that have been raised here for > *allowing* duplicate keyword arguments seem to apply just as well to > > f(**{'a': 1}, **{'a': 2}) Actually, I believe that is allowed now, although I'm not sure how far it goes. -- ~Ethan~ From ethan at stoneleaf.us Wed May 4 02:35:45 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 03 May 2016 23:35:45 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> Message-ID: <57299841.8020503@stoneleaf.us> On 05/03/2016 09:20 PM, Michael Selik wrote: > On Wed, May 4, 2016 at 12:04 AM Nick wrote: >> >> As such, the technical change being requested here (at least for >> CPython), is that BUILD_MAP be updated to do a PyDict_Contains() >> check prior to each key insertion, and: >> 1. Report a DeprecationWarning in 3.6 >> 2. Report a [RuntimeError? ValueError?] in 3.7+ > > To be clear, this implementation would potentially raise an error in the > random example -- ``{random(): 0, random(): 1}`` -- correct? If so, then > code smell or not, I think that's too much breakage. Either fix it all the way or don't bother. The rule "duplicates are allowed" or "duplicates are not allowed" is much simpler and easier to remember than "duplicates are allowed except when..." or, contrariwise, "duplicates are not allowed unless ...". -- ~Ethan~ From jcgoble3 at gmail.com Wed May 4 02:39:44 2016 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Wed, 4 May 2016 02:39:44 -0400 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <57299764.6080103@stoneleaf.us> References: <20160503235120.GA12028@ando.pearwood.info> <1462335539.2350378.597495545.720CC407@webmail.messagingengine.com> <57299764.6080103@stoneleaf.us> Message-ID: On Wed, May 4, 2016 at 2:32 AM, Ethan Furman wrote: > On 05/03/2016 09:18 PM, Random832 wrote: > >> To play devil's advocate for a minute here... >> >> Why *don't* we allow duplicate keyword arguments in a function call, >> anyway? CALL_FUNCTION doesn't seem to actually care, if I create this >> situation by monkey-patching the calling function's consts to get >> duplicate keywords. All the arguments that have been raised here for >> *allowing* duplicate keyword arguments seem to apply just as well to > >> >> >> f(**{'a': 1}, **{'a': 2}) > > > Actually, I believe that is allowed now, although I'm not sure how far it > goes. As of Python 3.5.1, it is still not allowed: C:\Users\Jonathan>python Python 3.5.1 (v3.5.1:37a07cee5969, Dec 6 2015, 01:54:25) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> def func(**kwargs): ... pass ... >>> func(**{'a': 1}, **{'a': 2}) Traceback (most recent call last): File "", line 1, in TypeError: func() got multiple values for keyword argument 'a' From pavol.lisy at gmail.com Wed May 4 02:47:00 2016 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Wed, 4 May 2016 08:47:00 +0200 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> Message-ID: 2016-05-04 6:03 GMT+02:00, Nick Coghlan : > I was curious as to whether or not it was technically feasible to > implement this "duplicate keys" check solely for dict displays in > CPython without impacting other dict use cases, and it turns out it > should be. [...] > 1. Report a DeprecationWarning in 3.6 > 2. Report a [RuntimeError? ValueError?] in 3.7+ But what about SyntaxError instead of RuntimeError? Could it be easily done too? PS. Probably nobody is interested in my tests with unicode literals which are "NFKC equivalent" but it help me to understand how things are done behind the scene. def f(**kwd): for i in kwd: print(i, kwd[i]) def tst(): f(ij=1, ?=2) # this is syntax error def tst2(): f(**{1:1}) # this is runtime error def tst3(): f(**{'ij':1, '?':2}) # this is legal (for me a little unexpected, maybe bug?) From leewangzhong+python at gmail.com Wed May 4 04:16:18 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Wed, 4 May 2016 04:16:18 -0400 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> Message-ID: On May 4, 2016 2:47 AM, "Pavol Lisy" wrote: > > 2016-05-04 6:03 GMT+02:00, Nick Coghlan : > > > I was curious as to whether or not it was technically feasible to > > implement this "duplicate keys" check solely for dict displays in > > CPython without impacting other dict use cases, and it turns out it > > should be. > > [...] > > > 1. Report a DeprecationWarning in 3.6 > > 2. Report a [RuntimeError? ValueError?] in 3.7+ > > But what about SyntaxError instead of RuntimeError? Could it be easily > done too? Is it really a SyntaxError? The compiler knows what the source code means, but it assumes that the meaning is not what you intended. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Wed May 4 05:05:53 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 4 May 2016 10:05:53 +0100 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <57299841.8020503@stoneleaf.us> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <57299841.8020503@stoneleaf.us> Message-ID: <5729BB71.9060703@btinternet.com> On 04/05/2016 07:35, Ethan Furman wrote: > On 05/03/2016 09:20 PM, Michael Selik wrote: >> On Wed, May 4, 2016 at 12:04 AM Nick wrote: >>> >>> As such, the technical change being requested here (at least for >>> CPython), is that BUILD_MAP be updated to do a PyDict_Contains() >>> check prior to each key insertion, and: >>> 1. Report a DeprecationWarning in 3.6 >>> 2. Report a [RuntimeError? ValueError?] in 3.7+ >> >> To be clear, this implementation would potentially raise an error in the >> random example -- ``{random(): 0, random(): 1}`` -- correct? If so, then >> code smell or not, I think that's too much breakage. > > Either fix it all the way or don't bother. The rule "duplicates are > allowed" or "duplicates are not allowed" is much simpler and easier to > remember than "duplicates are allowed except when..." or, > contrariwise, "duplicates are not allowed unless ...". > I disagree, on the grounds that 'practicality beats purity' (sorry to repeat myself). Here's a thought. We are rightly concerned with changes breaking existing code. The proposed change could lead to some existing code being *repaired*, by pointing out a mistake that the author doesn't know is there. Rob Cliffe From p.f.moore at gmail.com Wed May 4 05:48:22 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 May 2016 10:48:22 +0100 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <5729BB71.9060703@btinternet.com> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <57299841.8020503@stoneleaf.us> <5729BB71.9060703@btinternet.com> Message-ID: On 4 May 2016 at 10:05, Rob Cliffe wrote: >> Either fix it all the way or don't bother. The rule "duplicates are allowed" or >> "duplicates are not allowed" is much simpler and easier to remember than >> "duplicates are allowed except when..." or, contrariwise, "duplicates are not >> allowed unless ...". > > I disagree, on the grounds that 'practicality beats purity' (sorry to repeat > myself). > Here's a thought. > We are rightly concerned with changes breaking existing code. > The proposed change could lead to some existing code being *repaired*, by > pointing out a mistake that the author doesn't know is there. The problem is that it could also break code that works at the moment. Consider a mapping of constants to names: colors = { RED: "Red", BLUE: "Blue", PINK: "Pink", LIGHT_BLUE: "Light Blue", DARK_BLUE: "Dark Blue", } On a certain output device, there may be no "light blue", and in that case the code sets LIGHT_BLUE = BLUE. That's entirely reasonable, and the author may be perfectly happy with the current behavior in that case. So I think it's important for backward compatibility that this change be *only* for the case of literal values as keys (which is what the OP proposed, of course). And given that it needs to just be for literal values as keys, I personally think that makes it a case of "Special cases aren't special enough to break the rules", so I'm -0 on the proposal (I don't think it's a *bad* thing, in the constants-only form, just that it's not worth the effort - but if someone else makes the effort, I won't object). Paul From mal at egenix.com Wed May 4 07:56:56 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 4 May 2016 13:56:56 +0200 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <20160504010920.GC12028@ando.pearwood.info> References: <20160504010920.GC12028@ando.pearwood.info> Message-ID: <5729E388.9080603@egenix.com> On 04.05.2016 03:09, Steven D'Aprano wrote: > On Mon, May 02, 2016 at 02:36:35PM -0700, Luigi Semenzato wrote: > >> The original problem description: >> >> lives_in = { 'lion': ['Africa', 'America'], >> 'parrot': ['Europe'], >> #... 100+ more rows here >> 'lion': ['Europe'], >> #... 100+ more rows here >> } >> >> The above constructor overwrites the first 'lion' entry silently, >> often causing unexpected behavior. Issuing a warning for this would probably help, but raising an error introduced a backwards incompatibility which is not warranted IMO, given how seldom such situations occur in practice. For the implementation, there are two possibilities I can think of: 1. have STORE_MAP run a test for an already existing entry and issue a warning 2. add a final CHECK_MAP byte code to check that the number of keys in the mapping correspond to the number of keys added via the dict literal The first one causes a lot of overhead, esp. for larger mappings such as the ones used by the codecs or other static data tables. Most of the times these are generated anyway, so the warning case would not occur at all, but you'd still have to check for a collision N times. The second one is cheap regarding performance, but may not be accurate, since STORE_MAP may well be working on dynamically generated keys. It does work well for constants, and if we're just issuing a warning, may be good enough. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 04 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From jmcs at jsantos.eu Wed May 4 08:07:39 2016 From: jmcs at jsantos.eu (=?UTF-8?B?Sm/Do28gU2FudG9z?=) Date: Wed, 4 May 2016 14:07:39 +0200 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <57299841.8020503@stoneleaf.us> <5729BB71.9060703@btinternet.com> Message-ID: I think "special cases aren't special enough to break the rules" make this a -1, there's no reason for the behaviour change besides enabling bad code. On 4 May 2016 at 11:48, Paul Moore wrote: > On 4 May 2016 at 10:05, Rob Cliffe wrote: > >> Either fix it all the way or don't bother. The rule "duplicates are > allowed" or > >> "duplicates are not allowed" is much simpler and easier to remember than > >> "duplicates are allowed except when..." or, contrariwise, "duplicates > are not > >> allowed unless ...". > > > > I disagree, on the grounds that 'practicality beats purity' (sorry to > repeat > > myself). > > Here's a thought. > > We are rightly concerned with changes breaking existing code. > > The proposed change could lead to some existing code being *repaired*, by > > pointing out a mistake that the author doesn't know is there. > > The problem is that it could also break code that works at the moment. > Consider a mapping of constants to names: > > colors = { > RED: "Red", > BLUE: "Blue", > PINK: "Pink", > LIGHT_BLUE: "Light Blue", > DARK_BLUE: "Dark Blue", > } > > On a certain output device, there may be no "light blue", and in that > case the code sets LIGHT_BLUE = BLUE. That's entirely reasonable, and > the author may be perfectly happy with the current behavior in that > case. > > So I think it's important for backward compatibility that this change > be *only* for the case of literal values as keys (which is what the OP > proposed, of course). And given that it needs to just be for literal > values as keys, I personally think that makes it a case of "Special > cases aren't special enough to break the rules", so I'm -0 on the > proposal (I don't think it's a *bad* thing, in the constants-only > form, just that it's not worth the effort - but if someone else makes > the effort, I won't object). > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed May 4 08:08:39 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 4 May 2016 22:08:39 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <20160503235120.GA12028@ando.pearwood.info> Message-ID: <20160504120838.GD12028@ando.pearwood.info> On Tue, May 03, 2016 at 05:46:33PM -0700, Luigi Semenzato wrote: [...] > > Should duplicate keys be a SyntaxError at compile time, or a TypeError > > at runtime? Or something else? > > Is there such a thing as a SyntaxWarning? Yes there is. > From my perspective it > would be fine to make it a SyntaxError, but I am not sure it would be > overall a good choice for legacy code (i.e. as an error it might break > old code, and I don't know how many other things a new language > specification is going to break). > > It could also be a run-time error, but it might be nicer to detect it > earlier. Maybe both. There are serious limits to what the compiler can detect at compile-time. So unless you have your heart set on a compile-time SyntaxError (or Warning) it might be best to forget all about compile-time detection, ignore the question of "literals", and just focus on run-time TypeError if a duplicate key is detected. Otherwise, you will have the situation where Python only detects *some* duplicate keys, and your colleague will be cursing that Python does such a poor job of detecting duplicates. And it will probably be implementation-dependent, e.g. if your Python compiler does constant folding it might detect {1+1: None, 2: None} as duplicates, but if it doesn't have constant folding (or you have turned it off), it won't. So with implementation-dependent compile-time checks, you can't even guarantee what will be detected. In that case, you might as well use a linter. As far as I am concerned, a compile-time check is next-to-useless. It will be more annoying than useful, since it will give people a false sense of security, while still missing duplicates. So I intend to only discuss a run-time solution, which has the advantage that Python will detect a much wider range of duplicates: not just: {"SPAM": None, "SPAM": None} for example, but also: {"SPAM": None, ("spa".upper() + "AM"[1:]): None} But now we're getting out of the realm of "detect obvious typos and duplicates" and into a more judgemental area. Once you start prohibiting complex expressions or function calls that happen return duplicate keys, you can no longer be *quite* so sure that this is an error. Maybe the programmer has some reason for allowing duplicates. Not necessarily a *good* reason, but people write all sorts of ugly, bad, fragile, silly code without the compiler giving them an error. "Consenting adults" applies, and Python generally allows us to shoot ourselves in the foot. Let's say I write something like this: with open(path) as f: d = { f.write(a): 'alpha', f.write(b): 'beta', f.write(c): 'gamma', f.write(d): 'delta', } and purely by my bad luck, it turns out that len(b) and len(c) are equal, so that there are two duplicate keys. Personally, I think this is silly code, but there's no rule against silly code in Python, and maybe I have a (good/bad/stupid) reason for writing this. Maybe I don't care that duplicate keys are over-written. If we introduce your rule, that's the same as saying "this code is so awful, that we have to ban it... but only *sometimes*, when the lengths happen to be equal, the rest of the time it's fine". Are you okay with saying that? (Not a rhetorical question, and you are allowed to say "Yes".) So which is worse? - most of the time the code works fine, but sometimes it fails, raising an exception and leaving the file in a half-written state; or - the code always works fine, except that sometimes a duplicate key over-writes the previous value, which I may not even care about. I don't know which is worse. If there is no clear answer that is obviously right, then the status quo wins, even if the status quo is less than perfect. Even if the status quo is *awful*, it may be that all the alternatives are just as bad. I think a compile-time check is just enough to give people a false sense of security, and so is *worse* than what we have now. And I'm right on the fence regarding a run-time check. So to me, my judgement is: with no clear winner, the status quo stays. > > What counts as "duplicate keys"? I presume that you mean that two keys > > count as duplicate if they hash to the same value, and are equal. But > > you keep mentioning "literals" -- does this mean you care more about > > whether they look the same rather than are the same? > > Correct. The errors that I am guessing matter the most are those for > which folks copy-paste a key-value pair, where the key is a literal > string, intending to change the key, and then forget to change it. With respect, I think that is a harmful position to take. That leaves the job half-done: the constructor will complain about *some* duplicates, but not all, and worse, which ones it detects may depend on the implementation you happen to be using! If it is worth complaining about {0: None, 0: None} then it must also be worth complaining about: {0: None, 0.0: None, int(): None, 1-1: None, len(""): None} etc. Otherwise, I guarantee that somebody will be pulling their hair out as to why Python only sometimes detects duplicate keys. Better to never detect them at all (so you know you have to test for duplicates yourself) than to give a false sense of security. -- Steve From steve at pearwood.info Wed May 4 08:15:25 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 4 May 2016 22:15:25 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <5729E388.9080603@egenix.com> References: <20160504010920.GC12028@ando.pearwood.info> <5729E388.9080603@egenix.com> Message-ID: <20160504121525.GE12028@ando.pearwood.info> On Wed, May 04, 2016 at 01:56:56PM +0200, M.-A. Lemburg wrote: > Issuing a warning for this would probably help, but raising > an error introduced a backwards incompatibility which is > not warranted IMO, given how seldom such situations occur in > practice. > > For the implementation, there are two possibilities I can think > of: > > 1. have STORE_MAP run a test for an already existing entry > and issue a warning > > 2. add a final CHECK_MAP byte code to check that the number > of keys in the mapping correspond to the number of keys > added via the dict literal > > The first one causes a lot of overhead, esp. for larger mappings > such as the ones used by the codecs or other static data tables. > Most of the times these are generated anyway, so the warning > case would not occur at all, but you'd still have to check > for a collision N times. > > The second one is cheap regarding performance, but may not > be accurate, since STORE_MAP may well be working on dynamically > generated keys. It does work well for constants, and if we're > just issuing a warning, may be good enough. I'm not sure I understand what you mean by this. Are you suggesting that if I write: {1: None, 1: None} Python knows that it got two keys as argument, and so CHECK_MAP can determine that there is a duplicate? (i.e. that 2 keys were arguments, but the finished dict only has length 1). But if I write: {x: None, x: None} Python *cannot* tell that there were two keys given (since they are expressions, not constants) and CHECK_MAP can't do anything? -- Steve From steve at pearwood.info Wed May 4 08:22:52 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 4 May 2016 22:22:52 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <1462335539.2350378.597495545.720CC407@webmail.messagingengine.com> References: <20160503235120.GA12028@ando.pearwood.info> <1462335539.2350378.597495545.720CC407@webmail.messagingengine.com> Message-ID: <20160504122252.GF12028@ando.pearwood.info> On Wed, May 04, 2016 at 12:18:59AM -0400, Random832 wrote: > On Tue, May 3, 2016, at 23:31, Chris Angelico wrote: > > Duplicate keys in dict display > > is more similar to duplicate keyword arguments in a function call than > > to reassignment. > > To play devil's advocate for a minute here... > > Why *don't* we allow duplicate keyword arguments in a function call, > anyway? CALL_FUNCTION doesn't seem to actually care, if I create this > situation by monkey-patching the calling function's consts to get > duplicate keywords. All the arguments that have been raised here for > *allowing* duplicate keyword arguments seem to apply just as well to > f(**{'a': 1}, **{'a': 2}) [where 'a' may not be the only key present and > they may not be literal^Wdisplays], yet that is prohibited. Why not > remove that check, too? (1) Because status quo wins. In the absence of a really compelling reason to remove that check, we don't have to justify prohibing duplicate keyword args, we just have to note that it is already in place and so we shouldn't change it. (2) Duplicate keys in a dict may have side-effects, which mean they do something. Duplicate keyword arguments don't have side-effects (not in the keys at least). So that's a difference. (3) The analogy between function calls and dicts is not really that strong. The dict constuctor intentionally allows certain duplicates: py> dict({'a': None}, a=2) {'a': 2} for good reason (think of over-riding the values provided in the input dict with keyword arguments), but it's not clear that there is an analogy for function calls. Perhaps there is, and if it is a good enough analogy, we might have to re-think the duplicate kwargs situation. -- Steve From rosuav at gmail.com Wed May 4 08:24:06 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 May 2016 22:24:06 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <20160504120838.GD12028@ando.pearwood.info> References: <20160503235120.GA12028@ando.pearwood.info> <20160504120838.GD12028@ando.pearwood.info> Message-ID: On Wed, May 4, 2016 at 10:08 PM, Steven D'Aprano wrote: > Let's say I write something like this: > > with open(path) as f: > d = { > f.write(a): 'alpha', > f.write(b): 'beta', > f.write(c): 'gamma', > f.write(d): 'delta', > } > > and purely by my bad luck, it turns out that len(b) and len(c) are > equal, so that there are two duplicate keys. Personally, I think this is > silly code, but there's no rule against silly code in Python, and maybe > I have a (good/bad/stupid) reason for writing this. Maybe I don't care > that duplicate keys are over-written. > > If we introduce your rule, that's the same as saying "this code is > so awful, that we have to ban it... but only *sometimes*, when the > lengths happen to be equal, the rest of the time it's fine". > > Are you okay with saying that? (Not a rhetorical question, and you are > allowed to say "Yes".) > My response: Yes. I am okay with saying that. Because the question really is: What is this dictionary being used for? Why are you using keys that might, under some circumstances, overwrite each other? The point of a dict is to retrieve values when given keys. How can you do this, if those keys might collide? I've yet to see any real-world code that does this usefully, in a dict display. And the best "well, maybe if" examples have all involved generated code, so it wouldn't be hard to change your codegen to use this instead: d = dict(( (f.write(a), 'alpha'), (f.write(b), 'beta'), (f.write(c), 'gamma'), (f.write(d), 'delta'), )) which, of course, will continue to function as expected. ChrisA From ethan at stoneleaf.us Wed May 4 08:25:06 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 04 May 2016 05:25:06 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <20160503235120.GA12028@ando.pearwood.info> <1462335539.2350378.597495545.720CC407@webmail.messagingengine.com> <57299764.6080103@stoneleaf.us> Message-ID: <5729EA22.6020907@stoneleaf.us> On 05/03/2016 11:39 PM, Jonathan Goble wrote: > On Wed, May 4, 2016 at 2:32 AM, Ethan Furman wrote: >> On 05/03/2016 09:18 PM, Random832 wrote: >>> To play devil's advocate for a minute here... >>> >>> Why *don't* we allow duplicate keyword arguments in a function call, >>> anyway? CALL_FUNCTION doesn't seem to actually care, if I create this >>> situation by monkey-patching the calling function's consts to get >>> duplicate keywords. All the arguments that have been raised here for >>> *allowing* duplicate keyword arguments seem to apply just as well to >> >>> >>> >>> f(**{'a': 1}, **{'a': 2}) >> >> >> Actually, I believe that is allowed now, although I'm not sure how far it >> goes. > > As of Python 3.5.1, it is still not allowed: > > C:\Users\Jonathan>python > Python 3.5.1 (v3.5.1:37a07cee5969, Dec 6 2015, 01:54:25) [MSC v.1900 > 64 bit (AMD64)] on win32 > Type "help", "copyright", "credits" or "license" for more information. >>>> def func(**kwargs): > ... pass > ... >>>> func(**{'a': 1}, **{'a': 2}) > Traceback (most recent call last): > File "", line 1, in > TypeError: func() got multiple values for keyword argument 'a' Ah, right. The PEP was changed: http://bugs.python.org/issue2292#msg234413 -- ~Ethan~ From ethan at stoneleaf.us Wed May 4 08:35:53 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 04 May 2016 05:35:53 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <20160504120838.GD12028@ando.pearwood.info> References: <20160503235120.GA12028@ando.pearwood.info> <20160504120838.GD12028@ando.pearwood.info> Message-ID: <5729ECA9.50008@stoneleaf.us> On 05/04/2016 05:08 AM, Steven D'Aprano wrote: > If it is worth complaining about > > {0: None, 0: None} > > then it must also be worth complaining about: > > {0: None, 0.0: None, int(): None, 1-1: None, len(""): None} > > etc. Otherwise, I guarantee that somebody will be pulling their hair > out as to why Python only sometimes detects duplicate keys. Better to > never detect them at all (so you know you have to test for duplicates > yourself) than to give a false sense of security. Agreed. Either fix it all they way, or leave it alone. -- ~Ethan~ From mal at egenix.com Wed May 4 08:37:56 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 4 May 2016 14:37:56 +0200 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <20160504121525.GE12028@ando.pearwood.info> References: <20160504010920.GC12028@ando.pearwood.info> <5729E388.9080603@egenix.com> <20160504121525.GE12028@ando.pearwood.info> Message-ID: <5729ED24.2010206@egenix.com> On 04.05.2016 14:15, Steven D'Aprano wrote: > On Wed, May 04, 2016 at 01:56:56PM +0200, M.-A. Lemburg wrote: > >> Issuing a warning for this would probably help, but raising >> an error introduced a backwards incompatibility which is >> not warranted IMO, given how seldom such situations occur in >> practice. >> >> For the implementation, there are two possibilities I can think >> of: >> >> 1. have STORE_MAP run a test for an already existing entry >> and issue a warning >> >> 2. add a final CHECK_MAP byte code to check that the number >> of keys in the mapping correspond to the number of keys >> added via the dict literal >> >> The first one causes a lot of overhead, esp. for larger mappings >> such as the ones used by the codecs or other static data tables. >> Most of the times these are generated anyway, so the warning >> case would not occur at all, but you'd still have to check >> for a collision N times. >> >> The second one is cheap regarding performance, but may not >> be accurate, since STORE_MAP may well be working on dynamically >> generated keys. It does work well for constants, and if we're >> just issuing a warning, may be good enough. > > I'm not sure I understand what you mean by this. Are you suggesting that > if I write: > > {1: None, 1: None} > > Python knows that it got two keys as argument, and so > CHECK_MAP can determine that there is a duplicate? (i.e. that 2 keys > were arguments, but the finished dict only has length 1). But if I > write: > > {x: None, x: None} > > Python *cannot* tell that there were two keys given (since they are > expressions, not constants) and CHECK_MAP can't do anything? Well, the CHECK_MAP byte code would only check whether the number of assignments you are doing is equal to the number of keys found in the final dict. That's a very coarse test and not perfect. It can also not tell you which key causes the mismatch. On the plus side, the extra check is a simply size check and doesn't have to be applied per added mapping. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 04 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From ethan at stoneleaf.us Wed May 4 08:42:47 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 04 May 2016 05:42:47 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <57299841.8020503@stoneleaf.us> <5729BB71.9060703@btinternet.com> Message-ID: <5729EE47.7010400@stoneleaf.us> On 05/04/2016 02:48 AM, Paul Moore wrote: > The problem is that it could also break code that works at the moment. > Consider a mapping of constants to names: > > colors = { > RED: "Red", > BLUE: "Blue", > PINK: "Pink", > LIGHT_BLUE: "Light Blue", > DARK_BLUE: "Dark Blue", > } > > On a certain output device, there may be no "light blue", and in that > case the code sets LIGHT_BLUE = BLUE. That's entirely reasonable, and > the author may be perfectly happy with the current behavior in that > case. Actually, that code is setting BLUE = "Light Blue". ;) > So I think it's important for backward compatibility that this change > be *only* for the case of literal values as keys (which is what the OP > proposed, of course). And given that it needs to just be for literal > values as keys, I personally think that makes it a case of "Special > cases aren't special enough to break the rules", so I'm -0 on the > proposal (I don't think it's a *bad* thing, in the constants-only > form, just that it's not worth the effort - but if someone else makes > the effort, I won't object). I object. Having only /some/ duplicates checked for is overcomplicating without very good reason. Perhaps the way forward is to check for all duplicates, but issue a Warning instead of raising an Exception? To be honest, though, I'm starting to agree with Steven D'Aprano that sufficiently large dicts should be verified by other means, and those other means can also check for duplicates. -- ~Ethan~ From ethan at stoneleaf.us Wed May 4 08:44:23 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 04 May 2016 05:44:23 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <57299841.8020503@stoneleaf.us> <5729BB71.9060703@btinternet.com> Message-ID: <5729EEA7.4020809@stoneleaf.us> On 05/04/2016 05:07 AM, Jo?o Santos wrote: > I think "special cases aren't special enough to break the rules" make > this a -1, there's no reason for the behaviour change besides enabling > bad code. Actually, it /disables/ bad code. ;) -- ~Ethan~ From random832 at fastmail.com Wed May 4 09:30:03 2016 From: random832 at fastmail.com (Random832) Date: Wed, 04 May 2016 09:30:03 -0400 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <20160504122252.GF12028@ando.pearwood.info> References: <20160503235120.GA12028@ando.pearwood.info> <1462335539.2350378.597495545.720CC407@webmail.messagingengine.com> <20160504122252.GF12028@ando.pearwood.info> Message-ID: <1462368603.1710672.597896953.4F86465A@webmail.messagingengine.com> On Wed, May 4, 2016, at 08:22, Steven D'Aprano wrote: > (1) Because status quo wins. In the absence of a really compelling > reason to remove that check, we don't have to justify prohibing > duplicate keyword args, we just have to note that it is already in place > and so we shouldn't change it. > > (2) Duplicate keys in a dict may have side-effects, which mean they do > something. Duplicate keyword arguments don't have side-effects (not in > the keys at least). So that's a difference. f(a=1, b=2, **{foo(): bar()}) where foo may return 'a' or 'b', though this is the first that I've heard that the side-effect argument only applies to the keys. > > (3) The analogy between function calls and dicts is not really that > strong. The dict constuctor intentionally allows certain duplicates: > > py> dict({'a': None}, a=2) > {'a': 2} > > for good reason but not {**{'a': None}, **{'a': 2}}) > (think of over-riding the values provided in the input > dict with keyword arguments), but it's not clear that there is an > analogy for function calls. Perhaps there is, and if it is a good enough > analogy, we might have to re-think the duplicate kwargs situation. foo(**{'a':None}, a=2) seems like a perfect analogy to me. From steve at pearwood.info Wed May 4 09:50:12 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 4 May 2016 23:50:12 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> Message-ID: <20160504135012.GG12028@ando.pearwood.info> On Tue, May 03, 2016 at 06:58:43PM -0700, Luigi Semenzato wrote: > I still haven't heard a single > good argument in favor of *allowing* duplicate keys. Because duplicate keys are not necessarily an error. Maybe you don't care if there are duplicates. Maybe you expect duplicates, and want the last one seen to win. Sometimes experienced programmers forget what it is like to be a beginning programmer. One of the most useful things a beginner can do is experiment at the interactive interpreter. The more things the language prohibits, the more errors the beginner gets, the less they learn, and the more intimidating the language is. Duplicate keys are not necessarily an error. It just means one value or the other is dropped. How do dict displays construct the dict? Do they add keys from left to right, or right to left? I can read the source, if I can find it, and locate the right section of the right file, and if I can read C. Or I can do: py> {1: "left", 1: "right"} {1: 'right'} and in one line of code in the Python interactive interpreter I've learned more than in probably an hour of digging through C code. To a beginner, being able to do things like this without Big Brother shouting at them "No! That's Wrong! Have An Exception!!!" all the time is literally without price. Who cares that it's a useless thing to do in production code? "Useless" is not an error. Errors, and warnings, should be left for things which really are errors. Even if you think my example doesn't matter, and you think that duplicate keys are always an error, it may be that this case is not bad enough to prohibit. You don't get this check for free. It requires code, and that code has to run, and it will run for every single dict display whether needed or not. Depending on how it is implemented, it might be quite expensive. All those programmers who have carefully inspected their source code to ensure that the data used in the dict is correct, with no duplicates and no misspellings and no missed keys, they have no need of this check. But you would force it on them anyway, because one of your colleagues was sloppy and made a mistake and had difficulty debugging it. Okay, I have sympathy for you and your colleague, but that doesn't mean its okay for everyone to carry the cost of checking for your error. Maybe it is justified. Maybe it isn't. If you are really serious that you think duplicate keys are an error, then to be consistent you should also want to raise an error in the dict constructor, dict(obj, **kwargs). If not, what is your reasoning for why duplicate keys should be allowed in dict(...)? -- Steve From steve at pearwood.info Wed May 4 10:00:26 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 5 May 2016 00:00:26 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <5729ED24.2010206@egenix.com> References: <20160504010920.GC12028@ando.pearwood.info> <5729E388.9080603@egenix.com> <20160504121525.GE12028@ando.pearwood.info> <5729ED24.2010206@egenix.com> Message-ID: <20160504140026.GH12028@ando.pearwood.info> On Wed, May 04, 2016 at 02:37:56PM +0200, M.-A. Lemburg wrote: > On 04.05.2016 14:15, Steven D'Aprano wrote: > > On Wed, May 04, 2016 at 01:56:56PM +0200, M.-A. Lemburg wrote: > > > >> Issuing a warning for this would probably help, but raising > >> an error introduced a backwards incompatibility which is > >> not warranted IMO, given how seldom such situations occur in > >> practice. I agree with you that an error is unjustified but a warning might be. > >> For the implementation, there are two possibilities I can think > >> of: > >> > >> 1. have STORE_MAP run a test for an already existing entry > >> and issue a warning You have suggested that's too expensive, so I'm happy to rule that out. > >> 2. add a final CHECK_MAP byte code to check that the number > >> of keys in the mapping correspond to the number of keys > >> added via the dict literal [...] > >> The second one is cheap regarding performance, but may not > >> be accurate, since STORE_MAP may well be working on dynamically > >> generated keys. It does work well for constants, and if we're > >> just issuing a warning, may be good enough. > > > > I'm not sure I understand what you mean by this. Are you suggesting that > > if I write: > > > > {1: None, 1: None} > > > > Python knows that it got two keys as argument, and so > > CHECK_MAP can determine that there is a duplicate? (i.e. that 2 keys > > were arguments, but the finished dict only has length 1). But if I > > write: > > > > {x: None, x: None} > > > > Python *cannot* tell that there were two keys given (since they are > > expressions, not constants) and CHECK_MAP can't do anything? > > Well, the CHECK_MAP byte code would only check whether the number > of assignments you are doing is equal to the number of keys > found in the final dict. That's a very coarse test and not > perfect. It can also not tell you which key causes the > mismatch. > > On the plus side, the extra check is a simply size check > and doesn't have to be applied per added mapping. Sorry MAL, perhaps I'm a bit slow tonight, but I'm still lost. {x: None, x: None} -- would CHECK_MAP detect that as a duplicate or not? If the answer is "Yes", then this seems to me to be a cheap way of detecting the existence of at least one duplicate (but not which key was duplicate). That strikes me as good enough, in which case I would support raising a warning. -- Steve From mal at egenix.com Wed May 4 10:18:59 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 4 May 2016 16:18:59 +0200 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <20160504140026.GH12028@ando.pearwood.info> References: <20160504010920.GC12028@ando.pearwood.info> <5729E388.9080603@egenix.com> <20160504121525.GE12028@ando.pearwood.info> <5729ED24.2010206@egenix.com> <20160504140026.GH12028@ando.pearwood.info> Message-ID: <572A04D3.5050204@egenix.com> On 04.05.2016 16:00, Steven D'Aprano wrote: > On Wed, May 04, 2016 at 02:37:56PM +0200, M.-A. Lemburg wrote: >> On 04.05.2016 14:15, Steven D'Aprano wrote: >>> On Wed, May 04, 2016 at 01:56:56PM +0200, M.-A. Lemburg wrote: >>> >>>> Issuing a warning for this would probably help, but raising >>>> an error introduced a backwards incompatibility which is >>>> not warranted IMO, given how seldom such situations occur in >>>> practice. > > I agree with you that an error is unjustified but a warning might be. > > >>>> For the implementation, there are two possibilities I can think >>>> of: >>>> >>>> 1. have STORE_MAP run a test for an already existing entry >>>> and issue a warning > > You have suggested that's too expensive, so I'm happy to rule that out. > > >>>> 2. add a final CHECK_MAP byte code to check that the number >>>> of keys in the mapping correspond to the number of keys >>>> added via the dict literal > [...] >>>> The second one is cheap regarding performance, but may not >>>> be accurate, since STORE_MAP may well be working on dynamically >>>> generated keys. It does work well for constants, and if we're >>>> just issuing a warning, may be good enough. >>> >>> I'm not sure I understand what you mean by this. Are you suggesting that >>> if I write: >>> >>> {1: None, 1: None} >>> >>> Python knows that it got two keys as argument, and so >>> CHECK_MAP can determine that there is a duplicate? (i.e. that 2 keys >>> were arguments, but the finished dict only has length 1). But if I >>> write: >>> >>> {x: None, x: None} >>> >>> Python *cannot* tell that there were two keys given (since they are >>> expressions, not constants) and CHECK_MAP can't do anything? >> >> Well, the CHECK_MAP byte code would only check whether the number >> of assignments you are doing is equal to the number of keys >> found in the final dict. That's a very coarse test and not >> perfect. It can also not tell you which key causes the >> mismatch. >> >> On the plus side, the extra check is a simply size check >> and doesn't have to be applied per added mapping. > > Sorry MAL, perhaps I'm a bit slow tonight, but I'm still lost. {x: None, > x: None} -- would CHECK_MAP detect that as a duplicate or not? > > If the answer is "Yes", then this seems to me to be a cheap way of > detecting the existence of at least one duplicate (but not which key was > duplicate). That strikes me as good enough, in which case I would > support raising a warning. Yes: >>> dis.dis('{x:None, x:None}') 1 0 BUILD_MAP 2 3 LOAD_CONST 0 (None) 6 LOAD_NAME 0 (x) 9 STORE_MAP 10 LOAD_CONST 0 (None) 13 LOAD_NAME 0 (x) 16 STORE_MAP 17 RETURN_VALUE >>> d = {x:None, x:None} >>> len(d) 1 Since 1 != 2 -> issue warning We'd just have to tell the CHECK_MAP byte code to check for len(map) == 2. The byte code compiler will be able to determine this and add the correct byte code entry. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 04 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From steve at pearwood.info Wed May 4 10:20:41 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 5 May 2016 00:20:41 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> Message-ID: <20160504142041.GI12028@ando.pearwood.info> On Wed, May 04, 2016 at 02:03:59PM +1000, Nick Coghlan wrote: > I was curious as to whether or not it was technically feasible to > implement this "duplicate keys" check solely for dict displays in > CPython without impacting other dict use cases, and it turns out it > should be. [...] > All-in-all, the concept gets a +0 from me, although to allow for > implementations that don't share CPython's distinction between dict > displays and the normal dict constructor, any such behaviour (if > introduced) should likely be flagged as an implementation detail that > may vary across interpreters (as I would have been -1 if BUILD_MAP's > implementation wasn't already independent of dict_common_update). What do you think of MAL's suggestion of adding a CHECK_MAP op-code and raising a warning? I'd be much more comfortable with a warning than an error, because I really don't think that duplicate keys is *necessarily* an error. I think Paul Moore is onto something with his example of BLUE/LIGHT_BLUE, even if he got the order the wrong way around :-) This has been the documented behaviour of dicts since at least 1.5: https://docs.python.org/release/1.5/ref/ref-7.html#HEADING7-35 and it matches the behaviour of both the dict constructor and dict comprehensions. I don't think anyone has made the case for turning it into an error, although I concede that it's perhaps worthy of a warning. (At least, I won't object to it being a warning, provided we can turn it off :-) -- Steve From mertz at gnosis.cx Wed May 4 10:36:04 2016 From: mertz at gnosis.cx (David Mertz) Date: Wed, 4 May 2016 07:36:04 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <5729E388.9080603@egenix.com> References: <20160504010920.GC12028@ando.pearwood.info> <5729E388.9080603@egenix.com> Message-ID: After following the long thread, I'm solidly -1, at least on an error. A warning wouldn't be so bad, but would still feel like a nuisance not a feature. Such a warning should belong to linters, not to the language itself. Use those to automatically catch statically detectible probably-bad patterns like duplicate literals as keys of dictionary displays. The thing that nudged me to -1 is code something like the following, which I've actually written for completely valid use-cases: def code_generate(data_source, output=open('data.py','w')): """Write data to loadable Python module data_source may contain multiple occurrences of each key. Later occurrences are "better" than earlier ones """ print("data = {", file=output) for key, val in data_source: print(" %s: %s," % (key, val), file=output) print("}", file=output) I really don't want the Python interpreter to complain about my perfectly valid 'data.py' module that I auto-generated knowing full well it might have duplicate literal keys. Yes, of course I could write a more complex function that explicitly pruned keys if they were duplicates before writing to 'data.py'. But I do not want to have to think about that when a well defined semantics of Python since version 0.9 does not require that extra work. On Wed, May 4, 2016 at 4:56 AM, M.-A. Lemburg wrote: > Issuing a warning for this would probably help, but raising > an error introduced a backwards incompatibility which is > not warranted IMO, given how seldom such situations occur in > practice. > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed May 4 10:54:17 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 May 2016 15:54:17 +0100 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <5729EE47.7010400@stoneleaf.us> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <57299841.8020503@stoneleaf.us> <5729BB71.9060703@btinternet.com> <5729EE47.7010400@stoneleaf.us> Message-ID: On 4 May 2016 at 13:42, Ethan Furman wrote: > On 05/04/2016 02:48 AM, Paul Moore wrote: > >> The problem is that it could also break code that works at the moment. >> Consider a mapping of constants to names: >> >> colors = { >> RED: "Red", >> BLUE: "Blue", >> PINK: "Pink", >> LIGHT_BLUE: "Light Blue", >> DARK_BLUE: "Dark Blue", >> } >> >> On a certain output device, there may be no "light blue", and in that >> case the code sets LIGHT_BLUE = BLUE. That's entirely reasonable, and >> the author may be perfectly happy with the current behavior in that >> case. > > Actually, that code is setting BLUE = "Light Blue". ;) You missed my point, I think. BLUE and LIGHT_BLUE are "constant" values (set once at the top of the program). In some circumstances, the programmer deliberately wants BLUE and LIGHT_BLUE to be the same value (but not in others). The programmer is happy that a constant BLUE will be decoded as "Light Blue" by the colors mapping in that case. This is IMO perfectly valid code, and having it rejected by an over-zealous "duplicate check" is unacceptable. As far as I am concerned, this example clearly demonstrates that any duplicate check should only be applied to actual constant literal values used as keys. >> So I think it's important for backward compatibility that this change >> be *only* for the case of literal values as keys (which is what the OP >> proposed, of course). And given that it needs to just be for literal >> values as keys, I personally think that makes it a case of "Special >> cases aren't special enough to break the rules", so I'm -0 on the >> proposal (I don't think it's a *bad* thing, in the constants-only >> form, just that it's not worth the effort - but if someone else makes >> the effort, I won't object). > > I object. Having only /some/ duplicates checked for is overcomplicating > without very good reason. > > Perhaps the way forward is to check for all duplicates, but issue a Warning > instead of raising an Exception? Warning on perfectly good code is just as obnoxious IMO. > To be honest, though, I'm starting to agree with Steven D'Aprano that > sufficiently large dicts should be verified by other means, and those other > means can also check for duplicates. As far as I'm aware, the people who have looked at the implementation details seem to be saying that checking for duplicates on literals only is too hard to be practical, and that's the only variation I'd be willing to accept, so I agree - do duplicate checking by other means (assertions in the code being the obvious option). Paul From rosuav at gmail.com Wed May 4 10:54:22 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 May 2016 00:54:22 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <20160504010920.GC12028@ando.pearwood.info> <5729E388.9080603@egenix.com> Message-ID: On Thu, May 5, 2016 at 12:36 AM, David Mertz wrote: > The thing that nudged me to -1 is code something like the following, which > I've actually written for completely valid use-cases: > > def code_generate(data_source, output=open('data.py','w')): > """Write data to loadable Python module > > data_source may contain multiple occurrences of each key. > > Later occurrences are "better" than earlier ones > > """ > print("data = {", file=output) > for key, val in data_source: > print(" %s: %s," % (key, val), file=output) > print("}", file=output) > > > I really don't want the Python interpreter to complain about my perfectly > valid 'data.py' module that I auto-generated knowing full well it might have > duplicate literal keys. Yes, of course I could write a more complex > function that explicitly pruned keys if they were duplicates before writing > to 'data.py'. Or, much more simply, you could just dump it straight out like this: print("data = dict(%r)" % list(data_source), file=output) If the dict display starts rejecting duplicates, these kinds of code generators should be easily tweakable. The only risk would be if duplicates can *theoretically* happen, but are incredibly rare, such that the code runs for years without anyone noticing. This is a way of catching compile-time-detectable errors. Python has generally been pretty helpful with that kind of thing, even to the point of special-casing print-used-as-a-statement to tell you to put parens around it. While I fully understand that there are reasons against doing this (including "that's the job of linters, not the core language"), I do _not_ understand the apparent attitude that it's a fundamentally bad thing to do. Most cases where a dict display is used, it's intended to have unique keys. ChrisA From guido at python.org Wed May 4 11:16:53 2016 From: guido at python.org (Guido van Rossum) Date: Wed, 4 May 2016 08:16:53 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <20160504010920.GC12028@ando.pearwood.info> <5729E388.9080603@egenix.com> Message-ID: All this discussion is certainly very educational for the participants, but I want to remind everyone that this is not going to happen without a PEP. Personally I'm -0 on the whole business. Some quick notes: - In ABC, this was considered, and the rule was that {k1: v1; k2: v2} was allowed to have k1==k2 as long as v1==v2. Otherwise it was an error. (But ABC did not have comprehensions.) - Franklin? brought up sets and suggested the same thing should happen there. I'm not so sure though. When teaching Python it's nice to be able to demonstrate the fundamental property of sets by showing it: >>> {1, 1, 1} {1} >>> -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From klahnakoski at mozilla.com Wed May 4 11:27:14 2016 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Wed, 4 May 2016 11:27:14 -0400 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <20160504010920.GC12028@ando.pearwood.info> References: <20160504010920.GC12028@ando.pearwood.info> Message-ID: <7a99cdec-028e-d951-7357-4c9c26d795b8@mozilla.com> On 5/3/2016 9:09 PM, Steven D'Aprano wrote: > On Mon, May 02, 2016 at 02:36:35PM -0700, Luigi Semenzato wrote: > >> The original problem description: >> >> lives_in = { 'lion': ['Africa', 'America'], >> 'parrot': ['Europe'], >> #... 100+ more rows here >> 'lion': ['Europe'], >> #... 100+ more rows here >> } >> >> The above constructor overwrites the first 'lion' entry silently, >> often causing unexpected behavior. > Did your colleague really have 200+ items in the dict? No matter, I > suppose. The same principle applies. > > When you have significant amount of data in a dict (or any other data > structure, such as a list, tree, whatever), the programmer has to take > responsibility for the data validation. Not the compiler. Out of all the > possible errors, why is "duplicate key" so special? Your colleague could > have caused unexpected behaviour in many ways: I often use large literal dicts, with literal string keys. There are many times (a couple times a month) I add a duplicate key because I am prone to making mistakes like that. Now, I am lucky because I purchased an IDE that highlights those duplicates as errors. If I had to rely on a separate linter; with my duplicate keys hiding in a long list of other trivial formatting mistakes, then the time cost of reviewing the linting output on every code change is greater than the time cost of simply debugging the program when I see it inevitably malfunction. Instead, I would use data validation in my own code. > The data validation need not be a big burden. In my own code, unless the > dict is so small that I can easily see that it is correct with my own > eyes, I always follow it with an assertion: > > assert len(lives_in) == 250 > I would not use this check, it is error prone, given the number of times I update the dicts during development. Either I will loose count, or the line math is wrong (because of extra spaces), or I would count the duplicate. A better consistency check requires I construct all literal dicts like: def add_item(dict_, key, value): if key in dict_: raise Exception() dict_[key] = value lives_in = {} add_item(lives_in, 'lion', ['Africa', 'America']) add_item(lives_in, 'parrot', ['Europe']) # ... 100+ more rows here add_item(lives_in, 'lion', ['Europe']) # ... 100+ more rows here Which is inelegant, but necessary to minimize the programmer time wasted trying to detect these duplicate keys manually. From michael.selik at gmail.com Wed May 4 11:29:40 2016 From: michael.selik at gmail.com (Michael Selik) Date: Wed, 04 May 2016 15:29:40 +0000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <20160504010920.GC12028@ando.pearwood.info> <5729E388.9080603@egenix.com> Message-ID: On Wed, May 4, 2016 at 11:19 AM Guido van Rossum wrote: > - In ABC, this was considered, and the rule was that {k1: v1; k2: v2} was > allowed to have k1==k2 as long as v1==v2. Otherwise it was an error. (But > ABC did not have comprehensions.) > Was that enforced by comparing the expressions or the resulting values? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed May 4 11:32:58 2016 From: guido at python.org (Guido van Rossum) Date: Wed, 4 May 2016 08:32:58 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <20160504010920.GC12028@ando.pearwood.info> <5729E388.9080603@egenix.com> Message-ID: I honestly don't recall and I assume the docs are vague, but "comparing expressions" was not a thing in ABC. I'm sure they compared the values. (This is why I showed it with variables instead of constants.) On Wed, May 4, 2016 at 8:29 AM, Michael Selik wrote: > > > On Wed, May 4, 2016 at 11:19 AM Guido van Rossum wrote: > >> - In ABC, this was considered, and the rule was that {k1: v1; k2: v2} was >> allowed to have k1==k2 as long as v1==v2. Otherwise it was an error. (But >> ABC did not have comprehensions.) >> > > Was that enforced by comparing the expressions or the resulting values? > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed May 4 11:45:55 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 May 2016 01:45:55 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <7a99cdec-028e-d951-7357-4c9c26d795b8@mozilla.com> References: <20160504010920.GC12028@ando.pearwood.info> <7a99cdec-028e-d951-7357-4c9c26d795b8@mozilla.com> Message-ID: On Thu, May 5, 2016 at 1:27 AM, Kyle Lahnakoski wrote: > Now, I am lucky because I purchased > an IDE that highlights those duplicates as errors. If I had to rely on > a separate linter... For the record, an IDE with the capability to highlight this kind of error *is* a linter. When language discussions result in "let a linter pick that up", that includes smart editors. ChrisA From michael.selik at gmail.com Wed May 4 11:57:57 2016 From: michael.selik at gmail.com (Michael Selik) Date: Wed, 04 May 2016 15:57:57 +0000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <20160504010920.GC12028@ando.pearwood.info> <5729E388.9080603@egenix.com> Message-ID: On Wed, May 4, 2016 at 10:59 AM Chris Angelico wrote: > On Thu, May 5, 2016 at 12:36 AM, David Mertz wrote: > > The thing that nudged me to -1 is code something like the following, > which > > I've actually written for completely valid use-cases: > > def code_generate(data_source, output=open('data.py','w')): > > This is a way of catching compile-time-detectable errors. Python has > generally been pretty helpful with that kind of thing, even to the > point of special-casing print-used-as-a-statement to tell you to put > parens around it. While I fully understand that there are reasons > against doing this (including "that's the job of linters, not the core > language"), I do _not_ understand the apparent attitude that it's a > fundamentally bad thing to do. Most cases where a dict display is > used, it's intended to have unique keys. > You accidentally snipped out Mertz' statement that he didn't want the interpreter to complain about "probably-bad" patterns. This implies the distinction that linters should complain about probably-bad things while the interpreter only complains about always-bad things. I guess you could argue that warnings exist so that the interpreter can complain about probably-bad things as well. If that's what you'd like, we shouldn't call it a "compile-time-detectable error" but instead, uh, ... hard to find the right vocabulary here. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed May 4 12:06:24 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 04 May 2016 09:06:24 -0700 Subject: [Python-ideas] creating dicts with duplicates on purpose [was Re: dictionary constructor should not allow duplicate keys] In-Reply-To: References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <57299841.8020503@stoneleaf.us> <5729BB71.9060703@btinternet.com> <5729EE47.7010400@stoneleaf.us> Message-ID: <572A1E00.30406@stoneleaf.us> On 05/04/2016 07:54 AM, Paul Moore wrote: > On 4 May 2016 at 13:42, Ethan Furman wrote: >> On 05/04/2016 02:48 AM, Paul Moore wrote: >>> The problem is that it could also break code that works at the moment. >>> Consider a mapping of constants to names: >>> >>> colors = { >>> RED: "Red", >>> BLUE: "Blue", >>> PINK: "Pink", >>> LIGHT_BLUE: "Light Blue", >>> DARK_BLUE: "Dark Blue", >>> } >>> >>> On a certain output device, there may be no "light blue", and in that >>> case the code sets LIGHT_BLUE = BLUE. That's entirely reasonable, and >>> the author may be perfectly happy with the current behavior in that >>> case. >> >> Actually, that code is setting BLUE = "Light Blue". ;) > > You missed my point, I think. BLUE and LIGHT_BLUE are "constant" > values (set once at the top of the program). No, I got your point, and I agree. But one of us is confused about your explanation of that example (it could easily be me). Here's how I think it works: RED = 1 BLUE = 2 PINK = 3 LIGHT_BLUE = 4 DARK_BLUE = 5 # oops, light blue not supported on this machine, just use blue LIGHT_BLUE = 2 >>> colors = { RED: "Red", BLUE: "Blue", PINK: "Pink", LIGHT_BLUE: "Light Blue", DARK_BLUE: "Dark Blue", } >>> colors {1: 'Red', 2: 'Light Blue', 3: 'Pink', 5: 'Dark Blue'} So my issue was not with your point, but with your explanation of the example of your point. -- ~Ethan~ From ethan at stoneleaf.us Wed May 4 12:09:22 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 04 May 2016 09:09:22 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <20160504142041.GI12028@ando.pearwood.info> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <20160504142041.GI12028@ando.pearwood.info> Message-ID: <572A1EB2.6040004@stoneleaf.us> On 05/04/2016 07:20 AM, Steven D'Aprano wrote: > On Wed, May 04, 2016 at 02:03:59PM +1000, Nick Coghlan wrote: > This has been the documented behaviour of dicts since at least 1.5: > > https://docs.python.org/release/1.5/ref/ref-7.html#HEADING7-35 > > and it matches the behaviour of both the dict constructor and dict > comprehensions. Okay, we had dict displays and dicts, and now you say dict constructor... do you mean dict()? Because: >>> dict(a=1, a=1) File "", line 1 SyntaxError: keyword argument repeated is certainly not allowing duplicates. -- ~Ethan~ From klahnakoski at mozilla.com Wed May 4 12:15:40 2016 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Wed, 4 May 2016 12:15:40 -0400 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <20160504010920.GC12028@ando.pearwood.info> <7a99cdec-028e-d951-7357-4c9c26d795b8@mozilla.com> Message-ID: <9919f770-7461-c108-b519-3de6dea267ed@mozilla.com> On 5/4/2016 11:45 AM, Chris Angelico wrote: > On Thu, May 5, 2016 at 1:27 AM, Kyle Lahnakoski wrote: >> Now, I am lucky because I purchased >> an IDE that highlights those duplicates as errors. If I had to rely on >> a separate linter... > For the record, an IDE with the capability to highlight this kind of > error *is* a linter. When language discussions result in "let a linter > pick that up", that includes smart editors. Thinking about examples [1], it turns out I was mistaken. My IDE does NOT have an *integrated* linter that detects duplicate literal keys! My IDE does detect duplicate literal keys for Javascript, which I obviously got confused. Oh dear! That means my dicts are full of duplicates! https://github.com/klahnakoski/TestLog-ETL/blob/d7e8a99f7532a0551c98dfdccefe02c129a6123a/testlog_etl/imports/buildbot.py#L524 From steve at pearwood.info Wed May 4 12:24:13 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 5 May 2016 02:24:13 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <572A1EB2.6040004@stoneleaf.us> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <20160504142041.GI12028@ando.pearwood.info> <572A1EB2.6040004@stoneleaf.us> Message-ID: <20160504162413.GK12028@ando.pearwood.info> On Wed, May 04, 2016 at 09:09:22AM -0700, Ethan Furman wrote: > On 05/04/2016 07:20 AM, Steven D'Aprano wrote: > >and it matches the behaviour of both the dict constructor and dict > >comprehensions. > > Okay, we had dict displays and dicts, and now you say dict > constructor... do you mean dict()? Because: I was thinking of dict with a list of tuples as argument: py> dict([(1, 'a'), (2, 'b'), (1, 'c')]) {1: 'c', 2: 'b'} or an object with a keys method: py> class K: ... def keys(self): ... return [1, 2, 1] ... def __getitem__(self, i): ... return 'abcd'[i] ... py> dict(K()) {1: 'b', 2: 'c'} -- Steve From p.f.moore at gmail.com Wed May 4 12:42:33 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 May 2016 17:42:33 +0100 Subject: [Python-ideas] creating dicts with duplicates on purpose [was Re: dictionary constructor should not allow duplicate keys] In-Reply-To: <572A1E00.30406@stoneleaf.us> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <57299841.8020503@stoneleaf.us> <5729BB71.9060703@btinternet.com> <5729EE47.7010400@stoneleaf.us> <572A1E00.30406@stoneleaf.us> Message-ID: On 4 May 2016 at 17:06, Ethan Furman wrote: >>>> colors > {1: 'Red', 2: 'Light Blue', 3: 'Pink', 5: 'Dark Blue'} > > So my issue was not with your point, but with your explanation of the > example of your point. Ah, sorry. I hadn't checked and wasn't sure myself whether the first pairing or the last took priority. I tried to be vague enough that it didn't matter that I wasn't sure (my theoretical programmer who relied on this behaviour clearly *would* be sure :-)) but I obviously failed. The moral of the story is to check my examples before posting :-) Paul From luigi at semenzato.com Wed May 4 13:37:00 2016 From: luigi at semenzato.com (Luigi Semenzato) Date: Wed, 4 May 2016 10:37:00 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <20160504162413.GK12028@ando.pearwood.info> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <20160504142041.GI12028@ando.pearwood.info> <572A1EB2.6040004@stoneleaf.us> <20160504162413.GK12028@ando.pearwood.info> Message-ID: It's all about typing and reading. Suppose Python didn't have a special syntax for dict displays. Then folks would likely write this: dict(( (1, 11), (2, 22), ... )) but then suppose someone says: "hey, folks do this a lot, let's make it more convenient" and introduce a curly braces notation, and suppose it allows overwriting, like it does now. The new notation exists *exclusively* to make it easier to type and read dictionaries in the code---by a human. Except that with these semantics it makes it impossible to detect duplicate key errors, which the vast majority of users would prefer to notice (a bold and unproven statement, but a reasonable guess). What's worse, people don't realize the problem until they have many large dict displays, and many bugs. So they feel annoyed and betrayed. Suppose instead that the new notation does NOT allow overwriting. If folks rely on the overwrite semantics, in either human-written or machine-generated code, they will immediately notice the problem and use dict(), with very little inconvenience to code generators. (And some to the humans, but remember, they are a tiny minority :) The strict semantics are maybe marginally harder to explain, and maybe marginally harder to implement, but neither should be a show-stopper. However, that's not where we are. There is the legacy code issue, and the legacy mindset. I don't know how to evaluate these issues, and I don't even know how much experience the community has with backward-incompatible changes. So, while I am fairly positive that it would have been a better choice to start with, I have no idea whether, when, and how this proposal should have any effect. P.S. The issue of detecting duplicate keys at parse time is completely separate and it was my mistake to focus on it first. P.P.S. The issue of whether a set display should allow duplicate elements is less important, since allowing them does not have destructive consequences. From michael.selik at gmail.com Wed May 4 13:59:00 2016 From: michael.selik at gmail.com (Michael Selik) Date: Wed, 04 May 2016 17:59:00 +0000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <20160504142041.GI12028@ando.pearwood.info> <572A1EB2.6040004@stoneleaf.us> <20160504162413.GK12028@ando.pearwood.info> Message-ID: On Wed, May 4, 2016, 1:37 PM Luigi Semenzato wrote: > It's all about typing and reading. > > Suppose Python didn't have a special syntax for dict displays. Then > folks would likely write this: > > dict(( > (1, 11), > (2, 22), > ... > )) > > but then suppose someone says: "hey, folks do this a lot, let's make > it more convenient" and introduce a curly braces notation, and suppose > it allows overwriting, like it does now. > > The new notation exists *exclusively* to make it easier to type and > read dictionaries in the code---by a human. Except that with these > semantics it makes it impossible to detect duplicate key errors, which > the vast majority of users would prefer to notice (a bold and unproven > statement, but a reasonable guess). I'm part of the minority(?) that doesn't care. I hope that helps adjust your odds ratio for that guess. What's worse, people don't > realize the problem until they have many large dict displays, and many > bugs. So they feel annoyed and betrayed. > > Suppose instead that the new notation does NOT allow overwriting. If > folks rely on the overwrite semantics, in either human-written or > machine-generated code, they will immediately notice the problem and > use dict(), with very little inconvenience to code generators. (And > some to the humans, but remember, they are a tiny minority :) > Even a tiny minority of the Python community might be thousands of people. > The strict semantics are maybe marginally harder to explain, and maybe > marginally harder to implement, but neither should be a show-stopper. > > However, that's not where we are. There is the legacy code issue, and > the legacy mindset. I don't know how to evaluate these issues, and I > don't even know how much experience the community has with > backward-incompatible changes. Well, there's that transition from 2 to 3. You do that a few times and you get a reputation for instability. I like to imagine a hidden line in the Zen of Python. Between the title and "Beautiful is better..." I mentally insert, "Backwards compatibility is important." How important, well, that's why it's Zen and not Rules. So, while I am fairly positive that it > would have been a better choice to start with, I have no idea whether, > when, and how this proposal should have any effect. > > P.S. The issue of detecting duplicate keys at parse time is > completely separate and it was my mistake to focus on it first. > > P.P.S. The issue of whether a set display should allow duplicate > elements is less important, since allowing them does not have > destructive consequences. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Wed May 4 15:28:26 2016 From: mertz at gnosis.cx (David Mertz) Date: Wed, 4 May 2016 12:28:26 -0700 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <20160504142041.GI12028@ando.pearwood.info> <572A1EB2.6040004@stoneleaf.us> <20160504162413.GK12028@ando.pearwood.info> Message-ID: On May 4, 2016 10:59 AM, "Michael Selik" wrote: > > On Wed, May 4, 2016, 1:37 PM Luigi Semenzato wrote: >> read dictionaries in the code---by a human. Except that with these >> semantics it makes it impossible to detect duplicate key errors, which >> the vast majority of users would prefer to notice (a bold and unproven >> statement, but a reasonable guess). > > I'm part of the minority(?) that doesn't care. I hope that helps adjust your odds ratio for that guess. I'm also part of that group (I question whether it is minority) who wishes to allow duplicate keys at the syntax level, but to let linters complain. While *most* duplicate literal keys in dict displays probably are bugs, the legitimate and intended uses are not rare enough to prohibit them altogether. -------------- next part -------------- An HTML attachment was scrubbed... URL: From klahnakoski at mozilla.com Wed May 4 15:58:58 2016 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Wed, 4 May 2016 15:58:58 -0400 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> <5727C8B5.8040205@lucidity.plus.com> <5727CF4C.5000201@lucidity.plus.com> <5727E241.4020007@lucidity.plus.com> Message-ID: <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> Please excuse my nomenclature. I hope the community can correct the synonyms that clarify my proposal. Problem ------- I program defensively, and surround many of my code blocks with try blocks to catch expected and unexpected errors. Those unexpected errors seem to dominate in my code; I never really know how many ways my SQL library can fail, nor am I really sure that a particular key is in a `dict()`. Most of the time I can do nothing about those unexpected errors; I simply chain them, with some extra description about what the code block was attempting to do. I am using 2.7, so I have made my own convention for chaining exceptions. 3.x chains more elegantly: for t in todo: try: # do error prone stuff except Exception, e: raise ToDoError("oh dear!") from e The ?error prone stuff? can itself have more try blocks to catch known failure modes, maybe deal with them. Add some `with` blocks and a conditional, and the nesting gets ugly: def process_todo(todo): try: with Timer("todo processing"): # pre-processing for t in todo: try: # do error prone stuff except Exception, e: raise TodoError("oh dear!") from e # post-processing except Exception, e: raise OverallTodoError("Not expected") from e Not only is my code dominated by exception handling, the meaningful code is deeply nested. Solution -------- I would like Python to have a bare `except` statement, which applies from that line, to the end of enclosing block (or next `except` statement). Here is the same example using the new syntax: def process_todo(todo): except Exception, e: raise OverallTodoError("Not expected") from e with Timer("todo processing"): # pre-processing for t in todo: except Exception, e: raise TodoError("oh dear!") from e # do error prone stuff # post-processing Larger code blocks do a better job of portraying he visual impact of the reduced indentation. I would admit that some readability is lost because the error handling code precedes the happy path, but I believe the eye will overlook this with little practice. Multiple `except` statements are allowed. They apply as if they were used in a `try` statement; matched in the order declared: def process_todo(todo): pre_processing() # has no exception handling except SQLException, e: # effective until end of method raise Exception("Not expected") from e except Exception, e: raise OverallTodoError("Oh dear!") from e processing() A code block can have more than one `except` statement: def process_todo(todo): pre_processing() # no exception handling except SQLException, e: # covers lines from here to beginning of next except statement raise Exception("Not expected") from e except Exception, e: # catches other exception types raise Exception("Oh dear!") from e processing() # Exceptions caught except SQLException, e: # covers a lines to end of method raise Exception("Happens, sometimes") from e post_processing() # SQLException caught, but not Exception In these cases, a whole new block is effectively defined. Here is the same in legit Python: def process_todo(todo): pre_processing() # no exception handling try: processing() # Exceptions caught except SQLException, e: # covers all lines from here to beginning of next except statement raise Exception("Not expected") from e except Exception, e: # catches other exception types raise Exception("Oh dear!") from e try: post_processing() # SQLException caught, but not Exception except SQLException, e: # covers a lines to end of method raise Exception("Happens, sometimes") from e Other Thoughts -------------- I only propose this for replacing `try` blocks that have no `else` or `finally` clause. I am not limiting my proposal to exception chaining; Anything allowed in `except` clause would be allowed. I could propose adding `except` clauses to each of the major statement types (def, for, if, with, etc?). which would make the first example look like: def process_todo(todo): with Timer("todo processing"): # pre-processing for t in todo: # do error prone stuff except Exception, e: raise TodoError("oh dear!") from e # post-processing except Exception, e: raise OverallTodoError("Not expected") from e But, I am suspicious this is more complicated than it looks to implement, and the `except` statement does seem visually detached from the block it applies to. Thank you for your consideration! From 4kir4.1i at gmail.com Wed May 4 16:34:48 2016 From: 4kir4.1i at gmail.com (Akira Li) Date: Wed, 04 May 2016 23:34:48 +0300 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <20160504135012.GG12028@ando.pearwood.info> Message-ID: <8760ut4l5j.fsf@gmail.com> Steven D'Aprano writes: > How do dict displays construct the dict? Do they add keys from left to > right, or right to left? I can read the source, if I can find it, and > locate the right section of the right file, and if I can read C. Or I > can do: > > py> {1: "left", 1: "right"} > {1: 'right'} > > and in one line of code in the Python interactive interpreter I've > learned more than in probably an hour of digging through C code. The language reference explicitly says that the order from left to right and duplicate keys are also explicitly supported [1]: If a comma-separated sequence of key/datum pairs is given, they are evaluated from left to right to define the entries of the dictionary: each key object is used as a key into the dictionary to store the corresponding datum. This means that you can specify the same key multiple times in the key/datum list, and the final dictionary?s value for that key will be the last one given. [1] https://docs.python.org/3/reference/expressions.html#dictionary-displays Akira From random832 at fastmail.com Wed May 4 16:51:25 2016 From: random832 at fastmail.com (Random832) Date: Wed, 04 May 2016 16:51:25 -0400 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> <5727C8B5.8040205@lucidity.plus.com> <5727CF4C.5000201@lucidity.plus.com> <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> Message-ID: <1462395085.2738043.598361697.1BA830D6@webmail.messagingengine.com> On Wed, May 4, 2016, at 15:58, Kyle Lahnakoski wrote: > I would like Python to have a bare `except` statement, which applies > from that line, to the end of enclosing block (or next `except` > statement). Where does control resume if it doesn't raise? From klahnakoski at mozilla.com Wed May 4 18:34:04 2016 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Wed, 4 May 2016 18:34:04 -0400 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: <1462395085.2738043.598361697.1BA830D6@webmail.messagingengine.com> References: <57266698.7020602@bign.nl> <57270307.1040809@bign.nl> <57274BA2.3030907@sdamon.com> <5727C8B5.8040205@lucidity.plus.com> <5727CF4C.5000201@lucidity.plus.com> <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <1462395085.2738043.598361697.1BA830D6@webmail.messagingengine.com> Message-ID: <2b95dd95-3a8f-1119-4651-510320d4b110@mozilla.com> On 5/4/2016 4:51 PM, Random832 wrote: > Where does control resume if it doesn't raise? Control would resume just before the end of the block, not after it, much like in the case of a `try` statement: for t in todo: pre_process() except Exception, e: print "problem"+unicode(e) process() Would be the same as for t in todo: pre_process() try: process() except Exception, e: print "problem"+unicode(e) so, the control resumes at the end of the `for` block, but still inside the loop From robert at bign.nl Wed May 4 18:45:02 2016 From: robert at bign.nl (Robert van Geel) Date: Thu, 5 May 2016 00:45:02 +0200 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: References: Message-ID: <572A7B6E.4040503@bign.nl> It looks frankly a bit like an "on error goto errorhandler" statement but with a few drawbacks to that. How can you indicate the 'exception handled block' should end, when there's no indentation level involved? It would force the exception block to service all code until the end of that indentation level but what happens on within the indentation level is defined by other more functional criteria. The proposed construct could not replace the normal try/catch syntax because of that, but it can also not mix with the normal syntax well because it's too similar to it and hence confusing. The gain of the proposed construct seems to be to save 1 indentation level. Even when this would be worth the cost it would be more logical to still put the except block at the end of the functional block. So instead of the proposed: def process_todo(todo): except Exception, e: raise OverallTodoError("Not expected") from e with Timer("todo processing"): # pre-processing for t in todo: except Exception, e: raise TodoError("oh dear!") from e # do error p the below is more logical, since the main code remains the focus and the exception the afterthought: def process_todo(todo): with Timer("todo processing"): # pre-processing for t in todo: except Exception, e: raise TodoError("oh dear!") from e # do error p except Exception, e: raise OverallTodoError("Not expected") from e But this all leaves something implicit rather then implicit: the start (or in the proposal: the end) of the exception block. Robert > Message: 3 > Date: Wed, 4 May 2016 15:58:58 -0400 > From: Kyle Lahnakoski > To: python-ideas at python.org > Subject: [Python-ideas] Block-Scoped Exception Handlers > Message-ID: <00d21501-60fd-d756-c8a6-906a8b235221 at mozilla.com> > Content-Type: text/plain; charset=windows-1252 > > > Please excuse my nomenclature. I hope the community can correct the > synonyms that clarify my proposal. > > Problem > ------- > > I program defensively, and surround many of my code blocks with try > blocks to catch expected and unexpected errors. Those unexpected > errors seem to dominate in my code; I never really know how many ways my > SQL library can fail, nor am I really sure that a particular key is in a > `dict()`. Most of the time I can do nothing about those unexpected > errors; I simply chain them, with some extra description about what the > code block was attempting to do. > > I am using 2.7, so I have made my own convention for chaining > exceptions. 3.x chains more elegantly: > > for t in todo: > try: > # do error prone stuff > except Exception, e: > raise ToDoError("oh dear!") from e > > The ?error prone stuff? can itself have more try blocks to catch known > failure modes, maybe deal with them. Add some `with` blocks and a > conditional, and the nesting gets ugly: > > def process_todo(todo): > try: > with Timer("todo processing"): > # pre-processing > for t in todo: > try: > # do error prone stuff > except Exception, e: > raise TodoError("oh dear!") from e > # post-processing > except Exception, e: > raise OverallTodoError("Not expected") from e > > Not only is my code dominated by exception handling, the meaningful code > is deeply nested. > > > Solution > -------- > > I would like Python to have a bare `except` statement, which applies > from that line, to the end of enclosing block (or next `except` > statement). Here is the same example using the new syntax: > > def process_todo(todo): > except Exception, e: > raise OverallTodoError("Not expected") from e > > with Timer("todo processing"): > # pre-processing > for t in todo: > except Exception, e: > raise TodoError("oh dear!") from e > > # do error prone stuff > # post-processing > > Larger code blocks do a better job of portraying he visual impact of the > reduced indentation. I would admit that some readability is lost > because the error handling code precedes the happy path, but I believe > the eye will overlook this with little practice. > > Multiple `except` statements are allowed. They apply as if they were > used in a `try` statement; matched in the order declared: > > def process_todo(todo): > pre_processing() # has no exception handling > > except SQLException, e: # effective until end of method > raise Exception("Not expected") from e > except Exception, e: > raise OverallTodoError("Oh dear!") from e > > processing() > > A code block can have more than one `except` statement: > > def process_todo(todo): > pre_processing() # no exception handling > > except SQLException, e: # covers lines from here to beginning > of next except statement > raise Exception("Not expected") from e > except Exception, e: # catches other exception types > raise Exception("Oh dear!") from e > > processing() # Exceptions caught > > except SQLException, e: # covers a lines to end of method > raise Exception("Happens, sometimes") from e > > post_processing() # SQLException caught, but not Exception > > In these cases, a whole new block is effectively defined. Here is the > same in legit Python: > > def process_todo(todo): > pre_processing() # no exception handling > > try: > processing() # Exceptions caught > except SQLException, e: # covers all lines from here to > beginning of next except statement > raise Exception("Not expected") from e > except Exception, e: # catches other exception types > raise Exception("Oh dear!") from e > > try: > post_processing() # SQLException caught, but not Exception > except SQLException, e: # covers a lines to end of method > raise Exception("Happens, sometimes") from e > > Other Thoughts > -------------- > > I only propose this for replacing `try` blocks that have no `else` or > `finally` clause. I am not limiting my proposal to exception chaining; > Anything allowed in `except` clause would be allowed. > > I could propose adding `except` clauses to each of the major statement > types (def, for, if, with, etc?). which would make the first example > look like: > > def process_todo(todo): > with Timer("todo processing"): > # pre-processing > for t in todo: > # do error prone stuff > except Exception, e: > raise TodoError("oh dear!") from e > > # post-processing > except Exception, e: > raise OverallTodoError("Not expected") from e > > But, I am suspicious this is more complicated than it looks to > implement, and the `except` statement does seem visually detached from > the block it applies to. > > > Thank you for your consideration! > > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > > ------------------------------ > > End of Python-ideas Digest, Vol 114, Issue 35 > ********************************************* From cs at zip.com.au Wed May 4 19:05:53 2016 From: cs at zip.com.au (cs at zip.com.au) Date: Thu, 5 May 2016 09:05:53 +1000 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> References: <5727C8B5.8040205@lucidity.plus.com> <5727CF4C.5000201@lucidity.plus.com> <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> Message-ID: <20160504230553.GA22544@cskk.homeip.net> On 04May2016 15:58, Kyle Lahnakoski wrote: >Please excuse my nomenclature. I hope the community can correct the >synonyms that clarify my proposal. > >Problem >------- > >I program defensively, and surround many of my code blocks with try >blocks to catch expected and unexpected errors. Those unexpected >errors seem to dominate in my code; I never really know how many ways my >SQL library can fail, nor am I really sure that a particular key is in a >`dict()`. Most of the time I can do nothing about those unexpected >errors; I simply chain them, with some extra description about what the >code block was attempting to do. I also like context from close to where the exception occurred, while doing that catching at whatever the suitable outer layer may be, as normal. Let me show you what I do... I have a module "cs.logutils": https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/logutils.py which is also on PyPI, thus "pip install"able. It has a context manager called "Pfx", short for "prefix". I use it like this: from cs.logutils import Pfx ... with Pfx(filename): with open(filename) as fp: for lineno, line in enumerate(fp, 1): with Pfx(lineno): ... do stuff here ... If an exception occurs within a Pfx context manager, its message attributes get prepended with the strings of the active Pfx instances, joined by ": ". Then it is reriased and handled exaxtly as if there were no Pfxs in play. So if some ValueError occured while processing line 3 of the file "foo.txt" the ValueError's message would start "foo.txt: 3: " and proceed with the core ValueError message. This provides me with cheap runtime context for all exceptions, with minimal boilerplate in my code. All the catch-and-reraise stuff happens in the __exit__ method of the innermost Pfx instance, if an exception occurs. When no exceptions occur all it is doing is maintaining a thread local stack of current message prefixes. It looks like this might address your concerns without adding things to Python itself. You could certainly make a case for annotating the exceptions with an arbitrary extra object with whatever structured state eg a dict); suggestions there welcome. What do you think of this approach to your concerns? Cheers, Cameron Simpson >I am using 2.7, so I have made my own convention for chaining >exceptions. 3.x chains more elegantly: > > for t in todo: > try: > # do error prone stuff > except Exception, e: > raise ToDoError("oh dear!") from e > >The ?error prone stuff? can itself have more try blocks to catch known >failure modes, maybe deal with them. Add some `with` blocks and a >conditional, and the nesting gets ugly: > > def process_todo(todo): > try: > with Timer("todo processing"): > # pre-processing > for t in todo: > try: > # do error prone stuff > except Exception, e: > raise TodoError("oh dear!") from e > # post-processing > except Exception, e: > raise OverallTodoError("Not expected") from e > >Not only is my code dominated by exception handling, the meaningful code >is deeply nested. > > >Solution >-------- > >I would like Python to have a bare `except` statement, which applies >from that line, to the end of enclosing block (or next `except` >statement). Here is the same example using the new syntax: > > def process_todo(todo): > except Exception, e: > raise OverallTodoError("Not expected") from e > > with Timer("todo processing"): > # pre-processing > for t in todo: > except Exception, e: > raise TodoError("oh dear!") from e > > # do error prone stuff > # post-processing > >Larger code blocks do a better job of portraying he visual impact of the >reduced indentation. I would admit that some readability is lost >because the error handling code precedes the happy path, but I believe >the eye will overlook this with little practice. > >Multiple `except` statements are allowed. They apply as if they were >used in a `try` statement; matched in the order declared: > > def process_todo(todo): > pre_processing() # has no exception handling > > except SQLException, e: # effective until end of method > raise Exception("Not expected") from e > except Exception, e: > raise OverallTodoError("Oh dear!") from e > > processing() > >A code block can have more than one `except` statement: > > def process_todo(todo): > pre_processing() # no exception handling > > except SQLException, e: # covers lines from here to beginning >of next except statement > raise Exception("Not expected") from e > except Exception, e: # catches other exception types > raise Exception("Oh dear!") from e > > processing() # Exceptions caught > > except SQLException, e: # covers a lines to end of method > raise Exception("Happens, sometimes") from e > > post_processing() # SQLException caught, but not Exception > >In these cases, a whole new block is effectively defined. Here is the >same in legit Python: > > def process_todo(todo): > pre_processing() # no exception handling > > try: > processing() # Exceptions caught > except SQLException, e: # covers all lines from here to >beginning of next except statement > raise Exception("Not expected") from e > except Exception, e: # catches other exception types > raise Exception("Oh dear!") from e > > try: > post_processing() # SQLException caught, but not Exception > except SQLException, e: # covers a lines to end of method > raise Exception("Happens, sometimes") from e > >Other Thoughts >-------------- > >I only propose this for replacing `try` blocks that have no `else` or >`finally` clause. I am not limiting my proposal to exception chaining; >Anything allowed in `except` clause would be allowed. > >I could propose adding `except` clauses to each of the major statement >types (def, for, if, with, etc?). which would make the first example >look like: > > def process_todo(todo): > with Timer("todo processing"): > # pre-processing > for t in todo: > # do error prone stuff > except Exception, e: > raise TodoError("oh dear!") from e > > # post-processing > except Exception, e: > raise OverallTodoError("Not expected") from e > >But, I am suspicious this is more complicated than it looks to >implement, and the `except` statement does seem visually detached from >the block it applies to. > > >Thank you for your consideration! > > >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- From klahnakoski at mozilla.com Wed May 4 19:09:04 2016 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Wed, 4 May 2016 19:09:04 -0400 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: <572A7B6E.4040503@bign.nl> References: <572A7B6E.4040503@bign.nl> Message-ID: <3415efd1-6467-be52-d395-8525c0faf27b@mozilla.com> Oh dear! It appears I screwed up the indentation. Since I never see the emails I send to this list, I have no idea what you all were looking at. Maybe I should have used underscores for email: def process_todo(todo): ____except Exception, e: ________raise OverallTodoError("Not expected") from e ____with Timer("todo processing"): ________# pre-processing ________for t in todo: ____________except Exception, e: ________________raise TodoError("oh dear!") from e ____________# do error prone stuff ________# post-processing The exception handling code should be indented, but the code it applies to is not. On 5/4/2016 6:45 PM, Robert van Geel wrote: > It looks frankly a bit like an "on error goto errorhandler" statement > but with a few drawbacks to that. > > How can you indicate the 'exception handled block' should end, when > there's no indentation level involved? It would force the exception > block to service all code until the end of that indentation level but > what happens on within the indentation level is defined by other more > functional criteria. The proposed construct could not replace the > normal try/catch syntax because of that, but it can also not mix with > the normal syntax well because it's too similar to it and hence > confusing. > > The gain of the proposed construct seems to be to save 1 indentation > level. Even when this would be worth the cost it would be more logical > to still put the except block at the end of the functional block. So > instead of the proposed: > > def process_todo(todo): > except Exception, e: > raise OverallTodoError("Not expected") from e > with Timer("todo processing"): > # pre-processing > for t in todo: > except Exception, e: > raise TodoError("oh dear!") from e > > # do error p > > the below is more logical, since the main code remains the focus and > the exception the afterthought: > > def process_todo(todo): > with Timer("todo processing"): > # pre-processing > for t in todo: > except Exception, e: > raise TodoError("oh dear!") from e > > # do error p > > except Exception, e: > raise OverallTodoError("Not expected") from e > > But this all leaves something implicit rather then implicit: the start > (or in the proposal: the end) of the exception block. > > Robert > >> Message: 3 >> Date: Wed, 4 May 2016 15:58:58 -0400 >> From: Kyle Lahnakoski >> To: python-ideas at python.org >> Subject: [Python-ideas] Block-Scoped Exception Handlers >> Message-ID: <00d21501-60fd-d756-c8a6-906a8b235221 at mozilla.com> >> Content-Type: text/plain; charset=windows-1252 >> >> >> Please excuse my nomenclature. I hope the community can correct the >> synonyms that clarify my proposal. >> >> Problem >> ------- >> >> I program defensively, and surround many of my code blocks with try >> blocks to catch expected and unexpected errors. Those unexpected >> errors seem to dominate in my code; I never really know how many ways my >> SQL library can fail, nor am I really sure that a particular key is in a >> `dict()`. Most of the time I can do nothing about those unexpected >> errors; I simply chain them, with some extra description about what the >> code block was attempting to do. >> >> I am using 2.7, so I have made my own convention for chaining >> exceptions. 3.x chains more elegantly: >> >> for t in todo: >> try: >> # do error prone stuff >> except Exception, e: >> raise ToDoError("oh dear!") from e >> >> The ?error prone stuff? can itself have more try blocks to catch known >> failure modes, maybe deal with them. Add some `with` blocks and a >> conditional, and the nesting gets ugly: >> def process_todo(todo): >> try: >> with Timer("todo processing"): >> # pre-processing >> for t in todo: >> try: >> # do error prone stuff >> except Exception, e: >> raise TodoError("oh dear!") from e >> # post-processing >> except Exception, e: >> raise OverallTodoError("Not expected") from e >> >> Not only is my code dominated by exception handling, the meaningful code >> is deeply nested. >> >> >> Solution >> -------- >> >> I would like Python to have a bare `except` statement, which applies >> from that line, to the end of enclosing block (or next `except` >> statement). Here is the same example using the new syntax: >> >> def process_todo(todo): >> except Exception, e: >> raise OverallTodoError("Not expected") from e >> with Timer("todo processing"): >> # pre-processing >> for t in todo: >> except Exception, e: >> raise TodoError("oh dear!") from e >> >> # do error prone stuff >> # post-processing >> >> Larger code blocks do a better job of portraying he visual impact of the >> reduced indentation. I would admit that some readability is lost >> because the error handling code precedes the happy path, but I believe >> the eye will overlook this with little practice. >> >> Multiple `except` statements are allowed. They apply as if they were >> used in a `try` statement; matched in the order declared: >> >> def process_todo(todo): >> pre_processing() # has no exception handling >> >> except SQLException, e: # effective until end of method >> raise Exception("Not expected") from e >> except Exception, e: >> raise OverallTodoError("Oh dear!") from e >> >> processing() >> >> A code block can have more than one `except` statement: >> >> def process_todo(todo): >> pre_processing() # no exception handling >> except SQLException, e: # covers lines from here to >> beginning >> of next except statement >> raise Exception("Not expected") from e >> except Exception, e: # catches other exception types >> raise Exception("Oh dear!") from e >> processing() # Exceptions caught >> except SQLException, e: # covers a lines to end of method >> raise Exception("Happens, sometimes") from e >> post_processing() # SQLException caught, but not Exception >> In these cases, a whole new block is effectively defined. Here >> is the >> same in legit Python: >> >> def process_todo(todo): >> pre_processing() # no exception handling >> try: >> processing() # Exceptions caught >> except SQLException, e: # covers all lines from here to >> beginning of next except statement >> raise Exception("Not expected") from e >> except Exception, e: # catches other exception types >> raise Exception("Oh dear!") from e >> try: >> post_processing() # SQLException caught, but not Exception >> except SQLException, e: # covers a lines to end of method >> raise Exception("Happens, sometimes") from e >> Other Thoughts >> -------------- >> >> I only propose this for replacing `try` blocks that have no `else` or >> `finally` clause. I am not limiting my proposal to exception chaining; >> Anything allowed in `except` clause would be allowed. >> I could propose adding `except` clauses to each of the major statement >> types (def, for, if, with, etc?). which would make the first example >> look like: >> >> def process_todo(todo): >> with Timer("todo processing"): >> # pre-processing >> for t in todo: >> # do error prone stuff >> except Exception, e: >> raise TodoError("oh dear!") from e >> # post-processing >> except Exception, e: >> raise OverallTodoError("Not expected") from e >> >> But, I am suspicious this is more complicated than it looks to >> implement, and the `except` statement does seem visually detached from >> the block it applies to. >> >> >> Thank you for your consideration! >> >> >> >> ------------------------------ >> >> Subject: Digest Footer >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> >> >> ------------------------------ >> >> End of Python-ideas Digest, Vol 114, Issue 35 >> ********************************************* > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From klahnakoski at mozilla.com Wed May 4 19:15:39 2016 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Wed, 4 May 2016 19:15:39 -0400 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: <20160504230553.GA22544@cskk.homeip.net> References: <5727C8B5.8040205@lucidity.plus.com> <5727CF4C.5000201@lucidity.plus.com> <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> Message-ID: <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> On 5/4/2016 7:05 PM, cs at zip.com.au wrote: > I also like context from close to where the exception occurred, while > doing that catching at whatever the suitable outer layer may be, as > normal. Let me show you what I do... > > I have a module "cs.logutils": > I really like the idea of using a `with` clause for simplify exception chaining. I am concerned I would be missing some of the locals available at exception time, which the `with` clause would not have access to, but more introspection may solve that too. It does not solve the indent problem, but I could live with that if it made the code simpler in other ways. Many of my exception handlers are multiline, and I do not think the `with` clause strategy would work there. From klahnakoski at mozilla.com Wed May 4 19:29:37 2016 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Wed, 4 May 2016 19:29:37 -0400 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: <572A7B6E.4040503@bign.nl> References: <572A7B6E.4040503@bign.nl> Message-ID: <63bdeaad-3bdd-5a08-7add-ef4102a14e75@mozilla.com> To mitigate my indentation disaster, I posted the document on Github. https://github.com/klahnakoski/Block-Scoped-Exception-Handlers On 5/4/2016 6:45 PM, Robert van Geel wrote: > How can you indicate the 'exception handled block' should end, when > there's no indentation level involved? It would force the exception > block to service all code until the end of that indentation level but > what happens on within the indentation level is defined by other more > functional criteria. The proposed construct could not replace the > normal try/catch syntax because of that, but it can also not mix with > the normal syntax well because it's too similar to it and hence > confusing. > From ncoghlan at gmail.com Wed May 4 20:41:53 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 5 May 2016 10:41:53 +1000 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> References: <5727C8B5.8040205@lucidity.plus.com> <5727CF4C.5000201@lucidity.plus.com> <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> Message-ID: On 5 May 2016 9:16 am, "Kyle Lahnakoski" wrote: > > > On 5/4/2016 7:05 PM, cs at zip.com.au wrote: > > I also like context from close to where the exception occurred, while > > doing that catching at whatever the suitable outer layer may be, as > > normal. Let me show you what I do... > > > > I have a module "cs.logutils": > > > > I really like the idea of using a `with` clause for simplify exception > chaining. I am concerned I would be missing some of the locals > available at exception time, which the `with` clause would not have > access to, but more introspection may solve that too. It does not solve > the indent problem, but I could live with that if it made the code > simpler in other ways. > > Many of my exception handlers are multiline, and I do not think the > `with` clause strategy would work there. In combination with contextlib.contextmanager (and perhaps passing in references to relevant locals), with statements are designed to handle factoring out almost arbitrary exception handling. Chaining a consistent error, for example: @contextmanager def chain_errors(exc_to_raise): try: yield except Exception as e: raise exc_to_raise from e This is most useful for supplying state that's useful for debugging purposes (e.g. the codec infrastructure tries to do something like that in order to report the codec name) Cheers, Nick. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed May 4 21:36:05 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 4 May 2016 21:36:05 -0400 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> Message-ID: On 5/4/2016 12:03 AM, Nick Coghlan wrote: > On 4 May 2016 at 11:40, Ethan Furman wrote: >> On 05/03/2016 05:23 PM, Steven D'Aprano wrote: >>> I'm intentionally not giving you the values of a, b or c, or telling >>> you what spam() returns. Now you have the same information available >>> to you as the compiler has at compile time. What do you intend to do? >> >> Since the dict created by that dict display happens at run time, I am >> suggesting that during the process of creating that dict that any keys, >> however generated or retrieved, that are duplicates of keys already in the >> dict, cause an appropriate error (to be determined). > > I was curious as to whether or not it was technically feasible to > implement this "duplicate keys" check solely for dict displays in > CPython without impacting other dict use cases, and it turns out it > should be. > > The key point is that BUILD_MAP already has its own PyDict_SetItem() > loop in ceval.c (iterating over stack entries), and hence doesn't rely > on the "dict_common_update" code shared by dict.update and the dict > constructor(s). Changing only BUILD_MAP would invalidate current code equivalences and currently sensible optimizations. A toy example: >>> d1 = {'a': 'a1'} >>> d2 = {f(): 'a2'} >>> d1.update(d2) >>> d1 {'a': 'a2'} Sensible and comprehensible code transformation rule are important. Currently, the following rather trivial optimization of the above works. >>> d1 = {'a': 'a1', f(): 'a2'} >>> d1 {'a': 'a2'} The special rule for dict displays would invalidate this. In my post yesterday in response to Luigi (where I began 'The thread...'), I gave 4 equivalent other ways to initialize a dict using a Python loop (include a dict comprehension). Using a dict display amounts to un-rolling any of the loops and replacing the Python loop with the C loop buried in ceval. Changing the operation of just that loop would break the current equivalence. I suspect that the proposed change would introduce more bugs than it exposes. -- Terry Jan Reedy From tjreedy at udel.edu Wed May 4 21:49:47 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 4 May 2016 21:49:47 -0400 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <20160503235120.GA12028@ando.pearwood.info> Message-ID: On 5/3/2016 11:31 PM, Chris Angelico wrote: > On Wed, May 4, 2016 at 1:23 PM, Terry Reedy wrote: >> Why single out just one to raise an exception? My guess is that it is >> partly from mistakenly thinking of the one as a like a lexer literal rather >> than as runtime code. This was a preface to the main points that followed, and which were snipped from this followup. > But the lexer's definition of "literal" is extremely narrow, and > doesn't reflect the way people think about them. Most people's idea of > a literal is closer to what ast.literal_eval can accept: [snip] I am obviously aware of that. My main point, not addressed by this response, is that 1. dict displays are currently equivalence to several forms of Python code with more explicit looping, including at least two that are commonly used to initialize dicts; and 2. that changing the semantics of dict displays and only dict displays will break this equivalence. In my response just not to Nick, I was more explicit that breaking this equivalence now, after 20+ years, will break code that depends on the current equivalence and break people's currently true expectations. -- Terry Jan Reedy From michael.selik at gmail.com Wed May 4 22:17:25 2016 From: michael.selik at gmail.com (Michael Selik) Date: Thu, 05 May 2016 02:17:25 +0000 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: References: <5727C8B5.8040205@lucidity.plus.com> <5727CF4C.5000201@lucidity.plus.com> <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> Message-ID: On Wed, May 4, 2016, 8:42 PM Nick Coghlan wrote: > @contextmanager > def chain_errors(exc_to_raise): > try: > yield > except Exception as e: > raise exc_to_raise from e > Unfortunately, Kyle is using Python 2.7 still, so ``raise from`` won't help him. The exception context/cause is probably my favorite Python 3 feature. Or at least in my top 5. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed May 4 22:29:51 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 5 May 2016 12:29:51 +1000 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: References: <5727C8B5.8040205@lucidity.plus.com> <5727CF4C.5000201@lucidity.plus.com> <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> Message-ID: On 5 May 2016 at 12:17, Michael Selik wrote: > On Wed, May 4, 2016, 8:42 PM Nick Coghlan wrote: >> >> @contextmanager >> def chain_errors(exc_to_raise): >> try: >> yield >> except Exception as e: >> raise exc_to_raise from e > > > Unfortunately, Kyle is using Python 2.7 still, so ``raise from`` won't help > him. The exception context/cause is probably my favorite Python 3 feature. > Or at least in my top 5. For feature proposals on python-ideas it's the current capabilities on 3.x that matter and there, between exception chaining, contextlib.contextmanager, and contextlib.ExitStack, there are already some enormously powerful tools for exception stack manipulation without extensive code duplication. Python 2.7 doesn't have the implicit exception chaining, but it does have the other features (including ExitStack, by way of contextlib2). While the standard traceback display functions wouldn't show it, even explicit exception chaining can be emulated on Python 2.x (since that's mainly just a matter of setting the __cause__ attribute appropriately and using traceback2 to attach and display gc-friendly __traceback__ attributes ) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Wed May 4 22:31:49 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 5 May 2016 12:31:49 +1000 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: References: <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> Message-ID: <20160505023149.GL12028@ando.pearwood.info> On Thu, May 05, 2016 at 02:17:25AM +0000, Michael Selik wrote: > Unfortunately, Kyle is using Python 2.7 still, so ``raise from`` won't help > him. If Kyle is using Python 2.7, then a new feature which is only introduced to 3.6 or 3.7 isn't going to help him either. -- Steve From rob.cliffe at btinternet.com Wed May 4 22:38:05 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 5 May 2016 03:38:05 +0100 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> Message-ID: <572AB20D.1070403@btinternet.com> On 05/05/2016 02:36, Terry Reedy wrote: > On 5/4/2016 12:03 AM, Nick Coghlan wrote: >> On 4 May 2016 at 11:40, Ethan Furman wrote: >>> On 05/03/2016 05:23 PM, Steven D'Aprano wrote: >>>> I'm intentionally not giving you the values of a, b or c, or telling >>>> you what spam() returns. Now you have the same information available >>>> to you as the compiler has at compile time. What do you intend to do? >>> >>> Since the dict created by that dict display happens at run time, I am >>> suggesting that during the process of creating that dict that any keys, >>> however generated or retrieved, that are duplicates of keys already >>> in the >>> dict, cause an appropriate error (to be determined). >> >> I was curious as to whether or not it was technically feasible to >> implement this "duplicate keys" check solely for dict displays in >> CPython without impacting other dict use cases, and it turns out it >> should be. >> >> The key point is that BUILD_MAP already has its own PyDict_SetItem() >> loop in ceval.c (iterating over stack entries), and hence doesn't rely >> on the "dict_common_update" code shared by dict.update and the dict >> constructor(s). > > Changing only BUILD_MAP would invalidate current code equivalences and > currently sensible optimizations. A toy example: > > >>> d1 = {'a': 'a1'} > >>> d2 = {f(): 'a2'} > >>> d1.update(d2) > >>> d1 > {'a': 'a2'} > > Sensible and comprehensible code transformation rule are important. > Currently, the following rather trivial optimization of the above works. > > >>> d1 = {'a': 'a1', f(): 'a2'} > >>> d1 > {'a': 'a2'} > > The special rule for dict displays would invalidate this. > > In my post yesterday in response to Luigi (where I began 'The > thread...'), I gave 4 equivalent other ways to initialize a dict using > a Python loop (include a dict comprehension). Using a dict display > amounts to un-rolling any of the loops and replacing the Python loop > with the C loop buried in ceval. Changing the operation of just that > loop would break the current equivalence. > > I suspect that the proposed change would introduce more bugs than it > exposes. The OP mentioned (even if he didn't explicitly produce it, understandable if it was very long) a real-life use case where a warning/error would have aided debugging. I find this case realistic. Can anyone produce a single real-life use case where repeated literal keys needed to be accepted without a warning or error? Here I'm throwing down the gauntlet to those who theorise about what (breakable) code *might* be out in the wild, and asking them to produce just one real-life case. Rob Cliffe From michael.selik at gmail.com Wed May 4 23:01:00 2016 From: michael.selik at gmail.com (Michael Selik) Date: Thu, 05 May 2016 03:01:00 +0000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <572AB20D.1070403@btinternet.com> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <572AB20D.1070403@btinternet.com> Message-ID: On Wed, May 4, 2016 at 10:40 PM Rob Cliffe wrote: > The OP mentioned (even if he didn't explicitly produce it, > understandable if it was very long) a real-life use case where a > warning/error would have aided debugging. > I find this case realistic. > Certainly was. But it occurs relatively rarely and can be solved with linters and by stylistic changes like alphabetizing dict literals by key. Can anyone produce a single real-life use case where repeated literal > keys needed to be accepted without a warning or error? > Here I'm throwing down the gauntlet to those who theorise about what > (breakable) code *might* be out in the wild, and asking them to produce > just one real-life case. > I thought Paul Moore gave a good example. You can see him and Ethan Furman discussing it in a fork of this thread. There were a few code-generator examples. Were these not realistic enough? -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed May 4 23:29:08 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 5 May 2016 13:29:08 +1000 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <572AB20D.1070403@btinternet.com> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <572AB20D.1070403@btinternet.com> Message-ID: <20160505032908.GM12028@ando.pearwood.info> On Thu, May 05, 2016 at 03:38:05AM +0100, Rob Cliffe wrote: > Here I'm throwing down the gauntlet You can throw down whatever you like, but Guido has pronounced that this change is not going to happen without a PEP. So unless somebody is willing to write a PEP, further discussion is a waste of time. -- Steve From p.f.moore at gmail.com Thu May 5 04:22:26 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 5 May 2016 09:22:26 +0100 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: <572AB20D.1070403@btinternet.com> References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <572AB20D.1070403@btinternet.com> Message-ID: On 5 May 2016 at 03:38, Rob Cliffe wrote: > The OP mentioned (even if he didn't explicitly produce it, understandable if > it was very long) a real-life use case where a warning/error would have > aided debugging. > I find this case realistic. > Can anyone produce a single real-life use case where repeated literal keys > needed to be accepted without a warning or error? > Here I'm throwing down the gauntlet to those who theorise about what > (breakable) code *might* be out in the wild, and asking them to produce just > one real-life case. Not literal keys no. If the proposal is *solely* that duplicate literals are detected and objected to (I reserve judgement for now on the question of warning vs error) then I don't think there will be significant code breakage. But even knowing that, the next steps are: 1. Produce a PEP (Guido has said this is a requirement) 2. Explain how this would be implemented in that PEP - because the only implementation techniques so far mentioned in this thread fail to restrict the check to literals. 3. Manage discussion on said PEP. I'll await a PEP before contributing any further to the discussion. Paul From klahnakoski at mozilla.com Thu May 5 07:13:09 2016 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Thu, 5 May 2016 07:13:09 -0400 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: <20160505023149.GL12028@ando.pearwood.info> References: <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> Message-ID: <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> On 5/4/2016 10:31 PM, Steven D'Aprano wrote: > On Thu, May 05, 2016 at 02:17:25AM +0000, Michael Selik wrote: > >> Unfortunately, Kyle is using Python 2.7 still, so ``raise from`` won't help >> him. > If Kyle is using Python 2.7, then a new feature which is only introduced > to 3.6 or 3.7 isn't going to help him either. > It will help me! Eventually. :) From michael.selik at gmail.com Thu May 5 11:23:17 2016 From: michael.selik at gmail.com (Michael Selik) Date: Thu, 05 May 2016 15:23:17 +0000 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> References: <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> Message-ID: On Thu, May 5, 2016, 7:13 AM Kyle Lahnakoski wrote: > > > On 5/4/2016 10:31 PM, Steven D'Aprano wrote: > > On Thu, May 05, 2016 at 02:17:25AM +0000, Michael Selik wrote: > > > >> Unfortunately, Kyle is using Python 2.7 still, so ``raise from`` won't > help > >> him. > > If Kyle is using Python 2.7, then a new feature which is only introduced > > to 3.6 or 3.7 isn't going to help him either. > > > > It will help me! Eventually. :) > I thought you said the ``raise from`` syntax solved the problem. No? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From klahnakoski at mozilla.com Thu May 5 11:35:58 2016 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Thu, 5 May 2016 11:35:58 -0400 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: References: <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> Message-ID: On 5/5/2016 11:23 AM, Michael Selik wrote: > > > On Thu, May 5, 2016, 7:13 AM Kyle Lahnakoski > wrote: > > > > On 5/4/2016 10:31 PM, Steven D'Aprano wrote: > > On Thu, May 05, 2016 at 02:17:25AM +0000, Michael Selik wrote: > > > >> Unfortunately, Kyle is using Python 2.7 still, so ``raise > from`` won't help > >> him. > > If Kyle is using Python 2.7, then a new feature which is only > introduced > > to 3.6 or 3.7 isn't going to help him either. > > > > It will help me! Eventually. :) > > > I thought you said the ``raise from`` syntax solved the problem. No? I am jealous that Python 3.x has `raise from`, and I can not use it. `raise from` does solve the exception chaining problems in 2.7, but that can be worked around just as effectively [1]. Me being stuck in 2.7 will not last forever. `raise from` does not solve the excessive indentation problem: I have many `try` clauses, causing deep indentation in my code. The block-scoped exception handlers would mitigate this deep indentation, and make exception handling even easier to add. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu May 5 12:04:29 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 5 May 2016 17:04:29 +0100 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: References: <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> Message-ID: On 5 May 2016 at 16:35, Kyle Lahnakoski wrote: > `raise from` does not solve the excessive indentation problem: I have many > `try` clauses, causing deep indentation in my code. The block-scoped > exception handlers would mitigate this deep indentation, and make exception > handling even easier to add. contextmanager/ExitStack would likely help with the indentation problem. As would simply breaking out some of the deeply nested code into independent functions. It may be that there's a problem worth addressing here, but I suggest that you wait until you've had a chance to see if the existing features in Python 3.5 resolve your issues before taking this suggestion any further. Paul From klahnakoski at mozilla.com Thu May 5 15:17:43 2016 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Thu, 5 May 2016 15:17:43 -0400 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: References: <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> Message-ID: <9703c2ae-098d-53db-f28b-4db76b37cfa9@mozilla.com> On 5/5/2016 12:04 PM, Paul Moore wrote: > On 5 May 2016 at 16:35, Kyle Lahnakoski wrote: >> `raise from` does not solve the excessive indentation problem: I have many >> `try` clauses, causing deep indentation in my code. The block-scoped >> exception handlers would mitigate this deep indentation, and make exception >> handling even easier to add. > contextmanager/ExitStack would likely help with the indentation > problem. As would simply breaking out some of the deeply nested code > into independent functions. > May you provide me with an example of how contextmanager would help with the indentation? From what little I can glean, Python2.7 already has this, and I use it, but I do not see how replacing `try` blocks with `with` blocks reduces indentation. I do agree it looks cleaner than a `try/except` block though. I do agree that breaking out deeply nested code into independent functions can help with the indentation. I find breaking out functions that only have call site quite disappointing. My disappointment is proportional to the number of block-scoped variables. Thank you! From ncoghlan at gmail.com Thu May 5 22:17:52 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 6 May 2016 12:17:52 +1000 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: <9703c2ae-098d-53db-f28b-4db76b37cfa9@mozilla.com> References: <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> <9703c2ae-098d-53db-f28b-4db76b37cfa9@mozilla.com> Message-ID: On 6 May 2016 at 05:17, Kyle Lahnakoski wrote: > May you provide me with an example of how contextmanager would help with > the indentation? contextmanager doesn't, ExitStack does (which is in the standard library for 3.3+, and available via contextlib2 for earlier versions). > From what little I can glean, Python2.7 already has > this, and I use it, but I do not see how replacing `try` blocks with > `with` blocks reduces indentation. I do agree it looks cleaner than a > `try/except` block though. One of the cases that ExitStack handles is when you want to unwind all the contexts at the same point in the code, but enter them at different points. It does that by letting you write code like this: with ExitStack() as cm: cm.enter_context(the_first_cm) # Do some things cm.enter_context(the_second_cm) # Do some more things cm.enter_context(the_third_cm) # Do yet more things # All three context managers get unwound here The nested with equivalent would be: with the_first_cm: # Do some things with the_second_cm: # Do some more things with the_third_cm: # Do yet more things # All three context managers get unwound here As an added bonus, the ExitStack approach will also let you push arbitrary callbacks, enter contexts conditionally, and a few other things. Barry Warsaw has an excellent write-up here: http://www.wefearchange.org/2013/05/resource-management-in-python-33-or.html Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From leewangzhong+python at gmail.com Fri May 6 08:56:36 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Fri, 6 May 2016 08:56:36 -0400 Subject: [Python-ideas] dictionary constructor should not allow duplicate keys In-Reply-To: References: <57290328.8020408@stoneleaf.us> <572917B4.9000007@stoneleaf.us> <20160504002300.GB12028@ando.pearwood.info> <57295310.7080809@stoneleaf.us> <572AB20D.1070403@btinternet.com> Message-ID: Late, but here are a few more notes: 1. The gnumpy package (for Py2) used expressions in a `set` constructor which could evaluate to duplicates. It isn't hard to imagine a similar trick with `dict`, though I don't know if it'd be as useful. From gnumpy.py: ``` _floatTypes = set((types.FloatType, numpy.float64, numpy.float32, getattr(numpy, 'float128', numpy.float64), getattr(numpy, 'float96', numpy.float64))) ``` Refactored a bit: ``` _floatTypes = {float, numpy.float64, numpy.float32, getattr(numpy, 'float128', float), getattr(numpy, 'float96', float)} ``` 2. Even if the warning/error is compiler-only, `eval` and `exec` would have the same warning/error, and they are NOT always evaluating dev-written source code. (Though you can argue that `exec` and `eval` on strings are where you should be on high alert with regard to data validation.) On the other hand, a linter is only run on manually-written code. 3. A compile-time check doesn't necessarily give a false sense of security. It just needs the proper wording to indicate that it's a compile-time check. "Warning: Duplicate (constant|literal|string literal) keys detected." From klahnakoski at mozilla.com Fri May 6 10:50:30 2016 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Fri, 6 May 2016 10:50:30 -0400 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: References: <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> <9703c2ae-098d-53db-f28b-4db76b37cfa9@mozilla.com> Message-ID: <63dcceb5-f49d-2451-8048-12928bad88ae@mozilla.com> On 5/5/2016 10:17 PM, Nick Coghlan wrote: > On 6 May 2016 at 05:17, Kyle Lahnakoski wrote: >> May you provide me with an example of how contextmanager would help with >> the indentation? > contextmanager doesn't, ExitStack does (which is in the standard > library for 3.3+, and available via contextlib2 for earlier versions). Thank you for the examples, and they do reduce indention when you have callbacks defined, or dealing with resources. This does not seem to help in the case of exception handlers. Exception handlers have access to block scope variables, and have code blocks that do not require a `def`. I should have added different types of exception handlers to my initial email to distract from the chaining exception. Consider the exception handler that simply stops trying to process the todo items: |____def process_todo(todo): |||____||||____|pre_process() ||||____||||||____||||||||for t in todo: |||||____||||||________exc|||||||||||||||||||||||ept Exception, e: |||||_||||||___||||||||_||||||___||||||||_||||||___||||||||_||||||___|||break| ||||_||||||___||||||||_||||||___||||||||_||||||___|||process(t) |||||_||||||___||||||||_||||||___|||post_process() As an alternative to ||____def process_todo(todo): |||____||||____|pre_process() ||||____||||||____||||||||for t in todo: |||||____________try ||||||||||||_____||||||___||||||||_||||||___||||||||_||||||___|||process(t) |____||||||________exc|||||||||||||||||||||||ept Exception, e: |||||_||||||___||||||||_||||||___||||||||_||||||___||||||||_||||||___|||break| ||||_||||||___||||||||_||||||___|||post_process() | Of course, a couple `try` nested statements make the indentation worse. For example, we wish to ignore problems in dealing with todo items, like above, but the todo items are complicated; each is a dict() of details. Failure to deal with one of the details is ignored, but the rest of the details are still attempted: |def process_todo(todo): ____pre_process() ____for t in todo: ________except Exception, e: ____________break ________for u, v in t.items(): ____________except Exception, e: ________________continue ____________process() ____post_process() Which is better than what I do now: def process_todo(todo): ____pre_process() ____for t in todo: ________try: ____________for u, v in t.items(): ________________try: ____________________process() ________________except Exception, e: ____________________continue ________except Exception, e: ____________break ____post_process() | |I have not touched on more complicated except clauses; ones that have multiple lines, and ones that use, or update block-scoped variables. ExitStack is good for manipulating `with` clauses as first order objects, but they are limited to what you can do with `with` statements: Limited to naive exception handling, or simple chaining. If was to use ExitStack in the example above (using `continue` and `exit`), maybe I would write something like: def process_todo(todo): ____with ExitStack() as stack: ________pre_process() ________for t in todo: ____________stack.enter_context(BreakOnException()) ____________for u, v in t.items(): ________________stack.enter_context(ContinueOnException()) ________________process() ________________stack.pop() # DOES THIS EXIST? IT SHOULD ________post_process() | |This terrible piece of (not working) code assumes it is even possible to write a `BreakOnException` and `ContinueOnException`. It also requires a `pop` method, which is not documented, but really should exist. Without a `pop` method, I would need another ExitStack instance, and a `with` statement to hold it.| || || |I hope I have convinced you that ExitStack, and `with` block magic, does a poor job of covering the use cases for `try` statements. Maybe I should make examples that use the loop variables in the `except` clause for more examples?| || |At a high level: The `try/except/finally` is powerful structure. Realizing that the `try/finally` pair is very common leads to the creation of `with`. I am proposing the same logic for `try/catch` pair: They are commonly paired together and should have an optimized syntax. | |||| || -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.selik at gmail.com Fri May 6 13:28:46 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 06 May 2016 17:28:46 +0000 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: <63dcceb5-f49d-2451-8048-12928bad88ae@mozilla.com> References: <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> <9703c2ae-098d-53db-f28b-4db76b37cfa9@mozilla.com> <63dcceb5-f49d-2451-8048-12928bad88ae@mozilla.com> Message-ID: On Fri, May 6, 2016 at 10:50 AM Kyle Lahnakoski wrote: > > Of course, a couple `try` nested statements make the indentation worse. > For example, we wish to ignore problems in dealing with todo items, like > above, but the todo items are complicated; each is a dict() of details. > Failure to deal with one of the details is ignored, but the rest of the > details are still attempted: > > def process_todo(todo): > ____pre_process() > ____for t in todo: > ________try: > ____________for u, v in t.items(): > ________________try: > ____________________process() > ________________except Exception, e: > ____________________continue > ________except Exception, e: > ____________break > ____post_process() > > The refactoring that first comes to mind is to make the inner portion a separate function: def process_one(thing): for u, v in thing.items(): try: process() except Exception as e: continue def process_group(group): pre_process() for thing in group: try: process_one(thing) except Exception as e: break post_process() This avoids the excessive indentation and helps improve the reading of how an error in the group will break but an error in one will continue. Does that not satisfy? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri May 6 13:44:12 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 06 May 2016 10:44:12 -0700 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: <63dcceb5-f49d-2451-8048-12928bad88ae@mozilla.com> References: <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> <9703c2ae-098d-53db-f28b-4db76b37cfa9@mozilla.com> <63dcceb5-f49d-2451-8048-12928bad88ae@mozilla.com> Message-ID: <572CD7EC.6030204@stoneleaf.us> -1 on the whole idea. It's make code much less readable, and is way too implicit for my tastes, and, I think, for Python. My own dbf [1] module is roughly 10,000 lines, and I maintain a private copy of OpenERP which is at least 10 times that size, and no where does either system have so many nested try/except handlers. -- ~Ethan~ [1] https://pypi.python.org/pypi/dbf From klahnakoski at mozilla.com Fri May 6 13:56:19 2016 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Fri, 6 May 2016 13:56:19 -0400 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: References: <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> <9703c2ae-098d-53db-f28b-4db76b37cfa9@mozilla.com> <63dcceb5-f49d-2451-8048-12928bad88ae@mozilla.com> Message-ID: On 5/6/2016 1:28 PM, Michael Selik wrote: > > This avoids the excessive indentation and helps improve the reading of > how an error in the group will break but an error in one will > continue. Does that not satisfy? Yes, extracting deeply nested code into methods will reduce the indentation. It is disappointing that I must come up with a name for a method that has only one call site. More disappointment for each local variable I must pass to that method,. More disappointment for each variable the code bock update this locals. -------------- next part -------------- An HTML attachment was scrubbed... URL: From klahnakoski at mozilla.com Fri May 6 15:15:12 2016 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Fri, 6 May 2016 15:15:12 -0400 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: <572CD7EC.6030204@stoneleaf.us> References: <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> <9703c2ae-098d-53db-f28b-4db76b37cfa9@mozilla.com> <63dcceb5-f49d-2451-8048-12928bad88ae@mozilla.com> <572CD7EC.6030204@stoneleaf.us> Message-ID: On 5/6/2016 1:44 PM, Ethan Furman wrote: > -1 on the whole idea. > > It's make code much less readable, and is way too implicit for my > tastes, and, I think, for Python. > > My own dbf [1] module is roughly 10,000 lines, and I maintain a > private copy of OpenERP which is at least 10 times that size, and no > where does either system have so many nested try/except handlers. This is a good point. There are probably domains that have clear inputs, or have a mature codebase, with tests covering all input permutations. These domains do not need `try` statements to cover the unknown. Maybe these domains dominate the universe of source code, and I am in the minority. I can easily be convince this is the case: I have seen lots of code that really does not care if an exception gets raised. Understanding why it failed is a real pain, for there are no descriptions, no summary, original causes are not chained, or if they are, the stack trace is missing. My code has no hope of mitigating those errors: It can not retry on HTTP errors, or provide useful feedback if the original cause is a missing file. Your strategy of simply not using `try` statements, may also work. Although, I do not know how you trace down the cause of errors on production systems easily without them. Mature software will not have as many logic errors as my green code, so the cost of chasing down a problem is better amortized, and it more reasonable to leave out `try` blocks. For example, from ver_33.py (in Record._retrieve_field_value), ________try: ____________if null_data[byte] >> bit & 1: ________________return Null ________except IndexError: ____________print(null_data) ____________print(index) ____________print(byte, bit) ____________print(len(self._data), self._data) ____________print(null_def) ____________print(null_data) ____________raise It is not obvious to me that IndexError is the only Exception that can come from here. This code may raise file access exceptions, HTTP exceptions, I do not know. I would, at least, add a `raise Exception` clause to catch those unknown situations. Furthermore, since I do not know how deep the stack will be on those exceptions, I would chain-and-raise ________try: ____________if null_data[byte] >> bit & 1: ________________return Null ________except Exception, e: ____________raise CantGetValueFromFieldException( ________________null_data, ________________index, ________________byte, bit, ________________(len(self._data), self._data), ________________null_def, ________________null_data ____________) from e This is better than letting the SQLExceptions just propagate; I have described what I am doing (CantGetValueFromFieldException), so I can switch on it in a later exception handler, and I still have the original cause, which I can switch on also. Looking at Record.__getattr__(), and Record.__getitem__(), they use the above method, and can raise any number of other exceptions. This means the code that uses this library must catch those possible exceptions. Too many for me to keep track of now. I will catch them all, make some decisions, and re-raise the ones I can not deal with. def do_stuff(**kwargs): ____try: ________my_data = Record(**kwargs) ________value = my_data["value"] ________send(value) ____catch Exception, e: ________if in_causal_chain(SQLException, e): ____________Log.warning("Database problem, not doing anymore stuff", cause=e) ________elif in_causal_chain(CantGetValueFromFieldException, e): ____________raise AppearsToBeADataAccessProblem() from e ________else: ____________raise AlternatProblem() from e Thank you for the time you spend on this subject. From michael.selik at gmail.com Fri May 6 15:38:01 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 06 May 2016 19:38:01 +0000 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: References: <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> <9703c2ae-098d-53db-f28b-4db76b37cfa9@mozilla.com> <63dcceb5-f49d-2451-8048-12928bad88ae@mozilla.com> Message-ID: On Fri, May 6, 2016 at 1:56 PM Kyle Lahnakoski wrote: > On 5/6/2016 1:28 PM, Michael Selik wrote: > > This avoids the excessive indentation and helps improve the reading of how > an error in the group will break but an error in one will continue. Does > that not satisfy? > > Yes, extracting deeply nested code into methods will reduce the > indentation. It is disappointing that I must come up with a name for a > method that has only one call site. More disappointment for each local > variable I must pass to that method,. More disappointment for each > variable the code bock update this locals. > It is difficult to come up with a good name sometimes, but I feel that effort often gives me a better understanding of my code. I don't mind that there's only one call site. In fact, many of the functions I write only have one call site. I often break code out to a function so that I can give it a name and make the code easier to read. I share your distaste for passing the same set of arguments to a function, it's helper, that helper's helper, and so on down the chain. When it gets frustrating, that's more incentive to refactor. Sometimes it pushes me to realize a better design. It's often a sign of too much interdependence, which is hard to reason about. Regarding your proposal, even if it were a pleasant syntax, I think the alternatives are good enough that having both would go against the Zen of one obvious way. -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Fri May 6 19:57:42 2016 From: songofacandy at gmail.com (INADA Naoki) Date: Sat, 7 May 2016 08:57:42 +0900 Subject: [Python-ideas] Fwd: Block-Scoped Exception Handlers In-Reply-To: References: <5727E241.4020007@lucidity.plus.com> <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> <9703c2ae-098d-53db-f28b-4db76b37cfa9@mozilla.com> <63dcceb5-f49d-2451-8048-12928bad88ae@mozilla.com> Message-ID: > ____def process_todo(todo):________pre_process()________for t in todo:____________except Exception, e:________________break____________process(t)________post_process() > > > As an alternative to > ____def process_todo(todo):________pre_process()________for t in todo:____________try________________process(t)____________except Exception, e:________________break________post_process() > > I don't feel former is quite readable than later. Of course, a couple `try` nested statements make the indentation worse. > For example, we wish to ignore problems in dealing with todo items, like > above, but the todo items are complicated; each is a dict() of details. > Failure to deal with one of the details is ignored, but the rest of the > details are still attempted: > > def process_todo(todo): > ____pre_process() > ____for t in todo: > ________except Exception, e: > ____________break > ________for u, v in t.items(): > ____________except Exception, e: > ________________continue > ____________process() > ____post_process() > > Which is better than what I do now: > > def process_todo(todo): > ____pre_process() > ____for t in todo: > ________try: > ____________for u, v in t.items(): > ________________try: > ____________________process() > ________________except Exception, e: > ____________________continue > ________except Exception, e: > ____________break > ____post_process() > > > While later is more nested, it seems straightforward. Additionally, try-catch pair should have minimum try block unlike try-finally pair. I am afraid of proposed syntax sugar misleads people to expand try block. At a high level: The `try/except/finally` is powerful structure. > Realizing that the `try/finally` pair is very common leads to the creation > of `with`. I am proposing the same logic for `try/catch` pair: They are > commonly paired together and should have an optimized syntax. > > try-catch pair shouldn't used so many like try-finally pairs. When function call is very nested, only function near top level should catch Exceptions. Most of middle~bottom level functions use only finally (or with statement, off course). If many functions have catch clause, it's a very bad code smell. C# and Java have syntax sugar only for try-finally pair too. (using statement in C# and try with resource in Java). -- INADA Naoki -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri May 6 20:39:39 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 7 May 2016 10:39:39 +1000 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: References: <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> <9703c2ae-098d-53db-f28b-4db76b37cfa9@mozilla.com> <63dcceb5-f49d-2451-8048-12928bad88ae@mozilla.com> <572CD7EC.6030204@stoneleaf.us> Message-ID: On Sat, May 7, 2016 at 5:15 AM, Kyle Lahnakoski wrote: > This is a good point. There are probably domains that have clear > inputs, or have a mature codebase, with tests covering all input > permutations. These domains do not need `try` statements to cover the > unknown. Maybe these domains dominate the universe of source code, and > I am in the minority. I can easily be convince this is the case: I > have seen lots of code that really does not care if an exception gets > raised. Understanding why it failed is a real pain, for there are no > descriptions, no summary, original causes are not chained, or if they > are, the stack trace is missing. My code has no hope of mitigating > those errors: It can not retry on HTTP errors, or provide useful > feedback if the original cause is a missing file. > > Your strategy of simply not using `try` statements, may also work. > Although, I do not know how you trace down the cause of errors on > production systems easily without them. Mature software will not have > as many logic errors as my green code, so the cost of chasing down a > problem is better amortized, and it more reasonable to leave out `try` > blocks. It's nothing to do with the maturity of the codebase. It's more a question of effort-for-effort. You have a couple of options: 1) Put in heaps of effort up front, and during code editing; or 2) Put in a bit more effort in debugging. When you first write code, don't bother with any of the extra boiler-plate. Just let the exceptions propagate as they are, and worry about debugging when you get to it. Later on, follow the basic Rule of Three: if you've done the same thing three times, put in some effort to make it easier (because something you do three times is likely to happen a fourth). Most of your code won't need extra exception info - the traceback will serve you just fine. Once you've had three examples of some particular loop tripping you up (because your debugging work is harder due to not knowing which iteration of the loop raised the exception), you know where to put in a simple exception-chaining block: import random class InfoCarrier(Exception): pass for i in range(30): x = random.randrange(20) try: y = 1/x except: raise InfoCarrier("x = %s" % x) You almost certainly do _not_ need this kind of construct all through your code; that's too much effort in code maintenance for not enough benefit in debugging. If you really think you need this kind of locals inspection everywhere, pick up one of the execution frameworks that lets you do this - I think ipython does? - and have none of it in your code at all. You're using Python. So stop writing so much code. :) ChrisA From klahnakoski at mozilla.com Fri May 6 20:58:19 2016 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Fri, 6 May 2016 20:58:19 -0400 Subject: [Python-ideas] Fwd: Block-Scoped Exception Handlers In-Reply-To: References: <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> <9703c2ae-098d-53db-f28b-4db76b37cfa9@mozilla.com> <63dcceb5-f49d-2451-8048-12928bad88ae@mozilla.com> Message-ID: On 5/6/2016 7:57 PM, INADA Naoki wrote: > If many functions have catch clause, it's a very bad code smell. This is interesting, and the ?fourth? time it was alluded to! Maybe the lack of `try` statements is common, and people can still debug problems despite the loss of causes and stack traces. Another discussion made me realize what I am doing differently: My problems may be unique because of multi-threaded code and logging. I have programmed using multiple threads for a very long time, and have been catching my exceptions early-and-often so they, and their causes, are in a single structure. My exceptions are emitted to the log (with all causes and traces) all at once or not at all. This is necessary when multiple threads are writing to the log; writing out of order, and interlacing with other threads' log lines. Now, I mostly write Python, but I still spawn many threads despite the GIL. My exception handling strategy, and my logging strategy, has not changed. I see now. The log generated by a single thread would be much more readable, with cause preceding effect, and without obfuscating lines from other threads. -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Fri May 6 21:12:51 2016 From: songofacandy at gmail.com (INADA Naoki) Date: Sat, 7 May 2016 10:12:51 +0900 Subject: [Python-ideas] Fwd: Block-Scoped Exception Handlers In-Reply-To: References: <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> <9703c2ae-098d-53db-f28b-4db76b37cfa9@mozilla.com> <63dcceb5-f49d-2451-8048-12928bad88ae@mozilla.com> Message-ID: On Sat, May 7, 2016 at 9:58 AM, Kyle Lahnakoski wrote: > > On 5/6/2016 7:57 PM, INADA Naoki wrote: > > If many functions have catch clause, it's a very bad code smell. > > > This is interesting, and the ?fourth? time it was alluded to! Maybe the > lack of `try` statements is common, and people can still debug problems > despite the loss of causes and stack traces. > No. Catching at near top level doesn't loss cause and stack trace. Badly wrapped exception does loss important information. It is same to multi threaded program. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Fri May 6 21:41:24 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 07 May 2016 13:41:24 +1200 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: References: <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> <9703c2ae-098d-53db-f28b-4db76b37cfa9@mozilla.com> <63dcceb5-f49d-2451-8048-12928bad88ae@mozilla.com> <572CD7EC.6030204@stoneleaf.us> Message-ID: <572D47C4.6060509@canterbury.ac.nz> Kyle Lahnakoski wrote: > ________try: > ____________if null_data[byte] >> bit & 1: > ________________return Null > ________except IndexError: > ____________print(null_data) > ____________print(index) > ____________print(byte, bit) > ____________print(len(self._data), self._data) > ____________print(null_def) > ____________print(null_data) > ____________raise Using a suitably-defined context manager, it should be possible to write that something like this: with Activity("Getting a bit", lambda: (null_data, index, (byte, bit), (len(self._data), self._data), null_def, null_data): if null_data[byte] >> bit & 1: return Null -- Greg From ethan at stoneleaf.us Sat May 7 00:47:04 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 06 May 2016 21:47:04 -0700 Subject: [Python-ideas] Block-Scoped Exception Handlers In-Reply-To: References: <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> <9703c2ae-098d-53db-f28b-4db76b37cfa9@mozilla.com> <63dcceb5-f49d-2451-8048-12928bad88ae@mozilla.com> <572CD7EC.6030204@stoneleaf.us> Message-ID: <572D7348.2090002@stoneleaf.us> On 05/06/2016 12:15 PM, Kyle Lahnakoski wrote: > Your strategy of simply not using `try` statements, may also work. > Although, I do not know how you trace down the cause of errors on > production systems easily without them. Mature software will not have > as many logic errors as my green code, so the cost of chasing down a > problem is better amortized, and it more reasonable to leave out `try` > blocks. > > > > For example, from ver_33.py (in Record._retrieve_field_value), > > ________try: > ____________if null_data[byte] >> bit & 1: > ________________return Null > ________except IndexError: > ____________print(null_data) > ____________print(index) > ____________print(byte, bit) > ____________print(len(self._data), self._data) > ____________print(null_def) > ____________print(null_data) > ____________raise > > It is not obvious to me that IndexError is the only Exception that can > come from here. This code may raise file access exceptions, HTTP > exceptions, I do not know. I would, at least, add a `raise Exception` > clause to catch those unknown situations. Furthermore, since I do not > know how deep the stack will be on those exceptions, I would > chain-and-raise Firstly, my compliments for actually checking out the code I was referring to. I'm impressed! Secondly, that whole try/except, especially the multiple print statements, is an example of how to track down something -- but that is debugging code that I forgot take out. In other words, I was getting an IndexError, so I stuck that code in for testing, fixed the problem... and forgot to remove the code. To be fair, I was the primary consumer of that library for a long time. So, as others have said: Just write your code. When something breaks, then put in the debugging code to see what exactly is going on. If you don't already have a test suite, start one at that point: write the test that should succeed, watch it fail, fix your code, watch your test succeed, rest assured that if you break that test in the future you'll catch before you release your code in to the wild. -- ~Ethan~ From ncoghlan at gmail.com Sat May 7 09:31:20 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 7 May 2016 23:31:20 +1000 Subject: [Python-ideas] Fwd: Block-Scoped Exception Handlers In-Reply-To: References: <00d21501-60fd-d756-c8a6-906a8b235221@mozilla.com> <20160504230553.GA22544@cskk.homeip.net> <6110172f-f2a4-d694-a1ef-5ba84796e1cc@mozilla.com> <20160505023149.GL12028@ando.pearwood.info> <7c9d78fb-0cc7-952c-466a-e39b060cb872@mozilla.com> <9703c2ae-098d-53db-f28b-4db76b37cfa9@mozilla.com> <63dcceb5-f49d-2451-8048-12928bad88ae@mozilla.com> Message-ID: On 7 May 2016 10:59, "Kyle Lahnakoski" wrote: > > Another discussion made me realize what I am doing differently: My problems may be unique because of multi-threaded code and logging. I have programmed using multiple threads for a very long time, and have been catching my exceptions early-and-often so they, and their causes, are in a single structure. My exceptions are emitted to the log (with all causes and traces) all at once or not at all. This is necessary when multiple threads are writing to the log; writing out of order, and interlacing with other threads' log lines. Now, I mostly write Python, but I still spawn many threads despite the GIL. My exception handling strategy, and my logging strategy, has not changed. > > I see now. The log generated by a single thread would be much more readable, with cause preceding effect, and without obfuscating lines from other threads. In Python 2, you're also missing out on the ability to pass "stack_info=True" for any logging message to give it more context (although some 3rd party logging libraries may provide that). At the very least, traceback2 backports Python 3.5's improved stack walking features, including the ability to readily extract locals from a traceback or any other frame stack: https://docs.python.org/dev/library/traceback.html#stacksummary-objects Another feature potentially of interest to you is faulthandler.dump_tracebacks: https://docs.python.org/3/library/faulthandler.html#dumping-the-traceback (Also available via PyPI for 2.7) I'm not aware of any library yet that combines faulthandler's traceback dumping with traceback2's local variable capture, but that combination should be possible. Combine that with pyrasite, and I believe it would theoretically be possible to retrieve the repr of every local variable in every currently running frame in any process you're able to attach a remote debugger to :) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sat May 7 17:41:45 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 8 May 2016 00:41:45 +0300 Subject: [Python-ideas] Boolean parameters guidelines Message-ID: I propose to add following recommendations in PEP 8 or other documents: 1. It is preferable to pass boolean arguments as keyword arguments (if this is not the only argument). 2. It is preferable to declare boolean parameters as keyword-only parameters. What are you think about this? From joejev at gmail.com Sat May 7 18:04:52 2016 From: joejev at gmail.com (Joseph Jevnik) Date: Sat, 7 May 2016 18:04:52 -0400 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: Message-ID: +1: This is by far my biggest use case for keyword only parameters. On Sat, May 7, 2016 at 5:41 PM, Serhiy Storchaka wrote: > I propose to add following recommendations in PEP 8 or other documents: > > 1. It is preferable to pass boolean arguments as keyword arguments (if > this is not the only argument). > > 2. It is preferable to declare boolean parameters as keyword-only > parameters. > > What are you think about this? > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat May 7 21:59:33 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 8 May 2016 11:59:33 +1000 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: Message-ID: <20160508015932.GS12028@ando.pearwood.info> On Sun, May 08, 2016 at 12:41:45AM +0300, Serhiy Storchaka wrote: > I propose to add following recommendations in PEP 8 or other documents: > > 1. It is preferable to pass boolean arguments as keyword arguments (if > this is not the only argument). > > 2. It is preferable to declare boolean parameters as keyword-only > parameters. > > What are you think about this? I think it is preferable *not* to have boolean parameters at all. I don't remember if this is Guido's name for it, but I remember him expressing the guideline "No constant bool arguments". If you have an argument which takes a bool, and is used solely to switch between two different modes, and the caller will most likely call the function with the argument as a constant known when writing the code (rather than taking an expression or variable with value not known until runtime), then it is usually better to split the function into two, one for each mode. For example, in the statistics module, I could have written: def variance(data, pop=False): ... and have the `pop` argument switch between population variance and sample variance (the default). But that would fail the "No constant bool args" test, since you almost always know ahead of time which variance you want. So instead there are two separate functions, variance and pvariance. Obviously this is a guideline, not a hard rule, like all rules in PEP 8, and there may be exceptions, e.g. the closefd argument to open. So I think that if you are going to write a recommendation for treatment of bool arguments, it should start with "(1) Try to avoid bool args", explain the "No constant bool arguments" principle, and only then go on with "(2) if you still have bool arguments, then use keyword arguments..." as you suggest above. -- Steve From rosuav at gmail.com Sat May 7 22:27:18 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 8 May 2016 12:27:18 +1000 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: <20160508015932.GS12028@ando.pearwood.info> References: <20160508015932.GS12028@ando.pearwood.info> Message-ID: On Sun, May 8, 2016 at 11:59 AM, Steven D'Aprano wrote: > Obviously this is a guideline, not a hard rule, like all rules in PEP > 8, and there may be exceptions, e.g. the closefd argument to open. The difference with open() is that it's really a flag on the resulting file. It doesn't do something fundamentally different - it gives back a file object either way, and there's a hidden bit of state that says "hey, closefd was/wasn't set". There's actually a more fundamental difference between open(fn, "rb") and open(fn, "rt"), but for hysterical reasons that's all bound up in the "mode". IMO that would be better done with separate keyword arguments. (Obviously I'm not suggesting changing the existing function, but this is for PEP 8 and general advice.) > So I think that if you are going to write a recommendation for treatment > of bool arguments, it should start with "(1) Try to avoid bool args", > explain the "No constant bool arguments" principle, and only then go on > with "(2) if you still have bool arguments, then use keyword > arguments..." as you suggest above. #2 is particularly applicable if there's a whole matrix of possible options. Taking open() again, we have flag options for closefd, newline, read/write/append/update, and text/binary (these last two being packaged up into the "mode"); and more flags could easily be created in the future, same as closefd was. Creating separate functions for every combination would be ridiculously impractical, ergo the one master function is the cleaner way to do it. ChrisA From jsbueno at python.org.br Sun May 8 00:58:35 2016 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Sun, 8 May 2016 01:58:35 -0300 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: <20160508015932.GS12028@ando.pearwood.info> References: <20160508015932.GS12028@ando.pearwood.info> Message-ID: On 7 May 2016 at 22:59, Steven D'Aprano wrote: > So instead there are two separate functions, variance and > pvariance. Which, to avoid copy and pasting code, would obviously each consist of a call do a _variance private function, passing the operation mode as a ... Boolean parameter. And, if there are three booleans, one should actually write 8 different functions doing the same thing.... So, regardless of Guido or have or not having advised that, I think this is just bogus. From steve at pearwood.info Sun May 8 02:41:13 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 8 May 2016 16:41:13 +1000 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: <20160508015932.GS12028@ando.pearwood.info> Message-ID: <20160508064113.GU12028@ando.pearwood.info> On Sun, May 08, 2016 at 01:58:35AM -0300, Joao S. O. Bueno wrote: > On 7 May 2016 at 22:59, Steven D'Aprano wrote: > > So instead there are two separate functions, variance and > > pvariance. > > Which, to avoid copy and pasting code, would obviously each consist > of a call do a _variance private function, passing the operation mode as a ... > Boolean parameter. You know, the standard library is open source, and the source code is available :-) You could easily check to see whether that is what it does: https://hg.python.org/cpython/file/3.5/Lib/statistics.py Ignoring docstrings, this is the current implementation of the two functions: def variance(data, xbar=None): if iter(data) is data: data = list(data) n = len(data) if n < 2: raise StatisticsError('variance requires at least two data points') T, ss = _ss(data, xbar) return _convert(ss/(n-1), T) def pvariance(data, mu=None): if iter(data) is data: data = list(data) n = len(data) if n < 1: raise StatisticsError('pvariance requires at least one data point') ss = _ss(data, mu) T, ss = _ss(data, mu) return _convert(ss/n, T) So, no, it does not pass a hidden boolean argument. Virtually all the common code (apart from some trivial setup code) is already factored into private subfunctions, which I would have still done even if I had used a bool to set the mode. Even if I did use a private bool arg, I think that exposing it in the public interface would have been the wrong choice. Conceptually, sample variance and population variance are related but distinct things. It makes for a better API to have two distinct functions rather than passing a flag to choose between them. There are cases where using a bool to select a mode makes sense, which is why it is only a guideline, not a hard, unbending law that must always be obeyed, no exceptions. There may be cases where we choose to intentionally disregard the "No const bool args" rule, and that's okay, if we disregard it for good reasons. For example, I think that list.sort(reverse=True) is the right choice, in which case the other rules about using keyword arguments apply. But I think that we should start from a position of "No const bool args" being the preferred option. -- Steve From ncoghlan at gmail.com Sun May 8 08:14:51 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 May 2016 22:14:51 +1000 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: Message-ID: On 8 May 2016 at 07:41, Serhiy Storchaka wrote: > I propose to add following recommendations in PEP 8 or other documents: > > 1. It is preferable to pass boolean arguments as keyword arguments (if this > is not the only argument). > > 2. It is preferable to declare boolean parameters as keyword-only > parameters. > > What are you think about this? With Steven's suggestion to also articulate the "Would two functions be better?" question, I think it's a good idea. Suggested points to note: 1. Do not use a boolean toggle to change the return type of a function, define two different functions (e.g. os.walk vs os.fwalk, os.getcwd vs os.getcwdb) 2. If the boolean toggle is expected to be constant for any given call-site, consider defining two different functions (e.g. statistics.variance vs statistics.pvariance) 3. If a boolean toggle is deemed appropriate, it is preferable to make it keyword-only so call sites are always self-documenting 4. When calling a function that accepts a boolean toggle, it is preferable to pass it by keyword to make the call more self-documenting The reason I think this is worth documenting is that it comes up regularly in PEPs that require some associated API design, and these are the rules of thumb we generally apply. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From eric at trueblade.com Sun May 8 09:08:58 2016 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 8 May 2016 09:08:58 -0400 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: Message-ID: <77adf7cd-5cb6-9924-2990-7674fd733408@trueblade.com> On 5/8/2016 8:14 AM, Nick Coghlan wrote: ... > 2. If the boolean toggle is expected to be constant for any given > call-site, consider defining two different functions (e.g. > statistics.variance vs statistics.pvariance) Wouldn't that be "for every (or almost every) call-site"? Surely a few constants here and there are okay, it's when parameters are constants almost everywhere that the guidance comes in to play. > The reason I think this is worth documenting is that it comes up > regularly in PEPs that require some associated API design, and these > are the rules of thumb we generally apply. Agreed. Eric. From ncoghlan at gmail.com Sun May 8 09:16:34 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 May 2016 23:16:34 +1000 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: <77adf7cd-5cb6-9924-2990-7674fd733408@trueblade.com> References: <77adf7cd-5cb6-9924-2990-7674fd733408@trueblade.com> Message-ID: On 8 May 2016 at 23:08, Eric V. Smith wrote: > On 5/8/2016 8:14 AM, Nick Coghlan wrote: > ... > >> 2. If the boolean toggle is expected to be constant for any given >> call-site, consider defining two different functions (e.g. >> statistics.variance vs statistics.pvariance) > > Wouldn't that be "for every (or almost every) call-site"? Surely a few > constants here and there are okay, it's when parameters are constants > almost everywhere that the guidance comes in to play. Aye, that's a better way of phrasing it (it's what I meant, but I can see how "any given" could be read as "at least one", rather than "if you select a call site at random, it will be using a constant value"). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.barker at noaa.gov Mon May 9 13:57:58 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 9 May 2016 10:57:58 -0700 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: <77adf7cd-5cb6-9924-2990-7674fd733408@trueblade.com> References: <77adf7cd-5cb6-9924-2990-7674fd733408@trueblade.com> Message-ID: On Sun, May 8, 2016 at 6:08 AM, Eric V. Smith wrote: > > Wouldn't that be "for every (or almost every) call-site"? "most call sites" would be fine. The common use case matters, it's not that hard to conditionally call a different function, as long as that's the uncommon use-case. The other key here is that even if a boolean flag does essentially select between two different function, there are times where there is more than one such boolean flag for a single function, so you end up with 2**n "different" functions -- in this case, probably better to just use the flags and have one calling point. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From bzvi7919 at gmail.com Mon May 9 14:21:40 2016 From: bzvi7919 at gmail.com (Bar Harel) Date: Mon, 09 May 2016 18:21:40 +0000 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: <77adf7cd-5cb6-9924-2990-7674fd733408@trueblade.com> Message-ID: +1 on Steven D'Aprano's suggestion (And thus *Nick's* too). I have seen *way* too many complicated functions with lots of booleans that create an ugly 20 lines function call. I agree that splitting into 2 if possible is a good idea, and in the cases it's not, requesting keyword only arguments is a great solution. I do think though that in case of many possible options, using IntEnums and OR operators is the preferable method, unlike open() that uses a string, but that's again just a private opinion. Huge +1. On Mon, May 9, 2016 at 8:58 PM Chris Barker wrote: > On Sun, May 8, 2016 at 6:08 AM, Eric V. Smith wrote: > >> >> Wouldn't that be "for every (or almost every) call-site"? > > > "most call sites" would be fine. The common use case matters, it's not > that hard to conditionally call a different function, as long as that's the > uncommon use-case. > > The other key here is that even if a boolean flag does essentially select > between two different function, there are times where there is more than > one such boolean flag for a single function, so you end up with 2**n > "different" functions -- in this case, probably better to just use the > flags and have one calling point. > > -CHB > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon May 9 14:24:53 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 9 May 2016 11:24:53 -0700 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: <77adf7cd-5cb6-9924-2990-7674fd733408@trueblade.com> Message-ID: Let's just all remember that open() is a huge antipattern. Having a return type that depends on the value of an argument is terrible, and open() has two flags smushed into a single string (binary mode and read/write/both). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon May 9 15:34:09 2016 From: barry at python.org (Barry Warsaw) Date: Mon, 9 May 2016 14:34:09 -0500 Subject: [Python-ideas] Boolean parameters guidelines References: Message-ID: <20160509143409.73a50597@anarchist.wooz.org> On May 08, 2016, at 12:41 AM, Serhiy Storchaka wrote: >1. It is preferable to pass boolean arguments as keyword arguments (if this >is not the only argument). Even if it is! >2. It is preferable to declare boolean parameters as keyword-only parameters. Yes, unless the code needs to be bilingual. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Mon May 9 15:36:53 2016 From: barry at python.org (Barry Warsaw) Date: Mon, 9 May 2016 14:36:53 -0500 Subject: [Python-ideas] Boolean parameters guidelines References: <20160508015932.GS12028@ando.pearwood.info> Message-ID: <20160509143653.19d36e09@anarchist.wooz.org> On May 08, 2016, at 11:59 AM, Steven D'Aprano wrote: >I think it is preferable *not* to have boolean parameters at all. > >I don't remember if this is Guido's name for it, but I remember him >expressing the guideline "No constant bool arguments". If you have an >argument which takes a bool, and is used solely to switch between two >different modes, and the caller will most likely call the function with >the argument as a constant known when writing the code (rather than >taking an expression or variable with value not known until runtime), >then it is usually better to split the function into two, one for >each mode. I think Guido's point is stricter than that (but I don't want to put words in his mouth :). It's that a boolean flag argument shouldn't change the return type of the method. Flags can certainly change the behavior of a method and I think that's perfect fine. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From guido at python.org Mon May 9 15:47:08 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 9 May 2016 12:47:08 -0700 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: <20160509143653.19d36e09@anarchist.wooz.org> References: <20160508015932.GS12028@ando.pearwood.info> <20160509143653.19d36e09@anarchist.wooz.org> Message-ID: On Mon, May 9, 2016 at 12:36 PM, Barry Warsaw wrote: > On May 08, 2016, at 11:59 AM, Steven D'Aprano wrote: > > >I think it is preferable *not* to have boolean parameters at all. > > > >I don't remember if this is Guido's name for it, but I remember him > >expressing the guideline "No constant bool arguments". If you have an > >argument which takes a bool, and is used solely to switch between two > >different modes, and the caller will most likely call the function with > >the argument as a constant known when writing the code (rather than > >taking an expression or variable with value not known until runtime), > >then it is usually better to split the function into two, one for > >each mode. > > I think Guido's point is stricter than that (but I don't want to put words > in > his mouth :). It's that a boolean flag argument shouldn't change the > return > type of the method. Flags can certainly change the behavior of a method > and I > think that's perfect fine. > These are two different things. Any time the *value* of an argument affects the *type* of the return value there's a problem, not just for boolean parameters. E.g. open(..., "rb") vs. open(..., "r") is one of the worst offenders -- and "r" vs. "w" is another one for open()! The thing Steven quotes is specific to boolean parameters and pretty much exactly what I would say about them. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon May 9 16:00:04 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 9 May 2016 22:00:04 +0200 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: <20160508015932.GS12028@ando.pearwood.info> <20160509143653.19d36e09@anarchist.wooz.org> Message-ID: <5730EC44.7000702@egenix.com> On 09.05.2016 21:47, Guido van Rossum wrote: > On Mon, May 9, 2016 at 12:36 PM, Barry Warsaw wrote: > >> On May 08, 2016, at 11:59 AM, Steven D'Aprano wrote: >> >>> I think it is preferable *not* to have boolean parameters at all. >>> >>> I don't remember if this is Guido's name for it, but I remember him >>> expressing the guideline "No constant bool arguments". If you have an >>> argument which takes a bool, and is used solely to switch between two >>> different modes, and the caller will most likely call the function with >>> the argument as a constant known when writing the code (rather than >>> taking an expression or variable with value not known until runtime), >>> then it is usually better to split the function into two, one for >>> each mode. >> >> I think Guido's point is stricter than that (but I don't want to put words >> in >> his mouth :). It's that a boolean flag argument shouldn't change the >> return >> type of the method. Flags can certainly change the behavior of a method >> and I >> think that's perfect fine. >> > > These are two different things. Any time the *value* of an argument affects > the *type* of the return value there's a problem, not just for boolean > parameters. E.g. open(..., "rb") vs. open(..., "r") is one of the worst > offenders -- and "r" vs. "w" is another one for open()! > > The thing Steven quotes is specific to boolean parameters and pretty much > exactly what I would say about them. This seems overly strict, e.g. it's not uncommon to have functions enable debugging, verbose processing or similar processing variants using a boolean parameter. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 09 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From guido at python.org Mon May 9 16:07:41 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 9 May 2016 13:07:41 -0700 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: <5730EC44.7000702@egenix.com> References: <20160508015932.GS12028@ando.pearwood.info> <20160509143653.19d36e09@anarchist.wooz.org> <5730EC44.7000702@egenix.com> Message-ID: On Mon, May 9, 2016 at 1:00 PM, M.-A. Lemburg wrote: > On 09.05.2016 21:47, Guido van Rossum wrote: > > On Mon, May 9, 2016 at 12:36 PM, Barry Warsaw wrote: > >> On May 08, 2016, at 11:59 AM, Steven D'Aprano wrote: > >>> I think it is preferable *not* to have boolean parameters at all. > >>> > >>> I don't remember if this is Guido's name for it, but I remember him > >>> expressing the guideline "No constant bool arguments". If you have an > >>> argument which takes a bool, and is used solely to switch between two > >>> different modes, and the caller will most likely call the function with > >>> the argument as a constant known when writing the code (rather than > >>> taking an expression or variable with value not known until runtime), > >>> then it is usually better to split the function into two, one for > >>> each mode. > > This seems overly strict, e.g. it's not uncommon to have functions > enable debugging, verbose processing or similar processing variants > using a boolean parameter. > But usually the call site values for those wouldn't be constant -- they'd be computed from a command line flag for example. The key part of the phrasing is "the caller will most likely call the function with the argument as a constant known when writing the code". FWIW I disagree with the lead-in phrase "I think it is preferable *not* to have boolean parameters at all." -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon May 9 17:57:15 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 9 May 2016 23:57:15 +0200 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: <20160508015932.GS12028@ando.pearwood.info> <20160509143653.19d36e09@anarchist.wooz.org> <5730EC44.7000702@egenix.com> Message-ID: <573107BB.3040301@egenix.com> On 09.05.2016 22:07, Guido van Rossum wrote: > On Mon, May 9, 2016 at 1:00 PM, M.-A. Lemburg wrote: > >> On 09.05.2016 21:47, Guido van Rossum wrote: >>> On Mon, May 9, 2016 at 12:36 PM, Barry Warsaw wrote: >>>> On May 08, 2016, at 11:59 AM, Steven D'Aprano wrote: >>>>> I think it is preferable *not* to have boolean parameters at all. >>>>> >>>>> I don't remember if this is Guido's name for it, but I remember him >>>>> expressing the guideline "No constant bool arguments". If you have an >>>>> argument which takes a bool, and is used solely to switch between two >>>>> different modes, and the caller will most likely call the function with >>>>> the argument as a constant known when writing the code (rather than >>>>> taking an expression or variable with value not known until runtime), >>>>> then it is usually better to split the function into two, one for >>>>> each mode. >> >> This seems overly strict, e.g. it's not uncommon to have functions >> enable debugging, verbose processing or similar processing variants >> using a boolean parameter. >> > > But usually the call site values for those wouldn't be constant -- they'd > be computed from a command line flag for example. The key part of the > phrasing is "the caller will most likely call the function with the > argument as a constant known when writing the code". Hmm, so if you'd typically pass in a constant for the parameter it's deemed poor style, whereas when the value comes from some variable, it's fine ? This looks more like an API design question than a coding style one. E.g. take this example: def process(data, raise_errors=True): try: ... return result except ValueError: if raise_errors: raise else: return None It's not clear whether the caller would run the function with raise_errors=True or raise_errors=config.raise_errors more often. > FWIW I disagree with the lead-in phrase "I think it is preferable *not* to > have boolean parameters at all." Same here. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 09 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From guido at python.org Mon May 9 18:35:20 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 9 May 2016 15:35:20 -0700 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: <573107BB.3040301@egenix.com> References: <20160508015932.GS12028@ando.pearwood.info> <20160509143653.19d36e09@anarchist.wooz.org> <5730EC44.7000702@egenix.com> <573107BB.3040301@egenix.com> Message-ID: On Mon, May 9, 2016 at 2:57 PM, M.-A. Lemburg wrote: > Hmm, so if you'd typically pass in a constant for the parameter > it's deemed poor style, whereas when the value comes from some > variable, it's fine ? > Yes. > This looks more like an API design question than a coding > style one. > Yes. IIRC the question that started this thread was about API design guidance. > E.g. take this example: > > def process(data, raise_errors=True): > try: > ... > return result > except ValueError: > if raise_errors: > raise > else: > return None > > It's not clear whether the caller would run the function > with raise_errors=True or raise_errors=config.raise_errors > more often. > Yeah, that's for the API designer to guess based on how they envision the API to be used. It's often pretty clear once you start writing some example code though. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From contrebasse at gmail.com Tue May 10 09:15:40 2016 From: contrebasse at gmail.com (Joseph Martinot-Lagarde) Date: Tue, 10 May 2016 13:15:40 +0000 (UTC) Subject: [Python-ideas] Boolean parameters guidelines References: Message-ID: Serhiy Storchaka writes: > 1. It is preferable to pass boolean arguments as keyword arguments (if > this is not the only argument). > > 2. It is preferable to declare boolean parameters as keyword-only > parameters. > It is true for literal keywords arguments, but passing a named variable with a boolean value can be fine. # Bad my_func(True) # Good my_func(enable=True) # Good enable=True my_func(enable) # Meh enable=True my_func(enable=enable) # Bad w=True my_func(w) From ncoghlan at gmail.com Tue May 10 09:29:55 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 May 2016 23:29:55 +1000 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: <573107BB.3040301@egenix.com> References: <20160508015932.GS12028@ando.pearwood.info> <20160509143653.19d36e09@anarchist.wooz.org> <5730EC44.7000702@egenix.com> <573107BB.3040301@egenix.com> Message-ID: On 10 May 2016 at 07:57, M.-A. Lemburg wrote: > This looks more like an API design question than a coding > style one. This is actually a good point - at the moment, PEP 8 is a mix of coding style guidelines (i.e. how we use existing APIs) and API design guidelines (especially the parts on naming conventions and defining what is and isn't a public API). For code reviews, we want the former, but for PEP discussions we want the latter. Perhaps it would make sense to give API design guidelines a dedicated home in the developer guide, rather than overloading PEP 8 with them? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From barry at python.org Tue May 10 09:34:56 2016 From: barry at python.org (Barry Warsaw) Date: Tue, 10 May 2016 09:34:56 -0400 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: <20160508015932.GS12028@ando.pearwood.info> <20160509143653.19d36e09@anarchist.wooz.org> <5730EC44.7000702@egenix.com> <573107BB.3040301@egenix.com> Message-ID: <20160510093456.23688547@subdivisions.wooz.org> On May 10, 2016, at 11:29 PM, Nick Coghlan wrote: >This is actually a good point - at the moment, PEP 8 is a mix of >coding style guidelines (i.e. how we use existing APIs) and API design >guidelines (especially the parts on naming conventions and defining >what is and isn't a public API). > >For code reviews, we want the former, but for PEP discussions we want >the latter. Perhaps it would make sense to give API design guidelines >a dedicated home in the developer guide, rather than overloading PEP 8 >with them? PEP 8 is a *style guide*. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From k7hoven at gmail.com Tue May 10 09:43:03 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 10 May 2016 16:43:03 +0300 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: Message-ID: On Sun, May 8, 2016 at 3:14 PM, Nick Coghlan wrote: > On 8 May 2016 at 07:41, Serhiy Storchaka wrote: >> I propose to add following recommendations in PEP 8 or other documents: >> >> 1. It is preferable to pass boolean arguments as keyword arguments (if this >> is not the only argument). >> >> 2. It is preferable to declare boolean parameters as keyword-only >> parameters. >> >> What are you think about this? > > With Steven's suggestion to also articulate the "Would two functions > be better?" question, I think it's a good idea. > > Suggested points to note: > > 1. Do not use a boolean toggle to change the return type of a > function, define two different functions (e.g. os.walk vs os.fwalk, > os.getcwd vs os.getcwdb) > 2. If the boolean toggle is expected to be constant for any given > call-site, consider defining two different functions (e.g. > statistics.variance vs statistics.pvariance) > 3. If a boolean toggle is deemed appropriate, it is preferable to make > it keyword-only so call sites are always self-documenting > 4. When calling a function that accepts a boolean toggle, it is > preferable to pass it by keyword to make the call more > self-documenting > > The reason I think this is worth documenting is that it comes up > regularly in PEPs that require some associated API design, and these > are the rules of thumb we generally apply. > Regarding 3 and 4 above: More generally, would it not be a good thing to always consider using a keyword-only argument instead of a regular keyword argument (argument with a default value)? When using keyword-only arguments: A. It is possible to later turn a keyword-only argument into a regular (keyword) argument without breaking backwards compatibility of the API. or B. It is possible to add optional *positional* arguments to the function later without breaking the API. C. If the keyword-only argument later becomes redundant, it is possible to swallow it in **kwargs and make it effectively disappear (well, almost). -- Koos > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From steve at pearwood.info Tue May 10 09:48:59 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 10 May 2016 23:48:59 +1000 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: <20160510093456.23688547@subdivisions.wooz.org> References: <20160508015932.GS12028@ando.pearwood.info> <20160509143653.19d36e09@anarchist.wooz.org> <5730EC44.7000702@egenix.com> <573107BB.3040301@egenix.com> <20160510093456.23688547@subdivisions.wooz.org> Message-ID: <20160510134858.GC12028@ando.pearwood.info> On Tue, May 10, 2016 at 09:34:56AM -0400, Barry Warsaw wrote: > On May 10, 2016, at 11:29 PM, Nick Coghlan wrote: > > >This is actually a good point - at the moment, PEP 8 is a mix of > >coding style guidelines (i.e. how we use existing APIs) and API design > >guidelines (especially the parts on naming conventions and defining > >what is and isn't a public API). > > > >For code reviews, we want the former, but for PEP discussions we want > >the latter. Perhaps it would make sense to give API design guidelines > >a dedicated home in the developer guide, rather than overloading PEP 8 > >with them? > > PEP 8 is a *style guide*. Surely if we are discussing what PEP 8 *is*, then it is what Nick describes: a mix of coding style guidelines and API design guidelines. If people wish to separate the two into separate documents, that's one thing, but regardless of what PEP 8 claims to be, it does contain both style and API guides. Right now, that's what it is, whether that's what it was intended to be or not. (Personally, I think that coding style and API design overlap and that it is pointless to try to separate them. Coding style isn't only how to name your variables, but also includes whether you use map versus list comprehensions, whether you raise exceptions or return error codes, whether you write in a Java-influenced Design Pattern style or a functional style, when to use bool arguments and when not to.) -- Steve From rosuav at gmail.com Tue May 10 09:53:15 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 10 May 2016 23:53:15 +1000 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: <20160510134858.GC12028@ando.pearwood.info> References: <20160508015932.GS12028@ando.pearwood.info> <20160509143653.19d36e09@anarchist.wooz.org> <5730EC44.7000702@egenix.com> <573107BB.3040301@egenix.com> <20160510093456.23688547@subdivisions.wooz.org> <20160510134858.GC12028@ando.pearwood.info> Message-ID: On Tue, May 10, 2016 at 11:48 PM, Steven D'Aprano wrote: > > (Personally, I think that coding style and API design overlap and that > it is pointless to try to separate them. Coding style isn't only how to > name your variables, but also includes whether you use map versus list > comprehensions, whether you raise exceptions or return error codes, > whether you write in a Java-influenced Design Pattern style or a > functional style, when to use bool arguments and when not to.) If the API design guidelines get broken out, they should be closely associated with PEP 8. Maybe call it PEP 18, and have some references in PEP 8? But I'd rather see it all as one cohesive document - advice on how to structure your code to be Pythonic and maintainable. ChrisA From k7hoven at gmail.com Tue May 10 10:08:52 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 10 May 2016 17:08:52 +0300 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: <20160508015932.GS12028@ando.pearwood.info> <20160509143653.19d36e09@anarchist.wooz.org> <5730EC44.7000702@egenix.com> <573107BB.3040301@egenix.com> <20160510093456.23688547@subdivisions.wooz.org> <20160510134858.GC12028@ando.pearwood.info> Message-ID: On Tue, May 10, 2016 at 4:53 PM, Chris Angelico wrote: > On Tue, May 10, 2016 at 11:48 PM, Steven D'Aprano wrote: >> >> (Personally, I think that coding style and API design overlap and that >> it is pointless to try to separate them. Coding style isn't only how to >> name your variables, but also includes whether you use map versus list >> comprehensions, whether you raise exceptions or return error codes, >> whether you write in a Java-influenced Design Pattern style or a >> functional style, when to use bool arguments and when not to.) > > If the API design guidelines get broken out, they should be closely > associated with PEP 8. Maybe call it PEP 18, and have some references > in PEP 8? But I'd rather see it all as one cohesive document - advice > on how to structure your code to be Pythonic and maintainable. > One advantage of "PEP 18" might be that many style issues don't affect API design. And API design is more critical than style within the implementation, because the former can be very difficult to improve afterwards. And a shorter document may be easier to maintain in a self-consistent manner and to understand by the reader. -- Koos > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From barry at python.org Tue May 10 11:38:38 2016 From: barry at python.org (Barry Warsaw) Date: Tue, 10 May 2016 11:38:38 -0400 Subject: [Python-ideas] Is PEP 8 a style guide or API design? (was Re: Boolean parameters guidelines) References: <20160508015932.GS12028@ando.pearwood.info> <20160509143653.19d36e09@anarchist.wooz.org> <5730EC44.7000702@egenix.com> <573107BB.3040301@egenix.com> <20160510093456.23688547@subdivisions.wooz.org> <20160510134858.GC12028@ando.pearwood.info> Message-ID: <20160510113838.70bc8467@anarchist.wooz.org> On May 10, 2016, at 05:08 PM, Koos Zevenhoven wrote: >One advantage of "PEP 18" might be that many style issues don't affect >API design. And API design is more critical than style within the >implementation, because the former can be very difficult to improve >afterwards. And a shorter document may be easier to maintain in a >self-consistent manner and to understand by the reader. Let's be specific about PEP 8. What is "style" and what is "API design"? IMHO: * Code lay-out: style * String Quotes: style * Whitespace in Expressions and Statements: style * Comments: style * Version Bookkeeping: style * Naming Conventions: style & API intertwined, perhaps[1] - Designing for inheritance: API[2] - Public and internal interfaces: API[2] * Programming Recommendations: style[3] - Function Annotations: style[4] In short, I think style issues are those that you could imagine a tool enforcing , while API design issues are more for when humans communicate intent to each other. That's not a hard-and-fast rule, but I think it gets pretty close. So really, the major of PEP 8 does belong in a style guide. I'm not sure whether it's better to separate out the bits of API recommendations into a separate PEP, rather than perhaps reorganize or simplify PEP 8. [1] But I would argue that the naming conventions section is more aligned with style and *if* a PEP 8 split were to occur, the majority of this section would stay in PEP 8. [2] With a dash of style. [3] With a pinch of API [4] I put this in the style category because it doesn't make recommendations to *use* function annotations. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From guido at python.org Tue May 10 12:00:13 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 10 May 2016 09:00:13 -0700 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: Message-ID: On Tue, May 10, 2016 at 6:15 AM, Joseph Martinot-Lagarde < contrebasse at gmail.com> wrote: > Serhiy Storchaka writes: > > > 1. It is preferable to pass boolean arguments as keyword arguments (if > > this is not the only argument). > > > > 2. It is preferable to declare boolean parameters as keyword-only > > parameters. > > > > It is true for literal keywords arguments, but passing a named variable > with > a boolean value can be fine. > > # Bad > my_func(True) > > # Good > my_func(enable=True) > > # Good > enable=True > my_func(enable) > > # Meh > enable=True > my_func(enable=enable) > > # Bad > w=True > my_func(w) > That list of examples seems to be aligned along the wrong axis. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue May 10 12:02:30 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 10 May 2016 09:02:30 -0700 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: <20160508015932.GS12028@ando.pearwood.info> <20160509143653.19d36e09@anarchist.wooz.org> <5730EC44.7000702@egenix.com> <573107BB.3040301@egenix.com> Message-ID: On Tue, May 10, 2016 at 6:29 AM, Nick Coghlan wrote: > On 10 May 2016 at 07:57, M.-A. Lemburg wrote: > > This looks more like an API design question than a coding > > style one. > > This is actually a good point - at the moment, PEP 8 is a mix of > coding style guidelines (i.e. how we use existing APIs) and API design > guidelines (especially the parts on naming conventions and defining > what is and isn't a public API). > > For code reviews, we want the former, but for PEP discussions we want > the latter. Perhaps it would make sense to give API design guidelines > a dedicated home in the developer guide, rather than overloading PEP 8 > with them? > That seems a short-sighted approach to code reviews. At least where I work, early code review cycles are a great place for API design critique. I do agree that most API design guidelines are too complex to state categorically in a style PEP. The discussion and misunderstanding about the simple rule for boolean parameters is an example of how subtle the distinctions are. So API design is more a subject for blogs and books than for a PEP. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue May 10 13:50:01 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 10 May 2016 10:50:01 -0700 Subject: [Python-ideas] Is PEP 8 a style guide or API design? (was Re: Boolean parameters guidelines) In-Reply-To: <20160510113838.70bc8467@anarchist.wooz.org> References: <20160508015932.GS12028@ando.pearwood.info> <20160509143653.19d36e09@anarchist.wooz.org> <5730EC44.7000702@egenix.com> <573107BB.3040301@egenix.com> <20160510093456.23688547@subdivisions.wooz.org> <20160510134858.GC12028@ando.pearwood.info> <20160510113838.70bc8467@anarchist.wooz.org> Message-ID: On Tue, May 10, 2016 at 8:38 AM, Barry Warsaw wrote: > On May 10, 2016, at 05:08 PM, Koos Zevenhoven wrote: > > >One advantage of "PEP 18" might be that many style issues don't affect > >API design. And API design is more critical than style within the > >implementation, because the former can be very difficult to improve > >afterwards. And a shorter document may be easier to maintain in a > >self-consistent manner and to understand by the reader. > > Let's be specific about PEP 8. What is "style" and what is "API design"? > IMHO: > > * Code lay-out: style > * String Quotes: style > * Whitespace in Expressions and Statements: style > * Comments: style > * Version Bookkeeping: style > * Naming Conventions: style & API intertwined, perhaps[1] > - Designing for inheritance: API[2] > - Public and internal interfaces: API[2] > * Programming Recommendations: style[3] > - Function Annotations: style[4] > > In short, I think style issues are those that you could imagine a tool > enforcing , while API design issues are more for when humans > communicate > intent to each other. That's not a hard-and-fast rule, but I think it gets > pretty close. > I don't like that distinction, wink or no wink. It's not about whether you can mechanically enforce it (this is why I am pushing for the rename of the pep8 tool). E.g. how would the tool know whether a particular name is meant to be private? I think style issues are stuff that makes all code more readable. Naming conventions definitely fall in that category (they hint at what kind of thing something is when you see it used without having to look up the definition right away). I actually agree with your categorizations -- we can discuss inheritance vs. composition until the cows see blue in the face, but it's just much more complex, and it doesn't affect readability -- it affects what happens when large systems built out of diverse components evolve. > So really, the major of PEP 8 does belong in a style guide. I'm not sure > whether it's better to separate out the bits of API recommendations into a > separate PEP, rather than perhaps reorganize or simplify PEP 8. > I vote for keeping it together but I also vote to keep the more complex API design issues out of the PEP series altogether. Maybe 10 years from now there's a bunch of blog posts with sufficiently broad acceptance that we can add it to the PEPs, but until then I think it's pointless to try and pretend there's much of a standard there (given that even PEP 8 has some controversial bits and occasionally changes). Until then I recommend that everyone watches Josh Bloch's talk on API design yearly. ( https://www.youtube.com/watch?v=heh4OeB9A-c; you can find other versions and slides via this search; https://www.google.com/search?q=josh+bloch+api+design&ie=utf-8&oe=utf-8) > [1] But I would argue that the naming conventions section is more aligned > with > style and *if* a PEP 8 split were to occur, the majority of this section > would > stay in PEP 8. > > [2] With a dash of style. > > [3] With a pinch of API > > [4] I put this in the style category because it doesn't make > recommendations > to *use* function annotations. > This feels like unnecessary use of footnotes... -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From contrebasse at gmail.com Tue May 10 19:41:36 2016 From: contrebasse at gmail.com (Joseph Martinot-Lagarde) Date: Tue, 10 May 2016 23:41:36 +0000 (UTC) Subject: [Python-ideas] Boolean parameters guidelines References: Message-ID: > > That list of examples seems to be aligned along the wrong axis. > I don't understand what you mean. The examples should go from Good to Bad or something like that ? Or did I miss the point ? I just wanted to show that requiring a keyword for readability isn't always necessary. It can be up to the user to choose what is the most readable. From guido at python.org Tue May 10 19:44:02 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 10 May 2016 16:44:02 -0700 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: Message-ID: I meant that I disagreed in several cases with whether an example was Good or Bad and I couldn't figure out what rule you were using in your head to decide on Good/Bad, IOW what rule you were illustrating with these examples. On Tue, May 10, 2016 at 4:41 PM, Joseph Martinot-Lagarde < contrebasse at gmail.com> wrote: > > > > That list of examples seems to be aligned along the wrong axis. > > > I don't understand what you mean. The examples should go from Good to Bad > or > something like that ? Or did I miss the point ? > > I just wanted to show that requiring a keyword for readability isn't always > necessary. It can be up to the user to choose what is the most readable. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From contrebasse at gmail.com Tue May 10 19:59:47 2016 From: contrebasse at gmail.com (Joseph Martinot-Lagarde) Date: Tue, 10 May 2016 23:59:47 +0000 (UTC) Subject: [Python-ideas] Boolean parameters guidelines References: Message-ID: > I meant that I disagreed in several cases with whether an example was Good or Bad and I couldn't figure out what rule you were using in your head to decide on Good/Bad, IOW what rule you were illustrating with these examples. Alright, I'll add some explanation. I took an example already used in this thread to be more clear. # Bad: no indication on what the boolean argument does. process(data, True) # Good: the boolean is used to indicate that errors are raised. process(data, raise_errors=True) # Good: the variable name is used to indicate what the argument means. raise_errors=True process(data, raise_errors) # Meh: both the keyword and the variable name are used to indicate what the argument means. It's not really bad, but it adds no information while adding redundancy. This would happend if the keyword is required. raise_errors=True process(data, raise_errors=raise_errors) # Bad: a variable name is used but it doesn't indicate anything on its meaning. w=True process(data, w) From guido at python.org Tue May 10 20:20:53 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 10 May 2016 17:20:53 -0700 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: Message-ID: On Tue, May 10, 2016 at 4:59 PM, Joseph Martinot-Lagarde < contrebasse at gmail.com> wrote: > > > I meant that I disagreed in several cases with whether an example was > Good > or Bad and I couldn't figure out what rule you were using in your head to > decide on Good/Bad, IOW what rule you were illustrating with these > examples. > > Alright, I'll add some explanation. I took an example already used in this > thread to be more clear. > > # Bad: no indication on what the boolean argument does. > process(data, True) > > # Good: the boolean is used to indicate that errors are raised. > process(data, raise_errors=True) > So far so good. > # Good: the variable name is used to indicate what the argument means. > raise_errors=True > process(data, raise_errors) > Not so sure. If the variable raise_errors were set to False the process() call could seem misleading. > # Meh: both the keyword and the variable name are used to indicate what > the argument means. It's not really bad, but it adds no information while > adding redundancy. This would happend if the keyword is required. > raise_errors=True > process(data, raise_errors=raise_errors) > That's actually fine with me; there's often a reason to give the variable another name anyways. > # Bad: a variable name is used but it doesn't indicate anything on its > meaning. > w=True > process(data, w) > All depends. In any case the discussion was more about how to design the API in the first place than about what the call should look like. In particular, if the raise_errors parameter is nearly always given as the literal True or False, it might be better to have process(data) and process_raising_errors(data) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Wed May 11 08:31:28 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 11 May 2016 14:31:28 +0200 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: Message-ID: <57332620.8010408@egenix.com> On 11.05.2016 02:20, Guido van Rossum wrote: > In any case the discussion was more about how to design the API in the > first place than about what the call should look like. In particular, if > the raise_errors parameter is nearly always given as the literal True or > False, it might be better to have > > process(data) > > and > > process_raising_errors(data) This would work in the simple case where the try-except is used on the whole inner processing block (as in the example I posted), but in practice the variant you want to make selectable often requires adjusting multiple code sections or paths in the block. By having two separate functions, you'd then create a lot of duplicated code (with all the possibly issues that go with it, e.g. copy&paste errors, bugs only fixed in one copy, poor maintainability, etc.). Example: def complex_process(data, raise_errors=True): try: ...phase 1... except ValueError: if raise_errors: raise else: return None ...phase 2... try: ...phase3... except TypeError: if raise_errors: raise else: return None return result The obvious solution would be to wrap the raising variant in an outer function which catches the exceptions, but in the above example, you'd then have to catch both ValueError and TypeError, which may actually catch more errors than you really want to catch. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 11 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From srkunze at mail.de Wed May 11 12:10:23 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 11 May 2016 18:10:23 +0200 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: Message-ID: <5733596F.50803@mail.de> On 07.05.2016 23:41, Serhiy Storchaka wrote: > I propose to add following recommendations in PEP 8 or other documents: > > 1. It is preferable to pass boolean arguments as keyword arguments (if > this is not the only argument). > > 2. It is preferable to declare boolean parameters as keyword-only > parameters. As this proposal concerns the readability of code, the situation is arguably worse for boolean parameters; so +1. Additionally, it seems like this could be said for other primitive datatypes as well: >>> sleep(100) # minutes? Sven From dickinsm at gmail.com Wed May 11 14:51:23 2016 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 11 May 2016 19:51:23 +0100 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: <5733596F.50803@mail.de> References: <5733596F.50803@mail.de> Message-ID: On Wed, May 11, 2016 at 5:10 PM, Sven R. Kunze wrote: > Additionally, it seems like this could be said for other primitive datatypes > as well: > > >>>> sleep(100) # minutes? Don't be silly. Why on earth would that argument represent minutes? It's days, for consistency with `timedelta(100)`. Mark From ian.g.kelly at gmail.com Wed May 11 16:36:05 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 11 May 2016 14:36:05 -0600 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: <5733596F.50803@mail.de> Message-ID: On Wed, May 11, 2016 at 12:51 PM, Mark Dickinson wrote: > On Wed, May 11, 2016 at 5:10 PM, Sven R. Kunze wrote: >> Additionally, it seems like this could be said for other primitive datatypes >> as well: >> >> >>>>> sleep(100) # minutes? > > Don't be silly. Why on earth would that argument represent minutes? > > It's days, for consistency with `timedelta(100)`. Still best to be explicit: >>> sleep(timedelta(100)) It would be awesome if that actually worked. From random832 at fastmail.com Wed May 11 17:27:34 2016 From: random832 at fastmail.com (Random832) Date: Wed, 11 May 2016 17:27:34 -0400 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: <5733596F.50803@mail.de> Message-ID: <1463002054.1597916.605186377.578EA1AF@webmail.messagingengine.com> On Wed, May 11, 2016, at 16:36, Ian Kelly wrote: > >>> sleep(timedelta(100)) > > It would be awesome if that actually worked. Well, the recent __fspath__ proposal suggests a solution: How about a method for timedelta-like objects to return a float number of seconds? May also want a way to have datetime-like objects return a timestamp for e.g. utime. From zachary.ware+pyideas at gmail.com Wed May 11 17:40:45 2016 From: zachary.ware+pyideas at gmail.com (Zachary Ware) Date: Wed, 11 May 2016 16:40:45 -0500 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: <5733596F.50803@mail.de> Message-ID: On Wed, May 11, 2016 at 3:36 PM, Ian Kelly wrote: > On Wed, May 11, 2016 at 12:51 PM, Mark Dickinson wrote: >> On Wed, May 11, 2016 at 5:10 PM, Sven R. Kunze wrote: >>> Additionally, it seems like this could be said for other primitive datatypes >>> as well: >>> >>>>>> sleep(100) # minutes? >> >> Don't be silly. Why on earth would that argument represent minutes? >> >> It's days, for consistency with `timedelta(100)`. > > Still best to be explicit: > >>>> sleep(timedelta(100)) > > It would be awesome if that actually worked. You just weren't explicit enough: >>> sleep(timedelta(100).total_seconds()) This is drifting pretty far from the topic at hand, though. -- Zach From ethan at stoneleaf.us Wed May 11 17:57:30 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 May 2016 14:57:30 -0700 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: <5733596F.50803@mail.de> Message-ID: <5733AACA.4020702@stoneleaf.us> On 05/11/2016 02:40 PM, Zachary Ware wrote: > On Wed, May 11, 2016 at 3:36 PM, Ian Kelly wrote: >> On Wed, May 11, 2016 at 12:51 PM, Mark Dickinson wrote: >>> On Wed, May 11, 2016 at 5:10 PM, Sven R. Kunze wrote: >>>> Additionally, it seems like this could be said for other primitive datatypes >>>> as well: >>>> >>>>--> sleep(100) # minutes? >>> >>> Don't be silly. Why on earth would that argument represent minutes? >>> >>> It's days, for consistency with `timedelta(100)`. >> >> Still best to be explicit: >> >>--> sleep(timedelta(100)) >> >> It would be awesome if that actually worked. > > You just weren't explicit enough: > > --> sleep(timedelta(100).total_seconds()) > > This is drifting pretty far from the topic at hand, though. There was a topic at hand? Oh, yeah, keyword arguments! Actually, it's very apropos: each of those examples would have been either obviously correct or obviously wrong if the parameter names had been used. -- ~Ethan~ From rosuav at gmail.com Wed May 11 19:48:19 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 12 May 2016 09:48:19 +1000 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: <5733AACA.4020702@stoneleaf.us> References: <5733596F.50803@mail.de> <5733AACA.4020702@stoneleaf.us> Message-ID: On Thu, May 12, 2016 at 7:57 AM, Ethan Furman wrote: > On 05/11/2016 02:40 PM, Zachary Ware wrote: >> >> On Wed, May 11, 2016 at 3:36 PM, Ian Kelly wrote: >>> >>> On Wed, May 11, 2016 at 12:51 PM, Mark Dickinson wrote: >>>> >>>> On Wed, May 11, 2016 at 5:10 PM, Sven R. Kunze wrote: > > >>>>> Additionally, it seems like this could be said for other primitive >>>>> datatypes >>>>> as well: >>>>> >>>>> --> sleep(100) # minutes? >>>> >>>> >>>> Don't be silly. Why on earth would that argument represent minutes? >>>> >>>> It's days, for consistency with `timedelta(100)`. >>> >>> >>> Still best to be explicit: >>> >>> --> sleep(timedelta(100)) >>> >>> It would be awesome if that actually worked. >> >> >> You just weren't explicit enough: >> >> --> sleep(timedelta(100).total_seconds()) >> >> This is drifting pretty far from the topic at hand, though. > > > There was a topic at hand? > > Oh, yeah, keyword arguments! > > Actually, it's very apropos: each of those examples would have been either > obviously correct or obviously wrong if the parameter names had been used. Unfortunately it isn't always possible. rosuav at sikorsky:~$ PAGER=cat python3 Python 3.6.0a0 (default:98678738b7e9, May 2 2016, 13:37:04) [GCC 5.3.1 20160409] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from time import sleep >>> help(sleep) Help on built-in function sleep in module time: sleep(...) sleep(seconds) Delay execution for a given number of seconds. The argument may be a floating point number for subsecond precision. >>> sleep(seconds=1) Traceback (most recent call last): File "", line 1, in TypeError: sleep() takes no keyword arguments So style guides can't really recommend "always use keyword arguments", as there are too many functions implemented in C that don't take any keyword args. They're effectively implemented as: def sleep(*args): [seconds] = args This particular example might be worth using a keyword arg with, except that the confusion isn't within Python, it's across languages. And then there's the parallel question of whether or not fractional seconds are valid. A quick survey of sleep functions shows up: * C on POSIX: integer seconds (usleep and nanosleep for integer micro/nanoseconds) * C on Windows: integer milliseconds * Pike and Python: int or float seconds * REXX on OS/2: integer seconds [syssleep], milliseconds [rxsleep], or decimal seconds So if you're doing cross-language work, you'll need to check your docs anyway, and if you're not, it shouldn't be too hard to remember that sleep() in Python takes a number of seconds. (It's easy if you give it a non-integer value. You might see sleep(1000) and interpret that as milliseconds, but you would never see sleep(0.5) and mistake it for anything other than half a second.) ChrisA From alexander.belopolsky at gmail.com Wed May 11 20:16:21 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 11 May 2016 20:16:21 -0400 Subject: [Python-ideas] Sleep and timedelta Was: Boolean parameters guidelines Message-ID: On Wed, May 11, 2016 at 4:36 PM, Ian Kelly wrote: > Still best to be explicit: > > >>> sleep(timedelta(100)) > > It would be awesome if that actually worked. > FWIW, I would not mind seeing sleep() as a timedelta instance method: >>> timedelta(seconds=42).sleep() -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed May 11 22:14:07 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 11 May 2016 22:14:07 -0400 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: <5733596F.50803@mail.de> <5733AACA.4020702@stoneleaf.us> Message-ID: On 5/11/2016 7:48 PM, Chris Angelico wrote: > * C on POSIX: integer seconds (usleep and nanosleep for integer > micro/nanoseconds) > * C on Windows: integer milliseconds Ditto for tcl/tk 'sleep' function: widget.after(ms, func, *args) > * Pike and Python: int or float seconds > * REXX on OS/2: integer seconds [syssleep], milliseconds [rxsleep], or > decimal seconds > > So if you're doing cross-language work, you'll need to check your docs > anyway, Moving from sleep(seconds) to .after(ms) is a nuisance that sometimes trips up SO questioners, but it is near the bottom of the list of newbie problems with tkinter. -- Terry Jan Reedy From srkunze at mail.de Thu May 12 03:50:08 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 12 May 2016 09:50:08 +0200 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: <5733AACA.4020702@stoneleaf.us> References: <5733596F.50803@mail.de> <5733AACA.4020702@stoneleaf.us> Message-ID: <573435B0.7030507@mail.de> On 11.05.2016 23:57, Ethan Furman wrote: > On 05/11/2016 02:40 PM, Zachary Ware wrote: >> On Wed, May 11, 2016 at 3:36 PM, Ian Kelly wrote: >>> You just weren't explicit enough: >>> >>> --> sleep(timedelta(100).total_seconds()) >>> >>> This is drifting pretty far from the topic at hand, though. > > There was a topic at hand? > > Oh, yeah, keyword arguments! > > Actually, it's very apropos: each of those examples would have been > either obviously correct or obviously wrong if the parameter names had > been used. Indeed. It's something that's already baked in our internal style guidelines. Mostly because we are too lazy to read the docs or jump to definition. Sven PS: It's interesting to see how easy it is to get off-topic as soon as an example is present. From srkunze at mail.de Thu May 12 03:56:21 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 12 May 2016 09:56:21 +0200 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: <5733596F.50803@mail.de> <5733AACA.4020702@stoneleaf.us> Message-ID: <57343725.1060301@mail.de> On 12.05.2016 01:48, Chris Angelico wrote: > So style guides can't really recommend "always use keyword arguments", > as there are too many functions implemented in C that don't take any > keyword args. I don't think "sometimes technically impossible" prevents a recommendation. Do-it-when-possible-otherwise-don't suffices here. The same holds for boolean values. Sven From storchaka at gmail.com Thu May 12 04:28:16 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 12 May 2016 11:28:16 +0300 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: <5733596F.50803@mail.de> <5733AACA.4020702@stoneleaf.us> Message-ID: On 12.05.16 02:48, Chris Angelico wrote: >>>> from time import sleep >>>> help(sleep) > Help on built-in function sleep in module time: > > sleep(...) > sleep(seconds) > > Delay execution for a given number of seconds. The argument may be > a floating point number for subsecond precision. > >>>> sleep(seconds=1) > Traceback (most recent call last): > File "", line 1, in > TypeError: sleep() takes no keyword arguments Note that passing positional arguments is much faster than passing keyword arguments even if the function supports them. And the difference can be even larger after pushing Victor's optimization. [1] Other optimizations (like [2]) can be applied only for functions that support only positional parameters. Thus passing positional arguments and having functions that support only positional parameters is important for performance. [1] http://bugs.python.org/issue26814 [2] http://bugs.python.org/issue23867 From srkunze at mail.de Thu May 12 04:36:16 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 12 May 2016 10:36:16 +0200 Subject: [Python-ideas] Sleep and timedelta Was: Boolean parameters guidelines In-Reply-To: References: Message-ID: <57344080.5060200@mail.de> On 12.05.2016 02:16, Alexander Belopolsky wrote: > > On Wed, May 11, 2016 at 4:36 PM, Ian Kelly > wrote: > > Still best to be explicit: > > >>> sleep(timedelta(100)) > > It would be awesome if that actually worked. > > > FWIW, I would not mind seeing sleep() as a timedelta instance method: > > >>> timedelta(seconds=42).sleep() Some would suggest that introducing a new protocol like - timedelta(seconds=42).__seconds__ defining an ABC - class SecondLike(abc.ABC): - @abc.abstractmethod - def __seconds__(self): - raise NotImplementedError defining an extraction function - def os.seconds... and let all stdlib functions which accepts integer/float seconds also accept SecondLike objects as the solution to the problem. I have to admit that I like your idea more. Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu May 12 14:49:27 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 12 May 2016 11:49:27 -0700 Subject: [Python-ideas] Adding Type[C] support to PEP 484 and typing.py Message-ID: There's one new feature I'd like to add to PEP 484 and the typing.py module that will go out with Python 3.5.2. There's a long discssion in https://github.com/python/typing/issues/107, but I think I can explain the proposal very quickly. There's currently no good way to talk about class objects. If you have a function argument that's a class (as opposed to an instance) the only annotation you can give it is `type`. But I've encountered this pattern quite frequently, and what people invariably want to say is "it's a subclass of this here class C." Example: class User: ... class TeamUser(User): ... class ProUser(User): ... def new_user(user_class: Type[User]) -> User: ... The proposal is to let you spell that explicitly by setting the type or such an argument to `Type[C]`. The type checker will then understand that if you call it, the result is an instance of C. It will also know about all the class methods defined on C. There are some subtleties, e.g. in the above example we would actually like to know that the return type varies with the argument type: joe = new_user(ProUser) # Really, joe is a ProUser, not just a User This can be done using a type variable, e.g. U = TypeVar('U', bound=User) def new_user(user_class: Type[U]) -> U: ... joe = new_user(ProUser) The key part I want to settle quickly is the notation: I need to add a special generic `class Type` to typing.py so that we can start implementing this in mypy. Any objections? Or should I just go ahead and add the text to PEP 484 and post the diff to python-dev? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu May 12 14:59:37 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 May 2016 11:59:37 -0700 Subject: [Python-ideas] Adding Type[C] support to PEP 484 and typing.py In-Reply-To: References: Message-ID: <5734D299.2050406@stoneleaf.us> On 05/12/2016 11:49 AM, Guido van Rossum wrote: > Any objections? Or should I just go ahead and add the text to PEP 484 > and post the diff to python-dev? No objections from me! If (when?) I use typing I'll definitely want this ability. :) -- ~Ethan~ From kramm at google.com Thu May 12 16:28:00 2016 From: kramm at google.com (Matthias Kramm) Date: Thu, 12 May 2016 13:28:00 -0700 (PDT) Subject: [Python-ideas] Adding Type[C] support to PEP 484 and typing.py In-Reply-To: References: Message-ID: <7ff0aec2-60c4-4356-8f78-ee8a2eaf9ce0@googlegroups.com> On Thursday, May 12, 2016 at 11:50:32 AM UTC-7, Guido van Rossum wrote: > > There's one new feature I'd like to add to PEP 484 and the typing.py > module that will go out with Python 3.5.2. > > There's a long discssion in https://github.com/python/typing/issues/107, > but I think I can explain the proposal very quickly. > > There's currently no good way to talk about class objects. If you have a > function argument that's a class (as opposed to an instance) the only > annotation you can give it is `type`. But I've encountered this pattern > quite frequently, and what people invariably want to say is "it's a > subclass of this here class C." > I like it. I have wanted "Type[C]" many times, in pytype's output. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kramm at google.com Thu May 12 16:28:00 2016 From: kramm at google.com (Matthias Kramm) Date: Thu, 12 May 2016 13:28:00 -0700 (PDT) Subject: [Python-ideas] Adding Type[C] support to PEP 484 and typing.py In-Reply-To: References: Message-ID: <7ff0aec2-60c4-4356-8f78-ee8a2eaf9ce0@googlegroups.com> On Thursday, May 12, 2016 at 11:50:32 AM UTC-7, Guido van Rossum wrote: > > There's one new feature I'd like to add to PEP 484 and the typing.py > module that will go out with Python 3.5.2. > > There's a long discssion in https://github.com/python/typing/issues/107, > but I think I can explain the proposal very quickly. > > There's currently no good way to talk about class objects. If you have a > function argument that's a class (as opposed to an instance) the only > annotation you can give it is `type`. But I've encountered this pattern > quite frequently, and what people invariably want to say is "it's a > subclass of this here class C." > I like it. I have wanted "Type[C]" many times, in pytype's output. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pavol.lisy at gmail.com Thu May 12 17:51:53 2016 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Thu, 12 May 2016 23:51:53 +0200 Subject: [Python-ideas] Adding Type[C] support to PEP 484 and typing.py In-Reply-To: References: Message-ID: 2016-05-12 20:49 GMT+02:00, Guido van Rossum : > There are some subtleties, e.g. in the above example we would actually like > to know that the return type varies with the argument type: > > joe = new_user(ProUser) # Really, joe is a ProUser, not just a User > > This can be done using a type variable, e.g. > > U = TypeVar('U', bound=User) > def new_user(user_class: Type[U]) -> U: ... > joe = new_user(ProUser) It is very probably I dont understand. Is U here same type (=ProUser) for parameter and for return? Because in PEP484 is written: -> CT = TypeVar('CT', bound=Comparable) -> -> def min(x: CT, y: CT) -> CT: -> if x < y: -> return x -> else: -> return y -> -> min(1, 2) # ok, return type int -> min('x', 'y') # ok, return type str -> ->(Note that this is not ideal -- for example ``min('x', 1)`` is invalid ->at runtime but a type checker would simply infer the return type ->``Comparable``. Unfortunately, addressing this would require ->introducing a much more powerful and also much more complicated ->vconcept, F-bounded polymorphism. We may revisit this in the future.) which I understand that CT could be different type for x (=str) and y (=int). From guido at python.org Thu May 12 17:59:57 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 12 May 2016 14:59:57 -0700 Subject: [Python-ideas] Adding Type[C] support to PEP 484 and typing.py In-Reply-To: References: Message-ID: On Thu, May 12, 2016 at 2:51 PM, Pavol Lisy wrote: > 2016-05-12 20:49 GMT+02:00, Guido van Rossum : > > > There are some subtleties, e.g. in the above example we would actually > like > > to know that the return type varies with the argument type: > > > > joe = new_user(ProUser) # Really, joe is a ProUser, not just a User > > > > This can be done using a type variable, e.g. > > > > U = TypeVar('U', bound=User) > > def new_user(user_class: Type[U]) -> U: ... > > joe = new_user(ProUser) > > It is very probably I dont understand. Is U here same type (=ProUser) > for parameter and for return? > Their status is different though. The return type is inferred from the argument type but (usually) not the other way around. > Because in PEP484 is written: > > -> CT = TypeVar('CT', bound=Comparable) > -> > -> def min(x: CT, y: CT) -> CT: > -> if x < y: > -> return x > -> else: > -> return y > -> > -> min(1, 2) # ok, return type int > -> min('x', 'y') # ok, return type str > -> > ->(Note that this is not ideal -- for example ``min('x', 1)`` is invalid > ->at runtime but a type checker would simply infer the return type > ->``Comparable``. Unfortunately, addressing this would require > ->introducing a much more powerful and also much more complicated > ->vconcept, F-bounded polymorphism. We may revisit this in the future.) > > which I understand that CT could be different type for x (=str) and y > (=int). > No, *in a given call* CT will be the same for all. The example with Comparable is that for `min('x', 1)` there is a solution where CT *is* actually the same for both arguments -- both 'x' and 1 are instances of Comparable. In that example this is not ideal. But in the new_user(UserPro) example there's only one occurrence of U in the argument list, and that will be inferred as the return type. An example more similar to the CT example with users would be something like def pair_accounts(personal_user_class: Type[U], business_use_class: Type[U]) -> List[U]: ... Here the return type of pair_accounts(ProUser, ProUser) would be inferred as List[ProUser], but the return type of pair_accounts(ProUser, TeamUser) would be inferred as List[User], since User is the common base class of ProUser and TeamUser. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu May 12 22:52:37 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 12 May 2016 19:52:37 -0700 Subject: [Python-ideas] Adding Type[C] support to PEP 484 and typing.py In-Reply-To: <7ff0aec2-60c4-4356-8f78-ee8a2eaf9ce0@googlegroups.com> References: <7ff0aec2-60c4-4356-8f78-ee8a2eaf9ce0@googlegroups.com> Message-ID: Great. We'll be able to move this very quickly once we decide whether it should be called Type[C] or Class[C]. The original proposal is Type[C], and it's nice because it's a pun on `type`, just like Tuple[...] is a pun on `tuple` (and similar for List etc.). But OTOH it really annotates a class object, not an abstract type. I still prefer Type[C] -- anyone want to argue that we're making a mistake? (I know, nobody's listening, everyone's too busy arguing that Path should inherit from str. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu May 12 23:00:50 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 13 May 2016 13:00:50 +1000 Subject: [Python-ideas] Adding Type[C] support to PEP 484 and typing.py In-Reply-To: References: <7ff0aec2-60c4-4356-8f78-ee8a2eaf9ce0@googlegroups.com> Message-ID: On Fri, May 13, 2016 at 12:52 PM, Guido van Rossum wrote: > Great. We'll be able to move this very quickly once we decide whether it > should be called Type[C] or Class[C]. The original proposal is Type[C], and > it's nice because it's a pun on `type`, just like Tuple[...] is a pun on > `tuple` (and similar for List etc.). But OTOH it really annotates a class > object, not an abstract type. I still prefer Type[C] -- anyone want to argue > that we're making a mistake? (I know, nobody's listening, everyone's too > busy arguing that Path should inherit from str. :-) ISTM Type is the more obvious name here. Python doesn't have class objects any more than it has def objects - it has functions and types. ChrisA From robertc at robertcollins.net Thu May 12 23:04:42 2016 From: robertc at robertcollins.net (Robert Collins) Date: Fri, 13 May 2016 15:04:42 +1200 Subject: [Python-ideas] Adding Type[C] support to PEP 484 and typing.py In-Reply-To: References: <7ff0aec2-60c4-4356-8f78-ee8a2eaf9ce0@googlegroups.com> Message-ID: On 13 May 2016 at 15:00, Chris Angelico wrote: > On Fri, May 13, 2016 at 12:52 PM, Guido van Rossum wrote: >> Great. We'll be able to move this very quickly once we decide whether it >> should be called Type[C] or Class[C]. The original proposal is Type[C], and >> it's nice because it's a pun on `type`, just like Tuple[...] is a pun on >> `tuple` (and similar for List etc.). But OTOH it really annotates a class >> object, not an abstract type. I still prefer Type[C] -- anyone want to argue >> that we're making a mistake? (I know, nobody's listening, everyone's too >> busy arguing that Path should inherit from str. :-) > > ISTM Type is the more obvious name here. Python doesn't have class > objects any more than it has def objects - it has functions and types. > > ChrisA +1, FWIW - I don't generally chime in with Me Toos, but since Guido is asking :) -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From ncoghlan at gmail.com Fri May 13 00:14:35 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 13 May 2016 14:14:35 +1000 Subject: [Python-ideas] Adding Type[C] support to PEP 484 and typing.py In-Reply-To: References: <7ff0aec2-60c4-4356-8f78-ee8a2eaf9ce0@googlegroups.com> Message-ID: On 13 May 2016 at 12:52, Guido van Rossum wrote: > Great. We'll be able to move this very quickly once we decide whether it > should be called Type[C] or Class[C]. The original proposal is Type[C], and > it's nice because it's a pun on `type`, just like Tuple[...] is a pun on > `tuple` (and similar for List etc.). But OTOH it really annotates a class > object, not an abstract type. I still prefer Type[C] -- anyone want to argue > that we're making a mistake? +1 for Type - it's the typical term used by Python developers, even though it's technically a misnomer from a type theory perspective. My intuition is that being consistent with the misnomer will be less confusing overall, based on the assumption that many more Pythonistas will be familiar with the type() builtin than will be familiar with formal type theory. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Fri May 13 00:35:04 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 12 May 2016 21:35:04 -0700 Subject: [Python-ideas] Adding Type[C] support to PEP 484 and typing.py In-Reply-To: References: <7ff0aec2-60c4-4356-8f78-ee8a2eaf9ce0@googlegroups.com> Message-ID: On Thu, May 12, 2016 at 8:00 PM, Chris Angelico wrote: > On Fri, May 13, 2016 at 12:52 PM, Guido van Rossum > wrote: > > Great. We'll be able to move this very quickly once we decide whether it > > should be called Type[C] or Class[C]. The original proposal is Type[C], > and > > it's nice because it's a pun on `type`, just like Tuple[...] is a pun on > > `tuple` (and similar for List etc.). But OTOH it really annotates a class > > object, not an abstract type. I still prefer Type[C] -- anyone want to > argue > > that we're making a mistake? (I know, nobody's listening, everyone's too > > busy arguing that Path should inherit from str. :-) > > ISTM Type is the more obvious name here. Python doesn't have class > objects any more than it has def objects - it has functions and types. > Well, actually, I call them class objects all the time... Plus PEP 484 (at least the recent revisions, though this has always been the BDFL-delegate's intention) tries to make a clear terminological between classes (the things you have at runtime) and types (the things that type checkers care about). There's a big overlap because most classes are also types -- but not the other way around! E.g. Any is a type but not a class (you can neither inherit from Any nor instantiate it), and the same is true for unions and type variables. And it's pretty clear that when you have an argument or variable annotated with Type[C] it has to be a real class object, not Any or a union or a type variable, because in the common use case, functions taking class objects either instantiate that class, call class methods on it, or use it for isinstance() checks, and none of those things would work if at run time they received Any or a union or type variable. Let's look at that function new_user() again: def new_user(user_class: Type[User]) -> User: return user_class() We can call new_user(ProUser) or new_user(TeamUser), but calling new_user(Any) isn't going to work at runtime, not us new_user(Union[ProUser, TeamUser]). Note also that this is a unique twist for Type[C] that doesn't exist for non-type annotations. If we have a function argument annotated with a class, e.g. def first_name(u: User) -> str: ... it's totally reasonable to call it with an argument whose type is Union[ProUser, TeamUser]: def get_user_by_id(id: str) -> Union[ProUser, TeamUser]: ... name = first_name(get_user_by_id('joe')) This is because here the *type* returned by get_user_by_id() is a union, but the actual *values* it returns are always instances of one of the runtime classes ProUser or TeamUser. If we had had more consistent terminology in Python from the start, this naming issue would never have come up -- we'd never have used type(x) as (almost) equivalent to x.__class__, and we wouldn't have used class C(type): ... to define a metaclass (i.e. a class's class). But that's water under the bridge, and we might a well write Type[C] rather than Class[C] to pub with `type`. And it seems only two people care about the difference -- Mark Shannon and myself. :-) So Type[C] it is. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri May 13 01:58:54 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 May 2016 22:58:54 -0700 Subject: [Python-ideas] Adding Type[C] support to PEP 484 and typing.py In-Reply-To: References: <7ff0aec2-60c4-4356-8f78-ee8a2eaf9ce0@googlegroups.com> Message-ID: <57356D1E.9060402@stoneleaf.us> On 05/12/2016 09:35 PM, Guido van Rossum wrote: > Well, actually, I call them class objects all the time... Plus PEP 484 > (at least the recent revisions, though this has always been the > BDFL-delegate's intention) tries to make a clear terminological between > classes (the things you have at runtime) and types (the things that type > checkers care about). I definitely don't have enough CS background to understand the subtle distinction you're drawing here (at least, I hope it's subtle ;) . But to me, it's a type -- and when writing metaclasses, my decision for the name has always been between SomethingMeta or SomethingType, it's never been SomethingClass. > But that's water under the bridge, and we might a well write Type[C] > rather than Class[C] to pub with `type`. And it seems only two people > care about the difference -- Mark Shannon and myself. :-) So Type[C] it is. Cool. :) -- ~Ethan~ From k7hoven at gmail.com Fri May 13 05:56:35 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 13 May 2016 12:56:35 +0300 Subject: [Python-ideas] Adding Type[C] support to PEP 484 and typing.py In-Reply-To: References: <7ff0aec2-60c4-4356-8f78-ee8a2eaf9ce0@googlegroups.com> Message-ID: On Fri, May 13, 2016 at 5:52 AM, Guido van Rossum wrote: > Great. We'll be able to move this very quickly once we decide whether it > should be called Type[C] or Class[C]. The original proposal is Type[C], and > it's nice because it's a pun on `type`, just like Tuple[...] is a pun on > `tuple` (and similar for List etc.). But OTOH it really annotates a class > object, not an abstract type. I still prefer Type[C] -- anyone want to argue > that we're making a mistake? (I know, nobody's listening, everyone's too > busy arguing that Path should inherit from str. :-) > I'm definitely busy with things that have nothing to do with that, but why not SubType[C]. That's what it is, right? -- Koos From guido at python.org Fri May 13 11:29:19 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 13 May 2016 08:29:19 -0700 Subject: [Python-ideas] Adding Type[C] support to PEP 484 and typing.py In-Reply-To: References: <7ff0aec2-60c4-4356-8f78-ee8a2eaf9ce0@googlegroups.com> Message-ID: On Fri, May 13, 2016 at 2:56 AM, Koos Zevenhoven wrote: > On Fri, May 13, 2016 at 5:52 AM, Guido van Rossum > wrote: > > Great. We'll be able to move this very quickly once we decide whether it > > should be called Type[C] or Class[C]. The original proposal is Type[C], > and > > it's nice because it's a pun on `type`, just like Tuple[...] is a pun on > > `tuple` (and similar for List etc.). But OTOH it really annotates a class > > object, not an abstract type. I still prefer Type[C] -- anyone want to > argue > > that we're making a mistake? (I know, nobody's listening, everyone's too > > busy arguing that Path should inherit from str. :-) > > > > I'm definitely busy with things that have nothing to do with that, but > why not SubType[C]. That's what it is, right? > It's longer and it's not clearer. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Fri May 13 12:45:49 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 13 May 2016 19:45:49 +0300 Subject: [Python-ideas] Adding Type[C] support to PEP 484 and typing.py In-Reply-To: References: <7ff0aec2-60c4-4356-8f78-ee8a2eaf9ce0@googlegroups.com> Message-ID: On Fri, May 13, 2016 at 6:29 PM, Guido van Rossum wrote: > On Fri, May 13, 2016 at 2:56 AM, Koos Zevenhoven wrote: >> >> I'm definitely busy with things that have nothing to do with that, but >> why not SubType[C]. That's what it is, right? > > > It's longer and it's not clearer. > I'm under the impression that you wanted someone to convince you about a potential mistake being made in naming it 'Type', so: I've heard people use "of" or other prepositions when reading aloud subcripts: Type[C] could be read as "type of C" or something similar. So, I could imagine someone forgetting or failing to realize that there is type(C) that already means "type of C", and that giving the runtime type of an object as a static type hint does not make a lot of sense. I'm not sure Class[C] really solves this problem either. Luckily I can just do SubType = typing.Type ;). To me that is clearer. I can't argue against 'longer', since 7 characters is almost double compared to 4. In addition, I would not mind if a plain `SubType` without [] would be a type hint equivalent to `type`, if that would then be desired for consistency. (not that you would have said you would mind that either). -- Koos From guido at python.org Fri May 13 13:41:14 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 13 May 2016 10:41:14 -0700 Subject: [Python-ideas] Adding Type[C] support to PEP 484 and typing.py In-Reply-To: References: <7ff0aec2-60c4-4356-8f78-ee8a2eaf9ce0@googlegroups.com> Message-ID: Let's not pursue the naming issue further. On Fri, May 13, 2016 at 9:45 AM, Koos Zevenhoven wrote: > On Fri, May 13, 2016 at 6:29 PM, Guido van Rossum > wrote: > > On Fri, May 13, 2016 at 2:56 AM, Koos Zevenhoven > wrote: > >> > >> I'm definitely busy with things that have nothing to do with that, but > >> why not SubType[C]. That's what it is, right? > > > > > > It's longer and it's not clearer. > > > > I'm under the impression that you wanted someone to convince you about > a potential mistake being made in naming it 'Type', so: > > I've heard people use "of" or other prepositions when reading aloud > subcripts: Type[C] could be read as "type of C" or something similar. > So, I could imagine someone forgetting or failing to realize that > there is type(C) that already means "type of C", and that giving the > runtime type of an object as a static type hint does not make a lot of > sense. I'm not sure Class[C] really solves this problem either. > > Luckily I can just do SubType = typing.Type ;). To me that is clearer. > I can't argue against 'longer', since 7 characters is almost double > compared to 4. > > In addition, I would not mind if a plain `SubType` without [] would be > a type hint equivalent to `type`, if that would then be desired for > consistency. (not that you would have said you would mind that > either). > > -- Koos > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Fri May 13 16:30:18 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 13 May 2016 23:30:18 +0300 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: Message-ID: It turns out it has been almost a month since this, and the PEP draft is already looking good. It seems we might now be ready to discuss it. Should we add the generic type FSPath[str]? Again, there is a naming issue, and the question of including plain str and bytes. We'll need to address this, unless we want the type checker to not know whether os.path.* etc. return str or bytes and to carry around Union[str, bytes]. In theory, it would be possible to infer whether it is str or bytes, as described. -- Koos On Tue, Apr 19, 2016 at 3:40 AM, Koos Zevenhoven wrote: > I actually proposed this already in one of the pathlib threads on > python-dev, but I decided to repost here, because this is easily seen > as a separate issue. I'll start with some introduction, then moving on > to the actual type hinting part. > > In our seemingly never-ending discussions about pathlib support in the > stdlib in various threads, first here on python-ideas, then even more > extensively on python-dev, have perhaps almost converged. The required > changes involve a protocol method, probably named __fspath__, which > any path-like type could implement to return a more, let's say, > "classical" path object such as a str. However, the protocol is > polymorphic and may also return bytes, which has a lot do do with the > fact that the stdlib itself is polymophic and currently accepts str as > well as bytes paths almost everywhere, including the newly-introduced > os.scandir + DirEntry combination. The upcoming improvements will > further allow passing pathlib path objects as well as DirEntry objects > to any stdlib function that take paths. > > It came up, for instance here [1], that the function associated with > the protocol, potentially named os.fspath, will end up needing type > hints. This function takes pathlike objects and turns them into str or > bytes. There are various different scenarios [2] that can be > considered for code dealing with paths, but let's consider the case of > os.path.* and other traditional python path-related functions. > > Some examples: > > os.path.join > > Currently, it takes str or bytes paths and returns a joined path of > the same type (mixing different types raises an exception). > > In the future, it will also accept pathlib objects (underlying type > always str) and DirEntry (underlying type str or bytes) or third-party > path objects (underlying type str or bytes). The function will then > return a pathname of the underlying type. > > os.path.dirname > > Currently, it takes a str or bytes and returns the dirname of the same type. > In the future, it will also accept Path and DirEntry and return the > underlying type. > > Let's consider the type hint of os.path.dirname at present and in the future: > > Currently, one could write > > def dirname(p: Union[str, bytes]) -> Union[str, bytes]: > ... > > While this is valid, it could be more precise: > > pathstring = typing.TypeVar('pathstring', str, bytes) > > def dirname(p: pathstring) -> pathstring: > ... > > This now contains the information that the return type is the same as > the argument type. The name 'pathstring' may be considered slightly > misleading because "byte strings" are not actually strings in Python > 3, but at least it does not advertise the use of bytes as paths, which > is very rarely desirable. > > But what about the future. There are two kinds of rich path objects, > those with an underlying type of str and those with an underlying type > of bytes. These should implement the __fspath__() protocol and return > their underlying type. However, we do care about what (underlying) > type is provided by the protocol, so we might want to introduce > something like typing.FSPath[underlying_type]: > > FSPath[str] # str-based pathlike, including str > FSPath[bytes] # bytes-based pathlike, including bytes > > And now, using the above defined TypeVar pathstring, the future > version of dirname would be type annotated as follows: > > def dirname(p: FSPath[pathstring]) -> pathstring: > ... > > It's getting late. I hope this made sense :). > > -Koos > > [1] https://mail.python.org/pipermail/python-dev/2016-April/144246.html > [2] https://mail.python.org/pipermail/python-dev/2016-April/144239.html From ethan at stoneleaf.us Fri May 13 16:50:46 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 13 May 2016 13:50:46 -0700 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: Message-ID: <57363E26.6070908@stoneleaf.us> On 05/13/2016 01:30 PM, Koos Zevenhoven wrote: > It turns out it has been almost a month since this, and the PEP draft > is already looking good. It seems we might now be ready to discuss it. > Should we add the generic type FSPath[str]? Guido's post on one of the other threads: ---------------------------------------- > There's no need for typing.PathLike. So I'm gonna say no. ;) -- ~Ethan~ From k7hoven at gmail.com Fri May 13 18:08:37 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sat, 14 May 2016 01:08:37 +0300 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: <57363E26.6070908@stoneleaf.us> References: <57363E26.6070908@stoneleaf.us> Message-ID: On Fri, May 13, 2016 at 11:50 PM, Ethan Furman wrote: > On 05/13/2016 01:30 PM, Koos Zevenhoven wrote: > >> It turns out it has been almost a month since this, and the PEP draft >> is already looking good. It seems we might now be ready to discuss it. >> Should we add the generic type FSPath[str]? > > > Guido's post on one of the other threads: > ---------------------------------------- >> There's no need for typing.PathLike. > > So I'm gonna say no. ;) > Oh, it looks like a read those two emails in the wrong order ;). Anyway, I was going to suggest making the abstract base class subscriptable too like this: PathABC[str] is a str-based path ABC, and PathABC[bytes] a bytes-based one ;). I don't know if that should be called a generic type or not, though. -- Koos > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From k7hoven at gmail.com Fri May 13 18:45:13 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sat, 14 May 2016 01:45:13 +0300 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: On Sat, May 14, 2016 at 1:08 AM, Koos Zevenhoven wrote: > On Fri, May 13, 2016 at 11:50 PM, Ethan Furman wrote: >> On 05/13/2016 01:30 PM, Koos Zevenhoven wrote: >> >>> It turns out it has been almost a month since this, and the PEP draft >>> is already looking good. It seems we might now be ready to discuss it. >>> Should we add the generic type FSPath[str]? >> >> >> Guido's post on one of the other threads: >> ---------------------------------------- >>> There's no need for typing.PathLike. >> >> So I'm gonna say no. ;) >> > > Oh, it looks like a read those two emails in the wrong order ;). > > Anyway, I was going to suggest making the abstract base class > subscriptable too like this: PathABC[str] is a str-based path ABC, and > PathABC[bytes] a bytes-based one ;). I don't know if that should be > called a generic type or not, though. But maybe making it a generic type would be the way to make the type checker to understand it? Of course if subscripting the ABC is not desired, there could be three ABCs, something like os.StrPath (for str-based path types) and os.BytesPath (for bytes-based path types) and os.StrBytesPath (for str/bytes like DirEntry, unless such classes are split in two). But I suppose there is nothing that would help combining this with a TypeVar-like thing. Then again, when working in pure python with a pure python fspath and with the __fspath__ method properly annotated in the path class, the type checker should probably be able to infer the types. Maybe Guido was referring to this. -- Koos > -- Koos > > >> -- >> ~Ethan~ >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ From guido at python.org Fri May 13 15:31:04 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 13 May 2016 12:31:04 -0700 Subject: [Python-ideas] Adding Type[C] to PEP 484 Message-ID: I've got another proposed update to PEP 484 ready. There was previously some discussion on python-ideas (Bcc'ed for closure) and quite a bit on the typing issue tracker: https://github.com/python/typing/issues/107 I've worked all that into a patch for PEP 484: https://github.com/python/typing/pull/218 Once we're okay with that I will work on getting the implementation ready in time for Python 3.5.2. Here's the added text: Meta-types ---------- A meta-type can be used to indicate a value that is itself a class object that is a subclass of a given class. It is spelled as ``Type[C]`` where ``C`` is a class. To clarify: while ``C`` (when used as an annotation) refers to instances of class ``C``, ``Type[C]`` refers to *subclasses* of ``C``. (This is a similar distinction as between ``object`` and ``type``.) For example, suppose we have the following classes:: class User: ... # Abstract base for User classes class BasicUser(User): ... class ProUser(User): ... class TeamUser(User): ... And suppose we have a function that creates an instance of one of these classes if you pass it a class object:: def new_user(user_class): user = user_class() # (Here we could write the user object to a database) return user Without ``Type[]`` the best we could do to annotate ``new_user()`` would be:: def new_user(user_class: type) -> User: ... However using ``Type[]`` and a type variable with an upper bound we can do much better:: U = TypeVar('U', bound=User) def new_user(user_class: Type[U]) -> U: ... Now when we call ``new_user()`` with a specific subclass of ``User`` a type checker will infer the correct type of the result:: joe = new_user(BasicUser) # Inferred type is BasicUser At runtime the value corresponding to ``Type[C]`` must be an actual class object that's a subtype of ``C``, not a special form. IOW, in the above example calling e.g. ``new_user(Union[BasicUser, ProUser])`` is not allowed. There are some concerns with this feature: for example when ``new_user()`` calls ``user_class()`` this implies that all subclasses of ``User`` must support this in their constructor signature. However this is not unique to ``Type[]``: class methods have similar concerns. A type checker ought to flag violations of such assumptions, but by default constructor calls that match the constructor signature in the indicated base class (``User`` in the example above) should be allowed. A program containing a complex or extensible class hierarchy might also handle this by using a factory class method. Plain ``Type`` without brackets is equivalent to using ``type`` (the root of Python's metaclass hierarchy). This equivalence also motivates the name, ``Type``, as opposed to alternatives like ``Class`` or ``SubType``, which were proposed while this feature was under discussion; this is similar to the relationship between e.g. ``List`` and ``list``. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri May 13 19:05:20 2016 From: brett at python.org (Brett Cannon) Date: Fri, 13 May 2016 23:05:20 +0000 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: On Fri, 13 May 2016 at 15:09 Koos Zevenhoven wrote: > On Fri, May 13, 2016 at 11:50 PM, Ethan Furman wrote: > > On 05/13/2016 01:30 PM, Koos Zevenhoven wrote: > > > >> It turns out it has been almost a month since this, and the PEP draft > >> is already looking good. It seems we might now be ready to discuss it. > >> Should we add the generic type FSPath[str]? > > > > > > Guido's post on one of the other threads: > > ---------------------------------------- > >> There's no need for typing.PathLike. > > > > So I'm gonna say no. ;) > > > > Oh, it looks like a read those two emails in the wrong order ;). > > Anyway, I was going to suggest making the abstract base class > subscriptable too like this: PathABC[str] is a str-based path ABC, and > PathABC[bytes] a bytes-based one ;). I don't know if that should be > called a generic type or not, though. > The PEP already addresses this and says "no". -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Fri May 13 19:14:16 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sat, 14 May 2016 02:14:16 +0300 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: On Sat, May 14, 2016 at 2:05 AM, Brett Cannon wrote: > > > On Fri, 13 May 2016 at 15:09 Koos Zevenhoven wrote: >> >> On Fri, May 13, 2016 at 11:50 PM, Ethan Furman wrote: >> > On 05/13/2016 01:30 PM, Koos Zevenhoven wrote: >> > >> >> It turns out it has been almost a month since this, and the PEP draft >> >> is already looking good. It seems we might now be ready to discuss it. >> >> Should we add the generic type FSPath[str]? >> > >> > >> > Guido's post on one of the other threads: >> > ---------------------------------------- >> >> There's no need for typing.PathLike. >> > >> > So I'm gonna say no. ;) >> > >> >> Oh, it looks like a read those two emails in the wrong order ;). >> >> Anyway, I was going to suggest making the abstract base class >> subscriptable too like this: PathABC[str] is a str-based path ABC, and >> PathABC[bytes] a bytes-based one ;). I don't know if that should be >> called a generic type or not, though. > > > The PEP already addresses this and says "no". I obviously know the PEP very well, and it doesn't. But I'm probably just doing a bad job explaining what I mean right now, and should probably go to bed. Sorry. -- Koos From victor.stinner at gmail.com Fri May 13 19:27:45 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 14 May 2016 01:27:45 +0200 Subject: [Python-ideas] Sleep and timedelta Was: Boolean parameters guidelines In-Reply-To: <57344080.5060200@mail.de> References: <57344080.5060200@mail.de> Message-ID: See also the PEP 410 which is somehow related: http://legacy.python.org/dev/peps/pep-0410/ (and maybe also the PEP 418) Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri May 13 19:41:34 2016 From: brett at python.org (Brett Cannon) Date: Fri, 13 May 2016 23:41:34 +0000 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: On Fri, 13 May 2016 at 16:14 Koos Zevenhoven wrote: > On Sat, May 14, 2016 at 2:05 AM, Brett Cannon wrote: > > > > > > On Fri, 13 May 2016 at 15:09 Koos Zevenhoven wrote: > >> > >> On Fri, May 13, 2016 at 11:50 PM, Ethan Furman > wrote: > >> > On 05/13/2016 01:30 PM, Koos Zevenhoven wrote: > >> > > >> >> It turns out it has been almost a month since this, and the PEP draft > >> >> is already looking good. It seems we might now be ready to discuss > it. > >> >> Should we add the generic type FSPath[str]? > >> > > >> > > >> > Guido's post on one of the other threads: > >> > ---------------------------------------- > >> >> There's no need for typing.PathLike. > >> > > >> > So I'm gonna say no. ;) > >> > > >> > >> Oh, it looks like a read those two emails in the wrong order ;). > >> > >> Anyway, I was going to suggest making the abstract base class > >> subscriptable too like this: PathABC[str] is a str-based path ABC, and > >> PathABC[bytes] a bytes-based one ;). I don't know if that should be > >> called a generic type or not, though. > > > > > > The PEP already addresses this and says "no". > > I obviously know the PEP very well, and it doesn't. But I'm probably > just doing a bad job explaining what I mean right now, and should > probably go to bed. Sorry. > Ah, I now see what you're proposing: somehow making the ABC a generic like the generics types in the typing module are (which would be a new feature for ABCs and why I didn't realize what you were initially asking for; sorry about that). The answer is still "no". :) The generics support from the typing module is specific to that module and not applicable to ABCs themselves. Think of ABCs as helping guarantee that you implement specific methods and attributes. If your typing needs are met simply by that level of type information, then ABCs are fine as a type hint. But if you need something more specific like generics support then that's when it goes into the typing module (IOW don't think of the types in the typing module as ABCs or vice-versa but as separate things both dealing with duck-typing for their own purposes). Since the PEP already ruled out adding generics support for a special type representing os.PathLike or path-like objects, that also means it shouldn't be pushed into the ABC as an end-run around not going into the typing module. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sat May 14 01:28:51 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 14 May 2016 14:28:51 +0900 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: References: Message-ID: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> Executive summary: Inference of str vs bytes (in particular, which component of a "sum" type in general) is not always possible, and it's not obvious how beneficial it is. Proposal: have typecheckers recognize "suspicious" unions, and complain if they are not "converted" to a component type before passing to a polymorphic function. Koos Zevenhoven writes: > We'll need to address this, unless we want the type checker to not > know whether os.path.* etc. return str or bytes and to carry around > Union[str, bytes]. In theory, it would be possible to infer whether it > is str or bytes, as described. I'm -0.5 = "more trouble than it's worth". Rationale: *Sometimes* it would be possible. OTOH, modules like os.path *will* end up carrying around Union[str, bytes], precisely because they will be called in both contexts, and that's not predictable at the time of type-checking os.path. So what you really want is a Sum type constructor, where Sum is in the sense of category theory's sum functor. That is, not only is the result Type a (disjoint) union, but you're also guaranteed that each operation respects the component type. I could argue that this is why Guido was right to remove the restriction that os.fspath return str. The __fspath__ method is part of the "sum category os.path"[1], which is a collection of operations on bytes and on str that parallel each other because they are equivalent representations of filesystem paths. os.fspath therefore is also part of this category. If you want to take the result out of this category and make it str, use os.fsdecode (which is *not* part of this category *because* the codomain is str, not Sum[bytes,str]). Note that Union is still a useful type if we have Sum. For example, the operator division / : Tuple(Union(Float,Int),Union(Float,Int)) -> Union(Float,Int) might take 1.0/1.0 -> 1 (in Common Lisp it does, although in Python it doesn't). So the "sum" requirement that the map respect "original" type is a language design question, not a restriction we should apply for the fun of it. Now, how would the type checker treat __fspath__? Borrowing a type from Ethan, it needs to observe that there was bytes object in the main function that got passed to the Antipathy Sum[str,bytes] -> Path constructor. It would then tag that Path p as a bytes-y Path, and further tag the value of os.fspath(p) as bytes. So we would need to add the concept of a ComponentType-y object of Type SumType to the typing module. That sounds complicated to me, and typically not all that useful: people living at the level of bytes will be swimming in bytes, ditto str-people. Polymorphism "just works" for them. People working at the interface will be using functions like os.fsdecode that ensure type in the nature of things. They have to be more careful, but use of a Union[str, bytes] as an argument to foo(b: bytes) will already be flagged. I think TRT here is to provide a way to tell the type checker that certain Unions should be converted as soon as possible, and not allow passing them even to polymorphic functions. So: def foo(d: DirEntry) -> Int: s = os.fspath(d) with open(s) as f: # Flag use of s as Union[str, bytes]. # open() doesn't care, but the type # checker knows that such Unions # should only be passed to conversion # functions. return len(f) while def foo(d: DirEntry) -> Int: s = os.fspath(d) s = os.fsdecode(s) # Converted to str immediately, clear since # fsdecode Type is "SuspiciousUnion -> str". with open(s) as f: return len(f) is OK. This is on the "catch bugs early" principle, and respects the generally accepted principle that in "most" applications, encoded bytes should be decoded to str "at the boundary". Of course this would be optional (eg, it would be off in a standalone check of os.path for type sanity), and not all Unions are subject to this treatment (eg, it would make no sense for numbers). On the other hand, *inside* a function that is respectfully polymorphic, the annotation foo(s: Sum[str, bytes]) -> Sum[str, bytes] would tell the typechecker to ensure that in foo's implementation, if bytes go in, bytes come out, if str goes in, str comes out. That might be very useful (but I don't have a strong opinion). A nit: what would len(s: Sum[str, bytes]) -> Int mean? In this case it's obviously equivalent to len(s: Union[str, bytes]) -> Int, but I'm not sure it's all that obvious in other possible cases. Finally, in many cases Sum types just are not going to be useful. You might want to alias Number to Sum[Float, Int] because (in Python) Float + Float -> Float and Int + Int -> Int, but then you have two problems. First, you can't do inference on mixed-type arithmetic, Sum doesn't permit that.[2] Second, you can't do inference on Int / Int. Footnotes: [1] The quotes mean "I think I could make this precise but it would be boring. Bear with me, and don't be distracted by the fact that lots of stuff in this 'category' aren't in the os.path *module* -- eg, open() and __fspath__ itself." I hope that the categorical language will be a useful metaphor for those who understand category theory, but really the whole thing is a hand-wavy path to "what implementation would look like". [2] This is also a handwavy statement. Technically, the arithmetic we're talking about is binary: Tuple[Sum, Sum] -> Sum and it could do anything with mixed types, as Sum can only provide restrictions on unary: Sum -> Sum operation. So a more accurate way to put the point is that it's not really obvious what properties Tuple[Sum, Sum] should have, especially in the context of a function whose value is of the same Sum Type. And of course the problem with Int/Int is real -- it's hard to imagine how that could be handled by a general rule about Tuple[Sum, Sum] -> Sum: surely if both operands are of the same type, the value should be of that type! But we decided otherwise when designing Python 3. From k7hoven at gmail.com Sat May 14 06:32:52 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sat, 14 May 2016 13:32:52 +0300 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> Message-ID: First of all, thanks for posting, Stephen. I will be responding more thoroughly later, I'll first comment on some points. On Sat, May 14, 2016 at 8:28 AM, Stephen J. Turnbull wrote: [...] > I could argue that this is why Guido was right to remove the > restriction that os.fspath return str. The __fspath__ method is part > of the "sum category os.path"[1], which is a collection of operations > on bytes and on str that parallel each other because they are > equivalent representations of filesystem paths. os.fspath therefore > is also part of this category. If you want to take the result out of > this category and make it str, use os.fsdecode (which is *not* part of > this category *because* the codomain is str, not Sum[bytes,str]). Yes, indeed. This was also the reason why, in the python-dev discussions, I have been arguing for __fspath__ to be able to return both str and bytes *from the beginning*, although I have phrased it quite differently and I think in several different ways. An this is also the reason why, more recently, I've been arguing that os.fspath should *always* allow this too. Before that, I was compromizing towards desires to enforce str, while making the polymorphicity optionally available. I was very happy to learn Guido thought the same, because it meant I could stop arguing for that. If only the path discussion would have had a better signal-to-noise ratio, then this would perhaps have happened faster and many many many man hours would have been saved. -- Koos > Footnotes: > [1] The quotes mean "I think I could make this precise but it would > be boring. Bear with me, and don't be distracted by the fact that > lots of stuff in this 'category' aren't in the os.path *module* -- eg, > open() and __fspath__ itself." I hope that the categorical language > will be a useful metaphor for those who understand category theory, > but really the whole thing is a hand-wavy path to "what implementation > would look like". From guido at python.org Sat May 14 13:27:48 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 14 May 2016 10:27:48 -0700 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> Message-ID: I'm confused by the terminology in this discussion. PEP 484 has type variables with constraints, and the typing module predefines AnyStr = TypeVar('AnyStr', str, bytes). This is then used to define various polymorphic functions, e.g. in os.path: def abspath(path: AnyStr) -> AnyStr: ... This means that the type checker will know that the type of abspath(b'.') is bytes, the type of abspath('.') is str. There's also e.g. def relpath(path: AnyStr, start: AnyStr = ...) -> AnyStr: ... which means that path and start must be either both bytes or both str. For reference, this code lives in typeshed: https://github.com/python/typeshed/blob/master/stdlib/3/os/path.pyi Is that at all useful? On Fri, May 13, 2016 at 10:28 PM, Stephen J. Turnbull wrote: > Executive summary: > > Inference of str vs bytes (in particular, which component of a "sum" > type in general) is not always possible, and it's not obvious how > beneficial it is. Proposal: have typecheckers recognize "suspicious" > unions, and complain if they are not "converted" to a component type > before passing to a polymorphic function. > > Koos Zevenhoven writes: > > > We'll need to address this, unless we want the type checker to not > > know whether os.path.* etc. return str or bytes and to carry around > > Union[str, bytes]. In theory, it would be possible to infer whether it > > is str or bytes, as described. > > I'm -0.5 = "more trouble than it's worth". Rationale: > > *Sometimes* it would be possible. OTOH, modules like os.path *will* > end up carrying around Union[str, bytes], precisely because they will > be called in both contexts, and that's not predictable at the time of > type-checking os.path. > > So what you really want is a Sum type constructor, where Sum is in the > sense of category theory's sum functor. That is, not only is the > result Type a (disjoint) union, but you're also guaranteed that each > operation respects the component type. > > I could argue that this is why Guido was right to remove the > restriction that os.fspath return str. The __fspath__ method is part > of the "sum category os.path"[1], which is a collection of operations > on bytes and on str that parallel each other because they are > equivalent representations of filesystem paths. os.fspath therefore > is also part of this category. If you want to take the result out of > this category and make it str, use os.fsdecode (which is *not* part of > this category *because* the codomain is str, not Sum[bytes,str]). > > Note that Union is still a useful type if we have Sum. For example, > the operator division / : Tuple(Union(Float,Int),Union(Float,Int)) -> > Union(Float,Int) might take 1.0/1.0 -> 1 (in Common Lisp it does, > although in Python it doesn't). So the "sum" requirement that the map > respect "original" type is a language design question, not a > restriction we should apply for the fun of it. > > Now, how would the type checker treat __fspath__? Borrowing a type > from Ethan, it needs to observe that there was bytes object in the > main function that got passed to the Antipathy Sum[str,bytes] -> Path > constructor. It would then tag that Path p as a bytes-y Path, and > further tag the value of os.fspath(p) as bytes. So we would need to > add the concept of a ComponentType-y object of Type SumType to the > typing module. That sounds complicated to me, and typically not all > that useful: people living at the level of bytes will be swimming in > bytes, ditto str-people. Polymorphism "just works" for them. People > working at the interface will be using functions like os.fsdecode that > ensure type in the nature of things. They have to be more careful, > but use of a Union[str, bytes] as an argument to foo(b: bytes) will > already be flagged. > > I think TRT here is to provide a way to tell the type checker that > certain Unions should be converted as soon as possible, and not allow > passing them even to polymorphic functions. So: > > def foo(d: DirEntry) -> Int: > s = os.fspath(d) > with open(s) as f: # Flag use of s as Union[str, bytes]. > # open() doesn't care, but the type > # checker knows that such Unions > # should only be passed to conversion > # functions. > return len(f) > > while > > def foo(d: DirEntry) -> Int: > s = os.fspath(d) > s = os.fsdecode(s) # Converted to str immediately, clear since > # fsdecode Type is "SuspiciousUnion -> > str". > with open(s) as f: > return len(f) > > is OK. This is on the "catch bugs early" principle, and respects the > generally accepted principle that in "most" applications, encoded > bytes should be decoded to str "at the boundary". Of course this > would be optional (eg, it would be off in a standalone check of > os.path for type sanity), and not all Unions are subject to this > treatment (eg, it would make no sense for numbers). > > On the other hand, *inside* a function that is respectfully > polymorphic, the annotation foo(s: Sum[str, bytes]) -> Sum[str, bytes] > would tell the typechecker to ensure that in foo's implementation, if > bytes go in, bytes come out, if str goes in, str comes out. That > might be very useful (but I don't have a strong opinion). > > A nit: what would len(s: Sum[str, bytes]) -> Int mean? In this case > it's obviously equivalent to len(s: Union[str, bytes]) -> Int, but I'm > not sure it's all that obvious in other possible cases. > > Finally, in many cases Sum types just are not going to be useful. You > might want to alias Number to Sum[Float, Int] because (in Python) > Float + Float -> Float and Int + Int -> Int, but then you have two > problems. First, you can't do inference on mixed-type arithmetic, Sum > doesn't permit that.[2] Second, you can't do inference on Int / Int. > > > Footnotes: > [1] The quotes mean "I think I could make this precise but it would > be boring. Bear with me, and don't be distracted by the fact that > lots of stuff in this 'category' aren't in the os.path *module* -- eg, > open() and __fspath__ itself." I hope that the categorical language > will be a useful metaphor for those who understand category theory, > but really the whole thing is a hand-wavy path to "what implementation > would look like". > > [2] This is also a handwavy statement. Technically, the arithmetic > we're talking about is binary: Tuple[Sum, Sum] -> Sum and it could > do anything with mixed types, as Sum can only provide restrictions on > unary: Sum -> Sum operation. So a more accurate way to put the point > is that it's not really obvious what properties Tuple[Sum, Sum] should > have, especially in the context of a function whose value is of the > same Sum Type. And of course the problem with Int/Int is real -- it's > hard to imagine how that could be handled by a general rule about > Tuple[Sum, Sum] -> Sum: surely if both operands are of the same type, > the value should be of that type! But we decided otherwise when > designing Python 3. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From leewangzhong+python at gmail.com Sat May 14 19:26:50 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Sat, 14 May 2016 19:26:50 -0400 Subject: [Python-ideas] Boolean parameters guidelines In-Reply-To: References: <5733596F.50803@mail.de> <5733AACA.4020702@stoneleaf.us> Message-ID: On May 12, 2016 4:29 AM, "Serhiy Storchaka" wrote: > > On 12.05.16 02:48, Chris Angelico wrote: >>>>> >>>>> from time import sleep >>>>> help(sleep) >> >> Help on built-in function sleep in module time: >> >> sleep(...) >> sleep(seconds) >> >> Delay execution for a given number of seconds. The argument may be >> a floating point number for subsecond precision. >> >>>>> sleep(seconds=1) >> >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: sleep() takes no keyword arguments > > > Note that passing positional arguments is much faster than passing keyword arguments even if the function supports them. And the difference can be even larger after pushing Victor's optimization. [1] > Other optimizations (like [2]) can be applied only for functions that support only positional parameters. Thus passing positional arguments and having functions that support only positional parameters is important for performance. > > [1] http://bugs.python.org/issue26814 > [2] http://bugs.python.org/issue23867 Worth noting that a guarding optimizer, or a "hinted" optimizer (like a decorator on the caller which locks in specified names for functions), can theoretically optimize away those keyword args to positional args. So if performance is important, there could be a future where readability doesn't have to be sacrificed, and this idiom would put pressure on creating that future. (Victor's other work on optimization is about both of those kinds of optimizers, I think. I suggested this optimization, but never did anything about it. It's on the todo list: https://fatoptimizer.readthedocs.io/en/latest/todo.html#random) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat May 14 22:46:59 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 15 May 2016 12:46:59 +1000 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> Message-ID: <20160515024658.GJ12028@ando.pearwood.info> Sorry Stephen, I feel like I've fallen down the rabbit hole here... your posts are usually a paragon of clarity but this one is just beyond me :( On Sat, May 14, 2016 at 02:28:51PM +0900, Stephen J. Turnbull wrote: > Executive summary: > > Inference of str vs bytes (in particular, which component of a "sum" > type in general) is not always possible, and it's not obvious how > beneficial it is. Proposal: have typecheckers recognize "suspicious" > unions, and complain if they are not "converted" to a component type > before passing to a polymorphic function. [...] > So what you really want is a Sum type constructor, where Sum is in the > sense of category theory's sum functor. That is, not only is the > result Type a (disjoint) union, but you're also guaranteed that each > operation respects the component type. Are we supposed to know what category theory's sum functor is? I'm warning you, if you start talking about Monads I'm just going hit delete on your post :-) I'm afraid I don't understand what you mean by a Sum type constructor. Can you explain in more detail? I also don't understand what you mean by '"suspicious" unions' (quotes in original) or what makes them suspicious, or "suspicious" as the case may be. What's suspicious about Union[bytes, str]? Are you proposing a Sum contructor for the typing module, as an alternative to Union? What will it do? If not, why are you talking about Sum[bytes, str]? -- Steve From k7hoven at gmail.com Sun May 15 06:02:29 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sun, 15 May 2016 13:02:29 +0300 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> Message-ID: On Sat, May 14, 2016 at 8:28 AM, Stephen J. Turnbull wrote: > Executive summary: > > Inference of str vs bytes (in particular, which component of a "sum" > type in general) is not always possible, and it's not obvious how > beneficial it is. Proposal: have typecheckers recognize "suspicious" > unions, and complain if they are not "converted" to a component type > before passing to a polymorphic function. > Maybe you mean to propose that the concept of sum types should be introduced in typing/mypy. I don't think we want that, because annotating types with them would probably be clumsy, and there is too much overlap with things that are already available (unless of course some of that gets thrown away in favor of Sum). > Koos Zevenhoven writes: > > > We'll need to address this, unless we want the type checker to not > > know whether os.path.* etc. return str or bytes and to carry around > > Union[str, bytes]. In theory, it would be possible to infer whether it > > is str or bytes, as described. > > I'm -0.5 = "more trouble than it's worth". Rationale: I could imagine that being the conclusion at this point. I just feel bad for reducing the precision in the "type-hintability" of things like os.path.* as a consequence of the new fspath PEP, which I'm otherwise already quite happy with. That's the problem I'm trying to solve. I'm perfectly fine with it if the conclusion is that it's not worth it. I just don't want to say I broke the type hinting for os.path.* for people that want mypy to catch their errors of mixing bytes and str paths in the wrong way, or errors of mixing str/bytes-polymorphic path code with str concatenation etc. in the wrong way. But if I'm told Unions are sufficient, I'll be happy with that, because I'm not a significant user of type hints anyway. So please, someone tell me that. In my OP in the original thread, I describe how the os.path functions could presently be hinted (I guess someone may have already done this, but I did not check). In the future, It may be preferable to tell the type checker that passing a PurePath into os.path functions results in str being returned, as I also explain using a parametrizable type. > *Sometimes* it would be possible. OTOH, modules like os.path *will* > end up carrying around Union[str, bytes], precisely because they will > be called in both contexts, and that's not predictable at the time of > type-checking os.path. > My proposal is aimed at making it possible to *always* have mypy know the types. > So what you really want is a Sum type constructor, where Sum is in the > sense of category theory's sum functor. That is, not only is the > result Type a (disjoint) union, but you're also guaranteed that each > operation respects the component type. > (Side note: I don't know why you capitalize T in "Type" here, it does not seem to make sense in the context that Guido has just been describing as Type. Is this confusion an argument against calling Type "Type"?) My point was to parametrize the type (hint) for path objects [pretty much exactly (like) a generic], which roughly means that the type checker can be told how one disjoint union as the argument type gets turned into another disjoint union as the return type. I suppose one might write this in English as "In a typical os.path.* function such as os.path.dirname, Union[PathABC[str], str] and Union[PathABC[bytes], bytes] get turned into str and bytes, *respectively*." Mypy/PEP483-4 approaches this problem by using Generics and TypeVars. Parametrizing the path type by the underlying type such as str in PathABC[str] would allow using a TypeVar to indicate "respectively". However, there is also @overload to indicate this. > I could argue that this is why Guido was right to remove the > restriction that os.fspath return str. The __fspath__ method is part > of the "sum category os.path"[1], which is a collection of operations > on bytes and on str that parallel each other because they are > equivalent representations of filesystem paths. os.fspath therefore > is also part of this category. If you want to take the result out of > this category and make it str, use os.fsdecode (which is *not* part of > this category *because* the codomain is str, not Sum[bytes,str]). Indeed, as mentioned in my previous post, you seem to be describing exactly my earlier arguments, but using a different (or additional) set of terminology. > Note that Union is still a useful type if we have Sum. For example, > the operator division / : Tuple(Union(Float,Int),Union(Float,Int)) -> > Union(Float,Int) might take 1.0/1.0 -> 1 (in Common Lisp it does, > although in Python it doesn't). So the "sum" requirement that the map > respect "original" type is a language design question, not a > restriction we should apply for the fun of it. Yes, Unions will always be needed. > Now, how would the type checker treat __fspath__? Borrowing a type > from Ethan, it needs to observe that there was bytes object in the > main function that got passed to the Antipathy Sum[str,bytes] -> Path > constructor. It would then tag that Path p as a bytes-y Path, and > further tag the value of os.fspath(p) as bytes. So we would need to > add the concept of a ComponentType-y object of Type SumType to the > typing module. That sounds complicated to me, and typically not all > that useful: people living at the level of bytes will be swimming in > bytes, ditto str-people. Polymorphism "just works" for them. People > working at the interface will be using functions like os.fsdecode that > ensure type in the nature of things. They have to be more careful, > but use of a Union[str, bytes] as an argument to foo(b: bytes) will > already be flagged. Indeed, as mentioned, there are already approaches for this. > > I think TRT here is to provide a way to tell the type checker that > certain Unions should be converted as soon as possible, and not allow > passing them even to polymorphic functions. So: > > def foo(d: DirEntry) -> Int: > s = os.fspath(d) > with open(s) as f: # Flag use of s as Union[str, bytes]. > # open() doesn't care, but the type > # checker knows that such Unions > # should only be passed to conversion > # functions. > return len(f) > > while > > def foo(d: DirEntry) -> Int: > s = os.fspath(d) > s = os.fsdecode(s) # Converted to str immediately, clear since > # fsdecode Type is "SuspiciousUnion -> str". > with open(s) as f: > return len(f) > There is nothing suspicious about that Union there. "Union[str, bytes] -> str" is a perfectly good type hint for os.fsdecode. You can't get any more precise than that. Some nitpicking: Your function foo should probably be hinted to work with any path, not just DirEntry, besides, there seems to be no right way to currently import DirEntry, you won't actually be able to use that as a static type hint. I'm not sure this example has anything to do with what you (or I) are proposing, except that os.fsdecode and os.fsencode indeed make sure that the output type is no longer a Union. So maybe you are just calling Unions in general suspicious, because without a context you can't know exactly which runtime type it should be. Luckily our PEP will turn your example into: def foo(d: TheTypeHintForAnyPath) -> int: with open(d) as f: return len(f) > is OK. This is on the "catch bugs early" principle, and respects the > generally accepted principle that in "most" applications, encoded > bytes should be decoded to str "at the boundary". Of course this > would be optional (eg, it would be off in a standalone check of > os.path for type sanity), and not all Unions are subject to this > treatment (eg, it would make no sense for numbers). One definitely should not use os.fsdecode or os.fsencode just to help the type checker. I hope that is not what you meant. > On the other hand, *inside* a function that is respectfully > polymorphic, the annotation foo(s: Sum[str, bytes]) -> Sum[str, bytes] > would tell the typechecker to ensure that in foo's implementation, if > bytes go in, bytes come out, if str goes in, str comes out. That > might be very useful (but I don't have a strong opinion). Indeed, this is almost the same as TypeVar or an overload. > A nit: what would len(s: Sum[str, bytes]) -> Int mean? In this case > it's obviously equivalent to len(s: Union[str, bytes]) -> Int, but I'm > not sure it's all that obvious in other possible cases. > Also here. - Koos > Finally, in many cases Sum types just are not going to be useful. You > might want to alias Number to Sum[Float, Int] because (in Python) > Float + Float -> Float and Int + Int -> Int, but then you have two > problems. First, you can't do inference on mixed-type arithmetic, Sum > doesn't permit that.[2] Second, you can't do inference on Int / Int. > > > Footnotes: > [1] The quotes mean "I think I could make this precise but it would > be boring. Bear with me, and don't be distracted by the fact that > lots of stuff in this 'category' aren't in the os.path *module* -- eg, > open() and __fspath__ itself." I hope that the categorical language > will be a useful metaphor for those who understand category theory, > but really the whole thing is a hand-wavy path to "what implementation > would look like". > > [2] This is also a handwavy statement. Technically, the arithmetic > we're talking about is binary: Tuple[Sum, Sum] -> Sum and it could > do anything with mixed types, as Sum can only provide restrictions on > unary: Sum -> Sum operation. So a more accurate way to put the point > is that it's not really obvious what properties Tuple[Sum, Sum] should > have, especially in the context of a function whose value is of the > same Sum Type. And of course the problem with Int/Int is real -- it's > hard to imagine how that could be handled by a general rule about > Tuple[Sum, Sum] -> Sum: surely if both operands are of the same type, > the value should be of that type! But we decided otherwise when > designing Python 3. > From k7hoven at gmail.com Sun May 15 06:51:50 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sun, 15 May 2016 13:51:50 +0300 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: On Sat, May 14, 2016 at 2:41 AM, Brett Cannon wrote: > > On Fri, 13 May 2016 at 16:14 Koos Zevenhoven wrote: >> >> On Sat, May 14, 2016 at 2:05 AM, Brett Cannon wrote: >> > >> > On Fri, 13 May 2016 at 15:09 Koos Zevenhoven wrote: >> >> >> >> Anyway, I was going to suggest making the abstract base class >> >> subscriptable too like this: PathABC[str] is a str-based path ABC, and >> >> PathABC[bytes] a bytes-based one ;). I don't know if that should be >> >> called a generic type or not, though. >> > >> > >> > The PEP already addresses this and says "no". >> >> I obviously know the PEP very well, and it doesn't. But I'm probably >> just doing a bad job explaining what I mean right now, and should >> probably go to bed. Sorry. > > > Ah, I now see what you're proposing: somehow making the ABC a generic like > the generics types in the typing module are (which would be a new feature > for ABCs and why I didn't realize what you were initially asking for; sorry > about that). The answer is still "no". :) I'm not sure that this is strictly a new feature, although I suppose there is no example of such an ABC at the moment in the stdlib. But I suppose there is a reason why, for instance, typing.Sequence and collections.abc.Sequence are not merged together. Maybe that is to limit the complexity of the already complex type stuff at this point. The question of whether the ABC could be subscripted to determine the underlying type can be viewed as separate from whether it inherits from Generic[...] or not. But IIUC, inheriting from Generic[...] is the thing that Mypy understands. > The generics support from the typing module is specific to that module and > not applicable to ABCs themselves. Specific to that module? Maybe you mean the convention of having the stdlib generics in typing.py. > Think of ABCs as helping guarantee that > you implement specific methods and attributes. Yes, that's pretty much what I do ;-). And as I already suggested, one could also avoid the subscripting part by defining separate ABCs, os.StrPath, os.BytesPath. This still wouldn't allow parametrizing with a TypeVar, but at least one could write [for os(.path) functions] @overload def dirname(p: Union[str, StrPath]) -> str: ... @overload def dirname(p: Union[bytes, BytesPath] -> str: ... and @overload def fspath(p: Union[str, StrPath]) -> str: ... @overload def fspath(p: Union[bytes, BytesPath] -> str: ... - Koos P.S. The situation with DirEntry requires more considerations, because it can have either underlying type. From k7hoven at gmail.com Sun May 15 06:55:49 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sun, 15 May 2016 13:55:49 +0300 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: A copy-paste error of mine corrected below, sorry. On Sun, May 15, 2016 at 1:51 PM, Koos Zevenhoven wrote: > On Sat, May 14, 2016 at 2:41 AM, Brett Cannon wrote: >> >> On Fri, 13 May 2016 at 16:14 Koos Zevenhoven wrote: >>> >>> On Sat, May 14, 2016 at 2:05 AM, Brett Cannon wrote: >>> > >>> > On Fri, 13 May 2016 at 15:09 Koos Zevenhoven wrote: >>> >> >>> >> Anyway, I was going to suggest making the abstract base class >>> >> subscriptable too like this: PathABC[str] is a str-based path ABC, and >>> >> PathABC[bytes] a bytes-based one ;). I don't know if that should be >>> >> called a generic type or not, though. >>> > >>> > >>> > The PEP already addresses this and says "no". >>> >>> I obviously know the PEP very well, and it doesn't. But I'm probably >>> just doing a bad job explaining what I mean right now, and should >>> probably go to bed. Sorry. >> >> >> Ah, I now see what you're proposing: somehow making the ABC a generic like >> the generics types in the typing module are (which would be a new feature >> for ABCs and why I didn't realize what you were initially asking for; sorry >> about that). The answer is still "no". :) > > I'm not sure that this is strictly a new feature, although I suppose > there is no example of such an ABC at the moment in the stdlib. But I > suppose there is a reason why, for instance, typing.Sequence and > collections.abc.Sequence are not merged together. Maybe that is to > limit the complexity of the already complex type stuff at this point. > > The question of whether the ABC could be subscripted to determine the > underlying type can be viewed as separate from whether it inherits > from Generic[...] or not. But IIUC, inheriting from Generic[...] is > the thing that Mypy understands. > >> The generics support from the typing module is specific to that module and >> not applicable to ABCs themselves. > > Specific to that module? Maybe you mean the convention of having the > stdlib generics in typing.py. > >> Think of ABCs as helping guarantee that >> you implement specific methods and attributes. > > Yes, that's pretty much what I do ;-). > > And as I already suggested, one could also avoid the subscripting part > by defining separate ABCs, os.StrPath, os.BytesPath. This still > wouldn't allow parametrizing with a TypeVar, but at least one could > write [for os(.path) functions] > > @overload > def dirname(p: Union[str, StrPath]) -> str: > ... > @overload > def dirname(p: Union[bytes, BytesPath] -> bytes: > ... corrected to "-> bytes" > and > > @overload > def fspath(p: Union[str, StrPath]) -> str: > ... > @overload > def fspath(p: Union[bytes, BytesPath] -> bytes: > ... corrected to "-> bytes" > - Koos > > P.S. The situation with DirEntry requires more considerations, because > it can have either underlying type. From cesare.di.mauro at gmail.com Sun May 15 09:48:30 2016 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Sun, 15 May 2016 15:48:30 +0200 Subject: [Python-ideas] Exposing flat bytecode representation to optimizers In-Reply-To: References: <1763502835.2002313.1454702338876.JavaMail.yahoo.ref@mail.yahoo.com> <1763502835.2002313.1454702338876.JavaMail.yahoo@mail.yahoo.com> Message-ID: 2016-02-05 21:35 GMT+01:00 Serhiy Storchaka : > LGTM. I did have the same idea when added specialized 8-bit opcodes. > > Some optimization (like constant folding) it is worth to move yet one step > earlier, to AST. > In fact. In WPython I moved all constant folding to the ASDL. > Other idea - instead of EXTENDED_ARG have two sets of instructions: short > 16-bit with 8-bit arg, and long 32-bit with 24-bit arg. For simplicity > initially only long instructions are emitted (if it makes sense). Don't do it with all instructions, but only with the jump ones, which are the only susceptible of such changes. In all other cases you already know the size of the argument, which will not change. Regards, Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From cesare.di.mauro at gmail.com Sun May 15 09:51:58 2016 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Sun, 15 May 2016 15:51:58 +0200 Subject: [Python-ideas] Bytecode for calling function with keyword arguments In-Reply-To: References: Message-ID: With WPython I've solved this scenario with only a new opcode, LOAD_CONSTS, which basically unpacks the (constant) tuple into the stack. So, you don't need new opcodes for new function calls, and LOAD_CONSTS is only used when needed. Regards, Cesare 2016-04-21 17:52 GMT+02:00 Serhiy Storchaka : > Currently the call of function with keyword arguments (say `f(a, b, x=c, > y=d)`) is compiled to following bytecode: > > 0 LOAD_NAME 0 (f) > 3 LOAD_NAME 1 (a) > 6 LOAD_NAME 2 (b) > 9 LOAD_CONST 0 ('x') > 12 LOAD_NAME 3 (c) > 15 LOAD_CONST 1 ('y') > 18 LOAD_NAME 4 (d) > 21 CALL_FUNCTION 514 (2 positional, 2 keyword pair) > > For every positional argument its value is pushed on the stack, and for > every keyword argument its name and its value are pushed on the stack. But > keyword arguments are always constant strings! We can reorder the stack, > and push keyword argument values and names separately. And we can save all > keyword argument names for this call in a constant tuple and load it by one > bytecode command. > > 0 LOAD_NAME 0 (f) > 3 LOAD_NAME 1 (a) > 6 LOAD_NAME 2 (b) > 9 LOAD_NAME 3 (c) > 12 LOAD_NAME 4 (d) > 15 LOAD_CONST 0 (('x', 'y')) > 18 CALL_FUNCTION2 2 > > Benefits: > > 1. We save one command for every keyword parameter after the first. This > saves bytecode size and execution time. > > 2. Since the number of keyword arguments is obtained from tuple's size, > new CALL_FUNCTION opcode needs only the number of positional arguments. > It's argument is simpler and needs less bits (important for wordcode). > > Disadvantages: > > 1. Increases the number of constants. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sun May 15 13:20:08 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 16 May 2016 02:20:08 +0900 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: <20160515024658.GJ12028@ando.pearwood.info> References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <20160515024658.GJ12028@ando.pearwood.info> Message-ID: <22328.45000.977198.667396@turnbull.sk.tsukuba.ac.jp> Steven D'Aprano writes: > Are we supposed to know what category theory's sum functor is? No. There are Pythonistas who do know, that was for their benefit. But to clarify what I mean, I'll quote Guido's example (aside to Guido: yes, that helped!): > PEP 484 has type variables with constraints, and the typing module > predefines AnyStr = TypeVar('AnyStr', str, bytes). This is then > used to define various polymorphic functions, e.g. in os.path: > def abspath(path: AnyStr) -> AnyStr: ... > This means that the type checker will know that the type of abspath(b'.') > is bytes, the type of abspath('.') is str. My bad here. This has the effect of what I called a Sum, I just didn't recognize it as that when I last read PEP 484. > I also don't understand what you mean by '"suspicious" unions' > (quotes in original) or what makes them suspicious, or "suspicious" > as the case may be. What's suspicious about Union[bytes, str]? Union types doesn't allow you to trace the source of an unexpected type backward (or forward), thus, all instances of a Union type must be considered potential sources of the "wrong" type, even if you can infer some of the types of source variables. Compare Number, another union type: consider trying to figure out where an "unexpected" float came from in function performing a long sequence of arithmetic operations including many divisions. You can't do it knowing only the types of the arguments to the function, you need to know the values as well. But I shouldn't have taken the OP's Union[bytes,str] seriously. In fact the os.path functions and other such functions are defined with AnyStr, which is a *variable* type, not a Union type. In typeshed, we write os.path.basename(path: AnyStr) -> AnyStr: ... Ie, AnyStr can be either str or bytes, but it must be the same in all places it appears in the annotated signature. This allows us to reason forward from an appearance of bytes or str to determine what the value of any composition of os or os.path AnyStr->AnyStr functions would be. Eg, in def foo(path: str) -> str: return os.path.basename(os.path.dirname(os.realpath(path))) the composition is str->AnyStr->AnyStr->AnyStr, which resolves to str->str->str->str, and so foo() typechecks sucessfully. Back to typing and the __fspath__ PEP. The remaining question dealing with typing of __fspath__ as currently specified is determining the subtype of DirEntry you're looking at when its __fspath__ gets invoked. (A similar problem remains for the name attribute of file objects, but they aren't scheduled to get a __fspath__.) But AFAICT there is no way to do that yet (I can't find DirEntry in the typing module). Maybe I can contribute to resolving these issues. > Are you proposing a Sum contructor for the typing module, as an > alternative to Union? What will it do? Not any more. It would have done basically what constrained TypeVars like AnyStr do, but more obscurely, and it would have required much more effort and notation to work with functions with multiple arguments of different types and the like. > If not, why are you talking about Sum[bytes, str]? Because I didn't know better then. :-) From guido at python.org Sun May 15 13:21:29 2016 From: guido at python.org (Guido van Rossum) Date: Sun, 15 May 2016 10:21:29 -0700 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: I didn't have time to read the thread, but I read the PEP and thought about this a little bit. One key thing is that we can write the code CPython sees at runtime one way, and write the stubs that type checkers (like mypy) see a different way. The stubs go in the typeshed repo (https://github.com/python/typeshed) and I would add something like the following to the os module there (stdlib/3/os/__init__.pyi in the repo). First we need to add scandir() and DirEntry (this is not entirely unrelated -- DirEntry is an example of something that is PathLike). Disregarding the PathLike protocol for the moment, I think they can be defined like this: if sys.version_info >= (3, 5): class DirEntry(Generic[AnyStr]): name = ... # type: AnyStr path = ... # type: AnyStr def inode(self) -> int: ... def is_dir(self, *, follow_symlinks: bool = ...) -> bool: ... def is_file(self, *, follow_symlinks: bool = ...) -> bool: ... def is_symlink(self) -> bool: ... def stat(self, *, follow_symlinks: bool = ...) -> stat_result: ... @overload def scandir(path: str = ...) -> DirEntry[str]: ... @overload def scandir(path: bytes) -> DirEntry[bytes]: ... Note that the docs claim there's a type os.DirEntry, even though it doesn't currently exist -- I think we should fix that in 3.6 even if it may not make sense to instantiate it. Also note that a slightly different overload is also possible -- I think these are for all practical purposes the same: @overload def scandir() -> DirEntry[str]: ... @overload def scandir(path: AnyStr) -> DirEntry[AnyStr]: ... The reason we need the overload in all cases is that os.scandir() without arguments returns a str. Finally, a reminder that this is all stub code -- it's only ever seen by typecheckers. What we put in the actual os.py file in the stdlib can be completely different, and it doesn't need type annotations (type checkers always prefer the stubs over the real code). Now let's add PathLike. This first attempt doesn't address DirEntry yet: if sys.version_info >= (3, 6): from abc import abstractmethod class PathLike(Generic[AnyStr]): @abstractmethod def __fspath__(self) -> AnyStr: ... @overload def fspath(path: PathLike[AnyStr]) -> AnyStr: ... @overload def fspath(path: AnyStr) -> AnyStr: ... This tells a type checker enough so that it will know that e.g. os.fspath(b'.') returns a bytes object. Also, if we have a class C that derives from PathLike we can make it non-generic, e.g. the stubs for pathlib.Path would start with something like class Path(os.PathLike[str]): ... and now the type checker will know that in the following code `c` is always a str: a = ... # type: Any b = pathlib.Path(a) c = os.fspath(b) Finally let's redefine scandir(). We'll have to redefind DirEntry to inherit from PathLike, and it will remain generic: class DirEntry(PathLike[AnyStr], Generic[AnyStr]): # Everything else unchanged! Now the type checker should understand the following: for a in os.scandir('.'): b = os.fspath(a) ... Here it will know that `a` is a DirEntry[str] (because the argument given to os.scandir() is a str) and hence it will also know that b is a str. Now if then pass b to pathlib it will understand this cannot be a type error, and if you pass b to some os.path.* function (e.g. os.path.basename()) it will understand the return value is a str. If you pass some variable to os.scandir() then if the type checker can deduce that that variable is a str (e.g. because you've gotten it from pathlib) it will know that the results are DirEntry[str] instances. If you pass something to os.scandir() that's a bytes object it will know that the results are DirEntry[bytes] objects, and it knows that calling os.fspath() on those will return bytes. (And it will know that you can't pass those to pathlib, but you *can* pass them to most os and os.path functions.) Next, if the variable passed to os.scandir() has the declared or inferred type AnyStr then mypy will know that it can be either str or bytes and the types of results will also use AnyStr. I think in that case you'll get an error if you pass it to pathlib. Note that this can only happen inside a generic class or a generic function that has AnyStr as one of its parameters. (AnyStr is itself a type variable.) The story ought to be similar if the variable has the type Union[str, bytes], except that this can occur in non-generic code and the resulting types are similarly fuzzy. (I think there's a bug in mypy around this though, follow https://github.com/python/mypy/issues/1533 if you're interested how that turns out.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sun May 15 13:22:53 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 16 May 2016 02:22:53 +0900 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> Message-ID: <22328.45165.679858.812100@turnbull.sk.tsukuba.ac.jp> Koos Zevenhoven writes: > I just don't want to say I broke the type hinting for os.path.* Blame it on Brett. :-) > But if I'm told Unions are sufficient, Unions aren't. TypeVars are (and they're basically what you proposed as a solution, I just didn't recognize them in the context of "Union"). > There is nothing suspicious about that Union there. "Union[str, bytes] > -> str" is a perfectly good type hint for os.fsdecode. You can't get > any more precise than that. Unions are suspicious as *values*, not as arguments. Fortunately, AnyStr is not a Union. From greg.ewing at canterbury.ac.nz Sun May 15 19:37:46 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 16 May 2016 11:37:46 +1200 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> Message-ID: <5739084A.6040708@canterbury.ac.nz> Koos Zevenhoven wrote: > Maybe you mean to propose that the concept of sum types should be > introduced in typing/mypy. I'm not deeply into category theory, but the proposal seems to be that Sum would be a special kind of type that's assumed to be the same type wherever it appears in the signature of a function. That might work for the special case of a function of one parameter that returns the same type as its argument, but that's about all. As has been pointed out, type parameters allow the same thing to be expressed, and a lot more besides, so a Sum type is not needed. We already have everything we neeed. -- Greg From guido at python.org Sun May 15 21:57:07 2016 From: guido at python.org (Guido van Rossum) Date: Sun, 15 May 2016 18:57:07 -0700 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: <5739084A.6040708@canterbury.ac.nz> References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> Message-ID: There seems to be a movement (inspired by Haskell no doubt) that argues that sum types are the next fashion in language design. E.g. Googling for "sum types python" I found https://chadaustin.me/2015/07/sum-types/ (I should look up the author, he works a floor away from me :-). If you read that carefully, a sum type is mostly syntactic sugar for a tagged union. They're easy enough to add to compiled languages. I find it interesting that Austin mentions several types that the "Maybe" type is equivalent to the convention of returning NULL/null/None, and he uses it as a simple example (the simplest, in fact) of a sum type. So in PEP 484 that's called Optional[X], and indeed Optional[X] is syntactic sugar for Union[X, None], and of course unions in Python are always tagged (since you can always introspect the object type). So Python's equivalent for Sum types seems to be PEP 484's Union types. For matching, a sequence of isinstance() checks doesn't look too shabby: def add1(a: Union[int, str, bytes]) -> Union[int, str, bytes]: if isinstance(a, int): # In this block, the type of a is int return a+1 if isinstance(a, str): # Here a is a str return a + 'x' # In this block, the type of a is bytes (because it can't be anything else!) return a + b'x' Of course in this specific example, using a constrained type variable (similar to AnyStr) would be much better, since it declares that the return type in this case is the same as the argument type. However that's not what Sum types do in Haskell either, so we can't really blame them for this -- in Haskell as in PEP 484, Sum types and generics are pretty much orthogonal concepts. I think Sum/Union shine when they are either used for the argument (so the function must have a list of cases) or for the return value so the caller must have a list of cases), but not for both. That's not to say that it wouldn't be an interesting exercise to see if we can add a nice pattern matching syntax to Python. Austin claims that one of the advantages of Sum types in Haskell is that it supports a recursive matching syntax. It would probably look different than it looks in compiled languages though, because a Python compiler doesn't know much about the types occurring at runtime, so e.g. it couldn't tell you when you're missing a case. However, a type checker could -- this might be the first new Python feature that was in some sense enabled by PEP 484. (And here I thought I was off-topic for the thread, but this thread was actually started specifically to discuss Sum types, so now I feel better about this post.) PS. I recall vividly the classes in category theory I took in college. They were formative in my education -- I knew for sure then that I would never be a mathematician. On Sun, May 15, 2016 at 4:37 PM, Greg Ewing wrote: > Koos Zevenhoven wrote: > > Maybe you mean to propose that the concept of sum types should be >> introduced in typing/mypy. >> > > I'm not deeply into category theory, but the proposal > seems to be that Sum would be a special kind of type > that's assumed to be the same type wherever it appears > in the signature of a function. > > That might work for the special case of a function of > one parameter that returns the same type as its > argument, but that's about all. > > As has been pointed out, type parameters allow the > same thing to be expressed, and a lot more besides, > so a Sum type is not needed. We already have everything > we neeed. > > -- > Greg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Mon May 16 10:45:41 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Mon, 16 May 2016 17:45:41 +0300 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> Message-ID: On Mon, May 16, 2016 at 4:57 AM, Guido van Rossum wrote: [...] > unions in Python are always tagged (since you can always introspect the object type). I suppose that is true at runtime, at least if the pairwise intersections of the "arguments" of the union are empty as they usually are. But for a static type checker, the "tag" for the Union may be missing (which is the thing I was worried about). [...] > in Haskell as in PEP 484, Sum types and generics are pretty much orthogonal > concepts. Although the parametrization provided by generics make TypeVars more powerful for annotating functions that deal with the equivalent of sum types. -- Koos From k7hoven at gmail.com Mon May 16 10:53:25 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Mon, 16 May 2016 17:53:25 +0300 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: <22328.45000.977198.667396@turnbull.sk.tsukuba.ac.jp> References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <20160515024658.GJ12028@ando.pearwood.info> <22328.45000.977198.667396@turnbull.sk.tsukuba.ac.jp> Message-ID: On Sun, May 15, 2016 at 8:20 PM, Stephen J. Turnbull wrote: > > But I shouldn't have taken the OP's Union[bytes,str] seriously. In > fact the os.path functions and other such functions are defined with > AnyStr, which is a *variable* type, not a Union type. In typeshed, we > write If you mean my OP in the other thread, I was indeed just introducing the readers (potentially not familiar with PEP 484, mypy etc.) to the basic concepts such as Unions and TypeVars while going towards the actual proposal at the very end of the post, which was to avoid having to go from TypeVars "back to" Unions in os.fspath and os.path functions. -- Koos From k7hoven at gmail.com Mon May 16 11:05:24 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Mon, 16 May 2016 18:05:24 +0300 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: <5739084A.6040708@canterbury.ac.nz> References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> Message-ID: On Mon, May 16, 2016 at 2:37 AM, Greg Ewing wrote: > > I'm not deeply into category theory, but the proposal > seems to be that Sum would be a special kind of type > that's assumed to be the same type wherever it appears > in the signature of a function. Indeed, that's TypeVar. > That might work for the special case of a function of > one parameter that returns the same type as its > argument, but that's about all. I was actually trying to interpret the (vague) proposal as going even a step further, closer to what may have helped slightly in the fspath type hinting problem. Perhaps something like this: M = TagMatcher(3) # 3 is the number of "summed" types # (and yes, I just made up the term TagMatcher) def onehalf(value: M.sum[int, float, complex]) -> M.sum[float, float, complex]: return 0.5 * value That would then match the "tag" between the argument and return types (within the sum types which are tagged unions): int -> float float -> float complex -> complex As you can see, "onehalf(...)" will turn int, float and complex into float, float and complex, respectively. I suppose one or more "functors" could be seen in there to make this happen in theory XD. This is not solved by a TypeVar. > As has been pointed out, type parameters allow the > same thing to be expressed, and a lot more besides, > so a Sum type is not needed. We already have everything > we neeed. So, despite what I write above, you still seem to agree with me, assuming you did not literally mean *everything* ;-). The reason why we agree is that we can already do: @overload def onehalf(value: int) -> float: ... @overload def onehalf(value: float) -> float: ... @overload def onehalf(value: complex) -> complex: ... So, the sum type, even with my "TagMatcher" described above, does not provide anything new. After all, the whole point of sum types is to give *statically typed* languages something that resembles dynamic typing. -- Koos > > -- > Greg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From guido at python.org Mon May 16 11:19:12 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 16 May 2016 08:19:12 -0700 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> Message-ID: On Mon, May 16, 2016 at 7:45 AM, Koos Zevenhoven wrote: > On Mon, May 16, 2016 at 4:57 AM, Guido van Rossum > wrote: > [...] > > unions in Python are always tagged (since you can always introspect the > object type). > > I suppose that is true at runtime, at least if the pairwise > intersections of the "arguments" of the union are empty as they > usually are. But for a static type checker, the "tag" for the Union > may be missing (which is the thing I was worried about). > Have you thought about how a type checker works? *Of course* the tags are "missing" for it. When it sees that an argument (e.g.) is a union it has to type check the following code with the assumption that it can be any of those types. (However certain flow control using isinstance() can refine this knowledge, like the "if isinstance(...)" example I've given before; also "assert isinstance(...)" has a similar effect. When a type checker like mypy has a variable that it knows to be a str, and this variable is passed to a function whose signature is (Union[str, bytes]) -> Union[str, bytes], then the result has to be assumed to be the given union, because such a signature does not guarantee that the return type matches the input type. (E.g. if I had a function that turned bytes into str and str into bytes, this could be its signature!) For this reason, the type variable AnyStr exists, and the mechanism of constrained TypeVars that makes AnyStr possible -- you can make your own type variables with such behavior too. > [...] > > in Haskell as in PEP 484, Sum types and generics are pretty much > orthogonal > > concepts. > > Although the parametrization provided by generics make TypeVars more > powerful for annotating functions that deal with the equivalent of sum > types. > Yeah, it's not uncommon for two orthogonal features to combine into something even more powerful like that. It's like Hydrogen plus Oxygen... :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon May 16 12:15:50 2016 From: brett at python.org (Brett Cannon) Date: Mon, 16 May 2016 16:15:50 +0000 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: On Sun, 15 May 2016 at 10:21 Guido van Rossum wrote: > I didn't have time to read the thread, but I read the PEP and thought > about this a little bit. > > One key thing is that we can write the code CPython sees at runtime one > way, and write the stubs that type checkers (like mypy) see a different > way. The stubs go in the typeshed repo (https://github.com/python/typeshed) > and I would add something like the following to the os module there > (stdlib/3/os/__init__.pyi in the repo). > > First we need to add scandir() and DirEntry (this is not entirely > unrelated -- DirEntry is an example of something that is PathLike). > Disregarding the PathLike protocol for the moment, I think they can be > defined like this: > > if sys.version_info >= (3, 5): > class DirEntry(Generic[AnyStr]): > name = ... # type: AnyStr > path = ... # type: AnyStr > def inode(self) -> int: ... > def is_dir(self, *, follow_symlinks: bool = ...) -> bool: ... > def is_file(self, *, follow_symlinks: bool = ...) -> bool: ... > def is_symlink(self) -> bool: ... > def stat(self, *, follow_symlinks: bool = ...) -> stat_result: ... > > @overload > def scandir(path: str = ...) -> DirEntry[str]: ... > @overload > def scandir(path: bytes) -> DirEntry[bytes]: ... > > Note that the docs claim there's a type os.DirEntry, even though it > doesn't currently exist -- I think we should fix that in 3.6 even if it may > not make sense to instantiate it. > http://bugs.python.org/issue27038 (and AnyStr isn't documented, so http://bugs.python.org/issue26141). > > Also note that a slightly different overload is also possible -- I think > these are for all practical purposes the same: > > @overload > def scandir() -> DirEntry[str]: ... > @overload > def scandir(path: AnyStr) -> DirEntry[AnyStr]: ... > > The reason we need the overload in all cases is that os.scandir() without > arguments returns a str. > > Finally, a reminder that this is all stub code -- it's only ever seen by > typecheckers. What we put in the actual os.py file in the stdlib can be > completely different, and it doesn't need type annotations (type checkers > always prefer the stubs over the real code). > > Now let's add PathLike. This first attempt doesn't address DirEntry yet: > > if sys.version_info >= (3, 6): > from abc import abstractmethod > class PathLike(Generic[AnyStr]): > @abstractmethod > def __fspath__(self) -> AnyStr: ... > > @overload > def fspath(path: PathLike[AnyStr]) -> AnyStr: ... > @overload > def fspath(path: AnyStr) -> AnyStr: ... > > This tells a type checker enough so that it will know that e.g. > os.fspath(b'.') returns a bytes object. Also, if we have a class C that > derives from PathLike we can make it non-generic, e.g. the stubs for > pathlib.Path would start with something like > > class Path(os.PathLike[str]): > ... > > and now the type checker will know that in the following code `c` is > always a str: > > a = ... # type: Any > b = pathlib.Path(a) > c = os.fspath(b) > > Finally let's redefine scandir(). We'll have to redefind DirEntry to > inherit from PathLike, and it will remain generic: > > class DirEntry(PathLike[AnyStr], Generic[AnyStr]): > # Everything else unchanged! > > Now the type checker should understand the following: > > for a in os.scandir('.'): > b = os.fspath(a) > ... > > Here it will know that `a` is a DirEntry[str] (because the argument given > to os.scandir() is a str) > Which works because AnyStr is a TypeVar (if anyone else was wondering like I was why that worked since AnyStr isn't documented yet). > and hence it will also know that b is a str. Now if then pass b to pathlib > it will understand this cannot be a type error, and if you pass b to some > os.path.* function (e.g. os.path.basename()) it will understand the return > value is a str. > > If you pass some variable to os.scandir() then if the type checker can > deduce that that variable is a str (e.g. because you've gotten it from > pathlib) it will know that the results are DirEntry[str] instances. If you > pass something to os.scandir() that's a bytes object it will know that the > results are DirEntry[bytes] objects, and it knows that calling os.fspath() > on those will return bytes. (And it will know that you can't pass those to > pathlib, but you *can* pass them to most os and os.path functions.) > > Next, if the variable passed to os.scandir() has the declared or inferred > type AnyStr then mypy will know that it can be either str or bytes and the > types of results will also use AnyStr. I think in that case you'll get an > error if you pass it to pathlib. Note that this can only happen inside a > generic class or a generic function that has AnyStr as one of its > parameters. (AnyStr is itself a type variable.) > > The story ought to be similar if the variable has the type Union[str, > bytes], except that this can occur in non-generic code and the resulting > types are similarly fuzzy. (I think there's a bug in mypy around this > though, follow https://github.com/python/mypy/issues/1533 if you're > interested how that turns out.) > This might make a nice example in the docs and/or blog post since this is hitting the intermediate/advanced space for typing that almost none of us have hit. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon May 16 14:38:05 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 16 May 2016 11:38:05 -0700 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: On Mon, May 16, 2016 at 9:15 AM, Brett Cannon wrote: > This might make a nice example in the docs and/or blog post since this is > hitting the intermediate/advanced space for typing that almost none of us > have hit. > I'll see if I can follow up on this idea. But if anyone else feels like blogging about this, please go ahead! If you post a link here I'll retweet. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon May 16 16:48:31 2016 From: brett at python.org (Brett Cannon) Date: Mon, 16 May 2016 20:48:31 +0000 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: On Mon, 16 May 2016 at 11:38 Guido van Rossum wrote: > On Mon, May 16, 2016 at 9:15 AM, Brett Cannon wrote: > >> This might make a nice example in the docs and/or blog post since this is >> hitting the intermediate/advanced space for typing that almost none of us >> have hit. >> > > I'll see if I can follow up on this idea. But if anyone else feels like > blogging about this, please go ahead! If you post a link here I'll retweet. > If you don't get to it then maybe I will in a blog post on PEP 519, but I don't think I will be much faster than you for the foreseeable future. :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Mon May 16 16:19:30 2016 From: dinov at microsoft.com (Dino Viehland) Date: Mon, 16 May 2016 20:19:30 +0000 Subject: [Python-ideas] Adding a frame evaluation API to CPython Message-ID: Adding a frame evaluation API to CPython Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon , Dino Viehland https://github.com/Microsoft/Pyjion/blob/master/pep.rst Abstract This PEP proposes to expand CPython's C API https://github.com/Microsoft/Pyjion/blob/master/pep.rst#c-api to allow for the specification of a per-interpreter function pointer to handle the evaluation of frames https://github.com/Microsoft/Pyjion/blob/master/pep.rst#pyeval-evalframeex. This proposal also suggests adding a new field to code objects https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id21 to store arbitrary data for use by the frame evaluation function. Rationale One place where flexibility has been lacking in Python is in the direct execution of Python code. While CPython's C API https://github.com/Microsoft/Pyjion/blob/master/pep.rst#c-api allows for constructing the data going into a frame object and then evaluating it via PyEval_EvalFrameEx() https://github.com/Microsoft/Pyjion/blob/master/pep.rst#pyeval-evalframeex, control over the execution of Python code comes down to individual objects instead of a hollistic control of execution at the frame level. While wanting to have influence over frame evaluation may seem a bit too low-level, it does open the possibility for things such as a JIT to be introduced into CPython without CPython itself having to provide one. By allowing external C code to control frame evaluation, a JIT can participate in the execution of Python code at the key point where evaluation occurs. This then allows for a JIT to conditionally recompile Python bytecode to machine code as desired while still allowing for executing regular CPython bytecode when running the JIT is not desired. This can be accomplished by allowing interpreters to specify what function to call to evaluate a frame. And by placing the API at the frame evaluation level it allows for a complete view of the execution environment of the code for the JIT. This ability to specify a frame evaluation function also allows for other use-cases beyond just opening CPython up to a JIT. For instance, it would not be difficult to implement a tracing or profiling function at the call level with this API. While CPython does provide the ability to set a tracing or profiling function at the Python level, this would be able to match the data collection of the profiler and quite possibly be faster for tracing by simply skipping per-line tracing support. It also opens up the possibility of debugging where the frame evaluation function only performs special debugging work when it detects it is about to execute a specific code object. In that instance the bytecode could be theoretically rewritten in-place to inject a breakpoint function call at the proper point for help in debugging while not having to do a heavy-handed approach as required by sys.settrace(). To help facilitate these use-cases, we are also proposing the adding of a "scratch space" on code objects via a new field. This will allow per-code object data to be stored with the code object itself for easy retrieval by the frame evaluation function as necessary. The field itself will simply be a PyObject * type so that any data stored in the field will participate in normal object memory management. Proposal All proposed C API changes below will not be part of the stable ABI. Expanding PyCodeObject One field is to be added to the PyCodeObject struct https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id23: typedef struct { ... PyObject *co_extra; /* "Scratch space" for the code object. */ } PyCodeObject; The co_extra will be NULL by default and will not be used by CPython itself. Third-party code is free to use the field as desired. The field will be freed like all other fields on PyCodeObject during deallocation using Py_XDECREF(). It is not recommended that multiple users attempt to use the co_extra simultaneously. While a dictionary could theoretically be set to the field and various users could use a key specific to the project, there is still the issue of key collisions as well as performance degradation from using a dictionary lookup on every frame evaluation. Users are expected to do a type check to make sure that the field has not been previously set by someone else. Expanding PyInterpreterState The entrypoint for the frame evalution function is per-interpreter: // Same type signature as PyEval_EvalFrameEx(). typedef PyObject* (__stdcall *PyFrameEvalFunction)(PyFrameObject*, int); typedef struct { ... PyFrameEvalFunction eval_frame; } PyInterpreterState; By default, the eval_frame field will be initialized to a function pointer that represents what PyEval_EvalFrameEx() currently is (called PyEval_EvalFrameDefault(), discussed later in this PEP). Third-party code may then set their own frame evaluation function instead to control the execution of Python code. A pointer comparison can be used to detect if the field is set to PyEval_EvalFrameDefault() and thus has not been mutated yet. Changes to Python/ceval.c PyEval_EvalFrameEx() https://github.com/Microsoft/Pyjion/blob/master/pep.rst#pyeval-evalframeex as it currently stands will be renamed to PyEval_EvalFrameDefault(). The new PyEval_EvalFrameEx() will then become: PyObject * PyEval_EvalFrameEx(PyFrameObject *frame, int throwflag) { PyThreadState *tstate = PyThreadState_GET(); return tstate->interp->eval_frame(frame, throwflag); } This allows third-party code to place themselves directly in the path of Python code execution while being backwards-compatible with code already using the pre-existing C API. Performance impact As this PEP is proposing an API to add pluggability, performance impact is considered only in the case where no third-party code has made any changes. Several runs of pybench https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id25 consistently showed no performance cost from the API change alone. A run of the Python benchmark suite https://github.com/Microsoft/Pyjion/blob/master/pep.rst#py-benchmarks showed no measurable cost in performance. In terms of memory impact, since there are typically not many CPython interpreters executing in a single process that means the impact of co_extra being added to PyCodeObject is the only worry. According to https://github.com/Microsoft/Pyjion/blob/master/pep.rst#code-object-count, a run of the Python test suite results in about 72,395 code objects being created. On a 64-bit CPU that would result in 4,633,280 bytes of extra memory being used if all code objects were alive at once and had nothing set in their co_extra fields. Example Usage A JIT for CPython Pyjion The Pyjion project https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id18 has used this proposed API to implement a JIT for CPython using the CoreCLR's JIT https://github.com/Microsoft/Pyjion/blob/master/pep.rst#coreclr. Each code object has its co_extra field set to a PyjionJittedCode object which stores four pieces of information: 1. Execution count 2. A boolean representing whether a previous attempt to JIT failed 3. A function pointer to a trampoline (which can be type tracing or not) 4. A void pointer to any JIT-compiled machine code The frame evaluation function has (roughly) the following algorithm: def eval_frame(frame, throw_flag): pyjion_code = frame.code.co_extra if not pyjion_code: frame.code.co_extra = PyjionJittedCode() elif not pyjion_code.jit_failed: if not pyjion_code.jit_code: return pyjion_code.eval(pyjion_code.jit_code, frame) elif pyjion_code.exec_count > 20_000: if jit_compile(frame): return pyjion_code.eval(pyjion_code.jit_code, frame) else: pyjion_code.jit_failed = True pyjion_code.exec_count += 1 return PyEval_EvalFrameDefault(frame, throw_flag) The key point, though, is that all of this work and logic is separate from CPython and yet with the proposed API changes it is able to provide a JIT that is compliant with Python semantics (as of this writing, performance is almost equivalent to CPython without the new API). This means there's nothing technically preventing others from implementing their own JITs for CPython by utilizing the proposed API. Other JITs It should be mentioned that the Pyston team was consulted on an earlier version of this PEP that was more JIT-specific and they were not interested in utilizing the changes proposed because they want control over memory layout they had no interest in directly supporting CPython itself. An informal discusion with a developer on the PyPy team led to a similar comment. Numba https://github.com/Microsoft/Pyjion/blob/master/pep.rst#numba, on the other hand, suggested that they would be interested in the proposed change in a post-1.0 future for themselves https://github.com/Microsoft/Pyjion/blob/master/pep.rst#numba-interest. Debugging In conversations with the Python Tools for Visual Studio team (PTVS) https://github.com/Microsoft/Pyjion/blob/master/pep.rst#ptvs, they thought they would find these API changes useful for implementing more performant debugging. As mentioned in the https://github.com/Microsoft/Pyjion/blob/master/pep.rst#rationale section, this API would allow for switching on debugging functionality only in frames where it is needed. This could allow for either skipping information that sys.settrace() normally provides and even go as far as to dynamically rewrite bytecode prior to execution to inject e.g. breakpoints in the bytecode. Implementation A set of patches implementing the proposed API is available through the Pyjion project https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id18. In its current form it has more changes to CPython than just this proposed API, but that is for ease of development instead of strict requirements to accomplish its goals. Open Issues Allow eval_frame to be NULL Currently the frame evaluation function is expected to always be set. It could very easily simply default to NULL instead which would signal to use PyEval_EvalFrameDefault(). The current proposal of not special-casing the field seemed the most straight-forward, but it does require that the field not accidentally be cleared, else a crash may occur. Rejected Ideas A JIT-specific C API Originally this PEP was going to propose a much larger API change which was more JIT-specific. After soliciting feedback from the Numba team https://github.com/Microsoft/Pyjion/blob/master/pep.rst#numba, though, it became clear that the API was unnecessarily large. The realization was made that all that was truly needed was the opportunity to provide a trampoline function to handle execution of Python code that had been JIT-compiled and a way to attach that compiled machine code along with other critical data to the corresponding Python code object. Once it was shown that there was no loss in functionality or in performance while minimizing the API changes required, the proposal was changed to its current form. References [1] (https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id11, https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id16) Pyjion project (https://github.com/microsoft/pyjion) [2] (https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id1, https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id4) CPython's C API (https://docs.python.org/3/c-api/index.html) [3] PyCodeObject (https://docs.python.org/3/c-api/code.html#c.PyCodeObject) https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id12 .NET Core Runtime (CoreCLR) (https://github.com/dotnet/coreclr) [5] (https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id2, https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id5, https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id7) PyEval_EvalFrameEx() (https://docs.python.org/3/c-api/veryhigh.html?highlight=pyframeobject#c.PyEval_EvalFrameEx) [6] PyCodeObject (https://docs.python.org/3/c-api/code.html#c.PyCodeObject) [7] (https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id13, https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id17) Numba (http://numba.pydata.org/) https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id14 numba-users mailing list: "Would the C API for a JIT entrypoint being proposed by Pyjion help out Numba?" (https://groups.google.com/a/continuum.io/forum/#!topic/numba-users/yRl_0t8-m1g) https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id10 [Python-Dev] Opcode cache in ceval loop (https://mail.python.org/pipermail/python-dev/2016-February/143025.html) https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id9 Python benchmark suite (https://hg.python.org/benchmarks) [11] Pyston (http://pyston.org/) [12] PyPy (http://pypy.org/) https://github.com/Microsoft/Pyjion/blob/master/pep.rst#id15 Python Tools for Visual Studio (http://microsoft.github.io/PTVS/) Copyright This document has been placed in the public domain. From boekewurm at gmail.com Mon May 16 19:12:56 2016 From: boekewurm at gmail.com (Matthias welp) Date: Tue, 17 May 2016 01:12:56 +0200 Subject: [Python-ideas] Adding a frame evaluation API to CPython In-Reply-To: References: Message-ID: I won't comment on the content of this proposal, as I'm don't really see the implications, but I would like to correct one small mistake: Under 'performance impact' you say this: > In terms of memory impact, since there are typically not many CPython interpreters executing in a single process that means the impact of co_extra being added to PyCodeObject is the only worry. According to https://github.com/Microsoft/Pyjion/blob/master/pep.rst#code-object-count, a run of the Python test suite results in about 72,395 code objects being created. On a 64-bit CPU that would result in 4,633,280 bytes of extra memory being used if all code objects were alive at once and had nothing set in their co_extra fields. The extra RAM used is a bit off. One 64-bit pointer extra takes only 8 bytes, which translates to about 579160 bytes extra used in in the test suite. I did not consider padding for accessibility, but that was also not considered in the draft. -Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon May 16 19:34:38 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 17 May 2016 11:34:38 +1200 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> Message-ID: <573A590E.30708@canterbury.ac.nz> Guido van Rossum wrote: > On Mon, May 16, 2016 at 7:45 AM, Koos Zevenhoven > wrote: > > On Mon, May 16, 2016 at 4:57 AM, Guido van Rossum > wrote: > [...] >> unions in Python are always tagged (since you can always > introspect the object type). The "Sum" types talked about in the referenced article are what Haskell calls "algebraic types". They're not really the same as the Union[X,Y] types we're talking about here, because a Union type simply tells the type checker that one of a number of different types could be present at run time. The code might introspect on the type, but it doesn't have to do anything special to access one of the branches of the union -- it just goes ahead and uses the value. An algebraic type, on the other hand, is a new type of run-time object that has to be explicitly unpacked to access its contents. It's more like a class in that respect. -- Greg From jeanpierreda at gmail.com Mon May 16 20:00:05 2016 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 16 May 2016 17:00:05 -0700 Subject: [Python-ideas] Adding a frame evaluation API to CPython In-Reply-To: References: Message-ID: On Mon, May 16, 2016 at 1:19 PM, Dino Viehland wrote: > It also opens up the possibility of debugging where the frame evaluation function only performs special debugging work when it detects it is about to execute a specific code object. In that instance the bytecode could be theoretically rewritten in-place to inject a breakpoint function call at the proper point for help in debugging while not having to do a heavy-handed approach as required by sys.settrace(). To add to the debugging story: there are many debugging/profiling tools that look at C symbol names. When you have a mixed C/Python codebase, too often a problem straddles both sides -- the Python tools don't tell you anything about the C end, and the C tools just tell you that everything happened in PyEval_EvalFrameEx. Instead of plugging in a JIT-compiled version of a code object, one can plug in a shim, so that each code object has a function with a unique symbol name that only calls PyEval_EvalFrameDefault. This way, you get symbol names for all Python functions. So while it still says that 90% CPU is used by PyEval_EvalFrameEx, it also then that breaks down into 12% being used by PY_SYMBOL__json?load or similar, which is a symbol corresponding to the code object for the Python function json.load. I'm +1 for something that makes this possible, although for the specific API proposal here, no real opinions. -- Devin From rdmurray at bitdance.com Mon May 16 20:35:09 2016 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 16 May 2016 20:35:09 -0400 Subject: [Python-ideas] Application awareness of memory storage classes Message-ID: <20160517003511.AD7C9B14094@webabinitio.net> I'm currently working on a project for Intel involving Python and directly addressable non-volatile memory. See https://nvdimm.wiki.kernel.org/ for links to the Linux features underlying this work, and http://pmem.io/ for the Intel initiated open source project I'm working with whose intent is to bring support for DAX NVRAM to the application programming layer (nvml currently covers C, with C++ support in process, and there are python bindings for the lower level (non-libpmemobj) libraries). tldr: In the future (and the future is now) application programs will want to be aware of what class of memory they are allocating: normal DRAM, persistent RAM, fast DRAM, (etc?). High level languages like Python that manage memory for the application programmer will need to expose a way for programmers to declare which kind of memory backs an object, and define rules for interactions between objects that reside in different memory types. I'm interested in starting a discussion (not necessarily here, though I think that probably makes sense) about the challenges this presents and how Python-the-language and CPython the implementation might solve them. Feel free to tell me this is the wrong forum, and if so we can a new forum to continue the conversation if enough people are interested. Long version: While I mentioned fast DRAM as well as Persistent RAM above, really it is Persistent RAM that provides the most interesting challenges. This is because you actually have to program differently when your objects are backed by persistent memory. Consider the python script ('pop' is short for Persistent Object Pool): source_account = argv[1].lower() target_account = argv[2].lower() delta = float(argv[3]) pop = pypmemobj('/path/to/pool') if not hasattr(pop.ns, 'accounts'): pop.ns.accounts = dict() for acct in (source_account, target_account): if acct not in pop.ns.accounts: pop.ns.accounts[acct] = 0 pop.ns.accounts[source_account] -= delta pop.ns.accounts[target_account] += delta print("{:-10s} {:10.2f} {:-10s}".format( source_account, pop.ns.accounts[source_account], target_account, pop.ns.accounts[target_account] )) (I haven't bothered to test this code, forgive stupid errors.) This is a simple CLI ap that lets you manage a set of bank accounts: > acct deposits checking 10.00 deposits -10.00 savings 10.00 > acct checking savings 5.00 savings 5.00 checking 5.00 Obviously you'd have a lot more code in a real ap. The point here is that we've got *persistent* account objects, with values that are preserved between runs of the program. There's a big problem with this code. What happens if the program crashes or the machine crashes? Specifically, what happens if the machine crashes between the -= and the += lines? You could end up with a debit from one account with no corresponding credit to the other, leaving your accounts in an inconsistent state with no way to recover. The authors of the nvml library documented at the pmem.io site have discovered via theory and practice that what you need for reliable programming with persistent memory is almost exactly same kind of atomicity that is required when doing multi-threaded programming. (I can sense the asyncio programmers groaning...didn't we just solve that problem and now you are bringing it back? As you'll see below if you combine asyncio and persistent memory, providing the atomicity isn't nearly as bad as managing threading locks; although it does take more thought than writing DRAM-backed procedural code, it is relatively straightforward to reason about.) What we need for the above program fragment to work reliably in the face of crashes is for all of the operations that Python programmers think of as atomic (assignment, dictionary update, etc) to be atomic at the persistent memory level, and to surround any code that does multiple-object updates that are interdependent with a 'transaction' guard. Thus we'd rewrite the end of the above program like this: with pop: pop.ns.accounts[source_account] -= delta pop.ns.accounts[target_account] += delta print("{:-10s} {:10.2f} {:-10s}".format( source_account, pop.ns.accounts[source_account], target_account, pop.ns.accounts[target_account] )) If this were in fact a threaded program, we'd be holding a write lock on pop.ns.accounts in the same scope; but in that case (a) we'd also want to include the print statement and (b) we'd need to pay attention to what lock we were holding. The pmem transaction doesn't have to worry about locks, it is guarding *whatever* memory changes are made during the transaction, no mater which object they are made to. This is what I mean by asyncio + pmem transactions being simpler than threading locks. The transaction that is implied by 'with pop' is the heart of what the nvml libpmemobj does: it provides a transaction scope where any registered changes to the persistent memory will be rolled back on abort, or rolled back when the object pool is next opened if the transaction did not complete before the crash. Of course, in Python as it stands today, the above program would not do anything practical, since when the dict that becomes accounts is allocated, it gets allocated in DRAM, not in persistent memory. To use persistent memory, I'm about to start working on an extension module that basically re-implements most of the Python object classes so that they can be stored in persistent memory instead of RAM. Using my proposed API, the above program would actually look like this: source_account = argv[1].lower() target_account = argv[2].lower() delta = float(argv[3]) pop = pypmemobj('/path/to/pool') if not hasattr(pop.ns, 'accounts'): pop.ns.accounts = PersistentDict(pop) for acct in (source_account, target_account): if acct not in pop.ns.accounts: pop.ns.accounts[acct] = 0.0 with pop: pop.ns.accounts[source_account] -= delta pop.ns.accounts[target_account] += delta print("{:-10s} {:10.2f} {:-10s}".format( source_account, pop.ns.accounts[source_account], target_account, pop.ns.accounts[target_account] )) The difference here is creating a PersistentDict object, and telling it what persistent memory to allocate itself in. But what happens when we do: pop.ns.accounts[acct] = 0.0 We don't want to have to write pop.ns.accounts[acct] = PersistentFloat(pop, 0.0) but if we don't, we'd be trying to store a pointer to an float object that lives in normal RAM into our persistent list, and that pointer would be invalid on the next program run. Instead, for immutable objects we can make a copy in persistent ram when the assignment happens. But we have to reject any attempt to assign a mutable object in to a persistent object unless the mutable is itself persistent...we can't simply make a copy of a mutable object and make a persistent version, because of the following Python idiom: pop.ns.accounts = accounts = dict() If we copied the dict into a PersistentDict automatically, pop.ns.accounts and accounts would point to different objects, and the Python programmer's expectations about his program would be violated. So that's a brief description of what I'm planning to try to implement as an extension module (thoughts and feedback welcome). It's pretty clear that having to re-implement every mutable Python C type, as well as immutable collection types (so that we can handle pointers to other persistent objects correctly) will be a pain. What would it take for Python itself to support the notion of objects being backed by different kinds of memory? It seems to me that it would be possible, but that in CPython at least the cost would be a performance penalty for the all-DRAM case. I'm guessing this is not acceptable, but I'd like to present the notion anyway, in the hopes that someone cleverer than me can see a better way :) At the language level, support would mean two things, I think: there would need to be a way to declare which backing store an object belongs to, and there would need to be a notion of a memory hierarchy where, at least in the case of persistent vs non-persistent RAM, an object higher in the hierarchy could not point to an object lower in the hierarchy that was mutable, and that immutables would be copied from lower to higher. (Here I'm notionally defining "lower" as "closer to the CPU", since RAM is 'just there' whereas persistent memory goes through a driver layer before direct access can get exposed, and fast DRAM would be closer yet to the CPU :). In CPython I envision this being implemented by having every object be associated with a 'storage manager', with the DRAM storage manager obviously being the default. I think that which storage manager an object belongs to can be deduced from its address at runtime, although whether that would be better than storing a pointer is an open question. Object method implementations would then need to be expanded as follows: (1) wrapping any operations that comprise an 'atomic' operation with a 'start transaction' and 'end transaction' call on the storage manager. (2) any memory management function (malloc, etc) would be called indirectly through the storage manager, (3) any object pointer to be stored or retrieved would be passed through an appropriate call on the storage manager to give it an opportunity to block it or transform it, and (4) any memory range to be modified would be reported to the memory manager so that it can be registered with the transaction. I *think* that's the minimum set of changes. Clearly, however, making such calls is going to be less efficient in the DRAM case than the current code, even though the RAM implementation would be a noop. Summary: Providing Python language level support for directly addressable persistent RAM requires addressing the issues of how the application indicates when an object is persistent, and managing the interactions between objects that are persistent and those that are not. In addition, operations that a Python programmer expects to be atomic need to be made atomic from the viewpoint of persistent memory by bracketing them with implicit transactions, and a way to declare an explicit transaction needs to be exposed to the application programmer. These are all interesting design challenges, and I may not have managed to identify all the issues involved. Directly addressable persistent memory is going to become much more common, and probably relatively soon. An extension module such as I plan to work on can provide a way for Python to be used in this space, but are there things that we are able and willing to do to support it more directly in the language and, by implication, in the various Python implementations? Are there design considerations for this extension module that would make it easier or harder for more integrated language support to be added later? I realize this could be viewed as early days to be bringing up this subject, since I haven't even started the extension module yet. On the other hand, it seems to me that the community may have architectural insights that could inform the development, and that starting the conversation now without trying to nail anything down could be very helpful for facilitating the long term support of variant memory types in Python. From guido at python.org Mon May 16 20:44:05 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 16 May 2016 17:44:05 -0700 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: <573A590E.30708@canterbury.ac.nz> References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> <573A590E.30708@canterbury.ac.nz> Message-ID: > Guido van Rossum wrote: >> ... https://chadaustin.me/2015/07/sum-types/ ... >> unions in Python are always tagged (since you can alway introspect the object type). Greg Ewing replied: > The "Sum" types talked about in the referenced article are > what Haskell calls "algebraic types". They're not really the > same as the Union[X,Y] types we're talking about here, > because a Union type simply tells the type checker that > one of a number of different types could be present at > run time. The code might introspect on the type, but it > doesn't have to do anything special to access one of the > branches of the union -- it just goes ahead and uses the > value. > > An algebraic type, on the other hand, is a new type of > run-time object that has to be explicitly unpacked to > access its contents. It's more like a class in that > respect. I know. But this could be considered syntactic sugar. And the article insists that Sum types are not really the same as classes either (they're also a form of syntactic sugar). Anyway, I did set up a chat with the Chad (that article's author) and maybe we'll all get some more clarity. -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Mon May 16 23:49:17 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 17 May 2016 13:49:17 +1000 Subject: [Python-ideas] Adding a frame evaluation API to CPython In-Reply-To: References: Message-ID: This idea generally sounds reasonable to me, I just have some suggestions for other folks to approach specifically for feedback. On 17 May 2016 at 06:19, Dino Viehland wrote: > Other JITs > It should be mentioned that the Pyston team was consulted on an earlier version of this PEP that was more JIT-specific and they were not interested in utilizing the changes proposed because they want control over memory layout they had no interest in directly supporting CPython itself. An informal discusion with a developer on the PyPy team led to a similar comment. > Numba https://github.com/Microsoft/Pyjion/blob/master/pep.rst#numba, on the other hand, suggested that they would be interested in the proposed change in a post-1.0 future for themselves https://github.com/Microsoft/Pyjion/blob/master/pep.rst#numba-interest. Hopefully Victor will chime in anyway, but if he doesn't, I'd suggest asking him directly what impact this proposal might have on the bytecode transformation aspects of PEP 511. There also seems to be a potential overlap with the function specialisation proposal in PEP 510, so it would be good to see this PEP discussing its relationship with that one (the explanation may be "they're orthogonal proposals addressing different concerns", but it isn't immediately obvious to me that that's actually the case) > Debugging > In conversations with the Python Tools for Visual Studio team (PTVS) https://github.com/Microsoft/Pyjion/blob/master/pep.rst#ptvs, they thought they would find these API changes useful for implementing more performant debugging. As mentioned in the https://github.com/Microsoft/Pyjion/blob/master/pep.rst#rationale section, this API would allow for switching on debugging functionality only in frames where it is needed. This could allow for either skipping information that sys.settrace() normally provides and even go as far as to dynamically rewrite bytecode prior to execution to inject e.g. breakpoints in the bytecode. I'd suggest reaching out directly to Dave Malcolm on the GNU tools team in relation to this aspect, as I believe he did a lot of the work for the improved CPython runtime support in gdb 7+: https://docs.python.org/devguide/gdb.html That support currently mostly works at the frame level, so it would be interesting to know what additional capabilities might be enabled by being able to selectively intercept code execution at the bytecode level. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Tue May 17 00:08:55 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 16 May 2016 21:08:55 -0700 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: On its way. It's longer than I expected. Hopefully to be finished tomorrow... On Mon, May 16, 2016 at 1:48 PM, Brett Cannon wrote: > > > On Mon, 16 May 2016 at 11:38 Guido van Rossum wrote: >> >> On Mon, May 16, 2016 at 9:15 AM, Brett Cannon wrote: >>> >>> This might make a nice example in the docs and/or blog post since this is >>> hitting the intermediate/advanced space for typing that almost none of us >>> have hit. >> >> >> I'll see if I can follow up on this idea. But if anyone else feels like >> blogging about this, please go ahead! If you post a link here I'll retweet. > > > If you don't get to it then maybe I will in a blog post on PEP 519, but I > don't think I will be much faster than you for the foreseeable future. :) -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue May 17 01:09:33 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 16 May 2016 22:09:33 -0700 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: For the mailing list folks here's a draft of the promised blog post: https://paper.dropbox.com/doc/Adding-type-annotations-for-PEP-519-SQuovLc1ZyrC4EEIkeVzE Hopefully some of you can actually add comments in the sidebar (it requires a Dropbox login). I'll convert it to my usual Blogger format tomorrow. --Guido On Mon, May 16, 2016 at 9:08 PM, Guido van Rossum wrote: > On its way. It's longer than I expected. Hopefully to be finished tomorrow... > > On Mon, May 16, 2016 at 1:48 PM, Brett Cannon wrote: >> >> >> On Mon, 16 May 2016 at 11:38 Guido van Rossum wrote: >>> >>> On Mon, May 16, 2016 at 9:15 AM, Brett Cannon wrote: >>>> >>>> This might make a nice example in the docs and/or blog post since this is >>>> hitting the intermediate/advanced space for typing that almost none of us >>>> have hit. >>> >>> >>> I'll see if I can follow up on this idea. But if anyone else feels like >>> blogging about this, please go ahead! If you post a link here I'll retweet. >> >> >> If you don't get to it then maybe I will in a blog post on PEP 519, but I >> don't think I will be much faster than you for the foreseeable future. :) > > > > -- > --Guido van Rossum (python.org/~guido) -- --Guido van Rossum (python.org/~guido) From greg.ewing at canterbury.ac.nz Tue May 17 01:31:42 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 17 May 2016 17:31:42 +1200 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> <573A590E.30708@canterbury.ac.nz> Message-ID: <573AACBE.6070002@canterbury.ac.nz> Guido van Rossum wrote: > I know. But this could be considered syntactic sugar. I don't think it is just syntactic sugar. There is a real difference between having a box containing either an apple or a banana, and having just an apple or a banana. -- Greg From guido at python.org Tue May 17 01:56:05 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 16 May 2016 22:56:05 -0700 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: <573AACBE.6070002@canterbury.ac.nz> References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> <573A590E.30708@canterbury.ac.nz> <573AACBE.6070002@canterbury.ac.nz> Message-ID: Hmm... In Java there is a form of syntactic sugar that automatically deals with such boxes called auto-(un)boxing, IIUC. So I still think it can be called syntactic sugar. On Monday, May 16, 2016, Greg Ewing wrote: > Guido van Rossum wrote: > >> I know. But this could be considered syntactic sugar. >> > > I don't think it is just syntactic sugar. There is > a real difference between having a box containing > either an apple or a banana, and having just an > apple or a banana. > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Tue May 17 03:03:26 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 17 May 2016 16:03:26 +0900 Subject: [Python-ideas] Mention of "Sum" considered harmful [was: "Sum" Type hinting] In-Reply-To: References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> Message-ID: <22330.49726.660417.197837@turnbull.sk.tsukuba.ac.jp> Koos Zevenhoven writes: > On Mon, May 16, 2016 at 2:37 AM, Greg Ewing wrote: > > > > I'm not deeply into category theory, but the proposal > > seems to be that Sum would be a special kind of type > > that's assumed to be the same type wherever it appears > > in the signature of a function. > > Indeed, that's TypeVar. True as far as the statement itself makes sense (categorically, a type is what it is, so of course it is the same type wherever it appears!), but a Sum would not be a TypeVar. It's a type, it's far more limited[1] in analyzing Python programs, and I would like to withdraw the term itself, as well as any proposal to add it to typing, from this discussion. All concerned have my apologies for bringing it up at all. Footnotes: [1] And useful in category theory for that very reason -- it's easier to reason about in categorical contexts. But it makes things harder in Mypy. From greg.ewing at canterbury.ac.nz Tue May 17 05:38:10 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 17 May 2016 21:38:10 +1200 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> <573A590E.30708@canterbury.ac.nz> <573AACBE.6070002@canterbury.ac.nz> Message-ID: <573AE682.3040806@canterbury.ac.nz> Guido van Rossum wrote: > Hmm... In Java there is a form of syntactic sugar that automatically > deals with such boxes called auto-(un)boxing, IIUC. So I still think it > can be called syntactic sugar. That's not the same thing either. Boxing in Java is a hack to make up for the fact that some types are not objects, and the auto boxing and unboxing is there so that you can forget about the boxes and pretend that e.g. int and Integer are the same type (at least for some purposes). But with algebraic types, the boxes are entities in their own right whose presence conveys information. You can have more than one kind of box with the same contents: data Box = Matchbox Int | Shoebox Int Not only is a Matchbox distinguishable from a Shoebox at run time, but Box is a distinct type from Int -- you can't pass an Int directly to something expecting a Box or vice versa. A realisation of algebraic types in Python (or any other language, for that matter) would require carrying information at run time about which kind of box is present. This is in contrast to a union type, which is purely a compile-time concept and has no run-time implications. -- Greg From guettliml at thomas-guettler.de Tue May 17 07:09:09 2016 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Tue, 17 May 2016 13:09:09 +0200 Subject: [Python-ideas] Lib vs ( App | Project | ... ) Message-ID: <573AFBD5.6010902@thomas-guettler.de> This blog post from Donald Stufft explains the difference between a Library and a Application. https://caremad.io/2013/07/setup-vs-requirement/ The term "Library" is well known - ok But how to call the container for the libraries, the one thing providing the environment? Donald Stufft uses the term "Application" I would like an official term defined by the python docs. What do you think? Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ From k7hoven at gmail.com Tue May 17 07:28:08 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 17 May 2016 14:28:08 +0300 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: <573AE682.3040806@canterbury.ac.nz> References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> <573A590E.30708@canterbury.ac.nz> <573AACBE.6070002@canterbury.ac.nz> <573AE682.3040806@canterbury.ac.nz> Message-ID: On Tue, May 17, 2016 at 12:38 PM, Greg Ewing wrote: [...] > A realisation of algebraic types in Python (or any other > language, for that matter) would require carrying information > at run time about which kind of box is present. This is > in contrast to a union type, which is purely a compile-time > concept and has no run-time implications. The point about tagged unions was that Python *always* has the type information available at runtime. This type information corresponds to the "tag" in "tagged union". For a tagged union at runtime, a language does not need to carry around the information of what kind of box you have, but what kind of beast you have in that box. Unless of course you want a different kind of box type corresponding to each type, which would just be stupid. (Maybe there would be a separate box type for wrapping each of the boxes too ;-) Indeed I, on the other hand, was referring to the ambiguity of compile-time or static-type-checking unions, where the "tag" of a Union can only be known when a non-union/unambiguous type is explicitly passed to something that expects a union. But if the type hints do not "match the tags" [1], then the ambiguity of ("untagged") unions can "spread" in the static type-checking phase, while the runtime types are of course always well-defined. If a language simulates tagged unions by putting things in a box as you describe, and then comes up with a way to automatically release the bird out of a "Box" at runtime when you try to make it .quack(), then they would seem to be reinventing duck typing. - Koos [1] The way TypeVars or @overloads do, or my hypothetical TagMatcher sum example. From ncoghlan at gmail.com Tue May 17 09:20:30 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 17 May 2016 23:20:30 +1000 Subject: [Python-ideas] Lib vs ( App | Project | ... ) In-Reply-To: <573AFBD5.6010902@thomas-guettler.de> References: <573AFBD5.6010902@thomas-guettler.de> Message-ID: On 17 May 2016 at 21:09, Thomas G?ttler wrote: > This blog post from Donald Stufft explains the difference between > a Library and a Application. > > https://caremad.io/2013/07/setup-vs-requirement/ > > > The term "Library" is well known - ok > > But how to call the container for the libraries, the one thing providing the > environment? > > Donald Stufft uses the term "Application" > > I would like an official term defined by the python docs. Covering general principles of software engineering and the many and varied terms used for different kinds of software aggregation is *way* outside the scope of the Python language and standard library docs :) The closest we get is the PyPA/distutils-sig consensus that software is developed by and as projects (covering libraries, applications, and assorted other endeavours): https://packaging.python.org/en/latest/glossary/#term-project The install_requires vs requirements.txt question is covered as its own topic, without introducing any particular terminology: https://packaging.python.org/en/latest/requirements/ Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve.dower at python.org Tue May 17 09:55:25 2016 From: steve.dower at python.org (Steve Dower) Date: Tue, 17 May 2016 06:55:25 -0700 Subject: [Python-ideas] Lib vs ( App | Project | ... ) In-Reply-To: <573AFBD5.6010902@thomas-guettler.de> References: <573AFBD5.6010902@thomas-guettler.de> Message-ID: "how to call the container for the libraries, the one thing providing the environment?" In Visual Studio we call these Environments (in the source they're still referred to as Interpreters). A "virtual environment" is just a specific type of environment (one that inherits a stdlib and/or packages), as is an isolated environment (hosted in an app, or a conda env). They all specify a Python version and concrete versions of libraries - something directly executable, not something that needs to be actualised before use. Not sure that we need an official term, but the venv docs likely have or need a definition to work with. Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Thomas G?ttler" Sent: ?5/?17/?2016 4:28 To: "python-ideas" Subject: [Python-ideas] Lib vs ( App | Project | ... ) This blog post from Donald Stufft explains the difference between a Library and a Application. https://caremad.io/2013/07/setup-vs-requirement/ The term "Library" is well known - ok But how to call the container for the libraries, the one thing providing the environment? Donald Stufft uses the term "Application" I would like an official term defined by the python docs. What do you think? Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Tue May 17 10:44:43 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 17 May 2016 17:44:43 +0300 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: This seems to be what I thought too, except completely in the stubs. I might add some comments in the blog post draft. -- Koos On Sun, May 15, 2016 at 8:21 PM, Guido van Rossum wrote: > I didn't have time to read the thread, but I read the PEP and thought about > this a little bit. > > One key thing is that we can write the code CPython sees at runtime one way, > and write the stubs that type checkers (like mypy) see a different way. The > stubs go in the typeshed repo (https://github.com/python/typeshed) and I > would add something like the following to the os module there > (stdlib/3/os/__init__.pyi in the repo). > > First we need to add scandir() and DirEntry (this is not entirely unrelated > -- DirEntry is an example of something that is PathLike). Disregarding the > PathLike protocol for the moment, I think they can be defined like this: > > if sys.version_info >= (3, 5): > class DirEntry(Generic[AnyStr]): > name = ... # type: AnyStr > path = ... # type: AnyStr > def inode(self) -> int: ... > def is_dir(self, *, follow_symlinks: bool = ...) -> bool: ... > def is_file(self, *, follow_symlinks: bool = ...) -> bool: ... > def is_symlink(self) -> bool: ... > def stat(self, *, follow_symlinks: bool = ...) -> stat_result: ... > > @overload > def scandir(path: str = ...) -> DirEntry[str]: ... > @overload > def scandir(path: bytes) -> DirEntry[bytes]: ... > > Note that the docs claim there's a type os.DirEntry, even though it doesn't > currently exist -- I think we should fix that in 3.6 even if it may not make > sense to instantiate it. > > Also note that a slightly different overload is also possible -- I think > these are for all practical purposes the same: > > @overload > def scandir() -> DirEntry[str]: ... > @overload > def scandir(path: AnyStr) -> DirEntry[AnyStr]: ... > > The reason we need the overload in all cases is that os.scandir() without > arguments returns a str. > > Finally, a reminder that this is all stub code -- it's only ever seen by > typecheckers. What we put in the actual os.py file in the stdlib can be > completely different, and it doesn't need type annotations (type checkers > always prefer the stubs over the real code). > > Now let's add PathLike. This first attempt doesn't address DirEntry yet: > > if sys.version_info >= (3, 6): > from abc import abstractmethod > class PathLike(Generic[AnyStr]): > @abstractmethod > def __fspath__(self) -> AnyStr: ... > > @overload > def fspath(path: PathLike[AnyStr]) -> AnyStr: ... > @overload > def fspath(path: AnyStr) -> AnyStr: ... > > This tells a type checker enough so that it will know that e.g. > os.fspath(b'.') returns a bytes object. Also, if we have a class C that > derives from PathLike we can make it non-generic, e.g. the stubs for > pathlib.Path would start with something like > > class Path(os.PathLike[str]): > ... > > and now the type checker will know that in the following code `c` is always > a str: > > a = ... # type: Any > b = pathlib.Path(a) > c = os.fspath(b) > > Finally let's redefine scandir(). We'll have to redefind DirEntry to inherit > from PathLike, and it will remain generic: > > class DirEntry(PathLike[AnyStr], Generic[AnyStr]): > # Everything else unchanged! > > Now the type checker should understand the following: > > for a in os.scandir('.'): > b = os.fspath(a) > ... > > Here it will know that `a` is a DirEntry[str] (because the argument given to > os.scandir() is a str) and hence it will also know that b is a str. Now if > then pass b to pathlib it will understand this cannot be a type error, and > if you pass b to some os.path.* function (e.g. os.path.basename()) it will > understand the return value is a str. > > If you pass some variable to os.scandir() then if the type checker can > deduce that that variable is a str (e.g. because you've gotten it from > pathlib) it will know that the results are DirEntry[str] instances. If you > pass something to os.scandir() that's a bytes object it will know that the > results are DirEntry[bytes] objects, and it knows that calling os.fspath() > on those will return bytes. (And it will know that you can't pass those to > pathlib, but you *can* pass them to most os and os.path functions.) > > Next, if the variable passed to os.scandir() has the declared or inferred > type AnyStr then mypy will know that it can be either str or bytes and the > types of results will also use AnyStr. I think in that case you'll get an > error if you pass it to pathlib. Note that this can only happen inside a > generic class or a generic function that has AnyStr as one of its > parameters. (AnyStr is itself a type variable.) > > The story ought to be similar if the variable has the type Union[str, > bytes], except that this can occur in non-generic code and the resulting > types are similarly fuzzy. (I think there's a bug in mypy around this > though, follow https://github.com/python/mypy/issues/1533 if you're > interested how that turns out.) > > -- > --Guido van Rossum (python.org/~guido) From brett at python.org Tue May 17 11:10:47 2016 From: brett at python.org (Brett Cannon) Date: Tue, 17 May 2016 15:10:47 +0000 Subject: [Python-ideas] Adding a frame evaluation API to CPython In-Reply-To: References: Message-ID: On Mon, 16 May 2016 at 20:50 Nick Coghlan wrote: > This idea generally sounds reasonable to me, I just have some > suggestions for other folks to approach specifically for feedback. > > On 17 May 2016 at 06:19, Dino Viehland wrote: > > Other JITs > > It should be mentioned that the Pyston team was consulted on an earlier > version of this PEP that was more JIT-specific and they were not interested > in utilizing the changes proposed because they want control over memory > layout they had no interest in directly supporting CPython itself. An > informal discusion with a developer on the PyPy team led to a similar > comment. > > Numba https://github.com/Microsoft/Pyjion/blob/master/pep.rst#numba, on > the other hand, suggested that they would be interested in the proposed > change in a post-1.0 future for themselves > https://github.com/Microsoft/Pyjion/blob/master/pep.rst#numba-interest. > > Hopefully Victor will chime in anyway, but if he doesn't, I'd suggest > asking him directly what impact this proposal might have on the > bytecode transformation aspects of PEP 511. > There won't be any direct impact and would actually benefit from it as any improved bytecode emission would mean better JIT output. But that's getting rather JIT-specific and this PEP tries to only use JITs as a motivation, not the exact/only reason to add this API. > > There also seems to be a potential overlap with the function > specialisation proposal in PEP 510, so it would be good to see this > PEP discussing its relationship with that one (the explanation may be > "they're orthogonal proposals addressing different concerns", but it > isn't immediately obvious to me that that's actually the case) > "They're orthogonal proposals addressing different concerns". :) While a JIT may be able to benefit from it, the JIT aspect is simply a use of the proposed API (we have purposefully tried not to pain ourselves into a corner of being a JIT-only thing). IOW I purposefully didn't draw in other perf PEPs relating to bytecode as it isn't directly related to the overall proposal. > > > Debugging > > In conversations with the Python Tools for Visual Studio team (PTVS) > https://github.com/Microsoft/Pyjion/blob/master/pep.rst#ptvs, they > thought they would find these API changes useful for implementing more > performant debugging. As mentioned in the > https://github.com/Microsoft/Pyjion/blob/master/pep.rst#rationale > section, this API would allow for switching on debugging functionality only > in frames where it is needed. This could allow for either skipping > information that sys.settrace() normally provides and even go as far as to > dynamically rewrite bytecode prior to execution to inject e.g. breakpoints > in the bytecode. > > I'd suggest reaching out directly to Dave Malcolm on the GNU tools > team in relation to this aspect, as I believe he did a lot of the work > for the improved CPython runtime support in gdb 7+: > https://docs.python.org/devguide/gdb.html > > That support currently mostly works at the frame level, so it would be > interesting to know what additional capabilities might be enabled by > being able to selectively intercept code execution at the bytecode > level. > I'll shoot him an email (and based on how you phrased that I'm going to assume you said it with your work hat on and he's still at RH :). -Brett > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue May 17 11:11:44 2016 From: brett at python.org (Brett Cannon) Date: Tue, 17 May 2016 15:11:44 +0000 Subject: [Python-ideas] Adding a frame evaluation API to CPython In-Reply-To: References: Message-ID: On Mon, 16 May 2016 at 16:13 Matthias welp wrote: > I won't comment on the content of this proposal, as I'm don't really see > the implications, but I would like to correct one small mistake: > > Under 'performance impact' you say this: > > > > In terms of memory impact, since there are typically not many CPython > interpreters executing in a single process that means the impact of > co_extra being added to PyCodeObject is the only worry. According to > https://github.com/Microsoft/Pyjion/blob/master/pep.rst#code-object-count, > a run of the Python test suite results in about 72,395 code objects being > created. On a 64-bit CPU that would result in 4,633,280 bytes of extra > memory being used if all code objects were alive at once and had nothing > set in their co_extra fields. > > The extra RAM used is a bit off. One 64-bit pointer extra takes only 8 > bytes, which translates to about 579160 bytes extra used in in the test > suite. I did not consider padding for accessibility, but that was also not > considered in the draft. > Oops, my mistake. Thanks for pointing it out! -Brett > > -Matthias > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue May 17 11:24:34 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 17 May 2016 08:24:34 -0700 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> <573A590E.30708@canterbury.ac.nz> <573AACBE.6070002@canterbury.ac.nz> <573AE682.3040806@canterbury.ac.nz> Message-ID: <-7591141165036766983@unknownmsgid> The difference between a box with one apple in it and a single apple is critical. In fact, I think that is the source of the most common type issue in real Python code: You can't tell the difference (without type checking) between a sequence of strings and a string. Because, of course, a string IS a sequence of strings. In this case, there is no character type, i.e. no type to represent a single apple. (Kind of an infinite Russian doll of Apple boxes...) Type hinting will help address the issue for strings, but it seems a very useful distinction for all types. -CHB From guido at python.org Tue May 17 11:59:15 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 17 May 2016 08:59:15 -0700 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: <573AE682.3040806@canterbury.ac.nz> References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> <573A590E.30708@canterbury.ac.nz> <573AACBE.6070002@canterbury.ac.nz> <573AE682.3040806@canterbury.ac.nz> Message-ID: On Tue, May 17, 2016 at 2:38 AM, Greg Ewing wrote: > Guido van Rossum wrote: >> >> Hmm... In Java there is a form of syntactic sugar that automatically deals >> with such boxes called auto-(un)boxing, IIUC. So I still think it can be >> called syntactic sugar. > > > That's not the same thing either. Boxing in Java is a hack > to make up for the fact that some types are not objects, > and the auto boxing and unboxing is there so that you can > forget about the boxes and pretend that e.g. int and Integer > are the same type (at least for some purposes). > > But with algebraic types, the boxes are entities in their > own right whose presence conveys information. You can have > more than one kind of box with the same contents: > > data Box = Matchbox Int | Shoebox Int > > Not only is a Matchbox distinguishable from a Shoebox at > run time, but Box is a distinct type from Int -- you can't > pass an Int directly to something expecting a Box or > vice versa. > > A realisation of algebraic types in Python (or any other > language, for that matter) would require carrying information > at run time about which kind of box is present. This is > in contrast to a union type, which is purely a compile-time > concept and has no run-time implications. I'm sorry, I wasn't trying to claim that Java's auto-(un)boxing was anything like ADTs; I know better. I was just quipping that just because there's a difference between a box containing a piece of fruit and the piece of fruit itself, that doesn't mean handling the box can't be considered syntactic sugar -- your original remark claimed something wasn't syntactic sugar because of the difference between the box and its contents, and that's what I disagree with. -- --Guido van Rossum (python.org/~guido) From brett at python.org Tue May 17 12:33:08 2016 From: brett at python.org (Brett Cannon) Date: Tue, 17 May 2016 16:33:08 +0000 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: On Mon, 16 May 2016 at 22:09 Guido van Rossum wrote: > For the mailing list folks here's a draft of the promised blog post: > > https://paper.dropbox.com/doc/Adding-type-annotations-for-PEP-519-SQuovLc1ZyrC4EEIkeVzE > > Hopefully some of you can actually add comments in the sidebar (it > requires a Dropbox login). > No comments from me, although it's going to be interesting making this all work in Typeshed if we backport the PathLike support to the pathlib constructor to 3.4 and 3.5 but (obviously) not os.PathLike itself. -Brett > > I'll convert it to my usual Blogger format tomorrow. > > --Guido > > On Mon, May 16, 2016 at 9:08 PM, Guido van Rossum > wrote: > > On its way. It's longer than I expected. Hopefully to be finished > tomorrow... > > > > On Mon, May 16, 2016 at 1:48 PM, Brett Cannon wrote: > >> > >> > >> On Mon, 16 May 2016 at 11:38 Guido van Rossum wrote: > >>> > >>> On Mon, May 16, 2016 at 9:15 AM, Brett Cannon > wrote: > >>>> > >>>> This might make a nice example in the docs and/or blog post since > this is > >>>> hitting the intermediate/advanced space for typing that almost none > of us > >>>> have hit. > >>> > >>> > >>> I'll see if I can follow up on this idea. But if anyone else feels like > >>> blogging about this, please go ahead! If you post a link here I'll > retweet. > >> > >> > >> If you don't get to it then maybe I will in a blog post on PEP 519, but > I > >> don't think I will be much faster than you for the foreseeable future. > :) > > > > > > > > -- > > --Guido van Rossum (python.org/~guido) > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue May 17 13:33:21 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 17 May 2016 10:33:21 -0700 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: On Mon, May 16, 2016 at 10:09 PM, Guido van Rossum wrote: > For the mailing list folks here's a draft of the promised blog post: > https://paper.dropbox.com/doc/Adding-type-annotations-for-PEP-519-SQuovLc1ZyrC4EEIkeVzE > > Hopefully some of you can actually add comments in the sidebar (it > requires a Dropbox login). > > I'll convert it to my usual Blogger format tomorrow. I ended up breaking this in two. Part 1, about AnyStr, is now posted: http://neopythonic.blogspot.com/2016/05/the-anystr-type-variable.html Part 2 is still in draft and I'm hoping to get more feedback: https://paper.dropbox.com/doc/Adding-type-annotations-for-PEP-519-SQuovLc1ZyrC4EEIkeVzE -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue May 17 13:34:44 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 17 May 2016 10:34:44 -0700 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: On Tue, May 17, 2016 at 9:33 AM, Brett Cannon wrote: > No comments from me, although it's going to be interesting making this all > work in Typeshed if we backport the PathLike support to the pathlib > constructor to 3.4 and 3.5 but (obviously) not os.PathLike itself. By then mypy should support "if sys.version_info >= ..." checks. (There's some work required still: https://github.com/python/mypy/issues/698) -- --Guido van Rossum (python.org/~guido) From victor.stinner at gmail.com Tue May 17 14:51:18 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 17 May 2016 20:51:18 +0200 Subject: [Python-ideas] Adding a frame evaluation API to CPython In-Reply-To: References: Message-ID: Hi, Nice PEP. I would be nice to see how Yury Sulivanov's "implement per-opcode cache in ceval" patch would benefit from your PEP: * https://bugs.python.org/issue26219 * https://mail.python.org/pipermail/python-dev/2016-February/143025.html This patch depends on the "Speedup method calls 1.2x" patch: * https://bugs.python.org/issue26110 * https://mail.python.org/pipermail/python-dev/2016-January/142945.html Victor From tjreedy at udel.edu Tue May 17 17:07:59 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 17 May 2016 17:07:59 -0400 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: <20160517003511.AD7C9B14094@webabinitio.net> References: <20160517003511.AD7C9B14094@webabinitio.net> Message-ID: On 5/16/2016 8:35 PM, R. David Murray wrote: > I'm currently working on a project for Intel involving Python and directly > addressable non-volatile memory. See https://nvdimm.wiki.kernel.org/ > for links to the Linux features underlying this work, and http://pmem.io/ > for the Intel initiated open source project I'm working with whose intent > is to bring support for DAX NVRAM to the application programming layer > (nvml currently covers C, with C++ support in process, and there are > python bindings for the lower level (non-libpmemobj) libraries). How is non-volatile NVRAM different from static SRAM? > tldr: In the future (and the future is now) application programs will > want to be aware of what class of memory they are allocating: What I want is to be, if anything, less aware. I remember fiddling with register declarations in C. Then is was discovered that compilers can allocate registers better than move people, so that 'register' is deprecated for most C programmers. I have never had to worry about the L1, L2, L3 on chip caches, though someone has to. I have long thought that I should be able to randomly access data on disk the same way I would access that same data in RAM, and not have to fiddle with seek and so on. Virtual memory is sort of like this, except that it uses the disk as a volatile* cache for RAM objects. (* Volatile in the sense that when the program ends, the disk space is freed for other use, and is inaccessible even if not.) Whereas I am thinking of using RAM as a cache for a persistent disk object. A possible user API (assuming txt stored as a python string with k bytes per char): cons = str(None, path="c:/user/terry/gutenburg/us_constitution.txt") # now to slice like it was in ram preamble = cons[:cons.find(section_marker)] Perhaps you are pointing to being able to make this possible, from the implementer side. The generic interfaces would be bytes(None, path=) (read only) and bytearray(None, path=) (read-write). A list does not seem like a good candidate for static mem, unless insert and delete are suppressed/unused. [snip] If static objects were **always** aligned in 4-byte boundaries, then the lowest 2 bits could be used to indicate memory type. To not slow down programs, this should be supported by the CPU address decoder. Isn't Intel thinking/working on something like this? -- Terry Jan Reedy From ethan at stoneleaf.us Wed May 18 00:39:07 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 17 May 2016 21:39:07 -0700 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: <20160517003511.AD7C9B14094@webabinitio.net> References: <20160517003511.AD7C9B14094@webabinitio.net> Message-ID: <573BF1EB.9030303@stoneleaf.us> On 05/16/2016 05:35 PM, R. David Murray wrote: > I'm currently working on a project for Intel involving Python and directly > addressable non-volatile memory. Sounds cool. The first thing it reminded me of was database transactions. What are the expected speed comparisons? I would imagine that persistent RAM is going to be slower. How much would a typical configuration have? Enough for all a program's data? What are the expected use-cases? For example, loop variables might not be a prime target for being stored in persistent RAM. What are the advantages of using this persistent RAM directly as variables vs. keeping the access as it's own step via the driver layer? -- ~Ethan~ From g.rodola at gmail.com Wed May 18 01:29:40 2016 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Wed, 18 May 2016 01:29:40 -0400 Subject: [Python-ideas] Add contextlib.DummyContext Message-ID: This is one of those things which are so easy to implement which makes you think it is probably not worth adding to the stdlib, but then again, this is something I've ended up doing and rewriting pretty often over the years. Real world example: class DummyLock(object): def __init__(self, *args, **kwargs): pass def __enter__(self, *args, **kwargs): return self def __exit__(self, *args, **kwargs): pass def get_lock(name, bypass_lock=False): lock_cls = DummyLock if bypass_lock else RedisLock return lock with get_lock('foo', bypass_lock=True): ... Similarly to contextlib.closing and contextlib.suppress, perhaps it would be nice to have contextlib.DummyContext just because it's something which is done (I think) fairly often. On the bikeshedding front, in order to be consistent with closing(), redirect_stderr(), redirect_stdout() and suppress(), a better name for this would probably be contextlib.dummy_context. Extra: the same thing can be achieved by using mock.MagicMock, which probably makes this proposal useless and kills it entirely. The additional value is that it would be more explicit/clear/immediate to have this in contextlib itself as opposed to unittest module, which is kinda weird. But then again, "there should be (possibly) only one way to do it" so I'm not sure. OK, I should stop talking with myself. =) -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From joejev at gmail.com Wed May 18 01:34:23 2016 From: joejev at gmail.com (Joseph Jevnik) Date: Wed, 18 May 2016 01:34:23 -0400 Subject: [Python-ideas] Add contextlib.DummyContext In-Reply-To: References: Message-ID: You can use an ExitStack for this On Wed, May 18, 2016 at 1:29 AM, Giampaolo Rodola' wrote: > This is one of those things which are so easy to implement which makes you > think it is probably not worth adding to the stdlib, but then again, this > is something I've ended up doing and rewriting pretty often over the years. > Real world example: > > class DummyLock(object): > def __init__(self, *args, **kwargs): > pass > def __enter__(self, *args, **kwargs): > return self > def __exit__(self, *args, **kwargs): > pass > > def get_lock(name, bypass_lock=False): > lock_cls = DummyLock if bypass_lock else RedisLock > return lock > > with get_lock('foo', bypass_lock=True): > ... > > Similarly to contextlib.closing and contextlib.suppress, perhaps it would > be nice to have contextlib.DummyContext just because it's something which > is done (I think) fairly often. > > On the bikeshedding front, in order to be consistent with > closing(), redirect_stderr(), redirect_stdout() and suppress(), a better > name for this would probably be contextlib.dummy_context. > > Extra: the same thing can be achieved by using mock.MagicMock, which > probably makes this proposal useless and kills it entirely. The additional > value is that it would be more explicit/clear/immediate to have this in > contextlib itself as opposed to unittest module, which is kinda weird. But > then again, "there should be (possibly) only one way to do it" so I'm not > sure. > OK, I should stop talking with myself. =) > > -- > Giampaolo - http://grodola.blogspot.com > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Wed May 18 01:36:37 2016 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 17 May 2016 22:36:37 -0700 Subject: [Python-ideas] Add contextlib.DummyContext In-Reply-To: References: Message-ID: > This is one of those things which are so easy to implement which makes you > think it is probably not worth adding to the stdlib, but then again, this is > something I've ended up doing and rewriting pretty often over the years. > Real world example: > > class DummyLock(object): > def __init__(self, *args, **kwargs): > pass > def __enter__(self, *args, **kwargs): > return self > def __exit__(self, *args, **kwargs): > pass > > def get_lock(name, bypass_lock=False): > lock_cls = DummyLock if bypass_lock else RedisLock > return lock > > with get_lock('foo', bypass_lock=True): > ... with contextlib.ExitStack() as exit_stack: if not bypass_lock: exit_stack.enter_context(RedisLock()) And similar. In fact, ExitStack can itself be used as a no-op context manager, if you want. -- Devin From ncoghlan at gmail.com Wed May 18 02:30:43 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 18 May 2016 16:30:43 +1000 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: <20160517003511.AD7C9B14094@webabinitio.net> References: <20160517003511.AD7C9B14094@webabinitio.net> Message-ID: On 17 May 2016 at 10:35, R. David Murray wrote: > Feel free to tell me this is the wrong forum, and if so we can a new forum > to continue the conversation if enough people are interested. I think it's the right forum, although a SIG may turn out to be appropriate eventually (then again, there are plenty of arcane topics where we've managed to get by without ever creating a dedicated SIG). [snip background info] > At the language level, support would mean two things, I think: there > would need to be a way to declare which backing store an object belongs > to, and there would need to be a notion of a memory hierarchy where, at > least in the case of persistent vs non-persistent RAM, an object higher in > the hierarchy could not point to an object lower in the hierarchy that > was mutable, and that immutables would be copied from lower to higher. > (Here I'm notionally defining "lower" as "closer to the CPU", since RAM > is 'just there' whereas persistent memory goes through a driver layer > before direct access can get exposed, and fast DRAM would be closer yet > to the CPU :). > > In CPython I envision this being implemented by having every object > be associated with a 'storage manager', with the DRAM storage manager > obviously being the default. I think that which storage manager an > object belongs to can be deduced from its address at runtime, although > whether that would be better than storing a pointer is an open question. > Object method implementations would then need to be expanded as follows: > (1) wrapping any operations that comprise an 'atomic' operation with a > 'start transaction' and 'end transaction' call on the storage manager. > (2) any memory management function (malloc, etc) would be called > indirectly through the storage manager, (3) any object pointer to be > stored or retrieved would be passed through an appropriate call on the > storage manager to give it an opportunity to block it or transform it, > and (4) any memory range to be modified would be reported to the memory > manager so that it can be registered with the transaction. > > I *think* that's the minimum set of changes. Clearly, however, making > such calls is going to be less efficient in the DRAM case than the > current code, even though the RAM implementation would be a noop. The three main multiple-kinds-of-memory systems I'm personally familiar with are heap management in DSP/BIOS (now TI-RTOS), C++ memory allocators, and Rust's tiered memory management model (mainly for thread locals vs global heap objects). The DSP/BIOS model is probably too simplistic to help us out - it's just "allocate this memory on that heap", and then you're own your own at the semantic level. Rust's tiered memory management is really nice, but also completely foreign to the C-style memory management model that underpins CPython. However, it does suggest "thread local" as an allocator type worth considering along with "normal process heap" and "non-volatile storage". That leaves the C++ custom object allocator model, and if I recall correctly, allocators there are managed at the *type* level - if you want to use a different allocator, you need to declare a new type. For Python, I'd suggest looking at going one step further, and associating storage management with the *metaclass* - the idea there would be to take advantage of the metaclass conflict if someone attempts to combine data types with conflicting storage management expectations. So you might have "PersistentType", which would itself be a normal type instance (stored in RAM), and the classes it created would also be stored in RAM, but they'd be configured to handle the storage for individual instances differently from normal classes. I'm not sure if the current descriptor protocol and the GC management machinery would give you sufficient access to control all the things you care about for this purpose, but it should let you control quite a lot of it, and trying it out would give you more info on what's currently missing. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guettliml at thomas-guettler.de Wed May 18 02:38:43 2016 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Wed, 18 May 2016 08:38:43 +0200 Subject: [Python-ideas] Lib vs ( App | Project | ... ) In-Reply-To: References: <573AFBD5.6010902@thomas-guettler.de> Message-ID: <573C0DF3.1070000@thomas-guettler.de> Am 17.05.2016 um 15:20 schrieb Nick Coghlan: > On 17 May 2016 at 21:09, Thomas G?ttler wrote: >> This blog post from Donald Stufft explains the difference between >> a Library and a Application. >> >> https://caremad.io/2013/07/setup-vs-requirement/ >> >> >> The term "Library" is well known - ok >> >> But how to call the container for the libraries, the one thing providing the >> environment? >> >> Donald Stufft uses the term "Application" >> >> I would like an official term defined by the python docs. > > Covering general principles of software engineering and the many and > varied terms used for different kinds of software aggregation is *way* > outside the scope of the Python language and standard library docs :) AFAIK there can only be **one** sitecustomize.py. I think the docs should be able to have a term for the thing which contains/provides this file. The current vacuum gets filled with a lot of different terms meaning the same thing. This does not hurt, but still I **feel** it would help to have an agreement on a term. > The closest we get is the PyPA/distutils-sig consensus that software > is developed by and as projects (covering libraries, applications, and > assorted other endeavours): > https://packaging.python.org/en/latest/glossary/#term-project Thank you Nick, for providing this link. I hope your are not the author of this text. This is no definition according to my point of view. I read it, and don't get it. Example: I think "django" is a project according to this definition. But I search a term for the thing containing: settings.py sitecustomize.py logging-config, .... > The install_requires vs requirements.txt question is covered as its > own topic, without introducing any particular terminology: > https://packaging.python.org/en/latest/requirements/ It is hard to cover the meaning of requirements.txt if there is no common term to name the container which contains it :-) The above link calls it "complete python environment". Up to now I found these terms: - Project https://docs.djangoproject.com/en/1.8/ref/applications/#projects-and-applications - Application https://caremad.io/2013/07/setup-vs-requirement/ - site https://docs.python.org/3/library/site.html - complete python environment https://packaging.python.org/en/latest/requirements/ I guess there are more. "Standards are great: Everyone should have one" Is this the way you want it to be? I see no agreement on the name for the concrete installation holding all those nice python libraries. Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ From g.rodola at gmail.com Wed May 18 09:16:19 2016 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Wed, 18 May 2016 09:16:19 -0400 Subject: [Python-ideas] Add contextlib.DummyContext In-Reply-To: References: Message-ID: On Wed, May 18, 2016 at 1:36 AM, Devin Jeanpierre wrote: > > This is one of those things which are so easy to implement which makes > you > > think it is probably not worth adding to the stdlib, but then again, > this is > > something I've ended up doing and rewriting pretty often over the years. > > Real world example: > > > > class DummyLock(object): > > def __init__(self, *args, **kwargs): > > pass > > def __enter__(self, *args, **kwargs): > > return self > > def __exit__(self, *args, **kwargs): > > pass > > > > def get_lock(name, bypass_lock=False): > > lock_cls = DummyLock if bypass_lock else RedisLock > > return lock > > > > with get_lock('foo', bypass_lock=True): > > ... > > with contextlib.ExitStack() as exit_stack: > if not bypass_lock: > exit_stack.enter_context(RedisLock()) > > And similar. > > In fact, ExitStack can itself be used as a no-op context manager, if you > want. > > -- Devin > I didn't think about this. So yeah, my proposal is useless. Nevermind. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed May 18 10:09:56 2016 From: guido at python.org (Guido van Rossum) Date: Wed, 18 May 2016 07:09:56 -0700 Subject: [Python-ideas] Type hinting for path-related functions In-Reply-To: References: <57363E26.6070908@stoneleaf.us> Message-ID: Here's part 2: http://neopythonic.blogspot.com/2016/05/adding-type-annotations-for-fspath.html On Tue, May 17, 2016 at 10:33 AM, Guido van Rossum wrote: > On Mon, May 16, 2016 at 10:09 PM, Guido van Rossum wrote: >> For the mailing list folks here's a draft of the promised blog post: >> https://paper.dropbox.com/doc/Adding-type-annotations-for-PEP-519-SQuovLc1ZyrC4EEIkeVzE >> >> Hopefully some of you can actually add comments in the sidebar (it >> requires a Dropbox login). >> >> I'll convert it to my usual Blogger format tomorrow. > > I ended up breaking this in two. Part 1, about AnyStr, is now posted: > http://neopythonic.blogspot.com/2016/05/the-anystr-type-variable.html > > Part 2 is still in draft and I'm hoping to get more feedback: > https://paper.dropbox.com/doc/Adding-type-annotations-for-PEP-519-SQuovLc1ZyrC4EEIkeVzE > > -- > --Guido van Rossum (python.org/~guido) -- --Guido van Rossum (python.org/~guido) From ethan at stoneleaf.us Wed May 18 11:11:00 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 18 May 2016 08:11:00 -0700 Subject: [Python-ideas] Add contextlib.DummyContext In-Reply-To: References: Message-ID: <573C8604.9090808@stoneleaf.us> On 05/18/2016 06:16 AM, Giampaolo Rodola' wrote: > On Wed, May 18, 2016 at 1:36 AM, Devin Jeanpierre wrote: >> with contextlib.ExitStack() as exit_stack: >> if not bypass_lock: >> exit_stack.enter_context(RedisLock()) >> >> And similar. >> >> In fact, ExitStack can itself be used as a no-op context manager, if >> you want. > > I didn't think about this. So yeah, my proposal is useless. Nevermind. Not at all. It's a good use-case, and it helped introduce/remind us about ExitStack (which I suspect is a hidden gem). So thank you all! :) -- ~Ethan~ From chris.barker at noaa.gov Wed May 18 11:57:30 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 18 May 2016 08:57:30 -0700 Subject: [Python-ideas] Lib vs ( App | Project | ... ) In-Reply-To: <573C0DF3.1070000@thomas-guettler.de> References: <573AFBD5.6010902@thomas-guettler.de> <573C0DF3.1070000@thomas-guettler.de> Message-ID: On Tue, May 17, 2016 at 11:38 PM, Thomas G?ttler < guettliml at thomas-guettler.de> wrote: > > AFAIK there can only be **one** sitecustomize.py. I think the docs > should be able to have a term for the thing which contains/provides this > file. > uhm, the file system? it's a single file, and while a python module technically, it acts as a configuration file. so not a library, or a system or a package, or.... > The current vacuum gets filled with a lot of different terms meaning the > same > thing. This does not hurt, but still I **feel** it would help to have an > agreement on a term. sure, but where, how, etc does one gain consensus? since this all overlaps with non-python software development, I do'nt htink we'll ever get a consensus. > The closest we get is the PyPA/distutils-sig consensus that software >> is developed by and as projects (covering libraries, applications, and >> assorted other endeavours): >> https://packaging.python.org/en/latest/glossary/#term-project >> > > Thank you Nick, for providing this link. I hope your are not the author > of this text. This is no definition according to my point of view. > I read it, and don't get it. I"m sure PRs are considered :-) > Example: I think "django" is a > project according to this definition. I think it's a package. An application built with Django is a Project. See -- no consensus :-) > The install_requires vs requirements.txt question is covered as its >> own topic, without introducing any particular terminology: >> https://packaging.python.org/en/latest/requirements/ >> > > It is hard to cover the meaning of requirements.txt if there > is no common term to name the container which contains it :-) > I think this all comes down to guidelines and suggestions -- each developer is going to need to figure out what best fits their needs. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Wed May 18 13:11:20 2016 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 18 May 2016 13:11:20 -0400 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: References: <20160517003511.AD7C9B14094@webabinitio.net> Message-ID: <20160518171122.BB33BB1401C@webabinitio.net> On Tue, 17 May 2016 17:07:59 -0400, Terry Reedy wrote: > On 5/16/2016 8:35 PM, R. David Murray wrote: > > I'm currently working on a project for Intel involving Python and directly > > addressable non-volatile memory. See https://nvdimm.wiki.kernel.org/ > > for links to the Linux features underlying this work, and http://pmem.io/ > > for the Intel initiated open source project I'm working with whose intent > > is to bring support for DAX NVRAM to the application programming layer > > (nvml currently covers C, with C++ support in process, and there are > > python bindings for the lower level (non-libpmemobj) libraries). > > How is non-volatile NVRAM different from static SRAM? NVRAM retains the data even if it loses power. However, the programming issues involved in using direct memory access to battery backed SRAM should be similar, I think. > > tldr: In the future (and the future is now) application programs will > > want to be aware of what class of memory they are allocating: > > What I want is to be, if anything, less aware. I remember fiddling with > register declarations in C. Then is was discovered that compilers can > allocate registers better than move people, so that 'register' is > deprecated for most C programmers. I have never had to worry about the > L1, L2, L3 on chip caches, though someone has to. Yes, and I think for the hypothetical "fast memory" class that's probably what a dynamic language like python would ideally want to do, even if compiled languages didn't. So really I'm talking about NVRAM, the other was just a hypothetical example of another class of memory besides normal DRAM. For RAM of whatever flavor that retains its value between program invocations, the programmer has to be aware that the data is persistent and program accordingly. Indeed, to some degree it does not matter what the persistent store is, the issue is programming for persistence. The thing that NVRAM brings in to the picture is the motivation to do persistence *not* via serialization and deserialization, but via direct random access to memory that retains its state. Adding support for persistence to the language is actually more generically useful than just the NVRAM realm (consider the hoops the ZODB has to go through to support persistence of Python objects, for example). For current persistence schemes the persistence is file-system based, and the time costs of serialization are swamped by the time costs of the file system access (even when the file system is as fast as possible and/or SRAM or NVRAM based). What is different now, and what makes thinking about this now worthwhile, is that the *direct* access (ie: not driver mediated) to the NVRAM memory locations makes the time cost of serialization swamp the time costs of accessing the persistent data. > I have long thought that I should be able to randomly access data on > disk the same way I would access that same data in RAM, and not have to > fiddle with seek and so on. Virtual memory is sort of like this, except mmap + memoryview allows you to do this (in python3), does it not? > that it uses the disk as a volatile* cache for RAM objects. (* Volatile > in the sense that when the program ends, the disk space is freed for > other use, and is inaccessible even if not.) Whereas I am thinking of > using RAM as a cache for a persistent disk object. A possible user API > (assuming txt stored as a python string with k bytes per char): > > cons = str(None, path="c:/user/terry/gutenburg/us_constitution.txt") > # now to slice like it was in ram > preamble = cons[:cons.find(section_marker)] > > Perhaps you are pointing to being able to make this possible, from the > implementer side. > > The generic interfaces would be bytes(None, path=) (read only) and > bytearray(None, path=) (read-write). This is already possible by using the pynvm bindings to nvml and memoryviews, but it would indeed be interesting to provide a more convenient API, and we've discussed this a bit. There wouldn't be much motivation for any changes to python for that level of support, though, since it could be provided by a couple of new bytes-like classes defined in an external module. > A list does not seem like a good candidate for static mem, unless insert > and delete are suppressed/unused. Why not? The point isn't that the memory is *static*, it's that it is *persistent*. So whatever state your objects are in when your program ends, that's the state they are in when you next connect your program to that pool of objects. It's perfectly sensible to want to update a list and have your changes persist, that's why the ZODB, for example, provides a PersistentList class. > If static objects were **always** aligned in 4-byte boundaries, then the > lowest 2 bits could be used to indicate memory type. To not slow down > programs, this should be supported by the CPU address decoder. Isn't > Intel thinking/working on something like this? That's an interesting thought, thanks :) I'm not clear how the CPU address decoder would support Python in this context, but I'll ask the Intel folks if there's anything relevant. --David From brett at python.org Wed May 18 13:31:44 2016 From: brett at python.org (Brett Cannon) Date: Wed, 18 May 2016 17:31:44 +0000 Subject: [Python-ideas] Add contextlib.DummyContext In-Reply-To: <573C8604.9090808@stoneleaf.us> References: <573C8604.9090808@stoneleaf.us> Message-ID: On Wed, May 18, 2016, 10:12 Ethan Furman wrote: > On 05/18/2016 06:16 AM, Giampaolo Rodola' wrote: > > On Wed, May 18, 2016 at 1:36 AM, Devin Jeanpierre wrote: > > >> with contextlib.ExitStack() as exit_stack: > >> if not bypass_lock: > >> exit_stack.enter_context(RedisLock()) > >> > >> And similar. > >> > >> In fact, ExitStack can itself be used as a no-op context manager, if > >> you want. > > > > I didn't think about this. So yeah, my proposal is useless. Nevermind. > > Not at all. It's a good use-case, and it helped introduce/remind us > about ExitStack (which I suspect is a hidden gem). > > So thank you all! :) > Python 3.6 also has a context manager ABC so you could also use that in 2 lines for a dummy context manager. -brett > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed May 18 13:41:42 2016 From: brett at python.org (Brett Cannon) Date: Wed, 18 May 2016 17:41:42 +0000 Subject: [Python-ideas] Lib vs ( App | Project | ... ) In-Reply-To: <573C0DF3.1070000@thomas-guettler.de> References: <573AFBD5.6010902@thomas-guettler.de> <573C0DF3.1070000@thomas-guettler.de> Message-ID: On Wed, May 18, 2016, 01:39 Thomas G?ttler wrote: > > > Am 17.05.2016 um 15:20 schrieb Nick Coghlan: > > On 17 May 2016 at 21:09, Thomas G?ttler > wrote: > >> This blog post from Donald Stufft explains the difference between > >> a Library and a Application. > >> > >> https://caremad.io/2013/07/setup-vs-requirement/ > >> > >> > >> The term "Library" is well known - ok > >> > >> But how to call the container for the libraries, the one thing > providing the > >> environment? > >> > >> Donald Stufft uses the term "Application" > >> > >> I would like an official term defined by the python docs. > > > > Covering general principles of software engineering and the many and > > varied terms used for different kinds of software aggregation is *way* > > outside the scope of the Python language and standard library docs :) > > AFAIK there can only be **one** sitecustomize.py. I think the docs > should be able to have a term for the thing which contains/provides this > file. > The current vacuum gets filled with a lot of different terms meaning the > same > thing. This does not hurt, but still I **feel** it would help to have an > agreement on a term. > > > > The closest we get is the PyPA/distutils-sig consensus that software > > is developed by and as projects (covering libraries, applications, and > > assorted other endeavours): > > https://packaging.python.org/en/latest/glossary/#term-project > > Thank you Nick, for providing this link. I hope your are not the author > of this text. That's a rather negative statement to make. Someone took the time to write that glossary entry and their work should be appreciated, even if you disagree with whether it met your expectations for a definition. If you would like to clarify it then feel free to submit a PR to change it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Wed May 18 13:44:12 2016 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 18 May 2016 13:44:12 -0400 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: <573BF1EB.9030303@stoneleaf.us> References: <20160517003511.AD7C9B14094@webabinitio.net> <573BF1EB.9030303@stoneleaf.us> Message-ID: <20160518174416.6D39EB1400B@webabinitio.net> On Tue, 17 May 2016 21:39:07 -0700, Ethan Furman wrote: > On 05/16/2016 05:35 PM, R. David Murray wrote: > > > I'm currently working on a project for Intel involving Python and directly > > addressable non-volatile memory. > > Sounds cool. The first thing it reminded me of was database transactions. There are related concerns. For nvml, we are concerned about the A and the D of ACID, but not the C or the I. (There was also some discussion of a cloud-enabled ACID system using the equivalent of software transactional memory, but that's not what nvml is focused on.) To be clear about what I mean by Isolation not being involved: because DAX makes VMRAM behave like RAM (except persistent), when you update the data at a memory address in NVRAM, it is immediately visible to all other threads. Consistency in ACID terms is not something nvml is addressing, that's more of an application level property. So the guarantees NVML is providing are only about the Atomicity and Durability. > What are the expected speed comparisons? I would imagine that > persistent RAM is going to be slower. If I understand correctly the RAM itself is not *necessarily* slower. However, providing the crash-proof atomicity requires bookkeeping overhead that will slow down operations that need to be protected by a transaction (which will be lots of things but by no means everything) compared to DRAM access where you don't care about the atomicity in the face of crashes. The goal is obviously to keep the overhead as small as possible :) > How much would a typical configuration have? Enough for all a program's > data? Yes. I'm sure systems will start with "smaller" amounts of DAX NVRAM initially, and more and more will become common as costs drop. But if I understand correctly we're already talking about significant amounts. In fact, I played around with the idea of just pointing python3's memory allocation at nvram and running it, but that does not get you the transaction level support that would allow crash recovery. > What are the expected use-cases? For example, loop variables might not > be a prime target for being stored in persistent RAM. Probably not, but you might be surprised. The main target would of course be the data the program wants to preserve between runs, but one of the examples at pmem.io is a game program whose state is all in nvram. If the machine crashes in the middle of the game, upon restart you pick up exactly where you left off, including all the positioning information you might consider to be part of the "hot loop" part of the program. > What are the advantages of using this persistent RAM directly as > variables vs. keeping the access as it's own step via the driver layer? Speed. --David From rdmurray at bitdance.com Wed May 18 14:08:55 2016 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 18 May 2016 14:08:55 -0400 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: References: <20160517003511.AD7C9B14094@webabinitio.net> Message-ID: <20160518180856.D77AAB1401C@webabinitio.net> On Wed, 18 May 2016 16:30:43 +1000, Nick Coghlan wrote: > However, it does suggest "thread local" as an allocator type worth > considering along with "normal process heap" and "non-volatile > storage". Interesting thought. > That leaves the C++ custom object allocator model, and if I recall > correctly, allocators there are managed at the *type* level - if you > want to use a different allocator, you need to declare a new type. If I understand correctly (my C++ skills are almost non-existent), the nvml C++ support is using operator overloading on a custom pointer type. I don't think that's enough by itself, though; the code is still in progress. > For Python, I'd suggest looking at going one step further, and > associating storage management with the *metaclass* - the idea there > would be to take advantage of the metaclass conflict if someone > attempts to combine data types with conflicting storage management > expectations. So you might have "PersistentType", which would itself > be a normal type instance (stored in RAM), and the classes it created > would also be stored in RAM, but they'd be configured to handle the > storage for individual instances differently from normal classes. I was planning on a base class that implemented checks, but using a meta class is a good idea. > I'm not sure if the current descriptor protocol and the GC management > machinery would give you sufficient access to control all the things > you care about for this purpose, but it should let you control quite a > lot of it, and trying it out would give you more info on what's > currently missing. I'll give that a whirl and see what happens; thanks for the idea. But no matter what I do, I'm going to have to write a GC related cleanup pass when recovering from a crash, and Persistent versions of all the built-in classes. --David From barry at python.org Wed May 18 15:21:13 2016 From: barry at python.org (Barry Warsaw) Date: Wed, 18 May 2016 15:21:13 -0400 Subject: [Python-ideas] Add contextlib.DummyContext References: <573C8604.9090808@stoneleaf.us> Message-ID: <20160518152113.216d474a@anarchist.wooz.org> On May 18, 2016, at 08:11 AM, Ethan Furman wrote: >Not at all. It's a good use-case, and it helped introduce/remind us about >ExitStack (which I suspect is a hidden gem). A beautiful one. http://www.wefearchange.org/2013/05/resource-management-in-python-33-or.html Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From brett at python.org Wed May 18 17:39:34 2016 From: brett at python.org (Brett Cannon) Date: Wed, 18 May 2016 21:39:34 +0000 Subject: [Python-ideas] Adding a frame evaluation API to CPython In-Reply-To: References: Message-ID: On Tue, 17 May 2016 at 11:52 Victor Stinner wrote: > Hi, > > Nice PEP. > > I would be nice to see how Yury Sulivanov's "implement per-opcode > cache in ceval" patch would benefit from your PEP: > Or how we might benefit from his work. :) There's a possibility the JIT could pull data out of what he caches for even better perf. And Yury could actually have written his patch as a PoC using our proposed PEP if he wanted to (same goes for anyone who wants to do eval loop experiments like Falcon: https://github.com/rjpower/falcon). -Brett > > * https://bugs.python.org/issue26219 > * https://mail.python.org/pipermail/python-dev/2016-February/143025.html > > This patch depends on the "Speedup method calls 1.2x" patch: > > * https://bugs.python.org/issue26110 > * https://mail.python.org/pipermail/python-dev/2016-January/142945.html > > Victor > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed May 18 18:41:30 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 19 May 2016 10:41:30 +1200 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> <573A590E.30708@canterbury.ac.nz> <573AACBE.6070002@canterbury.ac.nz> <573AE682.3040806@canterbury.ac.nz> Message-ID: <573CEF9A.5080609@canterbury.ac.nz> Guido van Rossum wrote: > your original remark claimed > something wasn't syntactic sugar because of the difference between the > box and its contents, and that's what I disagree with. Maybe I misunderstood -- you seemed to be saying that algebraic types were just syntactic sugar for something. Perhaps I should have asked what you thought they were syntactic sugar *for*? I was also responding to a comment that values in Python are already tagged with their type, so tagged unions are unnecessary. But the type tag of a Python object is not equivalent to the tag of an algebraic type, because the latter conveys information over and above the type of its payload. -- Greg From greg.ewing at canterbury.ac.nz Wed May 18 18:41:37 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 19 May 2016 10:41:37 +1200 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: References: <20160517003511.AD7C9B14094@webabinitio.net> Message-ID: <573CEFA1.7070406@canterbury.ac.nz> Terry Reedy wrote: > I have long thought that I should be able to randomly access data on > disk the same way I would access that same data in RAM, and not have to > fiddle with seek and so on. I gather Multics was like that -- the only way to access a file was to map it into memory, like mmap() in unix. I believe that some unix kernels also work that way internally, with the file I/O layer being built on top of the VM layer. I can see some problems with trying to build that sort of thing too deeply into Python, though. One of them is what happens when two processes map in the same file of Python objects. Some kind of super-GIL that works between processes would be needed to manage the reference counts. Another thing is how to specify that particular objects should be allocated in particular memory areas. That could become quite tricky and tedious with object allocation being both very frequent and very implicit. Another other thing is how to handle references that cross different memory areas. Are they allowed? If so, how do you manage them? If not, how do you detect These problems are of course much reduced if you only allow a restricted set of types to live in one of these areas, such as only atomic types, and lists, arrays, etc. of them. But then it feels more like an I/O storage system like pickle or shelve than it does a VM system. -- Greg From robertc at robertcollins.net Wed May 18 18:52:51 2016 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 19 May 2016 10:52:51 +1200 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: <573CEFA1.7070406@canterbury.ac.nz> References: <20160517003511.AD7C9B14094@webabinitio.net> <573CEFA1.7070406@canterbury.ac.nz> Message-ID: On 19 May 2016 at 10:41, Greg Ewing wrote: > Terry Reedy wrote: >> >> I have long thought that I should be able to randomly access data on disk >> the same way I would access that same data in RAM, and not have to fiddle >> with seek and so on. > > > I gather Multics was like that -- the only way to access a > file was to map it into memory, like mmap() in unix. > > I believe that some unix kernels also work that way > internally, with the file I/O layer being built on > top of the VM layer. > > I can see some problems with trying to build that sort > of thing too deeply into Python, though. One of them > is what happens when two processes map in the same file > of Python objects. Some kind of super-GIL that works > between processes would be needed to manage the > reference counts. Another consideration is how to deal with processes that crash (or machines that crash). With transient memory, reference counts are always correct whenever the GIL is released...or the process has just segfaulted/kill -9/whatever. If reference counts are stored in NVRAM, then when a process fails while while it holds a reference from its transient memory to the NVRAM stored object, those references will leak. For me at least, I'd want to treat NVRAM as a form of disk, and not try to use transient ram idioms with it - they seem unsafe. -Rob From guido at python.org Wed May 18 18:53:25 2016 From: guido at python.org (Guido van Rossum) Date: Wed, 18 May 2016 15:53:25 -0700 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: <573CEF9A.5080609@canterbury.ac.nz> References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> <573A590E.30708@canterbury.ac.nz> <573AACBE.6070002@canterbury.ac.nz> <573AE682.3040806@canterbury.ac.nz> <573CEF9A.5080609@canterbury.ac.nz> Message-ID: On Wed, May 18, 2016 at 3:41 PM, Greg Ewing wrote: > Guido van Rossum wrote: >> >> your original remark claimed >> something wasn't syntactic sugar because of the difference between the >> box and its contents, and that's what I disagree with. > > Maybe I misunderstood -- you seemed to be saying that > algebraic types were just syntactic sugar for something. > Perhaps I should have asked what you thought they were > syntactic sugar *for*? That's a good question, for which I don't have a great answer. I've just discussed some of this with the author of that blog article (Chad Austin) and we came to the conclusion that there isn't a great future for ADTs or Sum types in Python because almost everything you can do with them already has a way to do it in Python that's good enough. Sometimes what you need is Haskell's Maybe (i.e. 'Nothing' or 'Just x'), and in most cases just checking for None is good enough. The strict optional checking that's coming to mypy soon will help here because it'll catch you when you're not checking for None before doing something else with the value. Sometimes you really do want to distinguish between the box and what's in it, and then you can use a bunch of simple classes, or maybe a bunch of namedtuples, to represent the different kinds of boxes. If you declare the unopened box type as a Union you can get mypy to check that you've covered all your bases. (If you don't want/need a check that you're handling all cases you can use a shared superclass instead of a union.) There are probably a few other cases. The one thing that Python doesn't have (and mypy doesn't add) would be a match statement. The design of a Pythonic match statement would be an interesting exercise; perhaps we should see how far we can get with that for Python 3.7. > I was also responding to a comment that values in > Python are already tagged with their type, so tagged > unions are unnecessary. But the type tag of a Python > object is not equivalent to the tag of an algebraic > type, because the latter conveys information over and > above the type of its payload. Right. And if you need that feature you can wrap it in a class or namedtuple. -- --Guido van Rossum (python.org/~guido) From tjreedy at udel.edu Wed May 18 21:30:41 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 18 May 2016 21:30:41 -0400 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: <20160518171122.BB33BB1401C@webabinitio.net> References: <20160517003511.AD7C9B14094@webabinitio.net> <20160518171122.BB33BB1401C@webabinitio.net> Message-ID: On 5/18/2016 1:11 PM, R. David Murray wrote: > On Tue, 17 May 2016 17:07:59 -0400, Terry Reedy wrote: >> How is non-volatile NVRAM different from static SRAM? > NVRAM retains the data even if it loses power. OK, I mis-remembered the battery-backup part, and was thinking of flash memory at 'static' rather than 'non-volatile'. A third term really is needed. >> A list does not seem like a good candidate for static mem, unless insert >> and delete are suppressed/unused. > > Why not? The point isn't that the memory is *static*, it's that it is > *persistent*. Because each is an O(n) operation. I was thinking of non-volatile writes as been relatively slow and in some sense more costly, but maybe that is not necessarily the case for new designs. -- Terry Jan Reedy From ncoghlan at gmail.com Wed May 18 23:06:14 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 19 May 2016 13:06:14 +1000 Subject: [Python-ideas] Lib vs ( App | Project | ... ) In-Reply-To: <573C0DF3.1070000@thomas-guettler.de> References: <573AFBD5.6010902@thomas-guettler.de> <573C0DF3.1070000@thomas-guettler.de> Message-ID: On 18 May 2016 at 16:38, Thomas G?ttler wrote: > Am 17.05.2016 um 15:20 schrieb Nick Coghlan: >> Covering general principles of software engineering and the many and >> varied terms used for different kinds of software aggregation is *way* >> outside the scope of the Python language and standard library docs :) > > AFAIK there can only be **one** sitecustomize.py. I think the docs > should be able to have a term for the thing which contains/provides this > file. `sitecustomize.py`, if provided at all, tends to be scoped to the Python installation. Since most (all?) installers don't provide it by default, it typically shows up as part of an organisation specific Standard Operating Environment, although you can technically add it to any virtual environment. > The current vacuum gets filled with a lot of different terms meaning the > same > thing. This does not hurt, but still I **feel** it would help to have an > agreement on a term. Agreement on terminology for pre-existing concepts can't be forced, only observed as an emergent phenomenon (one of the most prominent Python-specific examples of that being the multi-year effort to get people to stop using "package" for both import packages and distribution packages, before we finally conceded defeat and documented the ambiguity, rather than continuing to try to eliminate it) Real world categories are fuzzy, so the apparent precision of software development starts to break down once we start talking about the higher level assemblies that blur the lines between the software being developed and the collections of humans doing the development. (For one of the classic cognitive science examples of category fuzziness, first ask "What is a chair?". Then, whatever definition people give, present counter-examples that challenge their definition: "Is an overturned milk crate a chair?", "Is a cut log a chair?", "Is a foot stool a chair?", "Is a park bench a chair?", "What about rocking chairs?", "What about armchairs?", "What about desk chairs?", "What about folding chairs?"). >> The closest we get is the PyPA/distutils-sig consensus that software >> is developed by and as projects (covering libraries, applications, and >> assorted other endeavours): >> https://packaging.python.org/en/latest/glossary/#term-project > > Thank you Nick, for providing this link. I hope your are not the author > of this text. This is no definition according to my point of view. > I read it, and don't get it. Example: I think "django" is a > project according to this definition. Yes, Django is a project. > But I search a term > for the thing containing: > > settings.py > sitecustomize.py > logging-config, .... These are also projects: https://docs.djangoproject.com/en/1.9/ref/django-admin/#startproject Whether a given Django project is also a project in the PyPA sense will depend on how it is published. This is why Sphinx has glossary support - so projects can clearly define otherwise ambiguous terms in *their particular context*. >> The install_requires vs requirements.txt question is covered as its >> own topic, without introducing any particular terminology: >> https://packaging.python.org/en/latest/requirements/ > > It is hard to cover the meaning of requirements.txt if there > is no common term to name the container which contains it :-) The syntactic meaning of requirements.txt is straightforward: it's a series of installation commands for pip. The semantic meaning is context dependent - some projects will use it to express approximate runtime requirements (more like the recommended way of using install_requires in setup.py), while others will use it as a complete environment specification (e.g. by using a tool like pip-compile). > The above link calls it "complete python environment". > > Up to now I found these terms: > > - Project > https://docs.djangoproject.com/en/1.8/ref/applications/#projects-and-applications > - Application https://caremad.io/2013/07/setup-vs-requirement/ > - site https://docs.python.org/3/library/site.html > - complete python environment > https://packaging.python.org/en/latest/requirements/ > > I guess there are more. "Standards are great: Everyone should have one" > > Is this the way you want it to be? What I want has nothing to do with it - the emergence of ambiguity that requires context-specific clarification of terms is just a raw fact about human behaviour in the absence of centralised authoritarian control. > I see no agreement on the name for the concrete installation holding all > those nice python libraries. Correct, because there isn't one. Where the difference in perspective arises is that you appear to believe that python-dev has the collective power to define one, but we don't. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed May 18 23:17:14 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 19 May 2016 13:17:14 +1000 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> <573A590E.30708@canterbury.ac.nz> <573AACBE.6070002@canterbury.ac.nz> <573AE682.3040806@canterbury.ac.nz> <573CEF9A.5080609@canterbury.ac.nz> Message-ID: On 19 May 2016 at 08:53, Guido van Rossum wrote: > The one thing that Python doesn't have (and mypy doesn't add) would be > a match statement. The design of a Pythonic match statement would be > an interesting exercise; perhaps we should see how far we can get with > that for Python 3.7. If it's syntactic sugar for a particular variety of if/elif/else statement, I think that may be feasible, but you'd presumably want to avoid the "Can we precompute a lookup table?" quagmire that doomed PEP 3103's switch statement. That said, for the pre-computed lookup table case, whatever signature deconstruction design you came up with for a match statement might also be usable as the basis for a functools.multidispatch() decorator design. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Thu May 19 00:15:06 2016 From: guido at python.org (Guido van Rossum) Date: Wed, 18 May 2016 21:15:06 -0700 Subject: [Python-ideas] Match statement brainstorm Message-ID: On Wed, May 18, 2016 at 8:17 PM, Nick Coghlan wrote: > On 19 May 2016 at 08:53, Guido van Rossum wrote: >> The one thing that Python doesn't have (and mypy doesn't add) would be >> a match statement. The design of a Pythonic match statement would be >> an interesting exercise; perhaps we should see how far we can get with >> that for Python 3.7. > > If it's syntactic sugar for a particular variety of if/elif/else > statement, I think that may be feasible, but you'd presumably want to > avoid the "Can we precompute a lookup table?" quagmire that doomed PEP > 3103's switch statement. > > That said, for the pre-computed lookup table case, whatever signature > deconstruction design you came up with for a match statement might > also be usable as the basis for a functools.multidispatch() decorator > design. Let's give up on the pre-computed lookup table. Neither PEP 3103 nor PEP 275 (to which it compares itself) even gets into the unpacking part, which would be the interesting thing from the perspective of learning from Sum types and matching in other languages. Agreed on the idea of trying to reuse this for multidispatch! A few things that might be interesting to explore: - match by value or set of values (like those PEPs) - match by type (isinstance() checks) - match on tuple structure, including nesting and * unpacking (essentially, try a series of destructuring assignments until one works) - match on dict structure? (extension of destructuring to dicts) - match on instance variables or attributes by name? - match on generalized condition (predicate)? The idea is that many of these by themselves are better off using a classic if/elif/else structure, but a powerful matching should be allowed to alternate between e.g. destructuring matches and value or predicate matches. IIUC Haskell allows pattern matches as well as conditions, which they seem to call guards or where-clauses (see https://www.haskell.org/tutorial/patterns.html, or http://learnyouahaskell.com/syntax-in-functions if you like your pages more colorful). Or maybe we should be able to combine structure matches with guards. I guess the tuple structure matching idea is fairly easy to grasp. The attribute idea would be similar to a duck-type check, though more emphasizing data attributes. It would be nice if we could write a match that said "if it has attributes x and y, assign those to local variables p and q, and ignore other attributes". Strawman syntax could be like this: def demo(arg): switch arg: case (x=p, y=q): print('x=', p, 'y=', q) case (a, b, *_): print('a=', a, 'b=', b) else: print('Too bad') Now suppose we had a Point defined like this: Point = namedtuple('Point', 'x y z') and some variables like this: a = Point(x=1, y=2, z=3) b = (1, 2, 3, 4) c = 'hola' d = 42 then we could call demo with these variables: >>> demo(a) x= 1 y= 2 >>> demo(b) a= 1 b= 2 >>> demo(c) a= h b= o >>> demo(d) Too bad >>> (Note the slightly unfortunate outcome for 'hola', but that's what a destructuring assignment would do. Water under the bridge.) Someone else can try to fit simple value equality, set membership, isinstance, and guards into that same syntax. -- --Guido van Rossum (python.org/~guido) From stephen at xemacs.org Thu May 19 01:02:34 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 19 May 2016 14:02:34 +0900 Subject: [Python-ideas] Lib vs ( App | Project | ... ) In-Reply-To: <573C0DF3.1070000@thomas-guettler.de> References: <573AFBD5.6010902@thomas-guettler.de> <573C0DF3.1070000@thomas-guettler.de> Message-ID: <22333.18666.604476.908971@turnbull.sk.tsukuba.ac.jp> Thomas G?ttler writes: > I see no agreement on the name for the concrete installation > holding all those nice python libraries. I don't see how you could get it. The "concrete installation" you mention is apparently is defined by "holding all those nice python libraries" -- you make no mention of requirements.txt or any of the other "marker" files you've talked about. It's also unclear whether "concrete installation" includes things like the Python distribution itself (even the operating system), or perhaps things in site-packages but not the stdlib. And that is the problem. The variety of "collections of libraries" (some of which are singletons, others of which may be instantiated as empty!) and accompanying configuration (if any) that a developer might want denoted by the name you're looking for is pretty well infinite. It's not going to be the same as the distribution, since those often don't include configuration files (and does the "example.cfg" count?) In any case it's large enough that we're going to get substantial disagreement on the definition even before we try to choose a name. Steve From greg.ewing at canterbury.ac.nz Thu May 19 03:27:19 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 19 May 2016 19:27:19 +1200 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: References: <20160517003511.AD7C9B14094@webabinitio.net> <20160518171122.BB33BB1401C@webabinitio.net> Message-ID: <573D6AD7.4050309@canterbury.ac.nz> Terry Reedy wrote: > Because each is an O(n) operation. I was thinking of non-volatile > writes as been relatively slow and in some sense more costly, but maybe > that is not necessarily the case for new designs. Large flash memories are usually organised into blocks, like a disk. Writing a block is relatively slow, so you'd want to buffer up your changes and write as much as you can in one go. For Python objects, that would probably mean keeping a shadow copy in RAM and flushing it to NV storage periodically. Techniques similar to those used in relational databases might be needed to ensure that the contents of the storage remains consistent in the event of failures. -- Greg From greg.ewing at canterbury.ac.nz Thu May 19 04:20:04 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 19 May 2016 20:20:04 +1200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: Message-ID: <573D7734.20402@canterbury.ac.nz> Guido van Rossum wrote: > def demo(arg): > switch arg: > case (x=p, y=q): print('x=', p, 'y=', q) > case (a, b, *_): print('a=', a, 'b=', b) > else: print('Too bad') I would also add case Point(x=p, y=q): print('Point: x=', p, 'y=', q) Then if you also had Vector = namedtuple('x,y,z') then this case would match a Point but not a Vector. However, there's a problem with all this if you want to allow matching on specific values as well as structure. Haskell allows you to say things like myfunc [17, 42, x] = ... which will only match a 3-element list whose first 2 elements are 17 and 42. This would be fine in Python as long as the constants are literals. But if you want to name the constants, Seventeen = 17 FortyTwo = 42 case [Seventeen, FortyTwo, x]: it becomes ambiguous. Are the identifiers values to be matched or names to be bound? If we want this feature, it seems like we will need to explicitly mark either names to be bound or expressions to be evaluated and matched against. At first I thought maybe a name could be enclosed in parens to force it to be treated as an expression: case [(Seventeen), (FortyTwo), x]: but that wouldn't quite be consistent with assignment syntax, because (x) is actually valid on the left of an assignment and is equivalent to just x. >>>>demo(c) > > a= h b= o > > (Note the slightly unfortunate outcome for 'hola', If you wanted the second case to only match the tuple, you could write case tuple(a, b, *_): -- Greg From greg.ewing at canterbury.ac.nz Thu May 19 03:00:10 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 19 May 2016 19:00:10 +1200 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> <573A590E.30708@canterbury.ac.nz> <573AACBE.6070002@canterbury.ac.nz> <573AE682.3040806@canterbury.ac.nz> <573CEF9A.5080609@canterbury.ac.nz> Message-ID: <573D647A.8050105@canterbury.ac.nz> Guido van Rossum wrote: > Sometimes you really do want to distinguish between the box and what's > in it, and then you can use a bunch of simple classes, or maybe a > bunch of namedtuples, to represent the different kinds of boxes. If > you declare the unopened box type as a Union you can get mypy to check > that you've covered all your bases. Well, sort of. It will check that you don't pass anything outside of the set of allowed types, but it can't tell whether you've handled all the possible cases in the code. That's something Haskell can do because (1) it has special constructs for case analysis, and (2) its algebraic types can't be extended or subclassed. > The one thing that Python doesn't have (and mypy doesn't add) would be > a match statement. The design of a Pythonic match statement would be > an interesting exercise; Yes, if there were dedicated syntax for it, mypy could check that you covered all the branches of a Union. I've thought about this off and on a bit. Maybe I'll write about some ideas in another post. -- Greg From ppbbalcer at gmail.com Thu May 19 10:40:12 2016 From: ppbbalcer at gmail.com (Piotr Balcer) Date: Thu, 19 May 2016 07:40:12 -0700 (PDT) Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: <573D6AD7.4050309@canterbury.ac.nz> References: <20160517003511.AD7C9B14094@webabinitio.net> <20160518171122.BB33BB1401C@webabinitio.net> <573D6AD7.4050309@canterbury.ac.nz> Message-ID: <38b79838-2b9f-4299-8ce2-e468f7bccf02@googlegroups.com> In my infinite stupidity I've replied to this mailing group without properly subscribing. Sorry for the duplicates. Full disclosure: I work at Intel on the set of libraries mentioned above. Our project is fully hardware agnostic - any set of NVDIMMs is going to work. This is a new tier of memory between transient RAM and storage SSD's. It combines the features of both, NVDIMMs provide memory that is both byte addressable and persistent across power failures. The speed and other characterstics are obviously vendor dependent. The term I've seen some people use to describe this new memory is "storage class memory" and I think it really captures the intended use case. There are currently few NVDIMM providers - you might recall recent HP announcement about their new non-volatile memory enabled servers. Also a company called Plexistor is providing software-defined persistent memory solutions. As for Intel offering in this space: the 3D XPoint announcement happend few months back and it was widely covered by the media. Here are two nice presentations: http://www.snia.org/sites/default/files/NVM/2016/presentations/RickCoulson_All_the_Ways_3D_XPoint_Impacts.pdf http://www.snia.org/sites/default/files/SDC15_presentations/persistant_mem/JimHandy_Understanding_the-Intel.pdf If anyone finds this topic interesting I recommend checking out our website pmem.io and the going through the presentations presented at various SNIA organized NVM Summits: http://www.snia.org/events/non-volatile-memory-nvm-summit Also, the SNIA programming model is something else to look at after some initial reading: http://www.snia.org/tech_activities/standards/curr_standards/npm It's what the NVML is based on. Hope that this clarifies some things. Piotr W dniu czwartek, 19 maja 2016 09:28:07 UTC+2 u?ytkownik Greg Ewing napisa?: > > Terry Reedy wrote: > > Because each is an O(n) operation. I was thinking of non-volatile > > writes as been relatively slow and in some sense more costly, but maybe > > that is not necessarily the case for new designs. > > Large flash memories are usually organised into blocks, like > a disk. Writing a block is relatively slow, so you'd want to > buffer up your changes and write as much as you can in one go. > > For Python objects, that would probably mean keeping a > shadow copy in RAM and flushing it to NV storage periodically. > Techniques similar to those used in relational databases > might be needed to ensure that the contents of the storage > remains consistent in the event of failures. > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu May 19 10:46:23 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 19 May 2016 07:46:23 -0700 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: <573D647A.8050105@canterbury.ac.nz> References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> <573A590E.30708@canterbury.ac.nz> <573AACBE.6070002@canterbury.ac.nz> <573AE682.3040806@canterbury.ac.nz> <573CEF9A.5080609@canterbury.ac.nz> <573D647A.8050105@canterbury.ac.nz> Message-ID: On Thu, May 19, 2016 at 12:00 AM, Greg Ewing wrote: > Guido van Rossum wrote: >> >> Sometimes you really do want to distinguish between the box and what's >> in it, and then you can use a bunch of simple classes, or maybe a >> bunch of namedtuples, to represent the different kinds of boxes. If >> you declare the unopened box type as a Union you can get mypy to check >> that you've covered all your bases. > > > Well, sort of. It will check that you don't pass anything > outside of the set of allowed types, but it can't tell > whether you've handled all the possible cases in the > code. That's something Haskell can do because (1) it has > special constructs for case analysis, and (2) its > algebraic types can't be extended or subclassed. I'm still not sure I understand why this check that you've handled all cases is so important (I've met a few people who obsessed about it in other languages, but I don't really feel the need in my gut yet). I also don't think that subclasses cause problems (if there's a match for a particular class, it will match the subclass too). Anyway, I believe mypy could easily check that you're always returning a value in situations like this (even though it doesn't do that now): def foo(arg: Union[X, Y, Z]) -> int: if isinstance(arg, X): return arg.counter if isinstance(arg, Y): return 0 It should be possible to figure out that Z is missing here. >> The one thing that Python doesn't have (and mypy doesn't add) would be >> a match statement. The design of a Pythonic match statement would be >> an interesting exercise; > > > Yes, if there were dedicated syntax for it, mypy could > check that you covered all the branches of a Union. > > I've thought about this off and on a bit. Maybe I'll > write about some ideas in another post. Would love to hear your thoughts!. -- --Guido van Rossum (python.org/~guido) From ian.g.kelly at gmail.com Thu May 19 10:46:33 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 19 May 2016 08:46:33 -0600 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: Message-ID: On Wed, May 18, 2016 at 10:15 PM, Guido van Rossum wrote: > I guess the tuple structure matching idea is fairly easy to grasp. > > The attribute idea would be similar to a duck-type check, though more > emphasizing data attributes. It would be nice if we could write a > match that said "if it has attributes x and y, assign those to local > variables p and q, and ignore other attributes". Strawman syntax could > be like this: > > def demo(arg): > switch arg: > case (x=p, y=q): print('x=', p, 'y=', q) > case (a, b, *_): print('a=', a, 'b=', b) > else: print('Too bad') That the first case would match attributes and not named items seems surprising to me. That is, just by looking at it I would have expected it to match {'x': 1, 'y': 2}. This feels more in line with the way that keyword arguments work, and more consistent with the second case in that both would be matching contents of collections. From ppbbalcer at gmail.com Thu May 19 10:42:56 2016 From: ppbbalcer at gmail.com (Piotr Balcer) Date: Thu, 19 May 2016 07:42:56 -0700 (PDT) Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: <20160518180856.D77AAB1401C@webabinitio.net> References: <20160517003511.AD7C9B14094@webabinitio.net> <20160518180856.D77AAB1401C@webabinitio.net> Message-ID: To build on what David has said about the C++ - this in fact exactly what we've done for our language support :) Long story short, for C++ we introduce two new base types: a persistent smart pointer and a persistent memory resident variable. To complement that there's a support for transactions implemented in two ways: by providing a function to execute as transaction or by using RAII. Here's a very simple queue implemented using those features: https://github.com/pmem/nvml/tree/master/src/examples/libpmemobj/cpp But back on topic of custom allocators: we've prototyped a STL compatible allocator that allowed us to use the std::vector, std::list and other STL containers without any modifications by simply providing the allocator template argument. This was possible because the allocator provides the facility to provide a pointer type argument which is persistent_ptr in our case. Were the CPython implemented in C++ it would probably be possible to implement the collections in a way that creating a specialized ones would be very easy: class PersistentDictionary : Dictionary But I guess that's out of the question ;) I'm afraid I won't be much of a help for the python specific stuff but I'm happy to answer any questions about persistent memory programming in general. Piotr -------------- next part -------------- An HTML attachment was scrubbed... URL: From kulakov.ilya at gmail.com Thu May 19 14:49:52 2016 From: kulakov.ilya at gmail.com (Ilya Kulakov) Date: Thu, 19 May 2016 14:49:52 -0400 Subject: [Python-ideas] Add ability to get current event loop within asyncio coroutine Message-ID: <1115AEC1-29BD-4D89-A828-E3D57B711B74@gmail.com> asyncio is a great library which with recent changes in 3.5 was made even better. However there is an inconvenience that bothers me and looks very unnatural: inability to access event loop that executes a coroutine from within that coroutine. This is unnatural, because we have `self` and `cls` to access object from a method it is bound to, we have `current_thread` to get an instance of Thread that currently executes code, but for a coroutine we have no such method, we cannot get its context of execution. Current implementation of `get_event_loop` method is not sufficient, it will not work if thread has more than one event loop. I think Python would benefit from providing either a new method (e.g. `asyncio.current_event_loop`) or modifying `asyncio.get_event_loop` to return current event loop when called from a coroutine. It should, in my opinion, reduce necessity of passing event loop between coroutines in application's code as it is done in asyncio itself and 3rd party libraries (like aiohttp). Best Regards, Ilya Kulakov -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu May 19 15:09:58 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 19 May 2016 12:09:58 -0700 Subject: [Python-ideas] Add ability to get current event loop within asyncio coroutine In-Reply-To: <1115AEC1-29BD-4D89-A828-E3D57B711B74@gmail.com> References: <1115AEC1-29BD-4D89-A828-E3D57B711B74@gmail.com> Message-ID: Can you explain why asyncio.get_event_loop() doesn't do what you want? On Thu, May 19, 2016 at 11:49 AM, Ilya Kulakov wrote: > asyncio is a great library which with recent changes in 3.5 was made even > better. > > However there is an inconvenience that bothers me and looks very unnatural: > inability to access event loop > that executes a coroutine from within that coroutine. > > This is unnatural, because we have `self` and `cls` to access object from a > method it is bound to, > we have `current_thread` to get an instance of Thread that currently > executes code, but for a coroutine > we have no such method, we cannot get its context of execution. > Current implementation of `get_event_loop` method is not sufficient, it will > not work if thread has more > than one event loop. > > I think Python would benefit from providing either a new method (e.g. > `asyncio.current_event_loop`) or modifying `asyncio.get_event_loop` > to return current event loop when called from a coroutine. It should, in my > opinion, reduce necessity of passing event loop > between coroutines in application's code as it is done in asyncio itself and > 3rd party libraries (like aiohttp). > > > Best Regards, > Ilya Kulakov > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From ian.g.kelly at gmail.com Thu May 19 15:58:18 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 19 May 2016 13:58:18 -0600 Subject: [Python-ideas] Add ability to get current event loop within asyncio coroutine In-Reply-To: <1115AEC1-29BD-4D89-A828-E3D57B711B74@gmail.com> References: <1115AEC1-29BD-4D89-A828-E3D57B711B74@gmail.com> Message-ID: On Thu, May 19, 2016 at 12:49 PM, Ilya Kulakov wrote: > Current implementation of `get_event_loop` method is not sufficient, it will > not work if thread has more > than one event loop. Why do you need more than one event loop per thread? You couldn't run more than one of them at a time. > I think Python would benefit from providing either a new method (e.g. > `asyncio.current_event_loop`) or modifying `asyncio.get_event_loop` > to return current event loop when called from a coroutine. It should, in my > opinion, reduce necessity of passing event loop > between coroutines in application's code as it is done in asyncio itself and > 3rd party libraries (like aiohttp). To me this sounds like a case for a custom event loop policy, which is what determines the behavior of get_event_loop. From srkunze at mail.de Thu May 19 18:34:31 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 20 May 2016 00:34:31 +0200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: Message-ID: <573E3F77.3020907@mail.de> On 19.05.2016 16:46, Ian Kelly wrote: > On Wed, May 18, 2016 at 10:15 PM, Guido van Rossum wrote: >> I guess the tuple structure matching idea is fairly easy to grasp. That's true. So, I would love to have more capabilities baked into tuple structure matching. >> The attribute idea would be similar to a duck-type check, though more >> emphasizing data attributes. It would be nice if we could write a >> match that said "if it has attributes x and y, assign those to local >> variables p and q, and ignore other attributes". Strawman syntax could >> be like this: >> >> def demo(arg): >> switch arg: >> case (x=p, y=q): print('x=', p, 'y=', q) >> case (a, b, *_): print('a=', a, 'b=', b) >> else: print('Too bad') However, combining this (the tuple structure matching) with a switch-case statement is quite some change to my taste especially regarding Python's history with the switch-case statement. Maybe, that is not necessary and we can introduce more features to tuple structure matching like Erlang has without introducing switch-case. I think it's more powerful and flexible on its own. > That the first case would match attributes and not named items seems > surprising to me. That is, just by looking at it I would have expected > it to match {'x': 1, 'y': 2}. This feels more in line with the way > that keyword arguments work, and more consistent with the second case > in that both would be matching contents of collections. Maybe, here the .x and .y syntax, which has been proposed previously (another with-statement enhancement proposal), could come in handy. This would definitely refer to attribute matching. Matching via quotation marks (" or ') would rather hint at dictionary keys. Best, Sven From greg.ewing at canterbury.ac.nz Thu May 19 19:33:06 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 20 May 2016 11:33:06 +1200 Subject: [Python-ideas] "Sum" Type hinting [was: Type hinting for path-related functions] In-Reply-To: References: <22326.46995.258956.416661@turnbull.sk.tsukuba.ac.jp> <5739084A.6040708@canterbury.ac.nz> <573A590E.30708@canterbury.ac.nz> <573AACBE.6070002@canterbury.ac.nz> <573AE682.3040806@canterbury.ac.nz> <573CEF9A.5080609@canterbury.ac.nz> <573D647A.8050105@canterbury.ac.nz> Message-ID: <573E4D32.2060500@canterbury.ac.nz> Guido van Rossum wrote: > I'm still not sure I understand why this check that you've handled all > cases is so important (I've met a few people who obsessed about it in > other languages, but I don't really feel the need in my gut yet). If you're programming in the usual OO style, then when you add a new subclass, most of the code needed to support it goes into that class. You can easily go through all the methods of the base class and make sure you've overridden the ones you need to. But if you add a new branch to an algebraic type, you need to chase down all the pieces of code scattered about your program that operate on that type and update them. If you're happy to rely on testing to do so, that's fine. But if you're a static checking kind of person, I can see the attraction of having some help from your tools for it. > I also don't think that subclasses cause problems (if there's a match > for a particular class, it will match the subclass too). It's not subclassing an existing branch that's the problem, it's adding a new branch to the union. -- Greg From ncoghlan at gmail.com Thu May 19 22:37:35 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 20 May 2016 12:37:35 +1000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: Message-ID: On 19 May 2016 at 14:15, Guido van Rossum wrote: > The attribute idea would be similar to a duck-type check, though more > emphasizing data attributes. It would be nice if we could write a > match that said "if it has attributes x and y, assign those to local > variables p and q, and ignore other attributes". Strawman syntax could > be like this: > > def demo(arg): > switch arg: > case (x=p, y=q): print('x=', p, 'y=', q) > case (a, b, *_): print('a=', a, 'b=', b) > else: print('Too bad') For the destructuring assignment by attribute, I'd suggest the "value as name" idiom, since it's not quite a normal assignment, as well as a leading "." to more readily differentiate it from iterable unpacking: def demo(arg): switch arg: case (.x as p, .y as q): print('x=', p, 'y=', q) case (a, b, *_): print('a=', a, 'b=', b) else: print('Too bad') Whether to allow ".x" as a shorthand for ".x as x" would be an open question. > Someone else can try to fit simple value equality, set membership, > isinstance, and guards into that same syntax. For these, I'd guess the most flexible option would be to allow the switch expression to be bound to a name: switch expr as arg: case arg == value: ... case lower_bound <= arg <= upper_bound: ... case arg in container: ... Similar to with statement and for loops, this wouldn't create a new scope, it would just bind the name in the current scope (and hence the value would remain available after the switch statement ends) If we went down that path, then the "assign if you can, execute this case if you succeed" options would presumably need an explicit prefix to indicate they're not normal expressions, perhaps something like "?=": switch expr as arg: case ?= (.x as p, .y as q): print('x=', p, 'y=', q) case ?= (a, b, *_): print('a=', a, 'b=', b) case arg == value: ... case lower_bound <= arg <= upper_bound: ... case arg in container: ... else: print('Too bad') Which would then have the further implication that it might also make sense to support attribute unpacking as the LHS of normal assignment statements: (.x as p, .y as q) = expr In a similar vein, item unpacking might look like: (["x"] as p, ["y"] as q) = expr And unpacking different structures might look like: switch expr: case ?= (.x as x, .y as y): ... # x/y as attributes case ?= (["x"] as x, ["y"] as y): ... # x/y as mapping case ?= (x, y): ... # 2-tuple (or other iterable) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From zauddelig at gmail.com Fri May 20 04:02:37 2016 From: zauddelig at gmail.com (Fabrizio Messina) Date: Fri, 20 May 2016 10:02:37 +0200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: Message-ID: why the unpacking has to happen automatically? I would prefer something like: switch *args: case:... Also I'm not a big fan of adding two new keywords to the syntax, I would think something like: switch *args: &(a==1, ...) as a, *b: # code &(a,b) as a, b: # code &(a,b, ...) as a, b, *c: # code &(...) as a: # code This would reduce the numbers of new keywords needed to 1, it would make sense to use the & operator because all the conditions have to be TRUE and this use at the moment raises SyntaxError. For how I see it could also make sense to be able to pass the arguments to a callable. switch *args: &(a==1, ...): (lambda a, *b: ...) &(a,b): (lambda a, b: [a, b]) &(a,b, ...): (lambda a, b, *c: [ a+1, b+1, *c]) &(...) as a: (lambda *a: [*a]) On Fri, May 20, 2016 at 4:37 AM, Nick Coghlan wrote: > On 19 May 2016 at 14:15, Guido van Rossum wrote: > > The attribute idea would be similar to a duck-type check, though more > > emphasizing data attributes. It would be nice if we could write a > > match that said "if it has attributes x and y, assign those to local > > variables p and q, and ignore other attributes". Strawman syntax could > > be like this: > > > > def demo(arg): > > switch arg: > > case (x=p, y=q): print('x=', p, 'y=', q) > > case (a, b, *_): print('a=', a, 'b=', b) > > else: print('Too bad') > > For the destructuring assignment by attribute, I'd suggest the "value > as name" idiom, since it's not quite a normal assignment, as well as a > leading "." to more readily differentiate it from iterable unpacking: > > def demo(arg): > switch arg: > case (.x as p, .y as q): print('x=', p, 'y=', q) > case (a, b, *_): print('a=', a, 'b=', b) > else: print('Too bad') > > Whether to allow ".x" as a shorthand for ".x as x" would be an open > question. > > > Someone else can try to fit simple value equality, set membership, > > isinstance, and guards into that same syntax. > > For these, I'd guess the most flexible option would be to allow the > switch expression to be bound to a name: > > switch expr as arg: > case arg == value: ... > case lower_bound <= arg <= upper_bound: ... > case arg in container: ... > > Similar to with statement and for loops, this wouldn't create a new > scope, it would just bind the name in the current scope (and hence the > value would remain available after the switch statement ends) > > If we went down that path, then the "assign if you can, execute this > case if you succeed" options would presumably need an explicit prefix > to indicate they're not normal expressions, perhaps something like > "?=": > > switch expr as arg: > case ?= (.x as p, .y as q): print('x=', p, 'y=', q) > case ?= (a, b, *_): print('a=', a, 'b=', b) > case arg == value: ... > case lower_bound <= arg <= upper_bound: ... > case arg in container: ... > else: print('Too bad') > > Which would then have the further implication that it might also make > sense to support attribute unpacking as the LHS of normal assignment > statements: > > (.x as p, .y as q) = expr > > In a similar vein, item unpacking might look like: > > (["x"] as p, ["y"] as q) = expr > > And unpacking different structures might look like: > > switch expr: > case ?= (.x as x, .y as y): ... # x/y as attributes > case ?= (["x"] as x, ["y"] as y): ... # x/y as mapping > case ?= (x, y): ... # 2-tuple (or other iterable) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vxgmichel at gmail.com Fri May 20 05:02:38 2016 From: vxgmichel at gmail.com (Vincent Michel) Date: Fri, 20 May 2016 11:02:38 +0200 Subject: [Python-ideas] Add ability to get current event loop within asyncio coroutine In-Reply-To: References: Message-ID: <573ED2AE.9080101@gmail.com> On Thu, May 19, 2016 at 11:49 AM, Ilya Kulakov wrote: > I think Python would benefit from providing either a new method (e.g. > `asyncio.current_event_loop`) or modifying `asyncio.get_event_loop` > to return current event loop when called from a coroutine. It should, in my > opinion, reduce necessity of passing event loop > between coroutines in application's code as it is done in asyncio itself and > 3rd party libraries (like aiohttp). I was about to make the same proposal. I can add two practical arguments in favor. First, some coroutines could be simplified. For instance, asyncio.sleep takes an optional loop argument, defaulting to get_event_loop(). But get_event_loop is called inside the coroutine, and therefore inside a running event loop. This loop is actually the only valid choice, and using anything else should raise an exception. If get_event_loop is guaranteed to return the current running loop, then there is no reason to pass it as an argument to asyncio.sleep. Other coroutines could benefit from that, including wait and wait_for. It would also help for queues, locks and other task functions (gather, shield, as_completed, etc.). Here the loop argument is still useful, in case those objects are declared outside their event loop. However, it would be really convenient to assume loop=None always works as expected if the object is created inside a coroutine. In many cases, libraries tend to forward the loop argument from coroutine to coroutine, just in case someone writes: loop = asyncio.new_event_loop() loop.run_until_complete(my_coro(loop=loop)) instead of: loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) loop.run_until_complete(my_coro()) TL;DR `async def coro(*, loop=None)` doesn't really make sense, and could be discouraged if get_event_loop was guaranteed to return the running event loop when called inside a coroutine. From steve at pearwood.info Fri May 20 06:30:17 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 20 May 2016 20:30:17 +1000 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: <20160518174416.6D39EB1400B@webabinitio.net> References: <20160517003511.AD7C9B14094@webabinitio.net> <573BF1EB.9030303@stoneleaf.us> <20160518174416.6D39EB1400B@webabinitio.net> Message-ID: <20160520103016.GN12028@ando.pearwood.info> On Wed, May 18, 2016 at 01:44:12PM -0400, R. David Murray wrote: [...] > The main target would of > course be the data the program wants to preserve between runs, but one of > the examples at pmem.io is a game program whose state is all in nvram. > If the machine crashes in the middle of the game, upon restart you pick > up exactly where you left off, including all the positioning information > you might consider to be part of the "hot loop" part of the program. Wouldn't that mean you're restarting the game in the same state it was in just before it crashed? Which means (assuming the crash is deterministic) the very next thing the game will do is crash. -- Steve From leewangzhong+python at gmail.com Fri May 20 06:48:13 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Fri, 20 May 2016 06:48:13 -0400 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <573D7734.20402@canterbury.ac.nz> References: <573D7734.20402@canterbury.ac.nz> Message-ID: I think there should be different syntaxes for matching equality and binding patterns, and definitely different syntax for singular and plural cases. Let's try this: - Equality: `case 5:` - Conditional: `case if predicate(obj):` - Pattern-matching: `case as [a, b, *_]:` `case as Point(x, y):` Slightly-more-explicit checks, instead of simply `case 5:`. - `case == 5:` - `case is _sentinel:` - `case is None:` - `case < 6:` - `case in (1,2,3):` - `case in range(2, 5):` - `case in int:` The above uses an implicit object and is very different from how Python usually works. Though I think it's mentally consistent (as in, no contradictions), it's not an extrapolation of current syntax, and might go against "explicit self". It isn't necessary, though: require `case 5` and `case if obj == 5:`. I prefer `case == 5` and `case is 5`, though, even if that's not how other languages do it. The last one treats a type as a collection of its instances, which I just like. (I also like `SubClass < BaseClass`.) On Thu, May 19, 2016 at 4:20 AM, Greg Ewing wrote: > If we want this feature, it seems like we will need to > explicitly mark either names to be bound or expressions to > be evaluated and matched against. Possible pre-fix markers: case as Point(zero, =y): case as Point(zero, !y): case as Point(zero, $y): case as Point(zero, &y): case as Point(zero, ^y): case as Point(zero, :y): Possible post-fix markers: case as Point(zero, y=): case as Point(zero, y!): case as Point(zero, y:): I think the colon won't be ambiguous in dict displays (it's only a marker if one side of the colon is missing), but it might complicate the parser. It'd be ambiguous in slicing, but I don't see slice-matching being a thing. (My favorite is probably `&y`, for a bad reason: it's very C-like. That'd confuse C programmers when `&` doesn't work anywhere else and `*` won't work at all.) > If you wanted the second case to only match the tuple, > you could write > > case tuple(a, b, *_): Slightly naughty. The tuple constructor only takes one argument. On Thu, May 19, 2016 at 12:15 AM, Guido van Rossum wrote: > A few things that might be interesting to explore: I'll try out my syntax ideas on these. > - match by value or set of values (like those PEPs) `case == 1:` `case in (1, 2, 3):` > - match by type (isinstance() checks) `case in int:` `case if isinstance(obj, int):` `case if type(obj) == tuple:` > - match on tuple structure, including nesting and * unpacking > (essentially, try a series of destructuring assignments until one > works) `case in tuple as (first, *middle, last):` `case if isinstance(obj, tuple) as (first, *middle, last):` `case if type(obj) == tuple as (first, *middle, last):` > - match on dict structure? (extension of destructuring to dicts) I think it makes no sense to bind the keys to names, because they'd just be chosen arbitrarily (unless it's an OrderedDict), so let's say keys are evaluated and values are (by default) names to bind. Let `**_` mean "other items". `case as {'foo': x, 'bar': y, **_}:` `case as {key0: val0, key1: val1}: # Binds val0 and val1.` `case as {'foo': foo_val, var_holding_bar: bar_val, **_}:` ^ Ew. I'd like names-to-bind to require special syntax. `case as dict(foo=foo_val, **{var_holding_bar: bar_val}, **_):` If we had an OrderedDict syntax, binding keys makes more sense. case as [k0: v0, **_, klast: vlast]: P.S.: If `**_` is allowed in matching, it should be allowed in unpacking assignment. {'foo': x, 'bar': y, **_} = d > - match on instance variables or attributes by name? One of the following: `case as object(foo=x, bar=y):` `case as Object(foo=x, bar=y):` `case as SpecialAttrMatchingThing(foo=x, bar=y):` `SpecialAttrMatchingThing` objects would be special in the match system. > - match on generalized condition (predicate)? `case <= 4:` `case if is_something(the_obj):` `case as Point(x, y) if x == y:` `case if len(the_obj) < 5 as [first, second, *_] if isinstance(first, int):` == Matching user classes == What about using pickle dunders for this? In particular, `MyClass.__getnewargs__` and `MyClass.__getnewargs_ex__`. Pickling is kind of related to pattern matching. Note that the classes don't have to be immutable to be matchable. When matching `p` to `Point(x, y)`, the match system calls `Point.__getnewargs__(p)` (NOT `p.__getnewargs__`, thus allowing for some subclass matching). The result is matched against a two-tuple, and finally bound. # def match_constructor(obj, cls): if not isinstance(obj, cls): raise FailedMatch try: m = cls.__getnewargs_ex__ except AttributeError: pass else: args, kwargs = m(obj) # Even for subclasses. return args, kwargs try: m = cls.__getnewargs__ except AttributeError: raise FailedMatch args = m(obj) return args, {} # Back in the case: try: (x, y), [] = match_constructor(p, Point) except ValueError: raise FailedMatch run_block() The problem is that these functions return (args, kwargs), and optional args and pos_or_kw_params mess things up. Point(x, y) Point(x, y=y) # How can match system know that all of the following are valid patterns? Something(x) Something(x, y) Something(*args, **kwargs) Solution 1: Additional args to the pickle methods (or make a new method which could sometimes be used for pickling): - nargs: Number of args. An int, or a integer range (for `*args`). A `range` doesn't allow unbounded, so use `slice(min_nargs, None)`, `(min_nargs, ...)`, or `(min_nargs, None)`. - kws: Keywords. To represent **kwargs, either add another arg or pass `...` or `None` as a keyword. Solution 2: After receiving (args, kwargs) from the class, inspect the signature of the class constructor and just figure out how it works. == Additional Resources == * MacroPy implements case classes, which are similar to what Haskell uses for pattern matching on constructors. I'm not sure that it lends insight to the `Point(x, y)` case, because MacroPy doesn't have pattern matching, but maybe someone else would. https://github.com/lihaoyi/macropy#case-classes * "Pattern matching in Python" (2009) An attempt at making a matching thing. https://monkey.org/~marius/pattern-matching-in-python.html * PEP 275 -- Switching on Multiple Values https://www.python.org/dev/peps/pep-0275/ * PEP 3103 -- A Switch/Case Statement https://www.python.org/dev/peps/pep-3103/ From ppbbalcer at gmail.com Fri May 20 07:17:57 2016 From: ppbbalcer at gmail.com (Piotr Balcer) Date: Fri, 20 May 2016 13:17:57 +0200 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: <20160520103016.GN12028@ando.pearwood.info> References: <20160517003511.AD7C9B14094@webabinitio.net> <573BF1EB.9030303@stoneleaf.us> <20160518174416.6D39EB1400B@webabinitio.net> <20160520103016.GN12028@ando.pearwood.info> Message-ID: That's an excellent observation ;) This is actually what happend when we first wrote that game - there was a NULL-dereference in the game logic and it caused a 'persistent segmentation fault'. It's really important for programmers to step up when it comes to creating software that uses this type of memory directly. Everyone essentially becomes a file system developer, which is not necessarily a good thing. Our hope is that high level languages like python will also help in this regard by making sure that the programmer cannot shoot himself in the foot as easily as it is possible in C. Piotr 2016-05-20 12:30 GMT+02:00 Steven D'Aprano : > On Wed, May 18, 2016 at 01:44:12PM -0400, R. David Murray wrote: > > [...] > > The main target would of > > course be the data the program wants to preserve between runs, but one of > > the examples at pmem.io is a game program whose state is all in nvram. > > If the machine crashes in the middle of the game, upon restart you pick > > up exactly where you left off, including all the positioning information > > you might consider to be part of the "hot loop" part of the program. > > Wouldn't that mean you're restarting the game in the same state it was > in just before it crashed? Which means (assuming the crash is > deterministic) the very next thing the game will do is crash. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri May 20 07:36:43 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 20 May 2016 21:36:43 +1000 Subject: [Python-ideas] Lib vs ( App | Project | ... ) In-Reply-To: <573C0DF3.1070000@thomas-guettler.de> References: <573AFBD5.6010902@thomas-guettler.de> <573C0DF3.1070000@thomas-guettler.de> Message-ID: <20160520113643.GO12028@ando.pearwood.info> On Wed, May 18, 2016 at 08:38:43AM +0200, Thomas G?ttler wrote: > I see no agreement on the name for the concrete installation holding all > those nice python libraries. I don't think you have made the case why we need such an agreement. What is the *actual problem* you are hoping to solve here? -- Steve From mal at egenix.com Fri May 20 07:49:42 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 20 May 2016 13:49:42 +0200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <573D7734.20402@canterbury.ac.nz> Message-ID: <573EF9D6.9050704@egenix.com> Please see these PEPs... https://www.python.org/dev/peps/pep-3103/ https://www.python.org/dev/peps/pep-0275/ ...before starting yet another long thread on the same topic. On 20.05.2016 12:48, Franklin? Lee wrote: > I think there should be different syntaxes for matching equality and > binding patterns, and definitely different syntax for singular and > plural cases. > > Let's try this: > - Equality: > `case 5:` > - Conditional: > `case if predicate(obj):` > - Pattern-matching: > `case as [a, b, *_]:` > `case as Point(x, y):` > > Slightly-more-explicit checks, instead of simply `case 5:`. > - `case == 5:` > - `case is _sentinel:` > - `case is None:` > - `case < 6:` > - `case in (1,2,3):` > - `case in range(2, 5):` > - `case in int:` > > The above uses an implicit object and is very different from how > Python usually works. Though I think it's mentally consistent (as in, > no contradictions), it's not an extrapolation of current syntax, and > might go against "explicit self". It isn't necessary, though: require > `case 5` and `case if obj == 5:`. I prefer `case == 5` and `case is > 5`, though, even if that's not how other languages do it. > > The last one treats a type as a collection of its instances, which I > just like. (I also like `SubClass < BaseClass`.) > > > On Thu, May 19, 2016 at 4:20 AM, Greg Ewing wrote: >> If we want this feature, it seems like we will need to >> explicitly mark either names to be bound or expressions to >> be evaluated and matched against. > > Possible pre-fix markers: > case as Point(zero, =y): > case as Point(zero, !y): > case as Point(zero, $y): > case as Point(zero, &y): > case as Point(zero, ^y): > case as Point(zero, :y): > > Possible post-fix markers: > case as Point(zero, y=): > case as Point(zero, y!): > case as Point(zero, y:): > > I think the colon won't be ambiguous in dict displays (it's only a > marker if one side of the colon is missing), but it might complicate > the parser. It'd be ambiguous in slicing, but I don't see > slice-matching being a thing. > > (My favorite is probably `&y`, for a bad reason: it's very C-like. > That'd confuse C programmers when `&` doesn't work anywhere else and > `*` won't work at all.) > > >> If you wanted the second case to only match the tuple, >> you could write >> >> case tuple(a, b, *_): > > Slightly naughty. The tuple constructor only takes one argument. > > > On Thu, May 19, 2016 at 12:15 AM, Guido van Rossum wrote: >> A few things that might be interesting to explore: > > I'll try out my syntax ideas on these. > >> - match by value or set of values (like those PEPs) > > `case == 1:` > `case in (1, 2, 3):` > >> - match by type (isinstance() checks) > > `case in int:` > `case if isinstance(obj, int):` > `case if type(obj) == tuple:` > >> - match on tuple structure, including nesting and * unpacking >> (essentially, try a series of destructuring assignments until one >> works) > > `case in tuple as (first, *middle, last):` > `case if isinstance(obj, tuple) as (first, *middle, last):` > `case if type(obj) == tuple as (first, *middle, last):` > >> - match on dict structure? (extension of destructuring to dicts) > > I think it makes no sense to bind the keys to names, because > they'd just be chosen arbitrarily (unless it's an OrderedDict), so > let's say keys are evaluated and values are (by default) names to > bind. > > Let `**_` mean "other items". > > `case as {'foo': x, 'bar': y, **_}:` > `case as {key0: val0, key1: val1}: # Binds val0 and val1.` > `case as {'foo': foo_val, var_holding_bar: bar_val, **_}:` > ^ Ew. I'd like names-to-bind to require special syntax. > `case as dict(foo=foo_val, **{var_holding_bar: bar_val}, **_):` > > If we had an OrderedDict syntax, binding keys makes more sense. > case as [k0: v0, **_, klast: vlast]: > > P.S.: If `**_` is allowed in matching, it should be allowed in > unpacking assignment. > {'foo': x, 'bar': y, **_} = d > > >> - match on instance variables or attributes by name? > > One of the following: > `case as object(foo=x, bar=y):` > `case as Object(foo=x, bar=y):` > `case as SpecialAttrMatchingThing(foo=x, bar=y):` > > `SpecialAttrMatchingThing` objects would be special in the match system. > >> - match on generalized condition (predicate)? > > `case <= 4:` > `case if is_something(the_obj):` > `case as Point(x, y) if x == y:` > `case if len(the_obj) < 5 as [first, second, *_] if isinstance(first, int):` > > > == Matching user classes == > > What about using pickle dunders for this? In particular, > `MyClass.__getnewargs__` and `MyClass.__getnewargs_ex__`. Pickling is > kind of related to pattern matching. Note that the classes don't have > to be immutable to be matchable. > > When matching `p` to `Point(x, y)`, the match system calls > `Point.__getnewargs__(p)` (NOT `p.__getnewargs__`, thus allowing for > some subclass matching). The result is matched against a two-tuple, > and finally bound. > > # def match_constructor(obj, cls): > if not isinstance(obj, cls): > raise FailedMatch > try: > m = cls.__getnewargs_ex__ > except AttributeError: > pass > else: > args, kwargs = m(obj) # Even for subclasses. > return args, kwargs > try: > m = cls.__getnewargs__ > except AttributeError: > raise FailedMatch > args = m(obj) > return args, {} > > # Back in the case: > try: > (x, y), [] = match_constructor(p, Point) > except ValueError: > raise FailedMatch > run_block() > > > The problem is that these functions return (args, kwargs), and > optional args and pos_or_kw_params mess things up. > > Point(x, y) > Point(x, y=y) > # How can match system know that all of the following are > valid patterns? > Something(x) > Something(x, y) > Something(*args, **kwargs) > > Solution 1: Additional args to the pickle methods (or make a new > method which could sometimes be used for pickling): > - nargs: Number of args. An int, or a integer range (for `*args`). > A `range` doesn't allow unbounded, so use `slice(min_nargs, None)`, > `(min_nargs, ...)`, or `(min_nargs, None)`. > - kws: Keywords. To represent **kwargs, either add another arg or > pass `...` or `None` as a keyword. > > Solution 2: After receiving (args, kwargs) from the class, inspect the > signature of the class constructor and just figure out how it works. > > > == Additional Resources == > > * MacroPy implements case classes, which are similar to what Haskell > uses for pattern matching on constructors. I'm not sure that it lends > insight to the `Point(x, y)` case, because MacroPy doesn't have > pattern matching, but maybe someone else would. > https://github.com/lihaoyi/macropy#case-classes > > * "Pattern matching in Python" (2009) > An attempt at making a matching thing. > https://monkey.org/~marius/pattern-matching-in-python.html > > * PEP 275 -- Switching on Multiple Values > https://www.python.org/dev/peps/pep-0275/ > > * PEP 3103 -- A Switch/Case Statement > https://www.python.org/dev/peps/pep-3103/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 20 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From michael.selik at gmail.com Fri May 20 12:03:50 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 20 May 2016 16:03:50 +0000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: Message-ID: On Thu, May 19, 2016 at 10:38 PM Nick Coghlan wrote: > On 19 May 2016 at 14:15, Guido van Rossum wrote: > > The attribute idea would be similar to a duck-type check, though more > > emphasizing data attributes. It would be nice if we could write a > > match that said "if it has attributes x and y, assign those to local > > variables p and q, and ignore other attributes". > > If we went down that path, then the "assign if you can, execute this > case if you succeed" options would presumably need an explicit prefix > to indicate they're not normal expressions, perhaps something like > "?=": > > switch expr as arg: > case ?= (.x as p, .y as q): print('x=', p, 'y=', q) > If you don't mind adding a new operator, then an easier way to handle several of these situations would be to make ``?=`` an assignment expression that evaluates to True/False whether the assignment succeeded: def foo(obj): return a ?= obj.a Could be equivalent to: def foo(obj): try: a = obj.a except Exception: return False else: return True The use-cases of are somewhat overlapping with the idea of an inline try/except as in PEP 463 (https://www.python.org/dev/peps/pep-0463/). Which would then have the further implication that it might also make > sense to support attribute unpacking as the LHS of normal assignment > statements: > > (.x as p, .y as q) = expr > The ``as`` syntax flips the familiar variable-on-the-left and makes this one tough for me to read. I'd rather force a little repetition: o = expr p, q = o.x, o.y Using a temporary variable like ``o`` makes it even fewer characters than the proposed parens and ``as`` keyword. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gerald.britton at gmail.com Fri May 20 11:10:24 2016 From: gerald.britton at gmail.com (Gerald Britton) Date: Fri, 20 May 2016 15:10:24 +0000 Subject: [Python-ideas] Some food for thought here. (Just dashed this off from my phone) In-Reply-To: References: Message-ID: Needs pythonizing of course but I like the flexibility. ----- https://msdn.microsoft.com/visualfsharpdocs/conceptual/pattern-matching-%5bfsharp%5d -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri May 20 12:22:39 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 20 May 2016 09:22:39 -0700 Subject: [Python-ideas] Add ability to get current event loop within asyncio coroutine In-Reply-To: <573ED2AE.9080101@gmail.com> References: <573ED2AE.9080101@gmail.com> Message-ID: OK, I finally see the point. There are legitimate situations where get_event_loop() returns None (this is how most of asyncio's tests are running). So we want a separate function, e.g. get_current_event_loop(). Send a PR to the GitHub project. On Fri, May 20, 2016 at 2:02 AM, Vincent Michel wrote: > > On Thu, May 19, 2016 at 11:49 AM, Ilya Kulakov > wrote: >> >> I think Python would benefit from providing either a new method (e.g. >> `asyncio.current_event_loop`) or modifying `asyncio.get_event_loop` >> to return current event loop when called from a coroutine. It should, in >> my >> opinion, reduce necessity of passing event loop >> between coroutines in application's code as it is done in asyncio itself >> and >> 3rd party libraries (like aiohttp). > > > I was about to make the same proposal. I can add two practical arguments in > favor. > > First, some coroutines could be simplified. For instance, asyncio.sleep > takes an optional loop argument, defaulting to get_event_loop(). But > get_event_loop is called inside the coroutine, and therefore inside a > running event loop. This loop is actually the only valid choice, and using > anything else should raise an exception. If get_event_loop is guaranteed to > return the current running loop, then there is no reason to pass it as an > argument to asyncio.sleep. Other coroutines could benefit from that, > including wait and wait_for. > > It would also help for queues, locks and other task functions (gather, > shield, as_completed, etc.). Here the loop argument is still useful, in case > those objects are declared outside their event loop. However, it would be > really convenient to assume loop=None always works as expected if the object > is created inside a coroutine. In many cases, libraries tend to forward the > loop argument from coroutine to coroutine, just in case someone writes: > > loop = asyncio.new_event_loop() > loop.run_until_complete(my_coro(loop=loop)) > > instead of: > > loop = asyncio.new_event_loop() > asyncio.set_event_loop(loop) > loop.run_until_complete(my_coro()) > > TL;DR > `async def coro(*, loop=None)` doesn't really make sense, and could be > discouraged if get_event_loop was guaranteed to return the running event > loop when called inside a coroutine. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri May 20 12:32:48 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 20 May 2016 09:32:48 -0700 Subject: [Python-ideas] Some food for thought here. (Just dashed this off from my phone) In-Reply-To: References: Message-ID: That's just the Haskell matching translated to F# isn't it? PS. Next time even when you're on your mobile phone try replying to the existing thread. Your subject makes you look like you're a pointy-haired manager who barely knows how to use email on his phone. On Fri, May 20, 2016 at 8:10 AM, Gerald Britton wrote: > Needs pythonizing of course but I like the flexibility. > > > ----- > > > https://msdn.microsoft.com/visualfsharpdocs/conceptual/pattern-matching-%5bfsharp%5d > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From rymg19 at gmail.com Fri May 20 12:45:05 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 20 May 2016 11:45:05 -0500 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <573D7734.20402@canterbury.ac.nz> Message-ID: On Fri, May 20, 2016 at 5:48 AM, Franklin? Lee < leewangzhong+python at gmail.com> wrote: > I think there should be different syntaxes for matching equality and > binding patterns, and definitely different syntax for singular and > plural cases. > > Let's try this: > - Equality: > `case 5:` > - Conditional: > `case if predicate(obj):` > - Pattern-matching: > `case as [a, b, *_]:` > `case as Point(x, y):` > > Slightly-more-explicit checks, instead of simply `case 5:`. > - `case == 5:` > - `case is _sentinel:` > - `case is None:` > - `case < 6:` > - `case in (1,2,3):` > - `case in range(2, 5):` > - `case in int:` > > I personally really like the equality, conditional, and pattern-matching syntax you showed; it looks the most Pythonic IMO. However, the more explicit checks are a little overkill and feel like too much special-casing. I don't know of any languages that go so far as to allow stuff like `case < 6`; usually, there would instead be some syntax to assign the initial object to an intermediate variable, that way you can just use it with an `if` predicate. > The above uses an implicit object and is very different from how > Python usually works. Though I think it's mentally consistent (as in, > no contradictions), it's not an extrapolation of current syntax, and > might go against "explicit self". It isn't necessary, though: require > `case 5` and `case if obj == 5:`. I prefer `case == 5` and `case is > 5`, though, even if that's not how other languages do it. > > The last one treats a type as a collection of its instances, which I > just like. (I also like `SubClass < BaseClass`.) > > > On Thu, May 19, 2016 at 4:20 AM, Greg Ewing > wrote: > > If we want this feature, it seems like we will need to > > explicitly mark either names to be bound or expressions to > > be evaluated and matched against. > > Possible pre-fix markers: > case as Point(zero, =y): > case as Point(zero, !y): > case as Point(zero, $y): > case as Point(zero, &y): > case as Point(zero, ^y): > case as Point(zero, :y): > > Possible post-fix markers: > case as Point(zero, y=): > case as Point(zero, y!): > case as Point(zero, y:): > > I think the colon won't be ambiguous in dict displays (it's only a > marker if one side of the colon is missing), but it might complicate > the parser. It'd be ambiguous in slicing, but I don't see > slice-matching being a thing. > > (My favorite is probably `&y`, for a bad reason: it's very C-like. > That'd confuse C programmers when `&` doesn't work anywhere else and > `*` won't work at all.) > > > > If you wanted the second case to only match the tuple, > > you could write > > > > case tuple(a, b, *_): > > Slightly naughty. The tuple constructor only takes one argument. > > > On Thu, May 19, 2016 at 12:15 AM, Guido van Rossum > wrote: > > A few things that might be interesting to explore: > > I'll try out my syntax ideas on these. > > > - match by value or set of values (like those PEPs) > > `case == 1:` > `case in (1, 2, 3):` > > > - match by type (isinstance() checks) > > `case in int:` > `case if isinstance(obj, int):` > `case if type(obj) == tuple:` > > > - match on tuple structure, including nesting and * unpacking > > (essentially, try a series of destructuring assignments until one > > works) > > `case in tuple as (first, *middle, last):` > `case if isinstance(obj, tuple) as (first, *middle, last):` > `case if type(obj) == tuple as (first, *middle, last):` > > > - match on dict structure? (extension of destructuring to dicts) > > I think it makes no sense to bind the keys to names, because > they'd just be chosen arbitrarily (unless it's an OrderedDict), so > let's say keys are evaluated and values are (by default) names to > bind. > > Let `**_` mean "other items". > > `case as {'foo': x, 'bar': y, **_}:` > `case as {key0: val0, key1: val1}: # Binds val0 and val1.` > `case as {'foo': foo_val, var_holding_bar: bar_val, **_}:` > ^ Ew. I'd like names-to-bind to require special syntax. > `case as dict(foo=foo_val, **{var_holding_bar: bar_val}, **_):` > > If we had an OrderedDict syntax, binding keys makes more sense. > case as [k0: v0, **_, klast: vlast]: > > P.S.: If `**_` is allowed in matching, it should be allowed in > unpacking assignment. > {'foo': x, 'bar': y, **_} = d > > > > - match on instance variables or attributes by name? > > One of the following: > `case as object(foo=x, bar=y):` > `case as Object(foo=x, bar=y):` > `case as SpecialAttrMatchingThing(foo=x, bar=y):` > > `SpecialAttrMatchingThing` objects would be special in the match > system. > > > - match on generalized condition (predicate)? > > `case <= 4:` > `case if is_something(the_obj):` > `case as Point(x, y) if x == y:` > `case if len(the_obj) < 5 as [first, second, *_] if isinstance(first, > int):` > > > == Matching user classes == > > What about using pickle dunders for this? In particular, > `MyClass.__getnewargs__` and `MyClass.__getnewargs_ex__`. Pickling is > kind of related to pattern matching. Note that the classes don't have > to be immutable to be matchable. > > When matching `p` to `Point(x, y)`, the match system calls > `Point.__getnewargs__(p)` (NOT `p.__getnewargs__`, thus allowing for > some subclass matching). The result is matched against a two-tuple, > and finally bound. > > # def match_constructor(obj, cls): > if not isinstance(obj, cls): > raise FailedMatch > try: > m = cls.__getnewargs_ex__ > except AttributeError: > pass > else: > args, kwargs = m(obj) # Even for subclasses. > return args, kwargs > try: > m = cls.__getnewargs__ > except AttributeError: > raise FailedMatch > args = m(obj) > return args, {} > > # Back in the case: > try: > (x, y), [] = match_constructor(p, Point) > except ValueError: > raise FailedMatch > run_block() > > > The problem is that these functions return (args, kwargs), and > optional args and pos_or_kw_params mess things up. > > Point(x, y) > Point(x, y=y) > # How can match system know that all of the following are > valid patterns? > Something(x) > Something(x, y) > Something(*args, **kwargs) > > Solution 1: Additional args to the pickle methods (or make a new > method which could sometimes be used for pickling): > - nargs: Number of args. An int, or a integer range (for `*args`). > A `range` doesn't allow unbounded, so use `slice(min_nargs, None)`, > `(min_nargs, ...)`, or `(min_nargs, None)`. > - kws: Keywords. To represent **kwargs, either add another arg or > pass `...` or `None` as a keyword. > > Solution 2: After receiving (args, kwargs) from the class, inspect the > signature of the class constructor and just figure out how it works. > > > == Additional Resources == > > * MacroPy implements case classes, which are similar to what Haskell > uses for pattern matching on constructors. I'm not sure that it lends > insight to the `Point(x, y)` case, because MacroPy doesn't have > pattern matching, but maybe someone else would. > https://github.com/lihaoyi/macropy#case-classes > > * "Pattern matching in Python" (2009) > An attempt at making a matching thing. > https://monkey.org/~marius/pattern-matching-in-python.html > > * PEP 275 -- Switching on Multiple Values > https://www.python.org/dev/peps/pep-0275/ > > * PEP 3103 -- A Switch/Case Statement > https://www.python.org/dev/peps/pep-3103/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri May 20 13:45:28 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 20 May 2016 18:45:28 +0100 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <573D7734.20402@canterbury.ac.nz> Message-ID: On 20 May 2016 at 17:45, Ryan Gonzalez wrote: > On Fri, May 20, 2016 at 5:48 AM, Franklin? Lee > wrote: >> >> I think there should be different syntaxes for matching equality and >> binding patterns, and definitely different syntax for singular and >> plural cases. >> >> Let's try this: >> - Equality: >> `case 5:` >> - Conditional: >> `case if predicate(obj):` >> - Pattern-matching: >> `case as [a, b, *_]:` >> `case as Point(x, y):` >> >> Slightly-more-explicit checks, instead of simply `case 5:`. >> - `case == 5:` >> - `case is _sentinel:` >> - `case is None:` >> - `case < 6:` >> - `case in (1,2,3):` >> - `case in range(2, 5):` >> - `case in int:` >> > > I personally really like the equality, conditional, and pattern-matching > syntax you showed; it looks the most Pythonic IMO. Agreed. > However, the more explicit checks are a little overkill and feel like too > much special-casing. I don't know of any languages that go so far as to > allow stuff like `case < 6`; usually, there would instead be some syntax to > assign the initial object to an intermediate variable, that way you can just > use it with an `if` predicate. Also agreed. It seems that switch expr as name case if name is _sentinel: do_stuff() would be a reasonable syntax for naming the switch expression - although just giving it a name before using it in the switch is probably just as reasonable: name = expr switch name: case if name is _sentinel: do_stuff() BTW, there's also a semantic question - if multiple cases match, would only one (presumably the first) or all of them be executed? I'm sure this was discussed to death during the discussions on PEPs 3103 and 275, but I don't have the time or inclination to re-read those now, so I'll just leave the question open here for someone else to do the research... Paul From guido at python.org Fri May 20 13:47:19 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 20 May 2016 10:47:19 -0700 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <573D7734.20402@canterbury.ac.nz> Message-ID: It's definitely going to be "try cases in order until one matches". On Fri, May 20, 2016 at 10:45 AM, Paul Moore wrote: > On 20 May 2016 at 17:45, Ryan Gonzalez wrote: >> On Fri, May 20, 2016 at 5:48 AM, Franklin? Lee >> wrote: >>> >>> I think there should be different syntaxes for matching equality and >>> binding patterns, and definitely different syntax for singular and >>> plural cases. >>> >>> Let's try this: >>> - Equality: >>> `case 5:` >>> - Conditional: >>> `case if predicate(obj):` >>> - Pattern-matching: >>> `case as [a, b, *_]:` >>> `case as Point(x, y):` >>> >>> Slightly-more-explicit checks, instead of simply `case 5:`. >>> - `case == 5:` >>> - `case is _sentinel:` >>> - `case is None:` >>> - `case < 6:` >>> - `case in (1,2,3):` >>> - `case in range(2, 5):` >>> - `case in int:` >>> >> >> I personally really like the equality, conditional, and pattern-matching >> syntax you showed; it looks the most Pythonic IMO. > > Agreed. > >> However, the more explicit checks are a little overkill and feel like too >> much special-casing. I don't know of any languages that go so far as to >> allow stuff like `case < 6`; usually, there would instead be some syntax to >> assign the initial object to an intermediate variable, that way you can just >> use it with an `if` predicate. > > Also agreed. It seems that > > switch expr as name > case if name is _sentinel: do_stuff() > > would be a reasonable syntax for naming the switch expression - > although just giving it a name before using it in the switch is > probably just as reasonable: > > name = expr > switch name: > case if name is _sentinel: do_stuff() > > BTW, there's also a semantic question - if multiple cases match, would > only one (presumably the first) or all of them be executed? I'm sure > this was discussed to death during the discussions on PEPs 3103 and > 275, but I don't have the time or inclination to re-read those now, so > I'll just leave the question open here for someone else to do the > research... > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From kulakov.ilya at gmail.com Fri May 20 13:53:53 2016 From: kulakov.ilya at gmail.com (Ilya Kulakov) Date: Fri, 20 May 2016 13:53:53 -0400 Subject: [Python-ideas] Add ability to get current event loop within asyncio coroutine In-Reply-To: References: <573ED2AE.9080101@gmail.com> Message-ID: > On 20 May 2016, at 12:22, Guido van Rossum wrote: > > OK, I finally see the point. There are legitimate situations where > get_event_loop() returns None (this is how most of asyncio's tests are > running). Unless there are tests that expect that get_event_loop() would return None when called from within coroutine it should be fine. > So we want a separate function, e.g. > get_current_event_loop(). Send a PR to the GitHub project. Also see related bug: http://bugs.python.org/issue26969 Yuri did some work, but he argumented in favor of modifying get_event_loop. Should we continue discussion of implementation here or in that issue? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri May 20 14:04:25 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 20 May 2016 11:04:25 -0700 Subject: [Python-ideas] Add ability to get current event loop within asyncio coroutine In-Reply-To: References: <573ED2AE.9080101@gmail.com> Message-ID: On Fri, May 20, 2016 at 10:53 AM, Ilya Kulakov wrote: > > On 20 May 2016, at 12:22, Guido van Rossum wrote: > > > OK, I finally see the point. There are legitimate situations where > > get_event_loop() returns None (this is how most of asyncio's tests are > > running). > > > Unless there are tests that expect that get_event_loop() would return None > when called from within coroutine it should be fine. The tests are supposed to fail if something doesn't explicitly pass the loop; this is how we test that all code in asyncio itself always correctly passes the loop rather than accidentally relying on get_event_loop(). > > So we want a separate function, e.g. > > get_current_event_loop(). Send a PR to the GitHub project. > > > Also see related bug: http://bugs.python.org/issue26969 > Yuri did some work, but he argumented in favor of modifying get_event_loop. > > Should we continue discussion of implementation here or in that issue? I consider it a feature that you can make get_event_loop() return None. But I think it would be a bug (in the Policy class) if it returned some other loop that's not the one that's running. I also think it's fine for a particular library or app to request that get_event_loop() not return None. User code is simpler that way. -- --Guido van Rossum (python.org/~guido) From jim.baker at python.org Fri May 20 15:25:57 2016 From: jim.baker at python.org (Jim Baker) Date: Fri, 20 May 2016 13:25:57 -0600 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <573D7734.20402@canterbury.ac.nz> Message-ID: Agreed, if we want to support the structural pattern matching as seen in Haskell, Rust, or Scala, we should do the following: - Match in sequence against the provided cases. Cases should support optional guards (`if` clause) - Follow only one case - no fall through, no need to `break` - Raise a match error if the match is not exhaustive; an `else` clause is fine for exhaustion Using the typical switch pattern seen in C or Java is already idiomatically addressed in Python through dispatching through a dict. This type of matching is different. On Fri, May 20, 2016 at 11:47 AM, Guido van Rossum wrote: > It's definitely going to be "try cases in order until one matches". > > On Fri, May 20, 2016 at 10:45 AM, Paul Moore wrote: > > On 20 May 2016 at 17:45, Ryan Gonzalez wrote: > >> On Fri, May 20, 2016 at 5:48 AM, Franklin? Lee > >> wrote: > >>> > >>> I think there should be different syntaxes for matching equality and > >>> binding patterns, and definitely different syntax for singular and > >>> plural cases. > >>> > >>> Let's try this: > >>> - Equality: > >>> `case 5:` > >>> - Conditional: > >>> `case if predicate(obj):` > >>> - Pattern-matching: > >>> `case as [a, b, *_]:` > >>> `case as Point(x, y):` > >>> > >>> Slightly-more-explicit checks, instead of simply `case 5:`. > >>> - `case == 5:` > >>> - `case is _sentinel:` > >>> - `case is None:` > >>> - `case < 6:` > >>> - `case in (1,2,3):` > >>> - `case in range(2, 5):` > >>> - `case in int:` > >>> > >> > >> I personally really like the equality, conditional, and pattern-matching > >> syntax you showed; it looks the most Pythonic IMO. > > > > Agreed. > > > >> However, the more explicit checks are a little overkill and feel like > too > >> much special-casing. I don't know of any languages that go so far as to > >> allow stuff like `case < 6`; usually, there would instead be some > syntax to > >> assign the initial object to an intermediate variable, that way you can > just > >> use it with an `if` predicate. > > > > Also agreed. It seems that > > > > switch expr as name > > case if name is _sentinel: do_stuff() > > > > would be a reasonable syntax for naming the switch expression - > > although just giving it a name before using it in the switch is > > probably just as reasonable: > > > > name = expr > > switch name: > > case if name is _sentinel: do_stuff() > > > > BTW, there's also a semantic question - if multiple cases match, would > > only one (presumably the first) or all of them be executed? I'm sure > > this was discussed to death during the discussions on PEPs 3103 and > > 275, but I don't have the time or inclination to re-read those now, so > > I'll just leave the question open here for someone else to do the > > research... > > > > Paul > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri May 20 15:45:29 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 20 May 2016 20:45:29 +0100 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <573D7734.20402@canterbury.ac.nz> Message-ID: On 20 May 2016 at 18:47, Guido van Rossum wrote: > It's definitely going to be "try cases in order until one matches". Cool, thanks. Life's so much simpler with a BDFL pronouncement :-) Paul From tjreedy at udel.edu Fri May 20 16:06:03 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 20 May 2016 16:06:03 -0400 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: References: <20160517003511.AD7C9B14094@webabinitio.net> <573BF1EB.9030303@stoneleaf.us> <20160518174416.6D39EB1400B@webabinitio.net> <20160520103016.GN12028@ando.pearwood.info> Message-ID: On 5/20/2016 7:17 AM, Piotr Balcer wrote: [rearranged to undo top posting] > 2016-05-20 12:30 GMT+02:00 Steven D'Aprano > Wouldn't that mean you're restarting the game in the same state it was > in just before it crashed? Which means (assuming the crash is > deterministic) the very next thing the game will do is crash. > That's an excellent observation ;) This is actually what happend when we > first wrote that game - there was a NULL-dereference in the game logic and > it caused a 'persistent segmentation fault'. It's really important for > programmers to step up when it comes to creating software that uses > this type of memory directly. Everyone essentially becomes a file system > developer, which is not necessarily a good thing. > > Our hope is that high level languages like python will also help in this > regard by making sure that the programmer cannot shoot himself in the foot > as easily as it is possible in C. Unless one uses ctypes or buggy external C-coded modules, seg faulting is already harder in Python than C. One possibility to make using persistent memory easier might be a context manager. The __enter__ method would copy a persistent object to a volatile object and return the volatile object. The body of the with statement would manipulate the volatile object. *If there are no exceptions*, the __exit__ method would 'commit' the changes, presumed to be consisting, by copying back to the persistent object. with persistent(locator) as volatile: Ignoring the possibility of a machine crash during the writeback, a restart after an exception or crash during the block would start with the persistent object as it was before the block, which absent bugs would be a consistent state. -- Terry Jan Reedy From ben+python at benfinney.id.au Fri May 20 20:40:43 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 21 May 2016 10:40:43 +1000 Subject: [Python-ideas] Some food for thought here. (Just dashed this off from my phone) References: Message-ID: <858tz4b5v8.fsf@benfinney.id.au> Guido van Rossum writes: > PS. Next time even when you're on your mobile phone try replying to > the existing thread. Your subject makes you look like you're a > pointy-haired manager who barely knows how to use email on his phone. Yes. Or someone who barely knows how to use online discussion forums (such as this one). If you need to ?dash off? a reminder, send it to yourself as a prompt to take the time later to compose a useful message in a thread. -- \ ?Sunday School: A prison in which children do penance for the | `\ evil conscience of their parents.? ?Henry L. Mencken | _o__) | Ben Finney From steve at pearwood.info Sun May 22 04:45:58 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 22 May 2016 18:45:58 +1000 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? Message-ID: <20160522084556.GR12028@ando.pearwood.info> On the tutor mailing list, somebody asked a question about decimal.InvalidOperation. They were converting strings to Decimal using something like this: try: d = Decimal(the_string) except ValueError: handle_error() and were perplexed by the error that they got: decimal.InvalidOperation: [] This got me thinking. Normally, such a conversion error should raise ValueError, as ints, floats and Fractions do. Presumably decimal does something different as it must match the General Decimal Arithmetic Specification. But, is there any reason why InvalidOperation couldn't be a subclass of ValueError? Exception class hierarchies are not part of the GDAS, so this should be allowed. Looking at the docs, there are nine examples given of things which can raise InvalidOperation (if not trapped, in which case they return a NAN): Infinity - Infinity 0 * Infinity Infinity / Infinity x % 0 Infinity % x sqrt(-x) and x > 0 0 ** 0 x ** (non-integer) x ** Infinity plus invalid string conversion. To my mind, ValueError would be an acceptable error for all of these things. E.g. the root of a negative value raises ValueError in Python 2: >>> (-2.0)**0.5 Traceback (most recent call last): File "", line 1, in ValueError: negative number cannot be raised to a fractional power (In Python 3, it returns a complex number.) So I propose that InvalidOperation be changed to inherit from ValueError, to match the expected behaviour from other numeric types. The only tricky bit is that division by zero doesn't raise ValueError, but DivideByZeroError instead. But that's no worse than the current situation: # current - people expect Decimal(1)/0 to raise DivideByZeroError, but it raises InvalidOperation; # proposal - people expect Decimal(1)/0 to raise DivideByZero, but it raises InvalidOperation (subclass of ValueError). Oh, a further data point: if you pass an invalid list or tuple to Decimal, you get a ValueError: py> decimal.Decimal([]) Traceback (most recent call last): File "", line 1, in ValueError: argument must be a sequence of length 3 Thoughts? -- Steve From rosuav at gmail.com Sun May 22 05:01:30 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 22 May 2016 19:01:30 +1000 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: <20160522084556.GR12028@ando.pearwood.info> References: <20160522084556.GR12028@ando.pearwood.info> Message-ID: On Sun, May 22, 2016 at 6:45 PM, Steven D'Aprano wrote: > Looking at the docs, there are nine examples given of things which can > raise InvalidOperation (if not trapped, in which case they return a > NAN): > > Infinity - Infinity > 0 * Infinity > Infinity / Infinity > x % 0 > Infinity % x > sqrt(-x) and x > 0 > 0 ** 0 > x ** (non-integer) > x ** Infinity > > plus invalid string conversion. To my mind, ValueError would be an > acceptable error for all of these things. Hmm. Out of interest, I probed fractions.Fraction for comparable behaviours. AFAIK, there's no way to represent infinity as a Fraction; using a floating-point infinity just converts the Fraction to a float and continues as per float; and mixing Fraction and Decimal either instantly raises TypeError, or (in the case of x**infinity) converts to float and *then* raises TypeError. 0 ** 0 simply returns 1. I can't get a ValueError out of any of the given operations. Why is x ** (non-integer) an InvalidOperation? Is that specifically for a negative value for x? The example you quoted [1] isn't clear there. (I also don't know why it describes the square root example that way, and not as "sqrt(x) and x < 0", but presumably there's a good mathematical reason for that.) > So I propose that InvalidOperation be changed to inherit from > ValueError, to match the expected behaviour from other numeric types. +1. ChrisA [1] https://docs.python.org/3/library/decimal.html#decimal.InvalidOperation From stefan at bytereef.org Sun May 22 06:15:16 2016 From: stefan at bytereef.org (Stefan Krah) Date: Sun, 22 May 2016 10:15:16 +0000 (UTC) Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? References: <20160522084556.GR12028@ando.pearwood.info> Message-ID: Steven D'Aprano writes: > So I propose that InvalidOperation be changed to inherit from > ValueError, to match the expected behaviour from other numeric types. One problem is that often a *combination* of values triggers InvalidOperation. ValueError does not really fit here. In this scenario other modules use TypeError, but I'm not a fan of that use either: >>> Decimal("inf") * Decimal(0) Traceback (most recent call last): File "", line 1, in decimal.InvalidOperation: [] >>> Decimal("inf") * Decimal("inf") Decimal('Infinity') Additionally, InvalidOperation is part of Decimal's custom exception handling via the context, so it would be a bit odd if ValueError were part of that. > py> decimal.Decimal([]) > Traceback (most recent call last): > File "", line 1, in > ValueError: argument must be a sequence of length 3 That is a ValueError because in principle list and tuple types are accepted due to the somewhat arcane from_tuple() conversion: >>> Decimal([0, [], 'F']) Decimal('Infinity') Since this is a Python specific extension, ValueError looks appropriate here. Count me -1 on any changes here. Stefan Krah From steve at pearwood.info Sun May 22 09:31:52 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 22 May 2016 23:31:52 +1000 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: References: <20160522084556.GR12028@ando.pearwood.info> Message-ID: <20160522133150.GS12028@ando.pearwood.info> On Sun, May 22, 2016 at 10:15:16AM +0000, Stefan Krah wrote: > Steven D'Aprano writes: > > So I propose that InvalidOperation be changed to inherit from > > ValueError, to match the expected behaviour from other numeric types. > > One problem is that often a *combination* of values triggers InvalidOperation. > ValueError does not really fit here. I don't see why it matters whether it is one value or a combination of values, it's still a value error. > In this scenario other modules use > TypeError, but I'm not a fan of that use either: > > >>> Decimal("inf") * Decimal(0) > Traceback (most recent call last): > File "", line 1, in > decimal.InvalidOperation: [] I'm not sure what you mean by this example, and what it has to do with TypeError. The error shown is related to the *value* of the arguments, namely Decimal inf and Decimal zero, not their types. Both arguments are Decimal. > >>> Decimal("inf") * Decimal("inf") > Decimal('Infinity') Again, I don't see the relevance of this example. Multiplying Decimal infinity by another Decimal infinity is well-defined, and not an error. > Additionally, InvalidOperation is part of Decimal's custom exception handling > via the context, so it would be a bit odd if ValueError were part of that. Perhaps I am mistaken, but none of Decimal's custom exception handling would need to change. That's the beauty of multiple inheritence. If we add ValueError as a base class, InvalidOperation is a subclass of both DecimalException and ValueError. The custom exception handling will continue to work the same way it does now, and people who expect Decimal("spam") to raise ValueError will also be satisfied. -- Steve From stefan at bytereef.org Sun May 22 10:35:50 2016 From: stefan at bytereef.org (Stefan Krah) Date: Sun, 22 May 2016 14:35:50 +0000 (UTC) Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> Message-ID: Steven D'Aprano writes: [snipping excessively, since gmane.org started to lecture on this.] > The custom exception handling will continue to work the same way > it does now, and people who expect Decimal("spam") to raise ValueError > will also be satisfied. They won't be if they do setcontext(ExtendedContext). Stefan Krah From ncoghlan at gmail.com Sun May 22 11:14:36 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 23 May 2016 01:14:36 +1000 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> Message-ID: On 23 May 2016 at 00:35, Stefan Krah wrote: > Steven D'Aprano writes: > [snipping excessively, since gmane.org started to lecture on this.] >> The custom exception handling will continue to work the same way >> it does now, and people who expect Decimal("spam") to raise ValueError >> will also be satisfied. > > They won't be if they do setcontext(ExtendedContext). At that point they've explicitly asked for unknown strings to be converted to "NaN", and the behaviour won't be changed by Steven's proposal. However, while I can see Steven's point (folks expect to be able to catch malformed strings with ValueError), I don't believe the InvalidOperation would be the right place to adjust the hierarchy, as in terms of base classes, the decimal exceptions already inherit from ArithmeticError, which is the common base class of ZeroDivisionError, OverflowError, and FloatingPointError (which I would consider the closest analogy to decimal.InvalidOperation for binary floating point values), and those have the same behaviour of bypassing except clauses that catch ValueError: >>> def to_float(arg): ... try: ... return float(arg) ... except ValueError: ... return float("nan") ... >>> to_float(10*100) 1000.0 >>> to_float("spam") nan >>> to_float(10**10000) Traceback (most recent call last): File "", line 1, in File "", line 3, in to_float OverflowError: int too large to convert to float To address *just* the case of string conversion, it would suffice to introduce a dedicated decimal.ParsingError exception inheriting from both InvalidOperation and ValueError, specifically for failures converting from a string to a decimal value. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stefan at bytereef.org Sun May 22 11:27:34 2016 From: stefan at bytereef.org (Stefan Krah) Date: Sun, 22 May 2016 15:27:34 +0000 (UTC) Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> Message-ID: Nick Coghlan writes: > On 23 May 2016 at 00:35, Stefan Krah wrote: > > Steven D'Aprano ...> writes: > > [snipping excessively, since gmane.org started to lecture on this.] > >> The custom exception handling will continue to work the same way > >> it does now, and people who expect Decimal("spam") to raise ValueError > >> will also be satisfied. > > > > They won't be if they do setcontext(ExtendedContext). > > At that point they've explicitly asked for unknown strings to be > converted to "NaN", and the behaviour won't be changed by Steven's > proposal. Of course, but this thread was motivated by a question on the tutor mailing list. Another question "Why isn't ValueError raised if ..." may come up. The answer to all that is of course to read the documentation. :) Stefan Krah From random832 at fastmail.com Sun May 22 13:50:47 2016 From: random832 at fastmail.com (Random832) Date: Sun, 22 May 2016 13:50:47 -0400 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <573D7734.20402@canterbury.ac.nz> Message-ID: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> On Fri, May 20, 2016, at 15:25, Jim Baker wrote: > Agreed, if we want to support the structural pattern matching as seen in > Haskell, Rust, or Scala, we should do the following: > > - Raise a match error if the match is not exhaustive; an `else` clause > is fine for exhaustion This is required because the analogous constructs in those languages are expressions that return a value, and there's no reason to require it for a python construct which is a statement. From greg.ewing at canterbury.ac.nz Sun May 22 18:34:55 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 23 May 2016 10:34:55 +1200 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: <20160522133150.GS12028@ando.pearwood.info> References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> Message-ID: <5742340F.7080007@canterbury.ac.nz> >>Steven D'Aprano writes: >> >>>So I propose that InvalidOperation be changed to inherit from >>>ValueError, to match the expected behaviour from other numeric types. Related to this, is there any good reason that ArithmeticError doesn't derive from ValueError? If it did, it would fix this issue with Decimal and possibly others as well. I ran into this a while ago with a function like def convert(x, t): try: return t(x) except ValueError: ... and found that not all numeric types raise something derived from ValueError from their constructors. -- Greg From ncoghlan at gmail.com Sun May 22 22:56:40 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 23 May 2016 12:56:40 +1000 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: <5742340F.7080007@canterbury.ac.nz> References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> Message-ID: On 23 May 2016 at 08:34, Greg Ewing wrote: >>> Steven D'Aprano writes: >>> >>>> So I propose that InvalidOperation be changed to inherit from >>>> ValueError, to match the expected behaviour from other numeric types. > > > Related to this, is there any good reason that ArithmeticError > doesn't derive from ValueError? > > If it did, it would fix this issue with Decimal and possibly > others as well. > > I ran into this a while ago with a function like > > def convert(x, t): > try: > return t(x) > except ValueError: > ... > > and found that not all numeric types raise something derived from > ValueError from their constructors. I considered suggesting that, but got stuck on the question of how we quantify the compatibility implications - such a shift would change the behaviour of code that catches both, but handles them differently. That is, this code would be fine: try: ... except (ArithmeticError, ValueError): ... As would this: try: ... except ArithmeticError: ... except ValueError: ... But this would change to executing the first except clause rather than the second if the inheritance hierarchy changed: try: ... except ValueError: ... except ArithmeticError: ... While I can honestly say I've never seen an except clause catching ArithmeticError directly in the wild, I see more of a risk in the fact we'd potentially be silently changing the way ZeroDivisionError and OverflowError are handled. That said, this feels like the *right* answer to me, it's just the compatibility risk that worries me. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stefan at bytereef.org Mon May 23 04:11:10 2016 From: stefan at bytereef.org (Stefan Krah) Date: Mon, 23 May 2016 08:11:10 +0000 (UTC) Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> Message-ID: Greg Ewing writes: > >>>So I propose that InvalidOperation be changed to inherit from > >>>ValueError, to match the expected behaviour from other numeric types. > > Related to this, is there any good reason that ArithmeticError > doesn't derive from ValueError? For Decimal at least, Overflow, Underflow, Clamped etc. don't really map to ValueError. This includes InvalidOperation, which unfortunately *already* has a subclass ConversionSyntax for the use case that came up here. ConversionSyntax, however, is designated by the standard to be a "condition" rather than a "signal", so ConversionSyntax isn't raised -- it just exists, and in the C version you can see the "condition" in the list next to the exception that was raised. Adding a ValueError to all that just complicates things further. I wouldn't mind raising ConversionSyntax directly, but then again no one knows about that either and it's a small deviation from the standard. Stefan Krah From guettliml at thomas-guettler.de Mon May 23 05:43:41 2016 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Mon, 23 May 2016 11:43:41 +0200 Subject: [Python-ideas] Lib vs ( App | Project | ... ) In-Reply-To: <20160520113643.GO12028@ando.pearwood.info> References: <573AFBD5.6010902@thomas-guettler.de> <573C0DF3.1070000@thomas-guettler.de> <20160520113643.GO12028@ando.pearwood.info> Message-ID: <5742D0CD.4010700@thomas-guettler.de> Am 20.05.2016 um 13:36 schrieb Steven D'Aprano: > On Wed, May 18, 2016 at 08:38:43AM +0200, Thomas G?ttler wrote: > >> I see no agreement on the name for the concrete installation holding all >> those nice python libraries. > > I don't think you have made the case why we need such an agreement. > > What is the *actual problem* you are hoping to solve here? There is no actual problem. There is not uncaught exception and no segmentation fault. But still I fell that there is something wrong with "packaging python". And I guess the absence of an agreement on "app vs project vs site" is one of the roots of the problem. Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ From jsbueno at python.org.br Mon May 23 14:45:20 2016 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Mon, 23 May 2016 15:45:20 -0300 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> References: <573D7734.20402@canterbury.ac.nz> <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> Message-ID: I still fail to see what justifies violating The One Obvious Way to Do It which uses an if/elif sequence so blatantly beyond the feeble "Yet, some kind of switch statement is found in many languages and it is not unreasonable to expect that its addition to Python will allow us to write up certain code more cleanly and efficiently than before." in the rejected PEP. It is not like omitting the object reference name on the conditional expression (and therefore, a condition expression that is tied to one object) is more readable at all. But them, it is the BDLF cming with the proposal - I am just this Python develoepr teacher, that just last Wednesday was saying something along on a crash course: "We don need switch/case because it is just a special case of an if/elif sequence, that was meaningful when C compilers did not had resources to optimize that themselves". I argue no further, if people still see this as desirable.Ths is just my plain Python user "-1" on python-ideas. js -><- On 22 May 2016 at 14:50, Random832 wrote: > On Fri, May 20, 2016, at 15:25, Jim Baker wrote: >> Agreed, if we want to support the structural pattern matching as seen in >> Haskell, Rust, or Scala, we should do the following: >> >> - Raise a match error if the match is not exhaustive; an `else` clause >> is fine for exhaustion > > This is required because the analogous constructs in those languages are > expressions that return a value, and there's no reason to require it for > a python construct which is a statement. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rdmurray at bitdance.com Mon May 23 15:00:43 2016 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 23 May 2016 15:00:43 -0400 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: References: <20160517003511.AD7C9B14094@webabinitio.net> <573BF1EB.9030303@stoneleaf.us> <20160518174416.6D39EB1400B@webabinitio.net> <20160520103016.GN12028@ando.pearwood.info> Message-ID: <20160523190050.7A153B14028@webabinitio.net> On Fri, 20 May 2016 13:17:57 +0200, Piotr Balcer wrote: > That's an excellent observation ;) This is actually what happend when we > first wrote that game - there was a NULL-dereference in the game logic and > it caused a 'persistent segmentation fault'. It's really important for > programmers to step up when it comes to creating software that uses > this type of memory directly. Everyone essentially becomes a file system > developer, which is not necessarily a good thing. > > Our hope is that high level languages like python will also help in this > regard by making sure that the programmer cannot shoot himself in the foot > as easily as it is possible in C. > > Piotr > > 2016-05-20 12:30 GMT+02:00 Steven D'Aprano : > > > On Wed, May 18, 2016 at 01:44:12PM -0400, R. David Murray wrote: > > > > [...] > > > The main target would of > > > course be the data the program wants to preserve between runs, but one of > > > the examples at pmem.io is a game program whose state is all in nvram. > > > If the machine crashes in the middle of the game, upon restart you pick > > > up exactly where you left off, including all the positioning information > > > you might consider to be part of the "hot loop" part of the program. > > > > Wouldn't that mean you're restarting the game in the same state it was > > in just before it crashed? Which means (assuming the crash is > > deterministic) the very next thing the game will do is crash. Also note that that's a limited subset of the possible ways things might crash, even if it will likely be a common one in practice :) In the general case you'll need a way to clear the memory during development, and/or have a "test memory" setup that will be cleared before each test run and/or unit test. You'll also want ways to simulate crashes "at any point" (similar in principle to how Victor's memory tools can be used to simulate Python running out of memory "at any point"). You may need additional tools built into your application to manage the results of a crash. Exactly what tools you will want for updating the persistent state after various kinds of crashes is likely to be application specific, as is the "automatic recovery" code you'll build in to the application. I'm sure that we'll evolve best practices, test libraries, and helpful tools, for various problem domains, just as we have for working with disk-based databases. First we need to be able to write the code, though. --David From rdmurray at bitdance.com Mon May 23 15:33:13 2016 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 23 May 2016 15:33:13 -0400 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: References: <20160517003511.AD7C9B14094@webabinitio.net> <573BF1EB.9030303@stoneleaf.us> <20160518174416.6D39EB1400B@webabinitio.net> <20160520103016.GN12028@ando.pearwood.info> Message-ID: <20160523193314.7EEEAB14089@webabinitio.net> On Fri, 20 May 2016 16:06:03 -0400, Terry Reedy wrote: > On 5/20/2016 7:17 AM, Piotr Balcer wrote: > > [rearranged to undo top posting] > > > 2016-05-20 12:30 GMT+02:00 Steven D'Aprano > > Wouldn't that mean you're restarting the game in the same state it was > > in just before it crashed? Which means (assuming the crash is > > deterministic) the very next thing the game will do is crash. > > > That's an excellent observation ;) This is actually what happend when we > > first wrote that game - there was a NULL-dereference in the game logic and > > it caused a 'persistent segmentation fault'. It's really important for > > programmers to step up when it comes to creating software that uses > > this type of memory directly. Everyone essentially becomes a file system > > developer, which is not necessarily a good thing. > > > > Our hope is that high level languages like python will also help in this > > regard by making sure that the programmer cannot shoot himself in the foot > > as easily as it is possible in C. > > Unless one uses ctypes or buggy external C-coded modules, seg faulting > is already harder in Python than C. > > One possibility to make using persistent memory easier might be a > context manager. The __enter__ method would copy a persistent object to > a volatile object and return the volatile object. The body of the with > statement would manipulate the volatile object. *If there are no > exceptions*, the __exit__ method would 'commit' the changes, presumed to > be consisting, by copying back to the persistent object. > > with persistent(locator) as volatile: > > > Ignoring the possibility of a machine crash during the writeback, a > restart after an exception or crash during the block would start with > the persistent object as it was before the block, which absent bugs > would be a consistent state. Addressing consistency is what libpmemobj does, and what I'm designing the Python layer on top of it to do. But instead of copying the persistent data to volatile data and back again, what we have is a transaction scope inside of which libpmemobj records all object changes. If we never get to the end of the commit phase, then the changelog is used to roll back all of those changes, leaving the persistent objects in the state they were in before the transaction started. This applies across program restarts as well, which takes care of the "ignoring the possibility of a machine crash during writeback" (ie: that possibility is *not* ignored and dealing with it is in fact a primary goal of the design). So, we'll have: with persistent_store: and on exit from that code block either *all* of the changes are made and recorded, or *none* of them are, regardless of what happens inside the transaction, even if the machine crashes in the middle of the commit. Likewise, if a Python exception happens, the transaction commit is aborted, and the state of persistent memory is rolled back. I'm not coming up with anything that persistent->volatile->persistent copying would get you that these transactions don't get you, and transactions are more efficient because you don't have to copy the object data around or do the persistent/volatile conversions. There is one difference: in your version the changes to the volatile versions that happened before the python exception would still exist...but I don't see how that would be useful. On the other hand, the fact that *all* in-block persistent object state gets restored on block abort, regardless of where the exception occurred, could be somewhat confusing to Python programmers. --David From guido at python.org Mon May 23 15:43:29 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 23 May 2016 12:43:29 -0700 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <573D7734.20402@canterbury.ac.nz> <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> Message-ID: On Mon, May 23, 2016 at 11:45 AM, Joao S. O. Bueno wrote: > I still fail to see what justifies violating The One Obvious Way to Do It which > uses an if/elif sequence so blatantly beyond the > feeble "Yet, some kind of switch statement is found in many languages > and it is not unreasonable to expect that its addition to Python will > allow us to write up certain code more cleanly and efficiently than > before." in the rejected PEP. > > It is not like omitting the object reference name on the conditional > expression (and therefore, > a condition expression that is tied to one object) is more readable at all. > > But them, it is the BDLF cming with the proposal - I am just this > Python develoepr teacher, that just last Wednesday was saying > something along on a crash course: > "We don need switch/case because it is just a special case of an > if/elif sequence, > that was meaningful when C compilers did not had resources to optimize > that themselves". > > I argue no further, if people still see this as desirable.Ths is just my plain > Python user "-1" on python-ideas. Honestly I'm not at all convinced either! If it was easy to do all this with a sequence of if/elif clauses we wouldn't need it. The problem is that there are some types of matches that aren't so easy to write that way, e.g. combining an attempted tuple unpack with a guard, or "instance unpack" (check whether specific attributes exist) possibly combined with a guard. (The tricky thing is that the guard expression often needs to reference to the result of the unpacking.) It might become yet more interesting when combining the unpack syntax with type annotations, so you could check e.g. whether there's an x attribute that's a float and a y attribute that's a string. But it's about the most speculative piece of language design I've contemplated in a long time... -- --Guido van Rossum (python.org/~guido) From michael.selik at gmail.com Mon May 23 18:49:21 2016 From: michael.selik at gmail.com (Michael Selik) Date: Mon, 23 May 2016 22:49:21 +0000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <573D7734.20402@canterbury.ac.nz> <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> Message-ID: On Mon, May 23, 2016 at 3:58 PM Guido van Rossum wrote: > On Mon, May 23, 2016 at 11:45 AM, Joao S. O. Bueno > wrote: > > I still fail to see what justifies violating The One Obvious Way to Do > It which uses an if/elif sequence > > Honestly I'm not at all convinced either! If it was easy to do all > this with a sequence of if/elif clauses we wouldn't need it. The > problem is that there are some types of matches that aren't so easy to > write that way, e.g. combining an attempted tuple unpack with a guard, > or "instance unpack" (check whether specific attributes exist) > possibly combined with a guard. (The tricky thing is that the guard > expression often needs to reference to the result of the unpacking.) > Wouldn't an exception-catching expression (PEP 463) enable most of what you want with the existing if/elif/else chain? def demo(arg): if ((arg.x, arg.y) except AttributeError: False): print('x=', arg.x, 'y=', arg.y) elif ((arg[0], arg[1]) except IndexError: False): a, b, *_ = arg print('a=', a, 'b=', b) else: print('Too bad') To get everything you want, we could have an exception-catching assignment expression. While a plain ``=`` assignment would be problematic as an expression due to the common ``==`` typographical error, a less error-prone operator might be OK. I'd suggested ``?=`` earlier in the thread, but perhaps I didn't write out a good example at that time. Here's one that's closer to the original demo example. def demo(arg): if p, q ?= arg.x, arg.y: print('x=', p, 'y=', q) elif a, b, *_ ?= arg: print('a=', a, 'b=', b) else: print('Too bad') I figure it's better to solve the category of problems -- exception-catching expressions -- rather than the single problem of catching exceptions in an if/elif/else chain. Or do you think this is too much temptation for unreadable code golf? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon May 23 19:13:23 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 23 May 2016 16:13:23 -0700 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <573D7734.20402@canterbury.ac.nz> <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> Message-ID: On Mon, May 23, 2016 at 3:49 PM, Michael Selik wrote: > > > On Mon, May 23, 2016 at 3:58 PM Guido van Rossum wrote: >> >> On Mon, May 23, 2016 at 11:45 AM, Joao S. O. Bueno >> wrote: >> > I still fail to see what justifies violating The One Obvious Way to Do >> > It which uses an if/elif sequence >> >> Honestly I'm not at all convinced either! If it was easy to do all >> this with a sequence of if/elif clauses we wouldn't need it. The >> problem is that there are some types of matches that aren't so easy to >> write that way, e.g. combining an attempted tuple unpack with a guard, >> or "instance unpack" (check whether specific attributes exist) >> possibly combined with a guard. (The tricky thing is that the guard >> expression often needs to reference to the result of the unpacking.) > > > Wouldn't an exception-catching expression (PEP 463) enable most of what you > want with the existing if/elif/else chain? > > def demo(arg): > if ((arg.x, arg.y) except AttributeError: False): > print('x=', arg.x, 'y=', arg.y) > elif ((arg[0], arg[1]) except IndexError: False): > a, b, *_ = arg > print('a=', a, 'b=', b) > else: > print('Too bad') > > > To get everything you want, we could have an exception-catching assignment > expression. While a plain ``=`` assignment would be problematic as an > expression due to the common ``==`` typographical error, a less error-prone > operator might be OK. I'd suggested ``?=`` earlier in the thread, but > perhaps I didn't write out a good example at that time. Here's one that's > closer to the original demo example. > > def demo(arg): > if p, q ?= arg.x, arg.y: print('x=', p, 'y=', q) > elif a, b, *_ ?= arg: print('a=', a, 'b=', b) > else: print('Too bad') > > > I figure it's better to solve the category of problems -- exception-catching > expressions -- rather than the single problem of catching exceptions in an > if/elif/else chain. Or do you think this is too much temptation for > unreadable code golf? People are likely either going to put in exceptions that don't catch enough (e.g. IndexError isn't the only exception that example can throw) or, overreacting to that problm, that catch everything ("except Exception:" is an anti-pattern that's hard to fight). Plus there's the redundancy that you showed. -- --Guido van Rossum (python.org/~guido) From michael.selik at gmail.com Mon May 23 19:52:38 2016 From: michael.selik at gmail.com (Michael Selik) Date: Mon, 23 May 2016 23:52:38 +0000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <573D7734.20402@canterbury.ac.nz> <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> Message-ID: On Mon, May 23, 2016 at 7:13 PM Guido van Rossum wrote: > On Mon, May 23, 2016 at 3:49 PM, Michael Selik > wrote: > > > > > > On Mon, May 23, 2016 at 3:58 PM Guido van Rossum > wrote: > >> > >> On Mon, May 23, 2016 at 11:45 AM, Joao S. O. Bueno > >> wrote: > >> > I still fail to see what justifies violating The One Obvious Way to Do > >> > It which uses an if/elif sequence > >> > >> Honestly I'm not at all convinced either! If it was easy to do all > >> this with a sequence of if/elif clauses we wouldn't need it. The > >> problem is that there are some types of matches that aren't so easy to > >> write that way, e.g. combining an attempted tuple unpack with a guard, > >> or "instance unpack" (check whether specific attributes exist) > >> possibly combined with a guard. (The tricky thing is that the guard > >> expression often needs to reference to the result of the unpacking.) > > > > I figure it's better to solve the category of problems -- > exception-catching > > expressions -- rather than the single problem of catching exceptions in > an if/elif/else chain. > > People are likely either going to put in exceptions that don't catch > enough (e.g. IndexError isn't the only exception that example can > throw) or, overreacting to that problm, that catch everything ("except > Exception:" is an anti-pattern that's hard to fight). > It sounds like the justification for a switch/match syntax is to provide a special situation where generic Exception-catching assignment expressions are acceptable, because they're useful in a long elif chain but too dangerous for widespread use. Clearly it's beneficial enough to have appeared in other languages. However, languages like Haskell have already accepted the danger of assignment (and more) being an expression. Is there a language that makes the expression/statement distinction that has as powerful a matching syntax? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon May 23 20:22:55 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 23 May 2016 17:22:55 -0700 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <573D7734.20402@canterbury.ac.nz> <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> Message-ID: No, what I'm saying is that seeing it as sugar for exception handing is the wrong way to look at it. On Mon, May 23, 2016 at 4:52 PM, Michael Selik wrote: > > > On Mon, May 23, 2016 at 7:13 PM Guido van Rossum wrote: >> >> On Mon, May 23, 2016 at 3:49 PM, Michael Selik >> wrote: >> > >> > >> > On Mon, May 23, 2016 at 3:58 PM Guido van Rossum >> > wrote: >> >> >> >> On Mon, May 23, 2016 at 11:45 AM, Joao S. O. Bueno >> >> wrote: >> >> > I still fail to see what justifies violating The One Obvious Way to >> >> > Do >> >> > It which uses an if/elif sequence >> >> >> >> Honestly I'm not at all convinced either! If it was easy to do all >> >> this with a sequence of if/elif clauses we wouldn't need it. The >> >> problem is that there are some types of matches that aren't so easy to >> >> write that way, e.g. combining an attempted tuple unpack with a guard, >> >> or "instance unpack" (check whether specific attributes exist) >> >> possibly combined with a guard. (The tricky thing is that the guard >> >> expression often needs to reference to the result of the unpacking.) >> > >> > I figure it's better to solve the category of problems -- >> > exception-catching >> > expressions -- rather than the single problem of catching exceptions in >> > an if/elif/else chain. >> >> People are likely either going to put in exceptions that don't catch >> enough (e.g. IndexError isn't the only exception that example can >> throw) or, overreacting to that problm, that catch everything ("except >> Exception:" is an anti-pattern that's hard to fight). > > > It sounds like the justification for a switch/match syntax is to provide a > special situation where generic Exception-catching assignment expressions > are acceptable, because they're useful in a long elif chain but too > dangerous for widespread use. > > Clearly it's beneficial enough to have appeared in other languages. However, > languages like Haskell have already accepted the danger of assignment (and > more) being an expression. Is there a language that makes the > expression/statement distinction that has as powerful a matching syntax? -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Mon May 23 22:35:26 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 May 2016 12:35:26 +1000 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: <20160523193314.7EEEAB14089@webabinitio.net> References: <20160517003511.AD7C9B14094@webabinitio.net> <573BF1EB.9030303@stoneleaf.us> <20160518174416.6D39EB1400B@webabinitio.net> <20160520103016.GN12028@ando.pearwood.info> <20160523193314.7EEEAB14089@webabinitio.net> Message-ID: On 24 May 2016 at 05:33, R. David Murray wrote: > On the other hand, the fact that *all* in-block persistent object state > gets restored on block abort, regardless of where the exception occurred, > could be somewhat confusing to Python programmers. It occurs to me that you may want to talk to Mike Bayer and start digging into the way SQL Alchemy session objects work, as what you're describing here is starting to sound a *lot* like working with the SQL Alchemy ORM: - you have stateful objects declared as classes (SQL Alchemy models) - you have volatile state in memory (the SQL Alchemy session) - you have transactional commits to persistent storage (also the session) The difference would be that instead of working with a backing relational data store, you're working directly with persistent memory. It would also be interesting to see how much of this could be emulated with mmap and memoryview, permitting such code to have a slower fallback in cases where actual NVRAM wasn't available. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From michael.selik at gmail.com Mon May 23 22:57:39 2016 From: michael.selik at gmail.com (Michael Selik) Date: Tue, 24 May 2016 02:57:39 +0000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <573D7734.20402@canterbury.ac.nz> <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> Message-ID: On Mon, May 23, 2016 at 8:23 PM Guido van Rossum wrote: > No, what I'm saying is that seeing it as sugar for exception handing > is the wrong way to look at it. > Ok, let me refocus on the idea of assign-if-you-can during matching, not just sugar for try/except. The matching features of equality, inequality, predicates, and subtype are already available via if/elif. The matching features of destructuring, assignment, and post-assignment guards need new syntax. The other proposals in this thread suggest a new keyword or keyword pair like switch/case. Instead, why not extend if/elif with a new operator? def demo(arg): if p, q ?= arg.x, arg.y: # dict structure elif x ?= arg.x and isinstance(x, int) # assignment + guard elif a, b, *_ ?= arg: # tuple structure elif isinstance(arg, Mapping): # nothing new here A major advantage to merging these new matching features with the existing if/elif is that refactoring gets much easier. One could insert a single elif block near the top of the chain using the new assign-if-you-can operator/keyword, without needing to change any of the existing elif statements. Otherwise, I could imagine someone waffling back and forth between switch/case and if/elif as they add and remove cases. What would be the style recommendation if you have 5 boring cases that are all easy to read via if/elif? The disadvantage would be that the new syntax wouldn't stand out as much as it would with, say, switch/case, and someone might not notice its usage. I think a good choice of operator/keyword mitigates this. A keyword like ``as`` would stand out due to syntax highlighting. An operator like ``?=`` looks different enough and many similar-looking operators, like ``/=`` would be illegal. Operator precedence would affect the beauty of the code. Giving ``?=`` the same precedence as ``==`` seems best for post-assignment guards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon May 23 23:43:22 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 23 May 2016 20:43:22 -0700 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <573D7734.20402@canterbury.ac.nz> <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> Message-ID: On Mon, May 23, 2016 at 7:57 PM, Michael Selik wrote: > On Mon, May 23, 2016 at 8:23 PM Guido van Rossum wrote: >> >> No, what I'm saying is that seeing it as sugar for exception handing >> is the wrong way to look at it. > > > Ok, let me refocus on the idea of assign-if-you-can during matching, not > just sugar for try/except. > > The matching features of equality, inequality, predicates, and subtype are > already available via if/elif. The matching features of destructuring, > assignment, and post-assignment guards need new syntax. The other proposals > in this thread suggest a new keyword or keyword pair like switch/case. > Instead, why not extend if/elif with a new operator? > > def demo(arg): > if p, q ?= arg.x, arg.y: # dict structure > elif x ?= arg.x and isinstance(x, int) # assignment + guard > elif a, b, *_ ?= arg: # tuple structure > elif isinstance(arg, Mapping): # nothing new here > > > A major advantage to merging these new matching features with the existing > if/elif is that refactoring gets much easier. One could insert a single elif > block near the top of the chain using the new assign-if-you-can > operator/keyword, without needing to change any of the existing elif > statements. > > Otherwise, I could imagine someone waffling back and forth between > switch/case and if/elif as they add and remove cases. What would be the > style recommendation if you have 5 boring cases that are all easy to read > via if/elif? > > The disadvantage would be that the new syntax wouldn't stand out as much as > it would with, say, switch/case, and someone might not notice its usage. I > think a good choice of operator/keyword mitigates this. A keyword like > ``as`` would stand out due to syntax highlighting. An operator like ``?=`` > looks different enough and many similar-looking operators, like ``/=`` would > be illegal. > > Operator precedence would affect the beauty of the code. Giving ``?=`` the > same precedence as ``==`` seems best for post-assignment guards. This idea has some clear advantages -- it's unambiguous about the order of matching, and combines clearly with existing conditions. It also seems like it would support "recursive" matching nicely, by allowing you to chain additional unpacking operators. ("Recursive" because IIUC that's what Haskell calls matches inside matches -- similar to nested tuple unpackings like (a, (b, c)) = in Python.) The trick is to find a good syntax for the conditional assignment; using ? has been rejected by this group in the past for other conditionalisms. -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Tue May 24 00:35:51 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 May 2016 14:35:51 +1000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <573D7734.20402@canterbury.ac.nz> <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> Message-ID: On 24 May 2016 at 13:43, Guido van Rossum wrote: > On Mon, May 23, 2016 at 7:57 PM, Michael Selik wrote: >> Operator precedence would affect the beauty of the code. Giving ``?=`` the >> same precedence as ``==`` seems best for post-assignment guards. > > This idea has some clear advantages -- it's unambiguous about the > order of matching, and combines clearly with existing conditions. It > also seems like it would support "recursive" matching nicely, by > allowing you to chain additional unpacking operators. ("Recursive" > because IIUC that's what Haskell calls matches inside matches -- > similar to nested tuple unpackings like (a, (b, c)) = in Python.) > > The trick is to find a good syntax for the conditional assignment; > using ? has been rejected by this group in the past for other > conditionalisms. I don't think we've ever considered it specifically in the context of a "maybe-assign" expression, though. While such a conditional assignment syntax does sound appealing as a way to do runtime pattern matching, one key concern I'd have with it would be how it's defined in the case of a "can't fail" assignment to a single variable: if x ?= re.match(pattern, text): # Can x be None here? while x ?= expr(): # Can x be any non-truthy expression here? Assuming binding to a name was a "can't fail" operation that always returned either "True" or the LHS as a tuple (so assigning to a single name resulted in a 1-tuple, while only a failed assignment resulted in an empty tuple), then the correct spelling for those operations would be: if x ?= re.match(pattern, text) and x is not None: # x is definitely not None here while x ?= expr() and x: # x is always truthy The other tricky problem would be defining precisely how exception handling worked on the RHS. For iterable unpacking, clearly the TypeError from the implicit iter() call would be suppressed (if thrown), as would errors related to mismatches between the number of items and the number of unpacking targets, but catching arbitrary AttributeErrors from the RHS to support conditional attribute based destructuring assignment would be problematic. That suggests to me that some kind of explicit syntax for attribute based destructuring may still be needed to avoid overbroad suppression of exceptions. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From greg.ewing at canterbury.ac.nz Tue May 24 02:08:11 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 24 May 2016 18:08:11 +1200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> Message-ID: <5743EFCB.40401@canterbury.ac.nz> > On Mon, May 23, 2016 at 7:57 PM, Michael Selik wrote: > >>def demo(arg): >> if p, q ?= arg.x, arg.y: # dict structure >> elif x ?= arg.x and isinstance(x, int) # assignment + guard >> elif a, b, *_ ?= arg: # tuple structure >> elif isinstance(arg, Mapping): # nothing new here I'm unenthusiastic about this -- the above looks like an unreadable mess to me. Some other worries: * It appears that this would have to work by means of exception-catching in the rhs, so it's really a disguised form of the except: expression proposal; is that intentional? * It could also be abused to get an assigment-in- expression anywhere you wanted. Such things have been rejected before; are we changing our mind on that? * THere would be no hope of checking for coverage of all cases in a static checker. I feel like we've wandered a long way from the idea we started with, which is something to facilitate a Haskell-like case-analysis style of programming, and ended up with something much looser that tries to be all things to everyone. -- Greg From mal at egenix.com Tue May 24 03:59:42 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 24 May 2016 09:59:42 +0200 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: References: <20160517003511.AD7C9B14094@webabinitio.net> <573BF1EB.9030303@stoneleaf.us> <20160518174416.6D39EB1400B@webabinitio.net> <20160520103016.GN12028@ando.pearwood.info> <20160523193314.7EEEAB14089@webabinitio.net> Message-ID: <574409EE.6090407@egenix.com> On 24.05.2016 04:35, Nick Coghlan wrote: > On 24 May 2016 at 05:33, R. David Murray wrote: >> On the other hand, the fact that *all* in-block persistent object state >> gets restored on block abort, regardless of where the exception occurred, >> could be somewhat confusing to Python programmers. > > It occurs to me that you may want to talk to Mike Bayer and start > digging into the way SQL Alchemy session objects work, as what you're > describing here is starting to sound a *lot* like working with the SQL > Alchemy ORM: > > - you have stateful objects declared as classes (SQL Alchemy models) > - you have volatile state in memory (the SQL Alchemy session) > - you have transactional commits to persistent storage (also the session) > > The difference would be that instead of working with a backing > relational data store, you're working directly with persistent memory. > It would also be interesting to see how much of this could be emulated > with mmap and memoryview, permitting such code to have a slower > fallback in cases where actual NVRAM wasn't available. SQLAlchemy relies on underlying Python DB-API modules to work with the database, so in addition to the above you also have state in objects of those DB-API modules and these usually reference objects or buffers in the low-level database interface libraries to which Python has no direct access. As soon as you have memory in use which is not fully managed by Python, I don't think there's any way to implement transactions on memory in a meaningful way. The possible side effect in the unmanaged blocks would render such transactions meaningless, since a rollback in those would still leave you with the changes in the unmanaged blocks (other parts of the system). Now, back on topic: for writing to NVRAM, having a transaction mechanism in place does make sense, but it would have to be clear that only the bits stored in NVRAM are subject to the transaction. The reason here being that a failure while writing to NVRAM could potentially cause your machine to no longer boot. For volatile RAM, at worst, the process will die, but not have much effect on other running parts of the system, so there is less incentive to have transactions (unless, of course, you are deep into STM and want to work around the GIL :-)). Given that Armin Rigo has been working on STM for years, I'd suggest to talk to him about challenges and solutions for transactions on memory. My take on all this would be to work with NVRAM as block rather than single memory cells: allocate a lock on the NVRAM block try: copy the block into DRAM run manipulations in DRAM block write back DRAM block finally: release lock on NVRAM block so instead of worrying about a transaction failing while manipulating NVRAM, you only make sure that you can lock the NVRAM block and provide an atomic "block write to NVRAM" functionality. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 24 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From p.f.moore at gmail.com Tue May 24 04:25:02 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 May 2016 09:25:02 +0100 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <5743EFCB.40401@canterbury.ac.nz> References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> Message-ID: On 24 May 2016 at 07:08, Greg Ewing wrote: >> On Mon, May 23, 2016 at 7:57 PM, Michael Selik >> wrote: >> >>> def demo(arg): >>> if p, q ?= arg.x, arg.y: # dict structure >>> elif x ?= arg.x and isinstance(x, int) # assignment + guard >>> elif a, b, *_ ?= arg: # tuple structure >>> elif isinstance(arg, Mapping): # nothing new here > > > I'm unenthusiastic about this -- the above looks like > an unreadable mess to me. Agreed - the ?= operator looks like noise to me. One of the advantages of a switch/case statement is that it's keyword based, which IMO is inherently more readable. If we keep getting sucked into comparisons with "chain of if" constructs, maybe the problem is that the "switch" keyword is too strongly associated with that in people's minds? Maybe we could focus on the fact that it's matching that we're doing and make it a "match" statement? So match expr: argspec: xxx argspec: yyy else: default Whether we want to simply make "argspec" do standard tuple unpacking, or something a little more complex like attribute checking, can be up for debate. I'm inclined to start small and see if there's any value in a simple "try various unpackings" construct first. One big problem with this is that making "match" a keyword would be a real pain - I bet there's lots of code that uses "match" as a variable... Paul From mal at egenix.com Tue May 24 04:27:04 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 24 May 2016 10:27:04 +0200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <5743EFCB.40401@canterbury.ac.nz> References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> Message-ID: <57441058.3000709@egenix.com> On 24.05.2016 08:08, Greg Ewing wrote: >> On Mon, May 23, 2016 at 7:57 PM, Michael Selik >> wrote: >> >>> def demo(arg): >>> if p, q ?= arg.x, arg.y: # dict structure >>> elif x ?= arg.x and isinstance(x, int) # assignment + guard >>> elif a, b, *_ ?= arg: # tuple structure >>> elif isinstance(arg, Mapping): # nothing new here > > I'm unenthusiastic about this -- the above looks like > an unreadable mess to me. Seconded. It would be more readable to make the assignment explicit: if (p, q = arg.x, arg.y): ... but even then I find this error prone. Just think of the common typo of writing "if x = 1: ..." instead of "if x == 1: ...". This version p, q = arg.x, arg.y if (p, q): ... is just a bit more verbose, but comes without all the problems of having to allow implicit assignment-in-expression everywhere. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 24 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From walker_s at hotmail.co.uk Tue May 24 05:31:24 2016 From: walker_s at hotmail.co.uk (SW) Date: Tue, 24 May 2016 10:31:24 +0100 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <5743EFCB.40401@canterbury.ac.nz> References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> Message-ID: On 24/05/16 07:08, Greg Ewing wrote: >> On Mon, May 23, 2016 at 7:57 PM, Michael Selik >> wrote: >> >>> def demo(arg): >>> if p, q ?= arg.x, arg.y: # dict structure >>> elif x ?= arg.x and isinstance(x, int) # assignment + guard >>> elif a, b, *_ ?= arg: # tuple structure >>> elif isinstance(arg, Mapping): # nothing new here > > I'm unenthusiastic about this -- the above looks like > an unreadable mess to me. Agreed. S From k7hoven at gmail.com Tue May 24 08:03:17 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 24 May 2016 15:03:17 +0300 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <573D7734.20402@canterbury.ac.nz> <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> Message-ID: On Tue, May 24, 2016 at 6:43 AM, Guido van Rossum wrote: > On Mon, May 23, 2016 at 7:57 PM, Michael Selik wrote: >> Ok, let me refocus on the idea of assign-if-you-can during matching, not >> just sugar for try/except. >> >> The matching features of equality, inequality, predicates, and subtype are >> already available via if/elif. The matching features of destructuring, >> assignment, and post-assignment guards need new syntax. The other proposals >> in this thread suggest a new keyword or keyword pair like switch/case. >> Instead, why not extend if/elif with a new operator? >> >> def demo(arg): >> if p, q ?= arg.x, arg.y: # dict structure >> elif x ?= arg.x and isinstance(x, int) # assignment + guard >> elif a, b, *_ ?= arg: # tuple structure >> elif isinstance(arg, Mapping): # nothing new here >> >> [...] > This idea has some clear advantages -- it's unambiguous about the > order of matching, and combines clearly with existing conditions. It > also seems like it would support "recursive" matching nicely, by > allowing you to chain additional unpacking operators. ("Recursive" > because IIUC that's what Haskell calls matches inside matches -- > similar to nested tuple unpackings like (a, (b, c)) = in Python.) > > The trick is to find a good syntax for the conditional assignment; > using ? has been rejected by this group in the past for other > conditionalisms. Yeah, and indeed it's not obvious how to draw the line between whether the conditional assignment should return a truth value or raise an exception, as I think Greg was already implying. For instance, what would `a ?= data[0]` do, if data[0] raises, say, a TypeError. Should it be caught or be raised? Anyway, maybe it is possible to draw that line in a reasonable way. So here's a ?nother syntax : Introduce a ?new ? keyword `given`, which ?could be used as a prefix operator with roughly the following properties given a ? ? # True if the name a is bound to something given b.c ? ? # roughly equivalent to hasattr(b, 'c') given a, b.c ? ? # same as (given a and given b.c) given d, e = a, b.c ? ? # like `given a, b.c` but with added assignment Then one could do ( ?expanding on the above demo example) def demo(arg): if given p, q = arg.x, arg.y: # do something elif given x = arg.x and isinstance(x, int): # do somet??hing elif given a, b, *_ = arg: # do somet?h?ing elif isinstance(arg, Mapping): # do something ?# Or something like this: ? ? def demo2(arg):? ? global importan?t_value ? ? ? if not given important_value: important_value = compute_important_value() ?-- Koos? ?PS. I like seeing the word "brainstorm" used here. I'm not sure if I've seen that here before. I think that means that we try to see the good parts of the presented ideas and send further ideas and thoughts to see if we can find something useful -- even if the presented ideas are not yet optimal. (Of course that does not mean that the ideas should not be presented clearly!) Anyway, in the end, if nothing is found, it is easy to just drop the whole concept. ? > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue May 24 08:10:34 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 May 2016 22:10:34 +1000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> Message-ID: On 24 May 2016 at 18:25, Paul Moore wrote: > On 24 May 2016 at 07:08, Greg Ewing wrote: >>> On Mon, May 23, 2016 at 7:57 PM, Michael Selik >>> wrote: >>> >>>> def demo(arg): >>>> if p, q ?= arg.x, arg.y: # dict structure >>>> elif x ?= arg.x and isinstance(x, int) # assignment + guard >>>> elif a, b, *_ ?= arg: # tuple structure >>>> elif isinstance(arg, Mapping): # nothing new here >> >> >> I'm unenthusiastic about this -- the above looks like >> an unreadable mess to me. > > Agreed - the ?= operator looks like noise to me. One of the advantages > of a switch/case statement is that it's keyword based, which IMO is > inherently more readable. > > If we keep getting sucked into comparisons with "chain of if" > constructs, maybe the problem is that the "switch" keyword is too > strongly associated with that in people's minds? Maybe we could focus > on the fact that it's matching that we're doing and make it a "match" > statement? So > > match expr: > argspec: xxx > argspec: > yyy > else: > default New keywords that collide with standard library APIs (re.match in this case) are pretty much right out :) I suspect you're right that switch/case have too much baggage to be viable candidates for this use case, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue May 24 09:26:07 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 May 2016 23:26:07 +1000 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: <574409EE.6090407@egenix.com> References: <20160517003511.AD7C9B14094@webabinitio.net> <573BF1EB.9030303@stoneleaf.us> <20160518174416.6D39EB1400B@webabinitio.net> <20160520103016.GN12028@ando.pearwood.info> <20160523193314.7EEEAB14089@webabinitio.net> <574409EE.6090407@egenix.com> Message-ID: On 24 May 2016 at 17:59, M.-A. Lemburg wrote: > On 24.05.2016 04:35, Nick Coghlan wrote: >> On 24 May 2016 at 05:33, R. David Murray wrote: >>> On the other hand, the fact that *all* in-block persistent object state >>> gets restored on block abort, regardless of where the exception occurred, >>> could be somewhat confusing to Python programmers. >> >> It occurs to me that you may want to talk to Mike Bayer and start >> digging into the way SQL Alchemy session objects work, as what you're >> describing here is starting to sound a *lot* like working with the SQL >> Alchemy ORM: >> >> - you have stateful objects declared as classes (SQL Alchemy models) >> - you have volatile state in memory (the SQL Alchemy session) >> - you have transactional commits to persistent storage (also the session) >> >> The difference would be that instead of working with a backing >> relational data store, you're working directly with persistent memory. >> It would also be interesting to see how much of this could be emulated >> with mmap and memoryview, permitting such code to have a slower >> fallback in cases where actual NVRAM wasn't available. > > SQLAlchemy relies on underlying Python DB-API modules to > work with the database, so in addition to the above you > also have state in objects of those DB-API modules and > these usually reference objects or buffers in the low-level > database interface libraries to which Python has no direct > access. I'm not talking about using SQL Alchemy itself for this (as you say, the implementation details of the underlying persistence model are all wrong), I'm talking about studying the way SQL Alchemy session specifically, and the way it manages the in-memory state while a transaction is open, and then flushes that state to the database when the transaction is completed. It's a thought prompted by a presentation Mike gave at PyCon US 2013: https://speakerdeck.com/pyconslides/sqlalchemy-session-in-depth-by-mike-bayer Now, David might investigate that, and decide it's more complexity than is needed given persistent RAM, but it's the nearest already-in-existence thing I know to what I believe he's aiming to build, and it also handles the distinction between persistent state (database row for SQL Alchemy, memory allocated in NVRAM for this) and ephemeral state (elements dynamically allocated in normal RAM for both). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue May 24 09:27:56 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 May 2016 23:27:56 +1000 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: References: <20160517003511.AD7C9B14094@webabinitio.net> <573BF1EB.9030303@stoneleaf.us> <20160518174416.6D39EB1400B@webabinitio.net> <20160520103016.GN12028@ando.pearwood.info> <20160523193314.7EEEAB14089@webabinitio.net> <574409EE.6090407@egenix.com> Message-ID: On 24 May 2016 at 23:26, Nick Coghlan wrote: > I'm not talking about using SQL Alchemy itself for this (as you say, > the implementation details of the underlying persistence model are all > wrong), I'm talking about studying the way SQL Alchemy session > specifically, and the way it manages the in-memory state while a > transaction is open, and then flushes that state to the database when > the transaction is completed. It's a thought prompted by a > presentation Mike gave at PyCon US 2013: > https://speakerdeck.com/pyconslides/sqlalchemy-session-in-depth-by-mike-bayer > > Now, David might investigate that, and decide it's more complexity > than is needed given persistent RAM, but it's the nearest > already-in-existence thing I know to what I believe he's aiming to > build, and it also handles the distinction between persistent state > (database row for SQL Alchemy, memory allocated in NVRAM for this) and > ephemeral state (elements dynamically allocated in normal RAM for > both). Hah, now there's an interesting idea: you actually *could* use SQL Alchemy itself with an sqlite in-memory database, but ensure the latter was allocated in NVRAM :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From michael.selik at gmail.com Tue May 24 09:31:40 2016 From: michael.selik at gmail.com (Michael Selik) Date: Tue, 24 May 2016 13:31:40 +0000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <5743EFCB.40401@canterbury.ac.nz> References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> Message-ID: On Tue, May 24, 2016 at 2:08 AM Greg Ewing wrote: > > On Mon, May 23, 2016 at 7:57 PM, Michael Selik > wrote: > > > >>def demo(arg): > >> if p, q ?= arg.x, arg.y: # dict structure > >> elif x ?= arg.x and isinstance(x, int) # assignment + guard > >> elif a, b, *_ ?= arg: # tuple structure > >> elif isinstance(arg, Mapping): # nothing new here > > I'm unenthusiastic about this -- the above looks like > an unreadable mess to me. > Which is unreadable: ``if/elif`` keywords or ``?=`` operator? If it's the latter, please don't get hung up on that, as I intended that as a placeholder for whatever operator or keyword is best. My main point is that switch/case/matching is semantically identical to if/elif, so why use something different? > Some other worries: > > * It appears that this would have to work by means of > exception-catching in the rhs, so it's really a > disguised form of the except: expression proposal; > is that intentional? > No. The exception-catching expression is not meant to be available in any other context. However, exception-catching is an essential feature to matching via destructuring. I believe it's unavoidable, though *which* exceptions are suppressed could be made more clear. * It could also be abused to get an assigment-in- > expression anywhere you wanted. Such things have > been rejected before; are we changing our mind > on that? > No, using the assign-if-can operator would be syntax error outside of an if/elif, the reverse of how an assign operator is a syntax error inside an if/elif. * THere would be no hope of checking for coverage > of all cases in a static checker. > I believe you, though the logic isn't clear to me. Could you explain why? > I feel like we've wandered a long way from the idea > we started with, which is something to facilitate > a Haskell-like case-analysis style of programming, > and ended up with something much looser that tries > to be all things to everyone. > That was not my intention. If the assign-if-can operation is only available in an if/elif statement, I think it's exactly Haskell-like case-analysis and nothing more. Did I miss something? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue May 24 09:55:00 2016 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 24 May 2016 09:55:00 -0400 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: References: <20160517003511.AD7C9B14094@webabinitio.net> <573BF1EB.9030303@stoneleaf.us> <20160518174416.6D39EB1400B@webabinitio.net> <20160520103016.GN12028@ando.pearwood.info> <20160523193314.7EEEAB14089@webabinitio.net> Message-ID: <20160524135502.920BAB14027@webabinitio.net> On Tue, 24 May 2016 12:35:26 +1000, Nick Coghlan wrote: > On 24 May 2016 at 05:33, R. David Murray wrote: > > On the other hand, the fact that *all* in-block persistent object state > > gets restored on block abort, regardless of where the exception occurred, > > could be somewhat confusing to Python programmers. > > It occurs to me that you may want to talk to Mike Bayer and start > digging into the way SQL Alchemy session objects work, as what you're > describing here is starting to sound a *lot* like working with the SQL > Alchemy ORM: > > - you have stateful objects declared as classes (SQL Alchemy models) > - you have volatile state in memory (the SQL Alchemy session) > - you have transactional commits to persistent storage (also the session) > > The difference would be that instead of working with a backing > relational data store, you're working directly with persistent memory. There are similarities, so yes it will probably be helpful to look over sqlalchemy transactions in some detail. I've certainly been thinking about them while working on the design of the python layer. > It would also be interesting to see how much of this could be emulated > with mmap and memoryview, permitting such code to have a slower > fallback in cases where actual NVRAM wasn't available. The underlying library itself handles the no-NVRAM fallback using regular mmapped files. --David From k7hoven at gmail.com Tue May 24 10:03:51 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 24 May 2016 17:03:51 +0300 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> Message-ID: On Tue, May 24, 2016 at 4:31 PM, Michael Selik wrote: > > On Tue, May 24, 2016 at 2:08 AM Greg Ewing wrote: [...] >> * It could also be abused to get an assigment-in- >> expression anywhere you wanted. Such things have >> been rejected before; are we changing our mind >> on that? > > No, using the assign-if-can operator would be syntax error outside of an > if/elif, the reverse of how an assign operator is a syntax error inside an if/elif. I might be stating the obvious, but the `given` expression syntax could of course also be allowed only inside if/elif, even if I didn't include that. >> * THere would be no hope of checking for coverage >> of all cases in a static checker. > > I believe you, though the logic isn't clear to me. Could you explain why? > It's not clear to me either. Maybe there is hope, with a sophisticated enough checker at some point in the future? -- Koos From rdmurray at bitdance.com Tue May 24 10:17:55 2016 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 24 May 2016 10:17:55 -0400 Subject: [Python-ideas] Application awareness of memory storage classes In-Reply-To: <574409EE.6090407@egenix.com> References: <20160517003511.AD7C9B14094@webabinitio.net> <573BF1EB.9030303@stoneleaf.us> <20160518174416.6D39EB1400B@webabinitio.net> <20160520103016.GN12028@ando.pearwood.info> <20160523193314.7EEEAB14089@webabinitio.net> <574409EE.6090407@egenix.com> Message-ID: <20160524141757.27BFCB14027@webabinitio.net> On Tue, 24 May 2016 09:59:42 +0200, "M.-A. Lemburg" wrote: > As soon as you have memory in use which is not fully managed > by Python, I don't think there's any way to implement > transactions on memory in a meaningful way. The possible side > effect in the unmanaged blocks would render such transactions > meaningless, since a rollback in those would still leave you > with the changes in the unmanaged blocks (other parts of the > system). In this case all the memory will "managed" at the direction of the Python program (and the extension module). The issue is that while we have transactions on the NVRAM objects, the regular python objects don't get their state restored if the transaction block aborts. Which is part of why I was wondering about what it might look like to integrate awareness of storage classes into the language itself. > Now, back on topic: for writing to NVRAM, having a transaction > mechanism in place does make sense, but it would have to > be clear that only the bits stored in NVRAM are subject > to the transaction. Yes, exactly. > The reason here being that a failure while writing to NVRAM > could potentially cause your machine to no longer boot. I think you misunderstand. We're not talking about "regular" NVRAM, we're talking about memory banks that are exposed to user space via a DAX driver that uses file system semantics to set up the mapping from user space to the NVRAM, but after that some kernel magic allows the user space program to write directly to the NVRAM. We're not doing this with the NVRAM involved in booting the machine, it is separate dedicated storage. > For volatile RAM, at worst, the process will die, but not have > much effect on other running parts of the system, so there > is less incentive to have transactions (unless, of course, > you are deep into STM and want to work around the GIL :-)). STM is a different approach, and equally valid, but not the one the underlying library takes. > Given that Armin Rigo has been working on STM for years, > I'd suggest to talk to him about challenges and solutions > for transactions on memory. He's looking at what we might call the reverse of the type of transaction I'm dealing with. An STM transaction makes all changes pending, and throws them away on conflict. Our transaction makes all changes immediately, and *rolls them back* on *failure*. No conflicts are involved, so the things you have to worry about are different from the things you have to worry about in the STM case. I'm sure there are some commonalities, so it may well be worth talking to Armin, since he's thought deeply about this stuff. I'm being handed the transaction machinery by the underlying library, though, so I "only" have to think about how it impacts the Python level :) > My take on all this would be to work with NVRAM as block > rather than single memory cells: > > allocate a lock on the NVRAM block > try: > copy the block into DRAM > run manipulations in DRAM block > write back DRAM block > finally: > release lock on NVRAM block > > so instead of worrying about a transaction failing while > manipulating NVRAM, you only make sure that you can lock > the NVRAM block and provide an atomic "block write to NVRAM" > functionality. Which is the reverse of what the library actually does. It copies the existing data into an NVRAM rollback log, and then makes the changes to the visible memory (that is, the changes are immediately visible to all threads). The rollback log is then used to undo those changes if the transaction fails. And yes, this means that you need locks around your persistent object updates when doing threaded programming, as I mentioned in my original post. I'm personally also interested in the STM-style case, since that allows you to write multi-access, potentially distributed, DB-like applications. However, that's not what this particular project is about. A language that is supporting persistent storage should support both models, I think, because both are useful for different applications. But the primary difference is what happens during a transaction, so at the language syntax level there is probably no difference. I guess that means there are two different classes of persistent memory from the application's point of view, even if they can be backed by the same physical memory: rollback persistent, and STM persistent. --David From p.f.moore at gmail.com Tue May 24 10:35:04 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 May 2016 15:35:04 +0100 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> Message-ID: On 24 May 2016 at 14:31, Michael Selik wrote: > On Tue, May 24, 2016 at 2:08 AM Greg Ewing > wrote: >> >> > On Mon, May 23, 2016 at 7:57 PM, Michael Selik >> > wrote: >> > >> >>def demo(arg): >> >> if p, q ?= arg.x, arg.y: # dict structure >> >> elif x ?= arg.x and isinstance(x, int) # assignment + guard >> >> elif a, b, *_ ?= arg: # tuple structure >> >> elif isinstance(arg, Mapping): # nothing new here >> >> I'm unenthusiastic about this -- the above looks like >> an unreadable mess to me. > > > Which is unreadable: ``if/elif`` keywords or ``?=`` operator? If it's the > latter, please don't get hung up on that, as I intended that as a > placeholder for whatever operator or keyword is best. My main point is that > switch/case/matching is semantically identical to if/elif, so why use > something different? For me, it's the use of operators (punctuation) plus the fact that the overall layout doesn't in any way imply to me "I'm looking at *this* expression. Let's try a sequence of matches..." The key distinguishing feature for me of a switch/match statement is that it organises the "flow" of the statement differently from if/elif. The if/elif chain says "try this, then that, now something else". There's no implication that the tests are all looking at the same subject - to spot that, you need to be able to see (from the layout of the tests) the actual subject item on each line, and the ?= operator syntax obscures that because "arg" is used differently in each test. With a switch statement, however, the subject is stated once, at the top of the statement. The checks are then listed one after the other, and they are all by definition checks against the subject expression. And in fact, for me that's the distinguishing feature of a switch statement - that it's a series of tests against a single implied subject. That's also (I suspect) why it's hard to come up with a good syntax for tests - Python doesn't really do "implied subjects" anywhere else (explicit is better than implicit and all that). IMO, "Series of tests against a single expression" is a common enough pattern to be worth exploring in spite of the fact that it runs counter to EIBTI. But you may need to be Dutch to understand why :-) Paul From desmoulinmichel at gmail.com Tue May 24 10:59:55 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 24 May 2016 16:59:55 +0200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> Message-ID: <57446C6B.3030503@gmail.com> What about making it a function ? Pattern matching is complex, it can be like a mini language inside the language, just like regex. To match text we do: import re re.match('pattern', 'string') We could do the same for matching structures: from inspect import match def do(stuff): m = match(stuff): # m implements __call__ if m('whatever mini language you like'): return foo if m('again'): return m.thing_your_extracted Pros: - no need for a new syntax - very explicit - use well know constructs - we can easily start experimenting with the matching language in a lib in Pypy Cons: - much more verbose; - debugging your mini language and handling error is not as good; Le 24/05/2016 16:35, Paul Moore a ?crit : > On 24 May 2016 at 14:31, Michael Selik wrote: >> On Tue, May 24, 2016 at 2:08 AM Greg Ewing >> wrote: >>> >>>> On Mon, May 23, 2016 at 7:57 PM, Michael Selik >>>> wrote: >>>> >>>>> def demo(arg): >>>>> if p, q ?= arg.x, arg.y: # dict structure >>>>> elif x ?= arg.x and isinstance(x, int) # assignment + guard >>>>> elif a, b, *_ ?= arg: # tuple structure >>>>> elif isinstance(arg, Mapping): # nothing new here >>> >>> I'm unenthusiastic about this -- the above looks like >>> an unreadable mess to me. >> >> >> Which is unreadable: ``if/elif`` keywords or ``?=`` operator? If it's the >> latter, please don't get hung up on that, as I intended that as a >> placeholder for whatever operator or keyword is best. My main point is that >> switch/case/matching is semantically identical to if/elif, so why use >> something different? > > For me, it's the use of operators (punctuation) plus the fact that the > overall layout doesn't in any way imply to me "I'm looking at *this* > expression. Let's try a sequence of matches..." > > The key distinguishing feature for me of a switch/match statement is > that it organises the "flow" of the statement differently from > if/elif. The if/elif chain says "try this, then that, now something > else". There's no implication that the tests are all looking at the > same subject - to spot that, you need to be able to see (from the > layout of the tests) the actual subject item on each line, and the ?= > operator syntax obscures that because "arg" is used differently in > each test. > > With a switch statement, however, the subject is stated once, at the > top of the statement. The checks are then listed one after the other, > and they are all by definition checks against the subject expression. > And in fact, for me that's the distinguishing feature of a switch > statement - that it's a series of tests against a single implied > subject. That's also (I suspect) why it's hard to come up with a good > syntax for tests - Python doesn't really do "implied subjects" > anywhere else (explicit is better than implicit and all that). > > IMO, "Series of tests against a single expression" is a common enough > pattern to be worth exploring in spite of the fact that it runs > counter to EIBTI. But you may need to be Dutch to understand why :-) > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From desmoulinmichel at gmail.com Tue May 24 10:59:55 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 24 May 2016 16:59:55 +0200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> Message-ID: <57446C6B.3030704@gmail.com> What about making it a function ? Pattern matching is complex, it can be like a mini language inside the language, just like regex. To match text we do: import re re.match('pattern', 'string') We could do the same for matching structures: from inspect import match def do(stuff): m = match(stuff): # m implements __call__ if m('whatever mini language you like'): return foo if m('again'): return m.thing_your_extracted Pros: - no need for a new syntax - very explicit - use well know constructs - we can easily start experimenting with the matching language in a lib in Pypy Cons: - much more verbose; - debugging your mini language and handling error is not as good; Le 24/05/2016 16:35, Paul Moore a ?crit : > On 24 May 2016 at 14:31, Michael Selik wrote: >> On Tue, May 24, 2016 at 2:08 AM Greg Ewing >> wrote: >>> >>>> On Mon, May 23, 2016 at 7:57 PM, Michael Selik >>>> wrote: >>>> >>>>> def demo(arg): >>>>> if p, q ?= arg.x, arg.y: # dict structure >>>>> elif x ?= arg.x and isinstance(x, int) # assignment + guard >>>>> elif a, b, *_ ?= arg: # tuple structure >>>>> elif isinstance(arg, Mapping): # nothing new here >>> >>> I'm unenthusiastic about this -- the above looks like >>> an unreadable mess to me. >> >> >> Which is unreadable: ``if/elif`` keywords or ``?=`` operator? If it's the >> latter, please don't get hung up on that, as I intended that as a >> placeholder for whatever operator or keyword is best. My main point is that >> switch/case/matching is semantically identical to if/elif, so why use >> something different? > > For me, it's the use of operators (punctuation) plus the fact that the > overall layout doesn't in any way imply to me "I'm looking at *this* > expression. Let's try a sequence of matches..." > > The key distinguishing feature for me of a switch/match statement is > that it organises the "flow" of the statement differently from > if/elif. The if/elif chain says "try this, then that, now something > else". There's no implication that the tests are all looking at the > same subject - to spot that, you need to be able to see (from the > layout of the tests) the actual subject item on each line, and the ?= > operator syntax obscures that because "arg" is used differently in > each test. > > With a switch statement, however, the subject is stated once, at the > top of the statement. The checks are then listed one after the other, > and they are all by definition checks against the subject expression. > And in fact, for me that's the distinguishing feature of a switch > statement - that it's a series of tests against a single implied > subject. That's also (I suspect) why it's hard to come up with a good > syntax for tests - Python doesn't really do "implied subjects" > anywhere else (explicit is better than implicit and all that). > > IMO, "Series of tests against a single expression" is a common enough > pattern to be worth exploring in spite of the fact that it runs > counter to EIBTI. But you may need to be Dutch to understand why :-) > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From k7hoven at gmail.com Tue May 24 11:42:06 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 24 May 2016 18:42:06 +0300 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <57446C6B.3030503@gmail.com> References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> <57446C6B.3030503@gmail.com> Message-ID: On Tue, May 24, 2016 at 5:59 PM, Michel Desmoulin wrote: > What about making it a function ? > > Pattern matching is complex, it can be like a mini language inside the > language, just like regex. > > To match text we do: > > import re > re.match('pattern', 'string') > > We could do the same for matching structures: > > from inspect import match > > def do(stuff): > m = match(stuff): # m implements __call__ > if m('whatever mini language you like'): > return foo > if m('again'): > return m.thing_your_extracted > Or with methods: m = match(stuff) if m.by_type(SomeType): # handle SomeType elif m.by_attrs('x', 'y'): # do things with stuff.x and stuff.y elif m.by_len(3): x,y,z = stuff # do things with x, y, z > Pros: > > - no need for a new syntax > - very explicit > - use well know constructs > - we can easily start experimenting with the matching language in a lib > in Pypy + can be extended easily > > Cons: > > - much more verbose; > - debugging your mini language and handling error is not as good; + probably slower + need to separately get the desired attributes/elements after matching. -- Koos From steve at pearwood.info Tue May 24 11:45:07 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 25 May 2016 01:45:07 +1000 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> Message-ID: <20160524154506.GV12028@ando.pearwood.info> On Mon, May 23, 2016 at 12:56:40PM +1000, Nick Coghlan wrote: > On 23 May 2016 at 08:34, Greg Ewing wrote: > > Related to this, is there any good reason that ArithmeticError > > doesn't derive from ValueError? [...] > I considered suggesting that, but got stuck on the question of how we > quantify the compatibility implications - such a shift would change > the behaviour of code that catches both, but handles them differently. The same applies to decimal.InvalidOperation. Code like: try: ... except ValueError: ... except InvalidOperation: ... would change under my suggestion. > While I can honestly say I've never seen an except clause catching > ArithmeticError directly in the wild, I see more of a risk in the fact > we'd potentially be silently changing the way ZeroDivisionError and > OverflowError are handled. > > That said, this feels like the *right* answer to me, it's just the > compatibility risk that worries me. Indeed. -- Steve From gvanrossum at gmail.com Tue May 24 11:51:46 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue, 24 May 2016 08:51:46 -0700 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: <20160524154506.GV12028@ando.pearwood.info> References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> <20160524154506.GV12028@ando.pearwood.info> Message-ID: I've ignored this thread. Do you need help? --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue May 24 12:14:39 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 25 May 2016 02:14:39 +1000 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> Message-ID: <20160524161439.GW12028@ando.pearwood.info> On Mon, May 23, 2016 at 08:11:10AM +0000, Stefan Krah wrote: > Greg Ewing writes: > > >>>So I propose that InvalidOperation be changed to inherit from > > >>>ValueError, to match the expected behaviour from other numeric types. > > > > Related to this, is there any good reason that ArithmeticError > > doesn't derive from ValueError? > > For Decimal at least, Overflow, Underflow, Clamped etc. don't really > map to ValueError. I think they do. Take Overflow: py> Decimal("1e500000")**2 Traceback (most recent call last): File "", line 1, in decimal.Overflow: [] Obviously that's a problem with the *value*. It's not a type error. If the value were smaller, the calculation wouldn't have overflowed. > This includes InvalidOperation, which unfortunately *already* has > a subclass ConversionSyntax for the use case that came up here. > > ConversionSyntax, however, is designated by the standard to be a > "condition" rather than a "signal", so ConversionSyntax isn't > raised -- it just exists, and in the C version you can see the > "condition" in the list next to the exception that was raised. > > Adding a ValueError to all that just complicates things further. > I wouldn't mind raising ConversionSyntax directly, but then again > no one knows about that either and it's a small deviation from > the standard. I don't believe the standard says anything about class hierarchies for traps. It explicitly says: "This specification does not define the means by which flags and traps are reset or altered, respectively, or the means by which traps are effected." http://speleotrove.com/decimal/damodel.html#backref.11 As far as the standard itself cares, making InvalidOperation a subclass of ValueError is no better or worse than making InvalidOperation a subclass of Exception. -- Steve From steve at pearwood.info Tue May 24 12:19:13 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 25 May 2016 02:19:13 +1000 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> <20160524154506.GV12028@ando.pearwood.info> Message-ID: <20160524161913.GX12028@ando.pearwood.info> On Tue, May 24, 2016 at 08:51:46AM -0700, Guido van Rossum wrote: > I've ignored this thread. Do you need help? It might help if you ruled the change out completely, said it needs a PEP, or that you didn't care what we did so long as we don't break the module :-) I'd also want to see Raymond's and Tim's comments before any changes were made. But if you rule it out, no need to bother them. -- Steve From gvanrossum at gmail.com Tue May 24 12:56:10 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue, 24 May 2016 09:56:10 -0700 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: <20160524161913.GX12028@ando.pearwood.info> References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> <20160524154506.GV12028@ando.pearwood.info> <20160524161913.GX12028@ando.pearwood.info> Message-ID: If it's the right thing to do we should do it, and consider how to do it without breaking too much code too soon (or without clear indication of failure). I don't think a PEP is needed unless there's a lot of discussion about alternatives or disagreement. On Tue, May 24, 2016 at 9:19 AM, Steven D'Aprano wrote: > On Tue, May 24, 2016 at 08:51:46AM -0700, Guido van Rossum wrote: >> I've ignored this thread. Do you need help? > > It might help if you ruled the change out completely, said it needs a > PEP, or that you didn't care what we did so long as we don't break the > module :-) > > I'd also want to see Raymond's and Tim's comments before any changes > were made. But if you rule it out, no need to bother them. > > > -- > Steve -- --Guido van Rossum (python.org/~guido) From jim.baker at python.org Tue May 24 15:20:57 2016 From: jim.baker at python.org (Jim Baker) Date: Tue, 24 May 2016 15:20:57 -0400 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> <57446C6B.3030503@gmail.com> Message-ID: Structural pattern matching should address the following, somewhat similar to regex pattern matching: 1. Patterns are quoted. This can be implicitly done, because of built-in syntactic support; or one is supplying the pattern to a library using regular Python quoting. 2. Patterns should support bind variables for the unpacking. In languages like Scala, this is very natural to do, with bop, v1, v2 lexically scoped to the RHS of =>. ```scala case Binary(bop @ (Lt|Le|Gt|Ge), v1, v2) if isValue(v1) && isValue(v2) => doreturn(B(inequalityVal(bop, v1, v2)) ) ``` This is an example from an interpreter used in a lab exercise in a class I teach, with patterns against ASTs. The ASTs are themselves defined in terms of Scala's case classes, which are more or less equivalent to namedtuples in Python. Clearly the scoping of bind variables can be emulated by the match object, much as we do in regexes, at the cost of some minor overhead. 3. Some protocol to connect together objects like tuples, namedtuples, and arbitrary classes with the matching. In Scala, this is called unapply, and given that Scala has similar object-oriented aspects that are similar to Python in making life more difficult ;), when compared to Haskell's comparatively simple rules, it's probably close to what we need to do for __match__ or something like that. The challenge is making this work together, especially distinguishing patterns from bind variables. Haskell makes it easy by a requirement that algebraic types have constructors which start with an upper case letter. Scala can do simple scope analysis of names to determine if it's a case class or a bind variable. For Python, presumably more syntax is necessary in the pattern specifier. Maybe something like the following, which seems unambiguous, possibly not so ugly: ```python case Binary(bop @ (Lt|Le|Gt|Ge), v1@, v2@) if isValue(v1) and isValue(v2): # bop, v1, v2 are lexically scoped here ``` Am I missing anything? It seems to me that one can do structural pattern matching as a library (do obvious changes to above); although having it supported with specific syntax might make it much nicer. - Jim On Tue, May 24, 2016 at 11:42 AM, Koos Zevenhoven wrote: > On Tue, May 24, 2016 at 5:59 PM, Michel Desmoulin > wrote: > > What about making it a function ? > > > > Pattern matching is complex, it can be like a mini language inside the > > language, just like regex. > > > > To match text we do: > > > > import re > > re.match('pattern', 'string') > > > > We could do the same for matching structures: > > > > from inspect import match > > > > def do(stuff): > > m = match(stuff): # m implements __call__ > > if m('whatever mini language you like'): > > return foo > > if m('again'): > > return m.thing_your_extracted > > > > Or with methods: > > m = match(stuff) > if m.by_type(SomeType): > # handle SomeType > elif m.by_attrs('x', 'y'): > # do things with stuff.x and stuff.y > elif m.by_len(3): > x,y,z = stuff > # do things with x, y, z > > > Pros: > > > > - no need for a new syntax > > - very explicit > > - use well know constructs > > - we can easily start experimenting with the matching language in a lib > > in Pypy > + can be extended easily > > > > > Cons: > > > > - much more verbose; > > - debugging your mini language and handling error is not as good; > + probably slower > + need to separately get the desired attributes/elements after matching. > > -- Koos > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue May 24 19:03:13 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 25 May 2016 11:03:13 +1200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> Message-ID: <5744DDB1.5030101@canterbury.ac.nz> Michael Selik wrote: > Which is unreadable: ``if/elif`` keywords or ``?=`` operator? The whole thing. > If it's > the latter, please don't get hung up on that, as I intended that as a > placeholder for whatever operator or keyword is best. Syntax is an important part of readability, though. Any alternative syntax for the same semantics would have to be evaluated for readability on its own merits. > My main point is > that switch/case/matching is semantically identical to if/elif, so why > use something different? But then you say > No, using the assign-if-can operator would be syntax error outside of an > if/elif, the reverse of how an assign operator is a syntax error inside > an if/elif. So what you're proposing is *not* semantically equivalent to composing the existing if/elif with a new pattern matching assignment, which means re-using the existing keywords for it is misleading. > The exception-catching expression is not meant to be available in > any other context. However, exception-catching is an essential feature > to matching via destructuring. I believe it's unavoidable, though > *which* exceptions are suppressed could be made more clear. Perhaps I can elaborate on that a bit. In Guido's original proposal, you would say something like case Foo(a=x, b=y): to match something of class Foo having attributes a and b. Implementing that would involve using hasattr() on the object being matched, or something equivalent. That catches AttributeError, but only on the particular operation of testing for the attribute. Now with your x, y ?= arg.a, arg.b my assumption was that there was nothing special about the right hand side, so the implementation would have to just evaluate all of it and catch any AttribteErrors, KeyErrors, IndexErrors, etc. emanating from it, which is much looser and prone to catching too much. But if the right hand side is special, all bets are off and you'll have to explain exactly what would be allowed and how to interpret it. -- Greg From ncoghlan at gmail.com Tue May 24 20:26:45 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 May 2016 10:26:45 +1000 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> <20160524154506.GV12028@ando.pearwood.info> <20160524161913.GX12028@ando.pearwood.info> Message-ID: On 25 May 2016 at 02:56, Guido van Rossum wrote: > If it's the right thing to do we should do it, and consider how to do > it without breaking too much code too soon (or without clear > indication of failure). I don't think a PEP is needed unless there's a > lot of discussion about alternatives or disagreement. Recapping the thread, the specific problem Steven raised was that catching ValueError will handle "float('not-a-valid-float')", but not "Decimal('not-a-valid-float')", as the latter raises decimal.InvalidOperation, which inherits from ArithmeticError rather than ValueError. There are 3 possible ways of changing that: - make ArithmeticError a subclass of ValueError - make decimal.InvalidOperation inherit from both ArithmeticError & ValueError - raise a subclass of InvalidOperation that also inherits from ValueError for decimal string conversions The last one is clearly the lowest risk and lowest impact way to allow ValueError to be used to catch decimal string conversion problems, so in the absence of any other considerations, I'd just say "Let's do that". However, what came up in the course of the discussion is that those of us participating in the thread don't actually know the original rationale for ArithmeticError being entirely distinct from ValueError, rather than being a subclass of it, which means there are other cases where folks may be expecting ValueError to catch all conversion errors, but it may in fact be missing some. As a toy example: >>> def to_float(x): ... try: ... return float(x) ... except ValueError: ... return float("nan") ... >>> to_float("not-a-float") nan >>> to_float(10**1000) Traceback (most recent call last): File "", line 1, in File "", line 3, in to_float OverflowError: int too large to convert to float >>> from fractions import Fraction >>> class LazyFraction: ... def __float__(self): ... return float(Fraction(1, 0)) ... >>> to_float(LazyFraction()) Traceback (most recent call last): File "", line 1, in File "", line 3, in to_float File "", line 3, in __float__ File "/usr/lib64/python3.5/fractions.py", line 186, in __new__ raise ZeroDivisionError('Fraction(%s, 0)' % numerator) ZeroDivisionError: Fraction(1, 0) Before this thread, I would have said that the above "to_float()" function would reliably return either the converted value or NaN in the absence of coding bugs in a __float__ method implementation, but the OverflowError and ZeroDivisionError cases above show that I would have been wrong about that, and a more comprehensive exception clause would be "except (ValueError, ArithmeticError):". If you rule out turning ArithmeticError into a ValueError subclass in 3.6+, then we can go back to looking specifically at the decimal string conversion behaviour. However, if you thought the builtin exception hierarchy change was an idea worth considering further, then it would address decimal string conversions as a side effect (since those are already an ArithmeticError subclass). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From gvanrossum at gmail.com Tue May 24 22:04:46 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue, 24 May 2016 19:04:46 -0700 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> <20160524154506.GV12028@ando.pearwood.info> <20160524161913.GX12028@ando.pearwood.info> Message-ID: Is there really something here that needs to be fixed? It's very common for modules to define their own root exception class. And usually people who care about these kinds of exceptions just catch whatever exception is raised by the combination of argument they care about (e.g. calling float() with a string argument). The Decimal module already doesn't play all that much by the same rules as other modules (e.g. it doesn't participate in the numeric tower) so people trying to write algorithms independent from whether the numbers they're processing are floats, fractions or Decimals are already in a lot of pain, I imagine -- and the best solution is probably to just stick with Decimal (numerical specialists would cringe at attempts to write code without knowing how the arithmetic is done in detail anyways). If there's anything in this area that bugs me it would be that I'd worry that some obscure invalid input string to float() might not raise ValueError but something else (e.g. OverflowError -- while I cannot reproduce this, I've seen lots of code that catches that). But in Decimal I would expect this all to be specified by the standard, so I see no need to lose sleep there. On Tue, May 24, 2016 at 5:26 PM, Nick Coghlan wrote: > On 25 May 2016 at 02:56, Guido van Rossum wrote: >> If it's the right thing to do we should do it, and consider how to do >> it without breaking too much code too soon (or without clear >> indication of failure). I don't think a PEP is needed unless there's a >> lot of discussion about alternatives or disagreement. > > Recapping the thread, the specific problem Steven raised was that > catching ValueError will handle "float('not-a-valid-float')", but not > "Decimal('not-a-valid-float')", as the latter raises > decimal.InvalidOperation, which inherits from ArithmeticError rather > than ValueError. > > There are 3 possible ways of changing that: > > - make ArithmeticError a subclass of ValueError > - make decimal.InvalidOperation inherit from both ArithmeticError & ValueError > - raise a subclass of InvalidOperation that also inherits from > ValueError for decimal string conversions > > The last one is clearly the lowest risk and lowest impact way to allow > ValueError to be used to catch decimal string conversion problems, so > in the absence of any other considerations, I'd just say "Let's do > that". > > However, what came up in the course of the discussion is that those of > us participating in the thread don't actually know the original > rationale for ArithmeticError being entirely distinct from ValueError, > rather than being a subclass of it, which means there are other cases > where folks may be expecting ValueError to catch all conversion > errors, but it may in fact be missing some. As a toy example: > >>>> def to_float(x): > ... try: > ... return float(x) > ... except ValueError: > ... return float("nan") > ... >>>> to_float("not-a-float") > nan >>>> to_float(10**1000) > Traceback (most recent call last): > File "", line 1, in > File "", line 3, in to_float > OverflowError: int too large to convert to float >>>> from fractions import Fraction >>>> class LazyFraction: > ... def __float__(self): > ... return float(Fraction(1, 0)) > ... >>>> to_float(LazyFraction()) > Traceback (most recent call last): > File "", line 1, in > File "", line 3, in to_float > File "", line 3, in __float__ > File "/usr/lib64/python3.5/fractions.py", line 186, in __new__ > raise ZeroDivisionError('Fraction(%s, 0)' % numerator) > ZeroDivisionError: Fraction(1, 0) > > Before this thread, I would have said that the above "to_float()" > function would reliably return either the converted value or NaN in > the absence of coding bugs in a __float__ method implementation, but > the OverflowError and ZeroDivisionError cases above show that I would > have been wrong about that, and a more comprehensive exception clause > would be "except (ValueError, ArithmeticError):". > > If you rule out turning ArithmeticError into a ValueError subclass in > 3.6+, then we can go back to looking specifically at the decimal > string conversion behaviour. However, if you thought the builtin > exception hierarchy change was an idea worth considering further, then > it would address decimal string conversions as a side effect (since > those are already an ArithmeticError subclass). > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -- --Guido van Rossum (python.org/~guido) From stephen at xemacs.org Wed May 25 01:03:04 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 25 May 2016 14:03:04 +0900 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> Message-ID: <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> Paul Moore writes: > With a switch statement, however, the subject is stated once, at the > top of the statement. The checks are then listed one after the other, > and they are all by definition checks against the subject expression. > And in fact, for me that's the distinguishing feature of a switch > statement +1 That may also make some optimizations easier to spot. I'm also -1 on the ?= syntax, which reads "maybe assign" to me, but that covers way too much ground (and in particular has been proposed for "null-propagation" in the recent past). I admit I'm not in love with the "switch/case" syntax for the kind of matching intended here, as C's switch is way too burned into my thinking. I think it's gone right through the EEPROM silicon into the plastic case[sic, just can't get away from those preexisting "case"es!] How about for : try : pass try : pass Look Ma! No new keywords! Yeah, I know, "for" and "try" both have very strong connotations in Python already, so this may be a "not even the Dutch could like it" automatic-parser-only syntax. Steve From rosuav at gmail.com Wed May 25 01:11:10 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 25 May 2016 15:11:10 +1000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, May 25, 2016 at 3:03 PM, Stephen J. Turnbull wrote: > How about > > for : > try : > pass > try : > pass > > Look Ma! No new keywords! Yeah, I know, "for" and "try" both have > very strong connotations in Python already, so this may be a "not even > the Dutch could like it" automatic-parser-only syntax. I'd much prefer a different keyword instead of 'for'. If 'with' hadn't been used, that would be a better choice. Maybe 'using'... or 'switch'. But without iteration, 'for' is going to be very confusing. ChrisA From michael.selik at gmail.com Wed May 25 01:33:57 2016 From: michael.selik at gmail.com (Michael Selik) Date: Wed, 25 May 2016 05:33:57 +0000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <5744DDB1.5030101@canterbury.ac.nz> References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> <5744DDB1.5030101@canterbury.ac.nz> Message-ID: On Tue, May 24, 2016 at 7:03 PM Greg Ewing wrote: > Michael Selik wrote: > > My main point is > > that switch/case/matching is semantically identical to if/elif, so why > > use something different? > > But then you say > > > No, using the assign-if-can operator would be syntax error outside of an > > if/elif, the reverse of how an assign operator is a syntax error inside > > an if/elif. > > So what you're proposing is *not* semantically equivalent > to composing the existing if/elif with a new pattern matching > assignment, which means re-using the existing keywords for > it is misleading. > Perhaps I didn't explain my proposal correctly. I spent about 3 hours this evening writing out a more thorough proof of the semantic equivalence. As I did so, toying with different syntaxes, I realized that it might be better to break this problem apart and solve a few subproblems first. The first problem to solve is how to write a destructuring bind for more than just sequences. Once we're happy with that, it'll be easier to discuss pattern matching. # Iterable-destructuring bind (a.k.a unpacking) (a, b, *rest) = sequence # Subscriptable-destructuring bind {'x': p, 'y': q} = mapping {0: p, 42: q, **rest} = indexable # Attribute-destructuring bind (a few awkward ideas) (.x: p, .y: q) = obj (x=p, y=q) = obj Or would you call that "object-destructuring"? No clue what the best name is here. I think Clojure side-steps attributes and only provides destructuring sequences and mappings. If you want to do pattern matching on a more complex type, you must provide essentially a conversion from that type to a mapping. > The exception-catching expression is not meant to be available in > > any other context. However, exception-catching is an essential feature > > to matching via destructuring. I believe it's unavoidable, though > > *which* exceptions are suppressed could be made more clear. > > my assumption was that there was nothing special about > the right hand side, so the implementation would have to > just evaluate all of it and catch any AttribteErrors, > KeyErrors, IndexErrors, etc. emanating from it, which is > much looser and prone to catching too much. > > But if the right hand side is special, all bets are off > and you'll have to explain exactly what would be allowed > and how to interpret it. > I agree that catching all exceptions during the assign-if-can (via if/elif or switch/case) might be too error-prone. Whether it's the LHS of ``as`` or the RHS of ``?=``, it would help to restrict the kind of exceptions handled/suppressed as a failed match. However, I fail to see why ``case ... as ...`` would be restrictive and ``if ... ?= ...`` would not. They could have the same semantics, catching either a specific set of exceptions or all/most exceptions. -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.selik at gmail.com Wed May 25 01:38:58 2016 From: michael.selik at gmail.com (Michael Selik) Date: Wed, 25 May 2016 05:38:58 +0000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <57446C6B.3030503@gmail.com> References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> <57446C6B.3030503@gmail.com> Message-ID: On Tue, May 24, 2016 at 11:00 AM Michel Desmoulin wrote: > What about making it a function ? > Pattern matching is complex, it can be like a mini language inside the > language, just like regex. > What was that aphorism I heard -- "Any sufficiently complicated Python program contains an ad hoc implementation of Haskell."? Just kidding. Mostly. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed May 25 01:41:13 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 25 May 2016 17:41:13 +1200 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> <20160524154506.GV12028@ando.pearwood.info> <20160524161913.GX12028@ando.pearwood.info> Message-ID: <57453AF9.8050602@canterbury.ac.nz> Guido van Rossum wrote: > It's very > common for modules to define their own root exception class. And it's a nuisance when their exceptions only belong to their own private hierarchy and don't participate in any the standard classifications. I don't think Decimal is directly to blame here, because deriving its exceptions from ArithmeticError seems like a reasonable thing to do. What *doesn't* seem reasonable to me is that ArithmeticError doesn't derive from ValueError. So far nobody has explained why that's a good idea. -- Greg From guido at python.org Wed May 25 01:48:16 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 24 May 2016 22:48:16 -0700 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: <57453AF9.8050602@canterbury.ac.nz> References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> <20160524154506.GV12028@ando.pearwood.info> <20160524161913.GX12028@ando.pearwood.info> <57453AF9.8050602@canterbury.ac.nz> Message-ID: Well, ZeroDivisionError doesn't derive from ValueError, does it? So it may not be a good idea, but it's certainly a tradition. On Tuesday, May 24, 2016, Greg Ewing wrote: > Guido van Rossum wrote: > >> It's very >> common for modules to define their own root exception class. >> > > And it's a nuisance when their exceptions only belong > to their own private hierarchy and don't participate in > any the standard classifications. > > I don't think Decimal is directly to blame here, because > deriving its exceptions from ArithmeticError seems like > a reasonable thing to do. What *doesn't* seem reasonable > to me is that ArithmeticError doesn't derive from > ValueError. So far nobody has explained why that's > a good idea. > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed May 25 01:52:50 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 May 2016 15:52:50 +1000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> Message-ID: On 25 May 2016 at 15:11, Chris Angelico wrote: > On Wed, May 25, 2016 at 3:03 PM, Stephen J. Turnbull wrote: >> How about >> >> for : >> try : >> pass >> try : >> pass >> >> Look Ma! No new keywords! Yeah, I know, "for" and "try" both have >> very strong connotations in Python already, so this may be a "not even >> the Dutch could like it" automatic-parser-only syntax. > > I'd much prefer a different keyword instead of 'for'. If 'with' hadn't > been used, that would be a better choice. Maybe 'using'... or > 'switch'. But without iteration, 'for' is going to be very confusing. As a variant on Guido's original switch/case idea: given EXPR [as TARGET]: case MATCH_PATTERN [as TARGET] [and CONDITION]: ... case MATCH_PATTERN [as TARGET] [and CONDITION]: ... case if CONDITION: ... case MATCH_PATTERN [as TARGET]: ... else: ... Using the running demo: def demo(arg): given arg: case x, y, *_: # Tuple matching (implicit name binding) ... case (.x, .y) as p, q: # Attribute matching ... case (["x"], ["y"]) as p, q: # Item matching ... case (.x) as p and isinstance(p, int): # Match + condition ... case if isinstance(arg, int): # Condition only ... else: # Default ... The other key change there is introducing "as" to the individual cases in order to be able to separate the match pattern definition from the local name binding. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From greg.ewing at canterbury.ac.nz Wed May 25 01:58:17 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 25 May 2016 17:58:17 +1200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> <5744DDB1.5030101@canterbury.ac.nz> Message-ID: <57453EF9.2010505@canterbury.ac.nz> Michael Selik wrote: > I fail to see > why ``case ... as ...`` would be restrictive and ``if ... ?= ...`` would > not. They could have the same semantics, catching either a specific set > of exceptions or all/most exceptions. The difference is not which exceptions get caught, it's the size of the region of code around which the catching occurs. When I see if x, y ?= arg.a, arg.b: do_stuff() it suggests that something like this is going on: try: temp = arg.a, arg.b except AttributeError: pass else: x, y = temp do_stuff() Whereas the implementation I had in mind for switch arg: case (a = x, b = y): do_stuff() would be more like if hasattr(arg, "a") and hasattr(arg, "b"): x = arg.a y = arg.b do_stuff() They're equivalent if the only things you have on the RHS are attribute accesses, but not if the RHS is something more complicated. If you're intending to restrict the RHS so that you're not allowed anything more complicated, I think that would be weird and suprising. Another weird and surprising thing, that applies in either case, is that if x, y ?= arg.a, arg.b: ... would *not* be equivalent to z = arg.a, arg.b if x, y ?= z: ... With the switch/case syntax or something like it, that issue doesn't arise. -- Greg From ncoghlan at gmail.com Wed May 25 02:00:00 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 May 2016 16:00:00 +1000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> <5744DDB1.5030101@canterbury.ac.nz> Message-ID: On 25 May 2016 at 15:33, Michael Selik wrote: > On Tue, May 24, 2016 at 7:03 PM Greg Ewing > wrote: >> But if the right hand side is special, all bets are off >> and you'll have to explain exactly what would be allowed >> and how to interpret it. > > I agree that catching all exceptions during the assign-if-can (via if/elif > or switch/case) might be too error-prone. Whether it's the LHS of ``as`` or > the RHS of ``?=``, it would help to restrict the kind of exceptions > handled/suppressed as a failed match. However, I fail to see why ``case ... > as ...`` would be restrictive and ``if ... ?= ...`` would not. They could > have the same semantics, catching either a specific set of exceptions or > all/most exceptions. Having the RHS of an assignment operation be syntactically restricted isn't something Python has ever done before, while the LHS is already heavily restricted (since it describes name binding targets rather than normal expressions for evaluation). The syntactic restrictions then mean evaluation order can be controlled to ensure any exception handles only cover the desired step in the process (e.g. a particular key lookup), rather than the evaluation of arbitrary subexpressions. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From greg.ewing at canterbury.ac.nz Wed May 25 02:26:01 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 25 May 2016 18:26:01 +1200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> Message-ID: <57454579.5020907@canterbury.ac.nz> Nick Coghlan wrote: > case (.x, .y) as p, q: # Attribute matching I don't think I like the "as". Like the "?=", it separates the pattern from the names being bound too much. > case (["x"], ["y"]) as p, q: # Item matching That's even more confusing. The part before the "as" looks like it should match a tuple of two one-element lists containing the values "x" and "y". My feeling is that the patterns should look like constructors. The archetypal constructor for a mapping object is the dict display, so a pattern that matches a mapping having particular keys would be case {"x": p, "y": q}: ... > case MATCH_PATTERN [as TARGET] [and CONDITION]: That way of handling guards wouldn't allow for multiple guards on the same case. I would suggest case PATTERN: when CONDITION: ... when CONDITION: ... Note that this would be different from case PATTERN: if CONDITION: ... elif CONDITION: ... because failure of all the "when" clauses should cause the next case to be tried. -- Greg From michael.selik at gmail.com Wed May 25 02:35:08 2016 From: michael.selik at gmail.com (Michael Selik) Date: Wed, 25 May 2016 06:35:08 +0000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <57454579.5020907@canterbury.ac.nz> References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <57454579.5020907@canterbury.ac.nz> Message-ID: On Wed, May 25, 2016 at 2:26 AM Greg Ewing wrote: > > My feeling is that the patterns should look like > constructors. The archetypal constructor for a mapping > object is the dict display, so a pattern that matches > a mapping having particular keys would be > > case {"x": p, "y": q}: > This is starting to look really good to me. But if that's valid in a case statement, why not in a regular assignment? (a, b, *rest) = sequence {'x': p, 'y': q, **rest} = mapping -------------- next part -------------- An HTML attachment was scrubbed... URL: From casevh at gmail.com Wed May 25 03:22:21 2016 From: casevh at gmail.com (Case Van Horsen) Date: Wed, 25 May 2016 00:22:21 -0700 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> <20160524154506.GV12028@ando.pearwood.info> <20160524161913.GX12028@ando.pearwood.info> Message-ID: On Tue, May 24, 2016 at 5:26 PM, Nick Coghlan wrote: > On 25 May 2016 at 02:56, Guido van Rossum wrote: >> If it's the right thing to do we should do it, and consider how to do >> it without breaking too much code too soon (or without clear >> indication of failure). I don't think a PEP is needed unless there's a >> lot of discussion about alternatives or disagreement. > > Recapping the thread, the specific problem Steven raised was that > catching ValueError will handle "float('not-a-valid-float')", but not > "Decimal('not-a-valid-float')", as the latter raises > decimal.InvalidOperation, which inherits from ArithmeticError rather > than ValueError. > > There are 3 possible ways of changing that: > > - make ArithmeticError a subclass of ValueError > - make decimal.InvalidOperation inherit from both ArithmeticError & ValueError > - raise a subclass of InvalidOperation that also inherits from > ValueError for decimal string conversions FWIW, here is the exception hierarchy that I use in gmpy2. I've ignored the standard base classes, etc. gmpy2.InvalidOperationError - gmpyError - ArithmeticError - ValueError gmpy2.DivisionByZeroError - gmpyError - ZeroDivisionError - ArithmeticError gmpy2.InexactResultError - gmpyError - ArithmeticError gmpy2.OverflowResultError - InexactResultError - gmpyError - ArithmeticError gmpy2.UnderflowResultError - InexactResultError - gmpyError - ArithmeticError gmpy2.RangeError - gmpyError - ArithmeticError > > The last one is clearly the lowest risk and lowest impact way to allow > ValueError to be used to catch decimal string conversion problems, so > in the absence of any other considerations, I'd just say "Let's do > that". Of the 3 options proposed by Nick, the first option would change gmpy2.RangeError. The second option corresponds to the current approach used by gmpy2.InvalidOperation. The third option introduces a new hierarchy. > However, what came up in the course of the discussion is that those of > us participating in the thread don't actually know the original > rationale for ArithmeticError being entirely distinct from ValueError, > rather than being a subclass of it, which means there are other cases > where folks may be expecting ValueError to catch all conversion > errors, but it may in fact be missing some. As a toy example: > >>>> def to_float(x): > ... try: > ... return float(x) > ... except ValueError: > ... return float("nan") > ... >>>> to_float("not-a-float") > nan >>>> to_float(10**1000) > Traceback (most recent call last): > File "", line 1, in > File "", line 3, in to_float > OverflowError: int too large to convert to float >>>> from fractions import Fraction >>>> class LazyFraction: > ... def __float__(self): > ... return float(Fraction(1, 0)) > ... >>>> to_float(LazyFraction()) > Traceback (most recent call last): > File "", line 1, in > File "", line 3, in to_float > File "", line 3, in __float__ > File "/usr/lib64/python3.5/fractions.py", line 186, in __new__ > raise ZeroDivisionError('Fraction(%s, 0)' % numerator) > ZeroDivisionError: Fraction(1, 0) > > Before this thread, I would have said that the above "to_float()" > function would reliably return either the converted value or NaN in > the absence of coding bugs in a __float__ method implementation, but > the OverflowError and ZeroDivisionError cases above show that I would > have been wrong about that, and a more comprehensive exception clause > would be "except (ValueError, ArithmeticError):". > > If you rule out turning ArithmeticError into a ValueError subclass in > 3.6+, then we can go back to looking specifically at the decimal > string conversion behaviour. However, if you thought the builtin > exception hierarchy change was an idea worth considering further, then > it would address decimal string conversions as a side effect (since > those are already an ArithmeticError subclass). I've used the following guidelines for exceptions: 1) A TypeError is raised when there is no possible way to associate a meaning to an instance of that type. 2) A ValueError is raised when at least some instances of the given type can have a valid meaning but the particular value passed has no possible valid interpretation. 3) An ArithmeticError (or subclass) is raised when arithmetic operations on a valid input fail for a particular reason. Some examples: int(float) raises a TypeError because there is no possible way to assign a numeric value to a type. int(float("nan")) raises a ValueError because there is no way to perform arithmetic operations on NaN. int(float("inf")) raises OverflowError because while you can perform arithmetic operation on Infinity, it will eventually overflow. Following my guidelines, it makes sense that decimal.Decimal("not-valid") would raise a (subclass of) ValueError. casevh > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ethan at stoneleaf.us Wed May 25 03:38:34 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 May 2016 00:38:34 -0700 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> Message-ID: <5745567A.7070109@stoneleaf.us> On 05/24/2016 10:52 PM, Nick Coghlan wrote: > Using the running demo: > > def demo(arg): > given arg: > case x, y, *_: # Tuple matching (implicit name binding) > ... > case (.x, .y) as p, q: # Attribute matching > ... > case (["x"], ["y"]) as p, q: # Item matching > ... > case (.x) as p and isinstance(p, int): # Match + condition > ... > case if isinstance(arg, int): # Condition only > ... > else: # Default > ... > > The other key change there is introducing "as" to the individual cases > in order to be able to separate the match pattern definition from the > local name binding. With this one I have a clue as to what's going on. -- ~Ethan~ From rob.cliffe at btinternet.com Wed May 25 04:08:33 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 25 May 2016 09:08:33 +0100 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> <20160524154506.GV12028@ando.pearwood.info> <20160524161913.GX12028@ando.pearwood.info> <57453AF9.8050602@canterbury.ac.nz> Message-ID: <0239b3c2-f795-6418-5b27-f61673c8f643@btinternet.com> Or more generally, any operation on multiple values, where each value is valid individually, but the result of the operation is not: >>> p = Decimal('1E999999999') >>> q = Decimal('10') >>> try: ... p*q ... except ArithmeticError: ... print 'ArithmeticError' ... ArithmeticError On 25/05/2016 06:48, Guido van Rossum wrote: > Well, ZeroDivisionError doesn't derive from ValueError, does it? So it > may not be a good idea, but it's certainly a tradition. > > On Tuesday, May 24, 2016, Greg Ewing > wrote: > > Guido van Rossum wrote: > > It's very > common for modules to define their own root exception class. > > > And it's a nuisance when their exceptions only belong > to their own private hierarchy and don't participate in > any the standard classifications. > > I don't think Decimal is directly to blame here, because > deriving its exceptions from ArithmeticError seems like > a reasonable thing to do. What *doesn't* seem reasonable > to me is that ArithmeticError doesn't derive from > ValueError. So far nobody has explained why that's > a good idea. > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido (mobile) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed May 25 04:41:25 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 May 2016 18:41:25 +1000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <57454579.5020907@canterbury.ac.nz> Message-ID: On 25 May 2016 at 16:35, Michael Selik wrote: > On Wed, May 25, 2016 at 2:26 AM Greg Ewing > wrote: >> >> >> My feeling is that the patterns should look like >> constructors. The archetypal constructor for a mapping >> object is the dict display, so a pattern that matches >> a mapping having particular keys would be >> >> case {"x": p, "y": q}: > > > This is starting to look really good to me. But if that's valid in a case > statement, why not in a regular assignment? > > (a, b, *rest) = sequence > {'x': p, 'y': q, **rest} = mapping Aye, I'm warming to that as a syntax for item unpacking (although I'm not sold on the "**rest" notation, since that entails doing something like "{k: v for k, v in obj.items() if k not in explicit_keys}" to populate it, further elevating "items()" towards the status of being a protocol method without double-underscores) I also agree with the principle that any prospective structural pattern matching statement should align with assignment target notation. Unfortunately, we don't have anything like dictionary displays to inform possible structures for an implied getattribute as part of assignment to a particular target. The option I dislike least so far is : (p=.x, q=.y) = obj # Attribute unpacking Which is clearly distinct from: (p, q) = obj # Iterable unpacking {0: p, 1: q} = obj # Item unpacking {'x': p, 'y': q} = obj # Item unpacking However, at this point we've strayed a *long* way from the ideal of "executable pseudocode" :( It would be slightly more readable if the new unpacking options used "from" as their assignment keyword rather than the traditional "=": (p=.x, q=.y) from obj # Attribute unpacking {0: p, 1: q} from obj # Item unpacking {'x': p, 'y': q} from obj # Item unpacking Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From leewangzhong+python at gmail.com Wed May 25 05:04:35 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Wed, 25 May 2016 05:04:35 -0400 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, May 25, 2016 at 1:52 AM, Nick Coghlan wrote: > Using the running demo: > > def demo(arg): > given arg: > case x, y, *_: # Tuple matching (implicit name binding) > ... > case (.x, .y) as p, q: # Attribute matching > ... > case (["x"], ["y"]) as p, q: # Item matching > ... > case (.x) as p and isinstance(p, int): # Match + condition > ... > case if isinstance(arg, int): # Condition only > ... > else: # Default > ... > > The other key change there is introducing "as" to the individual cases > in order to be able to separate the match pattern definition from the > local name binding. I still don't like that `case THING` is a pattern, rather than a value to test against. Here's my modifications with "as", attributes, and def demo(arg): given arg: case as x, y, *_: # Tuple matching (implicit name binding) ... case as object(x=p, y=q, **_): # Attribute matching ... case as {'x': p, 'y', q, **_}: # Item matching ... case as object(x=p, **_) and isinstance(p, int): # Match + condition ... case if isinstance(arg, int): # Condition only ... else: # Default ... Here, the "as" pattern is a shape to fit the object's parts into, and it should be possible to get back the original (or something isomorphic to it) by evaluating the pattern expression (so `case as {'x': p, 'y', q, **_}": assert isomorphic({'x': p, 'y', q, **_}, arg)`). For attribute-matching, I think it's possible to make user types also play well with this syntax for attribute matching, and I made a proposal earlier for an API. (Section 'Matching user classes' here: https://mail.python.org/pipermail/python-ideas/2016-May/040343.html) (Proposal to make the `object` case more reasonable: `object(**kwargs)` constructor creates an object with the given attributes. It won't affect subclasses, because `object.__new__` can test whether `cls is object`.) Perhaps literal dicts could match against sequences, too. {0: x, 42: y, **rest} = some_list And this could be how you enforce types: dict({0: x, 42: y, **rest}) = some_list # Fails. tuple((x, y, z)) = some_list # Fails. though I haven't figured out how these constructor shapes would be implemented for user types. Big conceptual obstacle: iterating over a dict gives its keys (which has always bothered me), so as a value, `list({0: x, 42: y, **rest})` would just be a list of keys. (I'd give up the typecheck syntax, personally, and have you move the check into a guard. I'm not too attached to the indexing syntax, either.) Note that I force an explicit `**_, so that `object(x=p)` will fail if `arg` has (non-dunder) attributes other than `x` (which I think is a good thing). It's kinda wasteful to pack unused things into a variable, so `...` could specify ignored args (which has come up before on this list), and the matcher engine can tell the type's matchmaker that it doesn't care about the other args. Problem: `Point(x, y, ...)` is a legitimate function call, so if `Point(x, 0)` is a legal pattern (i.e. no distinguishing syntax between values and bindnames), you'd need the syntax to be `Point(x, y, *...)`. Personally, I'd require that everything is a bindname (unless it looks like a constructor call), and require checks to be in guards. From ethan at stoneleaf.us Wed May 25 05:17:51 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 May 2016 02:17:51 -0700 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> Message-ID: <57456DBF.5060900@stoneleaf.us> On 05/25/2016 02:04 AM, Franklin? Lee wrote: > On Wed, May 25, 2016 at 1:52 AM, Nick Coghlan wrote: >> Using the running demo: >> >> def demo(arg): >> given arg: >> case x, y, *_: # Tuple matching (implicit name binding) >> ... >> case (.x, .y) as p, q: # Attribute matching >> ... >> case (["x"], ["y"]) as p, q: # Item matching >> ... >> case (.x) as p and isinstance(p, int): # Match + condition >> ... >> case if isinstance(arg, int): # Condition only >> ... >> else: # Default >> ... >> >> The other key change there is introducing "as" to the individual cases >> in order to be able to separate the match pattern definition from the >> local name binding. > > I still don't like that `case THING` is a pattern, rather than a value > to test against. Here's my modifications with "as", attributes, and > > def demo(arg): > given arg: > case as x, y, *_: # Tuple matching (implicit name binding) > ... > case as object(x=p, y=q, **_): # Attribute matching > ... > case as {'x': p, 'y', q, **_}: # Item matching > ... > case as object(x=p, **_) and isinstance(p, int): # Match + condition > ... > case if isinstance(arg, int): # Condition only > ... > else: # Default > ... 'case' is the signal word, 'as' is the seperator word -- but it should be separating user stuff, not keyword and all the user stuff clumped together. `case blah blah as blah blah reads much better to me, and clearly says "the stuff between 'case' and 'as' will now be known as the stuff between 'as' and ':'". -- ~Ethan~ From stephen at xemacs.org Wed May 25 05:23:29 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 25 May 2016 18:23:29 +0900 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> <20160524154506.GV12028@ando.pearwood.info> <20160524161913.GX12028@ando.pearwood.info> <57453AF9.8050602@canterbury.ac.nz> Message-ID: <22341.28433.987988.688927@turnbull.sk.tsukuba.ac.jp> Guido van Rossum writes: > Well, ZeroDivisionError doesn't derive from ValueError, does it? So > it may not be a good idea, but it's certainly a tradition. I think it's deeper than merely tradition (though that doesn't make it a good idea, either). One way to look at these exceptions is that you shouldn't apply that operation to that value (ArithmeticError), and another is that you shouldn't pass that value to that operation (ValueError). A distinction without a difference, I suppose, after thinking about how I would apply it to Python. Eg (and most tellingly, I think) as Python implements division, first you ask the numerator if it knows how to divide itself by something. It says "sure, here's my method," the method asks "what number?", and *then* when you tell it "zero", it says, "sorry, I don't do zero!" and that's a ValueError. The example above is sort of an artifact of division being a binary operation, so even with an OO implementation you end up passing a value to function that then decides it doesn't like it. In a language like Haskell, where a function always takes one argument, and returns a new function if the program wants to handle more than one argument, the distinction might make more sense. But I don't see a reason to pursue that line of thought here, so I won't even try. From p.f.moore at gmail.com Wed May 25 05:26:14 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 25 May 2016 10:26:14 +0100 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <5745567A.7070109@stoneleaf.us> References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <5745567A.7070109@stoneleaf.us> Message-ID: On 25 May 2016 at 08:38, Ethan Furman wrote: > On 05/24/2016 10:52 PM, Nick Coghlan wrote: > >> Using the running demo: >> >> def demo(arg): >> given arg: >> case x, y, *_: # Tuple matching (implicit name binding) >> ... >> case (.x, .y) as p, q: # Attribute matching >> ... >> case (["x"], ["y"]) as p, q: # Item matching >> ... >> case (.x) as p and isinstance(p, int): # Match + condition >> ... >> case if isinstance(arg, int): # Condition only >> ... >> else: # Default >> ... >> >> The other key change there is introducing "as" to the individual cases >> in order to be able to separate the match pattern definition from the >> local name binding. > > > With this one I have a clue as to what's going on. On first reading, I felt the same way. But rereading, I find that a number of odd cases start bothering me: (["x"], ["y"]) doesn't look like an item match the more I think about it. And to match "a 2-tuple with ["x"] as the first item would be ["x"], y which is very close to the item matching case. As an *example* it works, but I don't see any way to describe the detailed *semantics* without it being a mess of special cases. One thought that this *does* prompt, though - the details of the statement syntax are more or less bikeshed material in all this. The *real* meat of the debate is around how we express matches. So maybe we should focus solely on pattern matching as a primitive construct. If we have an agreed syntax for "a match" then working out how we incorporate that into a switch statement (or a "does this match" maybe-assignment expression, or whatever) would likely be a lot easier. So looking solely at a "match" as a concept, let's see where that takes us: - Unpacking syntax (NAME, NAME, *REST) handles matching sequences. - Matches should probably nest, so NAME in the above could be a (sub-)match. - For matching constants, if NAME is a constant rather than a name, then that element must equal the literal for the match to succeed. We'd probably need to restrict this to literals (https://docs.python.org/3/reference/expressions.html#grammar-token-literal) to avoid ambiguity. - Matching mappings could be handled using dict syntax {literal: NAME, literal: NAME, **REST}. Again allow recursive submatches and literal matches? - There's no "obvious" syntax for object mappings - maybe use type constructor syntax TYPE(attr=NAME, attr=NAME). In the case where you don't care about the type, we could use "object" (in theory, I don't think it's ambiguous to omit the type, but that may be difficult for the reader to understand). Also, TYPE() would then be an isinstance check - do we want that? - Top-level matches can have "and CONDITION" to do further tests on the matched values (we don't want to allow this for nested matches, though!) Translating (and extending a bit) Nick's example: def demo(arg): given arg: case (x, y, *_): # Tuple matching (implicit name binding) ... case object(x=p, y=q): # Attribute matching ... case {"x": p, "y": q): # Item matching ... case object(x=p) and isinstance(p, int): # Match + condition ... case int(): # Type match ... case (1, p, {"key": 0, "match": q}): # Match a sequence of length 3, first item must be 1, last # must be a mapping with a key "key" with value 0 and a key "match" ... else: # Default The worst one to my mind is the object match (not just in this style, but basically everywhere) - that's because there's no existing display or unpacking syntax for objects, so whatever we come up with is unfamiliar. I'm still a bit meh on this, though. Every proposal I've seen now (including the above!) looks natural for simple examples - and would probably look natural for 99% of real-world uses, which are typically simple! - but gets awfully messy in the corner cases. It feels like "If the implementation is hard to explain, it's a bad idea." may apply here (although it's less the "implementation" and more the "detailed semantics" that's hard to explain). On 25 May 2016 at 10:04, Franklin? Lee wrote: > Problem: `Point(x, y, ...)` is a legitimate function call, so if > `Point(x, 0)` is a legal pattern (i.e. no distinguishing syntax > between values and bindnames), you'd need the syntax to be `Point(x, > y, *...)`. Personally, I'd require that everything is a bindname > (unless it looks like a constructor call), and require checks to be in > guards. With the above syntax, "bare" values aren't a valid match, so in a match, Point(x, y) can never be a function call, it must be a match "Object of type Point, with x and y attributes (which we check for but don't bind)". Paul From rob.cliffe at btinternet.com Wed May 25 07:10:19 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 25 May 2016 12:10:19 +0100 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: <0239b3c2-f795-6418-5b27-f61673c8f643@btinternet.com> References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> <20160524154506.GV12028@ando.pearwood.info> <20160524161913.GX12028@ando.pearwood.info> <57453AF9.8050602@canterbury.ac.nz> <0239b3c2-f795-6418-5b27-f61673c8f643@btinternet.com> Message-ID: What I was trying to say was: I think of ValueError as applying to a single ("bad") value whereas ArithmeticError can arise from trying to combine multiple ("good") values. It wouldn't be appropriate for ArithmeticError to be a subclass of ValueError, because there is no single erroneous value that you can point to. On 25/05/2016 09:08, Rob Cliffe wrote: > > Or more generally, any operation on multiple values, where each value > is valid individually, but the result of the operation is not: > > >>> p = Decimal('1E999999999') > > >>> q = Decimal('10') > > >>> try: > > ... p*q > > ... except ArithmeticError: > > ... print 'ArithmeticError' > > ... > > ArithmeticError > > > > On 25/05/2016 06:48, Guido van Rossum wrote: >> Well, ZeroDivisionError doesn't derive from ValueError, does it? So >> it may not be a good idea, but it's certainly a tradition. >> >> On Tuesday, May 24, 2016, Greg Ewing > > wrote: >> >> Guido van Rossum wrote: >> >> It's very >> common for modules to define their own root exception class. >> >> >> And it's a nuisance when their exceptions only belong >> to their own private hierarchy and don't participate in >> any the standard classifications. >> >> I don't think Decimal is directly to blame here, because >> deriving its exceptions from ArithmeticError seems like >> a reasonable thing to do. What *doesn't* seem reasonable >> to me is that ArithmeticError doesn't derive from >> ValueError. So far nobody has explained why that's >> a good idea. >> >> -- >> Greg >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> >> -- >> --Guido (mobile) >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct:http://python.org/psf/codeofconduct/ > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.selik at gmail.com Wed May 25 09:11:35 2016 From: michael.selik at gmail.com (Michael Selik) Date: Wed, 25 May 2016 13:11:35 +0000 Subject: [Python-ideas] Unpacking a dict Message-ID: Python's iterable unpacking is what Lispers might call a destructuring bind. py> iterable = 1, 2, 3, 4, 5 py> a, b, *rest = iterable py> a, b, rest (1, 2, (3, 4, 5)) Clojure also supports mapping destructuring. Let's add that to Python! py> mapping = {"a": 1, "b": 2, "c": 3} py> {"a": x, "b": y, "c": z} = mapping py> x, y, z (1, 2, 3) py> {"a": x, "b": y} = mapping Traceback: ValueError: too many keys to unpack This will be approximately as helpful as iterable unpacking was before PEP 3132 (https://www.python.org/dev/peps/pep-3132/). I hope to keep discussion in this thread focused on the most basic form of dict unpacking, but we could extended mapping unpacking similarly to how PEP 3132 extended iterable unpacking. Just brainstorming... py> mapping = {"a": 1, "b": 2, "c": 3} py> {"a": x, **rest} = mapping py> x, rest (1, {"b": 2, "c": 3}) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed May 25 09:21:25 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 25 May 2016 14:21:25 +0100 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: Message-ID: On 25 May 2016 at 14:11, Michael Selik wrote: > Clojure also supports mapping destructuring. Let's add that to Python! The part of me that likes new shiny things says "ooh, yes!" But the part of me that has to support software wants to know what improvements this would make to real-world code. Personally, I can't think of anywhere I'd have actually used a construct like this. Paul From srkunze at mail.de Wed May 25 09:58:58 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 25 May 2016 15:58:58 +0200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: Message-ID: <5745AFA2.402@mail.de> On 25.05.2016 15:11, Michael Selik wrote: > Clojure also supports mapping destructuring. Let's add that to Python! +1 for the general idea. Let's keep Python simple by providing flexible tools like the one you proposed instead of a monolithic switch-case. :) > > py> mapping = {"a": 1, "b": 2, "c": 3} > py> {"a": x, "b": y, "c": z} = mapping > py> x, y, z > (1, 2, 3) > py> {"a": x, "b": y} = mapping > Traceback: > ValueError: too many keys to unpack Nice! I like the error message. I could imagine how this can be generalized to attribute access BUT I think its easier to discuss dict unpacking to the end first. > This will be approximately as helpful as iterable unpacking was before > PEP 3132 (https://www.python.org/dev/peps/pep-3132/). I'd add that all benefits for iterable unpacking apply as well. So, what's true for iterable unpacking is true for dict unpacking, too. > I hope to keep discussion in this thread focused on the most basic > form of dict unpacking, but we could extended mapping unpacking > similarly to how PEP 3132 extended iterable unpacking. Just > brainstorming... > > py> mapping = {"a": 1, "b": 2, "c": 3} > py> {"a": x, **rest} = mapping > py> x, rest > (1, {"b": 2, "c": 3}) That's basically suppressing the the ValueError above with the same justification as for PEP 3132. I for one find this one of the shortest proposals compared to other recent proposals I have seen. :) Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Wed May 25 10:40:32 2016 From: random832 at fastmail.com (Random832) Date: Wed, 25 May 2016 10:40:32 -0400 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: Message-ID: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> On Wed, May 25, 2016, at 09:11, Michael Selik wrote: > Python's iterable unpacking is what Lispers might call a destructuring > bind. > > py> iterable = 1, 2, 3, 4, 5 > py> a, b, *rest = iterable > py> a, b, rest > (1, 2, (3, 4, 5)) > > Clojure also supports mapping destructuring. Let's add that to Python! > > py> mapping = {"a": 1, "b": 2, "c": 3} > py> {"a": x, "b": y, "c": z} = mapping > py> x, y, z > (1, 2, 3) > py> {"a": x, "b": y} = mapping > Traceback: > ValueError: too many keys to unpack How is this better than: >>> mapping = {"a": 1, "b": 2, "c": 3} >>> x, y, z = mapping[k] for k in ("a", "b", "c") From srkunze at mail.de Wed May 25 11:07:39 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 25 May 2016 17:07:39 +0200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: Message-ID: <5745BFBB.2060001@mail.de> On 25.05.2016 15:11, Michael Selik wrote: > I hope to keep discussion in this thread focused on the most basic > form of dict unpacking, but we could extended mapping unpacking > similarly to how PEP 3132 extended iterable unpacking. Just > brainstorming... Another idea (borrowed from Erlang): >>> mapping = {"a": 1, "b": 2, "c": 3} >>> {"a": x, "b": y, "c": 3} = mapping >>> x,y (1,2) >>> mapping = {"a": 1, "b": 2, "c": 3} >>> {"a": x, "b": y, "c": 2} = mapping Traceback: ValueError: key 'c' does not match to 2 Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed May 25 11:18:52 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 25 May 2016 16:18:52 +0100 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: Message-ID: On 25 May 2016 at 14:11, Michael Selik wrote: > Clojure also supports mapping destructuring. Let's add that to Python! > > py> mapping = {"a": 1, "b": 2, "c": 3} > py> {"a": x, "b": y, "c": z} = mapping > py> x, y, z > (1, 2, 3) > py> {"a": x, "b": y} = mapping > Traceback: > ValueError: too many keys to unpack > > > This will be approximately as helpful as iterable unpacking was before PEP > 3132 (https://www.python.org/dev/peps/pep-3132/). Neither this, nor the **rest extension you proposed, does what I think would be the most common requirement - get a set of elements and *ignore* the rest: >>> mapping = {"a": 1, "b": 2, "c": 3} >>> {"a": x, "b": y} = mapping >>> # Note no error! >>> x, y, z (1, 2) I'd still like to see some real-world use cases though - there are lots of options as the above example demonstrates, and nothing to guide us in deciding which would be the most useful one to choose. Paul From ethan at stoneleaf.us Wed May 25 12:08:45 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 May 2016 09:08:45 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: Message-ID: <5745CE0D.8030300@stoneleaf.us> On 05/25/2016 08:18 AM, Paul Moore wrote: > Neither this, nor the **rest extension you proposed, does what I think > would be the most common requirement - get a set of elements and > *ignore* the rest: > >--> mapping = {"a": 1, "b": 2, "c": 3} >--> {"a": x, "b": y} = mapping >--> # Note no error! >--> x, y # z discarded, should be NameError > (1, 2) > > I'd still like to see some real-world use cases though - there are > lots of options as the above example demonstrates, and nothing to > guide us in deciding which would be the most useful one to choose. Here's a real-world use-case: The main software app I support passes a `values` dict around like the plague. When doing interesting stuff I often unpack some of the values into local variables as that reads better, types better, and makes it easier to reason about the code (it's on 2.7 so for that I'll have to use Random's example). So for me, unpacking a dict with friendly syntax would be useful, and unpacking four or five keys from a 20-element dict will be far more useful than having to unpack them all. -- ~Ethan~ From leewangzhong+python at gmail.com Wed May 25 13:09:31 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Wed, 25 May 2016 13:09:31 -0400 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <5745567A.7070109@stoneleaf.us> Message-ID: On Wed, May 25, 2016 at 5:26 AM, Paul Moore wrote: > On 25 May 2016 at 10:04, Franklin? Lee wrote: >> Problem: `Point(x, y, ...)` is a legitimate function call, so if >> `Point(x, 0)` is a legal pattern (i.e. no distinguishing syntax >> between values and bindnames), you'd need the syntax to be `Point(x, >> y, *...)`. Personally, I'd require that everything is a bindname >> (unless it looks like a constructor call), and require checks to be in >> guards. > > With the above syntax, "bare" values aren't a valid match, so in a > match, Point(x, y) can never be a function call, it must be a match > "Object of type Point, with x and y attributes (which we check for but > don't bind)". I mean a function call, in particular a constructor call. Take these examples, written as constructor destructure assignment: Point(x, y) = obj Point(x=_, y=_) = obj I'd say that Point(x, y) binds two positional constructor arguments, and Point(x=_, y=_) binds attributes .x and .y to throwaway variables. (`object` takes no positional args, and I want it to start allowing keyword args.) Here's my proposed implementation again: Call Point.__asmatch__(obj, nargs, keys, args, kwargs) (where "asmatch" might be an enhanced "__getnewargs__"), with these params: - obj: The object to destructure, which is an instance of Point (possibly typechecked by the match engine before calling). (Note that we're not calling obj.__asmatch__(nargs, ...), to give superclasses a chance to destructure a subclass instance.) - nargs: Number of positional args. - keys: Keyword args specified. - args: Whether or not *rest was used. (Extension: Whether or not splat was used, and whether to discard them.) - kwargs: Whether or not **rest was used. (Extension: and whether to discard them.) It will return (args: tuple, kwargs: dict) if the match is possible. It will check whether there are enough positional args, and whether all keys were valid, and perform the logic about the optional params (giving up as few args and kwargs as possible). Contract: The returned tuple and dict _should_ be valid for Point(*args, **kwargs). This constraint is not necessary if "discard" was specified for anything. My problem is, this would then be ambiguous: Point(0, ...) = obj # Match if the first arg is 0 and the second is `...`? while this would not, but might be ugly. Point(0, *...) = obj From desmoulinmichel at gmail.com Wed May 25 13:49:57 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Wed, 25 May 2016 19:49:57 +0200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: Message-ID: <5745E5C5.3060801@gmail.com> I'm currently working on a data export softare using 1000th of OrderedDicts. Sometime I iterate, and sometime I get part of it. Currently I have a wrapper doing: x, y , z = d(data).unpack('x', 'y', 'z', default=None) So I can see a use for it. And it's not the first time I wish I could do this. Although I think for a shortcut to lookup automatically the vars with the same name as the dict keys by default would be nice: {x, y, z} = data Le 25/05/2016 17:18, Paul Moore a ?crit : > On 25 May 2016 at 14:11, Michael Selik wrote: >> Clojure also supports mapping destructuring. Let's add that to Python! >> >> py> mapping = {"a": 1, "b": 2, "c": 3} >> py> {"a": x, "b": y, "c": z} = mapping >> py> x, y, z >> (1, 2, 3) >> py> {"a": x, "b": y} = mapping >> Traceback: >> ValueError: too many keys to unpack >> >> >> This will be approximately as helpful as iterable unpacking was before PEP >> 3132 (https://www.python.org/dev/peps/pep-3132/). > > Neither this, nor the **rest extension you proposed, does what I think > would be the most common requirement - get a set of elements and > *ignore* the rest: > >>>> mapping = {"a": 1, "b": 2, "c": 3} >>>> {"a": x, "b": y} = mapping >>>> # Note no error! >>>> x, y, z > (1, 2) > > I'd still like to see some real-world use cases though - there are > lots of options as the above example demonstrates, and nothing to > guide us in deciding which would be the most useful one to choose. > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From p.f.moore at gmail.com Wed May 25 14:12:37 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 25 May 2016 19:12:37 +0100 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <5745CE0D.8030300@stoneleaf.us> References: <5745CE0D.8030300@stoneleaf.us> Message-ID: On 25 May 2016 at 17:08, Ethan Furman wrote: > Here's a real-world use-case: The main software app I support passes a > `values` dict around like the plague. When doing interesting stuff I often > unpack some of the values into local variables as that reads better, types > better, and makes it easier to reason about the code (it's on 2.7 so for > that I'll have to use Random's example). > > So for me, unpacking a dict with friendly syntax would be useful, and > unpacking four or five keys from a 20-element dict will be far more useful > than having to unpack them all. Thanks. That's the sort of use case I thought might exist - and it sounds to me as if you'd get much more benefit from a syntax that allowed "partial" unpacking: {"a": x, "b": y} = dict(a=1, b=2, c=3) gives x=1, y=2 with no error. I can't think of a good example where Michael's original proposal that this would give a ValueError would be a better approach. Paul From ethan at stoneleaf.us Wed May 25 14:21:00 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 May 2016 11:21:00 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <5745CE0D.8030300@stoneleaf.us> Message-ID: <5745ED0C.5080108@stoneleaf.us> On 05/25/2016 11:12 AM, Paul Moore wrote: > On 25 May 2016 at 17:08, Ethan Furman wrote: >> Here's a real-world use-case: The main software app I support passes a >> `values` dict around like the plague. When doing interesting stuff I often >> unpack some of the values into local variables as that reads better, types >> better, and makes it easier to reason about the code (it's on 2.7 so for >> that I'll have to use Random's example). >> >> So for me, unpacking a dict with friendly syntax would be useful, and >> unpacking four or five keys from a 20-element dict will be far more useful >> than having to unpack them all. > > Thanks. That's the sort of use case I thought might exist - and it > sounds to me as if you'd get much more benefit from a syntax that > allowed "partial" unpacking: > > {"a": x, "b": y} = dict(a=1, b=2, c=3) > > gives x=1, y=2 with no error. > > I can't think of a good example where Michael's original proposal that > this would give a ValueError would be a better approach. Agreed. The ValueError approach would make this useless for me. -- ~Ethan~ From steve at pearwood.info Wed May 25 14:42:08 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 26 May 2016 04:42:08 +1000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: Message-ID: <20160525184206.GZ12028@ando.pearwood.info> On Wed, May 25, 2016 at 01:11:35PM +0000, Michael Selik wrote: > py> mapping = {"a": 1, "b": 2, "c": 3} > py> {"a": x, "b": y, "c": z} = mapping > py> x, y, z > (1, 2, 3) I think that is too verbose and visually baffling. I'd rather see something less general and (in my opinion) more useful: a, b, c = **mapping being equivalent to: a = mapping['a'] b = mapping['b'] c = mapping['c'] It's less general, because you can only use the same names as the keys in the dict. But most of the time you'll probably want to do that anyway. Think of (for example) carrying around a "settings" or "preferences" dict: prefs = {'width': 80, 'height': 200, 'verbose': False, 'mode': PLAIN, 'name': 'Fnord', 'flags': spam|eggs|cheese, ... } # plus many more keys:values There's no need to unpack the entire dict, you can grab only the keys you need: width, height = **prefs # like width = prefs['width'] height = prefs['height'] Sure, you are forced to use the same variable names as the keys, but that's what you will probably do most of the time. You're not likely to write: foo = prefs['width'] bar = prefs['height'] although you might write: zip_code = prefs['zip code'] but probably shouldn't. (Just use 'zip_code' as the key.) Another awkward case is when a key is a keyword: except_ = prefs['except'] but I expect those cases will be relatively rare, and you can always manually unpack them the old fashioned way. Admittedly your syntax would allow those cases, at the cost of a more verbose statement: {'width': foo, 'height': bar, 'except': except_, 'zip code': zip_code} = mapping but I think that's mostly an over-generalisation and too hard to grasp what is going on. Naturally the order of the keys doesn't matter: mode, height, width = **prefs height, mode, width = **prefs etc are all the same. If you twist my arm and force me to come up with syntax for a "change of variable name", I'd consider: height, width, zip_code:'zip code', except_:'except' = **mapping In other words, if the target on the left is a plain name, the unpacking does: name = mapping['name'] If the target on the left has a colon, it is an identifier followed by key. The identifier can be any valid reference, including dots and [] subscripts. The key must be a string: identifier:'key' which performs: identifier = mapping['key'] Examples of valid colon targets: spam:'ham' # like spam = mapping['ham'] spam[1].eggs:'while' # like spam[1].eggs = mapping['while'] etc. If there's too many targets to comfortably fit on the one line, wrap them in parentheses to allow line wrapping: (height, flags, except_:'except', mymodule.obj.attribute[2].gamma:'gamma', alpha) = **prefs But the simple case, the case you'll use most of the time, is simple: height, width, verbose, flags = **prefs -- Steve From srkunze at mail.de Wed May 25 14:44:27 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 25 May 2016 20:44:27 +0200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <5745ED0C.5080108@stoneleaf.us> References: <5745CE0D.8030300@stoneleaf.us> <5745ED0C.5080108@stoneleaf.us> Message-ID: <5745F28B.8020508@mail.de> On 25.05.2016 20:21, Ethan Furman wrote: > On 05/25/2016 11:12 AM, Paul Moore wrote: >> On 25 May 2016 at 17:08, Ethan Furman wrote: > >> I can't think of a good example where Michael's original proposal that >> this would give a ValueError would be a better approach. > > Agreed. The ValueError approach would make this useless for me. I think that's why the * syntax might be useful as well. Best, Sven From ethan at stoneleaf.us Wed May 25 14:47:22 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 May 2016 11:47:22 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <20160525184206.GZ12028@ando.pearwood.info> References: <20160525184206.GZ12028@ando.pearwood.info> Message-ID: <5745F33A.1050907@stoneleaf.us> On 05/25/2016 11:42 AM, Steven D'Aprano wrote: > On Wed, May 25, 2016 at 01:11:35PM +0000, Michael Selik wrote: >> py> mapping = {"a": 1, "b": 2, "c": 3} >> py> {"a": x, "b": y, "c": z} = mapping >> py> x, y, z >> (1, 2, 3) > > I think that is too verbose and visually baffling. I'd rather see > something less general and (in my opinion) more useful: > > a, b, c = **mapping > > being equivalent to: > > a = mapping['a'] > b = mapping['b'] > c = mapping['c'] +1 Simplest, easiest to grok, probably solves 95+% of the use-cases. -- ~Ethan~ From ethan at stoneleaf.us Wed May 25 14:48:05 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 May 2016 11:48:05 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <5745F28B.8020508@mail.de> References: <5745CE0D.8030300@stoneleaf.us> <5745ED0C.5080108@stoneleaf.us> <5745F28B.8020508@mail.de> Message-ID: <5745F365.6090609@stoneleaf.us> On 05/25/2016 11:44 AM, Sven R. Kunze wrote: > On 25.05.2016 20:21, Ethan Furman wrote: >> On 05/25/2016 11:12 AM, Paul Moore wrote: >>> On 25 May 2016 at 17:08, Ethan Furman wrote: >> >>> I can't think of a good example where Michael's original proposal that >>> this would give a ValueError would be a better approach. >> >> Agreed. The ValueError approach would make this useless for me. > > I think that's why the * syntax might be useful as well. Maybe. Get the simple case added first; we can add more later if deemed appropriate. -- ~Ethan~ From rosuav at gmail.com Wed May 25 14:51:42 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 26 May 2016 04:51:42 +1000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <5745F33A.1050907@stoneleaf.us> References: <20160525184206.GZ12028@ando.pearwood.info> <5745F33A.1050907@stoneleaf.us> Message-ID: On Thu, May 26, 2016 at 4:47 AM, Ethan Furman wrote: > On 05/25/2016 11:42 AM, Steven D'Aprano wrote: >> >> On Wed, May 25, 2016 at 01:11:35PM +0000, Michael Selik wrote: > > >>> py> mapping = {"a": 1, "b": 2, "c": 3} >>> py> {"a": x, "b": y, "c": z} = mapping >>> py> x, y, z >>> (1, 2, 3) >> >> >> I think that is too verbose and visually baffling. I'd rather see >> something less general and (in my opinion) more useful: >> >> a, b, c = **mapping >> >> being equivalent to: >> >> a = mapping['a'] >> b = mapping['b'] >> c = mapping['c'] > > > +1 > > Simplest, easiest to grok, probably solves 95+% of the use-cases. Agreed, and also +1 on the proposal. It leaves room for future syntactic expansion in the same way as PEP 448. ChrisA From p.f.moore at gmail.com Wed May 25 14:52:53 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 25 May 2016 19:52:53 +0100 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <5745F33A.1050907@stoneleaf.us> References: <20160525184206.GZ12028@ando.pearwood.info> <5745F33A.1050907@stoneleaf.us> Message-ID: On 25 May 2016 at 19:47, Ethan Furman wrote: > On 05/25/2016 11:42 AM, Steven D'Aprano wrote: >> >> On Wed, May 25, 2016 at 01:11:35PM +0000, Michael Selik wrote: > > >>> py> mapping = {"a": 1, "b": 2, "c": 3} >>> py> {"a": x, "b": y, "c": z} = mapping >>> py> x, y, z >>> (1, 2, 3) >> >> >> I think that is too verbose and visually baffling. I'd rather see >> something less general and (in my opinion) more useful: >> >> a, b, c = **mapping >> >> being equivalent to: >> >> a = mapping['a'] >> b = mapping['b'] >> c = mapping['c'] > > > +1 > > Simplest, easiest to grok, probably solves 95+% of the use-cases. OTOH, you could also do x = SimpleNamespace(**mapping) and use x.a, x.b, x.c. Paul From brenbarn at brenbarn.net Wed May 25 14:55:10 2016 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Wed, 25 May 2016 11:55:10 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <20160525184206.GZ12028@ando.pearwood.info> References: <20160525184206.GZ12028@ando.pearwood.info> Message-ID: <5745F50E.2000607@brenbarn.net> On 2016-05-25 11:42, Steven D'Aprano wrote: > If the target on the left has a colon, it is an identifier followed by > key. The identifier can be any valid reference, including dots and [] > subscripts. The key must be a string: > > identifier:'key' > > which performs: > > identifier = mapping['key'] Why does the key have to be a string? I agree that the common case is where you want to assign to a local variable with the same name as the string key, but if you do allow specifying how keys map to assignment targets, it seems like you might as well allow non-string keys. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From ethan at stoneleaf.us Wed May 25 15:03:41 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 May 2016 12:03:41 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160525184206.GZ12028@ando.pearwood.info> <5745F33A.1050907@stoneleaf.us> Message-ID: <5745F70D.8000006@stoneleaf.us> On 05/25/2016 11:52 AM, Paul Moore wrote: > On 25 May 2016 at 19:47, Ethan Furman wrote: >> On 05/25/2016 11:42 AM, Steven D'Aprano wrote: >>> On Wed, May 25, 2016 at 01:11:35PM +0000, Michael Selik wrote: >>>> py> mapping = {"a": 1, "b": 2, "c": 3} >>>> py> {"a": x, "b": y, "c": z} = mapping >>>> py> x, y, z >>>> (1, 2, 3) >>> >>> >>> I think that is too verbose and visually baffling. I'd rather see >>> something less general and (in my opinion) more useful: >>> >>> a, b, c = **mapping >>> >>> being equivalent to: >>> >>> a = mapping['a'] >>> b = mapping['b'] >>> c = mapping['c'] >> >> >> +1 >> >> Simplest, easiest to grok, probably solves 95+% of the use-cases. > > OTOH, you could also do > > x = SimpleNamespace(**mapping) > > and use x.a, x.b, x.c. Beside visual clarity, the other big reason for using local variables instead constant dict access is speed -- which you lose by using another object. -- ~Ethan~ From srkunze at mail.de Wed May 25 15:38:44 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 25 May 2016 21:38:44 +0200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <20160525184206.GZ12028@ando.pearwood.info> References: <20160525184206.GZ12028@ando.pearwood.info> Message-ID: <5745FF44.60105@mail.de> On 25.05.2016 20:42, Steven D'Aprano wrote: > There's no need to unpack the entire dict, you can grab only the keys > you need: > > width, height = **prefs > > # like > width = prefs['width'] > height = prefs['height'] That's not the same behavior as it is for tuple unpacking. As a new user, I would expect them to work the same way. If I want to dismiss the remainder of the dict, I'd rather consider an explicit approach: width, height = **prefs # just those values width, height, *r = **prefs # r captures the rest This gives me two benefits: 1) I can further work with r from which width and height are extracted 2) a convenient way of verifying that a dict only contains certain values and extracting those at the same time > Sure, you are forced to use the same variable names as the keys, but > that's what you will probably do most of the time. You're not likely to > write: > > foo = prefs['width'] > bar = prefs['height'] > > although you might write: > > zip_code = prefs['zip code'] > > but probably shouldn't. (Just use 'zip_code' as the key.) "just use 'zip_code' as the key" won't do it when you have no influence on the data. I might remind you that most engineers are no gods who can change everything at their whim. > [...] > If you twist my arm and force me to come up with syntax for a "change of > variable name", I'd consider: Rest assured I will twist your arm very hard. ;) Also rest assured that we use non-string keys on a regular basis. Maybe, it's just me but I still tend to think that using unquoted strings should be reserved for attribute unpacking. > [...] > > If the target on the left has a colon, it is an identifier followed by > key. The identifier can be any valid reference, including dots and [] > subscripts. The key must be a string: > > identifier:'key' I would rather turn it around. It feels weird to have the "key" on the right side. One additional drawback of this solution is the fact that the "value" part of the colon is already taken. So, value matching like done in Erlang is not possible. OTOH, this applies to tuple unpacking in its current form as well Best, Sven From eric at trueblade.com Wed May 25 16:08:10 2016 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 25 May 2016 16:08:10 -0400 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <5745CE0D.8030300@stoneleaf.us> Message-ID: <5746062A.9010901@trueblade.com> On 05/25/2016 02:12 PM, Paul Moore wrote: > On 25 May 2016 at 17:08, Ethan Furman wrote: >> Here's a real-world use-case: The main software app I support passes a >> `values` dict around like the plague. When doing interesting stuff I often >> unpack some of the values into local variables as that reads better, types >> better, and makes it easier to reason about the code (it's on 2.7 so for >> that I'll have to use Random's example). >> >> So for me, unpacking a dict with friendly syntax would be useful, and >> unpacking four or five keys from a 20-element dict will be far more useful >> than having to unpack them all. > > Thanks. That's the sort of use case I thought might exist - and it > sounds to me as if you'd get much more benefit from a syntax that > allowed "partial" unpacking: > > {"a": x, "b": y} = dict(a=1, b=2, c=3) > > gives x=1, y=2 with no error. > > I can't think of a good example where Michael's original proposal that > this would give a ValueError would be a better approach. > Paul How is this an improvement over: def extract(mapping, *keys): return [mapping[key] for key in keys] mapping = {'a': 1, 'b': 2, 'c': 3} x, y = extract(mapping, 'a', 'b') print(x, y) 1, 2 Eric. From ethan at stoneleaf.us Wed May 25 16:11:11 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 May 2016 13:11:11 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <5745FF44.60105@mail.de> References: <20160525184206.GZ12028@ando.pearwood.info> <5745FF44.60105@mail.de> Message-ID: <574606DF.2060506@stoneleaf.us> On 05/25/2016 12:38 PM, Sven R. Kunze wrote: > On 25.05.2016 20:42, Steven D'Aprano wrote: >> There's no need to unpack the entire dict, you can grab only the keys >> you need: >> >> width, height = **prefs >> >> # like >> width = prefs['width'] >> height = prefs['height'] > > That's not the same behavior as it is for tuple unpacking. As a new > user, I would expect them to work the same way. > > If I want to dismiss the remainder of the dict, I'd rather consider an > explicit approach: > > width, height = **prefs # just those values > width, height, *r = **prefs # r captures the rest > > > This gives me two benefits: > > 1) I can further work with r from which width and height are extracted > 2) a convenient way of verifying that a dict only contains certain > values and extracting those at the same time Okay, those are good benefits. +1 > Maybe, it's just me but I still tend to think that using unquoted > strings should be reserved for attribute unpacking. lists, tuples, and dicts are basic containers -- having syntax to unpacak them easily is a clear win; attribute unpacking not so much: --> some_long_object_name = MyClass(blah, blah) --> s = some_long_object_name --> s.name some value --> s.value --> another value which is sufficiently readable. > One additional drawback of this solution is the fact that the "value" > part of the colon is already taken. So, value matching like done in > Erlang is not possible. OTOH, this applies to tuple unpacking in its > current form as well This is about unpacking, not matching. -- ~Ethan~ From ethan at stoneleaf.us Wed May 25 16:14:37 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 May 2016 13:14:37 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <5746062A.9010901@trueblade.com> References: <5745CE0D.8030300@stoneleaf.us> <5746062A.9010901@trueblade.com> Message-ID: <574607AD.3040207@stoneleaf.us> On 05/25/2016 01:08 PM, Eric V. Smith wrote: > How is this an improvement over: > > def extract(mapping, *keys): > return [mapping[key] for key in keys] > > mapping = {'a': 1, 'b': 2, 'c': 3} > > x, y = extract(mapping, 'a', 'b') > print(x, y) > 1, 2 Let's pretend you wrote: a, b = extract(mapping, 'a', 'b') since that's the way I would almost always be using it. The proposal is this: a, b = **mapping The advantages: - much more readable - less duplication Less duplication might not seem like that big a deal, but it's one of the motivators behind decorations and in-place operators, which are both wins. -- ~Ethan~ From Nikolaus at rath.org Wed May 25 16:48:44 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Wed, 25 May 2016 13:48:44 -0700 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> (Stephen J. Turnbull's message of "Wed, 25 May 2016 14:03:04 +0900") References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> Message-ID: <877fehuamr.fsf@thinkpad.rath.org> On May 25 2016, "Stephen J. Turnbull" wrote: > How about > > for : > try : > pass > try : > pass Or maybe: if ... matches : pass matches : pass Where "..." is an actual syntactic element that replaces the ":" in a regular if. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From eric at trueblade.com Wed May 25 17:01:56 2016 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 25 May 2016 17:01:56 -0400 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <574607AD.3040207@stoneleaf.us> References: <5745CE0D.8030300@stoneleaf.us> <5746062A.9010901@trueblade.com> <574607AD.3040207@stoneleaf.us> Message-ID: <3a4c2c02-2546-e115-4876-3c2e3226da20@trueblade.com> On 5/25/2016 4:14 PM, Ethan Furman wrote: > On 05/25/2016 01:08 PM, Eric V. Smith wrote: >> x, y = extract(mapping, 'a', 'b') >> print(x, y) >> 1, 2 > > Let's pretend you wrote: > > a, b = extract(mapping, 'a', 'b') > > since that's the way I would almost always be using it. > > The proposal is this: > > a, b = **mapping > > The advantages: > > - much more readable > - less duplication > > Less duplication might not seem like that big a deal, but it's one of > the motivators behind decorations and in-place operators, which are both > wins. I agree that would be a win. That's a lot of compiler magic, though. Eric. From michael.selik at gmail.com Wed May 25 18:52:01 2016 From: michael.selik at gmail.com (Michael Selik) Date: Wed, 25 May 2016 22:52:01 +0000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> References: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> Message-ID: I'm responding here to Sven, Random832, and Ethan. On Wed, May 25, 2016 at 10:08 AM Sven R. Kunze wrote: > I for one find this one of the shortest proposals compared to other recent > proposals I have seen. :) > If it's easy to explain, it might be a good idea :-) On Wed, May 25, 2016 at 10:40 AM Random832 wrote: > On Wed, May 25, 2016, at 09:11, Michael Selik wrote: > > Clojure also supports mapping destructuring. Let's add that to Python! > > > > py> mapping = {"a": 1, "b": 2, "c": 3} > > py> {"a": x, "b": y, "c": z} = mapping > > How is this better than: > py> mapping = {"a": 1, "b": 2, "c": 3} > py> x, y, z = mapping[k] for k in ("a", "b", "c") > I think the thread has formed a consensus that there are at least 2 clear use cases for unpacking. Not surprisingly, they're the same use cases for both tuple unpacking and dict unpacking. 1. declarative schema validation while simultaneously binding variables 2. declaratively extracting a subset of the elements In your example, what if the dict has more keys than you are looping over? Look at the other part of my proposal: py> mapping = {"a": 1, "b": 2, "c": 3} py> {"a": x, "b": y} = mapping Traceback: ValueError: too many keys to unpack I really like Sven's example. py> mapping = {"a": 1, "b": 2, "c": 3} py> {"a": x, "b": y, "c": 2} = mapping Traceback: ValueError: key 'c' does not match value 2 Even if we don't implement this feature in the first version of dict unpacking, we should keep the option open. On Wed, May 25, 2016 at 4:14 PM Ethan Furman wrote: > The proposal is this: > a, b = **mapping > > The advantages: > - much more readable > - less duplication > Why doesn't that work for tuple unpacking? py> a, b = *iterable SyntaxError Whatever the reasons, that syntax wasn't chosen for tuple unpacking. Dict unpacking should mimic tuple unpacking. If I saw ``a, b = **mapping`` I would expect ``a, b = *iterable``. Unpacking a tuple mirrors a tuple display. py> (a, b) = (1, 2) py> (a, b) = (1, 2, 3) ValueError: too many values to unpack, expected 2 Unpacking a dict should mirror a dict display. py> {'x': a, 'y': b} = {'x': 1, 'y': 2} py> {'x': a, 'y': b} = {'x': 1, 'y': 2, 'z': 3} ValueError: too many keys to unpack, expected {'x', 'y'} As Brendan and others have mentioned, the more concise syntax you're proposing will not support non-string keys and cannot be enhanced to support Erlang/Clojure/etc-style matching on values. On Wed, May 25, 2016 at 11:18 AM Paul Moore wrote: > get a set of elements and *ignore* the rest: If it's just one or two, that's easy. Use an underscore to indicate you don't care. Again, I'm trying to mirror a dict display. This is the same way that tuple unpacking solves the problem. py> (a, b, _) = (1, 2, 3) py> {'x': a, 'y': b, 'z': _} = {'x': 1, 'y': 2, 'z': 3} If you need to ignore many, we need to extend dict unpacking the same way that tuple unpacking was extended in PEP 3132. py> (a, b, *rest) = (1, 2, 3, 4) py> a, b, rest (1, 2, (3, 4)) py> {'x': a, 'y', b, **rest} = {'x': 1, 'y': 2, 'z': 3, 'w': 4} py> a, b, rest (1, 2, {'w': 4, 'z': 3}) Sure, **rest isn't valid in a dict display, but it's the same with tuple unpacking: *rest isn't valid in a tuple display. -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed May 25 19:03:40 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 26 May 2016 01:03:40 +0200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <574606DF.2060506@stoneleaf.us> References: <20160525184206.GZ12028@ando.pearwood.info> <5745FF44.60105@mail.de> <574606DF.2060506@stoneleaf.us> Message-ID: <57462F4C.8090908@mail.de> On 25.05.2016 22:11, Ethan Furman wrote: > On 05/25/2016 12:38 PM, Sven R. Kunze wrote: >> Maybe, it's just me but I still tend to think that using unquoted >> strings should be reserved for attribute unpacking. > > lists, tuples, and dicts are basic containers -- having syntax to > unpacak them easily is a clear win; attribute unpacking not so much: Wait a second. Reading your example below, do you imply that dicts actually should provide something like a attribute access to its keys? >>> d = {'a': 1,'b': 2} >>> d.b 2 Wouldn't this solution be equally readable/useful? I am just asking as it seems that you find accessing a dict to be straining. So, unpacking it would a one solution but providing attribute-like access to it would be another solution to that problem. > --> some_long_object_name = MyClass(blah, blah) > --> s = some_long_object_name > --> s.name > some value > --> s.value > --> another value > > which is sufficiently readable. I see your point. But let me explain how I approached the problem by considering it from the the "new users" perspective. Really fast, he will build up the following associations: attributes <-> .abc dict keys <-> ['abc'] or [other stuff] list/tuple keys <-> [123] So, from his perspective everything is clearly separated: lists/tuples use integers, dicts use mostly strings with quotes and objects have attributes which can be used like normal variables without quotes. Mixing things up (by removing/mixing the visual indicators), makes me feel nervous about the consistent perception of Python. Maybe, it's just FUD on my side but I can't help this feeling. Best, Sven From ethan at stoneleaf.us Wed May 25 19:25:47 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 May 2016 16:25:47 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <57462F4C.8090908@mail.de> References: <20160525184206.GZ12028@ando.pearwood.info> <5745FF44.60105@mail.de> <574606DF.2060506@stoneleaf.us> <57462F4C.8090908@mail.de> Message-ID: <5746347B.8020809@stoneleaf.us> On 05/25/2016 04:03 PM, Sven R. Kunze wrote: > On 25.05.2016 22:11, Ethan Furman wrote: >> On 05/25/2016 12:38 PM, Sven R. Kunze wrote: >>> Maybe, it's just me but I still tend to think that using unquoted >>> strings should be reserved for attribute unpacking. >> >> lists, tuples, and dicts are basic containers -- having syntax to >> unpacak them easily is a clear win; attribute unpacking not so much: > > Wait a second. Reading your example below, do you imply that dicts > actually should provide something like a attribute access to its keys? > > >>> d = {'a': 1,'b': 2} > >>> d.b > 2 No. I'm saying attribute access is already extremely easy, so we don't need to try and make it easier. > But let me explain how I approached the problem by considering it from > the the "new users" perspective. Really fast, he will build up the > following associations: > > attributes <-> .abc > dict keys <-> ['abc'] or [other stuff] > list/tuple keys <-> [123] > > So, from his perspective everything is clearly separated: lists/tuples > use integers, dicts use mostly strings with quotes and objects have > attributes which can be used like normal variables without quotes. And this newbie would be wrong, as [other stuff] for a mapping can easily be integers just like lists/tuples. > Mixing things up (by removing/mixing the visual indicators), makes me > feel nervous about the consistent perception of Python. Maybe, it's just > FUD on my side but I can't help this feeling. We already have that situation: some_var = [1, 2] a = some_var[0] b = some_var[1] is exactly the same as a, b = some_var and the visual indicators (the integers, the item access) are nowhere to be seen. -- ~Ethan~ From ethan at stoneleaf.us Wed May 25 19:41:57 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 May 2016 16:41:57 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> Message-ID: <57463845.8090709@stoneleaf.us> On 05/25/2016 03:52 PM, Michael Selik wrote: > I really like Sven's example. > > py> mapping = {"a": 1, "b": 2, "c": 3} > py>{"a": x, "b": y, "c": 2} = mapping > Traceback: > ValueError: key 'c' does not match value 2 > > Even if we don't implement this feature in the first version of dict > unpacking, we should keep the option open. Ugh, no. That is not dict unpacking, it's dict matching and unpacking, which is way beyond the simplicity of just unpacking. > On Wed, May 25, 2016 at 4:14 PM Ethan Furman wrote: >> >> The proposal is this: >> a, b = **mapping >> >> The advantages: >> - much more readable >> - less duplication > > > Why doesn't that work for tuple unpacking? > py> a, b = *iterable > SyntaxError > > Whatever the reasons, that syntax wasn't chosen for tuple unpacking. > Dict unpacking should mimic tuple unpacking. If I saw ``a, b = Good point. So it should just be: a, b = mapping or a, b, **_ = mapping # when you don't care about the rest > Unpacking a tuple mirrors a tuple display. > > py> (a, b) = (1, 2) > py> (a, b) = (1, 2, 3) > ValueError: too many values to unpack, expected 2 I think that most Pythonistas would say: a, b = 1, 2 # look ma! no round brackets! > Unpacking a dict should mirror a dict display. > > py> {'x': a, 'y': b} = {'x': 1, 'y': 2} Absolutely not, at least not for the simple case. A simple tuple/list unpack looks like a, b, c = an_iterable while a more complicated one looks like a, b, c = an_iterable[7], an_iterable[3], an_iterable[10] So a simple dict unpack should look like some_dict = dict(a=99, b=44, c=37) a, b, c = some_dict and if we don't need all the items a, b, **_ = some_dict and if we want to rename the keys on extraction x, y, z = some_dict['a'], some_dict['b'], some_dict['c'] or, as Random pointed out x, y, z = [some_dict[k] for k in ('a', 'b', 'c')] which has a nice symmetry to it. > py> {'x': a, 'y': b} = {'x': 1, 'y': 2, 'z': 3} > ValueError: too many keys to unpack, expected {'x', 'y'} > > As Brendan and others have mentioned, the more concise syntax you're > proposing will not support non-string keys and cannot be enhanced to > support Erlang/Clojure/etc-style matching on values. And as I have mentioned, matching syntax is out-of-scope for an unpacking proposal. At most, it should be a "let's not paint ourselves into a corner" type of concern -- and I must admit I don't see why a, b, c = some_dict rules out {'x': a, 'y':b} = some_dict as a pattern-matching construct. > On Wed, May 25, 2016 at 11:18 AM Paul Moore wrote: >> >> get a set of elements and *ignore* the rest: > > If it's just one or two, that's easy. Use an underscore to indicate you > don't care. Again, I'm trying to mirror a dict display. This is the same > way that tuple unpacking solves the problem. > > py> (a, b, _) = (1, 2, 3) > py> {'x': a, 'y': b, 'z': _} = {'x': 1, 'y': 2, 'z': 3} Since you've repeated yourself, I will too. ;) The parenthesis are legal, but usually unnecessary noise, when creating/unpacking a tuple. -- ~Ethan~ From python at mrabarnett.plus.com Wed May 25 19:58:55 2016 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 26 May 2016 00:58:55 +0100 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <57463845.8090709@stoneleaf.us> References: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> <57463845.8090709@stoneleaf.us> Message-ID: <97158294-6397-0b0a-aae3-f631f3133f43@mrabarnett.plus.com> On 2016-05-26 00:41, Ethan Furman wrote: > On 05/25/2016 03:52 PM, Michael Selik wrote: > >> I really like Sven's example. >> >> py> mapping = {"a": 1, "b": 2, "c": 3} >> py>{"a": x, "b": y, "c": 2} = mapping >> Traceback: >> ValueError: key 'c' does not match value 2 >> >> Even if we don't implement this feature in the first version of dict >> unpacking, we should keep the option open. > > Ugh, no. That is not dict unpacking, it's dict matching and unpacking, > which is way beyond the simplicity of just unpacking. > >> On Wed, May 25, 2016 at 4:14 PM Ethan Furman wrote: >>> >>> The proposal is this: >>> a, b = **mapping >>> >>> The advantages: >>> - much more readable >>> - less duplication >> >> >> Why doesn't that work for tuple unpacking? >> py> a, b = *iterable >> SyntaxError >> >> Whatever the reasons, that syntax wasn't chosen for tuple unpacking. >> Dict unpacking should mimic tuple unpacking. If I saw ``a, b = > > Good point. So it should just be: > > a, b = mapping > > or > a, b, **_ = mapping # when you don't care about the rest > > >> Unpacking a tuple mirrors a tuple display. >> >> py> (a, b) = (1, 2) >> py> (a, b) = (1, 2, 3) >> ValueError: too many values to unpack, expected 2 > > I think that most Pythonistas would say: > > a, b = 1, 2 # look ma! no round brackets! > >> Unpacking a dict should mirror a dict display. >> >> py> {'x': a, 'y': b} = {'x': 1, 'y': 2} > > Absolutely not, at least not for the simple case. A simple tuple/list > unpack looks like > > a, b, c = an_iterable > > while a more complicated one looks like > > a, b, c = an_iterable[7], an_iterable[3], an_iterable[10] > > So a simple dict unpack should look like > > some_dict = dict(a=99, b=44, c=37) > a, b, c = some_dict > > and if we don't need all the items > > a, b, **_ = some_dict > > and if we want to rename the keys on extraction > > x, y, z = some_dict['a'], some_dict['b'], some_dict['c'] > > or, as Random pointed out > > x, y, z = [some_dict[k] for k in ('a', 'b', 'c')] > > which has a nice symmetry to it. > Could we use 'as', which is already used for renaming in imports? a as x, b as y, c as z = some_dict or, perhaps: 'a' as x, 'b' as y, 'c' as z = some_dict which would cater for keys that aren't valid as identifiers. > >> py> {'x': a, 'y': b} = {'x': 1, 'y': 2, 'z': 3} >> ValueError: too many keys to unpack, expected {'x', 'y'} >> >> As Brendan and others have mentioned, the more concise syntax you're >> proposing will not support non-string keys and cannot be enhanced to >> support Erlang/Clojure/etc-style matching on values. > > And as I have mentioned, matching syntax is out-of-scope for an > unpacking proposal. At most, it should be a "let's not paint ourselves > into a corner" type of concern -- and I must admit I don't see why > > a, b, c = some_dict > > rules out > > {'x': a, 'y':b} = some_dict > > as a pattern-matching construct. > > >> On Wed, May 25, 2016 at 11:18 AM Paul Moore wrote: >>> >>> get a set of elements and *ignore* the rest: >> >> If it's just one or two, that's easy. Use an underscore to indicate you >> don't care. Again, I'm trying to mirror a dict display. This is the same >> way that tuple unpacking solves the problem. >> >> py> (a, b, _) = (1, 2, 3) >> py> {'x': a, 'y': b, 'z': _} = {'x': 1, 'y': 2, 'z': 3} > > Since you've repeated yourself, I will too. ;) > > The parenthesis are legal, but usually unnecessary noise, when > creating/unpacking a tuple. > From ethan at stoneleaf.us Wed May 25 20:14:27 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 May 2016 17:14:27 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <97158294-6397-0b0a-aae3-f631f3133f43@mrabarnett.plus.com> References: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> <57463845.8090709@stoneleaf.us> <97158294-6397-0b0a-aae3-f631f3133f43@mrabarnett.plus.com> Message-ID: <57463FE3.7020202@stoneleaf.us> On 05/25/2016 04:58 PM, MRAB wrote: > On 2016-05-26 00:41, Ethan Furman wrote: >> or, as Random pointed out >> >> x, y, z = [some_dict[k] for k in ('a', 'b', 'c')] >> >> which has a nice symmetry to it. > > Could we use 'as', which is already used for renaming in imports? > > a as x, b as y, c as z = some_dict I'm okay with that. > or, perhaps: > > 'a' as x, 'b' as y, 'c' as z = some_dict > > which would cater for keys that aren't valid as identifiers. That probably makes more sense, especially since it's already the rare(r) case of needing/wanting to rename the keys. -- ~Ethan~ From michael.selik at gmail.com Wed May 25 20:29:09 2016 From: michael.selik at gmail.com (Michael Selik) Date: Thu, 26 May 2016 00:29:09 +0000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <57463845.8090709@stoneleaf.us> References: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> <57463845.8090709@stoneleaf.us> Message-ID: On Wed, May 25, 2016 at 7:41 PM Ethan Furman wrote: > a, b = mapping > a, b, **_ = mapping # when you don't care about the rest > I'd grudgingly accept this as a second-best syntax. It doesn't support keys that aren't strs of valid identifiers, but I suppose ``def foo(**kwargs)`` doesn't either. I frequently have keys that are strs with punctuation, keys that are tuples of ints, etc. Being restricted to identifiers feels cramped. Also, I wonder if the implementation will be more difficult. In tuple unpacking, it's the LHS that provides the semantic meaning. The RHS is just duck-typed. What's the appropriate error message if you meant to do dict unpacking and didn't have a mapping on the RHS? py> a, b = 42 TypeError: 'int' object is not iterable py> {'a': x, 'b': y} = 42 TypeError: 'int' object is not subscriptable I think that most Pythonistas would say: > a, b = 1, 2 # look ma! no round brackets! > You need brackets for nested/recursive destructuring. py> a, b, (c, d) = 1, 2, (3, 4) py> {'a': x, 'b': {'c': y, 'd': z} = {'a': 1, 'b': {'c': 2, 'd': 3}} matching syntax is out-of-scope for an > unpacking proposal. At most, it should be a "let's not paint ourselves > into a corner" type of concern -- and I must admit I don't see why > a, b, c = some_dict > rules out > {'x': a, 'y':b} = some_dict > as a pattern-matching construct. > Yep. I just want to look ahead a bit. While you're considering dict unpacking syntax options, keep in mind that tuple matching will need to parallel dict matching. Also, our future selves will want one-way-to-do-it when considering matching syntaxes. py> a, b, c, 0 = 1, 2, 3, 4 ValueError: index 3 does not match value 0 py> {'x': a, 'y': 0} = {'x': 1, 'y': 2} ValueError: key 'y' does not match value 0 > On Wed, May 25, 2016 at 11:18 AM Paul Moore wrote: > >> get a set of elements and *ignore* the rest: > > Since you've repeated yourself, I will too. ;) > I figured repetition was more clear than "see above". :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.selik at gmail.com Wed May 25 20:37:58 2016 From: michael.selik at gmail.com (Michael Selik) Date: Thu, 26 May 2016 00:37:58 +0000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <57463845.8090709@stoneleaf.us> References: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> <57463845.8090709@stoneleaf.us> Message-ID: On Wed, May 25, 2016 at 7:41 PM Ethan Furman wrote: > it should just be: > a, b = mapping > Breaks backwards compatibility. py> a, b = {'a': 1, 'b': 2} py> a, b ('b', 'a') Sorry for the double-post. I should have pondered longer before sending... -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed May 25 21:56:07 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 26 May 2016 11:56:07 +1000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: Message-ID: <20160526015606.GB12028@ando.pearwood.info> On Wed, May 25, 2016 at 01:11:35PM +0000, Michael Selik wrote: > Clojure also supports mapping destructuring. Let's add that to Python! > > py> mapping = {"a": 1, "b": 2, "c": 3} > py> {"a": x, "b": y, "c": z} = mapping > py> x, y, z > (1, 2, 3) > py> {"a": x, "b": y} = mapping > Traceback: > ValueError: too many keys to unpack > > > This will be approximately as helpful as iterable unpacking was before PEP > 3132 (https://www.python.org/dev/peps/pep-3132/). What is your evidence for this claim? So far I've only seen one real- world use-case for this, and that single use-case would be well served by a simpler syntax: a, b, c = **mapping which just requires that a, b, c etc are legal names (not general identifiers). The dict is then unpacked: a = mapping['a'] etc. Two questions: (1) For the use-case we've already seen, a "preferences" or "settings" mapping, do you think users will be prepared to use your syntax? (2) Do you have any other concrete use cases for this, and if so, what are they? For (1), I've taken a prefs dict from a rather small command line script I've written. This is taken from actual code in use. To unpack the dict using your syntax, I would have to write: {'sort': sort, 'reverse': reverse, 'classify': classify, 'showlinks': showlinks, 'showsize': showsize, 'spacer': spacer, 'style': style, 'width': width} = prefs # next line is optional, but I might not wish to pollute the namespace del sort, reverse, showlinks, showsize, style in order to unpack the three keys I actually want. Here's my suggestion: classify, spacer, width = **prefs I don't know about you, but there's no way I'd use your suggested syntax as-shown. I'd rather unpack manually: classify, spacer, width = [prefs[key] for key in 'classify spacer width'.split()] or variations of same. For (2), here's another use-case I can think of. Unpacking **kwargs in functions/methods. def spam(self, a, b, **kwargs): ... I'd like to unpack a small number of keys:values from kwargs, extract them from the dict, and pass that on to another method: fnord = kwargs.pop('fnord', 'default') wibble = kwargs.pop('wibble', 42) super().spam(a, b, **kwargs) I don't have a concise syntax for this use-case, and yours won't work either. There's no way to supply defaults, nor can you list *all* the keys because you don't know what they will be. (The caller can provide arbitrary keyword arguments.) So I don't think this use-case can be handled by either your syntax or mine for dict unpacking. I think your syntax is too verbose and repetitive for the simple case. It does have the advantage that it can deal with keys which aren't identifiers: {'while': while_, 'foo bar': foobar} = mapping but it only looks good in toy examples. In real code, I wouldn't use it, it would be too painful and repetitive. If we add syntax to collect all the unused items, your syntax will be a bit less painful, but still repetitive: {'classify': classify, 'spacer': spacer, 'width': width, **whocares} = prefs but that has no advantage over what we already have: classify, spacer, width = [prefs[key] for key in ('classify', 'spacer', 'width')] (The two are almost the same length, and equally repetitive.) As far as changing names, we can already do that, and use arbitrary references: myobj.attr['key'][1], while_, foobar = [ mapping[key] for key in ('something', 'while', 'foo bar')] So I think that the problems your syntax solve are already easy to solve, and the things which are annoying to solve now, your syntax is too painful to use. I'd rather have syntax which is less general but more useful in practice, than something which solves dict unpacking in its full generality but a pain to use. -- Steve From ian at feete.org Wed May 25 10:06:22 2016 From: ian at feete.org (Ian Foote) Date: Wed, 25 May 2016 15:06:22 +0100 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: Message-ID: <5745B15E.5020703@feete.org> I can see this being useful if I have a Django Rest Framework validate method (http://www.django-rest-framework.org/api-guide/serializers/#object-level-validation) which takes a dictionary argument. def validate(self, data): {'name': name, 'address': address} = data Currently, this would look like: def validate(self, data): name, address = data['name'], data['address'] It does get more useful with the extension: def validate(self, data): {'name': name, 'address': address, **rest} = data instead of: def validate(self, data): rest = data.copy() name = rest.pop('name') address = rest.pop('address') In the rest framework case, mutating data directly might not be a problem, but this does feel like a nice syntax when avoiding mutation is required. Regards, Ian On 25/05/16 14:11, Michael Selik wrote: > Python's iterable unpacking is what Lispers might call a destructuring > bind. > > py> iterable = 1, 2, 3, 4, 5 > py> a, b, *rest = iterable > py> a, b, rest > (1, 2, (3, 4, 5)) > > Clojure also supports mapping destructuring. Let's add that to Python! > > py> mapping = {"a": 1, "b": 2, "c": 3} > py> {"a": x, "b": y, "c": z} = mapping > py> x, y, z > (1, 2, 3) > py> {"a": x, "b": y} = mapping > Traceback: > ValueError: too many keys to unpack > > > This will be approximately as helpful as iterable unpacking was before > PEP 3132 (https://www.python.org/dev/peps/pep-3132/). > > I hope to keep discussion in this thread focused on the most basic > form of dict unpacking, but we could extended mapping unpacking > similarly to how PEP 3132 extended iterable unpacking. Just > brainstorming... > > py> mapping = {"a": 1, "b": 2, "c": 3} > py> {"a": x, **rest} = mapping > py> x, rest > (1, {"b": 2, "c": 3}) > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Wed May 25 22:16:59 2016 From: mertz at gnosis.cx (David Mertz) Date: Wed, 25 May 2016 19:16:59 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <20160526015606.GB12028@ando.pearwood.info> References: <20160526015606.GB12028@ando.pearwood.info> Message-ID: On Wed, May 25, 2016 at 6:56 PM, Steven D'Aprano wrote: > What is your evidence for this claim? So far I've only seen one real- > world use-case for this, and that single use-case would be well served > by a simpler syntax: > > a, b, c = **mapping > > which just requires that a, b, c etc are legal names (not general > identifiers). The dict is then unpacked: > > a = mapping['a'] > I can see how this spelling might be intuitive at first brush. But the more I think about it, the more I recoil against the violation of a relatively uniform semantic principle in Python. In no other case in Python, does the RHS of an assignment "probe into" the LHS to figure out how to determine its value. Moreover, the idea that variable names are not just bindings, but also pseudo-literals, or maybe something akin to a Lisp 'symbol', feels enormously unpythonic to me. Moreover, given that comprehensions are already available, and can express every variation we might want simply, I see no point of having this mild syntax sugar. This includes binding in the usual style to arbitrary names, but also all the expected mechanisms of derived values and conditionals. You can write: a, b, c = (mapping[x] for x in ['a','b','c']) But equally you can write a natural extension like: x, y, z = (2*mapping[x] for x in get_keys() if x.isupper()) Special casing the very simplest thing to save a minimal number of characters does not seem worthwhile. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed May 25 22:18:41 2016 From: guido at python.org (Guido van Rossum) Date: Wed, 25 May 2016 19:18:41 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <20160526015606.GB12028@ando.pearwood.info> References: <20160526015606.GB12028@ando.pearwood.info> Message-ID: On Wed, May 25, 2016 at 6:56 PM, Steven D'Aprano wrote: > On Wed, May 25, 2016 at 01:11:35PM +0000, Michael Selik wrote: > What is your evidence for this claim? So far I've only seen one real- > world use-case for this, and that single use-case would be well served > by a simpler syntax: > > a, b, c = **mapping I have to warn here. This looks cool but it does something that AFAIK no other Python syntax uses -- it takes variable names and does something to those variables but *also* uses their actual names as string literals. I agree that the use cases for this seem pretty sweet, but perhaps we should give it a somewhat different syntax just so it's clear that the names on the LHS matter. The precedent that the targets must be actual names rather than anything you can assign to is also kind of scary. -- --Guido van Rossum (python.org/~guido) From stephen at xemacs.org Wed May 25 22:32:21 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 26 May 2016 11:32:21 +0900 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: References: <20160522084556.GR12028@ando.pearwood.info> <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> <20160524154506.GV12028@ando.pearwood.info> <20160524161913.GX12028@ando.pearwood.info> <57453AF9.8050602@canterbury.ac.nz> <0239b3c2-f795-6418-5b27-f61673c8f643@btinternet.com> Message-ID: <22342.24629.646814.622509@turnbull.sk.tsukuba.ac.jp> Rob Cliffe writes: > What I was trying to say was: I think of > > ValueError as applying to a single ("bad") value > > whereas ArithmeticError can arise from trying to combine multiple > ("good") values. I thought about that, and concluded that (1) a function taking multiple arguments is semantically equivalent to a function taking a single tuple as argument, and (2) offhand, I don't know of any errors of computation that require knowledge of more than one argument to recognize (and it's always the same argument). I realize that there are alternative ways of interpreting those facts (eg, in the implementation you will take the tuple apart and operate on the components as "multiple entities"). However, I found myself looping between those two, one or the other covered everything that wasn't a TypeError in the first place. > It wouldn't be appropriate for ArithmeticError to be a subclass of > ValueError, because there is no single erroneous value that you can > point to. Well, I *can* point to the *tuple*. You need to argue why I shouldn't, in view of the fact that the subclassing is correct (not only is the tuple as value invalid, the subclassing tells me that it's not just one of the components, but an interaction of more than one). Examples would help. ZeroDivisionError tells you that it's exactly one of the arguments by its name. Feeding a complex as either argument to atan2 gives a TypeError. Feeding a zero as either argument to atan2 gives a (correct!) answer (atan2(x,y) is not the same as atan(y/x), which raises ZeroDivisionError before atan is actually called). Feeding two zeros to atan2 gives an answer (but I'm way too far from 9th grade to remember if that's correct -- perhaps that "should" raise an ArithmeticError on your interpretation). Steve From steve at pearwood.info Wed May 25 22:32:29 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 26 May 2016 12:32:29 +1000 Subject: [Python-ideas] Should decimal.InvalidOperation subclass ValueError? In-Reply-To: References: <20160522133150.GS12028@ando.pearwood.info> <5742340F.7080007@canterbury.ac.nz> <20160524154506.GV12028@ando.pearwood.info> <20160524161913.GX12028@ando.pearwood.info> Message-ID: <20160526023227.GC12028@ando.pearwood.info> On Tue, May 24, 2016 at 07:04:46PM -0700, Guido van Rossum wrote: > Is there really something here that needs to be fixed? I'm not sure. As the OP, I was hoping that there would be a simple and obvious "yes, we can change this". But it's not that simple. It seems to me that the annoyance of the current inconsistent and idiosyncratic exceptions is not obviously greater than the pain of "fixing" the problem. Even if we agreed on what the fix should be. So I am prepared to let this go as a "won't fix": - people who try to guess what exceptions are raised are on shaky ground, especially beginners, and should read the docs or try things out in the REPL; - the current exception hierarchy may, or may not, be illogical, but it would be more painful to change it than live with it. -- Steve From stephen at xemacs.org Wed May 25 22:34:18 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 26 May 2016 11:34:18 +0900 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <877fehuamr.fsf@thinkpad.rath.org> References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <877fehuamr.fsf@thinkpad.rath.org> Message-ID: <22342.24746.141240.623012@turnbull.sk.tsukuba.ac.jp> Nikolaus Rath writes: > Or maybe: > > if ... > matches : > pass > matches : > pass > > Where "..." is an actual syntactic element that replaces the ":" in a > regular if. Interesting, but I don't like it because (1) ":" is *the* compound syntax indicator in Python, (2) "..." looks like somebody forgot to fill in a blank, *and it should* look that way, (3) "matches" is too close to "match" (which is a stdlib identifier), and it's an identifier I have used, and (4) it's an additional keyword. Leaving aside the spelling quibbles, following "if" I would "see" "matches" as an operator, and want to write if matches : pass elif matches : pass and I'm back in "if ... elif ... else ..." land. I don't really see much advantage to your particular spelling, as to me it doesn't have the connotation that Paul requires (correctly, IMHO) of a single "subject" tested against multiple "predicates". It just "looks wrong" to me. IMHO, YMMV, etc. It *is* interesting, I am trying to contribute to discussion, not end it. My mind could be updated. :-) From ethan at stoneleaf.us Wed May 25 22:43:22 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 May 2016 19:43:22 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> <57463845.8090709@stoneleaf.us> Message-ID: <574662CA.3060704@stoneleaf.us> [Including replies to On 05/25/2016 05:29 PM, Michael Selik wrote: > On Wed, May 25, 2016 at 7:41 PM Ethan Furman wrote: >> a, b = mapping >> a, b, **_ = mapping # when you don't care about the rest > > I'd grudgingly accept this as a second-best syntax. It doesn't support > keys that aren't strs of valid identifiers, but I suppose ``def > foo(**kwargs)`` doesn't either. I frequently have keys that are strs > with punctuation, keys that are tuples of ints, etc. Being restricted to > identifiers feels cramped. Doesn't have to be the only syntax, just the easiest. [from other email] ------------------ > Breaks backwards compatibility. > > py> a, b = {'a': 1, 'b': 2} > py> a, b > ('b', 'a') Ah. That is a very good point. I'll meet you halfway: {a, b} = some_dict which is currently a SyntaxError, so little backwards compatibility concerns, plus it clearly state the mapping should have two elements, similarly to [a] = some_iterable and (b) = some_iterable both clearly state that a one-element iterable is being unpacked. >> I think that most Pythonistas would say: >> a, b = 1, 2 # look ma! no round brackets! > > You need brackets for nested/recursive destructuring. > > py> a, b, (c, d) = 1, 2, (3, 4) Sure, but that isn't the simple case. > py> {'a': x, 'b': {'c': y, 'd': z} = {'a': 1, 'b': {'c': 2, 'd': 3}} With the above syntax: {a, b {c, d}} = a_mapping > Yep. I just want to look ahead a bit. While you're considering dict > unpacking syntax options, keep in mind that tuple matching will need to > parallel dict matching. Also, our future selves will want > one-way-to-do-it when considering matching syntaxes. > > py> a, b, c, 0 = 1, 2, 3, 4 > ValueError: index 3 does not match value 0 > py> {'x': a, 'y': 0} = {'x': 1, 'y': 2} > ValueError: key 'y' does not match value 0 The adage is actually one /obvious/ way to do it -- which can change depending on the circumstances. On 05/25/2016 07:16 PM, David Mertz wrote: > I can see how this spelling might be intuitive at first brush. But > the more I think about it, the more I recoil against the violation of > a relatively uniform semantic principle in Python. > > In no other case in Python, does the RHS of an assignment "probe > into" the LHS to figure out how to determine its value. Moreover, > the idea that variable names are not just bindings, but also pseudo- > literals, or maybe something akin to a Lisp 'symbol', feels enormously > unpythonic to me. On 05/25/2016 07:18 PM, Guido van Rossum wrote: > I have to warn here. This looks cool but it does something that AFAIK > no other Python syntax uses -- it takes variable names and does > something to those variables but *also* uses their actual names as > string literals. I agree that the use cases for this seem pretty > sweet, but perhaps we should give it a somewhat different syntax just > so it's clear that the names on the LHS matter. The precedent that the > targets must be actual names rather than anything you can assign to is > also kind of scary. Very good points. I'm glad we had the discussion, though, since it elicited Random's contribution -- which is what I will use from now on for dict unpacking. :) -- ~Ethan~ From rosuav at gmail.com Wed May 25 22:56:12 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 26 May 2016 12:56:12 +1000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <574662CA.3060704@stoneleaf.us> References: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> <57463845.8090709@stoneleaf.us> <574662CA.3060704@stoneleaf.us> Message-ID: On Thu, May 26, 2016 at 12:43 PM, Ethan Furman wrote: > I'll meet you halfway: > > {a, b} = some_dict > > which is currently a SyntaxError, so little backwards compatibility > concerns, plus it clearly state the mapping should have two elements, > similarly to > > [a] = some_iterable > > and > > (b) = some_iterable > > both clearly state that a one-element iterable is being unpacked. > Careful - the second one doesn't: >>> (b) = [1, 2, 3] >>> Parens don't make a tuple, and that includes with unpacking. You'd be correct if you had a comma in there, though. I hope there doesn't end up being a confusion between mapping unpacking and set display. Sets are a bit of an odd duck; are they like lists only unordered, or like mappings only without values? I've seen them used both ways, and the syntax is somewhere between the two. Having a syntax that constructs a set if used on the RHS but unpacks a dict if used on the LHS seems to violate syntactic purity, but I'd be happy to let practicality trump that. ChrisA From rosuav at gmail.com Wed May 25 22:59:07 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 26 May 2016 12:59:07 +1000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <22342.24746.141240.623012@turnbull.sk.tsukuba.ac.jp> References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <877fehuamr.fsf@thinkpad.rath.org> <22342.24746.141240.623012@turnbull.sk.tsukuba.ac.jp> Message-ID: On Thu, May 26, 2016 at 12:34 PM, Stephen J. Turnbull wrote: > Nikolaus Rath writes: > > > Or maybe: > > > > if ... > > matches : > > pass > > matches : > > pass > > > > Where "..." is an actual syntactic element that replaces the ":" in a > > regular if. > > Interesting, but I don't like it because (1) ":" is *the* compound > syntax indicator in Python, (2) "..." looks like somebody forgot to > fill in a blank, *and it should* look that way, (3) "matches" is too > close to "match" (which is a stdlib identifier), and it's an > identifier I have used, and (4) it's an additional keyword. > > Leaving aside the spelling quibbles, following "if" I would "see" > "matches" as an operator, and want to write > > if matches : > pass > elif matches : > pass > > and I'm back in "if ... elif ... else ..." land. What if it were like this: if ...: matches : pass matches : pass with a colon at the end of the first line too? (And 'matches' would still need to be bikeshedded into oblivion.) I'm not sold on it, but it would read reasonably well, and it still maintains the "colon then indented block" pattern of most Python code. ChrisA From ethan at stoneleaf.us Wed May 25 23:46:55 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 May 2016 20:46:55 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> <57463845.8090709@stoneleaf.us> <574662CA.3060704@stoneleaf.us> Message-ID: <574671AF.8040707@stoneleaf.us> On 05/25/2016 07:56 PM, Chris Angelico wrote: > On Thu, May 26, 2016 at 12:43 PM, Ethan Furman wrote: >> I'll meet you halfway: >> >> {a, b} = some_dict >> >> which is currently a SyntaxError, so little backwards compatibility >> concerns, plus it clearly state the mapping should have two elements, >> similarly to >> >> [a] = some_iterable >> >> and >> >> (b) = some_iterable >> >> both clearly state that a one-element iterable is being unpacked. >> > > Careful - the second one doesn't: > >--> (b) = [1, 2, 3] >--> Ack. Right you are, which is why I always use the list form, even on tuples. > I hope there doesn't end up being a confusion between mapping > unpacking and set display. Sets are a bit of an odd duck; are they > like lists only unordered, or like mappings only without values? I've > seen them used both ways, and the syntax is somewhere between the two. > Having a syntax that constructs a set if used on the RHS but unpacks a > dict if used on the LHS seems to violate syntactic purity, but I'd be > happy to let practicality trump that. Given the other issues, I'm happy to let this one die. Good discussion. -- ~Ethan~ From ethan at stoneleaf.us Wed May 25 23:50:13 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 May 2016 20:50:13 -0700 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <877fehuamr.fsf@thinkpad.rath.org> <22342.24746.141240.623012@turnbull.sk.tsukuba.ac.jp> Message-ID: <57467275.3010009@stoneleaf.us> On 05/25/2016 07:59 PM, Chris Angelico wrote: > On Thu, May 26, 2016 at 12:34 PM, Stephen J. Turnbull wrote: >> Leaving aside the spelling quibbles, following "if" I would "see" >> "matches" as an operator, and want to write >> >> if matches : >> pass >> elif matches : >> pass >> >> and I'm back in "if ... elif ... else ..." land. > > What if it were like this: > > if ...: > matches : > pass > matches : > pass > > with a colon at the end of the first line too? I would just as soon use the switch/case keywords, with extra matching oomph. It doesn't have to match C(-like) exactly, and would be immediately recognizable as to the intent (following tests are against the thing in the switch portion). I think it's okay to put our own Python spin on it. -- ~Ethan~ From leewangzhong+python at gmail.com Thu May 26 00:36:32 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Thu, 26 May 2016 00:36:32 -0400 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <574671AF.8040707@stoneleaf.us> References: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> <57463845.8090709@stoneleaf.us> <574662CA.3060704@stoneleaf.us> <574671AF.8040707@stoneleaf.us> Message-ID: Then is it too late to point out that the desired goal (of dumping part of a dict into the local namespace) is really similar to importing? Something I am most definitely not suggesting: from some_dict dict_import a, b, c Related, a version that works for Enums: from some_Enum attr_import * from vars(some_enum) dict_import * Again, I'm not suggesting these syntaxes (and I hope everyone else hates them), just pointing out that it's more similar to imports than destructuring. On Wed, May 25, 2016 at 11:46 PM, Ethan Furman wrote: > On 05/25/2016 07:56 PM, Chris Angelico wrote: >> >> On Thu, May 26, 2016 at 12:43 PM, Ethan Furman wrote: > > >>> I'll meet you halfway: >>> >>> {a, b} = some_dict >>> >>> which is currently a SyntaxError, so little backwards compatibility >>> concerns, plus it clearly state the mapping should have two elements, >>> similarly to >>> >>> [a] = some_iterable >>> >>> and >>> >>> (b) = some_iterable >>> >>> both clearly state that a one-element iterable is being unpacked. >>> >> >> Careful - the second one doesn't: >> >> --> (b) = [1, 2, 3] >> --> > > > Ack. Right you are, which is why I always use the list form, even on > tuples. > >> I hope there doesn't end up being a confusion between mapping >> unpacking and set display. Sets are a bit of an odd duck; are they >> like lists only unordered, or like mappings only without values? I've >> seen them used both ways, and the syntax is somewhere between the two. >> Having a syntax that constructs a set if used on the RHS but unpacks a >> dict if used on the LHS seems to violate syntactic purity, but I'd be >> happy to let practicality trump that. > > > Given the other issues, I'm happy to let this one die. Good discussion. > > -- > ~Ethan~ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From tritium-list at sdamon.com Thu May 26 00:39:02 2016 From: tritium-list at sdamon.com (tritium-list at sdamon.com) Date: Thu, 26 May 2016 00:39:02 -0400 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <5745F33A.1050907@stoneleaf.us> References: <20160525184206.GZ12028@ando.pearwood.info> <5745F33A.1050907@stoneleaf.us> Message-ID: <05eb01d1b708$876eec30$964cc490$@hotmail.com> I am -1 on the whole idea. What is either asking for the identifier being assigned to having semantic meaning in the language, something we do not have anywhere else. (yes, we have special names, but the language does not actually care about what object you assign to the special name, and the name itself does not change the behavior of the assignment.) OR is totally redundant to something we can already do: a, b, c, *r = mapping.values() -----Original Message----- From: Python-ideas [mailto:python-ideas-bounces+tritium-list=sdamon.com at python.org] On Behalf Of Ethan Furman Sent: Wednesday, May 25, 2016 2:47 PM To: python-ideas at python.org Subject: Re: [Python-ideas] Unpacking a dict On 05/25/2016 11:42 AM, Steven D'Aprano wrote: > On Wed, May 25, 2016 at 01:11:35PM +0000, Michael Selik wrote: >> py> mapping = {"a": 1, "b": 2, "c": 3} >> py> {"a": x, "b": y, "c": z} = mapping >> py> x, y, z >> (1, 2, 3) > > I think that is too verbose and visually baffling. I'd rather see > something less general and (in my opinion) more useful: > > a, b, c = **mapping > > being equivalent to: > > a = mapping['a'] > b = mapping['b'] > c = mapping['c'] +1 Simplest, easiest to grok, probably solves 95+% of the use-cases. -- ~Ethan~ _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ From michael.selik at gmail.com Thu May 26 00:47:47 2016 From: michael.selik at gmail.com (Michael Selik) Date: Thu, 26 May 2016 04:47:47 +0000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <05eb01d1b708$876eec30$964cc490$@hotmail.com> References: <20160525184206.GZ12028@ando.pearwood.info> <5745F33A.1050907@stoneleaf.us> <05eb01d1b708$876eec30$964cc490$@hotmail.com> Message-ID: On Thu, May 26, 2016 at 12:39 AM wrote: > I am -1 on the whole idea. What is either asking for the identifier being > assigned to having semantic meaning in the language, something we do > not have anywhere else. No? Tuple unpacking has semantic meaning for the left-hand side. Or did I misunderstand you? > a, b, c, *r = mapping.values() > Unless it's an OrderedDict *and* I know the order, I wouldn't want to do that. -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.selik at gmail.com Thu May 26 01:09:02 2016 From: michael.selik at gmail.com (Michael Selik) Date: Thu, 26 May 2016 05:09:02 +0000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <20160526015606.GB12028@ando.pearwood.info> References: <20160526015606.GB12028@ando.pearwood.info> Message-ID: On Wed, May 25, 2016 at 9:56 PM Steven D'Aprano wrote: > > This will be approximately as helpful as iterable unpacking > > What is your evidence for this claim? [For extracting from a large dictionary] I'd rather unpack manually: > classify, spacer, width = (prefs[k] for k in ('classify', 'spacer', > 'width')) > I agree that comprehensions plus tuple unpacking handle many possible use cases for dict unpacking. There are many marginally-better situations that would just drum up endless back-and-forth about aesthetics. So let's look for a situation where dict unpacking handles well what current syntax struggles with. An example of schema validation plus binding in the current syntax: ``` py> mapping = {'a': 1, 'b': 2} py> schema = ('a', 'b') py> unexpected = mapping.keys() - set(schema) py> if unexpected: ... raise ValueError('unexpected keys %r' % unexpected) ... py> x, y = (mapping[key] for key in schema) ``` With sets and comprehensions, that's very nice. More pleasant than most if not all other mainstream languages. Yet dict unpacking can be just a bit better -- more declarative, more say-what-you-mean, less thinking about algorithms. Fewer steps, too, but that's not very important. The proposed syntax, examples of missing and excess keys. ``` py> mapping = {'a': 1, 'b': 2, 'c': 3} py> {'a': x, 'b': y, 'c': z, 'd': s, 'e': t} = mapping ValueError: missing 2 required keys 'd' and 'e' py> {'a': x, 'b': y} = mapping ValueError: got an unexpected key 'c' ``` If you tell me that's not enough better to justify changing the language, I won't argue very much. I agree that example, if that's the only use case, would need to be backed up by extensive review of code of major projects to see how much improvement it'd really provide. Unpacking really starts to shine when doing nested/recursive destructuring. My proposed syntax: ``` py> d = {'a': 1, ... 'b': {'c': 2, 'd': 3}} py> {'a': x, 'b': {'c': y, 'd': y}} = d py> x, y, z 1, 2, 3 ``` In current syntax, even simply specifying the schema is troublesome if the order of keys is to be preserved for binding to the desired names. ``` >>> mapping = {'a': 1, 'b': {'c': 2, 'd': 3}} >>> schema = OrderedDict([('a', None), ... ('b', OrderedDict([('c', None), ('d', None)]))]) >>> x, y, z = ... ``` I tried writing out a couple comprehensions for the binding, but they were *ugly*. The traversal of nested mapping for validation, flattening and assignment needs a recursive function call or you'll end up with a disaster of nested loops. And lastly, if you have your eye on the prize (pattern matching) then establishing a full-featured dict unpacking is a big step in the right direction. I may not have stated it when I started this thread, but the initial motivation for dict unpacking was the trouble we were having in our discussion of pattern matching. I wanted to break that big problem apart into smaller problems. A possible (not proposed) syntax for dict and tuple pattern matching: ``` py> {'a': x, 'b': 0} = {'a': 1, 'b': 2} ValueError: key 'b' does not match value 0 py> (a, 0) = (1, 2) ValueError: index 1 does not match value 0 ``` def spam(self, a, b, **kwargs): > ... > > I'd like to unpack a small number of keys:values from kwargs, extract > them from the dict, and pass that on to another method: > > fnord = kwargs.pop('fnord', 'default') > wibble = kwargs.pop('wibble', 42) > super().spam(a, b, **kwargs) > > > I don't have a concise syntax for this use-case, and yours won't work > either. There's no way to supply defaults, nor can you list *all* the > keys because you don't know what they will be. (The caller can provide > arbitrary keyword arguments.) > You're right, I haven't thought about defaults. Hmm. Tuple unpacking doesn't support defaults either, so I guess I'll let this one go as not appropriate for dict unpacking. I think your syntax is too verbose and repetitive for the simple case. > It's no more repetitive than str.format with keyword arguments :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu May 26 01:41:00 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 26 May 2016 17:41:00 +1200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <57456DBF.5060900@stoneleaf.us> References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <57456DBF.5060900@stoneleaf.us> Message-ID: <57468C6C.7060409@canterbury.ac.nz> Ethan Furman wrote: > `case blah blah as blah blah reads much better to me, and clearly says > "the stuff between 'case' and 'as' will now be known as the stuff > between 'as' and ':'". But that separates the things being bound from the names they're being bound to, so that you have to match them up positionally. That hurts readability for me. -- Greg From tjreedy at udel.edu Thu May 26 01:48:05 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 26 May 2016 01:48:05 -0400 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> Message-ID: On 5/26/2016 1:09 AM, Michael Selik wrote: > I agree that comprehensions plus tuple unpacking handle many possible > use cases for dict unpacking. There is another dict method that has been ignored in this discussion. >>> mapping = {'a': 1, 'b': 2, 'c': 3} >>> x = mapping.pop('a') >>> mapping {'c': 3, 'b': 2} We have both subscripting and popping to handle getting a value and either leaving or removing the pair. -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Thu May 26 02:09:43 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 26 May 2016 18:09:43 +1200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> Message-ID: <57469327.4070600@canterbury.ac.nz> Guido van Rossum wrote: > it does something that AFAIK > no other Python syntax uses -- it takes variable names and does > something to those variables but *also* uses their actual names as > string literals. The names in def and class statements also end up in the __name__ attributes of the created objects -- does that count? -- Greg From greg.ewing at canterbury.ac.nz Thu May 26 02:24:40 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 26 May 2016 18:24:40 +1200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> <57463845.8090709@stoneleaf.us> <574662CA.3060704@stoneleaf.us> Message-ID: <574696A8.3070403@canterbury.ac.nz> Chris Angelico wrote: > I hope there doesn't end up being a confusion between mapping > unpacking and set display. Sets are a bit of an odd duck; are they > like lists only unordered, or like mappings only without values? Let's just hope nobody says they want set unpacking... -- Greg From k7hoven at gmail.com Thu May 26 07:45:07 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 26 May 2016 14:45:07 +0300 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> Message-ID: On Thu, May 26, 2016 at 1:52 AM, Michael Selik wrote: [...] > On Wed, May 25, 2016 at 4:14 PM Ethan Furman wrote: >> >> The proposal is this: >> a, b = **mapping >> >> The advantages: >> - much more readable >> - less duplication > > > Why doesn't that work for tuple unpacking? > py> a, b = *iterable > SyntaxError > > Whatever the reasons, that syntax wasn't chosen for tuple unpacking. Dict > unpacking should mimic tuple unpacking. If I saw ``a, b = **mapping`` I > would expect ``a, b = *iterable``. > I think it would make sense to allow a, b = *iterable That would be more explicit about unpacking the iterable. Still, in clear cases like `a, b = 1, 2`, one could omit the asterisk. After all, this already works: a, b, c = 1, *(2, 3) Some more examples: a, b = 1, 2 # this is clear a, b = *(1,2) # could be legal and equivalent to the above a = *(1, 2) # would fail, and should fail! a = 1, 2 # does not fail So why not allow being more explicit about unpacking? -- Koos From k7hoven at gmail.com Thu May 26 08:27:55 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 26 May 2016 15:27:55 +0300 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> Message-ID: On Thu, May 26, 2016 at 5:18 AM, Guido van Rossum wrote: > On Wed, May 25, 2016 at 6:56 PM, Steven D'Aprano wrote: >> On Wed, May 25, 2016 at 01:11:35PM +0000, Michael Selik wrote: >> What is your evidence for this claim? So far I've only seen one real- >> world use-case for this, and that single use-case would be well served >> by a simpler syntax: >> >> a, b, c = **mapping > > I have to warn here. This looks cool but it does something that AFAIK > no other Python syntax uses -- it takes variable names and does > something to those variables but *also* uses their actual names as > string literals. I agree that the use cases for this seem pretty > sweet, but perhaps we should give it a somewhat different syntax just > so it's clear that the names on the LHS matter. The precedent that the > targets must be actual names rather than anything you can assign to is > also kind of scary. I understand the concern, and maybe you are right. However ?, this:? ? ? def func(y, z, x) ? ? print(x, y, z) ? ? func(**dict(x=1, y=2, z=3)) prints "1 2 3" ?, a ?nd so doe ?s? ? func(x=1, y=2, z=3)? and ? func(z=3, x=1, y=2) ?So `a, b, c = **mapping` would be perfectly in line with this. Of course there may still be confusion, but that would mean the user would probably already be confused about whether dicts are ordered or not, so that confusion would need to be fixed anyway. I think the key is that ** should _never_ be interpreted as unpack/repack by order. Or in other words, it always means unpack/repack _by name_. That said, here's a couple of suggestions: **(a, b, c) = **mapping **{a, b, c} = **mapping Although `a, b, c = **mapping` would still be more convenient. ?-- Koos PS. For even more explicitness: a from 'a', b from 'b', c from 'c' = **mapping Which would allow even b from 1, a from 0, x from 2 = **iterable Or a, b, c from 'a', 'b', 'c' in mapping b, a, c from 1, 0, 2 in mapping > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Thu May 26 10:10:10 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Thu, 26 May 2016 16:10:10 +0200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> Message-ID: <574703C2.7070409@gmail.com> We could provide several options though. a, b, c = **mapping a, b, **c = **mapping For the simple case. AND {"1": a, "foo.bar": b, **c} = **mapping For more complex cases. We already have suitabilities with regular unpacking such as: a, (b, c) = 1, range(2) The thing with those details is that you can completly ignore them, and don't know they exist, and simply look it up when you need it. But I must say: {"1": a, "foo.bar": b, **c} = **mapping Looks very ugly. Le 26/05/2016 07:48, Terry Reedy a ?crit : > On 5/26/2016 1:09 AM, Michael Selik wrote: >> I agree that comprehensions plus tuple unpacking handle many possible >> use cases for dict unpacking. > > There is another dict method that has been ignored in this discussion. > >>>> mapping = {'a': 1, 'b': 2, 'c': 3} >>>> x = mapping.pop('a') >>>> mapping > {'c': 3, 'b': 2} > > We have both subscripting and popping to handle getting a value and > either leaving or removing the pair. > From python+python_ideas at discontinuity.net Thu May 26 11:25:10 2016 From: python+python_ideas at discontinuity.net (Davin Potts) Date: Thu, 26 May 2016 10:25:10 -0500 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> Message-ID: > So `a, b, c = **mapping` would be perfectly in line with this. Your `func` example is a great connection to have made but I would not reach the same conclusion. When calling `func(**some_dict)` we are not performing variable assignment but with the `a, b, c = **mapping` we would be. Whereas I accept that `func(**some_dict)` imposes some constraints on the nature of the keys in the dict because functions must have proper variable names as input parameters, I find it difficult to accept a similar constraint on general dict unpacking. Davin On Thu, May 26, 2016 at 7:27 AM, Koos Zevenhoven wrote: > On Thu, May 26, 2016 at 5:18 AM, Guido van Rossum > wrote: > > On Wed, May 25, 2016 at 6:56 PM, Steven D'Aprano > wrote: > >> On Wed, May 25, 2016 at 01:11:35PM +0000, Michael Selik wrote: > >> What is your evidence for this claim? So far I've only seen one real- > >> world use-case for this, and that single use-case would be well served > >> by a simpler syntax: > >> > >> a, b, c = **mapping > > > > I have to warn here. This looks cool but it does something that AFAIK > > no other Python syntax uses -- it takes variable names and does > > something to those variables but *also* uses their actual names as > > string literals. I agree that the use cases for this seem pretty > > sweet, but perhaps we should give it a somewhat different syntax just > > so it's clear that the names on the LHS matter. The precedent that the > > targets must be actual names rather than anything you can assign to is > > also kind of scary. > > I understand the concern, and maybe you are right. However > ?, this:? > > ? ? > def func(y, z, x) > ? ? > print(x, y, z) > > ? ? > func(**dict(x=1, y=2, z=3)) > > prints "1 2 3" > ?, a > ?nd so doe > ?s? > ? > func(x=1, y=2, z=3)? > > and > > ? func(z=3, x=1, y=2) > > ?So `a, b, c = **mapping` would be perfectly in line with this. Of course > there may still be confusion, but that would mean the user would probably > already be confused about whether dicts are ordered or not, so that > confusion would need to be fixed anyway. I think the key is that ** should > _never_ be interpreted as unpack/repack by order. Or in other words, it > always means unpack/repack _by name_. > > That said, here's a couple of suggestions: > > **(a, b, c) = **mapping > > **{a, b, c} = **mapping > > Although `a, b, c = **mapping` would still be more convenient. > > ?-- Koos > > > > PS. For even more explicitness: > > a from 'a', b from 'b', c from 'c' = **mapping > > Which would allow even > > b from 1, a from 0, x from 2 = **iterable > > Or > > a, b, c from 'a', 'b', 'c' in mapping > b, a, c from 1, 0, 2 in mapping > > > > > > -- > > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu May 26 11:29:31 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 26 May 2016 08:29:31 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <574696A8.3070403@canterbury.ac.nz> References: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> <57463845.8090709@stoneleaf.us> <574662CA.3060704@stoneleaf.us> <574696A8.3070403@canterbury.ac.nz> Message-ID: <5747165B.1070808@stoneleaf.us> On 05/25/2016 11:24 PM, Greg Ewing wrote: > Chris Angelico wrote: >> I hope there doesn't end up being a confusion between mapping >> unpacking and set display. Sets are a bit of an odd duck; are they >> like lists only unordered, or like mappings only without values? > > Let's just hope nobody says they want set unpacking... We already have that: --> a, b, c = {7, 99, 22} --> a, b, c (99, 22, 7) Of course, the order gets all messed up... ;) -- ~Ethan~ From srkunze at mail.de Thu May 26 12:21:25 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 26 May 2016 18:21:25 +0200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <57467275.3010009@stoneleaf.us> References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <877fehuamr.fsf@thinkpad.rath.org> <22342.24746.141240.623012@turnbull.sk.tsukuba.ac.jp> <57467275.3010009@stoneleaf.us> Message-ID: <57472285.1000406@mail.de> On 26.05.2016 05:50, Ethan Furman wrote: > I would just as soon use the switch/case keywords, with extra matching > oomph. It doesn't have to match C(-like) exactly, and would be > immediately recognizable as to the intent (following tests are against > the thing in the switch portion). > > I think it's okay to put our own Python spin on it. I think "test one thing against multiple conditions" is a quite abstract thing to do. I would like to see some real-world use-case for this kind of syntax. Best, Sven From srkunze at mail.de Thu May 26 12:32:15 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 26 May 2016 18:32:15 +0200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <5745B15E.5020703@feete.org> References: <5745B15E.5020703@feete.org> Message-ID: <5747250F.70904@mail.de> On 25.05.2016 16:06, Ian Foote wrote: > I can see this being useful if I have a Django Rest Framework validate > method > (http://www.django-rest-framework.org/api-guide/serializers/#object-level-validation) > which takes a dictionary argument. > > def validate(self, data): > {'name': name, 'address': address} = data > > Currently, this would look like: > > def validate(self, data): > name, address = data['name'], data['address'] Now, that you mention it (the restframework), I remember a concrete usecase for our systems as well. We receive a YAML resource (via the restframework -- that's the association) and we know it must be a dict and contain several items. Additionally, we need the remaining items to store somewhere else depending on some of the received data. {'needed1': needed1, 'needed2': needed2, **rest} = yaml_data store_there(needed1, rest) store_somewhere(needed2, rest) Please note, that needed1 and needed2 are not allowed to be part of the data which is supposed to be stored. These are mere flags. > > It does get more useful with the extension: > > def validate(self, data): > {'name': name, 'address': address, **rest} = data @Michael Does using ** instead of * seem more appropriate? > > instead of: > > def validate(self, data): > rest = data.copy() > name = rest.pop('name') > address = rest.pop('address') > > In the rest framework case, mutating data directly might not be a > problem, but this does feel like a nice syntax when avoiding mutation > is required. > > Regards, > Ian Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Thu May 26 12:41:42 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 26 May 2016 18:41:42 +0200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <5746347B.8020809@stoneleaf.us> References: <20160525184206.GZ12028@ando.pearwood.info> <5745FF44.60105@mail.de> <574606DF.2060506@stoneleaf.us> <57462F4C.8090908@mail.de> <5746347B.8020809@stoneleaf.us> Message-ID: <57472746.9040005@mail.de> On 26.05.2016 01:25, Ethan Furman wrote: > We already have that situation: > > some_var = [1, 2] > a = some_var[0] > b = some_var[1] > > is exactly the same as > > a, b = some_var > > and the visual indicators (the integers, the item access) are nowhere > to be seen. Good point. However, don't you think this case is different? As a human (at least the ones I know) can easily relate position and index number. So, 1 is first, 2 is second and so on. That's pretty straightforward. So, the item access is quite naturally mapped. With keys of a dictionary it is less clear. I prefer the explicit key variant over some implicitly working one. Maybe there will come up a better argument for or against this. We will see. Best, Sven From srkunze at mail.de Thu May 26 12:50:12 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 26 May 2016 18:50:12 +0200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> Message-ID: <57472944.7020102@mail.de> On 26.05.2016 14:27, Koos Zevenhoven wrote: > That said, here's a couple of suggestions: > > **(a, b, c) = **mapping > > **{a, b, c} = **mapping > > Although `a, b, c = **mapping` would still be more convenient. > > ?-- Koos Do you see what you just did there? **{a, b, c} = **mapping That seems like a mathematical equation where the ** appear to be superfluous (besides I don't really like special characters ;-) ). So it yields: {a, b, c} = mapping However and additionally, here the LHS (and the LHS in your suggestion) reminds me of a set. That's not good I guess. I one now adds the keys back in we go back to {'a': s1, 'b': s2, 'c': s3} = mapping So far, all proposals which deviate from Michael's one are just "optimizations in terms of characters". The only one I would find not necessarily too restrictive were: 'a': s1, 'b': s2, 'c': s3 = mapping # no braces :) That looks quite good to me. What do you think? Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Thu May 26 12:51:48 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 26 May 2016 18:51:48 +0200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <57463FE3.7020202@stoneleaf.us> References: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> <57463845.8090709@stoneleaf.us> <97158294-6397-0b0a-aae3-f631f3133f43@mrabarnett.plus.com> <57463FE3.7020202@stoneleaf.us> Message-ID: <574729A4.2030408@mail.de> On 26.05.2016 02:14, Ethan Furman wrote: > >> or, perhaps: >> >> 'a' as x, 'b' as y, 'c' as z = some_dict >> >> which would cater for keys that aren't valid as identifiers. > > That probably makes more sense, especially since it's already the > rare(r) case of needing/wanting to rename the keys. What do you think about? 'a': x, 'b': y, 'c': z = some_dict Best, Sven From k7hoven at gmail.com Thu May 26 12:56:45 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 26 May 2016 19:56:45 +0300 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <574729A4.2030408@mail.de> References: <1464187232.64628.618401001.6FE9D99A@webmail.messagingengine.com> <57463845.8090709@stoneleaf.us> <97158294-6397-0b0a-aae3-f631f3133f43@mrabarnett.plus.com> <57463FE3.7020202@stoneleaf.us> <574729A4.2030408@mail.de> Message-ID: On May 26, 2016 7:52 PM, "Sven R. Kunze" wrote: > > On 26.05.2016 02:14, Ethan Furman wrote: >> >> >>> or, perhaps: >>> >>> 'a' as x, 'b' as y, 'c' as z = some_dict >>> >>> which would cater for keys that aren't valid as identifiers. >> >> >> That probably makes more sense, especially since it's already the rare(r) case of needing/wanting to rename the keys. > > > What do you think about? > > 'a': x, 'b': y, 'c': z = some_dict > Maybe keep that for potential non-comment type hints in the future... -- Koos (mobile) > Best, > Sven > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu May 26 13:11:21 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 26 May 2016 18:11:21 +0100 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <57472285.1000406@mail.de> References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <877fehuamr.fsf@thinkpad.rath.org> <22342.24746.141240.623012@turnbull.sk.tsukuba.ac.jp> <57467275.3010009@stoneleaf.us> <57472285.1000406@mail.de> Message-ID: On 26 May 2016 at 17:21, Sven R. Kunze wrote: > I think "test one thing against multiple conditions" is a quite abstract thing to do. > > I would like to see some real-world use-case for this kind of syntax Today, I was parsing a job output file. For each line in the file, I tested it against various patterns/conditions to see what type of line it was. Depending on the type of line, I did something different. I used a chain of if/elif statements, because that's what Python currently (3.5) provides, but it would have been an obvious case for a match/switch statement. If the important thing here is *structural* matching then what I actually did was call split() on the line to get a list of elements, then wanted some to be constants, and I picked data out of others. Something like match line_parts: case _, 'Job', 'Start', start_time: do_something1(start_time) case _, 'Job', 'End', end_time: do_something2(end_time) case view, 'size', 'is', kb, 'KB': do_something3(view, kb) Is that the type of example you had in mind? Paul From steve at pearwood.info Thu May 26 13:40:17 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 27 May 2016 03:40:17 +1000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <57472944.7020102@mail.de> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> Message-ID: <20160526174016.GD12028@ando.pearwood.info> On Thu, May 26, 2016 at 06:50:12PM +0200, Sven R. Kunze wrote: > So far, all proposals which deviate from Michael's one are just > "optimizations in terms of characters". The only one I would find not > necessarily too restrictive were: > > > 'a': s1, 'b': s2, 'c': s3 = mapping # no braces :) > > > That looks quite good to me. What do you think? I think that if you submitted code to me with keys 'a', 'b', 'c' and variables s1, s2, s3, I'd probably reject it and tell you to use descriptive, meaningful keys and names. I wish people would stop giving toy examples as examples of how nice the syntax looks, and instead try to use it with descriptive names taken from real code. I believe that, by far the majority of the time, you will be repeating the same names twice, and likely exceeding most reasonable line lengths: 'referer': referer, 'useragent': useragent, 'use_proxy': use_proxy, 'follow_links': follow_links, 'clobber': clobber, 'timeout': timeout = mapping Still think it looks quite good? If you do, that's your right, of course, it's a matter of personal taste. But using toy examples with one or two letter variable names is not a fair or realistic test of what it will be like to use this syntax in real code. -- Steve From steve at pearwood.info Thu May 26 13:51:00 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 27 May 2016 03:51:00 +1000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <574703C2.7070409@gmail.com> References: <20160526015606.GB12028@ando.pearwood.info> <574703C2.7070409@gmail.com> Message-ID: <20160526175100.GE12028@ando.pearwood.info> On Thu, May 26, 2016 at 04:10:10PM +0200, Michel Desmoulin wrote: > The thing with those details is that you can completly ignore them, and > don't know they exist, and simply look it up when you need it. What Google search terms would a Python programmer use to find out what {"1": a, "foo.bar": b, **c} = **mapping does? Please don't dismiss the effect of unfamiliar syntax on the reader. Adding more magic syntax increases the cost and difficulty of reading the code and learning the language. That cost might be justified if the new syntax is useful enough, but so far this syntax appears to be of very marginal usefulness. There's no obvious use-case where it would be an overwhelming benefit, at least not yet. I believe this syntax comes from Clojure. Can you give some examples of real-world code using this syntax in Clojure? (Not toy demonstrations of how it works, but working code that uses it to solve real problems.) If Clojure programmers don't use it, then I expect neither will Python programmers. -- Steve From ethan at stoneleaf.us Thu May 26 13:55:28 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 26 May 2016 10:55:28 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <20160526174016.GD12028@ando.pearwood.info> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> Message-ID: <57473890.6050400@stoneleaf.us> On 05/26/2016 10:40 AM, Steven D'Aprano wrote: > On Thu, May 26, 2016 at 06:50:12PM +0200, Sven R. Kunze wrote: >> So far, all proposals which deviate from Michael's one are ju >> >> 'a': s1, 'b': s2, 'c': s3 = mapping # no braces :) >> >> That looks quite good to me. What do you think? > > I think that if you submitted code to me with keys 'a', 'b', 'c' and > variables s1, s2, s3, I'd probably reject it and tell you to use > descriptive, meaningful keys and names. > > I wish people would stop giving toy examples as examples of how nice the > syntax looks, and instead try to use it with descriptive names taken > from real code. I believe that, by far the majority of the time, you > will be repeating the same names twice, and likely exceeding most > reasonable line lengths: > > 'referer': referer, 'useragent': useragent, 'use_proxy': use_proxy, 'follow_links': follow_links, 'clobber': clobber, 'timeout': timeout = mapping > > Still think it looks quite good? If you do, that's your right, of > course, it's a matter of personal taste. But using toy examples with one > or two letter variable names is not a fair or realistic test of what it > will be like to use this syntax in real code. With the simple syntax that I could live with, a real example could be: {active_id, active_ids, active_model} = context or {partner_id, product_id, ship_to, product_ids} = values which is more readable than (partner_id, product_id, ship_to, product_ids = (values[k] for k in ['partner_id', 'product_id', 'ship_to', 'product_ids'])) Wow. That's a lot of room for typos and wrong order. -- ~Ethan~ From Nikolaus at rath.org Thu May 26 14:21:50 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Thu, 26 May 2016 11:21:50 -0700 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: (Franklin's message of "Wed, 25 May 2016 05:04:35 -0400") References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> Message-ID: <87shx4d6ip.fsf@thinkpad.rath.org> On May 25 2016, "Franklin? Lee" wrote: > On Wed, May 25, 2016 at 1:52 AM, Nick Coghlan wrote: >> Using the running demo: >> >> def demo(arg): >> given arg: >> case x, y, *_: # Tuple matching (implicit name binding) >> ... >> case (.x, .y) as p, q: # Attribute matching >> ... >> case (["x"], ["y"]) as p, q: # Item matching >> ... >> case (.x) as p and isinstance(p, int): # Match + condition >> ... >> case if isinstance(arg, int): # Condition only >> ... >> else: # Default >> ... >> >> The other key change there is introducing "as" to the individual cases >> in order to be able to separate the match pattern definition from the >> local name binding. > > I still don't like that `case THING` is a pattern, rather than a value > to test against. Here's my modifications with "as", attributes, and > > def demo(arg): > given arg: > case as x, y, *_: # Tuple matching (implicit name binding) > ... > case as object(x=p, y=q, **_): # Attribute matching > ... > case as {'x': p, 'y', q, **_}: # Item matching > ... > case as object(x=p, **_) and isinstance(p, int): # Match + condition > ... > case if isinstance(arg, int): # Condition only > ... > else: # Default I think all the ideas with "as" are difficult to read. I think its much better to embed the target variables in the pattern - we just need a way to mark them as such. Mathematica has the same problem and solves it with a trailing _, but we can't do that because our variables names may contain them. But maybe we could use $? Most people already strongly associate this with variables. Example: given foo case (x,y): # matches if foo == (x,y) case (x, $y): # matches if len(foo) == 2 and foo[0] == x, # and assigns y = foo[1] case {'bar': $x, y: $z}: # matches if foo is a map that has 'bar' and y keys # and assigns x = foo['bar'], z = foo[y] case $x.bar: # matches if hasattr(foo, 'bar') and assigns x = foo Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From Nikolaus at rath.org Thu May 26 14:35:41 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Thu, 26 May 2016 11:35:41 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <20160526175100.GE12028@ando.pearwood.info> (Steven D'Aprano's message of "Fri, 27 May 2016 03:51:00 +1000") References: <20160526015606.GB12028@ando.pearwood.info> <574703C2.7070409@gmail.com> <20160526175100.GE12028@ando.pearwood.info> Message-ID: <87oa7sd5vm.fsf@thinkpad.rath.org> On May 27 2016, Steven D'Aprano wrote: > On Thu, May 26, 2016 at 04:10:10PM +0200, Michel Desmoulin wrote: > >> The thing with those details is that you can completly ignore them, and >> don't know they exist, and simply look it up when you need it. > > What Google search terms would a Python programmer use to find out what > > {"1": a, "foo.bar": b, **c} = **mapping > > does? "python syntax reference" That said, I don't like the idea either. But anything that introduces new syntax is going to be hard to Google. How do you Google what def foo(bar) -> "something obscure" means in Python? Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From p.f.moore at gmail.com Thu May 26 15:15:57 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 26 May 2016 20:15:57 +0100 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <87shx4d6ip.fsf@thinkpad.rath.org> References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <87shx4d6ip.fsf@thinkpad.rath.org> Message-ID: On 26 May 2016 at 19:21, Nikolaus Rath wrote: > I think all the ideas with "as" are difficult to read. I think its much > better to embed the target variables in the pattern - we just need a way > to mark them as such. > > Mathematica has the same problem and solves it with a trailing _, but we > can't do that because our variables names may contain them. But maybe we > could use $? Most people already strongly associate this with variables. > > Example: > > given foo > case (x,y): > # matches if foo == (x,y) > case (x, $y): > # matches if len(foo) == 2 and foo[0] == x, > # and assigns y = foo[1] > case {'bar': $x, y: $z}: > # matches if foo is a map that has 'bar' and y keys > # and assigns x = foo['bar'], z = foo[y] > case $x.bar: > # matches if hasattr(foo, 'bar') and assigns x = foo That's quite a nice idea.I'm not sure whether it feels pythonic to me, though - I'd be very interested to see whether Guido loves or hates this. The "case $x.bar" example is a bit unfortunate - ideally, after testing for existence of the attribute, you would want to be able to name the value of the attribute, not the containing object. Of course you can access the attribute as x.bar, but that costs an additional lookup - hardly the end of the world, but not ideal. Oh, and "given...case" is a pretty nice keyword pair, too. Not a disastrous clash with a common variable name (like match) and no pre-existing connotations (like switch). I think it's been mentioned as a possibility earlier in the thread, but seemed to get lost in the noise. Paul From p.f.moore at gmail.com Thu May 26 15:25:19 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 26 May 2016 20:25:19 +0100 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <57473890.6050400@stoneleaf.us> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> Message-ID: On 26 May 2016 at 18:55, Ethan Furman wrote: > With the simple syntax that I could live with, a real example could be: > > {active_id, active_ids, active_model} = context > > or > > {partner_id, product_id, ship_to, product_ids} = values The behaviour of using the names of the variables from the LHS to introspect the value on the RHS is, to me, extremely magical and unlike anything I've seen in any other language. I don't think it sits well in Python, even though it is certainly a very readable idiom for the sort of unpacking we're talking about here. One other disadvantage of this syntax is that it would break badly if someone refactored the code and renamed one of the variables. Of course, the semantics of this construct means that renaming the variables changes the meaning - but once again, that's not something I can recall ever having seen in any language. Having said all that... > which is more readable than > > (partner_id, product_id, ship_to, product_ids = > (values[k] for k in > ['partner_id', 'product_id', 'ship_to', 'product_ids'])) > > Wow. That's a lot of room for typos and wrong order. I agree - this is pretty horrible. Although marginally better than the proposed {'partner_id': partner_id, ...} form with explicit naming of the keys. Personally, though, I don't see *that* much wrong with partner_id = values['partner_id'] product_id = values['product_id'] ship_to = values['ship_to'] product_ids = values['product_ids'] It's a bit repetitive, and maybe a little verbose, but nothing a good editor or IDE (or anything better than gmail's web interface :-)) wouldn't make straightforward to manage. Paul From ethan at stoneleaf.us Thu May 26 15:49:16 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 26 May 2016 12:49:16 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> Message-ID: <5747533C.7080202@stoneleaf.us> On 05/26/2016 12:25 PM, Paul Moore wrote: > On 26 May 2016 at 18:55, Ethan Furman wrote: >> With the simple syntax that I could live with, a real example could be: >> >> {active_id, active_ids, active_model} = context >> >> or >> >> {partner_id, product_id, ship_to, product_ids} = values > > The behaviour of using the names of the variables from the LHS to > introspect the value on the RHS is, to me, extremely magical and > unlike anything I've seen in any other language. I don't think it sits > well in Python, even though it is certainly a very readable idiom for > the sort of unpacking we're talking about here. > > One other disadvantage of this syntax is that it would break badly if > someone refactored the code and renamed one of the variables. Of > course, the semantics of this construct means that renaming the > variables changes the meaning - but once again, that's not something I > can recall ever having seen in any language. > > Having said all that... > >> which is more readable than >> >> (partner_id, product_id, ship_to, product_ids = >> (values[k] for k in >> ['partner_id', 'product_id', 'ship_to', 'product_ids'])) >> >> Wow. That's a lot of room for typos and wrong order. > > I agree - this is pretty horrible. Although marginally better than the > proposed {'partner_id': partner_id, ...} form with explicit naming of > the keys. > > Personally, though, I don't see *that* much wrong with > > partner_id = values['partner_id'] > product_id = values['product_id'] > ship_to = values['ship_to'] > product_ids = values['product_ids'] And this is what I currently do. But a fellow can dream, right? :) -- ~Ethan~ From python at mrabarnett.plus.com Thu May 26 15:48:36 2016 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 26 May 2016 20:48:36 +0100 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> Message-ID: <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> On 2016-05-26 20:25, Paul Moore wrote: [snip] > Personally, though, I don't see *that* much wrong with > > partner_id = values['partner_id'] > product_id = values['product_id'] > ship_to = values['ship_to'] > product_ids = values['product_ids'] > > It's a bit repetitive, and maybe a little verbose, but nothing a good > editor or IDE (or anything better than gmail's web interface :-)) > wouldn't make straightforward to manage. > Could we use semicolons in the subscript to create a tuple? They could be used for packing or unpacking: partner_id, product_id, ship_to, product_ids = values['partner_id'; 'product_id'; 'ship_to'; 'product_ids'] my_dict['partner_id'; 'product_id'; 'ship_to'; 'product_ids'] = partner_id, product_id, ship_to, product_ids Or they would that be too easily confused with commas? From p.f.moore at gmail.com Thu May 26 16:18:14 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 26 May 2016 21:18:14 +0100 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> Message-ID: On 26 May 2016 at 20:48, MRAB wrote: > On 2016-05-26 20:25, Paul Moore wrote: > [snip] > >> Personally, though, I don't see *that* much wrong with >> >> partner_id = values['partner_id'] >> product_id = values['product_id'] >> ship_to = values['ship_to'] >> product_ids = values['product_ids'] >> >> It's a bit repetitive, and maybe a little verbose, but nothing a good >> editor or IDE (or anything better than gmail's web interface :-)) >> wouldn't make straightforward to manage. >> > Could we use semicolons in the subscript to create a tuple? They could be > used for packing or unpacking: > > partner_id, product_id, ship_to, product_ids = values['partner_id'; > 'product_id'; 'ship_to'; 'product_ids'] > > my_dict['partner_id'; 'product_id'; 'ship_to'; 'product_ids'] = > partner_id, product_id, ship_to, product_ids > > Or they would that be too easily confused with commas? I'd imagine it would be confusing. And personally, I *still* find that syntax less readable than the sequence of assignments. Full disclosure - I've written that sort of sequence of assignments quite a few times, and it's annoyed me every time I have. So I sympathise with the desire for "something better". But now that we're using real-world names, I'm finding that none of the proposed options are actually qualifying as "better" - just "different"... Paul From srkunze at mail.de Thu May 26 17:39:05 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 26 May 2016 23:39:05 +0200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <877fehuamr.fsf@thinkpad.rath.org> <22342.24746.141240.623012@turnbull.sk.tsukuba.ac.jp> <57467275.3010009@stoneleaf.us> <57472285.1000406@mail.de> Message-ID: <57476CF9.2060207@mail.de> On 26.05.2016 19:11, Paul Moore wrote: > On 26 May 2016 at 17:21, Sven R. Kunze wrote: >> I think "test one thing against multiple conditions" is a quite abstract thing to do. >> >> I would like to see some real-world use-case for this kind of syntax > Today, I was parsing a job output file. For each line in the file, I > tested it against various patterns/conditions to see what type of line > it was. Depending on the type of line, I did something different. I > used a chain of if/elif statements, because that's what Python > currently (3.5) provides, but it would have been an obvious case for a > match/switch statement. If the important thing here is *structural* > matching then what I actually did was call split() on the line to get > a list of elements, then wanted some to be constants, and I picked > data out of others. Something like > > match line_parts: > case _, 'Job', 'Start', start_time: do_something1(start_time) > case _, 'Job', 'End', end_time: do_something2(end_time) > case view, 'size', 'is', kb, 'KB': do_something3(view, kb) > > Is that the type of example you had in mind? > Paul Interesting to see that you chose chains of if-elses. I wouldn't. Especially because our guidelines tells us to avoid *elses* and *ifs* if possible. But here, it's also my personal preference of avoiding a switch-case-like mess. So, in this particular case, I would have used a datastructure for describing how to work with the incoming data. That gives me three advantages over a syntax-using solution: 1) a datastructure (like a dict) would be first-class and I could move it around, manipulate it etc for whatever reason 2) I am able to run separate tests for the matching algorithm (aka finding the right mapping) without applying mocks for *all* possible match-cases 3) no monolithic code block that tends to grow It is also interesting to see that it worked for you. I can remember a similar task quite some time ago. However, there, I needed regular expressions to solve the issue at hand. So, a simple structural unpacking didn't suffice. However, a loop over a list of (regex, action)s with a single check made it work for me. Btw. Django uses this way of matching for the URLs system. And because it needs to be modular (different apps can contribute to it), it would be pointless to have a syntax for it. Even dynamic changes would not be possible with syntax which we do in various places because of, well let's call it, our customer requirements. ;-) My point here is not that not somebody could make use of it (apparently it would for you). However, the absence of it forces people to invent other more flexible mechanisms and I'd actually like it this way. This said, I could rather imagine a new function introduced to functools which provides some sort of match-and-dispatch functionality which you actually do above. Not sure if somebody already suggested this on this thread. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu May 26 17:49:52 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 26 May 2016 22:49:52 +0100 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <57476CF9.2060207@mail.de> References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <877fehuamr.fsf@thinkpad.rath.org> <22342.24746.141240.623012@turnbull.sk.tsukuba.ac.jp> <57467275.3010009@stoneleaf.us> <57472285.1000406@mail.de> <57476CF9.2060207@mail.de> Message-ID: On 26 May 2016 at 22:39, Sven R. Kunze wrote: > However, the absence of it forces people to invent other more flexible > mechanisms and I'd actually like it this way. For a bigger project maybe. If I were writing a proper parser for my file, I wouldn't have done it like I did. But this was a relatively quick, essentially one-off, data extraction task (that's what I do 99% of the time) and for that, simple solutions are much more appropriate. If I had a matching statement in my toolkit, I would have used it. If I had a matching *library* I would have used it. (But not regex, it's too messy for this particular task). But I wouldn't build my own "more flexible" mechanism - my job today was to get the data out, not to write reusable (or even long-term maintainable, if I'm honest!) code. This does make one good point though - if this sort of matching could be provided with a library, that would be just as viable an option for my use case. But I've not seen such a library yet. Paul From Nikolaus at rath.org Thu May 26 18:10:59 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Thu, 26 May 2016 15:10:59 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <57473890.6050400@stoneleaf.us> (Ethan Furman's message of "Thu, 26 May 2016 10:55:28 -0700") References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> Message-ID: <878tywzczw.fsf@thinkpad.rath.org> On May 26 2016, Ethan Furman wrote: > On 05/26/2016 10:40 AM, Steven D'Aprano wrote: >> On Thu, May 26, 2016 at 06:50:12PM +0200, Sven R. Kunze wrote: > >>> So far, all proposals which deviate from Michael's one are ju >>> >>> 'a': s1, 'b': s2, 'c': s3 = mapping # no braces :) >>> >>> That looks quite good to me. What do you think? >> >> I think that if you submitted code to me with keys 'a', 'b', 'c' and >> variables s1, s2, s3, I'd probably reject it and tell you to use >> descriptive, meaningful keys and names. >> >> I wish people would stop giving toy examples as examples of how nice the >> syntax looks, and instead try to use it with descriptive names taken >> from real code. I believe that, by far the majority of the time, you >> will be repeating the same names twice, and likely exceeding most >> reasonable line lengths: >> >> 'referer': referer, 'useragent': useragent, 'use_proxy': use_proxy, >> 'follow_links': follow_links, 'clobber': clobber, 'timeout': timeout >> = mapping >> >> Still think it looks quite good? If you do, that's your right, of >> course, it's a matter of personal taste. But using toy examples with one >> or two letter variable names is not a fair or realistic test of what it >> will be like to use this syntax in real code. > > With the simple syntax that I could live with, a real example could be: > > {active_id, active_ids, active_model} = context > > or > > {partner_id, product_id, ship_to, product_ids} = values > > which is more readable than > > (partner_id, product_id, ship_to, product_ids = > (values[k] for k in > ['partner_id', 'product_id', 'ship_to', 'product_ids'])) > > Wow. That's a lot of room for typos and wrong order. Where possible, I use for n in ('partner_id', 'product_id', 'ship_to', 'product_ids'): globals()[n] = values[n] but it would be nice to have a solution that's friendlier for static analyzers (the above code almost always produces warnings when e.g. partner_id is first accessed afterwards). Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From michael.selik at gmail.com Thu May 26 18:14:47 2016 From: michael.selik at gmail.com (Michael Selik) Date: Thu, 26 May 2016 22:14:47 +0000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <5747250F.70904@mail.de> References: <5745B15E.5020703@feete.org> <5747250F.70904@mail.de> Message-ID: On Thu, May 26, 2016 at 12:32 PM Sven R. Kunze wrote: > On 25.05.2016 16:06, Ian Foote wrote: > > def validate(self, data): > {'name': name, 'address': address, **rest} = data > > @Michael > Does using ** instead of * seem more appropriate? > It does. ``*args`` creates a tuple. ``**kwargs`` creates a dict. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu May 26 18:20:44 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 27 May 2016 10:20:44 +1200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <20160526174016.GD12028@ando.pearwood.info> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> Message-ID: <574776BC.9020209@canterbury.ac.nz> Steven D'Aprano wrote: > I believe that, by far the majority of the time, you will be repeating the > same names twice, and likely exceeding most reasonable line lengths: > > 'referer': referer, 'useragent': useragent, 'use_proxy': use_proxy, > 'follow_links': follow_links, 'clobber': clobber, 'timeout': timeout = > mapping I think a more realistic usage would be to give short local names to something having long keys. ('referer': ref, 'useragent': ua, 'use_proxy': prox, 'follow_links': fl, 'clobber': clob, 'timeout': to) = mapping So tying the keys to the names of the target variables might be too restrictive. -- Greg From neatnate at gmail.com Thu May 26 18:28:25 2016 From: neatnate at gmail.com (Nathan Schneider) Date: Thu, 26 May 2016 23:28:25 +0100 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> Message-ID: On Thu, May 26, 2016 at 8:48 PM, MRAB wrote: > On 2016-05-26 20:25, Paul Moore wrote: > [snip] > > Personally, though, I don't see *that* much wrong with >> >> partner_id = values['partner_id'] >> product_id = values['product_id'] >> ship_to = values['ship_to'] >> product_ids = values['product_ids'] >> >> It's a bit repetitive, and maybe a little verbose, but nothing a good >> editor or IDE (or anything better than gmail's web interface :-)) >> wouldn't make straightforward to manage. >> >> Could we use semicolons in the subscript to create a tuple? They could be > used for packing or unpacking: > > partner_id, product_id, ship_to, product_ids = values['partner_id'; > 'product_id'; 'ship_to'; 'product_ids'] > > my_dict['partner_id'; 'product_id'; 'ship_to'; 'product_ids'] = > partner_id, product_id, ship_to, product_ids > > Instead of special syntax, what if dict.values() returned a tuple when given keys as arguments: partner_id, product_id, ship_to, product_ids = my_dict.values('partner_id', 'product_id', 'ship_to', 'product_ids') That avoids repeating the dict variable, at least. And as there is dict.update(), I don't see the need for a new syntax for assigning to multiple keys. Nathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu May 26 19:10:08 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 27 May 2016 11:10:08 +1200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <87shx4d6ip.fsf@thinkpad.rath.org> References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <87shx4d6ip.fsf@thinkpad.rath.org> Message-ID: <57478250.6000707@canterbury.ac.nz> Nikolaus Rath wrote: > I think its much > better to embed the target variables in the pattern - we just need a way > to mark them as such. > > maybe we could use $? I'd like to suggest using '?' instead. It looks more pattern- matchy to me. Trying to come up with some real-looking examples: switch obj: case Point(?x, ?y): print("Point at", x, y) case Line(start = ?p1, end = ?p2, width = ?w): print("Line from", p1, "to", p2, "with width", w) case Intersection(?obj1, ?obj2): print("Intersection of", obj1, "and", obj2) switch command: case ("frobulate", ?inp, ?out): print("Frobulating", inp", "to give", out) case ("frobulate", "-v", ?inp, ?out): print("Frobulating", inp, "verbosely to give", out) case ("xyzzy", ?) print("Xyzzying not implemented, sorry.") else: print("Couldn't grok your command, master.") Here I've also used '?' without a following name as a "don't care" marker. -- Greg From rob.cliffe at btinternet.com Thu May 26 22:34:54 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 27 May 2016 03:34:54 +0100 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <20160526175100.GE12028@ando.pearwood.info> References: <20160526015606.GB12028@ando.pearwood.info> <574703C2.7070409@gmail.com> <20160526175100.GE12028@ando.pearwood.info> Message-ID: <8d2131cd-ee5a-0518-81df-0aad73638746@btinternet.com> On 26/05/2016 18:51, Steven D'Aprano wrote: > On Thu, May 26, 2016 at 04:10:10PM +0200, Michel Desmoulin wrote: > >> The thing with those details is that you can completly ignore them, and >> don't know they exist, and simply look it up when you need it. > What Google search terms would a Python programmer use to find out what > > {"1": a, "foo.bar": b, **c} = **mapping > > does? > > Please don't dismiss the effect of unfamiliar syntax on the reader. > Adding more magic syntax increases the cost and difficulty of reading > the code and learning the language. That cost might be justified if the > new syntax is useful enough, but so far this syntax appears to be of > very marginal usefulness. There's no obvious use-case where it would be > an overwhelming benefit, at least not yet. +1 With tuple and list unpacking (a,b) = (1,2) [a,b] = [1,2] # or even mixing: [a,b] = (1,2) the structure of the LHS and RHS mirror each other, making the meaning intuitive/obvious. Doing the same thing for dicts: { 'a' : a, 'b' : b } = { 'a' : 1, 'b' : 2 } makes the LHS too verbose to be very useful IMO. All the examples so far can be done in other, arguably better, ways. While a more concise syntax would break the "mirroring" and be confusing, or at least more to learn as Steven says. > > I believe this syntax comes from Clojure. Can you give some examples of > real-world code using this syntax in Clojure? (Not toy demonstrations of > how it works, but working code that uses it to solve real problems.) > > If Clojure programmers don't use it, then I expect neither will Python > programmers. > > > From rob.cliffe at btinternet.com Thu May 26 22:43:19 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 27 May 2016 03:43:19 +0100 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <57473890.6050400@stoneleaf.us> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> Message-ID: On 26/05/2016 18:55, Ethan Furman wrote: > > With the simple syntax that I could live with, a real example could be: > > {active_id, active_ids, active_model} = context > > or > > {partner_id, product_id, ship_to, product_ids} = values > > which is more readable than > > (partner_id, product_id, ship_to, product_ids = > (values[k] for k in > ['partner_id', 'product_id', 'ship_to', 'product_ids'])) > > Wow. That's a lot of room for typos and wrong order. How about locals().update(values) or if you just want a subset for k in ['partner_id', 'product_id', 'ship_to', 'product_ids']: locals()[k] = values[k] # Look Ma, DRY From rosuav at gmail.com Thu May 26 22:57:56 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 27 May 2016 12:57:56 +1000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> Message-ID: On Fri, May 27, 2016 at 12:43 PM, Rob Cliffe wrote: > How about > > locals().update(values) Because locals() can't be updated unless it happens to be the same dict as globals(), in which case it's clearer to use that name. ChrisA From ethan at stoneleaf.us Thu May 26 23:05:09 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 26 May 2016 20:05:09 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> Message-ID: <5747B965.1000405@stoneleaf.us> On 05/26/2016 07:57 PM, Chris Angelico wrote: > On Fri, May 27, 2016 at 12:43 PM, Rob Cliffe wrote: >> How about >> >> locals().update(values) > > Because locals() can't be updated unless it happens to be the same > dict as globals(), in which case it's clearer to use that name. Actually, the dict returned by locals() can be updated, but at least in cPython those updates don't make it back to the function's locals (although they do in other Python's). -- ~Ethan~ From rosuav at gmail.com Thu May 26 23:15:29 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 27 May 2016 13:15:29 +1000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <5747B965.1000405@stoneleaf.us> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <5747B965.1000405@stoneleaf.us> Message-ID: On Fri, May 27, 2016 at 1:05 PM, Ethan Furman wrote: > On 05/26/2016 07:57 PM, Chris Angelico wrote: >> >> On Fri, May 27, 2016 at 12:43 PM, Rob Cliffe wrote: > > >>> How about >>> >>> locals().update(values) >> >> >> Because locals() can't be updated unless it happens to be the same >> dict as globals(), in which case it's clearer to use that name. > > > Actually, the dict returned by locals() can be updated, but at least in > cPython those updates don't make it back to the function's locals (although > they do in other Python's). Sorry, yeah. The dict is a dict, so of course it can be updated, but the changes aren't guaranteed to have effect. Point is, it's not a suitable way to do this. :) ChrisA From steve at pearwood.info Thu May 26 23:39:31 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 27 May 2016 13:39:31 +1000 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> Message-ID: <20160527033930.GF12028@ando.pearwood.info> I think this is important enough to get a change in subject line, lest it be lost in the dict unpacking thread. On Thu, May 26, 2016 at 11:28:25PM +0100, Nathan Schneider wrote: > Instead of special syntax, what if dict.values() returned a tuple when > given keys as arguments: > > partner_id, product_id, ship_to, product_ids = my_dict.values( > 'partner_id', 'product_id', 'ship_to', 'product_ids') > > That avoids repeating the dict variable, at least. And as there is > dict.update(), I don't see the need for a new syntax for assigning to > multiple keys. I like this idea. I think it beats the status quo: partner_id = my_dict['partner_id'] product_id = my_dict['product_id'] # etc. and the various "dict unpacking" syntax suggested, e.g.: {'partner_id': partner_id, 'product_id': product_id, **catch_all} = my_dict and it's less magical than variants that extract the names from the left hand side: partner_id, product_id, ship_to, product_ids = **my_dict It naturally and trivially supports the case where assignment targets aren't names, and where keys are not identifiers: obj.partner_id, products[the_id] = my_dict.values('partner id', 'is') It's still a bit repetitive in the simple case where the keys are the same as the variable names, but without compiler magic, what else are you going to do? +1 -- Steve From guido at python.org Fri May 27 00:26:59 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 26 May 2016 21:26:59 -0700 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: <20160527033930.GF12028@ando.pearwood.info> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> Message-ID: On Thursday, May 26, 2016, Steven D'Aprano wrote: > I think this is important enough to get a change in subject line, lest > it be lost in the dict unpacking thread. > > > On Thu, May 26, 2016 at 11:28:25PM +0100, Nathan Schneider wrote: > > > Instead of special syntax, what if dict.values() returned a tuple when > > given keys as arguments: > > > > partner_id, product_id, ship_to, product_ids = my_dict.values( > > 'partner_id', 'product_id', 'ship_to', 'product_ids') > > > > That avoids repeating the dict variable, at least. And as there is > > dict.update(), I don't see the need for a new syntax for assigning to > > multiple keys. > > I like this idea. I think it beats the status quo: > > partner_id = my_dict['partner_id'] > product_id = my_dict['product_id'] > # etc. > > and the various "dict unpacking" syntax suggested, e.g.: > > {'partner_id': partner_id, 'product_id': product_id, **catch_all} = my_dict > > > and it's less magical than variants that extract the names from the left > hand side: > > partner_id, product_id, ship_to, product_ids = **my_dict > > It naturally and trivially supports the case where assignment targets > aren't names, and where keys are not identifiers: > > obj.partner_id, products[the_id] = my_dict.values('partner id', 'is') > > It's still a bit repetitive in the simple case where the keys are the > same as the variable names, but without compiler magic, what else are > you going to do? > > +1 > Interesting. It should probably have a different name. What type should it return? Iterator? Sequence? It can't really be a ValuesView because that class is just a view on the hash table. Even though you technically *could* combine this functionality into values(), I don't think it would be helpful to do so -- if only because of the surprising edge case where if you were to pass it a list of keys to extract using **args, if the list is empty, values() would default to its original behavior or returning all keys, in hash table order. --Guido -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri May 27 00:57:09 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 26 May 2016 21:57:09 -0700 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> Message-ID: <5747D3A5.9050504@stoneleaf.us> On 05/26/2016 09:26 PM, Guido van Rossum wrote: > On Thursday, May 26, 2016, Steven D'Apranowrote: >> On Thu, May 26, 2016 at 11:28:25PM +0100, Nathan Schneider wrote: >> I think this is important enough to get a change in subject line, lest >> it be lost in the dict unpacking thread. > > > >>> Instead of special syntax, what if dict.values() returned a tuple >> when given keys as arguments: >>> >>> partner_id, product_id, ship_to, product_ids = my_dict.values( >>> 'partner_id', 'product_id', 'ship_to', 'product_ids') >>> >>> That avoids repeating the dict variable, at least. And as there is >>> dict.update(), I don't see the need for a new syntax for assigning to >>> multiple keys. >> >> I like this idea. I think it beats the status quo: >> >> +1 > > Interesting. It should probably have a different name. What type should > it return? Iterator? Sequence? It can't really be a ValuesView because > that class is just a view on the hash table. Even though you technically > *could* combine this functionality into values(), I don't think it would > be helpful to do so -- if only because of the surprising edge case where > if you were to pass it a list of keys to extract using **args, if the > list is empty, values() would default to its original behavior or > returning all keys, in hash table order. Good point. The time bomb would be even worse if sometimes the dict had the same number of elements as were being asked for, as then it would be an intermittent problem. However, if we make a new method we could just as easily make a new function: def get_values_from(a_dict, keys=()): if not keys: raise an_error yield a_dict[k] for k in keys Hmmm. That could even be handy for a list/tuple: offset, name = get_values_from(a_list, [1, 7]) ;) At any rate, the return type should be an iterator. -- ~Ethan~ From k7hoven at gmail.com Fri May 27 01:25:27 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 27 May 2016 08:25:27 +0300 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: <5747D3A5.9050504@stoneleaf.us> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> <5747D3A5.9050504@stoneleaf.us> Message-ID: On Fri, May 27, 2016 at 7:57 AM, Ethan Furman wrote: > On 05/26/2016 09:26 PM, Guido van Rossum wrote: > >> On Thursday, May 26, 2016, Steven D'Apranowrote: >> >>> On Thu, May 26, 2016 at 11:28:25PM +0100, Nathan Schneider wrote: >>> >> > I think this is important enough to get a change in subject line, lest >>> it be lost in the dict unpacking thread. >>> >> >> >> >> Instead of special syntax, what if dict.values() returned a tuple >>>> >>> when given keys as arguments: >>> >>>> >>>> partner_id, product_id, ship_to, product_ids = my_dict.values( >>>> 'partner_id', 'product_id', 'ship_to', 'product_ids') >>>> >>>> That avoids repeating the dict variable, at least. And as there is >>>> dict.update(), I don't see the need for a new syntax for assigning to >>>> multiple keys. >>>> >>> >>> I like this idea. I think it beats the status quo: >>> >>> +1 >>> >> >> Interesting. It should probably have a different name. What type should >> it return? Iterator? Sequence? It can't really be a ValuesView because >> that class is just a view on the hash table. Even though you technically >> *could* combine this functionality into values(), I don't think it would >> be helpful to do so -- if only because of the surprising edge case where >> if you were to pass it a list of keys to extract using **args, if the >> list is empty, values() would default to its original behavior or >> returning all keys, in hash table order. >> > > Good point. The time bomb would be even worse if sometimes the dict had > the same number of elements as were being asked for, as then it would be an > intermittent problem. > > However, if we make a new method we could just as easily make a new > function: > > def get_values_from(a_dict, keys=()): > if not keys: > raise an_error > yield a_dict[k] for k in keys > > Hmmm. That could even be handy for a list/tuple: > > offset, name = get_values_from(a_list, [1, 7]) > > ;) > > ?getitems(obj, subscripts) ? We almost have this: from operator import itemgetter itemgetter(1,7)(a_list) ?-- Koos? > At any rate, the return type should be an iterator. > > -- > ~Ethan~ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Fri May 27 03:20:33 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 27 May 2016 16:20:33 +0900 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <57478250.6000707@canterbury.ac.nz> References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <87shx4d6ip.fsf@thinkpad.rath.org> <57478250.6000707@canterbury.ac.nz> Message-ID: <22343.62785.209111.270692@turnbull.sk.tsukuba.ac.jp> Greg Ewing writes: > Trying to come up with some real-looking examples: > > switch obj: > case Point(?x, ?y): > print("Point at", x, y) > case Line(start = ?p1, end = ?p2, width = ?w): > print("Line from", p1, "to", p2, "with width", w) > case Intersection(?obj1, ?obj2): > print("Intersection of", obj1, "and", obj2) Suppose I want to destructure the Points determining the Lines determining an Intersection when I print an Intersection, say: print("""Intersection of Line from ({1},{2}) to ({3},{4}) with width {5} and Line from ({6},{7}) to ({8},{9}) with width {10} """.format(obj.obj1.p1.x, obj.obj1.p1.y, obj.obj1.p2.x, obj.obj1.p2.y, obj.obj1.w, obj.obj2.p1.x, obj.obj2.p1.y, obj.obj2.p2.x, obj.obj2.p2.y, obj.obj2.w)) What would the full destructuring bind of an Intersection look like? (I recognize that in your use case, obj1 might be a Circle. Let's not worry about that yet, unless it's easy. I also recognize that most likely you intended this routine to be called recursively, but let's assume that isn't so: it is the nested case where I would really like destructuring. If it must be done recursively, I wouldn't care if I had to write code like that below.) If I can't do that, I won't really get too upset about using if isinstance(obj, Point): print("Point at", obj.x, obj.y) elif isinstance(obj, Line): print("Line from", obj.p1, "to", obj.p2, "with width", obj.w) elif isinstance(obj, Intersection): print("Intersection of", obj.obj1, "and", obj.obj2) instead of the switch ... case syntax. From steve.dower at python.org Fri May 27 09:43:35 2016 From: steve.dower at python.org (Steve Dower) Date: Fri, 27 May 2016 06:43:35 -0700 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> <5747D3A5.9050504@stoneleaf.us> Message-ID: Neat! My bikeshed is coloured `dict.getmany(*keys, default=None)`, and while returning a namedtuple might be cool, an iterator is probably the way to go. But +1 regardless of colour. Top-posted from my Windows Phone -----Original Message----- From: "Koos Zevenhoven" Sent: ?5/?26/?2016 22:27 To: "Ethan Furman" Cc: "python-ideas" Subject: Re: [Python-ideas] Enhancing dict.values On Fri, May 27, 2016 at 7:57 AM, Ethan Furman wrote: On 05/26/2016 09:26 PM, Guido van Rossum wrote: On Thursday, May 26, 2016, Steven D'Apranowrote: On Thu, May 26, 2016 at 11:28:25PM +0100, Nathan Schneider wrote: I think this is important enough to get a change in subject line, lest it be lost in the dict unpacking thread. Instead of special syntax, what if dict.values() returned a tuple when given keys as arguments: partner_id, product_id, ship_to, product_ids = my_dict.values( 'partner_id', 'product_id', 'ship_to', 'product_ids') That avoids repeating the dict variable, at least. And as there is dict.update(), I don't see the need for a new syntax for assigning to multiple keys. I like this idea. I think it beats the status quo: +1 Interesting. It should probably have a different name. What type should it return? Iterator? Sequence? It can't really be a ValuesView because that class is just a view on the hash table. Even though you technically *could* combine this functionality into values(), I don't think it would be helpful to do so -- if only because of the surprising edge case where if you were to pass it a list of keys to extract using **args, if the list is empty, values() would default to its original behavior or returning all keys, in hash table order. Good point. The time bomb would be even worse if sometimes the dict had the same number of elements as were being asked for, as then it would be an intermittent problem. However, if we make a new method we could just as easily make a new function: def get_values_from(a_dict, keys=()): if not keys: raise an_error yield a_dict[k] for k in keys Hmmm. That could even be handy for a list/tuple: offset, name = get_values_from(a_list, [1, 7]) ;) ?getitems(obj, subscripts) ? We almost have this: from operator import itemgetter itemgetter(1,7)(a_list) ?-- Koos? At any rate, the return type should be an iterator. -- ~Ethan~ _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.selik at gmail.com Fri May 27 10:55:37 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 27 May 2016 14:55:37 +0000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <22343.62785.209111.270692@turnbull.sk.tsukuba.ac.jp> References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <87shx4d6ip.fsf@thinkpad.rath.org> <57478250.6000707@canterbury.ac.nz> <22343.62785.209111.270692@turnbull.sk.tsukuba.ac.jp> Message-ID: On Fri, May 27, 2016, 3:21 AM Stephen J. Turnbull wrote: > it is the nested case where I would really like destructuring. After more than 100 notes in the dict unpacking thread, I've come to the same conclusion. Python already has great tools via tuple unpacking, comprehensions, sets, and the various well-thought methods. Destructuring even a shallowly nested dictionary needs either ugly nested loops or helper functions. -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.selik at gmail.com Fri May 27 11:02:15 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 27 May 2016 15:02:15 +0000 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: <20160527033930.GF12028@ando.pearwood.info> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> Message-ID: On Thu, May 26, 2016 at 11:39 PM Steven D'Aprano wrote: > On Thu, May 26, 2016 at 11:28:25PM +0100, Nathan Schneider wrote: > > > Instead of special syntax, what if dict.values() returned a tuple when > > given keys as arguments: > > > > partner_id, product_id, ship_to, product_ids = my_dict.values( > > 'partner_id', 'product_id', 'ship_to', 'product_ids') > > > I like this idea. I think it beats the status quo: > Isn't this the status quo? a, b, c = [mapping[k] for k in ('a', 'b', 'c')] That was already someone's argument that a special dict unpacking syntax is unnecessary. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Fri May 27 11:09:51 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Fri, 27 May 2016 08:09:51 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: (Paul Moore's message of "Thu, 26 May 2016 20:25:19 +0100") References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> Message-ID: <87d1o77d1c.fsf@thinkpad.rath.org> On May 26 2016, Paul Moore wrote: > On 26 May 2016 at 18:55, Ethan Furman wrote: >> With the simple syntax that I could live with, a real example could be: >> >> {active_id, active_ids, active_model} = context >> >> or >> >> {partner_id, product_id, ship_to, product_ids} = values > > The behaviour of using the names of the variables from the LHS to > introspect the value on the RHS is, to me, extremely magical and > unlike anything I've seen in any other language. I don't think it sits > well in Python, even though it is certainly a very readable idiom for > the sort of unpacking we're talking about here. Very true. But as someone else already said (I can't find the email right now), we have a different construct that everyone is familiar with and that's easily adapted for this situation: from dict context import active_id, active_ids, active_model or more general: "from dict" "import" Everyone knows that "from .. import .." modifies the local namespace. We just have to extend it to work not just on modules, but also on dictionaries. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From Nikolaus at rath.org Fri May 27 11:14:25 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Fri, 27 May 2016 08:14:25 -0700 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <57478250.6000707@canterbury.ac.nz> (Greg Ewing's message of "Fri, 27 May 2016 11:10:08 +1200") References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <87shx4d6ip.fsf@thinkpad.rath.org> <57478250.6000707@canterbury.ac.nz> Message-ID: <878tyv7ctq.fsf@thinkpad.rath.org> On May 27 2016, Greg Ewing wrote: > Nikolaus Rath wrote: >> I think its much >> better to embed the target variables in the pattern - we just need a way >> to mark them as such. >> >> maybe we could use $? > > I'd like to suggest using '?' instead. It looks more pattern- > matchy to me. > > Trying to come up with some real-looking examples: > > switch obj: > case Point(?x, ?y): > print("Point at", x, y) > case Line(start = ?p1, end = ?p2, width = ?w): > print("Line from", p1, "to", p2, "with width", w) > case Intersection(?obj1, ?obj2): > print("Intersection of", obj1, "and", obj2) I think that's a little confusing (but I don't have a better idea). Assuming you have this class: class Point: def __init__(self, y, x): self.y = y self.x = x would your first example assign x == obj.x or x = obj.y? In ML-like languages these constructs typically match against the "constructor", so you would expect x == obj.y. But I think the mechanics of Python would force a translation to x == obj.x. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From michael.selik at gmail.com Fri May 27 11:19:29 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 27 May 2016 15:19:29 +0000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <8d2131cd-ee5a-0518-81df-0aad73638746@btinternet.com> References: <20160526015606.GB12028@ando.pearwood.info> <574703C2.7070409@gmail.com> <20160526175100.GE12028@ando.pearwood.info> <8d2131cd-ee5a-0518-81df-0aad73638746@btinternet.com> Message-ID: On Thu, May 26, 2016, 10:46 PM Rob Cliffe wrote: > On 26/05/2016 18:51, Steven D'Aprano wrote: > > On Thu, May 26, 2016 at 04:10:10PM +0200, Michel Desmoulin wrote: > With tuple and list unpacking > (a,b) = (1,2) > the structure of the LHS and RHS mirror each other, making the meaning > intuitive/obvious. > Doing the same thing for dicts: > { 'a' : a, 'b' : b } = { 'a' : 1, 'b' : 2 } > makes the LHS too verbose to be very useful IMO. But is it intuitive/obvious? Sounds like you think so. > All the examples so > far can be done in other, arguably better, ways. > Not for nested destructuring. Maybe my example got lost in the shuffle. No one replied to it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.ware+pyideas at gmail.com Fri May 27 11:27:23 2016 From: zachary.ware+pyideas at gmail.com (Zachary Ware) Date: Fri, 27 May 2016 10:27:23 -0500 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <87d1o77d1c.fsf@thinkpad.rath.org> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> Message-ID: On Fri, May 27, 2016 at 10:09 AM, Nikolaus Rath wrote: > Everyone knows that "from .. import .." modifies the local namespace. We > just have to extend it to work not just on modules, but also on > dictionaries. Here's a crazy thought that might be best dismissed out of hand: what about extending 'from name import other_names' to accept any object for ? First try to get values via __getitem__() (possibly only for dict/dict subclasses?), next try getattr(), finally try to import the module and pull values from it as per usual. Pros: - solves dict unpacking - solves attribute unpacking Cons: - it's weird - has the potential to be confusing: "from name import other_name" will do different things depending on whether "name" is already bound in current scope - we'd suddenly have 'from ... import ...' statements littered throughout code, rather than nearly always all at the top of the module - dilutes the meaning of "importing" The two pros are nice, but I'm not sure they beat the four cons. FTR, I've not seen anything else in this thread that excites me, but I have only been skimming. -- Zach From michael.selik at gmail.com Fri May 27 11:32:32 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 27 May 2016 15:32:32 +0000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> Message-ID: On Fri, May 27, 2016 at 11:28 AM Zachary Ware < zachary.ware+pyideas at gmail.com> wrote: > Here's a crazy thought that might be best dismissed out of hand: what > about extending 'from name import other_names' to accept any object > for ? First try to get values via __getitem__() (possibly only > for dict/dict subclasses?), next try getattr(), finally try to import > the module and pull values from it as per usual. > > Pros: > - solves dict unpacking > Would it solve nested dict unpacking? -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.ware+pyideas at gmail.com Fri May 27 11:37:27 2016 From: zachary.ware+pyideas at gmail.com (Zachary Ware) Date: Fri, 27 May 2016 10:37:27 -0500 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> Message-ID: On Fri, May 27, 2016 at 10:32 AM, Michael Selik wrote: > > > On Fri, May 27, 2016 at 11:28 AM Zachary Ware > wrote: >> >> Here's a crazy thought that might be best dismissed out of hand: what >> about extending 'from name import other_names' to accept any object >> for ? First try to get values via __getitem__() (possibly only >> for dict/dict subclasses?), next try getattr(), finally try to import >> the module and pull values from it as per usual. >> >> Pros: >> - solves dict unpacking > > > Would it solve nested dict unpacking? How do you mean? Replacing `some_name = some_dict['some_key']['some_name']` with `from some_dict['some_key'] import some_name`? Sure, why not? :) -- Zach From michael.selik at gmail.com Fri May 27 11:41:41 2016 From: michael.selik at gmail.com (Michael Selik) Date: Fri, 27 May 2016 15:41:41 +0000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> Message-ID: On Fri, May 27, 2016 at 11:38 AM Zachary Ware < zachary.ware+pyideas at gmail.com> wrote: > On Fri, May 27, 2016 at 10:32 AM, Michael Selik > wrote: > > > > > > On Fri, May 27, 2016 at 11:28 AM Zachary Ware > > wrote: > >> > >> Here's a crazy thought that might be best dismissed out of hand: what > >> about extending 'from name import other_names' to accept any object > >> for ? First try to get values via __getitem__() (possibly only > >> for dict/dict subclasses?), next try getattr(), finally try to import > >> the module and pull values from it as per usual. > >> > >> Pros: > >> - solves dict unpacking > > > > > > Would it solve nested dict unpacking? > > How do you mean? Replacing `some_name = > some_dict['some_key']['some_name']` with `from some_dict['some_key'] > import some_name`? > No, more like py> {'name': name, 'location': {0: row, 1: col} = mapping py> name, row, col ('Mike', 3, 5) -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.ware+pyideas at gmail.com Fri May 27 11:44:46 2016 From: zachary.ware+pyideas at gmail.com (Zachary Ware) Date: Fri, 27 May 2016 10:44:46 -0500 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> Message-ID: On Fri, May 27, 2016 at 10:41 AM, Michael Selik wrote: > On Fri, May 27, 2016 at 11:38 AM Zachary Ware > wrote: >> On Fri, May 27, 2016 at 10:32 AM, Michael Selik >> wrote: >> >> - solves dict unpacking >> > >> > Would it solve nested dict unpacking? >> >> How do you mean? Replacing `some_name = >> some_dict['some_key']['some_name']` with `from some_dict['some_key'] >> import some_name`? > > No, more like > py> {'name': name, 'location': {0: row, 1: col} = mapping > py> name, row, col > ('Mike', 3, 5) That reads as gibberish to me (and also as SyntaxError, you're missing a '}'), so probably not :) -- Zach From ethan at stoneleaf.us Fri May 27 12:12:57 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 27 May 2016 09:12:57 -0700 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> Message-ID: <57487209.6060901@stoneleaf.us> On 05/27/2016 08:02 AM, Michael Selik wrote: > On Thu, May 26, 2016 at 11:39 PM Steven D'Aprano wrote: >> On Thu, May 26, 2016 at 11:28:25PM +0100, Nathan Schneider wrote: >>> Instead of special syntax, what if dict.values() returned a tuple >>> when given keys as arguments: >>> >>> partner_id, product_id, ship_to, product_ids = my_dict.values( >>> 'partner_id', 'product_id', 'ship_to', 'product_ids') >> >> I like this idea. I think it beats the status quo: > > Isn't this the status quo? > a, b, c = [mapping[k] for k in ('a', 'b', 'c')] Yes. > That was already someone's argument that a special dict unpacking syntax > is unnecessary. It looks really cool at first blush, but when substituting real names in for the place-holders a, b, and c it gets ugly fast. -- ~Ethan~ From steve at pearwood.info Fri May 27 12:17:40 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 28 May 2016 02:17:40 +1000 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: References: <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> <5747D3A5.9050504@stoneleaf.us> Message-ID: <20160527161739.GG12028@ando.pearwood.info> On Fri, May 27, 2016 at 06:43:35AM -0700, Steve Dower wrote: > Neat! > > My bikeshed is coloured `dict.getmany(*keys, default=None)`, "getmany" doesn't tell you, many of what? > and while > returning a namedtuple might be cool, an iterator is probably the way > to go. The disadvantage of a namedtuple is that every invocation would create a new class, which is then used once and once only for a singleton instance. Could get very expensive. I don't think an iterator would be needed. The motivating use-case is for sequence unpacking: # apologies for breaking my own rule about realistic names fee, fi, fo, fum = mydict.getmany('fee', 'fi', 'fo', 'fum') so I don't think the lazy aspect of an iterator is useful. It's going to be consumed eagerly, and immediately. I think a regular tuple is better. That also matches the behaviour of itemgetter: py> d = {'fe': 1, 'fi': 2, 'fo': 3, 'fum': 4} py> from operator import itemgetter py> f = itemgetter('fe', 'fi', 'fo', 'fum') py> f(d) (1, 2, 3, 4) So the core functionality already exists, it's just hidden away in the operator module. (Guido's time machine strikes again.) Open questions: - Is it worth making this a dict method? +1 from me. - Name? "getvalues"? - Any other functionality? Possibly a keyword-only "default" argument: mydict.getvalues(*keys, default=None) On more shaky ground, how about a "pop" argument? mydict.getvalues(*keys, pop=True) will delete the keys as well as return the values. Use-case: methods (particularly __init__ or __new__) which take extra keyword args which need to be popped before calling super. def __init__(self, **kwargs): colour, height = kwargs.getvalues('colour', 'height', pop=True) super().__init__(**kwargs) self.process(colour, height) -- Steve From chris.barker at noaa.gov Fri May 27 12:36:57 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 27 May 2016 09:36:57 -0700 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: <20160527161739.GG12028@ando.pearwood.info> References: <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> <5747D3A5.9050504@stoneleaf.us> <20160527161739.GG12028@ando.pearwood.info> Message-ID: On Fri, May 27, 2016 at 9:17 AM, Steven D'Aprano wrote: > # apologies for breaking my own rule about realistic names > fee, fi, fo, fum = mydict.getmany('fee', 'fi', 'fo', 'fum') > so how again is the much better than: > Isn't this the status quo? > a, b, c = [mapping[k] for k in ('a', 'b', 'c')] the objection to that was that it gets ugly when you've got longer, more realistic names (and maybe more of them.) fee, fi, fo, fum = [mapping[k] for k in ('fee', 'fi', 'fo', 'fum')] only a little more typing, and in both cases, you need to specify the names twice. (of course, they don't need to be the same names...) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri May 27 12:42:44 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 27 May 2016 17:42:44 +0100 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: <20160527161739.GG12028@ando.pearwood.info> References: <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> <5747D3A5.9050504@stoneleaf.us> <20160527161739.GG12028@ando.pearwood.info> Message-ID: On 27 May 2016 at 17:17, Steven D'Aprano wrote: > Open questions: > > - Is it worth making this a dict method? +1 from me. Quite possibly, but maybe a 3rd-party function implementing this would be a worthwhile test of how useful it is in practice (although conceded, a big part of its usefulness would be that it doesn't need a dependency). > - Any other functionality? Nested unpacking: location = = { 'name': 'The north pole', 'position': { 'x': 0, 'y': 0 }} location_name, x, y = location.getvalues('name', 'position.x', 'position.y') Paul. From ethan at stoneleaf.us Fri May 27 12:51:07 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 27 May 2016 09:51:07 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <87d1o77d1c.fsf@thinkpad.rath.org> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> Message-ID: <57487AFB.8020800@stoneleaf.us> On 05/27/2016 08:09 AM, Nikolaus Rath wrote: > On May 26 2016, Paul Moore wrote: >> On 26 May 2016 at 18:55, Ethan Furman wrote: >>> With the simple syntax that I could live with, a real example could be: >>> >>> {active_id, active_ids, active_model} = context >>> >>> or >>> >>> {partner_id, product_id, ship_to, product_ids} = values >> >> The behaviour of using the names of the variables from the LHS to >> introspect the value on the RHS is, to me, extremely magical and >> unlike anything I've seen in any other language. I don't think it sits >> well in Python, even though it is certainly a very readable idiom for >> the sort of unpacking we're talking about here. > > Very true. But as someone else already said (I can't find the email > right now), we have a different construct that everyone is familiar with > and that's easily adapted for this situation: > > from dict context import active_id, active_ids, active_model > > or more general: > > "from dict" "import" > > Everyone knows that "from .. import .." modifies the local namespace. We > just have to extend it to work not just on modules, but also on > dictionaries. -1 import works with modules. Having it work with other things would muddy the concept, plus make module/object naming conflicts an even bigger hassle. -- ~Ethan~ From mal at egenix.com Fri May 27 12:52:15 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 27 May 2016 18:52:15 +0200 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: <57487209.6060901@stoneleaf.us> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> <57487209.6060901@stoneleaf.us> Message-ID: <57487B3F.7030207@egenix.com> On 27.05.2016 18:12, Ethan Furman wrote: > On 05/27/2016 08:02 AM, Michael Selik wrote: >> On Thu, May 26, 2016 at 11:39 PM Steven D'Aprano wrote: >>> On Thu, May 26, 2016 at 11:28:25PM +0100, Nathan Schneider wrote: > >>>> Instead of special syntax, what if dict.values() returned a tuple >>>> when given keys as arguments: >>>> >>>> partner_id, product_id, ship_to, product_ids = my_dict.values( >>>> 'partner_id', 'product_id', 'ship_to', 'product_ids') >>> >>> I like this idea. I think it beats the status quo: >> >> Isn't this the status quo? >> a, b, c = [mapping[k] for k in ('a', 'b', 'c')] > > Yes. Uhm, what about... from operator import itemgetter a, b, c = itemgetter('a', 'b', 'c')(mapping) (itemgetter is not the best name, but it does get the job done) >> That was already someone's argument that a special dict unpacking syntax >> is unnecessary. > > It looks really cool at first blush, but when substituting real names in > for the place-holders a, b, and c it gets ugly fast. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 27 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From ethan at stoneleaf.us Fri May 27 12:59:38 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 27 May 2016 09:59:38 -0700 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: <57487B3F.7030207@egenix.com> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> <57487209.6060901@stoneleaf.us> <57487B3F.7030207@egenix.com> Message-ID: <57487CFA.2070601@stoneleaf.us> On 05/27/2016 09:52 AM, M.-A. Lemburg wrote: > On 27.05.2016 18:12, Ethan Furman wrote: >> On 05/27/2016 08:02 AM, Michael Selik wrote: >>> On Thu, May 26, 2016 at 11:39 PM Steven D'Aprano wrote: >>>> On Thu, May 26, 2016 at 11:28:25PM +0100, Nathan Schneider wrote: >> >>>>> Instead of special syntax, what if dict.values() returned a tuple >>>>> when given keys as arguments: >>>>> >>>>> partner_id, product_id, ship_to, product_ids = my_dict.values( >>>>> 'partner_id', 'product_id', 'ship_to', 'product_ids') >>>> >>>> I like this idea. I think it beats the status quo: >>> >>> Isn't this the status quo? >>> a, b, c = [mapping[k] for k in ('a', 'b', 'c')] >> >> Yes. > > Uhm, what about... > > from operator import itemgetter > > a, b, c = itemgetter('a', 'b', 'c')(mapping) I may have misspoken. The probably most used status quo would be: partner_id = values['partner_id'] state = values['state'] due_date = values['due_date'] ... So far, the only syntax I've seen that is both readable and has even a remote shot at acceptance would be: {partner_id, state, due_date, **rest} = values The consensus is that that is too magical (which I can't argue with ;). itemgetter, comprehensions, name changes, etc., are all (extremely) verbose. -- ~Ethan~ From steve.dower at python.org Fri May 27 13:17:52 2016 From: steve.dower at python.org (Steve Dower) Date: Fri, 27 May 2016 10:17:52 -0700 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: <20160527161739.GG12028@ando.pearwood.info> References: <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> <5747D3A5.9050504@stoneleaf.us> <20160527161739.GG12028@ando.pearwood.info> Message-ID: ""getmany" doesn't tell you, many of what?" Eh, neither does "get", but like I said, I like the idea regardless of colour. Top-posted from my Windows Phone -----Original Message----- From: "Steven D'Aprano" Sent: ?5/?27/?2016 9:24 To: "python-ideas at python.org" Subject: Re: [Python-ideas] Enhancing dict.values On Fri, May 27, 2016 at 06:43:35AM -0700, Steve Dower wrote: > Neat! > > My bikeshed is coloured `dict.getmany(*keys, default=None)`, "getmany" doesn't tell you, many of what? > and while > returning a namedtuple might be cool, an iterator is probably the way > to go. The disadvantage of a namedtuple is that every invocation would create a new class, which is then used once and once only for a singleton instance. Could get very expensive. I don't think an iterator would be needed. The motivating use-case is for sequence unpacking: # apologies for breaking my own rule about realistic names fee, fi, fo, fum = mydict.getmany('fee', 'fi', 'fo', 'fum') so I don't think the lazy aspect of an iterator is useful. It's going to be consumed eagerly, and immediately. I think a regular tuple is better. That also matches the behaviour of itemgetter: py> d = {'fe': 1, 'fi': 2, 'fo': 3, 'fum': 4} py> from operator import itemgetter py> f = itemgetter('fe', 'fi', 'fo', 'fum') py> f(d) (1, 2, 3, 4) So the core functionality already exists, it's just hidden away in the operator module. (Guido's time machine strikes again.) Open questions: - Is it worth making this a dict method? +1 from me. - Name? "getvalues"? - Any other functionality? Possibly a keyword-only "default" argument: mydict.getvalues(*keys, default=None) On more shaky ground, how about a "pop" argument? mydict.getvalues(*keys, pop=True) will delete the keys as well as return the values. Use-case: methods (particularly __init__ or __new__) which take extra keyword args which need to be popped before calling super. def __init__(self, **kwargs): colour, height = kwargs.getvalues('colour', 'height', pop=True) super().__init__(**kwargs) self.process(colour, height) -- Steve _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri May 27 13:42:52 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 28 May 2016 03:42:52 +1000 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: References: <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> <5747D3A5.9050504@stoneleaf.us> <20160527161739.GG12028@ando.pearwood.info> Message-ID: <20160527174252.GI12028@ando.pearwood.info> On Fri, May 27, 2016 at 05:42:44PM +0100, Paul Moore wrote: > > - Any other functionality? > > Nested unpacking: > > location = = { 'name': 'The north pole', 'position': { 'x': 0, 'y': 0 }} > location_name, x, y = location.getvalues('name', 'position.x', 'position.y') How do you distinguish between a key 'position.x' and a key 'position' with a dict with a key 'x'? Nested unpacking seems too Javascripty for my liking. -- Steve From guido at python.org Fri May 27 13:50:48 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 27 May 2016 10:50:48 -0700 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: References: <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> <5747D3A5.9050504@stoneleaf.us> <20160527161739.GG12028@ando.pearwood.info> Message-ID: On Fri, May 27, 2016 at 9:36 AM, Chris Barker wrote: > > On Fri, May 27, 2016 at 9:17 AM, Steven D'Aprano > wrote: >> >> # apologies for breaking my own rule about realistic names >> fee, fi, fo, fum = mydict.getmany('fee', 'fi', 'fo', 'fum') > > so how again is the much better than: > >> Isn't this the status quo? >> a, b, c = [mapping[k] for k in ('a', 'b', 'c')] > > the objection to that was that it gets ugly when you've got longer, more > realistic names (and maybe more of them.) > > fee, fi, fo, fum = [mapping[k] for k in ('fee', 'fi', 'fo', 'fum')] > > only a little more typing, and in both cases, you need to specify the names > twice. > > (of course, they don't need to be the same names...) Honestly I don't like the comprehension version much; it reeks of cleverness. The advantage of the getnamy() spelling is that you can Google for it more easily. (But I don't know how it would handle nested unpacking, which some have said is an important use case to warrant adding complexity to the language.) If this was a Dropbox code review I'd say rewrite it as fee = mapping['fee'] fi = mapping['fi'] etc. -- then nobody will have any trouble understanding what it does. -- --Guido van Rossum (python.org/~guido) From brenbarn at brenbarn.net Fri May 27 13:58:40 2016 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Fri, 27 May 2016 10:58:40 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <87d1o77d1c.fsf@thinkpad.rath.org> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> Message-ID: <57488AD0.4090106@brenbarn.net> On 2016-05-27 08:09, Nikolaus Rath wrote: > Very true. But as someone else already said (I can't find the email > right now), we have a different construct that everyone is familiar with > and that's easily adapted for this situation: > > from dict context import active_id, active_ids, active_model > > or more general: > > "from dict" "import" > > Everyone knows that "from .. import .." modifies the local namespace. We > just have to extend it to work not just on modules, but also on > dictionaries. One problem with this is that currently "from" doesn't operate on names or objects. It operates on module paths. You can't do this: import somepackage as othername from othername import something (You'll get "No module named othername".) If we change this, it will be confusing, because when you see "from blah import stuff" you won't know offhand whether it's going to just read a value from a dict or go searching the filesystem, which are quite different operations. It might be possible, though, to leverage "from" without also having to use "import": from somedict get thiskey, thatkey I'm not sure I actually like this idea, but I like it more than repurposing "import". "import" has a quite idiosyncratic meaning in Python that has to do with looking for files on disk and running them; overloading this with totally file-local operations like getting keys from dicts seems too wild to me. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From Nikolaus at rath.org Fri May 27 14:07:59 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Fri, 27 May 2016 11:07:59 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <57487AFB.8020800@stoneleaf.us> (Ethan Furman's message of "Fri, 27 May 2016 09:51:07 -0700") References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> Message-ID: <87vb1z5q80.fsf@thinkpad.rath.org> On May 27 2016, Ethan Furman wrote: > On 05/27/2016 08:09 AM, Nikolaus Rath wrote: >> On May 26 2016, Paul Moore wrote: >>> On 26 May 2016 at 18:55, Ethan Furman wrote: > >>>> With the simple syntax that I could live with, a real example could be: >>>> >>>> {active_id, active_ids, active_model} = context >>>> >>>> or >>>> >>>> {partner_id, product_id, ship_to, product_ids} = values >>> >>> The behaviour of using the names of the variables from the LHS to >>> introspect the value on the RHS is, to me, extremely magical and >>> unlike anything I've seen in any other language. I don't think it sits >>> well in Python, even though it is certainly a very readable idiom for >>> the sort of unpacking we're talking about here. >> >> Very true. But as someone else already said (I can't find the email >> right now), we have a different construct that everyone is familiar with >> and that's easily adapted for this situation: >> >> from dict context import active_id, active_ids, active_model >> >> or more general: >> >> "from dict" "import" >> >> Everyone knows that "from .. import .." modifies the local namespace. We >> just have to extend it to work not just on modules, but also on >> dictionaries. > > -1 > > import works with modules. You don't think of it as "*import*ing something from another namespace into the local namespace"? That's the first thing that I associate with it. > Having it work with other things would > muddy the concept, plus make module/object naming conflicts an even > bigger hassle. You did see that I proposed "from dict <> import ..", instead of "from <> import ..", right? The latter would continue to work only for modules. The form would be new syntax and only work for dicts. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From gvanrossum at gmail.com Fri May 27 14:13:44 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Fri, 27 May 2016 11:13:44 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <87vb1z5q80.fsf@thinkpad.rath.org> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> Message-ID: I am against this. Try something else please. Keep import for modules. --Guido (mobile) On May 27, 2016 11:08 AM, "Nikolaus Rath" wrote: > On May 27 2016, Ethan Furman < > ethan-gcWI5d7PMXnvaiG9KC9N7Q at public.gmane.org> wrote: > > On 05/27/2016 08:09 AM, Nikolaus Rath wrote: > >> On May 26 2016, Paul Moore wrote: > >>> On 26 May 2016 at 18:55, Ethan Furman wrote: > > > >>>> With the simple syntax that I could live with, a real example could > be: > >>>> > >>>> {active_id, active_ids, active_model} = context > >>>> > >>>> or > >>>> > >>>> {partner_id, product_id, ship_to, product_ids} = values > >>> > >>> The behaviour of using the names of the variables from the LHS to > >>> introspect the value on the RHS is, to me, extremely magical and > >>> unlike anything I've seen in any other language. I don't think it sits > >>> well in Python, even though it is certainly a very readable idiom for > >>> the sort of unpacking we're talking about here. > >> > >> Very true. But as someone else already said (I can't find the email > >> right now), we have a different construct that everyone is familiar with > >> and that's easily adapted for this situation: > >> > >> from dict context import active_id, active_ids, active_model > >> > >> or more general: > >> > >> "from dict" "import" > >> > >> Everyone knows that "from .. import .." modifies the local namespace. We > >> just have to extend it to work not just on modules, but also on > >> dictionaries. > > > > -1 > > > > import works with modules. > > You don't think of it as "*import*ing something from another namespace > into the local namespace"? That's the first thing that I associate with > it. > > > Having it work with other things would > > muddy the concept, plus make module/object naming conflicts an even > > bigger hassle. > > You did see that I proposed "from dict <> import ..", instead of "from > <> import ..", right? The latter would continue to work only for > modules. The form would be new syntax and only work for dicts. > > > Best, > -Nikolaus > > -- > GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F > Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F > > ?Time flies like an arrow, fruit flies like a Banana.? > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri May 27 14:28:15 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 27 May 2016 19:28:15 +0100 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: <20160527174252.GI12028@ando.pearwood.info> References: <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> <5747D3A5.9050504@stoneleaf.us> <20160527161739.GG12028@ando.pearwood.info> <20160527174252.GI12028@ando.pearwood.info> Message-ID: On 27 May 2016 at 18:42, Steven D'Aprano wrote: > On Fri, May 27, 2016 at 05:42:44PM +0100, Paul Moore wrote: > >> > - Any other functionality? >> >> Nested unpacking: >> >> location = = { 'name': 'The north pole', 'position': { 'x': 0, 'y': 0 }} >> location_name, x, y = location.getvalues('name', 'position.x', 'position.y') > > How do you distinguish between a key 'position.x' and a key 'position' > with a dict with a key 'x'? > > Nested unpacking seems too Javascripty for my liking. Good point. The problem is that *without* handling nested unpacking (and in particular, the error checking needed - what if location['position'] isn't a dict? I'd like a uniform error I can trap or report on) I don't see any practical advantage over location_name = location['name'] x = location['position']['x'] y = location['position']['y'] So let's turn the question round. What advantages does the getvalues() proposal have over the above sequence of assignments? I'm genuinely no longer sure. Paul From Nikolaus at rath.org Fri May 27 14:29:16 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Fri, 27 May 2016 11:29:16 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <57488AD0.4090106@brenbarn.net> (Brendan Barnwell's message of "Fri, 27 May 2016 10:58:40 -0700") References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57488AD0.4090106@brenbarn.net> Message-ID: <87shx35p8j.fsf@thinkpad.rath.org> On May 27 2016, Brendan Barnwell wrote: > On 2016-05-27 08:09, Nikolaus Rath wrote: >> Very true. But as someone else already said (I can't find the email >> right now), we have a different construct that everyone is familiar with >> and that's easily adapted for this situation: >> >> from dict context import active_id, active_ids, active_model >> >> or more general: >> >> "from dict" "import" >> >> Everyone knows that "from .. import .." modifies the local namespace. We >> just have to extend it to work not just on modules, but also on >> dictionaries. > > One problem with this is that currently "from" doesn't operate > on names or objects. It operates on module paths. You can't do this: > > import somepackage as othername > from othername import something > > (You'll get "No module named othername".) > > If we change this I'm not proposing to change this. I'm proposing to add a new "from dict import " statement that provides the new functionality. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From storchaka at gmail.com Fri May 27 15:04:34 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 27 May 2016 22:04:34 +0300 Subject: [Python-ideas] Add support for version objects Message-ID: I propose to add support for version objects in the stdlib. Version object is a namedtuple-like object with special attrbutes major, minor, etc for representing semantic version in the form used by most open source (and not only) software. The sys module already contains two version objects: sys.version_info and the result of sys.getwindowsversion(). There is a support of version objects in Perl [1] and .NET [2]. There are third-party Python implementations ([3], [4], [5]). Version objects can be used for representing: 1. Version of Python itself (sys.version_info). 2. Version of OS and system libraries. 3. Static and runtime versions of wrapped libraries (zlib, expat, sqlite, curses, Tcl/Tk, libffi, libmpdec). 4. Versions of third-party modules. The benefit of version objects against plain strings is that version objects are comparable with expected rules (1.2.3 < 1.10.1). The benefit of version objects against tuples is human readable string representation and named attributes for components. [1] http://search.cpan.org/perldoc?version [2] https://msdn.microsoft.com/en-us/library/system.version%28v=vs.110%29.aspx [3] https://pypi.python.org/pypi/semantic_version [4] https://pypi.python.org/pypi/versions [3] https://pypi.python.org/pypi/version https://pypi.python.org/pypi/SemVerPy From phd at phdru.name Fri May 27 15:30:41 2016 From: phd at phdru.name (Oleg Broytman) Date: Fri, 27 May 2016 21:30:41 +0200 Subject: [Python-ideas] Add support for version objects In-Reply-To: References: Message-ID: <20160527193041.GA1010@phdru.name> On Fri, May 27, 2016 at 10:04:34PM +0300, Serhiy Storchaka wrote: > I propose to add support for version objects in the stdlib. Version object > is a namedtuple-like object with special attrbutes major, minor, etc for > representing semantic version in the form used by most open source (and not > only) software. The sys module already contains two version objects: > sys.version_info and the result of sys.getwindowsversion(). > > There is a support of version objects in Perl [1] and .NET [2]. There are > third-party Python implementations ([3], [4], [5]). > > Version objects can be used for representing: > > 1. Version of Python itself (sys.version_info). > 2. Version of OS and system libraries. > 3. Static and runtime versions of wrapped libraries (zlib, expat, sqlite, > curses, Tcl/Tk, libffi, libmpdec). > 4. Versions of third-party modules. > > The benefit of version objects against plain strings is that version objects > are comparable with expected rules (1.2.3 < 1.10.1). > > The benefit of version objects against tuples is human readable string > representation and named attributes for components. > > [1] http://search.cpan.org/perldoc?version > [2] > https://msdn.microsoft.com/en-us/library/system.version%28v=vs.110%29.aspx > [3] https://pypi.python.org/pypi/semantic_version > [4] https://pypi.python.org/pypi/versions > [3] https://pypi.python.org/pypi/version > https://pypi.python.org/pypi/SemVerPy Something like https://hg.python.org/cpython/file/tip/Lib/distutils/version.py https://hg.python.org/cpython/file/tip/Lib/distutils/versionpredicate.py ??? Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From donald at stufft.io Fri May 27 15:37:55 2016 From: donald at stufft.io (Donald Stufft) Date: Fri, 27 May 2016 15:37:55 -0400 Subject: [Python-ideas] Add support for version objects In-Reply-To: References: Message-ID: > On May 27, 2016, at 3:04 PM, Serhiy Storchaka wrote: > > I propose to add support for version objects in the stdlib. Version object is a namedtuple-like object with special attrbutes major, minor, etc for representing semantic version in the form used by most open source (and not only) software. The sys module already contains two version objects: sys.version_info and the result of sys.getwindowsversion(). If Python adds a version objection, it should not be one that implements SemVer, but one that implements PEP 440 which is what PyPI, pip, setuptools, etc all use. SemVer, while a nice idea, is too simplistic to represent the variations of versioning that the Python community uses. There?s also a pure Python implementation of it available [1]. To be honest though, I don?t see a lot of benefit to adding it to the standard library. [1] http://packaging.readthedocs.io/en/latest/version/ ? Donald Stufft From greg.ewing at canterbury.ac.nz Fri May 27 21:32:26 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 28 May 2016 13:32:26 +1200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <878tyv7ctq.fsf@thinkpad.rath.org> References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <87shx4d6ip.fsf@thinkpad.rath.org> <57478250.6000707@canterbury.ac.nz> <878tyv7ctq.fsf@thinkpad.rath.org> Message-ID: <5748F52A.9020504@canterbury.ac.nz> Nikolaus Rath wrote: > class Point: > def __init__(self, y, x): > self.y = y > self.x = x > > would your first example assign x == obj.x or x = obj.y? The intention is that positional args in the pattern correspond to positional args in the object's constructor. To make that work, there will have to be some protocol for reversing the construction of an object. It's been suggested that pickle's __getinitargs__ could be re-used for this purpose. Or we could add a new one called __unpack__ or __uninit__ or something. If the object doesn't explicitly say how to unpack itself, iteration could be used as a fallback. Since your example above doesn't do either of those, it would raise an exception when used in a match statement. -- Greg From greg.ewing at canterbury.ac.nz Fri May 27 21:37:31 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 28 May 2016 13:37:31 +1200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> Message-ID: <5748F65B.20407@canterbury.ac.nz> Zachary Ware wrote: > Here's a crazy thought that might be best dismissed out of hand: what > about extending 'from name import other_names' to accept any object > for ? > - has the potential to be confusing: "from name import other_name" > will do different things depending on whether "name" is already bound > in current scope Indeed. It would be better to have some syntactic marker to distinguish this new kind of import from a normal import. -- Greg From greg.ewing at canterbury.ac.nz Fri May 27 21:41:36 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 28 May 2016 13:41:36 +1200 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: <20160527161739.GG12028@ando.pearwood.info> References: <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> <5747D3A5.9050504@stoneleaf.us> <20160527161739.GG12028@ando.pearwood.info> Message-ID: <5748F750.6040507@canterbury.ac.nz> Steven D'Aprano wrote: > "getmany" doesn't tell you, many of what? I think the idea is that it would get the same kind of things that get() gets, i.e. items by their keys. -- Greg From brenbarn at brenbarn.net Fri May 27 21:44:17 2016 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Fri, 27 May 2016 18:44:17 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <87shx35p8j.fsf@thinkpad.rath.org> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57488AD0.4090106@brenbarn.net> <87shx35p8j.fsf@thinkpad.rath.org> Message-ID: <5748F7F1.1010603@brenbarn.net> On 2016-05-27 11:29, Nikolaus Rath wrote: > On May 27 2016, Brendan Barnwell wrote: >> On 2016-05-27 08:09, Nikolaus Rath wrote: >>> Very true. But as someone else already said (I can't find the email >>> right now), we have a different construct that everyone is familiar with >>> and that's easily adapted for this situation: >>> >>> from dict context import active_id, active_ids, active_model >>> >>> or more general: >>> >>> "from dict" "import" >>> >>> Everyone knows that "from .. import .." modifies the local namespace. We >>> just have to extend it to work not just on modules, but also on >>> dictionaries. >> >> One problem with this is that currently "from" doesn't operate >> on names or objects. It operates on module paths. You can't do this: >> >> import somepackage as othername >> from othername import something >> >> (You'll get "No module named othername".) >> >> If we change this > > > I'm not proposing to change this. I'm proposing to add a new "from dict > import " statement that provides the new functionality. Ah, I had missed that. I agree that resolves the ambiguity, but I don't care much for that way of doing it. Does the object have to be a dict? What about some other mapping type? If new syntax were to be added, it makes more sense to me to change the "verb", a la "from someobject get somename". -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From gvanrossum at gmail.com Fri May 27 21:45:41 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Fri, 27 May 2016 18:45:41 -0700 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: <5748F750.6040507@canterbury.ac.nz> References: <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> <5747D3A5.9050504@stoneleaf.us> <20160527161739.GG12028@ando.pearwood.info> <5748F750.6040507@canterbury.ac.nz> Message-ID: Also it probably plays on the dbapi convention fetchmany(). --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Fri May 27 22:13:33 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 28 May 2016 14:13:33 +1200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <22343.62785.209111.270692@turnbull.sk.tsukuba.ac.jp> References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <87shx4d6ip.fsf@thinkpad.rath.org> <57478250.6000707@canterbury.ac.nz> <22343.62785.209111.270692@turnbull.sk.tsukuba.ac.jp> Message-ID: <5748FECD.2030203@canterbury.ac.nz> Stephen J. Turnbull wrote: > Suppose I want to destructure the Points determining the Lines > determining an Intersection when I print an Intersection, For the non-Dutch at least, the obvious way to spell that would be switch obj: case Intersection( Line(Point(?x1, ?y1), Point(?x2, ?y2)), Line(Point(?x3, ?y3), Point(?x4, ?y4))): However, that would mean there was something special about expressions in the pattern not preceded by ?, i.e. things that look like function calls are really sub-patterns. That would make it difficult to use a real function call to calculate a value to match against. An alternative would be to require these kinds of sub-patterns to be marked, e.g. switch obj: case Intersection( class Line(class Point(?x1, ?y1), class Point(?x2, ?y2)), class Line(class Point(?x3, ?y3), class Point(?x4, ?y4))): For consistency, we should probably apply the same rule to the main pattern as well: switch obj: case class Intersection( class Line(class Point(?x1, ?y1), class Point(?x2, ?y2)), class Line(class Point(?x3, ?y3), class Point(?x4, ?y4))): Finally we could allow "case class" to be abbreviated to just "class": switch obj: class Intersection( class Line(class Point(?x1, ?y1), class Point(?x2, ?y2)), class Line(class Point(?x3, ?y3), class Point(?x4, ?y4))): Is that unacceptably verbose? I don't know. We could use more random punctuation instead, but that would increase the line-noise factor. -- Greg From robertc at robertcollins.net Fri May 27 22:28:13 2016 From: robertc at robertcollins.net (Robert Collins) Date: Sat, 28 May 2016 14:28:13 +1200 Subject: [Python-ideas] Add support for version objects In-Reply-To: References: Message-ID: On 28 May 2016 at 07:37, Donald Stufft wrote: > >> On May 27, 2016, at 3:04 PM, Serhiy Storchaka wrote: >> >> I propose to add support for version objects in the stdlib. Version object is a namedtuple-like object with special attrbutes major, minor, etc for representing semantic version in the form used by most open source (and not only) software. The sys module already contains two version objects: sys.version_info and the result of sys.getwindowsversion(). > > > If Python adds a version objection, it should not be one that implements SemVer, but one that implements PEP 440 which is what PyPI, pip, setuptools, etc all use. SemVer, while a nice idea, is too simplistic to represent the variations of versioning that the Python community uses. > > There?s also a pure Python implementation of it available [1]. > > To be honest though, I don?t see a lot of benefit to adding it to the standard library. Same. Further, I don't see any need for the stdlib itself to use it - so its not being drawn in as a dependency, or do you have something in mind Serhiy? -Rob From ethan at stoneleaf.us Fri May 27 22:34:41 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 27 May 2016 19:34:41 -0700 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <5748F52A.9020504@canterbury.ac.nz> References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <87shx4d6ip.fsf@thinkpad.rath.org> <57478250.6000707@canterbury.ac.nz> <878tyv7ctq.fsf@thinkpad.rath.org> <5748F52A.9020504@canterbury.ac.nz> Message-ID: <574903C1.2010200@stoneleaf.us> On 05/27/2016 06:32 PM, Greg Ewing wrote: > Nikolaus Rath wrote: > >> class Point: >> def __init__(self, y, x): >> self.y = y >> self.x = x >> >> would your first example assign x == obj.x or x = obj.y? > > The intention is that positional args in the pattern > correspond to positional args in the object's constructor. > To make that work, there will have to be some protocol for > reversing the construction of an object. Meaning we can *only* match on constructor arguments? What about attributes that are not set in the constructor, are we just out of luck for those? -- ~Ethan~ From ncoghlan at gmail.com Fri May 27 23:22:53 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 28 May 2016 13:22:53 +1000 Subject: [Python-ideas] Add support for version objects In-Reply-To: References: Message-ID: On 27 May 2016 19:29, "Robert Collins" wrote: > > On 28 May 2016 at 07:37, Donald Stufft wrote: > > > >> On May 27, 2016, at 3:04 PM, Serhiy Storchaka wrote: > >> > >> I propose to add support for version objects in the stdlib. Version object is a namedtuple-like object with special attrbutes major, minor, etc for representing semantic version in the form used by most open source (and not only) software. The sys module already contains two version objects: sys.version_info and the result of sys.getwindowsversion(). > > > > > > If Python adds a version objection, it should not be one that implements SemVer, but one that implements PEP 440 which is what PyPI, pip, setuptools, etc all use. SemVer, while a nice idea, is too simplistic to represent the variations of versioning that the Python community uses. > > > > There?s also a pure Python implementation of it available [1]. > > > > To be honest though, I don?t see a lot of benefit to adding it to the standard library. > > Same. Further, I don't see any need for the stdlib itself to use it - > so its not being drawn in as a dependency, or do you have something in > mind Serhiy? The main advantage I'd see to stdlib inclusion is providing "one obvious way to do it" - it isn't immediately obvious to a newcomer that PEP 440 and the implementation in packaging are the preferred approach for SemVer-like version numbers in Python projects. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat May 28 00:38:58 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 27 May 2016 21:38:58 -0700 Subject: [Python-ideas] Add support for version objects In-Reply-To: References: Message-ID: <574920E2.6040600@stoneleaf.us> On 05/27/2016 08:22 PM, Nick Coghlan wrote: > The main advantage I'd see to stdlib inclusion is providing "one obvious > way to do it" - it isn't immediately obvious to a newcomer that PEP 440 > and the implementation in packaging are the preferred approach for > SemVer-like version numbers in Python projects. Yup, I'll second that! -- ~Ethan~ From michael.selik at gmail.com Sat May 28 03:22:02 2016 From: michael.selik at gmail.com (Michael Selik) Date: Sat, 28 May 2016 07:22:02 +0000 Subject: [Python-ideas] pattern matching proof-of-concept Message-ID: My original dict unpacking proposal was very short and lacked a motivating usage. Toy examples made my proposal look unnecessarily verbose and suggested obvious alternatives with easy current syntax. Nested/recursive unpacking is much more troublesome, especially when combined with name-binding. I wrote an example to compare my proposal with current syntax. Example usage. https://github.com/selik/destructure/blob/master/examples/fips.py Implementation. https://github.com/selik/destructure/blob/master/destructure.py The design of my module I'm least happy with is the name-binding. I extended a SimpleNamespace to create an Erlang-style distinction between bound and unbound names. Though the API is a bit awkward, now that the module is built, I'm less enthusiastic about introducing new syntax. Funny how that works. I haven't yet decided how to add post-binding guards to the cases. -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sat May 28 03:45:59 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 28 May 2016 10:45:59 +0300 Subject: [Python-ideas] Add support for version objects In-Reply-To: References: Message-ID: On 28.05.16 05:28, Robert Collins wrote: > On 28 May 2016 at 07:37, Donald Stufft wrote: >> To be honest though, I don?t see a lot of benefit to adding it to the standard library. > > Same. Further, I don't see any need for the stdlib itself to use it - > so its not being drawn in as a dependency, or do you have something in > mind Serhiy? My use case is the need of testing versions of libraries, mainly for testing, but not only. For example Tkinter provides version in two forms: _tkinter.TK_VERSION is a string ('8.6') and tkinter.TkVersion is a decimal number (8.6). Both forms wouldn't work for Tk 8.10. We need to convert string version to a tuple of integers. In Lib/tkinter/test/support.py: tcl_version = tuple(map(int, _tkinter.TCL_VERSION.split('.'))) But two digits is not enough, because the behavior often depends on the pathlevel number. We need to retrieve and parse runtime version. In Lib/idlelib/macosxSupport.py: tkversion = root.tk.eval('info patchlevel') if tuple(map(int, tkversion.split('.'))) < (8, 4, 14): ... This code doesn't work with alpha, beta and rc versions. There is more correct complex code in Lib/tkinter/test/support.py: tcl = tkinter.Tcl() patchlevel = tcl.call('info', 'patchlevel') m = re.fullmatch(r'(\d+)\.(\d+)([ab.])(\d+)', patchlevel) major, minor, releaselevel, serial = m.groups() major, minor, serial = int(major), int(minor), int(serial) releaselevel = {'a':'alpha','b':'beta','.':'final'}[releaselevel] if releaselevel == 'final': _tk_patchlevel = major, minor, serial, releaselevel, 0 else: _tk_patchlevel = major, minor, 0, releaselevel, serial I think it is worth to provide structured comparable version in the tkinter module. But this is just one library. There are similar problems with other libraries. zlib tests contain following complicated code: v = (zlib.ZLIB_RUNTIME_VERSION + ".0").split(".", 4) supports_wbits_0 = int(v[0]) > 1 or int(v[0]) == 1 \ and (int(v[1]) > 2 or int(v[1]) == 2 and (int(v[2]) > 3 or int(v[2]) == 3 and int(v[3]) >= 5)) It is so complex because zlib.ZLIB_RUNTIME_VERSION can contain not only digits (like "1.2.8.1-motley"). See also complex _requires_unix_version() and requires_mac_ver() in Lib/test/support/__init__.py. There is other code that parses string representation to a tuple of numbers. While parsing code can be library specific, I think it is worth to provide standard type for representing the result. Libraries provide versions as string at Python level, but they often provide separate version components, thus we can create version object without parsing string representation. From storchaka at gmail.com Sat May 28 03:55:16 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 28 May 2016 10:55:16 +0300 Subject: [Python-ideas] Add support for version objects In-Reply-To: <20160527193041.GA1010@phdru.name> References: <20160527193041.GA1010@phdru.name> Message-ID: On 27.05.16 22:30, Oleg Broytman wrote: > Something like > https://hg.python.org/cpython/file/tip/Lib/distutils/version.py > https://hg.python.org/cpython/file/tip/Lib/distutils/versionpredicate.py > ??? Thank you. I was not aware of this class when wrote my message. This class provide a part of the feature. It doesn't provide nor tuple-like interface, nor attributes like major and minor, it doesn't allow to create a version object from components, and LooseVersion objects are not comparable in general case: >>> LooseVersion('2.2beta29') < LooseVersion('2.2.0') Traceback (most recent call last): File "", line 1, in File "/home/serhiy/py/cpython/Lib/distutils/version.py", line 52, in __lt__ c = self._cmp(other) File "/home/serhiy/py/cpython/Lib/distutils/version.py", line 337, in _cmp if self.version < other.version: TypeError: '<' not supported between instances of 'str' and 'int' From greg.ewing at canterbury.ac.nz Sat May 28 04:47:58 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 28 May 2016 20:47:58 +1200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <574903C1.2010200@stoneleaf.us> References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <87shx4d6ip.fsf@thinkpad.rath.org> <57478250.6000707@canterbury.ac.nz> <878tyv7ctq.fsf@thinkpad.rath.org> <5748F52A.9020504@canterbury.ac.nz> <574903C1.2010200@stoneleaf.us> Message-ID: <57495B3E.7050902@canterbury.ac.nz> Ethan Furman wrote: > Meaning we can *only* match on constructor arguments? What about > attributes that are not set in the constructor, are we just out of luck > for those? No, keyword pattern args should probably match either keyword constructor args or attributes. -- Greg From neatnate at gmail.com Sat May 28 06:22:40 2016 From: neatnate at gmail.com (Nathan Schneider) Date: Sat, 28 May 2016 11:22:40 +0100 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: <5748F750.6040507@canterbury.ac.nz> References: <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> <5747D3A5.9050504@stoneleaf.us> <20160527161739.GG12028@ando.pearwood.info> <5748F750.6040507@canterbury.ac.nz> Message-ID: On Sat, May 28, 2016 at 2:41 AM, Greg Ewing wrote: > Steven D'Aprano wrote: > > "getmany" doesn't tell you, many of what? >> > > I think the idea is that it would get the same kind > of things that get() gets, i.e. items by their keys. > > > My slight hesitation about "many" is that it's a subjective quantity. (Are 2 or 3 keys enough to count as "many"?) Another option would be 'geteach'?i.e., for each key provided, get a value. Or 'getmult' (multiple), but that could be mistaken as multiplication. Nathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat May 28 06:26:09 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 28 May 2016 11:26:09 +0100 Subject: [Python-ideas] pattern matching proof-of-concept In-Reply-To: References: Message-ID: On 28 May 2016 at 08:22, Michael Selik wrote: > My original dict unpacking proposal was very short and lacked a motivating > usage. Toy examples made my proposal look unnecessarily verbose and > suggested obvious alternatives with easy current syntax. > > Nested/recursive unpacking is much more troublesome, especially when > combined with name-binding. I wrote an example to compare my proposal with > current syntax. > > Example usage. > https://github.com/selik/destructure/blob/master/examples/fips.py > > Implementation. > https://github.com/selik/destructure/blob/master/destructure.py > > The design of my module I'm least happy with is the name-binding. I extended > a SimpleNamespace to create an Erlang-style distinction between bound and > unbound names. Though the API is a bit awkward, now that the module is > built, I'm less enthusiastic about introducing new syntax. Funny how that > works. > > I haven't yet decided how to add post-binding guards to the cases. Interesting! Thanks for taking the time to make a real-world use case. I haven't looked at the module yet, just the example, but the code does look pretty clean and readable. The example is certainly complex enough that I'd probably end up with pretty messy and fragile code if I just tried to put something together with pure Python code. And yes, it's interesting how finding a good API for a module can make the need for a dedicated syntax less pressing. But working out that good API can be really hard (I don't think I'd ever have thought of doing it the way you did). Paul From p.f.moore at gmail.com Sat May 28 06:35:50 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 28 May 2016 11:35:50 +0100 Subject: [Python-ideas] Add support for version objects In-Reply-To: <574920E2.6040600@stoneleaf.us> References: <574920E2.6040600@stoneleaf.us> Message-ID: On 28 May 2016 at 05:38, Ethan Furman wrote: > On 05/27/2016 08:22 PM, Nick Coghlan wrote: > >> The main advantage I'd see to stdlib inclusion is providing "one obvious >> way to do it" - it isn't immediately obvious to a newcomer that PEP 440 >> and the implementation in packaging are the preferred approach for >> SemVer-like version numbers in Python projects. > > > Yup, I'll second that! Agreed. Having a version object in the stdlib would be a good way of directing people to the standard approach. Some points: - It pretty much has to be PEP 440, as that's the standard in the Python packaging ecosystem. Adding a *different* standard to the stdlib would be a big problem IMO. - Packaging tools like pip will still need to bundle an external implementation (at the moment we use the "packaging" library) as they'll need to support older versions of Python for some time yet. So there would need to be a maintained backport version on PyPI (or the packaging library could be sanctioned as that backport). - Assuming a separate backport, it's not clear to me how we'd manage the transition between packaging and the backport (maintain the 2 in parallel? packaging falls back to the stdlib/backport if present? packaging gains a conditional dependency on the backport?) - PEP 440, and the implementation, is pretty stable. But are we happy for it to move to the development pace of the stdlib? The distutils version classes are pretty old, and fairly limited - the strict version is frequently *too* strict, but the loose version is rather too lenient... Paul From ceronman at gmail.com Sat May 28 07:07:10 2016 From: ceronman at gmail.com (=?UTF-8?Q?Manuel_Cer=C3=B3n?=) Date: Sat, 28 May 2016 13:07:10 +0200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <1463939447.1831850.615253577.7371CAA7@webmail.messagingengine.com> <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, May 25, 2016 at 7:52 AM, Nick Coghlan wrote: > On 25 May 2016 at 15:11, Chris Angelico wrote: > > On Wed, May 25, 2016 at 3:03 PM, Stephen J. Turnbull > wrote: > >> How about > >> > >> for : > >> try : > >> pass > >> try : > >> pass > >> > >> Look Ma! No new keywords! Yeah, I know, "for" and "try" both have > >> very strong connotations in Python already, so this may be a "not even > >> the Dutch could like it" automatic-parser-only syntax. > > > > I'd much prefer a different keyword instead of 'for'. If 'with' hadn't > > been used, that would be a better choice. Maybe 'using'... or > > 'switch'. But without iteration, 'for' is going to be very confusing. > > As a variant on Guido's original switch/case idea: > > given EXPR [as TARGET]: > case MATCH_PATTERN [as TARGET] [and CONDITION]: > ... > case MATCH_PATTERN [as TARGET] [and CONDITION]: > ... > case if CONDITION: > ... > case MATCH_PATTERN [as TARGET]: > ... > else: Just a small comment regarding the possible new keyworks: I very much prefer to use 'given' and 'when' instead of 'switch' and 'case': given EXPR [as TARGET]: when MATCH_PATTERN [as TARGET] [and CONDITION]: ... when MATCH_PATTERN [as TARGET] [and CONDITION]: ... when if CONDITION: ... when MATCH_PATTERN [as TARGET]: ... else: One reason for this is that 'given' and 'when' are less likely to collide with existing variable names. In fact, there are already variables named 'switch' and 'case' in the standard Python distribution. I haven't found any for 'given' and 'when'. Also, given .. when is already used in Perl 5 [1] and 6 [2]. Another alternative is to reuse 'if' instead of 'when'. But I'm not sure if that would make things very confusing. [1] http://perldoc.perl.org/perlsyn.html#Switch-Statements [2] https://doc.perl6.org/language/control#given Manuel. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Sat May 28 13:34:16 2016 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Sat, 28 May 2016 14:34:16 -0300 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> Message-ID: On 27 May 2016 at 12:37, Zachary Ware wrote: > On Fri, May 27, 2016 at 10:32 AM, Michael Selik wrote: >> >> >> On Fri, May 27, 2016 at 11:28 AM Zachary Ware >> wrote: >>> >>> Here's a crazy thought that might be best dismissed out of hand: what >>> about extending 'from name import other_names' to accept any object >>> for ? First try to get values via __getitem__() (possibly only >>> for dict/dict subclasses?), next try getattr(), finally try to import >>> the module and pull values from it as per usual. >>> >>> Pros: >>> - solves dict unpacking >> >> >> Would it solve nested dict unpacking? > > How do you mean? Replacing `some_name = > some_dict['some_key']['some_name']` with `from some_dict['some_key'] > import some_name`? > > Sure, why not? :) That is the best idea I've seem on this thread. And them, why having to specify the keys, just to violate DRY? Maybe just allwoing Mappings to be usedwith `from ... import ...` syntax will work nicely, unambiguously, with no new weird syntaxes introduced - and the syntax even allows one to rename the dict keys to other variables, with the `from mymapping import a as c, b as d " variant. That would be certainly nice. > > -- > Zach > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From jsbueno at python.org.br Sat May 28 13:38:05 2016 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Sat, 28 May 2016 14:38:05 -0300 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> Message-ID: On 28 May 2016 at 14:34, Joao S. O. Bueno wrote: > On 27 May 2016 at 12:37, Zachary Ware wrote: >> On Fri, May 27, 2016 at 10:32 AM, Michael Selik wrote: >>> >>> >>> On Fri, May 27, 2016 at 11:28 AM Zachary Ware >>> wrote: >>>> >>>> Here's a crazy thought that might be best dismissed out of hand: what >>>> about extending 'from name import other_names' to accept any object >>>> for ? First try to get values via __getitem__() (possibly only >>>> for dict/dict subclasses?), next try getattr(), finally try to import >>>> the module and pull values from it as per usual. >>>> >>>> Pros: >>>> - solves dict unpacking >>> >>> >>> Would it solve nested dict unpacking? >> >> How do you mean? Replacing `some_name = >> some_dict['some_key']['some_name']` with `from some_dict['some_key'] >> import some_name`? >> >> Sure, why not? :) > > That is the best idea I've seem on this thread. > > And them, why having to specify the keys, just to violate DRY? > > Maybe just allwoing Mappings to be usedwith `from ... import ...` syntax > will work nicely, unambiguously, with no new weird syntaxes introduced - > and the syntax even allows one to rename the dict keys to other > variables, with the > `from mymapping import a as c, b as d " variant. > > That would be certainly nice. Well, I jsut replied upon hitting the "import" suggestion for the first time. Distinguishing it from module imports, of course, is a must. And them, even if using another keyword than "import" (and requiring a specfic name after from) I stil find it much better than the proposals introducing brackets on the LHS ,and loaded with DRY violations. > > >> >> -- >> Zach >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ From mertz at gnosis.cx Sat May 28 14:03:21 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 28 May 2016 11:03:21 -0700 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: References: <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> <5747D3A5.9050504@stoneleaf.us> <20160527161739.GG12028@ando.pearwood.info> <5748F750.6040507@canterbury.ac.nz> Message-ID: Worrying about how many .getmany() is seems silly. As Guido notes, it follows the pattern of .fetchmany() in the DBAPI. That "many" might be one, or even zero, which is fine. On May 28, 2016 3:23 AM, "Nathan Schneider" wrote: > > > On Sat, May 28, 2016 at 2:41 AM, Greg Ewing > wrote: > >> Steven D'Aprano wrote: >> >> "getmany" doesn't tell you, many of what? >>> >> >> I think the idea is that it would get the same kind >> of things that get() gets, i.e. items by their keys. >> >> >> > My slight hesitation about "many" is that it's a subjective quantity. (Are > 2 or 3 keys enough to count as "many"?) > > Another option would be 'geteach'?i.e., for each key provided, get a > value. Or 'getmult' (multiple), but that could be mistaken as > multiplication. > > Nathan > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Sat May 28 16:15:50 2016 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Sat, 28 May 2016 17:15:50 -0300 Subject: [Python-ideas] Enhancing dict.values In-Reply-To: References: <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <921a93c7-9775-a80a-2a50-c7b138f5e31a@mrabarnett.plus.com> <20160527033930.GF12028@ando.pearwood.info> <5747D3A5.9050504@stoneleaf.us> <20160527161739.GG12028@ando.pearwood.info> <5748F750.6040507@canterbury.ac.nz> Message-ID: So, apart from whatever idea you find more suitable, as I was playing around with a module for dictionary utils just this week, I made a proof of concept thingy that uses context managers and import from - Whoever want to try it is welcome, of course: (env)[gwidion at localhost tmp30]$ pip install extradict ... Successfully installed extradict-0.1.9 (env)[gwidion at localhost tmp30]$ python ... >>> from extradict import MapGetter >>> a = dict(b="test", c="another test") >>> with MapGetter(a) as a: ... from a import b, c ... >>> print (b, c) test another test ---------- For the time being it works as a naive implementation monkey patching "__import__" - When (and if) I get a proper thing using the importlib machinery , I will upgrade "extradict" to 0.2 On 28 May 2016 at 15:03, David Mertz wrote: > Worrying about how many .getmany() is seems silly. As Guido notes, it > follows the pattern of .fetchmany() in the DBAPI. That "many" might be one, > or even zero, which is fine. > > On May 28, 2016 3:23 AM, "Nathan Schneider" wrote: >> >> >> >> On Sat, May 28, 2016 at 2:41 AM, Greg Ewing >> wrote: >>> >>> Steven D'Aprano wrote: >>> >>>> "getmany" doesn't tell you, many of what? >>> >>> >>> I think the idea is that it would get the same kind >>> of things that get() gets, i.e. items by their keys. >>> >>> >> >> My slight hesitation about "many" is that it's a subjective quantity. (Are >> 2 or 3 keys enough to count as "many"?) >> >> Another option would be 'geteach'?i.e., for each key provided, get a >> value. Or 'getmult' (multiple), but that could be mistaken as >> multiplication. >> >> Nathan >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From eric at trueblade.com Sun May 29 08:35:46 2016 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 29 May 2016 08:35:46 -0400 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> Message-ID: On 5/28/2016 1:34 PM, Joao S. O. Bueno wrote: > On 27 May 2016 at 12:37, Zachary Ware wrote: >> On Fri, May 27, 2016 at 10:32 AM, Michael Selik wrote: >>> >>> >>> On Fri, May 27, 2016 at 11:28 AM Zachary Ware >>> wrote: >>>> >>>> Here's a crazy thought that might be best dismissed out of hand: what >>>> about extending 'from name import other_names' to accept any object >>>> for ? First try to get values via __getitem__() (possibly only >>>> for dict/dict subclasses?), next try getattr(), finally try to import >>>> the module and pull values from it as per usual. >>>> >>>> Pros: >>>> - solves dict unpacking >>> >>> >>> Would it solve nested dict unpacking? >> >> How do you mean? Replacing `some_name = >> some_dict['some_key']['some_name']` with `from some_dict['some_key'] >> import some_name`? >> >> Sure, why not? :) > > That is the best idea I've seem on this thread. > > And them, why having to specify the keys, just to violate DRY? > > Maybe just allwoing Mappings to be usedwith `from ... import ...` syntax > will work nicely, unambiguously, with no new weird syntaxes introduced - > and the syntax even allows one to rename the dict keys to other > variables, with the > `from mymapping import a as c, b as d " variant. > > That would be certainly nice. Wouldn't this approach require that the keys be constants? That is, you couldn't implement a replacement for: val = d[key+'bar'] I'm not sure that's a reasonable restriction. Eric. From jsbueno at python.org.br Sun May 29 10:28:02 2016 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Sun, 29 May 2016 11:28:02 -0300 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> Message-ID: On 29 May 2016 at 09:35, Eric V. Smith wrote: > On 5/28/2016 1:34 PM, Joao S. O. Bueno wrote: >> >> On 27 May 2016 at 12:37, Zachary Ware >> wrote: >>> >>> On Fri, May 27, 2016 at 10:32 AM, Michael Selik >>> wrote: >>>> >>>> >>>> >>>> On Fri, May 27, 2016 at 11:28 AM Zachary Ware >>>> wrote: >>>>> >>>>> >>>>> Here's a crazy thought that might be best dismissed out of hand: what >>>>> about extending 'from name import other_names' to accept any object >>>>> for ? First try to get values via __getitem__() (possibly only >>>>> for dict/dict subclasses?), next try getattr(), finally try to import >>>>> the module and pull values from it as per usual. >>>>> >>>>> Pros: >>>>> - solves dict unpacking >>>> >>>> >>>> >>>> Would it solve nested dict unpacking? >>> >>> >>> How do you mean? Replacing `some_name = >>> some_dict['some_key']['some_name']` with `from some_dict['some_key'] >>> import some_name`? >>> >>> Sure, why not? :) >> >> >> That is the best idea I've seem on this thread. >> >> And them, why having to specify the keys, just to violate DRY? >> >> Maybe just allwoing Mappings to be usedwith `from ... import ...` >> syntax >> will work nicely, unambiguously, with no new weird syntaxes introduced - >> and the syntax even allows one to rename the dict keys to other >> variables, with the >> `from mymapping import a as c, b as d " variant. >> >> That would be certainly nice. > > > Wouldn't this approach require that the keys be constants? That is, you > couldn't implement a replacement for: > > val = d[key+'bar'] > > I'm not sure that's a reasonable restriction. As I posted up on the other thread, I've implemented a poof of concept for this in a somewhat toyish package I've started earlier. So right now, one can do $ pip install extradict $ python >>> from extradict import MapGetter >>> with MapGetter({"a": 1, "b": 2}) as mydict: ... from mydict import a, b ... >>> print(a, b) The code is at http://github.com/jsbueno/extradict - (btw, since that e-mail, I've studied the import hook mechanisms and decided to keep my first design of temporarily replacing __import__. ) If more people decide to use it, it would be easy to include some more mapping parameters to the call to MapGetter to overcome static restrictions of the "import" syntax with MapGetter(mapping, keysuffix="bar"): ... or rather: with MapGetter(mapping, keytransform=lambda key: key + 'bar'): ... js -><- From pavol.lisy at gmail.com Sun May 29 13:10:33 2016 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Sun, 29 May 2016 19:10:33 +0200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> Message-ID: I see backward compatibility problem with import : sys = dict(version_info=4.0) from sys import version_info This is legal and using import for unpacking dict could change this behavior. In case we put higher priority to import from module then unexpected module in PYTHONPATH could change unpacked value. Sort of problem not easy to found. From library maintainer point of view not easy to avoid too. From guido at python.org Sun May 29 13:23:51 2016 From: guido at python.org (Guido van Rossum) Date: Sun, 29 May 2016 10:23:51 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> Message-ID: Any idea using import syntax or even syntax similar to import is dead. Import is about modules and needs to stay about that. On Sun, May 29, 2016 at 10:10 AM, Pavol Lisy wrote: > I see backward compatibility problem with import : > > sys = dict(version_info=4.0) > from sys import version_info > > This is legal and using import for unpacking dict could change this behavior. > > In case we put higher priority to import from module then unexpected > module in PYTHONPATH could change unpacked value. Sort of problem not > easy to found. From library maintainer point of view not easy to avoid > too. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From pavol.lisy at gmail.com Sun May 29 13:49:08 2016 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Sun, 29 May 2016 19:49:08 +0200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> Message-ID: Sorry I just tried to help understand why it is wrong idea. :) But (and sorry it is another story) what surprised me during my analysis of this problem is that import variable from module is more likely unpacking value than getting access to module's variable. my_test.py --------- VAR = 1 def setter(a): global VAR VAR = a def getter(): return VAR -------- from my_test import VAR, setter, getter print(VAR) # 1 print(getter()) # 1 setter(7) print(VAR) # 1 ! print(getter()) # 7 from my_test import VAR print(VAR) # 7 ! The more I understand python the more I see that I don't understand enough. :) 2016-05-29 19:23 GMT+02:00, Guido van Rossum : > Any idea using import syntax or even syntax similar to import is dead. > Import is about modules and needs to stay about that. > > On Sun, May 29, 2016 at 10:10 AM, Pavol Lisy wrote: >> I see backward compatibility problem with import : >> >> sys = dict(version_info=4.0) >> from sys import version_info >> >> This is legal and using import for unpacking dict could change this >> behavior. >> >> In case we put higher priority to import from module then unexpected >> module in PYTHONPATH could change unpacked value. Sort of problem not >> easy to found. From library maintainer point of view not easy to avoid >> too. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido van Rossum (python.org/~guido) > From leewangzhong+python at gmail.com Sun May 29 15:11:43 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Sun, 29 May 2016 15:11:43 -0400 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <57495B3E.7050902@canterbury.ac.nz> References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <87shx4d6ip.fsf@thinkpad.rath.org> <57478250.6000707@canterbury.ac.nz> <878tyv7ctq.fsf@thinkpad.rath.org> <5748F52A.9020504@canterbury.ac.nz> <574903C1.2010200@stoneleaf.us> <57495B3E.7050902@canterbury.ac.nz> Message-ID: On May 28, 2016 4:48 AM, "Greg Ewing" wrote: > > Ethan Furman wrote: >> >> Meaning we can *only* match on constructor arguments? What about attributes that are not set in the constructor, are we just out of luck for those? > > > No, keyword pattern args should probably match either > keyword constructor args or attributes. That would make things ambiguous and tricky, wouldn't it? Any criticism for using object(attrname0=v0, attrname1=v1) for generalized attribute unpacking? -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Sun May 29 16:05:43 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Sun, 29 May 2016 13:05:43 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: (Guido van Rossum's message of "Sun, 29 May 2016 10:23:51 -0700") References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> Message-ID: <87twhgr5ns.fsf@kosh.rath.org> On May 29 2016, Guido van Rossum wrote: > Any idea using import syntax or even syntax similar to import is dead. Does that mean the whole idea is dead? Because as I see it, there are only two ways this could possibly be implemented: 1. As a statement similar to import, which you declared dead. 2. As an assignment, which you declared dead IIRC primarily because the RHS should not be "peeking" into the LHS. Or am I interpreting "similar to import" too broadly? Is a statement using other words still on the table? d = {"foo": 42} d 42 # or the other way around assert foo == 42 Best, Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From michael.selik at gmail.com Sun May 29 16:37:54 2016 From: michael.selik at gmail.com (Michael Selik) Date: Sun, 29 May 2016 20:37:54 +0000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> Message-ID: On Sun, May 29, 2016 at 1:49 PM Pavol Lisy wrote: > from my_test import VAR, setter, getter > > The more I understand python the more I see that I don't understand enough. > Seems like you expected the ``VAR`` in your __main__ module would be the same variable as the one in the ``my_test`` module Each module's globals are separate namespaces. ``from module import name`` is roughly equivalent to 3 lines: import module name = module.name del module -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sun May 29 19:00:00 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 30 May 2016 11:00:00 +1200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <87shx4d6ip.fsf@thinkpad.rath.org> <57478250.6000707@canterbury.ac.nz> <878tyv7ctq.fsf@thinkpad.rath.org> <5748F52A.9020504@canterbury.ac.nz> <574903C1.2010200@stoneleaf.us> <57495B3E.7050902@canterbury.ac.nz> Message-ID: <574B7470.20307@canterbury.ac.nz> Franklin? Lee wrote: > On May 28, 2016 4:48 AM, "Greg Ewing" > wrote: > > > > No, keyword pattern args should probably match either > > keyword constructor args or attributes. > > That would make things ambiguous and tricky, wouldn't it? My feeling is that for any well-designed object they should be compatible. If an object has a constructor keyword arg and an attribute with the same name but they mean different things, that object is confusing to begin with. -- Greg From guido at python.org Sun May 29 19:49:07 2016 From: guido at python.org (Guido van Rossum) Date: Sun, 29 May 2016 16:49:07 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <87twhgr5ns.fsf@kosh.rath.org> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> <87twhgr5ns.fsf@kosh.rath.org> Message-ID: On Sun, May 29, 2016 at 1:05 PM, Nikolaus Rath wrote: > On May 29 2016, Guido van Rossum wrote: >> Any idea using import syntax or even syntax similar to import is dead. > > Does that mean the whole idea is dead? Because as I see it, there are > only two ways this could possibly be implemented: > > 1. As a statement similar to import, which you declared dead. Any use of the word 'import' (even in combination with other words) is verboten. Anything *starting* with 'from' also sounds like a bad idea. > 2. As an assignment, which you declared dead IIRC primarily because the > RHS should not be "peeking" into the LHS. Right. > Or am I interpreting "similar to import" too broadly? Is a statement > using other words still on the table? > > d = {"foo": 42} > d 42 # or the other way around > assert foo == 42 This I don't understand -- why would the '42' appear in the extraction syntax? I guess you meant "foo"? Maybe we can riff on extract foo from d ? Though honestly that looks like it would be extracting d.foo, no d['foo']. -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Mon May 30 02:37:38 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 30 May 2016 16:37:38 +1000 Subject: [Python-ideas] Add support for version objects In-Reply-To: References: <574920E2.6040600@stoneleaf.us> Message-ID: On 28 May 2016 3:36 am, "Paul Moore" wrote: > > On 28 May 2016 at 05:38, Ethan Furman wrote: > > On 05/27/2016 08:22 PM, Nick Coghlan wrote: > > > >> The main advantage I'd see to stdlib inclusion is providing "one obvious > >> way to do it" - it isn't immediately obvious to a newcomer that PEP 440 > >> and the implementation in packaging are the preferred approach for > >> SemVer-like version numbers in Python projects. > > > > > > Yup, I'll second that! > > Agreed. Having a version object in the stdlib would be a good way of > directing people to the standard approach. Some points: > > - It pretty much has to be PEP 440, as that's the standard in the > Python packaging ecosystem. Adding a *different* standard to the > stdlib would be a big problem IMO. > - Packaging tools like pip will still need to bundle an external > implementation (at the moment we use the "packaging" library) as > they'll need to support older versions of Python for some time yet. So > there would need to be a maintained backport version on PyPI (or the > packaging library could be sanctioned as that backport). > - Assuming a separate backport, it's not clear to me how we'd manage > the transition between packaging and the backport (maintain the 2 in > parallel? packaging falls back to the stdlib/backport if present? > packaging gains a conditional dependency on the backport?) > - PEP 440, and the implementation, is pretty stable. But are we happy > for it to move to the development pace of the stdlib? The other issue is the intended use case - I *think* PEP 440 (including the normalisation steps) will cover the version numbering for external dependencies like Tkinter and zlib, but it's not the precise context that spec was designed to cover (i.e. versioning distribution packages for Python projects). I still think the idea is worth considering, it's just a more complex problem than it appears to be at first glance. Cheers, Nick. > > The distutils version classes are pretty old, and fairly limited - the > strict version is frequently *too* strict, but the loose version is > rather too lenient... > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.selik at gmail.com Mon May 30 02:52:33 2016 From: michael.selik at gmail.com (Michael Selik) Date: Mon, 30 May 2016 06:52:33 +0000 Subject: [Python-ideas] pattern matching proof-of-concept In-Reply-To: References: Message-ID: I'm sure y'all are too busy with PyCon to be checking email. But if you're curious, I added attribute unpacking and post-binding guards ( https://github.com/selik/destructure). The object unpacking works as most folks have suggested, via kwargs. The guards unfortunately need to be functions that take no parameters. match(schema=Foo(a=1, b=bind.x), data=Foo(a=1, b=2), lambda : bind.x > 0) On Sat, May 28, 2016 at 6:26 AM Paul Moore wrote: > On 28 May 2016 at 08:22, Michael Selik wrote: > > My original dict unpacking proposal was very short and lacked a > motivating > > usage. Toy examples made my proposal look unnecessarily verbose and > > suggested obvious alternatives with easy current syntax. > > > > Nested/recursive unpacking is much more troublesome, especially when > > combined with name-binding. I wrote an example to compare my proposal > with > > current syntax. > > > > Example usage. > > https://github.com/selik/destructure/blob/master/examples/fips.py > > > > Implementation. > > https://github.com/selik/destructure/blob/master/destructure.py > > > > The design of my module I'm least happy with is the name-binding. I > extended > > a SimpleNamespace to create an Erlang-style distinction between bound and > > unbound names. Though the API is a bit awkward, now that the module is > > built, I'm less enthusiastic about introducing new syntax. Funny how that > > works. > > > > I haven't yet decided how to add post-binding guards to the cases. > > Interesting! Thanks for taking the time to make a real-world use case. > I haven't looked at the module yet, just the example, but the code > does look pretty clean and readable. The example is certainly complex > enough that I'd probably end up with pretty messy and fragile code if > I just tried to put something together with pure Python code. > > And yes, it's interesting how finding a good API for a module can make > the need for a dedicated syntax less pressing. But working out that > good API can be really hard (I don't think I'd ever have thought of > doing it the way you did). > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.selik at gmail.com Mon May 30 05:07:37 2016 From: michael.selik at gmail.com (Michael Selik) Date: Mon, 30 May 2016 09:07:37 +0000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <574B7470.20307@canterbury.ac.nz> References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <87shx4d6ip.fsf@thinkpad.rath.org> <57478250.6000707@canterbury.ac.nz> <878tyv7ctq.fsf@thinkpad.rath.org> <5748F52A.9020504@canterbury.ac.nz> <574903C1.2010200@stoneleaf.us> <57495B3E.7050902@canterbury.ac.nz> <574B7470.20307@canterbury.ac.nz> Message-ID: On Sun, May 29, 2016 at 7:00 PM Greg Ewing wrote: > Franklin? Lee wrote: > > On May 28, 2016 4:48 AM, "Greg Ewing" > > wrote: > > > > > > No, keyword pattern args should probably match either > > > keyword constructor args or attributes. > > > > That would make things ambiguous and tricky, wouldn't it? > > My feeling is that for any well-designed object they > should be compatible. If an object has a constructor > keyword arg and an attribute with the same name but > they mean different things, that object is confusing > to begin with. > What about Fraction -- it has ``numerator`` and ``denominator`` for keyword arguments, but also has an extra stuff like ``real`` and ``imag`` inherited from Complex? -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon May 30 08:52:22 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 31 May 2016 00:52:22 +1200 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: References: <5743EFCB.40401@canterbury.ac.nz> <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <87shx4d6ip.fsf@thinkpad.rath.org> <57478250.6000707@canterbury.ac.nz> <878tyv7ctq.fsf@thinkpad.rath.org> <5748F52A.9020504@canterbury.ac.nz> <574903C1.2010200@stoneleaf.us> <57495B3E.7050902@canterbury.ac.nz> <574B7470.20307@canterbury.ac.nz> Message-ID: <574C3786.2030306@canterbury.ac.nz> Michael Selik wrote: > On Sun, May 29, 2016 at 7:00 PM Greg Ewing > wrote: > > My feeling is that for any well-designed object they > should be compatible. If an object has a constructor > keyword arg and an attribute with the same name but > they mean different things, that object is confusing > to begin with. > > What about Fraction -- it has ``numerator`` and ``denominator`` for > keyword arguments, but also has an extra stuff like ``real`` and > ``imag`` inherited from Complex? All of those names are distinct, so there's no problem there. All of them would be usable as keywords when unpacking. Additionally you would be able to unpack 'numerator' and 'denominator' positionally, because the constructor accepts those positionally. -- Greg From Nikolaus at rath.org Mon May 30 11:07:48 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Mon, 30 May 2016 08:07:48 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: (Guido van Rossum's message of "Sun, 29 May 2016 16:49:07 -0700") References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> <87twhgr5ns.fsf@kosh.rath.org> Message-ID: <87bn3na8jf.fsf@kosh.rath.org> On May 29 2016, Guido van Rossum wrote: >> using other words still on the table? >> >> d = {"foo": 42} >> d 42 # or the other way around >> assert foo == 42 > > This I don't understand -- why would the '42' appear in the extraction > syntax? I guess you meant "foo"? Yes, sorry. > Maybe we can riff on > > extract foo from d > > ? Though honestly that looks like it would be extracting d.foo, no > d['foo']. Yeah, but that might be useful too :-). How about: extract key foo from d extract attribute foo import d or export key foo from d export attribute foo import d As for "import", with both foo and d required to be identifiers. Best, Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From p.f.moore at gmail.com Mon May 30 11:23:30 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 30 May 2016 16:23:30 +0100 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <87bn3na8jf.fsf@kosh.rath.org> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> <87twhgr5ns.fsf@kosh.rath.org> <87bn3na8jf.fsf@kosh.rath.org> Message-ID: On 30 May 2016 at 16:07, Nikolaus Rath wrote: > Yeah, but that might be useful too :-). How about: > > extract key foo from d > extract attribute foo import d > > or > > export key foo from d > export attribute foo import d > > As for "import", with both foo and d required to be identifiers. At this point, the question has to be, how is this any better than foo = d.foo foo = d['foo'] ??? Paul From boekewurm at gmail.com Mon May 30 11:39:38 2016 From: boekewurm at gmail.com (Matthias welp) Date: Mon, 30 May 2016 17:39:38 +0200 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <87bn3na8jf.fsf@kosh.rath.org> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> <87twhgr5ns.fsf@kosh.rath.org> <87bn3na8jf.fsf@kosh.rath.org> Message-ID: On 30 May 2016 at 17:07, Nikolaus Rath wrote: > extract attribute foo import d Does this mean that attributes, values and properties from objects will be extractable too? If so, with this syntax, will the getter/setter properties still work as if it was in that object, or will the imported value contain the property.getter() value returned? -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon May 30 14:35:17 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 31 May 2016 04:35:17 +1000 Subject: [Python-ideas] Match statement brainstorm In-Reply-To: <5748FECD.2030203@canterbury.ac.nz> References: <22341.12808.512932.821488@turnbull.sk.tsukuba.ac.jp> <87shx4d6ip.fsf@thinkpad.rath.org> <57478250.6000707@canterbury.ac.nz> <22343.62785.209111.270692@turnbull.sk.tsukuba.ac.jp> <5748FECD.2030203@canterbury.ac.nz> Message-ID: <20160530183517.GQ12028@ando.pearwood.info> On Sat, May 28, 2016 at 02:13:33PM +1200, Greg Ewing wrote: [...] > Finally we could allow "case class" to be abbreviated to > just "class": > > switch obj: > class Intersection( > class Line(class Point(?x1, ?y1), class Point(?x2, ?y2)), > class Line(class Point(?x3, ?y3), class Point(?x4, ?y4))): > > Is that unacceptably verbose? I don't know. Verbose, not so much. Cryptic? Hell yes! Who is going to be able to guess what it means? We're no longer even in the same galaxy as executable pseudo-code. Try this as an exercise: given the above, explain in plain English what it does and what the result will be. What exactly is being matched? Those familiar with C-like switches are going to be totally confused. They'll probably wonder if you are matching "if obj == the class Intersection", and have no idea what's going on with the nested Line and Point stuff. Those familiar with classes and the class keyword are going to wonder what class definitions are doing inside the class declaration. Does this mean we can now do this? class Child(class Parent: pass): def method(self): ... Obviously not, but that's what it looks like. I wonder whether the real problem here is that pattern matching as a concept is simply too concise for mere mortals? Once you get past the trivial Haskell-esque: def factorial(n): switch n: 0: return 1 n: return n*fact(n-1) its just doing too much in too little code to be easily comprehensible. -- Steve From pavol.lisy at gmail.com Mon May 30 16:48:45 2016 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Mon, 30 May 2016 22:48:45 +0200 Subject: [Python-ideas] pattern matching proof-of-concept In-Reply-To: References: Message-ID: I see a little problem if we want to match more cases: from destructure import Binding, Switch o = Binding() schema1 = [2, o.x, 3] schema2 = [2, 4, o.x] s = Switch([2, 4, 3]) if s.case(schema1): print(o.x) if s.case(schema2): print(o.x) 4 --------------------------------------------------------------------------- BindError Traceback (most recent call last) [...] BindError: name 'x' has already been bound to 3 I prefer to enable multiple matches. But if you really like to forbid it then you would probably like to change next line -> raise BindError(fmt.format(name=name, value=value)) to line -> raise BindError(fmt.format(name=name, value=getattr(self, name))) Anyway I really like your proof of concept! :) To improve it maybe you could distinguish cases in readme.rst. For example: schema1 = [1, o.x, 3] schema2 = [2, 4, o.x] s = Switch([2, 4, 6]) if s.case(schema1): print(o.x) elif s.case(schema2): print(o.x) else: print('otherwise') 6 2016-05-30 8:52 GMT+02:00, Michael Selik : > I'm sure y'all are too busy with PyCon to be checking email. But if you're > curious, I added attribute unpacking and post-binding guards ( > https://github.com/selik/destructure). > > The object unpacking works as most folks have suggested, via kwargs. > The guards unfortunately need to be functions that take no parameters. > > match(schema=Foo(a=1, b=bind.x), data=Foo(a=1, b=2), lambda : bind.x > > 0) > > > On Sat, May 28, 2016 at 6:26 AM Paul Moore wrote: > >> On 28 May 2016 at 08:22, Michael Selik wrote: >> > My original dict unpacking proposal was very short and lacked a >> motivating >> > usage. Toy examples made my proposal look unnecessarily verbose and >> > suggested obvious alternatives with easy current syntax. >> > >> > Nested/recursive unpacking is much more troublesome, especially when >> > combined with name-binding. I wrote an example to compare my proposal >> with >> > current syntax. >> > >> > Example usage. >> > https://github.com/selik/destructure/blob/master/examples/fips.py >> > >> > Implementation. >> > https://github.com/selik/destructure/blob/master/destructure.py >> > >> > The design of my module I'm least happy with is the name-binding. I >> extended >> > a SimpleNamespace to create an Erlang-style distinction between bound >> > and >> > unbound names. Though the API is a bit awkward, now that the module is >> > built, I'm less enthusiastic about introducing new syntax. Funny how >> > that >> > works. >> > >> > I haven't yet decided how to add post-binding guards to the cases. >> >> Interesting! Thanks for taking the time to make a real-world use case. >> I haven't looked at the module yet, just the example, but the code >> does look pretty clean and readable. The example is certainly complex >> enough that I'd probably end up with pretty messy and fragile code if >> I just tried to put something together with pure Python code. >> >> And yes, it's interesting how finding a good API for a module can make >> the need for a dedicated syntax less pressing. But working out that >> good API can be really hard (I don't think I'd ever have thought of >> doing it the way you did). >> >> Paul >> > From michael.selik at gmail.com Mon May 30 18:22:24 2016 From: michael.selik at gmail.com (Michael Selik) Date: Mon, 30 May 2016 22:22:24 +0000 Subject: [Python-ideas] pattern matching proof-of-concept In-Reply-To: References: Message-ID: On Mon, May 30, 2016 at 4:48 PM Pavol Lisy wrote: > I prefer to enable multiple matches. > Ok. Preventing rebinding was inspired by Erlang, but I can see why you might want to match multiple cases. I'll move that feature to a FreezeBinding class; I figure it'll be helpful in multithreading. > But if you really like to forbid it then you would probably like to > change next line -> > raise BindError(fmt.format(name=name, value=value)) > to line -> > raise BindError(fmt.format(name=name, value=getattr(self, name))) > Fixed. > Anyway I really like your proof of concept! :) > Thanks for reading. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon May 30 21:16:26 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 30 May 2016 18:16:26 -0700 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name Message-ID: In mypy we have a need for type variables, which are created like this: from typing import TypeVar T = TypeVar('T') I just saw a lightning talk about sympy where they define symbols to be used in mathematical equations, like this: from sympy import Symbol x = Symbol('x') I'm sure this is not a new idea, but so far I've always thought that this is pretty esoteric and the approach here is good enough. But maybe we can actually do better, and whatever solution we come up with might also be useful for extracting attributes from an object or values from a mapping? -- --Guido van Rossum (python.org/~guido) From bruce at leban.us Mon May 30 21:59:46 2016 From: bruce at leban.us (Bruce Leban) Date: Mon, 30 May 2016 18:59:46 -0700 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: References: Message-ID: On May 30, 2016 6:20 PM, "Guido van Rossum" wrote: > > In mypy we have a need for type variables, which are created like this: > > from typing import TypeVar > > T = TypeVar('T') > > I just saw a lightning talk about sympy where they define symbols to > be used in mathematical equations, like this: > > from sympy import Symbol > > x = Symbol('x') > > I'm sure this is not a new idea, but so far I've always thought that > this is pretty esoteric and the approach here is good enough. But > maybe we can actually do better.... One of the negatives of the above usages is that the two names don't have to match. I can do: Q = TypeVar('U') for example. Dedicated syntax could fix that, e.g., from typing import TypeVar as T$ And instead of writing x I write T$x. That compiles to a call to TypeVar('x'). A function imported this way would be required to always return the same value. That is T$xyz is T$xyz and the compiler would be free to optimize calls away or not. If I want a runtime eval that won't be optimized away I can write T$('x' + 'yz') which is T$xyz. Of course $ could be :: or ! or something else.x Nothing prevents me from writing x = T$x but I can be assured that this is the same as T$x elsewhere (barring different code importing different functions under the same name). --- Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon May 30 23:00:41 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 30 May 2016 20:00:41 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <87bn3na8jf.fsf@kosh.rath.org> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> <87twhgr5ns.fsf@kosh.rath.org> <87bn3na8jf.fsf@kosh.rath.org> Message-ID: <574CFE59.2050904@stoneleaf.us> On 05/30/2016 08:07 AM, Nikolaus Rath wrote: > On May 29 2016, Guido van Rossum wrote: >>> using other words still on the table? >>> >>> d = {"foo": 42} >>> d 42 # or the other way around >>> assert foo == 42 >> >> This I don't understand -- why would the '42' appear in the extraction >> syntax? I guess you meant "foo"? > > Yes, sorry. > >> Maybe we can riff on >> >> extract foo from d >> >> ? Though honestly that looks like it would be extracting d.foo, no >> d['foo']. > > Yeah, but that might be useful too :-). How about: > > extract key foo from d > extract attribute foo import d > > or > > export key foo from d > export attribute foo import d > > As for "import", with both foo and d required to be identifiers. The versions with "import" in them are DAAP (dead-as-a-parrot). No point in even talking about them. -- ~Ethan~ From steve at pearwood.info Mon May 30 23:08:10 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 31 May 2016 13:08:10 +1000 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: References: Message-ID: <20160531030810.GR12028@ando.pearwood.info> On Mon, May 30, 2016 at 06:16:26PM -0700, Guido van Rossum wrote: [...] > T = TypeVar('T') > x = Symbol('x') > > I'm sure this is not a new idea, but so far I've always thought that > this is pretty esoteric and the approach here is good enough. But > maybe we can actually do better, and whatever solution we come up with > might also be useful for extracting attributes from an object or > values from a mapping? This comes up a lot and it would be nice to clean it up. T = type('T', bases, ns) Record = namedtuple('Record', fields) It came up in the discussion on dict unpacking: a, b, c = **dict # unpacks keys 'a', 'b', 'c' which was rejected as too magical. But maybe it will seem less magical if we have a special assignment operator that takes the left hand symbol(s) and copies them to the right? How do you feel about an arrow operator? T -> TypeVar() x -> Symbol() T -> type(bases, ns) Record -> namedtuple(fields) In general: name [, name, ...] -> callable(...) where "callable" is some expression, e.g. a name, a dot lookup, etc. # okay name -> sympy.Symbol() The easy case ------------- All assignment targets are plain names. The arrow operator takes the names, turns them into strings, and passes them to the callable on the right as a tuple of strings, given as the first positional argument. Since the most common case will be a single target, I think it is worth while to treat it as a special case and pass just a string: T -> Type() # like Type('T') a, b, c -> Spam(arg) # like Spam(('a', 'b', 'c'), arg) The callable being called doesn't need to know or care whether it is being called with the -> or = assignment. It just receives the name(s) as the first argument. This will mean that callables that take the name as (say) the second argument cannot be used with this syntax. Using the arrow operator with arbitrary expressions on the right will be a SyntaxError: x -> y + 1 my_name -> spam*f() my_name -> f() or g() I reject that third example as it is unclear whether both f and g, or only f, get the name as first argument. But this is allowed: my_name -> f(g()) # like f('my_name', g()) If f isn't *actually* a callable, you get a runtime TypeError as usual. The hard case ------------- Assignment target(s) is not a plain name. spam[2].eggs().attr -> Symbol() I'm going to take the cowards way out and just say "we don't allow that". -- Steve From guido at python.org Tue May 31 00:45:48 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 30 May 2016 21:45:48 -0700 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <20160531030810.GR12028@ando.pearwood.info> References: <20160531030810.GR12028@ando.pearwood.info> Message-ID: That all sounds not unreasonable, except the -> operator feels kind of wrong given its use in function signatures. In terms of restricted syntax, the precedent for such syntax restrictions would be decorators, which also don't allow arbitrary expressions after the @. In fact, maybe we could somehow unifying this with structural unpacking *and* variable decorators (also oft-requested)? I know you must think "Guido is high" or "someone broke into Guido's account". My actual excuse is that I'm procrastinating on rehearsing my PyCon keynote, which is in about 11 hours. So don't take all this as too much of a ringing endorsement. :-) On Mon, May 30, 2016 at 8:08 PM, Steven D'Aprano wrote: > On Mon, May 30, 2016 at 06:16:26PM -0700, Guido van Rossum wrote: > > [...] >> T = TypeVar('T') >> x = Symbol('x') >> >> I'm sure this is not a new idea, but so far I've always thought that >> this is pretty esoteric and the approach here is good enough. But >> maybe we can actually do better, and whatever solution we come up with >> might also be useful for extracting attributes from an object or >> values from a mapping? > > This comes up a lot and it would be nice to clean it up. > > T = type('T', bases, ns) > Record = namedtuple('Record', fields) > > It came up in the discussion on dict unpacking: > > a, b, c = **dict # unpacks keys 'a', 'b', 'c' > > which was rejected as too magical. But maybe it will seem less magical > if we have a special assignment operator that takes the left hand symbol(s) > and copies them to the right? > > How do you feel about an arrow operator? > > > T -> TypeVar() > x -> Symbol() > T -> type(bases, ns) > Record -> namedtuple(fields) > > In general: > > name [, name, ...] -> callable(...) > > where "callable" is some expression, e.g. a name, a dot lookup, etc. > > # okay > name -> sympy.Symbol() > > > > The easy case > ------------- > > All assignment targets are plain names. > > The arrow operator takes the names, turns them into strings, and passes > them to the callable on the right as a tuple of strings, given as the first > positional argument. > > Since the most common case will be a single target, I think it is worth > while to treat it as a special case and pass just a string: > > T -> Type() # like Type('T') > > a, b, c -> Spam(arg) # like Spam(('a', 'b', 'c'), arg) > > The callable being called doesn't need to know or care whether it is > being called with the -> or = assignment. It just receives the name(s) > as the first argument. > > This will mean that callables that take the name as (say) the second > argument cannot be used with this syntax. > > Using the arrow operator with arbitrary expressions on the > right will be a SyntaxError: > > x -> y + 1 > my_name -> spam*f() > my_name -> f() or g() > > I reject that third example as it is unclear whether both f and g, or > only f, get the name as first argument. > > But this is allowed: > > my_name -> f(g()) # like f('my_name', g()) > > If f isn't *actually* a callable, you get a runtime TypeError as usual. > > > The hard case > ------------- > > Assignment target(s) is not a plain name. > > spam[2].eggs().attr -> Symbol() > > > I'm going to take the cowards way out and just say "we don't allow > that". > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From joshua.morton13 at gmail.com Tue May 31 00:57:59 2016 From: joshua.morton13 at gmail.com (Joshua Morton) Date: Tue, 31 May 2016 04:57:59 +0000 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <20160531030810.GR12028@ando.pearwood.info> References: <20160531030810.GR12028@ando.pearwood.info> Message-ID: I like the arrow idea, another option is a syntax more in line with the one being discussed for dict unpacking, something like adding a __define__ dunder that returns a tuple of (name, value). So that say, Typevar's __define__ returns the tuple `(self.__name__, self)`. Which is then converted into an assignment. The downside of this being that its assignment overloading(which can be abused), but at least it can only be done when specifically asked for. The other downside is that the syntax define TypeVar('T') # and I assume we can also declare a list of them def ident(t: T) -> T: return t looks both foreign and a bit magical. Also new keyword, but I have a feeling that the same keyword could probably be used here and with dict unpacking. (the *really* big downside is for a library implementer to do something devious like have __define__ always return ("constant", self), or generally something that wasn't clear to the caller, because we get names appearing out of no where with no clear history, but my argument would be that we should trust libraries not to do that and say that it is a very bad thing) The upside is that it avoids the problem of Steven's implementation, since the name is decided internally by the callable. Like with his example, define `Typevar('7')` and similar invalid names would throw an error, in this case probably a TypeError. --Josh On Mon, May 30, 2016 at 11:08 PM Steven D'Aprano wrote: > On Mon, May 30, 2016 at 06:16:26PM -0700, Guido van Rossum wrote: > > [...] > > T = TypeVar('T') > > x = Symbol('x') > > > > I'm sure this is not a new idea, but so far I've always thought that > > this is pretty esoteric and the approach here is good enough. But > > maybe we can actually do better, and whatever solution we come up with > > might also be useful for extracting attributes from an object or > > values from a mapping? > > This comes up a lot and it would be nice to clean it up. > > T = type('T', bases, ns) > Record = namedtuple('Record', fields) > > It came up in the discussion on dict unpacking: > > a, b, c = **dict # unpacks keys 'a', 'b', 'c' > > which was rejected as too magical. But maybe it will seem less magical > if we have a special assignment operator that takes the left hand symbol(s) > and copies them to the right? > > How do you feel about an arrow operator? > > > T -> TypeVar() > x -> Symbol() > T -> type(bases, ns) > Record -> namedtuple(fields) > > In general: > > name [, name, ...] -> callable(...) > > where "callable" is some expression, e.g. a name, a dot lookup, etc. > > # okay > name -> sympy.Symbol() > > > > The easy case > ------------- > > All assignment targets are plain names. > > The arrow operator takes the names, turns them into strings, and passes > them to the callable on the right as a tuple of strings, given as the first > positional argument. > > Since the most common case will be a single target, I think it is worth > while to treat it as a special case and pass just a string: > > T -> Type() # like Type('T') > > a, b, c -> Spam(arg) # like Spam(('a', 'b', 'c'), arg) > > The callable being called doesn't need to know or care whether it is > being called with the -> or = assignment. It just receives the name(s) > as the first argument. > > This will mean that callables that take the name as (say) the second > argument cannot be used with this syntax. > > Using the arrow operator with arbitrary expressions on the > right will be a SyntaxError: > > x -> y + 1 > my_name -> spam*f() > my_name -> f() or g() > > I reject that third example as it is unclear whether both f and g, or > only f, get the name as first argument. > > But this is allowed: > > my_name -> f(g()) # like f('my_name', g()) > > If f isn't *actually* a callable, you get a runtime TypeError as usual. > > > The hard case > ------------- > > Assignment target(s) is not a plain name. > > spam[2].eggs().attr -> Symbol() > > > I'm going to take the cowards way out and just say "we don't allow > that". > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mtanous22 at gmail.com Tue May 31 01:27:53 2016 From: mtanous22 at gmail.com (Matthew Tanous) Date: Mon, 30 May 2016 23:27:53 -0600 Subject: [Python-ideas] Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects Message-ID: <574D20D9.7060504@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Currently, the use of the (*) operator on a list is to duplicate a list by creating multiple references to the same object. While this works intuitively for immutable objects (like [True] * 5) as these immutable references are replaced when the list is assigned to, it makes the operator nigh unusable for mutable objects. The most obvious case is when the operator is duplicated in a sequence like this: arr = [[True] * 5] * 5 This does not create a matrix-like arrangement of the immutable truth variable, but instead creates a list of 5 references to the same list, such that a following assignment like arr[2][3] = False will not change just that one index, but every 4th element of each list in the outer list. This also makes the sequence construction using a mutable type a problem. For example, assume a class Foo: class Foo: def __init__(self): self.val = True def set(self): self.val = False def __repr__(self): return str(self.val) If I then use sequence repetition to create a list of these like so: arr = [Foo()] * 5 This will create a list of references to the same Foo instance, making the list construction itself effectively meaningless. Running the set() method on any of the instances in the list is the same as running it on all the instances in the list. It is my opinion that the sequence repetition operator should be modified to make copies of the objects it is repeating, rather than copying references alone. I believe this would both be more intuitive from a semantic point of view and more useful for the developer. This would change the operator in a way that is mostly unseen in current usage ([5] * 3 would still result in [5, 5, 5]) while treating mutable nesting in a way that is more understandable from the apparent intent of the syntax construction. Reference regarding previous discussion: https://bugs.python.org/issue27135 -----BEGIN PGP SIGNATURE----- iQEcBAEBCAAGBQJXTSDZAAoJEF14rZEhZ/cMd24H/1p24+EYIALc7pBR5qbGpW20 oxHUWGfVaERizkhvuDAbO/n5sXUB5QHbh6MMwe2tn3TCWLstnvRhvJDR9ahKx7gm EChB4sIAKM/npUQge6ljLqP61m88p7LpnIVV6gF4PC0Wkyz8g2iSMjVwFv4XEBYZ /PNWXa4QLlNmqksrcQ7pYKZObYSjU8lNAEsCmtRy8PbBTvWq2f+YB9kcc79byFIs W0bhSI7x2iaicU24UC7FJAo4bSFKNZ8LDSEMZhu7gWhFxJ7wVsyxk6/RrZkptdCx z/DNqo9/Ggs4UJ9vo4cfCoX0723bejT0VG1K/EuYxWAXYOlNuICYIMQVNiZhkoo= =2/BD -----END PGP SIGNATURE----- From duda.piotr at gmail.com Tue May 31 02:35:25 2016 From: duda.piotr at gmail.com (Piotr Duda) Date: Tue, 31 May 2016 08:35:25 +0200 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: References: <20160531030810.GR12028@ando.pearwood.info> Message-ID: 3 years ago, I've proposed syntax def name = expr for binding names to arbitrary objects https://mail.python.org/pipermail/python-ideas/2013-May/020501.html 2016-05-31 6:57 GMT+02:00 Joshua Morton : > I like the arrow idea, another option is a syntax more in line with the > one being discussed for dict unpacking, something like adding a __define__ > dunder that returns a tuple of (name, value). So that say, Typevar's > __define__ returns the tuple `(self.__name__, self)`. Which is then > converted into an assignment. The downside of this being that its > assignment overloading(which can be abused), but at least it can only be > done when specifically asked for. The other downside is that the syntax > > define TypeVar('T') # and I assume we can also declare a list of them > def ident(t: T) -> T: > return t > > looks both foreign and a bit magical. Also new keyword, but I have a > feeling that the same keyword could probably be used here and with dict > unpacking. (the *really* big downside is for a library implementer to do > something devious like have __define__ always return ("constant", self), or > generally something that wasn't clear to the caller, because we get names > appearing out of no where with no clear history, but my argument would be > that we should trust libraries not to do that and say that it is a very bad > thing) > > The upside is that it avoids the problem of Steven's implementation, since > the name is decided internally by the callable. > > Like with his example, define `Typevar('7')` and similar invalid names > would throw an error, in this case probably a TypeError. > > --Josh > > > On Mon, May 30, 2016 at 11:08 PM Steven D'Aprano > wrote: > >> On Mon, May 30, 2016 at 06:16:26PM -0700, Guido van Rossum wrote: >> >> [...] >> > T = TypeVar('T') >> > x = Symbol('x') >> > >> > I'm sure this is not a new idea, but so far I've always thought that >> > this is pretty esoteric and the approach here is good enough. But >> > maybe we can actually do better, and whatever solution we come up with >> > might also be useful for extracting attributes from an object or >> > values from a mapping? >> >> This comes up a lot and it would be nice to clean it up. >> >> T = type('T', bases, ns) >> Record = namedtuple('Record', fields) >> >> It came up in the discussion on dict unpacking: >> >> a, b, c = **dict # unpacks keys 'a', 'b', 'c' >> >> which was rejected as too magical. But maybe it will seem less magical >> if we have a special assignment operator that takes the left hand >> symbol(s) >> and copies them to the right? >> >> How do you feel about an arrow operator? >> >> >> T -> TypeVar() >> x -> Symbol() >> T -> type(bases, ns) >> Record -> namedtuple(fields) >> >> In general: >> >> name [, name, ...] -> callable(...) >> >> where "callable" is some expression, e.g. a name, a dot lookup, etc. >> >> # okay >> name -> sympy.Symbol() >> >> >> >> The easy case >> ------------- >> >> All assignment targets are plain names. >> >> The arrow operator takes the names, turns them into strings, and passes >> them to the callable on the right as a tuple of strings, given as the >> first >> positional argument. >> >> Since the most common case will be a single target, I think it is worth >> while to treat it as a special case and pass just a string: >> >> T -> Type() # like Type('T') >> >> a, b, c -> Spam(arg) # like Spam(('a', 'b', 'c'), arg) >> >> The callable being called doesn't need to know or care whether it is >> being called with the -> or = assignment. It just receives the name(s) >> as the first argument. >> >> This will mean that callables that take the name as (say) the second >> argument cannot be used with this syntax. >> >> Using the arrow operator with arbitrary expressions on the >> right will be a SyntaxError: >> >> x -> y + 1 >> my_name -> spam*f() >> my_name -> f() or g() >> >> I reject that third example as it is unclear whether both f and g, or >> only f, get the name as first argument. >> >> But this is allowed: >> >> my_name -> f(g()) # like f('my_name', g()) >> >> If f isn't *actually* a callable, you get a runtime TypeError as usual. >> >> >> The hard case >> ------------- >> >> Assignment target(s) is not a plain name. >> >> spam[2].eggs().attr -> Symbol() >> >> >> I'm going to take the cowards way out and just say "we don't allow >> that". >> >> >> -- >> Steve >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- ???????? ?????? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue May 31 03:46:57 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 31 May 2016 09:46:57 +0200 Subject: [Python-ideas] Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects In-Reply-To: <574D20D9.7060504@gmail.com> References: <574D20D9.7060504@gmail.com> Message-ID: <574D4171.3060301@egenix.com> On 31.05.2016 07:27, Matthew Tanous wrote: > > Currently, the use of the (*) operator on a list is to duplicate a list > by creating multiple references to the same object. While this works > intuitively for immutable objects (like [True] * 5) as these immutable > references are replaced when the list is assigned to, it makes the > operator nigh unusable for mutable objects. > > The most obvious case is when the operator is duplicated in a sequence > like this: > > arr = [[True] * 5] * 5 > > This does not create a matrix-like arrangement of the immutable truth > variable, but instead creates a list of 5 references to the same list, > such that a following assignment like arr[2][3] = False will not change > just that one index, but every 4th element of each list in the outer list. > > ... > > It is my opinion that the sequence repetition operator should be > modified to make copies of the objects it is repeating, rather than > copying references alone. I believe this would both be more intuitive > from a semantic point of view and more useful for the developer. > > This would change the operator in a way that is mostly unseen in current > usage ([5] * 3 would still result in [5, 5, 5]) while treating mutable > nesting in a way that is more understandable from the apparent intent of > the syntax construction. Some questions: * How would you determine whether a list element is mutable or not ? * How would you copy the elements ? * For which object types would you want to change the behavior ? I agree that the repeat operator can sometimes create confusing and unwanted object structures if not used correctly, but it's main purpose it to repeat the already existing objects, not to copy them, so the current behavior still is conceptually correct. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 31 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From p.f.moore at gmail.com Tue May 31 04:06:02 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 31 May 2016 09:06:02 +0100 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <20160531030810.GR12028@ando.pearwood.info> References: <20160531030810.GR12028@ando.pearwood.info> Message-ID: On 31 May 2016 at 04:08, Steven D'Aprano wrote: > How do you feel about an arrow operator? > > > T -> TypeVar() > x -> Symbol() > T -> type(bases, ns) > Record -> namedtuple(fields) I like this. Simplifying it, how about name [, name ...] -> callable which is equivalent to name [, name ...] = callable(('name' [, 'name' ... ])) I.e., the RHS is a callable, and we call it with the names. No need to inject an argument into an existing call. This avoids any confusion over the RHS being "too complex". For callables that have extra arguments, just use functools.partial. I agree with the special case of a single name becoming a string (rather than a tuple of strings). If you want a 1-tuple, do "name, -> function" (the same trick as for unpacking assignment). As for the operator name, I like -> but I see why Guido might object because of the type signature usage (although is there actually any clash?) Alternatives could be := (from Pascal, although that doesn't have the implication of "sending the LHS to the right") or => (which might work, but is easily confused with >=/<=). If a 3-character symbol is acceptable, ->> might work. Keywords don't really sit right with me for this. Paul From stephen at xemacs.org Tue May 31 04:56:11 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 31 May 2016 17:56:11 +0900 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: References: <20160531030810.GR12028@ando.pearwood.info> Message-ID: <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> Paul Moore writes: > On 31 May 2016 at 04:08, Steven D'Aprano wrote: > > How do you feel about an arrow operator? > > > > T -> TypeVar() > > x -> Symbol() > > T -> type(bases, ns) > > Record -> namedtuple(fields) > > I like this. Well, it's cuter than Hello Kitty, what's not to like? But ... It give me no clue what it's supposed to mean. In math the forward arrow is often used to name a function type (including the domain preceding the arrow and the codomain following it in the name), although what the parentheses and their contents mean, I don't know. The codomain type is the value of TypeVar()? What's that? In Pascal and R, the reverse arrow T <- TypeVar() is used to mean assignment (which you could read as "T receives TypeVar()", but the implicit argument on the RHS is a double-take for me in both syntaxes -- the argument to TypeVar is not optional! IIRC, we have two syntaxes in Python itself that take the name of an identifier and reify it as a string (ie, inject it into a namespace): def and class. I know you don't think a keyword works for you, but either the recently reraised "def = " or perhaps "type : " make more sense to me right out of the box. Another problem is that none of these syntaxes (including the keyword- based ones) provides a clue as to whether the type is a distinct type or a type alias. It's not that one couldn't learn, but it doesn't seem very Pythonic to me: not conformant to EIBTI and creating ambiguity that forces the programmer to guess (or even more painful, read the doc!) I'm +1 for stopping the bikeshedding until we've all got a lot of stubfile reading under our belts. From p.f.moore at gmail.com Tue May 31 05:05:35 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 31 May 2016 10:05:35 +0100 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> Message-ID: On 31 May 2016 at 09:56, Stephen J. Turnbull wrote: > I know you don't think a keyword works for you, but > either the recently reraised "def = " or perhaps > "type : " make more sense to me right out of the box. I was thinking along the lines of "name callable", which I don't think works because it needs some "punctuation" to separate the name from the callable. But "def name = callable" (or some other preceding keyword combined with =) might work. I don't like "type" though, as the point here (I thought - see below) is to come up with a construct useful for more than just types. > I'm +1 for stopping the bikeshedding until we've all got a lot of > stubfile reading under our belts. If this was simply about type definitions, I'd agree. But I thought the point of Guido's post was that having seen two examples (TypeVar and Symbol) is there a more general approach that might cover these two cases as well as others? So just looking at the problem in terms of stub files isn't really the point here. Paul From code at lorenzquack.de Tue May 31 07:39:47 2016 From: code at lorenzquack.de (Lorenz) Date: Tue, 31 May 2016 11:39:47 +0000 (UTC) Subject: [Python-ideas] Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects References: <574D20D9.7060504@gmail.com> Message-ID: Matthew Tanous writes: > It is my opinion that the sequence repetition operator should be > modified to make copies of the objects it is repeating, rather than > copying references alone. I believe this would both be more intuitive > from a semantic point of view and more useful for the developer. > > This would change the operator in a way that is mostly unseen in current > usage ([5] * 3 would still result in [5, 5, 5]) while treating mutable > nesting in a way that is more understandable from the apparent intent of > the syntax construction. > Hi Matthew, I agree that when I was starting out with python this tripped me up a couple of times but Marc-Andre pointed out it is not necessarily easy to solve in a generic way. As a point of reference, I currently tend to do this via generators arr = list(Foo() for i in range(5)) or list comprehensions arr = [Foo() for i in range(5)] note that this works for the mutable and immutable case. Kind Regards, Lorenz From srkunze at mail.de Tue May 31 08:55:04 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 31 May 2016 14:55:04 +0200 Subject: [Python-ideas] Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects In-Reply-To: References: <574D20D9.7060504@gmail.com> Message-ID: <574D89A8.3060306@mail.de> On 31.05.2016 13:39, Lorenz wrote: > Hi Matthew, > > I agree that when I was starting out with python this tripped me up a couple > of times but Marc-Andre pointed out it is not necessarily easy to solve in a > generic way. I can only agree here. Even today, despite knowing the fact, it's causing some headaches in some cases. However, I think that's the reason why it's not so easy to do in a general manner. Maybe, somebody has a good idea how to. Sven > > As a point of reference, I currently tend to do this via generators > arr = list(Foo() for i in range(5)) > or list comprehensions > arr = [Foo() for i in range(5)] > > note that this works for the mutable and immutable case. > > Kind Regards, > Lorenz > From srkunze at mail.de Tue May 31 08:58:22 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 31 May 2016 14:58:22 +0200 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> Message-ID: <574D8A6E.5040609@mail.de> On 31.05.2016 11:05, Paul Moore wrote: > If this was simply about type definitions, I'd agree. But I thought > the point of Guido's post was that having seen two examples (TypeVar > and Symbol) is there a more general approach that might cover these > two cases as well as others? So just looking at the problem in terms > of stub files isn't really the point here. I don't know why this needs special syntax anyway. Maybe, somebody could explain. Even Guido said it was just for procrastinating. So, I don't give much weight to it. Anyway, what's the difference between: a = and def a = From srkunze at mail.de Tue May 31 09:01:48 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 31 May 2016 15:01:48 +0200 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> Message-ID: <574D8B3C.5060406@mail.de> And here we go again. Sorry for posting unfinished stuff: On 31.05.2016 11:05, Paul Moore wrote: > If this was simply about type definitions, I'd agree. But I thought > the point of Guido's post was that having seen two examples (TypeVar > and Symbol) is there a more general approach that might cover these > two cases as well as others? So just looking at the problem in terms > of stub files isn't really the point here. > I don't know why this needs special syntax anyway. Maybe, somebody could explain. Even Guido said it was just for procrastinating. So, I don't give much weight to it. Anyway, what's the difference between: a = and def a = ? Both evaluate RHS and assign the result to a name (if not already defined, define the name) The only benefit I can see is some a syntax like: def a which defines a name in the scope without content. If that's useful? Maybe. Best, Sven From greg.ewing at canterbury.ac.nz Tue May 31 09:02:57 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 01 Jun 2016 01:02:57 +1200 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: References: <20160531030810.GR12028@ando.pearwood.info> Message-ID: <574D8B81.9070108@canterbury.ac.nz> > On 31 May 2016 at 04:08, Steven D'Aprano wrote: > >>T -> TypeVar() >>x -> Symbol() >>T -> type(bases, ns) >>Record -> namedtuple(fields) The arrow seems back to front. It should point towards the name being assigned. T <- TypeVar() x <- Symbol() T <- type(bases, ns) Record <- namedtuple(fields) Also, if the RHS is going to be called as part of the <- operation, shouldn't the first two just be T <- TypeVar x <- Symbol The main drawback is that '<-' doesn't suggest in any way what's going on. An alternative might be def x = Symbol since 'def' has the precedent of attaching the name being bound to the object being created. -- Greg From cyscoyote at gmail.com Tue May 31 09:04:11 2016 From: cyscoyote at gmail.com (Yongsheng Cheng) Date: Tue, 31 May 2016 13:04:11 +0000 Subject: [Python-ideas] (no subject) Message-ID: to begin with: i a deep python fun i want to suguest python a feature to support Fuctional Program :,here is the toy code i need a more powerful decrote in moudle functools instead of functools.partial,thanks do you think it is a nice idea? waitting for your reply def minpara(func_1,args): num=0 try: func_1(*args) except Exception as error: num,given=re.findall(r'\d+',str(error))[:2] if num: #return int(num),int(given) return (num,given) else: #return func(*args) return (0,0) def curried(func): #@wraps(func) def new_func(*args): num,given=minpara(func,args) #print num,given if (num,given)==(0,0): return func(*args) else: return curried(functools.partial(func,*args)) return new_func @curried def sun_1(a,b,c): return a+b+c @curried def sun_2(a,b,c): return a*b*c if __name__=="__main__": print sun_2(1,)(2)(44) print sun_1(1,)(2)(44) print sun_2(1,2)(23) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- def minpara(func_1,args): num=0 try: func_1(*args) except Exception as error: num,given=re.findall(r'\d+',str(error))[:2] if num: #return int(num),int(given) return (num,given) else: #return func(*args) return (0,0) def curried(func): #@wraps(func) def new_func(*args): num,given=minpara(func,args) #print num,given if (num,given)==(0,0): return func(*args) else: return curried(functools.partial(func,*args)) return new_func @curried def sun_1(a,b,c): return a+b+c @curried def sun_2(a,b,c): return a*b*c if __name__=="__main__": print sun_2(1,)(2)(44) print sun_1(1,)(2)(44) print sun_2(1,2)(23) From michael.selik at gmail.com Tue May 31 09:13:32 2016 From: michael.selik at gmail.com (Michael Selik) Date: Tue, 31 May 2016 13:13:32 +0000 Subject: [Python-ideas] Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects In-Reply-To: <574D89A8.3060306@mail.de> References: <574D20D9.7060504@gmail.com> <574D89A8.3060306@mail.de> Message-ID: On Tue, May 31, 2016 at 8:55 AM Sven R. Kunze wrote: > On 31.05.2016 13:39, Lorenz wrote: > > I agree that when I was starting out with python this tripped me up a > couple of times but Marc-Andre pointed out it is not necessarily easy to > solve in a generic way. > > However, I think that's the reason why it's not so easy to do in a > general manner. > I think comprehensions are doing pretty well at solving the general case. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue May 31 09:22:25 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 31 May 2016 23:22:25 +1000 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> Message-ID: <20160531132225.GS12028@ando.pearwood.info> On Tue, May 31, 2016 at 05:56:11PM +0900, Stephen J. Turnbull wrote: > Paul Moore writes: > > On 31 May 2016 at 04:08, Steven D'Aprano wrote: > > > How do you feel about an arrow operator? > > > > > > T -> TypeVar() > > > x -> Symbol() > > > T -> type(bases, ns) > > > Record -> namedtuple(fields) > > > > I like this. > > Well, it's cuter than Hello Kitty, what's not to like? But ... > > It give me no clue what it's supposed to mean. That's harsh. No clue *at all*? It shouldn't be so difficult to see an arrow as assignment. Even if you can't *guess* what it means, at least it should be easy enough to understand and remember once explained. The arrow operator represents a value "moving into" a name, or a name becoming a value: x <- 999 # move 999 into the slot called "x" x -> 999 # x becomes 999 As you yourself pointed out, R uses <- as the assignment operator (as do OCaml, F#, but not Pascal, as it uses := instead). I've seen plenty of pseudo-code using <- as the assignment operator, and *lots* of whiteboard discussions using arrows in either direction for assignment. I've never had somebody claim to be perplexed or confused by what I mean if I write `foo -> value`. So I think there are plenty of clues, even if the arrow points the wrong way for R users. There's another analogy here that will help: x -> spam() # copy 'x' to the RHS I don't expect anyone to intuit what -> does, but I think that it will be easy to learn and memorable -- more so than @ for decorators, ^ for bitwise xor, or {} for dicts. Of course Python cannot use <- because it's ambiguous with (less than) (unary minus), e.g.: `x<-f()` could mean either `x <- f()` or `x < -f()`. > In math the forward > arrow is often used to name a function type (including the domain > preceding the arrow and the codomain following it in the name), > although what the parentheses and their contents mean, I don't know. > The codomain type is the value of TypeVar()? What's that? I don't think that the average Python programmer is going to be thinking of domains and codomains when they see an arrow :-) Do you think that was a problem for C++ ? But if it makes you happy, perhaps => instead :-) > In Pascal and R, the reverse arrow > > T <- TypeVar() > > is used to mean assignment (which you could read as "T receives > TypeVar()", but the implicit argument on the RHS is a double-take for > me in both syntaxes -- the argument to TypeVar is not optional! And neither is the `self` argument to methods. But we learn that self is implicitly provided as if by magic :-) Another similar example of magic is decorator syntax: def decorator(func): ... which again is not optional and gets provided implicitly by the interpreter. The thing is, regardless of whether we use a symbol or a keyword, the functionality here is going to be magical. That's its point: it is to avoid the need to write the name twice. > IIRC, we have two syntaxes in Python itself that take the name of an > identifier and reify it as a string (ie, inject it into a namespace): > def and class. Plus: - import spam - from module import spam - for spam in ... - @decorator - and of course regular assignment. So there's a nice mix of keywords and symbols. > I know you don't think a keyword works for you, but > either the recently reraised "def = " or perhaps > "type : " make more sense to me right out of the box. Don't think of the RHS as necessarily a type expression. Guido has already given one other example: x = sympy.Symbol('x') This is a proposal to solve the general problem of passing a name from the left hand side to the right just prior to doing an assignment, regardless of what the purpose of that name is. So "type" is right out: the motivating use-case might be TypeVar, but this isn't limited to types by any means. I could probably learn to live with def x = sympy.Symbol() if I really needed to, put it looks strange to use def for defining something which isn't necessarily a function. And it's a tad wordy: four extra keypresses, in order to avoid three. But most critically, "def x =" doesn't really suggest copying the "x" onto the RHS in any way. > Another problem is that none of these syntaxes (including the keyword- > based ones) provides a clue as to whether the type is a distinct type > or a type alias. The type will be whatever TypeVar returns. That's no different from the status quo `T = TypeVar('T')`. -- Steve From python at mrabarnett.plus.com Tue May 31 09:33:09 2016 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 31 May 2016 14:33:09 +0100 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> Message-ID: <76a47ed1-4fad-fc2d-f535-ed8df86b6a29@mrabarnett.plus.com> On 2016-05-31 09:56, Stephen J. Turnbull wrote: [snip] > It give me no clue what it's supposed to mean. In math the forward > arrow is often used to name a function type (including the domain > preceding the arrow and the codomain following it in the name), > although what the parentheses and their contents mean, I don't know. > The codomain type is the value of TypeVar()? What's that? In Pascal > and R, the reverse arrow > > T <- TypeVar() > > is used to mean assignment (which you could read as "T receives > TypeVar()", but the implicit argument on the RHS is a double-take for > me in both syntaxes -- the argument to TypeVar is not optional! > FYI, in Pascal the assignment operator is ":=". The assignment operator in APL is "?". From contrebasse at gmail.com Tue May 31 09:36:31 2016 From: contrebasse at gmail.com (Joseph Martinot-Lagarde) Date: Tue, 31 May 2016 13:36:31 +0000 (UTC) Subject: [Python-ideas] Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects References: <574D20D9.7060504@gmail.com> <574D89A8.3060306@mail.de> Message-ID: > I can only agree here. Even today, despite knowing the fact, it's > causing some headaches in some cases. How about raising an exception if mutable objects are in the list ? It's a pretty big backward incompatible change, but: Pros: - it "works" in the general case - it avoids hidden bugs - it avoids a transition period between new and old behavior - it is possible to propose using a list comprehension instead in the exception message Cons: - backward incompatible (but changing the behavior of * is backward incompatible anyway) - maybe there are useful use cases of duplicating the reference ? Note that the same arguments could be taken for mutable default arguments, but an actual use case is to create a cache variable. Joseph From steve at pearwood.info Tue May 31 09:37:17 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 31 May 2016 23:37:17 +1000 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <574D8B3C.5060406@mail.de> References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> <574D8B3C.5060406@mail.de> Message-ID: <20160531133717.GT12028@ando.pearwood.info> On Tue, May 31, 2016 at 03:01:48PM +0200, Sven R. Kunze wrote: > And here we go again. Sorry for posting unfinished stuff: > > On 31.05.2016 11:05, Paul Moore wrote: > >If this was simply about type definitions, I'd agree. But I thought > >the point of Guido's post was that having seen two examples (TypeVar > >and Symbol) is there a more general approach that might cover these > >two cases as well as others? So just looking at the problem in terms > >of stub files isn't really the point here. > > > > I don't know why this needs special syntax anyway. Maybe, somebody could > explain. Any time you have an object that needs to know its own name, you have to provide it as a string, AND as an assignment target: T = TypeVar('T') x = sympy.Symbol('x') myclass = namedtuple("myclass", fields) klass = type('klass', bases, ns) The only exceptions are when you can use compiler magic do to it for you. We don't have to write these: math = import math func = def func(arg): ... MyClass = class MyClass(Parent): ... because the compiler does it for us. Likewise we have @ decorator syntax to avoid writing the function name three times: def spam(): ... spam = decorate(spam) This solves the same problem for ordinary assignment: how to get the name or names on the left hand side over to the right hand side without re-typing them as strings? > Even Guido said it was just for procrastinating. So, I don't > give much weight to it. This comes up from time to time. It was one of the motives for adding @ decorator syntax, so maybe its the right time for it now. > Anyway, what's the difference between: > > a = > > and > > def a = > > ? > > Both evaluate RHS and assign the result to a name (if not already > defined, define the name) The "def a = ..." is not my suggested syntax, so I can't tell you exactly what it will do, but *my* suggested syntax is: name -> Function(args) will be expanded to: name = Function('name', args) by the compiler. The "def a = ..." syntax will probably be similar. Somehow, in some fashion, the name "a" on the LHS will be passed as a string to something on the RHS. -- Steve From duda.piotr at gmail.com Tue May 31 09:50:41 2016 From: duda.piotr at gmail.com (Piotr Duda) Date: Tue, 31 May 2016 15:50:41 +0200 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <574D8B3C.5060406@mail.de> References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> <574D8B3C.5060406@mail.de> Message-ID: 2016-05-31 15:01 GMT+02:00 Sven R. Kunze : > > Anyway, what's the difference between: > > a = > > and > > def a = > def a = will bind name 'a' to , there was proposal to make it equivalent for a = .__def__('a', , ) -- ???????? ?????? -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue May 31 10:04:16 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 1 Jun 2016 00:04:16 +1000 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <574D8B81.9070108@canterbury.ac.nz> References: <20160531030810.GR12028@ando.pearwood.info> <574D8B81.9070108@canterbury.ac.nz> Message-ID: <20160531140416.GU12028@ando.pearwood.info> On Wed, Jun 01, 2016 at 01:02:57AM +1200, Greg Ewing wrote: > >On 31 May 2016 at 04:08, Steven D'Aprano wrote: > > > >>T -> TypeVar() > >>x -> Symbol() > >>T -> type(bases, ns) > >>Record -> namedtuple(fields) > > The arrow seems back to front. It should point towards > the name being assigned. See my response to Stephen, but in a nutshell, no, we can't use <- because that's already legal for (less than) (unary minus). Besides, the concepts I'm hinting at with -> are: (1) "x becomes the value of the right hand side" (2) "copy the name 'x' over to the right hand side" > T <- TypeVar() > x <- Symbol() > T <- type(bases, ns) > Record <- namedtuple(fields) > > Also, if the RHS is going to be called as part of > the <- operation, shouldn't the first two just be > > T <- TypeVar > x <- Symbol No, because in the general case they might require more than one argument. Besides, I want to make it clear that the right hand side is being called. And I don't want to make the same mistake as decorator syntax. I'm not saying that decorator syntax is bad (it's been really successful!) but the choice to leave off the parens does make things a bit more complicated in some cases. E.g. if your decorator wants to take extra arguments, you have to write a decorator factory: def factory(args): process(args) def decorator(func): @wraps(func) def inner(*args, **kwargs): ... return inner return decorator @factory(args) def spam(): ... That's not awful, but it does lead to a trap: what if you make the args optional? Then you can leave them out when calling the factory: @factory() def spam(): ... but in practice that tends to be a bug magnet, because people invariably forget the parens. And *that* leads to spam being bound to the decorator itself, which is wrong, but you don't find out until you actually call spam. I don't think this suggested syntax will be used as often as decorators, so I don't think it's worth the risk of complicating matters by making the parens optional. Keep them explicit and then you never need to worry about whether they are needed or not, they're always needed. > The main drawback is that '<-' doesn't suggest in any > way what's going on. Well, apart from the languages and pseudo-code that already uses <- as an assignment operator :-) > An alternative might be > > def x = Symbol > > since 'def' has the precedent of attaching the name > being bound to the object being created. I never thought of it like that before. To me, I don't think def is a good match because def doesn't have a left and right hand side. Assignment does. I know that *technically* def is an assignment (a name binding) but it doesn't look like one. It looks like a definition or declaration. -- Steve From contrebasse at gmail.com Tue May 31 10:12:05 2016 From: contrebasse at gmail.com (Joseph Martinot-Lagarde) Date: Tue, 31 May 2016 14:12:05 +0000 (UTC) Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name References: <20160531030810.GR12028@ando.pearwood.info> Message-ID: Steven D'Aprano writes: > How do you feel about an arrow operator? > > T -> TypeVar() > x -> Symbol() > T -> type(bases, ns) > Record -> namedtuple(fields) > > In general: > > name [, name, ...] -> callable(...) > > where "callable" is some expression, e.g. a name, a dot lookup, etc. > It could work with indexing, like: key -> my_dict[] instead of key = my_dict[key] Or even attribute lookup with a trailing dot: attr -> my_object. instead of attr = my_object.attr Joseph From steve at pearwood.info Tue May 31 10:16:25 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 1 Jun 2016 00:16:25 +1000 Subject: [Python-ideas] Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects In-Reply-To: References: <574D20D9.7060504@gmail.com> <574D89A8.3060306@mail.de> Message-ID: <20160531141625.GV12028@ando.pearwood.info> On Tue, May 31, 2016 at 01:36:31PM +0000, Joseph Martinot-Lagarde wrote: > > I can only agree here. Even today, despite knowing the fact, it's > > causing some headaches in some cases. > > How about raising an exception if mutable objects are in the list ? -1 It's a gratuitous breakage that cannot solve the problem, because you cannot tell in advance which objects are mutable and which are not. The best you can do is recognise known built-ins. ("list is mutable, tuple is not, except when it contains a mutable item, but mymodule.MySequence may or may not be, there's no way to tell in advance.") > - maybe there are useful use cases of duplicating the reference ? Absolutely. That's a standard way of grouping items taken from an iterable: py> it = iter("hello world!") py> for a, b, c in zip(*[it]*3): # groups of three ... print(a, b, c) ... h e l l o w o r l d ! This wouldn't work if the iterators were three independent copies. -- Steve From steve at pearwood.info Tue May 31 10:24:38 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 1 Jun 2016 00:24:38 +1000 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: References: <20160531030810.GR12028@ando.pearwood.info> Message-ID: <20160531142438.GW12028@ando.pearwood.info> On Tue, May 31, 2016 at 02:12:05PM +0000, Joseph Martinot-Lagarde wrote: > It could work with indexing, like: > > key -> my_dict[] > > instead of > > key = my_dict[key] That would have to be key = my_dict['key'] With the possible exception of the typing module, I don't see when that would be useful. There are proven uses for passing a name to a function. Any time that an object needs to know its own name, you have to pass it to the constructor. But I can't think of any cases where you would want to do something similar with item lookup. > Or even attribute lookup with a trailing dot: > > attr -> my_object. That fails the "grit on Uncle Timmy's monitor" test. The trailing dot is easy to miss. -- Steve From srkunze at mail.de Tue May 31 10:37:26 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 31 May 2016 16:37:26 +0200 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <20160531133717.GT12028@ando.pearwood.info> References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> <574D8B3C.5060406@mail.de> <20160531133717.GT12028@ando.pearwood.info> Message-ID: <574DA1A6.8090906@mail.de> On 31.05.2016 15:37, Steven D'Aprano wrote: > because the compiler does it for us. Likewise we have @ decorator syntax > to avoid writing the function name three times: > > def spam(): > ... > > spam = decorate(spam) > > > This solves the same problem for ordinary assignment: how to get the > name or names on the left hand side over to the right hand side without > re-typing them as strings? So, why don't do it for all assignments without a special syntax? Btw. there would be huge use-case for Django development. So, given your explanation, it sounds useful. Sven From p.f.moore at gmail.com Tue May 31 10:39:14 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 31 May 2016 15:39:14 +0100 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <20160531142438.GW12028@ando.pearwood.info> References: <20160531030810.GR12028@ando.pearwood.info> <20160531142438.GW12028@ando.pearwood.info> Message-ID: On 31 May 2016 at 15:24, Steven D'Aprano wrote: > On Tue, May 31, 2016 at 02:12:05PM +0000, Joseph Martinot-Lagarde wrote: > >> It could work with indexing, like: >> >> key -> my_dict[] >> >> instead of >> >> key = my_dict[key] > > That would have to be > > key = my_dict['key'] > > > With the possible exception of the typing module, I don't see when that > would be useful. It's reminiscent of the recent "dictionary unpacking" discussion. But it's doable without special-casing [] anyway: key -> my_dict.get [I still prefer "call the RHS with the name of the LHS as the argument" over "inject the name of the LHS as the first argument of the call present on the RHS", btw - mostly because it has fewer restrictions and special cases to define] Paul From cyscoyote at gmail.com Tue May 31 10:48:54 2016 From: cyscoyote at gmail.com (Yongsheng Cheng) Date: Tue, 31 May 2016 14:48:54 +0000 Subject: [Python-ideas] functional programming curried Message-ID: i want to suguest python a feature to support Fuctional Program :,here is the toy code i need a more powerful decrote in moudle functools instead of functools.partial,thanks do you think it is a nice idea? waitting for your reply,import port some features like scala. def minpara(func_1,args): num=0 try: func_1(*args) except Exception as error: num,given=re.findall(r'\d+',str(error))[:2] if num: #return int(num),int(given) return (num,given) else: #return func(*args) return (0,0) def curried(func): #@wraps(func) def new_func(*args): num,given=minpara(func,args) #print num,given if (num,given)==(0,0): return func(*args) else: return curried(functools.partial(func,*args)) return new_func @curried def sun_1(a,b,c): return a+b+c @curried def sun_2(a,b,c): return a*b*c if __name__=="__main__": print sun_2(1,)(2)(44) print sun_1(1,)(2)(44) print sun_2(1,2)(23) _______________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- def minpara(func_1,args): num=0 try: func_1(*args) except Exception as error: num,given=re.findall(r'\d+',str(error))[:2] if num: #return int(num),int(given) return (num,given) else: #return func(*args) return (0,0) def curried(func): #@wraps(func) def new_func(*args): num,given=minpara(func,args) #print num,given if (num,given)==(0,0): return func(*args) else: return curried(functools.partial(func,*args)) return new_func @curried def sun_1(a,b,c): return a+b+c @curried def sun_2(a,b,c): return a*b*c if __name__=="__main__": print sun_2(1,)(2)(44) print sun_1(1,)(2)(44) print sun_2(1,2)(23) From steve at pearwood.info Tue May 31 11:08:58 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 1 Jun 2016 01:08:58 +1000 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: References: <20160531030810.GR12028@ando.pearwood.info> <20160531142438.GW12028@ando.pearwood.info> Message-ID: <20160531150858.GX12028@ando.pearwood.info> On Tue, May 31, 2016 at 03:39:14PM +0100, Paul Moore wrote: > [I still prefer "call the RHS with the name of the LHS as the > argument" over "inject the name of the LHS as the first argument of > the call present on the RHS", btw - Do I understand that you think that the RHS should be limited to callables that take a single argument only? So far we have four real use-cases: T = TypeVar('T') x = sympy.Symbol('x') cls = type('cls', bases, ns) Record = nametuple('Record', fields) You would only support the first two cases and leave the others with the status quo? That seems like a strange restriction to make. What problem do you think the other two use-cases would cause that the first two don't? The fundamental use-case here is "any object that needs to know its own name". Most objects need more than just a name. If you're worried that it will be confusing that we define a callable to take (say) three arguments: class Widget: def __init__(self, name, foo, bar): ... but then call it with only two: spanner -> Widget(foo=1, bar=2) I think you're worrying over nothing. In fact, I played a trick on you... I said we define a callable that takes three arguments, but in fact I defined it with FOUR. We're constantly writing methods that take an extra argument, self, that is provided by the compiler. A similar argument applies to decorators: # defined with a single argument def decorator(func): ... # called with no explicit argument (or even parens!) @decorator And both cases can be used without compiler magic, using an unbound method versus a bound method, decorator syntax versus calling the decorator by hand. So while this is magic, it is the same sort of magic we deal with frequently in Python. So frequently that we forget just how magical it is that the "self" param is provided automatically for us. -- Steve From mertz at gnosis.cx Tue May 31 11:27:28 2016 From: mertz at gnosis.cx (David Mertz) Date: Tue, 31 May 2016 08:27:28 -0700 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> Message-ID: This is bikeshedding a bit, but a keyword that looks good to me: expose Typevar as T expose Symbol as X On May 31, 2016 2:06 AM, "Paul Moore" wrote: > On 31 May 2016 at 09:56, Stephen J. Turnbull wrote: > > I know you don't think a keyword works for you, but > > either the recently reraised "def = " or perhaps > > "type : " make more sense to me right out of the box. > > I was thinking along the lines of "name callable", > which I don't think works because it needs some "punctuation" to > separate the name from the callable. > > But "def name = callable" (or some other preceding keyword combined > with =) might work. I don't like "type" though, as the point here (I > thought - see below) is to come up with a construct useful for more > than just types. > > > I'm +1 for stopping the bikeshedding until we've all got a lot of > > stubfile reading under our belts. > > If this was simply about type definitions, I'd agree. But I thought > the point of Guido's post was that having seen two examples (TypeVar > and Symbol) is there a more general approach that might cover these > two cases as well as others? So just looking at the problem in terms > of stub files isn't really the point here. > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Tue May 31 11:35:12 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 31 May 2016 10:35:12 -0500 Subject: [Python-ideas] A function currying decorator ((Was: no subject)) Message-ID: So you're referring to function currying, correct? I changed the thread title to reflect that. -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ On May 31, 2016 8:04 AM, "Yongsheng Cheng" wrote: to begin with: i a deep python fun i want to suguest python a feature to support Fuctional Program :,here is the toy code i need a more powerful decrote in moudle functools instead of functools.partial,thanks do you think it is a nice idea? waitting for your reply def minpara(func_1,args): num=0 try: func_1(*args) except Exception as error: num,given=re.findall(r'\d+',str(error))[:2] if num: #return int(num),int(given) return (num,given) else: #return func(*args) return (0,0) def curried(func): #@wraps(func) def new_func(*args): num,given=minpara(func,args) #print num,given if (num,given)==(0,0): return func(*args) else: return curried(functools.partial(func,*args)) return new_func @curried def sun_1(a,b,c): return a+b+c @curried def sun_2(a,b,c): return a*b*c if __name__=="__main__": print sun_2(1,)(2)(44) print sun_1(1,)(2)(44) print sun_2(1,2)(23) _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue May 31 11:43:27 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 1 Jun 2016 01:43:27 +1000 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> Message-ID: <20160531154327.GY12028@ando.pearwood.info> On Tue, May 31, 2016 at 08:27:28AM -0700, David Mertz wrote: > This is bikeshedding a bit, but a keyword that looks good to me: > > expose Typevar as T > expose Symbol as X "expose" sounds like it is taking something hidden or private and exposing it to the public. It doesn't give any hint that it takes the name T and uses it as an argument to Typevar. I would take it as meaning: take the (possibly internal or private) thing Typevar, and bind it to the name T with no hint that Typevar was being called. -- Steve From p.f.moore at gmail.com Tue May 31 11:46:51 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 31 May 2016 16:46:51 +0100 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <20160531150858.GX12028@ando.pearwood.info> References: <20160531030810.GR12028@ando.pearwood.info> <20160531142438.GW12028@ando.pearwood.info> <20160531150858.GX12028@ando.pearwood.info> Message-ID: On 31 May 2016 at 16:08, Steven D'Aprano wrote: > On Tue, May 31, 2016 at 03:39:14PM +0100, Paul Moore wrote: > >> [I still prefer "call the RHS with the name of the LHS as the >> argument" over "inject the name of the LHS as the first argument of >> the call present on the RHS", btw - > > Do I understand that you think that the RHS should be limited to > callables that take a single argument only? Yes. Or more specifically, expressions that evaluate to a callable. Because as you said in your original post, once you start allowing injection into anything more complicated than a "simple call" the whole thing gets messy. It's possible you could somehow restrict what's allowed, but not easy - IIRC, the complications around what was originally allowed in the position of a decorator were similar. Also, needing to have the "wrong" number of arguments on the RHS seems like it would lead to confusion: T => TypeVar() x => sympy.Symbol() cls => type(bases, ns) Record => nametuple(fields) Compare the calls on the RHS to the documentation. It'd probably also play havoc with IDEs that do signature introspection. > So far we have four real use-cases: > > T = TypeVar('T') > x = sympy.Symbol('x') > cls = type('cls', bases, ns) > Record = nametuple('Record', fields) > > You would only support the first two cases and leave the others with the > status quo? That seems like a strange restriction to make. What problem > do you think the other two use-cases would cause that the first two > don't? The idea was that the second pair can easily be handled using functools.partial. So there's no loss of functionality, just a (small) inconvenience. Having said that, I've just checked functools.partial, and it doesn't appear to allow for inserting an argument at the *start* of the argument list. Bah. But you could still do def mktype(bases, ns): return lambda cls: type(cls, bases, ns) def mknamedtuple(fields) return lambda name: namedtuple(name, fields) cls => mktype(bases, ns) Record => mknamedtuple(fields) ... and if this were a language feature, there's nothing stopping mknamedtuple being added to the collections module (and mktype somewhere, maybe operator). But conceded, it's a limitation. My feeling is that the benefit in terms of simpler semantics is worth it, but I can see your POV as well. Paul From cyscoyote at gmail.com Tue May 31 11:59:07 2016 From: cyscoyote at gmail.com (Yongsheng Cheng) Date: Tue, 31 May 2016 15:59:07 +0000 Subject: [Python-ideas] A function currying decorator ((Was: no subject)) In-Reply-To: References: Message-ID: Yes, that's what I meant. thanks and do you think we need such decorate in funtools? wiatting for reply Ryan Gonzalez ?2016?5?31??? ??11:35??? > So you're referring to function currying, correct? I changed the thread > title to reflect that. > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your > program. Something?s wrong. > http://kirbyfan64.github.io/ > On May 31, 2016 8:04 AM, "Yongsheng Cheng" wrote: > > to begin with: i a deep python fun > i want to suguest python a feature to support Fuctional Program :,here is > the toy code i need a more powerful decrote in moudle functools instead of > functools.partial,thanks do you think it is a nice idea? > waitting for your reply > > def minpara(func_1,args): > num=0 > try: > func_1(*args) > except Exception as error: > num,given=re.findall(r'\d+',str(error))[:2] > if num: > #return int(num),int(given) > return (num,given) > else: > #return func(*args) > return (0,0) > > def curried(func): > #@wraps(func) > def new_func(*args): > num,given=minpara(func,args) > #print num,given > if (num,given)==(0,0): > return func(*args) > else: > return curried(functools.partial(func,*args)) > return new_func > @curried > def sun_1(a,b,c): > return a+b+c > @curried > def sun_2(a,b,c): > return a*b*c > if __name__=="__main__": > print sun_2(1,)(2)(44) > print sun_1(1,)(2)(44) > print sun_2(1,2)(23) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.cristh at gmail.com Tue May 31 12:09:05 2016 From: alan.cristh at gmail.com (Alan Cristhian) Date: Tue, 31 May 2016 13:09:05 -0300 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> Message-ID: > The fundamental use-case here is "any object that needs to know its own > name". Most objects need more than just a name. A more general solution could be: def x, y, z as Where should return a callable that accept only one argument. So, you can do some like: def Type(bases, namespace): def get_name(name): return type(name, bases, namespace) return get_name def x, y, z as Type(bases, namespace) That is a syntax sugar of: dummy = Type(bases, namespace) x = dummy("x") y = dummy("y") z = dummy("z") 2016-05-31 12:27 GMT-03:00 David Mertz : > This is bikeshedding a bit, but a keyword that looks good to me: > > expose Typevar as T > expose Symbol as X > On May 31, 2016 2:06 AM, "Paul Moore" wrote: > >> On 31 May 2016 at 09:56, Stephen J. Turnbull wrote: >> > I know you don't think a keyword works for you, but >> > either the recently reraised "def = " or perhaps >> > "type : " make more sense to me right out of the box. >> >> I was thinking along the lines of "name callable", >> which I don't think works because it needs some "punctuation" to >> separate the name from the callable. >> >> But "def name = callable" (or some other preceding keyword combined >> with =) might work. I don't like "type" though, as the point here (I >> thought - see below) is to come up with a construct useful for more >> than just types. >> >> > I'm +1 for stopping the bikeshedding until we've all got a lot of >> > stubfile reading under our belts. >> >> If this was simply about type definitions, I'd agree. But I thought >> the point of Guido's post was that having seen two examples (TypeVar >> and Symbol) is there a more general approach that might cover these >> two cases as well as others? So just looking at the problem in terms >> of stub files isn't really the point here. >> >> Paul >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Tue May 31 12:10:13 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Tue, 31 May 2016 09:10:13 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: (Paul Moore's message of "Mon, 30 May 2016 16:23:30 +0100") References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> <87twhgr5ns.fsf@kosh.rath.org> <87bn3na8jf.fsf@kosh.rath.org> Message-ID: <87pos28aze.fsf@thinkpad.rath.org> On May 30 2016, Paul Moore wrote: > On 30 May 2016 at 16:07, Nikolaus Rath wrote: >> Yeah, but that might be useful too :-). How about: >> >> extract key foo from d >> extract attribute foo import d >> >> or >> >> export key foo from d >> export attribute foo import d >> >> As for "import", with both foo and d required to be identifiers. > > At this point, the question has to be, how is this any better than > > foo = d.foo > foo = d['foo'] Huh? I thought this has been discussed at length. It's not better in toy examples, but consider this: r = query_result product_id = r['product_id'] quantity = r['quantity'] distributor = r['distributor'] description = r['destription'] Compare to export key (product_id, quantity, distributor, description) from query_result Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From Nikolaus at rath.org Tue May 31 12:11:52 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Tue, 31 May 2016 09:11:52 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <574CFE59.2050904@stoneleaf.us> (Ethan Furman's message of "Mon, 30 May 2016 20:00:41 -0700") References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> <87twhgr5ns.fsf@kosh.rath.org> <87bn3na8jf.fsf@kosh.rath.org> <574CFE59.2050904@stoneleaf.us> Message-ID: <87mvn68awn.fsf@thinkpad.rath.org> On May 30 2016, Ethan Furman wrote: > On 05/30/2016 08:07 AM, Nikolaus Rath wrote: >> On May 29 2016, Guido van Rossum wrote: >>>> using other words still on the table? >>>> >>>> d = {"foo": 42} >>>> d 42 # or the other way around >>>> assert foo == 42 >>> >>> This I don't understand -- why would the '42' appear in the extraction >>> syntax? I guess you meant "foo"? >> >> Yes, sorry. >> >>> Maybe we can riff on >>> >>> extract foo from d >>> >>> ? Though honestly that looks like it would be extracting d.foo, no >>> d['foo']. >> >> Yeah, but that might be useful too :-). How about: >> >> extract key foo from d >> extract attribute foo import d >> >> or >> >> export key foo from d >> export attribute foo import d >> >> As for "import", with both foo and d required to be identifiers. > > The versions with "import" in them are DAAP (dead-as-a-parrot). No > point in even talking about them. Aeh, yeah. No idea how I ended up writing that. What I meant was extract key foo from d extract attribute foo from d or export key foo from d export attribute foo from d i.e, contrast "extract" with "export" for attribute access and dict access. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From greg.ewing at canterbury.ac.nz Tue May 31 08:55:37 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 01 Jun 2016 00:55:37 +1200 Subject: [Python-ideas] Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects In-Reply-To: <574D4171.3060301@egenix.com> References: <574D20D9.7060504@gmail.com> <574D4171.3060301@egenix.com> Message-ID: <574D89C9.4030502@canterbury.ac.nz> > On 31.05.2016 07:27, Matthew Tanous wrote: >> >>arr = [[True] * 5] * 5 >> >>This does not create a matrix-like arrangement of the immutable truth >>variable, but instead creates a list of 5 references to the same list I think this would be better addressed by an extension to list comprehensions, e.g. arr = [[True times 5] times 5] where [x times n] would be equivalent to [x for i in range(n)] Since this would evaluate x afresh for each elment, it would avoid all the thorny issues involved in trying to automatically deep-copy arbitrary objects. -- Greg From tjreedy at udel.edu Tue May 31 12:59:00 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 31 May 2016 12:59:00 -0400 Subject: [Python-ideas] Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects In-Reply-To: <20160531141625.GV12028@ando.pearwood.info> References: <574D20D9.7060504@gmail.com> <574D89A8.3060306@mail.de> <20160531141625.GV12028@ando.pearwood.info> Message-ID: On 5/31/2016 10:16 AM, Steven D'Aprano wrote: > On Tue, May 31, 2016 at 01:36:31PM +0000, Joseph Martinot-Lagarde wrote: >>> I can only agree here. Even today, despite knowing the fact, it's >>> causing some headaches in some cases. >> >> How about raising an exception if mutable objects are in the list ? > > -1 > > It's a gratuitous breakage that cannot solve the problem, because you > cannot tell in advance which objects are mutable and which are not. The > best you can do is recognise known built-ins. > > ("list is mutable, tuple is not, except when it contains a mutable > item, but mymodule.MySequence may or may not be, there's no way to > tell in advance.") > > >> - maybe there are useful use cases of duplicating the reference ? > > Absolutely. That's a standard way of grouping items taken from an > iterable: > > py> it = iter("hello world!") > py> for a, b, c in zip(*[it]*3): # groups of three > ... print(a, b, c) > ... > h e l > l o > w o r > l d ! > > > This wouldn't work if the iterators were three independent copies. -1 also. As Steven has pointed out several times in other threads, Python is consistent in not copying objects unless requested. Thus a = [1,2,3] b = a # a reference copy, not an object copy, as some expect a[1] = 4 print(b[1]) # 4 -- a surprise for those who expect '=' to copy the list object Sequence * is an elaboration of the same issue. -- Terry Jan Reedy From barry at barrys-emacs.org Tue May 31 12:20:36 2016 From: barry at barrys-emacs.org (Barry Scott) Date: Tue, 31 May 2016 17:20:36 +0100 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: References: Message-ID: <793900F8-38DC-4D21-823A-ABB72D4A0E04@barrys-emacs.org> > On 31 May 2016, at 02:16, Guido van Rossum wrote: > > In mypy we have a need for type variables, which are created like this: > > from typing import TypeVar > > T = TypeVar('T') > > I just saw a lightning talk about sympy where they define symbols to > be used in mathematical equations, like this: > > from sympy import Symbol > > x = Symbol('x') > > I'm sure this is not a new idea, but so far I've always thought that > this is pretty esoteric and the approach here is good enough. But > maybe we can actually do better, and whatever solution we come up with > might also be useful for extracting attributes from an object or > values from a mapping? > Looks like the C #define is what is being asked for. #define SYM( name ) name = Symbol( #name ) SYM( x ) Does python need macro preprocessor? Barry From Nikolaus at rath.org Tue May 31 13:12:12 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Tue, 31 May 2016 10:12:12 -0700 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <20160531030810.GR12028@ando.pearwood.info> (Steven D'Aprano's message of "Tue, 31 May 2016 13:08:10 +1000") References: <20160531030810.GR12028@ando.pearwood.info> Message-ID: <87k2ia8843.fsf@thinkpad.rath.org> On May 31 2016, Steven D'Aprano wrote: > How do you feel about an arrow operator? > > > T -> TypeVar() > x -> Symbol() > T -> type(bases, ns) > Record -> namedtuple(fields) Solves the problem nicely, but difficult to memorize. I'd always expect the RHS to be evaluated first, and *then* somehow be connected to the LHS. How about a special placeholder? Record -> namedtuple($me, fields) Record := namedtuple($me, fields) or maybe even Record = namedtuple($me, fields) *duck* (yeah, I guess I'd like to see $ get semantics in Python) Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From p.f.moore at gmail.com Tue May 31 13:22:24 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 31 May 2016 18:22:24 +0100 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <87pos28aze.fsf@thinkpad.rath.org> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> <87twhgr5ns.fsf@kosh.rath.org> <87bn3na8jf.fsf@kosh.rath.org> <87pos28aze.fsf@thinkpad.rath.org> Message-ID: On 31 May 2016 at 17:10, Nikolaus Rath wrote: > On May 30 2016, Paul Moore wrote: >> On 30 May 2016 at 16:07, Nikolaus Rath wrote: >>> Yeah, but that might be useful too :-). How about: >>> >>> extract key foo from d >>> extract attribute foo import d >>> >>> or >>> >>> export key foo from d >>> export attribute foo import d >>> >>> As for "import", with both foo and d required to be identifiers. >> >> At this point, the question has to be, how is this any better than >> >> foo = d.foo >> foo = d['foo'] > > Huh? I thought this has been discussed at length. It's not better in toy > examples, but consider this: > > r = query_result > product_id = r['product_id'] > quantity = r['quantity'] > distributor = r['distributor'] > description = r['destription'] > > > Compare to > > export key (product_id, quantity, distributor, > description) from query_result > Not saying the current idiom can't be improved, but I don't like this approach - it seems to be focused on compressing the extraction into one line, which isn't a benefit to me I typically find it easier to line up, and scan, information vertically rather than horizontally. So the "export" approach for me would need to be written export key ( product_id, quantity, distributor, description ) \ from query_result That backslash after the close bracket is ugly. But not having a "blank" line before the "from query_result" line is also difficult to read. The list of repeated assignments is too repetitive, but nevertheless can be lined up more neatly. Technically, the "export key" approach could probably me made readable in a way I'd be OK with, for example: export key from query_result ( product_id, quantity, distributor, description ) but I'm not at all sure that needing 2 new keywords ("export" and "key") and a reuse of an existing one ("from") is going to fly - it's too wordy, feels like SQL or COBOL to me. Maybe if someone comes up with a one-word option for "export key from" then it would be viable... Paul From mertz at gnosis.cx Tue May 31 13:35:01 2016 From: mertz at gnosis.cx (David Mertz) Date: Tue, 31 May 2016 10:35:01 -0700 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <20160531154327.GY12028@ando.pearwood.info> References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> <20160531154327.GY12028@ando.pearwood.info> Message-ID: Bracketing for a moment what keyword might be used, a thing I like about this syntax is that it can work for multiple names naturally. E.g. expose Symbol as x, y, z Or with a different word: define Typevar as T, S, R For things like namedtuple that need more arguments, you'd need to use functools.partial to create the single argument callable to match the syntax. On May 31, 2016 8:44 AM, "Steven D'Aprano" wrote: > On Tue, May 31, 2016 at 08:27:28AM -0700, David Mertz wrote: > > This is bikeshedding a bit, but a keyword that looks good to me: > > > > expose Typevar as T > > expose Symbol as X > > "expose" sounds like it is taking something hidden or private and > exposing it to the public. It doesn't give any hint that it takes the > name T and uses it as an argument to Typevar. I would take it as > meaning: > > take the (possibly internal or private) thing Typevar, and bind it to > the name T > > with no hint that Typevar was being called. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vgr255 at live.ca Tue May 31 14:08:08 2016 From: vgr255 at live.ca (=?utf-8?Q?=C3=89manuel_Barry?=) Date: Tue, 31 May 2016 14:08:08 -0400 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <87k2ia8843.fsf@thinkpad.rath.org> References: <20160531030810.GR12028@ando.pearwood.info> <87k2ia8843.fsf@thinkpad.rath.org> Message-ID: I think that the idea of implicit call or magically inserting items at the front (or any arbitrary position, for that matter) isn't going to work. For it to work, we need to address cases such as those: x -> foo("bar")("baz") # is this foo("x", "bar")("baz") or foo("bar")("x", "baz") ? x -> foo() # is this foo("x") or foo()("x") ? x -> foo # hmm, are we doing foo("x") here, or should this be a syntax error? I think that whatever we go for, we need to be explicit where the name goes - but that completely breaks the purpose of the proposal to begin with. Or we could get rid of ambiguity and something like this instead: x -> foo("bar")("baz") # actually foo("x", "bar")("x", "baz") x -> foo() # wait, this is actually foo("x")("x") Yuck! I tend to like implicit black magic in some of my code, but that's pushing it too far. Or wait, maybe we can settle for a syntax like this (hypothetic pseudo-code): x -> foo(?, "bar") # this is foo("x", "bar") x -> foo(?) # foo("x") x -> foo(?, "bar", ?) # is this foo("x", "bar", "x") or a syntax error? This looks fine. What about this, though? a, b, c -> foo(?) # probably foo(("a", "b", "c")), right? a, b, c -> foo(?, "bar", ?, "baz", ?) # is this foo("a", "bar", "b", "baz", "c") or a syntax error? In the face of ambiguity, refuse the temptation to guess and throw a syntax error in all ambiguous cases. Oh wait, that's how Python works right now - every case is ambiguous for that. The main point is that there's no way this is going to fly if there isn't an explicit way to declare the implicitness - at that point I'd rather hand-roll my own function (which I can probably customize to fit the needs of my module). Something like: def set_global(func, name, *args, **kwargs): globals()[name] = func(name, *args, **kwargs) In fact, I would go for some decorator magic (I love decorators): def call_with_name(func): fn, args, kwargs = func() return fn(func.__name__, *args, **kwargs) @call_with_name def x(): return namedtuple, (fields,), {} There you go, and you only typed the name once, in the function definition. Magic enough for you? I love syntactic sugar and Python's magic, and this would be just fine for me. And, as everyone knows, code is read much more often than it is written. As such, x -> foo() is easier to write, but much harder to read, turning the fun of writing something that works using compiler magic into a maintenance hell. While x = foo("x") takes more keystrokes to type, it's exactly in line with what I'm used to see in Python, doesn't require any magic, is very easy to get right and makes maintaining the module straightforward for everyone involved. ---- I agree that something like T = TypeVar("T") isn't pretty, but explicit is better than implicit here. If someone comes with an obviously unambiguous syntax for this, then I'll be in favour. Until then, count me -1 on the whole thing. Finding the appropriate syntax for this isn't bikeshedding here, it's the most important part of the whole thing. As I stated above, most of the ideas mentioned here fall into the "obviously unobvious" category. I also think that adding new syntax might become an attractive nuisance, where people use it all the time everywhere and code quickly becomes unreadable. It also makes one new concept for newbies to learn, possibly quite early if most start to adopt this. All in all, while I see and feel the need, I don't think the syntax change is a good idea, sadly. - Emanuel From brenbarn at brenbarn.net Tue May 31 14:13:30 2016 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Tue, 31 May 2016 11:13:30 -0700 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <20160531030810.GR12028@ando.pearwood.info> References: <20160531030810.GR12028@ando.pearwood.info> Message-ID: <574DD44A.8020609@brenbarn.net> On 2016-05-30 20:08, Steven D'Aprano wrote: > How do you feel about an arrow operator? > > > T -> TypeVar() > x -> Symbol() > T -> type(bases, ns) > Record -> namedtuple(fields) > > In general: > > name [, name, ...] -> callable(...) > > where "callable" is some expression, e.g. a name, a dot lookup, etc. > > # okay > name -> sympy.Symbol() I don't like this that much, because it looks odd to have the expression on the right without the argument. The syntax doesn't make it at all clear how the T gets used on the RHS. > Using the arrow operator with arbitrary expressions on the > right will be a SyntaxError: > > x -> y + 1 > my_name -> spam*f() > my_name -> f() or g() > > I reject that third example as it is unclear whether both f and g, or > only f, get the name as first argument. > > But this is allowed: > > my_name -> f(g()) # like f('my_name', g()) So. . . what exactly will be allowed? Above you said the RHS would be "a name, a dot lookup, etc.". If you can't do it with at least dot lookups I don't see much point; you'd definitely want to be able to do this to create names from method calls. But it could get confusing to start splitting the expression syntax into portions that are allowed in this new "name-passing" syntax and portions that aren't. > The hard case > ------------- > > Assignment target(s) is not a plain name. > > spam[2].eggs().attr -> Symbol() > > > I'm going to take the cowards way out and just say "we don't allow > that". That seems like the best way out. :-) It seems like the main purpose of this is as a shortcut to create an object with a simple, readable name. That goal isn't served by trying to assign some complicated expression. Anyway, I still have a problem with solutions like this that separate the name from the expression in which it will be used. It seems crude to restrict this syntax to cases where the outermost callable takes the name as its first positional argument only. What if we want to pass the name by keyword or something? Or what if you want to do `f(g())` as in your example, but pass the name argument to g instead of f? Because of that, I'm more inclined towards something that involves a placeholder in the actual expression (as someone else suggested elsewhere on this thread). Something like: Symbol(def x) An expression could contain at most one "def name" token, and the result of the entire expression would be assigned to that name. I acknowledge that this particular syntax isn't ideal, because the "def x" could be difficult to see in a complicated expression (although syntax highlighting would mitigate that). The construct could perhaps be introduced by a keyword to call out its unusual behavior: assign Symbol(def x) I'm just making up "assign" right now, not actually saying that's what should be used. Maybe some existing keyword could be repurposed. The upshot is that, to my eye, any solution that pulls the "name" argument out of the expression where it's used is to going to A) be confusing; and B) painfully restrict the use cases to those where the RHS is a very simple expression into which the name can be inserted at an easily-definable place. So I incline towards solutions that "tag" the name in-place in the expression, rather than pulling it out. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From stephen at xemacs.org Tue May 31 14:58:36 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 1 Jun 2016 03:58:36 +0900 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <20160531132225.GS12028@ando.pearwood.info> References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> <20160531132225.GS12028@ando.pearwood.info> Message-ID: <22349.57052.500629.614001@turnbull.sk.tsukuba.ac.jp> Steven D'Aprano writes: > > It give me no clue what it's supposed to mean. > > That's harsh. No clue *at all*? What harsh? Frank, yes. As for the question, of course not "*at all*", obviously it's an operator that does something. But what and to which, no clue, except from the context of what you're trying to accomplish -- not from the notation itself. The variety of examples and hints you give just makes it worse. > And neither is the `self` argument to methods. But we learn that self is > implicitly provided as if by magic :-) No, it's explicitly provided by the attribute reference syntax that binds a positional parameter when the attribute is a method. Binding positional parameters is a concept familiar from all programming languages I know of (except Haskell, which has only one parameter so "position" is a vacuous concept). I will remind you that ".attribute" syntax to avoid typing self in method definitions and similar anaphoric syntaxes for avoiding typing the name of a object whose attributes are repeatedly accessed have been repeatedly vetoed. > Another similar example of magic is decorator syntax: I can't explain why that doesn't bother me, but it doesn't. It's also much more powerful. Not the syntax itself, but what decorators can do. > The thing is, regardless of whether we use a symbol or a keyword, the > functionality here is going to be magical. That's its point: it is to > avoid the need to write the name twice. Yes, I could learn this magic. No, I find the syntaxes proposed so far unintuitive, by which I mean quite arbitrary and lacking connotations that lead to the intended meaning. So, not good enough. -1 on forward arrow syntax: as you point out, mention of a variable injects that name into the appropriate namespace, and there's important. Extremely concise notation for variable evaluation is incredibly important, and having that string in locals() or globals() is necessary for introspection. Thus variable *mention* does the injection. But the other syntaxes that inject names of identifiers into namespaces are all statements. -0.5 on keywords: this doesn't clear the bar for keyword introduction yet AFAICS. Not even if it's only used in stubfiles. From bzvi7919 at gmail.com Tue May 31 15:11:03 2016 From: bzvi7919 at gmail.com (Bar Harel) Date: Tue, 31 May 2016 19:11:03 +0000 Subject: [Python-ideas] Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects In-Reply-To: References: <574D20D9.7060504@gmail.com> <574D89A8.3060306@mail.de> <20160531141625.GV12028@ando.pearwood.info> Message-ID: -1 as well. I have used the generator multiplication many times before. The use of the same object helps a lot in certain cases. Changing that will cause a confusion among those who know the phenomenon and would break compatibility. Regarding your matrix example, maybe we can make use of the new '@' operator for copying new objects, what do you say? -- Bar Harel On Tue, May 31, 2016 at 8:01 PM Terry Reedy wrote: > On 5/31/2016 10:16 AM, Steven D'Aprano wrote: > > On Tue, May 31, 2016 at 01:36:31PM +0000, Joseph Martinot-Lagarde wrote: > >>> I can only agree here. Even today, despite knowing the fact, it's > >>> causing some headaches in some cases. > >> > >> How about raising an exception if mutable objects are in the list ? > > > > -1 > > > > It's a gratuitous breakage that cannot solve the problem, because you > > cannot tell in advance which objects are mutable and which are not. The > > best you can do is recognise known built-ins. > > > > ("list is mutable, tuple is not, except when it contains a mutable > > item, but mymodule.MySequence may or may not be, there's no way to > > tell in advance.") > > > > > >> - maybe there are useful use cases of duplicating the reference ? > > > > Absolutely. That's a standard way of grouping items taken from an > > iterable: > > > > py> it = iter("hello world!") > > py> for a, b, c in zip(*[it]*3): # groups of three > > ... print(a, b, c) > > ... > > h e l > > l o > > w o r > > l d ! > > > > > > This wouldn't work if the iterators were three independent copies. > > -1 also. As Steven has pointed out several times in other threads, > Python is consistent in not copying objects unless requested. Thus > > a = [1,2,3] > b = a # a reference copy, not an object copy, as some expect > a[1] = 4 > print(b[1]) > # 4 -- a surprise for those who expect '=' to copy the list object > > Sequence * is an elaboration of the same issue. > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Tue May 31 15:57:04 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Tue, 31 May 2016 12:57:04 -0700 Subject: [Python-ideas] Unpacking a dict In-Reply-To: (Paul Moore's message of "Tue, 31 May 2016 18:22:24 +0100") References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> <87twhgr5ns.fsf@kosh.rath.org> <87bn3na8jf.fsf@kosh.rath.org> <87pos28aze.fsf@thinkpad.rath.org> Message-ID: <87bn3m80hb.fsf@thinkpad.rath.org> On May 31 2016, Paul Moore wrote: > Technically, the "export key" approach could probably me made readable > in a way I'd be OK with, for example: > > export key from query_result ( > product_id, > quantity, > distributor, > description > ) > > but I'm not at all sure that needing 2 new keywords ("export" and > "key") and a reuse of an existing one ("from") is going to fly - it's > too wordy, feels like SQL or COBOL to me. Maybe if someone comes up > with a one-word option for "export key from" then it would be > viable... How about unravel query_result (this, that, something) Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From leewangzhong+python at gmail.com Tue May 31 15:58:09 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Tue, 31 May 2016 15:58:09 -0400 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> <87twhgr5ns.fsf@kosh.rath.org> <87bn3na8jf.fsf@kosh.rath.org> <87pos28aze.fsf@thinkpad.rath.org> Message-ID: On Tue, May 31, 2016 at 1:22 PM, Paul Moore wrote: > but I'm not at all sure that needing 2 new keywords ("export" and > "key") and a reuse of an existing one ("from") is going to fly - it's > too wordy, feels like SQL or COBOL to me. Maybe if someone comes up > with a one-word option for "export key from" then it would be > viable... As two words: fromdict import [ as ][, [, ...]] IMO: - The desired feature really is more like import than unpacking. It's only different in that one is a lookup to d.__getitem__ while the other is a file lookup, then a load, then (I believe) a lookup to vars(module).__getitem__. - Don't repeat the key. - The names being introduced should be syntactic tokens, [so that || because] the compiler knows to make cells in locals() for those names. - The mapping should be named first. (TBF, I also think that, in comprehensions, the item should be named before the expression using it, so maybe my senses are wrong.) - The syntax should make it obvious that a dict-like object is expected, even to those who haven't seen it before. Leading `extract` and `export` don't satisfy that. (`extract key` makes me itch when I see more than one key follow, and it isn't clear without reading further that "key" isn't what's being extracted.) - As long as it is clear from the start of the line whether it is a module or a dict being imported, it's okay to use `import`. It's too bad "fromdict" is used in the wild (https://pythonhosted.org/dictalchemy/#dictalchemy.utils.fromdict), and "frommapping" is too long (in letters, syllables, and constituent parts: "from", "map", and "-ing"). `fromdict d import *`: People might expect this, for symmetry with `from ... import *`. In CPy, there's no sense of dynamically-created local names (since locals are looked up in the compiler), and you couldn't use them anyway. (We don't talk about `eval` and `exec`.) In global scope, `import *` makes more sense, but I think you can just say, "If you need something like that, you should've created a module instead." Possible (gross) extensions: fromdict d import 'literalstring' as name2, (evaluatedname) as name2 From leewangzhong+python at gmail.com Tue May 31 15:59:17 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Tue, 31 May 2016 15:59:17 -0400 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> <87twhgr5ns.fsf@kosh.rath.org> <87bn3na8jf.fsf@kosh.rath.org> <87pos28aze.fsf@thinkpad.rath.org> Message-ID: In Tue, May 31, 2016 at 3:58 PM, Franklin? Lee wrote: > In CPy, there's no sense of dynamically-created local names (since > locals are looked up in the compiler) I mean, in an array. From p.f.moore at gmail.com Tue May 31 16:00:15 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 31 May 2016 21:00:15 +0100 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <22349.57052.500629.614001@turnbull.sk.tsukuba.ac.jp> References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> <20160531132225.GS12028@ando.pearwood.info> <22349.57052.500629.614001@turnbull.sk.tsukuba.ac.jp> Message-ID: On 31 May 2016 at 19:58, Stephen J. Turnbull wrote: > -0.5 on keywords: this doesn't clear the bar for keyword introduction > yet AFAICS. Not even if it's only used in stubfiles. If anything that comes from this is only for stubfiles, I'm -1 on it. To be worth implementing, any new feature here would have to have a much wider applicability than one small area to get my vote. Paul From p.f.moore at gmail.com Tue May 31 16:06:08 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 31 May 2016 21:06:08 +0100 Subject: [Python-ideas] Unpacking a dict In-Reply-To: <87bn3m80hb.fsf@thinkpad.rath.org> References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> <87twhgr5ns.fsf@kosh.rath.org> <87bn3na8jf.fsf@kosh.rath.org> <87pos28aze.fsf@thinkpad.rath.org> <87bn3m80hb.fsf@thinkpad.rath.org> Message-ID: On 31 May 2016 at 20:57, Nikolaus Rath wrote: > On May 31 2016, Paul Moore wrote: >> Technically, the "export key" approach could probably me made readable >> in a way I'd be OK with, for example: >> >> export key from query_result ( >> product_id, >> quantity, >> distributor, >> description >> ) >> >> but I'm not at all sure that needing 2 new keywords ("export" and >> "key") and a reuse of an existing one ("from") is going to fly - it's >> too wordy, feels like SQL or COBOL to me. Maybe if someone comes up >> with a one-word option for "export key from" then it would be >> viable... > > How about > > unravel query_result (this, that, something) Meh. Don't assume that if you come up with a good word, I'll be in favour. At best, I'll go from -1 to -0 or maybe +0. Should we get to something that looks reasonably attractive to me, there's still issues with the whole thing being a niche problem, limited applicability (by leaping at "unravel" you lost the ability to extract attributes from an object - did you mean to do that?), etc. Basically, don't waste too much time trying to convince me. A better bet would be to get sufficient support from others that my opinion is irrelevant (which it may well be anyway :-)) Paul From egregius313 at gmail.com Tue May 31 17:09:30 2016 From: egregius313 at gmail.com (Ed Minnix) Date: Tue, 31 May 2016 17:09:30 -0400 Subject: [Python-ideas] A function currying decorator ((Was: no subject)) In-Reply-To: References: Message-ID: <3DC8E640-E359-4BD8-840B-0CC5BF8D6E4F@gmail.com> It requires downloading a 3rd party package, but both toolz and funcy provide curry classes/functions which can decorate a function and create a function that is automatically curried. - Ed > On May 31, 2016, at 11:59 AM, Yongsheng Cheng > wrote: > > Yes, that's what I meant. thanks and do you think we need such decorate in funtools? wiatting for reply > > Ryan Gonzalez >?2016?5?31??? ??11:35??? > So you're referring to function currying, correct? I changed the thread title to reflect that. > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. > http://kirbyfan64.github.io/ > On May 31, 2016 8:04 AM, "Yongsheng Cheng" > wrote: > to begin with: i a deep python fun > i want to suguest python a feature to support Fuctional Program :,here is the toy code i need a more powerful decrote in moudle functools instead of functools.partial,thanks do you think it is a nice idea? > waitting for your reply > > On May 31, 2016, at 11:59 AM, Yongsheng Cheng wrote: > > Yes, that's what I meant. thanks and do you think we need such decorate in funtools? wiatting for reply > > Ryan Gonzalez >?2016?5?31??? ??11:35??? > So you're referring to function currying, correct? I changed the thread title to reflect that. > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. > http://kirbyfan64.github.io/ > On May 31, 2016 8:04 AM, "Yongsheng Cheng" > wrote: > to begin with: i a deep python fun > i want to suguest python a feature to support Fuctional Program :,here is the toy code i need a more powerful decrote in moudle functools instead of functools.partial,thanks do you think it is a nice idea? > waitting for your reply > > def minpara(func_1,args): > num=0 > try: > func_1(*args) > except Exception as error: > num,given=re.findall(r'\d+',str(error))[:2] > if num: > #return int(num),int(given) > return (num,given) > else: > #return func(*args) > return (0,0) > > def curried(func): > #@wraps(func) > def new_func(*args): > num,given=minpara(func,args) > #print num,given > if (num,given)==(0,0): > return func(*args) > else: > return curried(functools.partial(func,*args)) > return new_func > @curried > def sun_1(a,b,c): > return a+b+c > @curried > def sun_2(a,b,c): > return a*b*c > if __name__=="__main__": > print sun_2(1,)(2)(44) > print sun_1(1,)(2)(44) > print sun_2(1,2)(23) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue May 31 19:10:19 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 01 Jun 2016 11:10:19 +1200 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <20160531133717.GT12028@ando.pearwood.info> References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> <574D8B3C.5060406@mail.de> <20160531133717.GT12028@ando.pearwood.info> Message-ID: <574E19DB.7050307@canterbury.ac.nz> Steven D'Aprano wrote: > name -> Function(args) > > will be expanded to: > > name = Function('name', args) Another possibility would be to expand it to name = Function(args) name.__name__ = "name" -- Greg From greg.ewing at canterbury.ac.nz Tue May 31 19:20:16 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 01 Jun 2016 11:20:16 +1200 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <20160531142438.GW12028@ando.pearwood.info> References: <20160531030810.GR12028@ando.pearwood.info> <20160531142438.GW12028@ando.pearwood.info> Message-ID: <574E1C30.8030508@canterbury.ac.nz> Steven D'Aprano wrote: > On Tue, May 31, 2016 at 02:12:05PM +0000, Joseph Martinot-Lagarde wrote: > >> attr -> my_object. > > > That fails the "grit on Uncle Timmy's monitor" test. ALso it looks like somebody lapsed into writing COBOL. CLOSE INTERNET-POST WITH SMILEY LY YRS, GREG. From greg.ewing at canterbury.ac.nz Tue May 31 19:31:52 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 01 Jun 2016 11:31:52 +1200 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <20160531140416.GU12028@ando.pearwood.info> References: <20160531030810.GR12028@ando.pearwood.info> <574D8B81.9070108@canterbury.ac.nz> <20160531140416.GU12028@ando.pearwood.info> Message-ID: <574E1EE8.1080601@canterbury.ac.nz> Steven D'Aprano wrote: > I know that *technically* def is > an assignment (a name binding) but it doesn't look like one. It looks > like a definition or declaration. Well, the use cases we've been considering are effectivly definitions, they just don't look like it because we don't have a syntax that generalises the definitioniness of 'def' and 'class'. > To me, I don't think def is a good match because def doesn't have a left > and right hand side. Assignment does. Maybe that means assignment isn't a good model for the new syntax? Maybe it should be more like: def Symbol x If you want to give it args, you do def Symbol x(args) The idea is that this is modelled after class name(args): with the keyword 'class' replaced by another keyword together with the type of thing you're defining. -- Greg From greg.ewing at canterbury.ac.nz Tue May 31 19:43:05 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 01 Jun 2016 11:43:05 +1200 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <20160531150858.GX12028@ando.pearwood.info> References: <20160531030810.GR12028@ando.pearwood.info> <20160531142438.GW12028@ando.pearwood.info> <20160531150858.GX12028@ando.pearwood.info> Message-ID: <574E2189.6050205@canterbury.ac.nz> Steven D'Aprano wrote: > If you're worried that it will be confusing that we define a callable to > take (say) three arguments ... but then call it with only two ... > I think you're worrying over nothing. The situation with "self" is a bit different. When you do something like foo.method(args) it's obvious that something is being done *before* the call, i.e. looking up the 'method' attribute of foo, and that step has the opportunity to manufacture and return a callable having the appropriate number of args. However, with x -> TypeVar() it looks very much like whatever is bound to TypeVar is being called directly, before anything else happens. For that not to be the case would be a rather higher order of weirdness, IMO. -- Greg From greg.ewing at canterbury.ac.nz Tue May 31 19:45:53 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 01 Jun 2016 11:45:53 +1200 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: References: <20160531030810.GR12028@ando.pearwood.info> <20160531142438.GW12028@ando.pearwood.info> <20160531150858.GX12028@ando.pearwood.info> Message-ID: <574E2231.9090907@canterbury.ac.nz> Paul Moore wrote: > I've just checked functools.partial, > and it doesn't appear to allow for inserting an argument at the > *start* of the argument list. Maybe we could do with a functools.laitrap function? -- Greg From ian at feete.org Tue May 31 19:46:36 2016 From: ian at feete.org (Ian Foote) Date: Wed, 1 Jun 2016 00:46:36 +0100 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <574E1EE8.1080601@canterbury.ac.nz> References: <20160531030810.GR12028@ando.pearwood.info> <574D8B81.9070108@canterbury.ac.nz> <20160531140416.GU12028@ando.pearwood.info> <574E1EE8.1080601@canterbury.ac.nz> Message-ID: <574E225C.6090400@feete.org> On 01/06/16 00:31, Greg Ewing wrote: > Steven D'Aprano wrote: > >> I know that *technically* def is an assignment (a name binding) but it >> doesn't look like one. It looks like a definition or declaration. > > Well, the use cases we've been considering are effectivly > definitions, they just don't look like it because we don't > have a syntax that generalises the definitioniness of > 'def' and 'class'. > >> To me, I don't think def is a good match because def doesn't have a left >> and right hand side. Assignment does. > > Maybe that means assignment isn't a good model for the > new syntax? > > Maybe it should be more like: > > def Symbol x > > If you want to give it args, you do > > def Symbol x(args) > > The idea is that this is modelled after > > class name(args): > > with the keyword 'class' replaced by another keyword > together with the type of thing you're defining. > This is the first syntax I've seen in this thread that seems reasonably intuitive and pythonic. I don't mind particularly the keyword used (def or something else), but this structure draws the right parallels in my mind. Regards, Ian F -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: OpenPGP digital signature URL: From michael.selik at gmail.com Tue May 31 19:51:47 2016 From: michael.selik at gmail.com (Michael Selik) Date: Tue, 31 May 2016 23:51:47 +0000 Subject: [Python-ideas] Unpacking a dict In-Reply-To: References: <20160526015606.GB12028@ando.pearwood.info> <57472944.7020102@mail.de> <20160526174016.GD12028@ando.pearwood.info> <57473890.6050400@stoneleaf.us> <87d1o77d1c.fsf@thinkpad.rath.org> <57487AFB.8020800@stoneleaf.us> <87vb1z5q80.fsf@thinkpad.rath.org> <87twhgr5ns.fsf@kosh.rath.org> <87bn3na8jf.fsf@kosh.rath.org> <87pos28aze.fsf@thinkpad.rath.org> Message-ID: On Tue, May 31, 2016 at 3:59 PM Franklin? Lee wrote: > The desired feature really is more like import than unpacking. > That's the feature some people desire, but not what I originally proposed. I did want unpacking, including recursive unpacking and too-many/too-few errors. I'm satisfied with the result that I'll need to use my own module for now. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue May 31 19:57:27 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 01 Jun 2016 11:57:27 +1200 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> <20160531154327.GY12028@ando.pearwood.info> Message-ID: <574E24E7.2070402@canterbury.ac.nz> David Mertz wrote: > define Typevar as T, S, R I quite like this, but it should be the other way around: define T, S, R as TypeVar Yes, I *know* the name being bound comes after 'as' in other places, but I think consistency with the obvious English meaning is more important. One other thing, 'define' is so similar to 'def' that I think it would be needlessly confusing to have both, so just make it def T, S, R as TypeVar Also, yes, you would have to curry if you want the constructor to have arguments. I think that's a small price to pay for the benefits: less magic, no restrictions on the form of the constructor expression, and the ability to re-use it for multiple bound names. -- Greg From cs at zip.com.au Tue May 31 20:09:56 2016 From: cs at zip.com.au (cs at zip.com.au) Date: Wed, 1 Jun 2016 10:09:56 +1000 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: References: Message-ID: <20160601000956.GA65302@cskk.homeip.net> On 31May2016 04:57, Joshua Morton wrote: >I like the arrow idea, another option is a syntax more in line with the one >being discussed for dict unpacking, something like adding a __define__ >dunder that returns a tuple of (name, value). So that say, Typevar's >__define__ returns the tuple `(self.__name__, self)`. Which is then >converted into an assignment. The downside of this being that its >assignment overloading(which can be abused), but at least it can only be >done when specifically asked for. The other downside is that the syntax > >define TypeVar('T') # and I assume we can also declare a list of them >def ident(t: T) -> T: > return t [...] >The upside is that it avoids the problem of Steven's implementation, since >the name is decided internally by the callable. Which to me is a downside. A big downside. Ignoring "from blah import *", when I write a Python module I have complete control over the local names: from os.path import join from os.path import join as joinpath x = 3 def y(): All of these are only capable of defining names that I have typed literally into my code. If I understand Joshua's suggestion: define TypeVar('T') might define _any_ name, because the __define__ dunder can return any name. Maybe I misunderstand, but otherwise I think this is like a little landmine. Yes, code can monkey patch. But this feels more insidious. Why not require the __dunder__ to always return __name__? And then perhaps support both: define TypeVar('T') define TypeVar('T') as Not_T so that providing the local name (which we're trying to avoid needing to do) _is_ possible, as the special case as it is with import. Cheers, Cameron Simpson From michael.selik at gmail.com Tue May 31 20:52:58 2016 From: michael.selik at gmail.com (Michael Selik) Date: Wed, 01 Jun 2016 00:52:58 +0000 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <20160531030810.GR12028@ando.pearwood.info> References: <20160531030810.GR12028@ando.pearwood.info> Message-ID: On Mon, May 30, 2016 at 11:08 PM Steven D'Aprano wrote: > On Mon, May 30, 2016 at 06:16:26PM -0700, Guido van Rossum wrote: > > [...] > > T = TypeVar('T') > > x = Symbol('x') > > > > I'm sure this is not a new idea, but so far I've always thought that > > this is pretty esoteric and the approach here is good enough. > > This comes up a lot and it would be nice to clean it up. > > T = type('T', bases, ns) > Record = namedtuple('Record', fields) > I'm not enthusiastic about cleaning it up. It feels appropriately awkward. I don't want my colleagues having too easy a time dynamically creating types or functions. You brought up decorators as an example, but I think that was more about how decorators tend to be used essentially as part of the function definition. They might dramatically alter the behavior of the function and yet appear as an afterthought without the decorator syntax. From the PEP 318, "This becomes less readable with longer [functions]". The motivation said nothing about avoiding typing the function name twice. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Tue May 31 21:40:41 2016 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 31 May 2016 21:40:41 -0400 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <574E19DB.7050307@canterbury.ac.nz> References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> <574D8B3C.5060406@mail.de> <20160531133717.GT12028@ando.pearwood.info> <574E19DB.7050307@canterbury.ac.nz> Message-ID: On 5/31/2016 7:10 PM, Greg Ewing wrote: > Steven D'Aprano wrote: >> name -> Function(args) >> >> will be expanded to: >> >> name = Function('name', args) This is sounding a lot like PEP 359, the "make" statement. There's also a lot of discussion on the mailing lists back then about similar constructs. The make statement was focused on blocks, but it did have similar semantics with a callable. Eric. From mertz at gnosis.cx Tue May 31 21:48:35 2016 From: mertz at gnosis.cx (David Mertz) Date: Tue, 31 May 2016 18:48:35 -0700 Subject: [Python-ideas] Quick idea: defining variables from functions that take the variable name In-Reply-To: <574E24E7.2070402@canterbury.ac.nz> References: <20160531030810.GR12028@ando.pearwood.info> <22349.20907.341754.999142@turnbull.sk.tsukuba.ac.jp> <20160531154327.GY12028@ando.pearwood.info> <574E24E7.2070402@canterbury.ac.nz> Message-ID: I agree, I saw this after I posted. If the keyword is 'define' the names should definitely come before 'as'. If the keyword is 'expose' as I first thought of, the names should come after 'as'. Other possible words with names after 'as': reveal, publish, reify, create, etc. If some word is intuitive enough, I'd prefer this order. It's closer to what 'import' and 'with' do in conjunction with 'as'. On May 31, 2016 4:58 PM, "Greg Ewing" wrote: > David Mertz wrote: > > define Typevar as T, S, R >> > > I quite like this, but it should be the other way around: > > define T, S, R as TypeVar > > Yes, I *know* the name being bound comes after 'as' in > other places, but I think consistency with the obvious > English meaning is more important. > > One other thing, 'define' is so similar to 'def' that > I think it would be needlessly confusing to have both, > so just make it > > def T, S, R as TypeVar > > Also, yes, you would have to curry if you want the > constructor to have arguments. I think that's a small > price to pay for the benefits: less magic, no > restrictions on the form of the constructor expression, > and the ability to re-use it for multiple bound names. > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cyscoyote at gmail.com Tue May 31 21:56:50 2016 From: cyscoyote at gmail.com (Yongsheng Cheng) Date: Wed, 01 Jun 2016 01:56:50 +0000 Subject: [Python-ideas] Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects In-Reply-To: References: <574D20D9.7060504@gmail.com> <574D89A8.3060306@mail.de> <20160531141625.GV12028@ando.pearwood.info> Message-ID: simplicity matters Bar Harel ?2016?6?1??? ??3:11??? > -1 as well. > > I have used the generator multiplication many times before. The use of the > same object helps a lot in certain cases. > Changing that will cause a confusion among those who know the phenomenon > and would break compatibility. > > Regarding your matrix example, maybe we can make use of the new '@' > operator for copying new objects, what do you say? > > -- Bar Harel > > On Tue, May 31, 2016 at 8:01 PM Terry Reedy wrote: > >> On 5/31/2016 10:16 AM, Steven D'Aprano wrote: >> > On Tue, May 31, 2016 at 01:36:31PM +0000, Joseph Martinot-Lagarde wrote: >> >>> I can only agree here. Even today, despite knowing the fact, it's >> >>> causing some headaches in some cases. >> >> >> >> How about raising an exception if mutable objects are in the list ? >> > >> > -1 >> > >> > It's a gratuitous breakage that cannot solve the problem, because you >> > cannot tell in advance which objects are mutable and which are not. The >> > best you can do is recognise known built-ins. >> > >> > ("list is mutable, tuple is not, except when it contains a mutable >> > item, but mymodule.MySequence may or may not be, there's no way to >> > tell in advance.") >> > >> > >> >> - maybe there are useful use cases of duplicating the reference ? >> > >> > Absolutely. That's a standard way of grouping items taken from an >> > iterable: >> > >> > py> it = iter("hello world!") >> > py> for a, b, c in zip(*[it]*3): # groups of three >> > ... print(a, b, c) >> > ... >> > h e l >> > l o >> > w o r >> > l d ! >> > >> > >> > This wouldn't work if the iterators were three independent copies. >> >> -1 also. As Steven has pointed out several times in other threads, >> Python is consistent in not copying objects unless requested. Thus >> >> a = [1,2,3] >> b = a # a reference copy, not an object copy, as some expect >> a[1] = 4 >> print(b[1]) >> # 4 -- a surprise for those who expect '=' to copy the list object >> >> Sequence * is an elaboration of the same issue. >> >> -- >> Terry Jan Reedy >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: