From steve at pearwood.info Sun Nov 1 01:06:30 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 1 Nov 2015 17:06:30 +1100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: References: <201510291559.t9TFxi4i003479@fido.openend.se> Message-ID: <20151101060629.GE10946@ando.pearwood.info> CC'ing Python-Ideas. Follow-ups to Python-Ideas please. On Thu, Oct 29, 2015 at 09:22:15PM -0400, Terry Reedy wrote: > Leaving IDLE aside, the reason '' is added to sys.path is so that people > can import their own modules. This is very useful. Shadowing is the > result of putting it at the front. I have long thought this a dubious > choice. If '' were instead appended, people could still import modules > that did not duplicate stdlib names. Anyone who wanted shadowing could > move '' to the front. But then shadowing would be intentional, not an > accident. Terry is right. Shadowing should be possible, and it should require a deliberate decision on the part of the programmer. Consider the shell, say, bash or similar. My understanding is that the shell PATH deliberately excludes the current directory because of the possibility of malicious software shadowing usual commands in /bin etc. If you want to run an executable in the current directory, you have to explicitly provide the path to it: ./myscript rather than just myscript. Now Python isn't exactly the shell, and so I'm not proposing that Python does the same thing. But surely we can agree on the following? - Shadowing explicitly installed packages, including the stdlib, is *occasionally* useful. - But when shadowing occurs, it is *nearly always* accidental. - Such accidental shadowing often causes problems. - And further more, debugging shadowing problems is sometimes tricky even for experienced coders, and almost impossible for beginners. (It's not until you've been burned once or thrice by shadowing that you recognise the symptoms, at which point it is then usually easy to debug.) - Hence, we should put the onus on those who want to shadow installed packages) to do so *explicitly*, or at least make it easier to avoid accidental shadowing. I propose the following two changes: (1) Beginning with Python 3.6, the default is that the current directory is put at the end of sys.path rather than the beginning. Instead of: >>> print(sys.path) ['', '/this', '/that', '/another'] we will have this instead: >>> print(sys.path) ['/this', '/that', '/another', ''] Those who don't shadow installed packages won't notice any difference. Scripts which deliberately or unintentionally shadow installed packages will break from this change. I don't have a problem with this. You can't fix harmful behaviour without breaking code that depends on that harmful behaviour. Additionally, I expect that those who rely on the current behaviour will be in a small minority, much fewer than those who will be bitten by accidental shadowing into the indefinite future. And if you want the old behaviour back, it is easy to do so, by changing the path before doing your imports: import sys if sys.path[-1] == "": sys.path = [""] + sys.path[:-1] or equivalent. I do not belive that it is onerous for those who want shadowing to have to take steps to do so explicitly. That can be added to your scripts on a case-by-case basis, or your PYTHONSTARTUP file, by modifying your site.py, or (I think) by putting the code into the sitecustomize or usercustomize modules. (2) IDLE doesn't need to wait for Python 3.6 to make this change. I believe that IDLE is permitted to make backwards incompatible changes in minor releases, so there is no reason why it can't change the path effective immediately. That's a simpler fix than scanning the entire path, raising warnings (which beginners won't understand and will either ignore or panic over) or other complex solutions. It may not prevent *every* shadowing incident, but it will improve the situation immeasurably. Thoughts? -- Steve From storchaka at gmail.com Sun Nov 1 01:41:03 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 1 Nov 2015 08:41:03 +0200 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <20151101060629.GE10946@ando.pearwood.info> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> Message-ID: On 01.11.15 08:06, Steven D'Aprano wrote: > (1) Beginning with Python 3.6, the default is that the current directory > is put at the end of sys.path rather than the beginning. Instead of: > > >>> print(sys.path) > ['', '/this', '/that', '/another'] > > we will have this instead: > > >>> print(sys.path) > ['/this', '/that', '/another', ''] > > Those who don't shadow installed packages won't notice any > difference. > > Scripts which deliberately or unintentionally shadow installed packages > will break from this change. I don't have a problem with this. You can't > fix harmful behaviour without breaking code that depends on that harmful > behaviour. Additionally, I expect that those who rely on the current > behaviour will be in a small minority, much fewer than those who will be > bitten by accidental shadowing into the indefinite future. Unfortunately this is not such small minority https://code.openhub.net/search?s=%22sys.path.pop(0)%22&p=0 But we can workaround this problem by adding harmless path at the strt of sys.path. sys.path.insert(0, '/non-existing-stub-path') or sys.path.insert(0, sys.path[0]) From tjreedy at udel.edu Sun Nov 1 02:26:35 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 1 Nov 2015 02:26:35 -0500 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <20151101060629.GE10946@ando.pearwood.info> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> Message-ID: On 11/1/2015 1:06 AM, Steven D'Aprano wrote: > CC'ing Python-Ideas. Follow-ups to Python-Ideas please. > > On Thu, Oct 29, 2015 at 09:22:15PM -0400, Terry Reedy wrote: > >> Leaving IDLE aside, the reason '' is added to sys.path is so that people >> can import their own modules. This is very useful. Shadowing is the >> result of putting it at the front. I have long thought this a dubious >> choice. If '' were instead appended, people could still import modules >> that did not duplicate stdlib names. Anyone who wanted shadowing could >> move '' to the front. But then shadowing would be intentional, not an >> accident. > > Terry is right. Shadowing should be possible, and it should require a > deliberate decision on the part of the programmer. > > Consider the shell, say, bash or similar. My understanding is that the > shell PATH deliberately excludes the current directory because of the > possibility of malicious software shadowing usual commands in /bin etc. > If you want to run an executable in the current directory, you have to > explicitly provide the path to it: ./myscript rather than just myscript. > > Now Python isn't exactly the shell, and so I'm not proposing that Python > does the same thing. But surely we can agree on the following? > > - Shadowing explicitly installed packages, including the stdlib, is > *occasionally* useful. > > - But when shadowing occurs, it is *nearly always* accidental. > > - Such accidental shadowing often causes problems. > > - And further more, debugging shadowing problems is sometimes tricky > even for experienced coders, and almost impossible for beginners. > > (It's not until you've been burned once or thrice by shadowing that > you recognise the symptoms, at which point it is then usually easy to > debug.) > > - Hence, we should put the onus on those who want to shadow installed > packages) to do so *explicitly*, or at least make it easier to avoid > accidental shadowing. > > > I propose the following two changes: > > > (1) Beginning with Python 3.6, the default is that the current directory > is put at the end of sys.path rather than the beginning. Instead of: > > >>> print(sys.path) > ['', '/this', '/that', '/another'] > > we will have this instead: > > >>> print(sys.path) > ['/this', '/that', '/another', ''] > > Those who don't shadow installed packages won't notice any > difference. Serhiy pointed out that code that unconditionally executes sys.path.pop(0) to avoid shadowing of its stdlib imports might break if it imports from what is now sys.path[1]. This currently (2.7, 3.4, 3.5) is pythonxy.zip. (In 2.7 and 3.4 on Windows, somewhere = /windows/system32, In 3.5, the install directory). I say 'might' because I don't know if the presence of pythonxy.zip always means that /lib is absent. If it is gone, the imports will fail. Since use of zip is rare, the transition might be doable. Adding "if sys.path[0] == '':" is backwark compatible. If the import system had available the set of /lib modules, as I proposed on python-list, then the error handler could diagnose the situation of a lib module import failing because of an existing pythonxy.zip not being on sys.path. I believe the set would also help for generating deprecation messages. > Scripts which deliberately or unintentionally shadow installed packages > will break from this change. I don't have a problem with this. You can't > fix harmful behaviour without breaking code that depends on that harmful > behaviour. Additionally, I expect that those who rely on the current > behaviour will be in a small minority, much fewer than those who will be > bitten by accidental shadowing into the indefinite future. And if you > want the old behaviour back, it is easy to do so, by changing the path > before doing your imports: > > import sys > if sys.path[-1] == "": sys.path = [""] + sys.path[:-1] > > or equivalent. > > I do not belive that it is onerous for those who want shadowing to have > to take steps to do so explicitly. The question is whether it would be too onerous for those explicitly avoiding shadowing now to have to change their explicit code. > That can be added to your scripts on > a case-by-case basis, or your PYTHONSTARTUP file, by modifying your > site.py, or (I think) by putting the code into the sitecustomize or > usercustomize modules. > > (2) IDLE doesn't need to wait for Python 3.6 to make this change. I > believe that IDLE is permitted to make backwards incompatible changes in > minor releases, so there is no reason why it can't change the path > effective immediately. IDLE can change how it operates with its own code. I plan to add a conditional pop before IDLE does its own imports. However, IDLE is not free to change how user code is executed, except to better imitate CPython than it does now. Hence '' will have to be put back for user code execution. -- Terry Jan Reedy From rosuav at gmail.com Sun Nov 1 02:36:08 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 1 Nov 2015 18:36:08 +1100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <20151101060629.GE10946@ando.pearwood.info> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> Message-ID: On Sun, Nov 1, 2015 at 5:06 PM, Steven D'Aprano wrote: > (1) Beginning with Python 3.6, the default is that the current directory > is put at the end of sys.path rather than the beginning. Instead of: > > >>> print(sys.path) > ['', '/this', '/that', '/another'] > > we will have this instead: > > >>> print(sys.path) > ['/this', '/that', '/another', ''] > > Those who don't shadow installed packages won't notice any > difference. > > Scripts which deliberately or unintentionally shadow installed packages > will break from this change. I don't have a problem with this. You can't > fix harmful behaviour without breaking code that depends on that harmful > behaviour. Additionally, I expect that those who rely on the current > behaviour will be in a small minority, much fewer than those who will be > bitten by accidental shadowing into the indefinite future. +1. As Serhiy says, though, this will additionally break scripts that _protect against_ shadowing. We could have an easily-recognized shim in slot zero, as a compatibility measure; not duplicating the next one, but a clearly invalid entry. I suggest None, which appears to work: rosuav at sikorsky:~$ python3 Python 3.6.0a0 (default:9ca59b3cc18b, Oct 16 2015, 15:25:11) [GCC 4.9.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path[0]=None; sys.path.append("") >>> open("string.py","w").write("print('I am a string!')") 23 >>> import string >>> string >>> import doesntexist Traceback (most recent call last): File "", line 1, in ImportError: No module named 'doesntexist' >>> open("uniquename.py","w").write("print('I have a unique name.')") 30 >>> import uniquename I have a unique name. The None entry is happily ignored. Shim removal with "if not sys.path[0]: sys.path.pop(0)" will still remove it. But I had a quick look at the link Serhiy posted, and two of the three entries that I dug into were actually not removing the blank entry (the other removed it and replaced it with a hard path); they had previously inserted something at the beginning of sys.path, and were removing _that_ entry. They won't need to be changed. I did a similar search on GitHub and, again, most of the results weren't actually removing the '' entry from the beginning. So the None could be put there as a compatibility measure for 3.6, and then dropped in 3.7 - or even just left out altogether, so a small number of programs break. Would have been nice to do this at Python 3.0, as it'd have been on par with absolute imports and other edge cases. Oh well. ChrisA From lac at openend.se Sun Nov 1 02:53:17 2015 From: lac at openend.se (Laura Creighton) Date: Sun, 01 Nov 2015 08:53:17 +0100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <20151101060629.GE10946@ando.pearwood.info> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> Message-ID: <201511010753.tA17rHs3025816@fido.openend.se> In a message of Sun, 01 Nov 2015 17:06:30 +1100, "Steven D'Aprano" writes: >I propose the following two changes: > > >(1) Beginning with Python 3.6, the default is that the current directory >is put at the end of sys.path rather than the beginning. Instead of: > > >>> print(sys.path) > ['', '/this', '/that', '/another'] > >we will have this instead: > > >>> print(sys.path) > ['/this', '/that', '/another', ''] > >Those who don't shadow installed packages won't notice any >difference. > >Scripts which deliberately or unintentionally shadow installed packages >will break from this change. I don't have a problem with this. You can't >fix harmful behaviour without breaking code that depends on that harmful >behaviour. This is a bad idea, if you mean 'shadows anything in site-packages'. I write a perfectly good working program, which then silently breaks because somebody happens to install a site package with a name conflict with my code. Can you imagine being in the middle of writing and debugging such a thing and have everything start failing because suddenly your program isn't the one being found? How long is it going to take you to stop looking at your own code, and your own setup for the problem and begin looking at what packages got installed by anybody else sharing this machine, and what they are named? It is not necessarily going to make the teachers' lives any better. They will trade the confusion of 'things are acting strangely around here' for 'I just wrote a program and the stupid langauge cannot find it'. People whose code inadvertantly shadow something are better off with a warning about the potential problem. Laura From rosuav at gmail.com Sun Nov 1 02:58:36 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 1 Nov 2015 18:58:36 +1100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <201511010753.tA17rHs3025816@fido.openend.se> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> Message-ID: On Sun, Nov 1, 2015 at 6:53 PM, Laura Creighton wrote: > In a message of Sun, 01 Nov 2015 17:06:30 +1100, "Steven D'Aprano" writes: > >>I propose the following two changes: >> >> >>(1) Beginning with Python 3.6, the default is that the current directory >>is put at the end of sys.path rather than the beginning. Instead of: >> >> >>> print(sys.path) >> ['', '/this', '/that', '/another'] >> >>we will have this instead: >> >> >>> print(sys.path) >> ['/this', '/that', '/another', ''] >> >>Those who don't shadow installed packages won't notice any >>difference. >> >>Scripts which deliberately or unintentionally shadow installed packages >>will break from this change. I don't have a problem with this. You can't >>fix harmful behaviour without breaking code that depends on that harmful >>behaviour. > > This is a bad idea, if you mean 'shadows anything in site-packages'. > I write a perfectly good working program, which then silently breaks > because somebody happens to install a site package with a name > conflict with my code. Can you imagine being in the middle of > writing and debugging such a thing and have everything start > failing because suddenly your program isn't the one being found? > How long is it going to take you to stop looking at your own code, > and your own setup for the problem and begin looking at what > packages got installed by anybody else sharing this machine, and > what they are named? I'm fairly sure Steven's talking about the standard library, not arbitrary packages people might happen to have installed. Whether he thought about it or not, the solution to both your problems seems to be this: sys.path = ['/usr/local/lib/python36.zip', '/usr/local/lib/python3.6', '/usr/local/lib/python3.6/plat-linux', '/usr/local/lib/python3.6/lib-dynload', '', '/home/rosuav/.local/lib/python3.6/site-packages', '/usr/local/lib/python3.6/site-packages'] Equivalently, for the system Python: sys.path = ['/usr/lib/python3.4', '/usr/lib/python3.4/plat-x86_64-linux-gnu', '/usr/lib/python3.4/lib-dynload', '', '/usr/local/lib/python3.4/dist-packages', '/usr/lib/python3/dist-packages'] So the current directory is after everything that's logically the standard library, but before things that would be installed separately (site-packages, dist-packages, etc). It'd no longer be possible to shadow wave.py, but you could shadow sqlalchemy.py. ChrisA From lac at openend.se Sun Nov 1 03:08:36 2015 From: lac at openend.se (Laura Creighton) Date: Sun, 01 Nov 2015 09:08:36 +0100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> Message-ID: <201511010808.tA188a77027105@fido.openend.se> In a message of Sun, 01 Nov 2015 18:58:36 +1100, Chris Angelico writes: >sys.path = ['/usr/local/lib/python36.zip', '/usr/local/lib/python3.6', >'/usr/local/lib/python3.6/plat-linux', >'/usr/local/lib/python3.6/lib-dynload', '', >'/home/rosuav/.local/lib/python3.6/site-packages', >'/usr/local/lib/python3.6/site-packages'] > >Equivalently, for the system Python: > >sys.path = ['/usr/lib/python3.4', >'/usr/lib/python3.4/plat-x86_64-linux-gnu', >'/usr/lib/python3.4/lib-dynload', '', >'/usr/local/lib/python3.4/dist-packages', >'/usr/lib/python3/dist-packages'] > > >So the current directory is after everything that's logically the >standard library, but before things that would be installed separately >(site-packages, dist-packages, etc). It'd no longer be possible to >shadow wave.py, but you could shadow sqlalchemy.py. > >ChrisA But the kid who just wrote string.py or turtle.py will still have the 'why isn't this working at all?' experience instead of something that warns her what her problem is. Laura From rosuav at gmail.com Sun Nov 1 04:02:23 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 1 Nov 2015 20:02:23 +1100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <201511010808.tA188a77027105@fido.openend.se> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> <201511010808.tA188a77027105@fido.openend.se> Message-ID: On Sun, Nov 1, 2015 at 7:08 PM, Laura Creighton wrote: > In a message of Sun, 01 Nov 2015 18:58:36 +1100, Chris Angelico writes: >>sys.path = ['/usr/local/lib/python36.zip', '/usr/local/lib/python3.6', >>'/usr/local/lib/python3.6/plat-linux', >>'/usr/local/lib/python3.6/lib-dynload', '', >>'/home/rosuav/.local/lib/python3.6/site-packages', >>'/usr/local/lib/python3.6/site-packages'] >> >>Equivalently, for the system Python: >> >>sys.path = ['/usr/lib/python3.4', >>'/usr/lib/python3.4/plat-x86_64-linux-gnu', >>'/usr/lib/python3.4/lib-dynload', '', >>'/usr/local/lib/python3.4/dist-packages', >>'/usr/lib/python3/dist-packages'] >> >> >>So the current directory is after everything that's logically the >>standard library, but before things that would be installed separately >>(site-packages, dist-packages, etc). It'd no longer be possible to >>shadow wave.py, but you could shadow sqlalchemy.py. >> >>ChrisA > > But the kid who just wrote string.py or turtle.py will still have > the 'why isn't this working at all?' experience instead of something > that warns her what her problem is. Right. The warning when you save a file of that name is still a useful thing; it's orthogonal to this, though. (FWIW I think the warning's a good idea, but it's no panacea.) ChrisA From lac at openend.se Sun Nov 1 04:49:49 2015 From: lac at openend.se (Laura Creighton) Date: Sun, 01 Nov 2015 10:49:49 +0100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> <201511010808.tA188a77027105@fido.openend.se> Message-ID: <201511010949.tA19nnwj001815@fido.openend.se> In a message of Sun, 01 Nov 2015 20:02:23 +1100, Chris Angelico writes: >> But the kid who just wrote string.py or turtle.py will still have >> the 'why isn't this working at all?' experience instead of something >> that warns her what her problem is. > >Right. The warning when you save a file of that name is still a useful >thing; it's orthogonal to this, though. (FWIW I think the warning's a >good idea, but it's no panacea.) > >ChrisA Adding warnings to IDLE when you save a file is a fine idea, and will help IDLE users avoid pain. I want to help _everybody_ with a change to Python, so that it issues a warning when you shadow something in the standard library. Something like: Warning: local file /u/lac/junk/string.py shadows module named string in the Standard Library Note, for somebody implementing this. Teachers will find the message slightly more useful if it prints the full path name. Laura From agtoever at hotmail.com Sun Nov 1 06:08:44 2015 From: agtoever at hotmail.com (Albert ten Oever) Date: Sun, 1 Nov 2015 11:08:44 +0000 Subject: [Python-ideas] Include partitioning in itertools Message-ID: Hi all, I really hope this isn't proposed before - I couldn't find anything in the archives. I want to propose to include a 'partition' (mathematically more correct: a set partition) function in itertools. To my knowledge, partitioning of a set (iterable) is quite a common thing to do and a logic extension of the combinatoric generators in itertools. It is implemented as a Python recipe (http://code.activestate.com/recipes/576795-partitioning-a-sequence/). I found that implementing a partition generator of an iterable isn't very straightforward, which, in my opinion, strengthens the case for implementing it as a separate function. Definitions of partitions: http://mathworld.wolfram.com/SetPartition.html or https://en.wikipedia.org/wiki/Partition_of_a_set Humble regards, Albert. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Nov 1 06:26:30 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 1 Nov 2015 22:26:30 +1100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> <201511010808.tA188a77027105@fido.openend.se> Message-ID: <20151101112630.GF10946@ando.pearwood.info> On Sun, Nov 01, 2015 at 08:02:23PM +1100, Chris Angelico wrote: > On Sun, Nov 1, 2015 at 7:08 PM, Laura Creighton wrote: [...] > > But the kid who just wrote string.py or turtle.py will still have > > the 'why isn't this working at all?' experience instead of something > > that warns her what her problem is. > > Right. The warning when you save a file of that name is still a useful > thing; it's orthogonal to this, though. (FWIW I think the warning's a > good idea, but it's no panacea.) I disagree. As shown by the tutor and python-list mailing lists, beginners don't read error messages. Warning them that they're about to save a file using the name of a stdlib module will be no different. They'll either ignore the warning, and still be in the dark as to why their code is broken, or they'll panic that they did something wrong, and possibly lose their as yet unsaved work. What produces the warning? Python, or the editor? If Python, you're annoying experienced programmers who intend to do what they do. If the editor, you do nothing about people using a different editor, or people who move and rename files in the shell. -- Steve From steve at pearwood.info Sun Nov 1 06:42:50 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 1 Nov 2015 22:42:50 +1100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <201511010949.tA19nnwj001815@fido.openend.se> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> <201511010808.tA188a77027105@fido.openend.se> <201511010949.tA19nnwj001815@fido.openend.se> Message-ID: <20151101114250.GG10946@ando.pearwood.info> On Sun, Nov 01, 2015 at 10:49:49AM +0100, Laura Creighton wrote: > Adding warnings to IDLE when you save a file is a fine idea, and will > help IDLE users avoid pain. IDLE users must be different from most computer users then, because most computer users don't read warnings or errors. http://ignorethecode.net/blog/2008/10/31/nobody-reads/ http://stackoverflow.com/questions/125269/how-would-you-handle-users-who-dont-read-dialog-boxes > I want to help _everybody_ with a change > to Python, so that it issues a warning when you shadow something in > the standard library. It's not enough to protect the stdlib. I've seen people accidently shadow numpy, or other third-party modules. -- Steve From steve at pearwood.info Sun Nov 1 06:44:03 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 1 Nov 2015 22:44:03 +1100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> Message-ID: <20151101114402.GH10946@ando.pearwood.info> On Sun, Nov 01, 2015 at 08:41:03AM +0200, Serhiy Storchaka wrote: > On 01.11.15 08:06, Steven D'Aprano wrote: [...] > >Scripts which deliberately or unintentionally shadow installed packages > >will break from this change. I don't have a problem with this. You can't > >fix harmful behaviour without breaking code that depends on that harmful > >behaviour. Additionally, I expect that those who rely on the current > >behaviour will be in a small minority, much fewer than those who will be > >bitten by accidental shadowing into the indefinite future. > > Unfortunately this is not such small minority > > https://code.openhub.net/search?s=%22sys.path.pop(0)%22&p=0 The search results contain MANY duplicates. For example, in the first ten results, there are three duplicates of "common.py" from "kongove's autotest", and two duplicates of "common.py" from "Chromium OS". The first hit does: sys.path.insert(0, path.dirname(__file__)) import objects from objects import constants sys.path.pop(0) which seems to be a very common pattern: insert something into the start of the path, then pop it out later. That's harmless, and won't be effected by shifting where "" in inserted. So I think that this search is not a good test for code that will be effected. Besides, anyone who unconditionally pops the first item from sys.path is already on shakey ground. You should not assume that the first item will always be "", since it may have been changed before your code runs, e.g. by the PYTHONSTARTUP file, usercustomize, etc. Now, we shouldn't break people's code for no good reason, not even it it is already broken, but we have a good reason: having "" at the start of sys.path breaks code that inadvertently shadows other modules. (And it may even be a security risk.) -- Steve From lac at openend.se Sun Nov 1 07:20:43 2015 From: lac at openend.se (Laura Creighton) Date: Sun, 01 Nov 2015 13:20:43 +0100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <20151101114250.GG10946@ando.pearwood.info> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> <201511010808.tA188a77027105@fido.openend.se> <201511010949.tA19nnwj001815@fido.openend.se> <20151101114250.GG10946@ando.pearwood.info> Message-ID: <201511011220.tA1CKhsS013079@fido.openend.se> In a message of Sun, 01 Nov 2015 22:42:50 +1100, "Steven D'Aprano" writes: >On Sun, Nov 01, 2015 at 10:49:49AM +0100, Laura Creighton wrote: > >> Adding warnings to IDLE when you save a file is a fine idea, and will >> help IDLE users avoid pain. > >IDLE users must be different from most computer users then, because most >computer users don't read warnings or errors. > >http://ignorethecode.net/blog/2008/10/31/nobody-reads/ > >http://stackoverflow.com/questions/125269/how-would-you-handle-users-who-dont-read-dialog-boxes This is actually my old field of research. The reason that most users don't read dialog boxes is that they are using operating systems who chat them up using dialog boxes all the time. They are used to having a dialog box mean 'the computer is working as usual'. They do not associate this with warnings or errors. Thus they are oblivious to them. This is the argument against running your C code with lint every single time. People get used to seeing: __Warning: long assignment may lose accuracy__ treat it as noise, and then when you actually want them to find such errors are unable to find them. It's quite astonishing. They can read the lint output to you out loud, and _still_ not be able to do the exercise where they were to find rounding errors in their programs. However, when you divide people into groups and make one group use lint all the time, and tell the others not to use it, until they get to this question -- poof. The second group finds the rounding errors. So if, unknownst to me, lots and lots of people are shadowing the stdlib, then if we issue warnings we may blunt their ability to see warnings in general. And that would be a downside. But the upside is that all of the people who did this inadvertantly would get a warning that actually explained what is going on. And the other thing that is well understood is that people who are learning tend to see warnings and whatnot -- simply because they haven't had the time and the experience to file the warnings as 'just more noise'. >> I want to help _everybody_ with a change >> to Python, so that it issues a warning when you shadow something in >> the standard library. > >It's not enough to protect the stdlib. I've seen people accidently >shadow numpy, or other third-party modules. Yes, but I very much don't want any warnings when you shadow third party modules. If shadowing third party modules produce a warning, then we will end up with 'all warnings get ignored' in very short order. We want warnings to be rare, rare enough that people don't get used to seeing them. If you can point me at a large community of regular intentional shadowers of the standard library, that quite likely would be enough for me to think that warning people is a bad idea -- having people get into the habit of ignoring warnings is a really dreadful outcome. But the argument isn't 'we shouldn't issue warnings because people don't read them' but rather 'we shouldn't make warnings commonplace because that makes people incapable of reading them'. >-- >Steve (This ignores a different group of learners who have been studied, people who cannot learn and read at the same time. Only some of these people have the sorts of problems that get called dyslexia. But video tutorials are making life a lot easier for these people nowadays. And it is hard to see how adding a warning could harm these learners.) Laura From tritium-list at sdamon.com Sun Nov 1 08:25:54 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Sun, 01 Nov 2015 08:25:54 -0500 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> Message-ID: <563612E2.1060201@sdamon.com> Honestly, shadowing modules is something that should be solved by renaming modules. If you are worrying about shadowing ONLY the standard library - guess what? those names don't change often, and are well known. Don't use those names. If you are talking about shadowing site-packages, or any package anywhere on sys.path, you are bound to break something somewhere in a hard to debug way. It is sufficient to allow things to break and instruct people to jump in the repl, import a module and check __file__. On 11/1/2015 02:58, Chris Angelico wrote: > On Sun, Nov 1, 2015 at 6:53 PM, Laura Creighton wrote: >> In a message of Sun, 01 Nov 2015 17:06:30 +1100, "Steven D'Aprano" writes: >> >>> I propose the following two changes: >>> >>> >>> (1) Beginning with Python 3.6, the default is that the current directory >>> is put at the end of sys.path rather than the beginning. Instead of: >>> >>> >>> print(sys.path) >>> ['', '/this', '/that', '/another'] >>> >>> we will have this instead: >>> >>> >>> print(sys.path) >>> ['/this', '/that', '/another', ''] >>> >>> Those who don't shadow installed packages won't notice any >>> difference. >>> >>> Scripts which deliberately or unintentionally shadow installed packages >>> will break from this change. I don't have a problem with this. You can't >>> fix harmful behaviour without breaking code that depends on that harmful >>> behaviour. >> This is a bad idea, if you mean 'shadows anything in site-packages'. >> I write a perfectly good working program, which then silently breaks >> because somebody happens to install a site package with a name >> conflict with my code. Can you imagine being in the middle of >> writing and debugging such a thing and have everything start >> failing because suddenly your program isn't the one being found? >> How long is it going to take you to stop looking at your own code, >> and your own setup for the problem and begin looking at what >> packages got installed by anybody else sharing this machine, and >> what they are named? > I'm fairly sure Steven's talking about the standard library, not > arbitrary packages people might happen to have installed. Whether he > thought about it or not, the solution to both your problems seems to > be this: > > sys.path = ['/usr/local/lib/python36.zip', '/usr/local/lib/python3.6', > '/usr/local/lib/python3.6/plat-linux', > '/usr/local/lib/python3.6/lib-dynload', '', > '/home/rosuav/.local/lib/python3.6/site-packages', > '/usr/local/lib/python3.6/site-packages'] > > Equivalently, for the system Python: > > sys.path = ['/usr/lib/python3.4', > '/usr/lib/python3.4/plat-x86_64-linux-gnu', > '/usr/lib/python3.4/lib-dynload', '', > '/usr/local/lib/python3.4/dist-packages', > '/usr/lib/python3/dist-packages'] > > > So the current directory is after everything that's logically the > standard library, but before things that would be installed separately > (site-packages, dist-packages, etc). It'd no longer be possible to > shadow wave.py, but you could shadow sqlalchemy.py. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rosuav at gmail.com Sun Nov 1 08:44:27 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 Nov 2015 00:44:27 +1100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <563612E2.1060201@sdamon.com> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> <563612E2.1060201@sdamon.com> Message-ID: On Mon, Nov 2, 2015 at 12:25 AM, Alexander Walters wrote: > If you are worrying about shadowing ONLY the standard library - guess > what? those names don't change often, and are well known. Don't use those > names. Well known? Okay. No cheating now! Which of these names will shadow something from the standard library, and which don't? code.py cgi.py chunk.py cmd.py cprofile.py gc.py html.py imp.py mailbox.py numbers.py test.py this.py types.py wave.py Every one of these is a reasonably plausible name for a quick throw-away script (maybe a test script as you learn how to use something). How many of them will be problematic? ChrisA From python at lucidity.plus.com Sun Nov 1 08:45:21 2015 From: python at lucidity.plus.com (Erik) Date: Sun, 1 Nov 2015 13:45:21 +0000 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <201511010949.tA19nnwj001815@fido.openend.se> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> <201511010808.tA188a77027105@fido.openend.se> <201511010949.tA19nnwj001815@fido.openend.se> Message-ID: <56361771.9090702@lucidity.plus.com> On 01/11/15 09:49, Laura Creighton wrote: > Adding warnings to IDLE when you save a file is a fine idea, and will > help IDLE users avoid pain. I want to help _everybody_ with a change > to Python, so that it issues a warning when you shadow something in > the standard library. I think that a solution to this could be something along the lines of having a command line switch (let's say '-w') or other way of enabling a "shadow module checking" mechanism. When an exception is thrown that is caught by the default handler and output to stderr, an additional line of output could be appended: "Note: This problem could be caused by a module that shadows a standard library or third party module. Run Python with the '-w' switch for more detail." I'm not sure if this would be suitable for _all_ exceptions or only those from certain points in the exception hierarchy. When the '-w' switch is enabled, the default exception handler would instead enumerate all active modules and, for any which has another module with the same name further down the search path than the one that is loaded, would output some sort of diagnostic message (i.e., a list of which loaded modules are shadowing _something_else_ that would have been loaded otherwise) - similar to your example below: > Something like: > Warning: local file /u/lac/junk/string.py shadows module named string in the > Standard Library This would prevent "power users" from always having to see a potentially large diagnostic after each uncaught exception, but generally remind everyone that such a thing is possible and "here's the easy way to check for it". A bit like Valgrind's memcheck tool's "--leak-check=full" option - when run without it, if there are leaks then memcheck reminds the user that that's the way to start digging down into what might be causing them. E. From lac at openend.se Sun Nov 1 08:53:23 2015 From: lac at openend.se (Laura Creighton) Date: Sun, 01 Nov 2015 14:53:23 +0100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <563612E2.1060201@sdamon.com> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> <563612E2.1060201@sdamon.com> Message-ID: <201511011353.tA1DrNkI019815@fido.openend.se> In a message of Sun, 01 Nov 2015 08:25:54 -0500, Alexander Walters writes: >Honestly, shadowing modules is something that should be solved by >renaming modules. If you are worrying about shadowing ONLY the standard >library - guess what? those names don't change often, and are well >known. Don't use those names. The problem is that the sort of people who make these errors don't know the names. Knowing that 'turtle.py' is a bad name for 'my very first program that does turtle graphics' is a more advanced skill than the turtle program writers, or even their teachers, can be expected to have. I'm willing to have the kids and their teachers (and all the other beginners) suffer if the downside is that an even only moderately-sized set of legitimate stdlib shadowers end up with an inability to notice warning messages, because they are now ubiquitous. I just think that legitimate stdlib shadowers are rare. And I suspect that many of them do this once, for a short period, not every day of their lives. Thus the 'ignore this noise' is much less likely to develop with them. Python warnings are rare. Choosing to add a new one is a serious step. I do not want to open the floodgates and end up warning about all possibly-silly practice -- I am one of the cognitive psychology boffins who go around asking/nudging/reminding people to warn less often if you want your most serious warnings to be heeded. I just think the tradeoffs here are worth it. Laura From steve at pearwood.info Sun Nov 1 10:06:33 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 2 Nov 2015 02:06:33 +1100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> <563612E2.1060201@sdamon.com> Message-ID: <20151101150633.GI10946@ando.pearwood.info> On Mon, Nov 02, 2015 at 12:44:27AM +1100, Chris Angelico wrote: > On Mon, Nov 2, 2015 at 12:25 AM, Alexander Walters > wrote: > > If you are worrying about shadowing ONLY the standard library - guess > > what? those names don't change often, and are well known. Don't use those > > names. > > Well known? Okay. No cheating now! Which of these names will shadow > something from the standard library, and which don't? > > code.py > cgi.py > chunk.py [...] Does it matter? It really only matters if you shadow something that gets imported. It's not exactly great practice to name your script "code.py", but if you do, and it doesn't directly or indirectly try to import the code library, it's harmless. That's why it is harmful to present users with an alert (which they won't read) warning them that they're shadowing a stdlib module. -- Steve From rosuav at gmail.com Sun Nov 1 10:24:37 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 Nov 2015 02:24:37 +1100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <20151101150633.GI10946@ando.pearwood.info> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> <563612E2.1060201@sdamon.com> <20151101150633.GI10946@ando.pearwood.info> Message-ID: On Mon, Nov 2, 2015 at 2:06 AM, Steven D'Aprano wrote: > On Mon, Nov 02, 2015 at 12:44:27AM +1100, Chris Angelico wrote: >> On Mon, Nov 2, 2015 at 12:25 AM, Alexander Walters >> wrote: >> > If you are worrying about shadowing ONLY the standard library - guess >> > what? those names don't change often, and are well known. Don't use those >> > names. >> >> Well known? Okay. No cheating now! Which of these names will shadow >> something from the standard library, and which don't? >> >> code.py >> cgi.py >> chunk.py > [...] > > Does it matter? It really only matters if you shadow something that gets > imported. > > It's not exactly great practice to name your script "code.py", but if > you do, and it doesn't directly or indirectly try to import the code > library, it's harmless. That's why it is harmful to present users with > an alert (which they won't read) warning them that they're shadowing a > stdlib module. It matters under two circumstances: 1) Currently, if the stdlib module ends up getting imported *even indirectly* by something you use. The indirect import is the subtle one here. 2) Under the proposal to reorder sys.path, if you want to import *your* module, you wouldn't be able to. I'm responding to the suggestion that the standard library names are "well known" to the point where people can be told not to use them. I say they are not; sure, everyone knows that 'sys.py' would be a bad name, and you wouldn't call your module "pprint.py" unless you deliberately want to shadow the normal standard library module, but there are a lot that are more obscure. Suppose you start learning about Python's data types, and you create a "types.py" that lets you play with strings, integers, floats, and so on. Then, in the next lesson, you meet enumerations, and enum.py. $ touch types.py; python3 -c "import enum" Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.6/enum.py", line 2, in from types import MappingProxyType, DynamicClassAttribute ImportError: cannot import name 'MappingProxyType' Tell me, is this an implausible scenario? Should a person just learning about Python's type system have to also learn not to use the name "types.py"? ChrisA From steve at pearwood.info Sun Nov 1 10:44:10 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 2 Nov 2015 02:44:10 +1100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <201511010753.tA17rHs3025816@fido.openend.se> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> Message-ID: <20151101154410.GJ10946@ando.pearwood.info> On Sun, Nov 01, 2015 at 08:53:17AM +0100, Laura Creighton wrote: > In a message of Sun, 01 Nov 2015 17:06:30 +1100, "Steven D'Aprano" writes: > >Scripts which deliberately or unintentionally shadow installed packages > >will break from this change. I don't have a problem with this. You can't > >fix harmful behaviour without breaking code that depends on that harmful > >behaviour. > > This is a bad idea, if you mean 'shadows anything in site-packages'. > I write a perfectly good working program, which then silently breaks > because somebody happens to install a site package with a name > conflict with my code. I'm willing to consider that "" should appear in the middle of sys.path: [standard-library-directories, "", site-package-directories] but that still allows accidental shadowing of third-party packages. I don't think that the problem of accidently shadowing numpy is less deserving of a solution than accidently shadowing random. They're both problems. The downside of using a search path, instead of explicit full pathnames, is that you can have conflicts where two or more modules have the same name, and only the first can be imported. That's unavoidable. We can only try to minimize the risk of accidental shadowing, not prevent it altogether. > Can you imagine being in the middle of > writing and debugging such a thing and have everything start failing > because suddenly your program isn't the one being found? How long is > it going to take you to stop looking at your own code, and your own > setup for the problem and begin looking at what packages got installed > by anybody else sharing this machine, and what they are named? Now just a minute, I'm not proposing that this change occur at random, in the middle of a debugging session. Not even in a point (micro) release. So the code will only break (if it breaks) when you upgrade from version 3.x to 3.6 or higher. Moving to a new minor version number is a big step, and people expect that things *could* break. (They don't necessarily expect that they *will* break, but it's certainly a possibility that they might.) As an experienced developer, what do you do when code that works in one version stops working in the next? Everyone I know immediately assumes that something has changed in the new Python version, and proceeds from there. In this case, they'll be right. Hopefully they will read the release notes. Or the documentation. Or ask on StackOverflow. The same applies for when you install a new package: "Everything worked fine yesterday." "What changed?" "Well, I installed a new package with the same name as one of my modules." "Well there you go." Of course we can come up with scenarios that are more confusing. Suppose you have a program that you don't use very often, say, once a year, that relies on "" being at the front of the path, and then somebody who isn't you installs a site-wide third-party package that now shadows one of your modules, and you don't find out about for six months. But this sort of thing can already happen: any time you add a third-party module, it may shadow *other* third party modules that are found later in the path. So I'm not creating a brand new failure mode, I'm just changing how it applies a little. There are all sorts of debugging strategies available to an experienced developer, starting from "print out module.__file__" to "use the debugger", which a beginner who has just written "turtle.py" would never think of. If I'm going to break anyone's code, I'd rather it be yours than theirs, since I have every confidence you can debug any problems fairly quickly. I don't suggest breaking working code lightly, but the issue of shadowing the stdlib is an ever-present millstone around every Python developer's neck. The stdlib alone, never mind site packages, is too big to expect everyone to memorise what names they should not use. We can't realistically expect people to avoid shadowing. But we can shift it from "easy to shadow, and mostly affect those who are least able to cope" to "harder to shadow, and mostly affect those who can cope". -- Steve From lac at openend.se Sun Nov 1 11:59:40 2015 From: lac at openend.se (Laura Creighton) Date: Sun, 01 Nov 2015 17:59:40 +0100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <20151101154410.GJ10946@ando.pearwood.info> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> <20151101154410.GJ10946@ando.pearwood.info> Message-ID: <201511011659.tA1GxeiG001458@fido.openend.se> In a message of Mon, 02 Nov 2015 02:44:10 +1100, "Steven D'Aprano" writes: >On Sun, Nov 01, 2015 at 08:53:17AM +0100, Laura Creighton wrote: >> In a message of Sun, 01 Nov 2015 17:06:30 +1100, "Steven D'Aprano" writes: >but that still allows accidental shadowing of third-party packages. I >don't think that the problem of accidently shadowing numpy is less >deserving of a solution than accidently shadowing random. They're both >problems. And I do. seriously. I think that shadowing third party packages is something I do all the time. I am not willing to share a namespace with all the package writers in the universe, and should I write package name with the same name as somebody else's third party package, I damn well want mine, not some other one. But learners shadowing the stdlib are a separate case. We've had similar arguments before, the slipperly slope sort. Either you don't want to solve any unless all get solved, or else you are afraid that minor precedent here will lead to all hell breaking loose when it is extended to its ultimate conclusion. But I think the entire purpose of being a human being is to make such judgments. And I think that this is an easy one. Shadowing the stdlib can be treated as a catchable mistake. Shadowing an outside package is something we want to do all the time. >> because suddenly your program isn't the one being found? How long is >> it going to take you to stop looking at your own code, and your own >> setup for the problem and begin looking at what packages got installed >> by anybody else sharing this machine, and what they are named? > >Now just a minute, I'm not proposing that this change occur at random, >in the middle of a debugging session. Not even in a point (micro) >release. So the code will only break (if it breaks) when you upgrade >from version 3.x to 3.6 or higher. Moving to a new minor version number >is a big step, and people expect that things *could* break. Ah, I think your thinking is less than clear on this one. 1. my sysadmin does an update and suddently the system python is a different one than the one where I wrote my code. some poor sysadmin soul who never wrote anything gets to deal with the fact that my code now has name conflicts with packages that we have, but I never knew about and never imported. 2. things fall over because the new verision of python, in effect, shadows my code. I am not willing to share my namespace with every other python package creator on the planet. I am a consultant. When my code breaks, they call me. I am unwilling to have my code break every time some human being decides to make a package named something I already used over the last 18 years of writing python code. (I know that 3.x has not been out that long, but 18 years from now, the same condition holds.) Laura From srkunze at mail.de Sun Nov 1 12:14:20 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Sun, 1 Nov 2015 18:14:20 +0100 Subject: [Python-ideas] Include partitioning in itertools In-Reply-To: References: Message-ID: <5636486C.6080401@mail.de> Hi Albert, sounds useful. I can remember needing it myself. On 01.11.2015 12:08, Albert ten Oever wrote: > Hi all, > > I really hope this isn't proposed before - I couldn't find anything in > the archives. > > I want to propose to include a 'partition' (mathematically more > correct: a set partition) function in itertools. This operates over iterables, right? So, it's not quite about sets alone. > To my knowledge, partitioning of a set (iterable) is quite a common > thing to do and a logic extension of the combinatoric generators in > itertools. > > It is implemented as a Python recipe > (http://code.activestate.com/recipes/576795-partitioning-a-sequence/). > I found that implementing a partition generator of an iterable isn't > very straightforward, which, in my opinion, strengthens the case for > implementing it as a separate function. Not quite sure if I understand the implementation. Reading this http://mathworld.wolfram.com/BellNumber.html for the given example of 6, it should have 203 possible partitions. What am I missing? The implementation over there reads as if one gets a list of all possible partitions. Picking a specific subset ("all 4-part-partitions") could be useful as well. > Definitions of partitions: > http://mathworld.wolfram.com/SetPartition.html or > https://en.wikipedia.org/wiki/Partition_of_a_set > > Humble regards, > > Albert. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Nov 1 12:21:23 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 1 Nov 2015 09:21:23 -0800 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <20151101154410.GJ10946@ando.pearwood.info> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> <20151101154410.GJ10946@ando.pearwood.info> Message-ID: On Nov 1, 2015 7:44 AM, "Steven D'Aprano" wrote: > > On Sun, Nov 01, 2015 at 08:53:17AM +0100, Laura Creighton wrote: > > In a message of Sun, 01 Nov 2015 17:06:30 +1100, "Steven D'Aprano" writes: > > > >Scripts which deliberately or unintentionally shadow installed packages > > >will break from this change. I don't have a problem with this. You can't > > >fix harmful behaviour without breaking code that depends on that harmful > > >behaviour. > > > > This is a bad idea, if you mean 'shadows anything in site-packages'. > > I write a perfectly good working program, which then silently breaks > > because somebody happens to install a site package with a name > > conflict with my code. > > I'm willing to consider that "" should appear in the middle of sys.path: > > [standard-library-directories, "", site-package-directories] > > but that still allows accidental shadowing of third-party packages. I > don't think that the problem of accidently shadowing numpy is less > deserving of a solution than accidently shadowing random. They're both > problems. In this case there actually is a common use case [1] that involves issuing using the "" entry to intentionally shadow third-party packages: if you're hacking on foo v1.3.dev, and want to test out the code you're writing as opposed to the installed version of foo v1.2, then switching to the root of your source checkout before running python or nosetests or whatever suffices. (OTOH it is also the true that numpy has had enough users run into problems due to trying to run python while accidentally in the root of a source checkout that numpy/__init__.py contains special case code to check for this case and error out early.) -n [1] Commonality here is assessed using the standard experimental procedure, i.e., "*I* do it so it's common". -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Nov 1 12:40:23 2015 From: brett at python.org (Brett Cannon) Date: Sun, 01 Nov 2015 17:40:23 +0000 Subject: [Python-ideas] Add 'module' module, similar to 'keyword' module In-Reply-To: References: Message-ID: On Sat, 31 Oct 2015 at 11:33 Steve Dower wrote: > Could we add a context manager to importlib (or perhaps site or sys) to > temporarily disable imports from non-standard paths? I don't see any safe > way to change the default behavior, but no reason we can't make it easy for > applications to self-isolate. > There's https://www.python.org/dev/peps/pep-0406/ which tried to get isolated import data. As for an immediate solution, trick is whether we have stored somewhere what the standard path entries are or if we have a reliable way to figure out which paths are standard or not. There is also the problem of not being thread-safe when swapping out sys.path entries. -Brett > > Top-posted from my Windows Phone > ------------------------------ > From: Brett Cannon > Sent: ?10/?31/?2015 9:50 > To: Terry Reedy ; python-ideas at python.org > Subject: Re: [Python-ideas] Add 'module' module, similar to 'keyword' > module > > > > On Fri, 30 Oct 2015 at 22:57 Terry Reedy wrote: > >> On 10/30/2015 9:19 PM, Guido van Rossum wrote: >> > There's sys.builtin_module_names which returns the names of the >> > hardcoded builtin modules. >> >> Great. With this solved, I opened an issue for IDLE. >> https://bugs.python.org/issue25522 >> >> > Dynamically loaded modules can be found by >> > searching sys.path in the usual way -- importlib shoul know. I wonder if >> > just asking importlib whether it can locate a given module would be >> enough? >> >> The default search order is stdlib builtins, local user files, /lib >> files, so the shadowing issue the opposite for builtin and /lib modules. >> Hence a different message is needed. >> > > Quick and dirty way to use importlib is to get the location of the stdlib > (os.__file__ should work since it's hard-coded in the interpreter as > representing where the stdlib is) and then use importlib.find_spec() for a > module name to check if the file location in the spec has the same location > prefix as os or not (make sure you use absolute paths since it isn't > guaranteed if you don't execute site.py). > > I've now seen this use case, the logging one, and the 2to3 module rename. > I'm starting to wonder if there some general solution that should get added > to the import machinery that can serve these cases more easily than with > importers or __import__ overrides that can be tricky to get right. > > -Brett > > >> >> > On Fri, Oct 30, 2015 at 6:09 PM, Terry Reedy >> > > > > wrote: >> > >> > This idea results from issue of user files shadowing stdlib files on >> > import. There was a thread on pydev about this yesterday. There is >> > also an opposite issue of builtin modules shadowing user files. >> > >> > The keyword module provides kwlist and iskeyword function. One use >> > of kwlist is used in some other stdlib modules and can be used by >> > syntax highlighters (as in IDLE). Kwlist is updated by the main >> > function. >> > >> > A module module would have at least liblist and islibmodule >> > function. Liblist would contain all directories with __init__.py and >> > all .py files. (I don't think files within package directories >> > should be included, as there is no direct shadowing problem.) A >> > python oriented editor could then warn on save requests "This name >> > matches a stdlib name in /Lib. If you run python in this directory, >> > you will not be able to import the stdlib module. Continue?". >> > >> > The module should also have binlist and isbinmodule for builtin >> > modules. (I do not know how to get such a list. If necessary, an >> > api could be added.) An editor could than warn "This name matches a >> > builtin stdlib name. You will not be able to import this file. >> > Continue?". >> > >> > -- >> > Terry Jan Reedy >> > >> > _______________________________________________ >> > Python-ideas mailing list >> > Python-ideas at python.org >> > >> > https://mail.python.org/mailman/listinfo/python-ideas >> > Code of Conduct: http://python.org/psf/codeofconduct/ >> > >> > >> > >> > >> > -- >> > --Guido van Rossum (python.org/~guido ) >> > >> > >> > _______________________________________________ >> > Python-ideas mailing list >> > Python-ideas at python.org >> > https://mail.python.org/mailman/listinfo/python-ideas >> > Code of Conduct: http://python.org/psf/codeofconduct/ >> > >> >> >> -- >> Terry Jan Reedy >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sun Nov 1 13:10:08 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 1 Nov 2015 20:10:08 +0200 Subject: [Python-ideas] Include partitioning in itertools In-Reply-To: References: Message-ID: On 01.11.15 13:08, Albert ten Oever wrote: > I really hope this isn't proposed before - I couldn't find anything in > the archives. > > I want to propose to include a 'partition' (mathematically more correct: > a set partition) function in itertools. To my knowledge, partitioning of > a set (iterable) is quite a common thing to do and a logic extension of > the combinatoric generators in itertools. > > It is implemented as a Python recipe > (http://code.activestate.com/recipes/576795-partitioning-a-sequence/). I > found that implementing a partition generator of an iterable isn't very > straightforward, which, in my opinion, strengthens the case for > implementing it as a separate function. The implementation can be simpler and don't require itertools: def partition(seq): if seq: yield [seq] for i in range(1, len(seq)): s = seq[i:] for p in partition(seq[:i]): p.append(s) yield p >>> from pprint import pprint >>> pprint(sorted(partition('abcde'), key=len)) [['abcde'], ['a', 'bcde'], ['ab', 'cde'], ['abc', 'de'], ['abcd', 'e'], ['a', 'b', 'cde'], ['a', 'bc', 'de'], ['ab', 'c', 'de'], ['a', 'bcd', 'e'], ['ab', 'cd', 'e'], ['abc', 'd', 'e'], ['a', 'b', 'c', 'de'], ['a', 'b', 'cd', 'e'], ['a', 'bc', 'd', 'e'], ['ab', 'c', 'd', 'e'], ['a', 'b', 'c', 'd', 'e']] From lukasjuhrich at wh2.tu-dresden.de Sun Nov 1 13:30:09 2015 From: lukasjuhrich at wh2.tu-dresden.de (Lukas Juhrich) Date: Sun, 1 Nov 2015 19:30:09 +0100 Subject: [Python-ideas] Include partitioning in itertools In-Reply-To: <5636486C.6080401@mail.de> References: <5636486C.6080401@mail.de> Message-ID: <56365A31.6020803@wh2.tu-dresden.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi Sven, The issue here is that such an implementation does not allow ?gaps? and implicitly enforces that the elements of such a partition only contain elements directly following each other, thus making it not a complete implementation of producing a set partition. To give a specific counterexample, `['acd', 'bef']` will never be produced. The question to ask is whether you want to implement exact mathematical set partition, or if you implement something with the above constraint, and call it something different like ?iterable partition?. If choosing the latter, however, it should be made clear by the function name and in the documentation that this is *not* comparable to set partition. Perhaps it will be useful to implement both concepts, since such an ?iterable partition? seems to be easier to implement and will perhaps be faster (which, however, is a blind guess and depends on the implementation of producing a set partition). - -- Sincerely, Lukas -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJWNlotAAoJENBP4KTXGFXeSJ8P/33IfUqVe+LqKjdjghOVrb2A 2wcj85GJHV5hu8UIYNfchVwvBML7w0LpgvHsopD7NkXLYepOy1AuSDTlePYZGu2g TvVFLd861HZVAQFtFwuuY1bAkNlCDDtNOUZYrortkc7F05TFBaGiTu7Z7zrHG9PN 5eLZ7dpr5y9Ok8nCaerUVnxAn81+A+mr0O2KPAAOfTLA0C0bo4uWJsp1WJMcLF1N o0RpzrZ68Qy1vzfnRaf5T/MpcxIFaPxTYee8wxg/pGse5gPJidvK83G5MV/ylHbK INqG7iPTcD8DJ0OpvR/462QaxBf225fP3Dg+xUtKvpBw2SMrSfc8X9VXOE7EFMJ+ m9Wajx5d1+wVvlTzdEa5Nu5A7I5DGDO+zW7jeuZ2Ri6q6aeKHRt1JLsi/5VIJ+/B 0xb7/a5JKcjskrZuQrlCzVKlvOUq2Wf1SZKlPLq6jsB3xkT1qJhF2QssoR3s2aNn 998gwRl3sJzrdVh9+QxCXjoQ92aqQCLRsvHll+uZ9bjJEhD3SXlyFhqmhLf396fq BMWpxFrSuf4ltqke2WQrzrvpJnxd1MH7yH+2tVQej8/kUHJyCWdBLZeQJeFljiJa 3zYh6BQ8KjwtjNcSHzmhANO/0FSwzVDPSvP2ocEbjYjv5WMXXk0/sNdRgsb3gkQ1 5zuBS6j2n5n9SeUZ0IcM =Ki11 -----END PGP SIGNATURE----- From tjreedy at udel.edu Sun Nov 1 20:10:57 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 1 Nov 2015 20:10:57 -0500 Subject: [Python-ideas] Add 'module' module, similar to 'keyword' module In-Reply-To: <56354996.8030807@brenbarn.net> References: <20151031081450.GD10946@ando.pearwood.info> <56354996.8030807@brenbarn.net> Message-ID: On 10/31/2015 7:07 PM, Brendan Barnwell wrote: > On 2015-10-31 15:45, Terry Reedy wrote: >> Revised and reduced proposal: If other people would find it useful, add >> all_stdlib_toplevel_module_names - builtin_module_names to sys as >> something equivalent to .other_stdlib_module_names or >> .python_coded_module_names. Anyone wanting all_toplevel_module names >> could add the two. Or add the latter, and let others subtract. > > One thing I've sometimes wondered about for Python 4000 is the idea > of putting the whole standard library under a single top-level package > (e.g., from stdlib import sys). This would be a big change but would > reduce the surprises that can arise from stdlib modules named things > like "string", "parser", etc. I see a brief mention of this in PEP 3108 > but that's it. Was there more discussion of the idea? It has been discussed and rejected by Guido. I think it would be more sensible to require 'from . import mymod' for local imports. -- Terry Jan Reedy From stephen at xemacs.org Sun Nov 1 20:25:21 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 2 Nov 2015 10:25:21 +0900 Subject: [Python-ideas] Include partitioning in itertools In-Reply-To: References: Message-ID: <22070.48001.916960.173974@turnbull.sk.tsukuba.ac.jp> Albert ten Oever writes: > I want to propose to include a 'partition' (mathematically more > correct: a set partition) function in itertools. To my knowledge, > partitioning of a set (iterable) is quite a common thing to do and > a logic extension of the combinatoric generators in itertools. Even after other discussion, I don't understand. Sets and iterables are rather different things. The recipe from ActiveState would be perfectly happy with partitioning "aaaaaa", but that sequence is not a reasonable representation of a set (I would normally say it has only one element despite being repeated 6 times), and the results makes no sense as a set, either. For example, as sets ["aa", "aaaa"] and ["aaaa", "aa"] are the same. And I suspect that you might get anomolies using a sequence or iterable-based algorithm (for example, (["a"] is not ["a"]) and ("a" is "a") both evaluate to true because of the way the CPython implementation handles different objects). I'm also not clear on what you would hope to get from an infinite iterator. I assume passing an infinite iterator would be a programmer error, but you could also restrict the function to working on concrete sequences. All that doesn't mean it's a bad idea, but at this point it's definitely underspecified. I'd also prefer a more explicit name, such as enumerate_partitions. Steve From brenbarn at brenbarn.net Sun Nov 1 20:29:28 2015 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Sun, 01 Nov 2015 17:29:28 -0800 Subject: [Python-ideas] Add 'module' module, similar to 'keyword' module In-Reply-To: References: <20151031081450.GD10946@ando.pearwood.info> <56354996.8030807@brenbarn.net> Message-ID: <5636BC78.1040200@brenbarn.net> On 2015-11-01 17:10, Terry Reedy wrote: > On 10/31/2015 7:07 PM, Brendan Barnwell wrote: >> On 2015-10-31 15:45, Terry Reedy wrote: >>> Revised and reduced proposal: If other people would find it useful, add >>> all_stdlib_toplevel_module_names - builtin_module_names to sys as >>> something equivalent to .other_stdlib_module_names or >>> .python_coded_module_names. Anyone wanting all_toplevel_module names >>> could add the two. Or add the latter, and let others subtract. >> >> One thing I've sometimes wondered about for Python 4000 is the idea >> of putting the whole standard library under a single top-level package >> (e.g., from stdlib import sys). This would be a big change but would >> reduce the surprises that can arise from stdlib modules named things >> like "string", "parser", etc. I see a brief mention of this in PEP 3108 >> but that's it. Was there more discussion of the idea? > > It has been discussed and rejected by Guido. I think it would be more > sensible to require 'from . import mymod' for local imports. That doesn't handle the situation where you want to import a third-party module that is neither part of the stdlib nor part of your own project. But I guess my question is answered anyhow. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From mertz at gnosis.cx Sun Nov 1 20:49:22 2015 From: mertz at gnosis.cx (David Mertz) Date: Sun, 1 Nov 2015 17:49:22 -0800 Subject: [Python-ideas] Include partitioning in itertools In-Reply-To: References: Message-ID: This doesn't make sense to me for `itertools`. There is no way to "partition" an infinite sequence in the style of the recipe you show (nor any sensible one I can think of). I think a better name would be `powerset()` anyway; but in either case, it's not an `itertools` type of operation. I think what might be more useful is to make a mix-in class that gave concrete collections a `.powerset()` method that preserved the types of the original collection. I think you can even do this as an iterable without having to concretize the whole set of subsets (but not from an iterable, from a concrete collection, as input). On Sun, Nov 1, 2015 at 3:08 AM, Albert ten Oever wrote: > Hi all, > > I really hope this isn't proposed before - I couldn't find anything in the > archives. > > I want to propose to include a 'partition' (mathematically more correct: a > set partition) function in itertools. To my knowledge, partitioning of a > set (iterable) is quite a common thing to do and a logic extension of the > combinatoric generators in itertools. > > It is implemented as a Python recipe ( > http://code.activestate.com/recipes/576795-partitioning-a-sequence/). I > found that implementing a partition generator of an iterable isn't very > straightforward, which, in my opinion, strengthens the case for > implementing it as a separate function. > > Definitions of partitions: http://mathworld.wolfram.com/SetPartition.html > or https://en.wikipedia.org/wiki/Partition_of_a_set > > Humble regards, > > Albert. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Sun Nov 1 20:54:22 2015 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 1 Nov 2015 19:54:22 -0600 Subject: [Python-ideas] Include partitioning in itertools In-Reply-To: <5636486C.6080401@mail.de> References: <5636486C.6080401@mail.de> Message-ID: [Albert ten Oever] > ... > I want to propose to include a 'partition' (mathematically more correct: a > set partition) function in itertools. > > To my knowledge, partitioning of a set (iterable) is quite a common thing to > do and a logic extension of the combinatoric generators in itertools. > > It is implemented as a Python recipe > (http://code.activestate.com/recipes/576795-partitioning-a-sequence/). [Sven R. Kunze] > ... > Not quite sure if I understand the implementation. Reading this > http://mathworld.wolfram.com/BellNumber.html for the given example of 6, it > should have 203 possible partitions. What am I missing? The ActiveState recipe has nothing to do with set partitions. Instead it returns all ways of breaking a sequence into a sequence of subsequences whose sum (catenation) is the original sequence. If there are N elements in the original sequence, there are 2**(N-1) ways to do that. That's why there are 32 results in the recipe's example output (2**(len("abcdef")-1) == 32). There would be 203 "set partitions" of a 6-element set. From mertz at gnosis.cx Sun Nov 1 20:58:42 2015 From: mertz at gnosis.cx (David Mertz) Date: Sun, 1 Nov 2015 17:58:42 -0800 Subject: [Python-ideas] Include partitioning in itertools In-Reply-To: References: Message-ID: Oh, yeah... I see what you are asking for isn't quite the same thing as power set. But still, it's not something you can do in finite time with infinite iterators (nor in tractable time with *large* iterators). So `itertools` isn't where it belongs. On Sun, Nov 1, 2015 at 5:49 PM, David Mertz wrote: > This doesn't make sense to me for `itertools`. There is no way to > "partition" an infinite sequence in the style of the recipe you show (nor > any sensible one I can think of). I think a better name would be > `powerset()` anyway; but in either case, it's not an `itertools` type of > operation. > > I think what might be more useful is to make a mix-in class that gave > concrete collections a `.powerset()` method that preserved the types of the > original collection. I think you can even do this as an iterable without > having to concretize the whole set of subsets (but not from an iterable, > from a concrete collection, as input). > > On Sun, Nov 1, 2015 at 3:08 AM, Albert ten Oever > wrote: > >> Hi all, >> >> I really hope this isn't proposed before - I couldn't find anything in >> the archives. >> >> I want to propose to include a 'partition' (mathematically more correct: >> a set partition) function in itertools. To my knowledge, partitioning of a >> set (iterable) is quite a common thing to do and a logic extension of the >> combinatoric generators in itertools. >> >> It is implemented as a Python recipe ( >> http://code.activestate.com/recipes/576795-partitioning-a-sequence/). I >> found that implementing a partition generator of an iterable isn't very >> straightforward, which, in my opinion, strengthens the case for >> implementing it as a separate function. >> >> Definitions of partitions: http://mathworld.wolfram.com/SetPartition.html >> or https://en.wikipedia.org/wiki/Partition_of_a_set >> >> Humble regards, >> >> Albert. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brenbarn at brenbarn.net Sun Nov 1 21:37:15 2015 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Sun, 01 Nov 2015 18:37:15 -0800 Subject: [Python-ideas] Include partitioning in itertools In-Reply-To: References: Message-ID: <5636CC5B.7020806@brenbarn.net> On 2015-11-01 17:58, David Mertz wrote: > Oh, yeah... I see what you are asking for isn't quite the same thing as > power set. But still, it's not something you can do in finite time with > infinite iterators (nor in tractable time with *large* iterators). So > `itertools` isn't where it belongs. I don't think that reasoning is sound. Itertools already includes several combinatoric generators like permutations and combinations, which aren't computable for infinite iterators and which take intractable time for large finite iterators. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From mertz at gnosis.cx Sun Nov 1 22:42:42 2015 From: mertz at gnosis.cx (David Mertz) Date: Sun, 1 Nov 2015 19:42:42 -0800 Subject: [Python-ideas] Include partitioning in itertools In-Reply-To: <5636CC5B.7020806@brenbarn.net> References: <5636CC5B.7020806@brenbarn.net> Message-ID: I'm surprised to have just noticed that `itertools` is oddly broken this way (under Python 3.4.3): This is perfectly happy: >>> def fibs(): a, b = 1, 1 while True: yield b b, a = a+b, b ... >>> f = ((a,b) for a in fibs() for b in fibs()) >>> next(f) Out[3]: (1, 1) >>> next(f) Out[4]: (1, 2) >>> next(f) Out[5]: (1, 3) And yet this freezes be greedily trying to go through the infinite iterator: >>> from itertools import product >>> product(fibs(), fibs()) On Sun, Nov 1, 2015 at 6:37 PM, Brendan Barnwell wrote: > On 2015-11-01 17:58, David Mertz wrote: > >> Oh, yeah... I see what you are asking for isn't quite the same thing as >> power set. But still, it's not something you can do in finite time with >> infinite iterators (nor in tractable time with *large* iterators). So >> `itertools` isn't where it belongs. >> > > I don't think that reasoning is sound. Itertools already includes > several combinatoric generators like permutations and combinations, which > aren't computable for infinite iterators and which take intractable time > for large finite iterators. > > -- > Brendan Barnwell > "Do not follow where the path may lead. Go, instead, where there is no > path, and leave a trail." > --author unknown > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sun Nov 1 18:22:39 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 02 Nov 2015 12:22:39 +1300 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <20151101060629.GE10946@ando.pearwood.info> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> Message-ID: <56369EBF.2000806@canterbury.ac.nz> Steven D'Aprano wrote: > (1) Beginning with Python 3.6, the default is that the current directory > is put at the end of sys.path rather than the beginning. Instead of: > > Scripts which deliberately or unintentionally shadow installed packages > will break from this change. I don't have a problem with this. You can't > fix harmful behaviour without breaking code that depends on that harmful > behaviour. It will also break scripts that rely on looking at the first element of sys.path to find the directory they are running from. Such code is NOT relying on harmful behaviour. -- Greg From rosuav at gmail.com Sun Nov 1 23:43:43 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 Nov 2015 15:43:43 +1100 Subject: [Python-ideas] Include partitioning in itertools In-Reply-To: References: <5636CC5B.7020806@brenbarn.net> Message-ID: On Mon, Nov 2, 2015 at 2:42 PM, David Mertz wrote: > I'm surprised to have just noticed that `itertools` is oddly broken this way > (under Python 3.4.3): > > This is perfectly happy: > >>>> def fibs(): > > a, b = 1, 1 > while True: > yield b > b, a = a+b, b > ... >>>> f = ((a,b) for a in fibs() for b in fibs()) >>>> next(f) > Out[3]: (1, 1) >>>> next(f) > Out[4]: (1, 2) >>>> next(f) > Out[5]: (1, 3) > > And yet this freezes be greedily trying to go through the infinite iterator: > >>>> from itertools import product >>>> product(fibs(), fibs()) Yeah. The problem is that you're looking at a simplified version of the reference implementation. Further down it says that it's equivalent to a rather longer function, which (critically) includes a call to tuple(), to coalesce the iterables into their values. When you use the naive genexp, what you're actually doing is constructing a new instance of b's fibs() for every iteration of a's one (you won't see that on an infinite iterator, but try a finite iterator with some print() calls and you'll see); itertools.product obviously can't do that, as it's given the iterable and not the means of constructing more. The tiny example up the top is inaccurate in its handling of edge cases, but for stable, repeatable iterables, it will return the same results. ChrisA From tritium-list at sdamon.com Mon Nov 2 01:54:48 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Mon, 02 Nov 2015 01:54:48 -0500 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <201511011353.tA1DrNkI019815@fido.openend.se> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> <563612E2.1060201@sdamon.com> <201511011353.tA1DrNkI019815@fido.openend.se> Message-ID: <563708B8.8070808@sdamon.com> On 11/1/2015 08:53, Laura Creighton wrote: > In a message of Sun, 01 Nov 2015 08:25:54 -0500, Alexander Walters writes: >> Honestly, shadowing modules is something that should be solved by >> renaming modules. If you are worrying about shadowing ONLY the standard >> library - guess what? those names don't change often, and are well >> known. Don't use those names. > The problem is that the sort of people who make these errors don't > know the names. Knowing that 'turtle.py' is a bad name for 'my > very first program that does turtle graphics' is a more advanced > skill than the turtle program writers, or even their teachers, can > be expected to have. This is what is called a teachable moment. It is BETTER that python does nothing to prevent or warn the user. Let them fail, and fail hard. Its better for them to fail. I see any argument against this as pure detriment to the user. Wanting these warnings is wanting to disadvantage the new user - debugging shadowed names is a skill you need in your toolbox, and this deprives people of that vital early lesson. From toddrjen at gmail.com Mon Nov 2 03:19:41 2015 From: toddrjen at gmail.com (Todd) Date: Mon, 2 Nov 2015 09:19:41 +0100 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <56361771.9090702@lucidity.plus.com> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <201511010753.tA17rHs3025816@fido.openend.se> <201511010808.tA188a77027105@fido.openend.se> <201511010949.tA19nnwj001815@fido.openend.se> <56361771.9090702@lucidity.plus.com> Message-ID: On Nov 1, 2015 14:45, "Erik" wrote: > > On 01/11/15 09:49, Laura Creighton wrote: >> >> Adding warnings to IDLE when you save a file is a fine idea, and will >> help IDLE users avoid pain. I want to help _everybody_ with a change >> to Python, so that it issues a warning when you shadow something in >> the standard library. > > > I think that a solution to this could be something along the lines of having a command line switch (let's say '-w') or other way of enabling a "shadow module checking" mechanism. > > When an exception is thrown that is caught by the default handler and output to stderr, an additional line of output could be appended: > > "Note: This problem could be caused by a module that shadows a standard library or third party module. Run Python with the '-w' switch for more detail." > > I'm not sure if this would be suitable for _all_ exceptions or only those from certain points in the exception hierarchy. > > When the '-w' switch is enabled, the default exception handler would instead enumerate all active modules and, for any which has another module with the same name further down the search path than the one that is loaded, would output some sort of diagnostic message (i.e., a list of which loaded modules are shadowing _something_else_ that would have been loaded otherwise) - similar to your example below: > > >> Something like: >> Warning: local file /u/lac/junk/string.py shadows module named string in the >> Standard Library > > > This would prevent "power users" from always having to see a potentially large diagnostic after each uncaught exception, but generally remind everyone that such a thing is possible and "here's the easy way to check for it". > > A bit like Valgrind's memcheck tool's "--leak-check=full" option - when run without it, if there are leaks then memcheck reminds the user that that's the way to start digging down into what might be causing them. I was thinking of something like this too. A list of stdlib modules could be created statically when python is originally built (if it isn't already). This would have no changes in behaviour unless an error actually happens. And maybe to avoid any performance cost the message can only be inserted when the interpreter actually exits, if that makes any sense at all. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Nov 2 06:14:12 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 2 Nov 2015 06:14:12 -0500 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <56369EBF.2000806@canterbury.ac.nz> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <56369EBF.2000806@canterbury.ac.nz> Message-ID: On 11/1/2015 6:22 PM, Greg Ewing wrote: > It will also break scripts that rely on looking at the > first element of sys.path to find the directory they > are running from. '' is not very informative ;-). I suspect the full path added in the past, but it is not now. import os; os.getcwd() works better -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Mon Nov 2 19:02:56 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 03 Nov 2015 13:02:56 +1300 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <56369EBF.2000806@canterbury.ac.nz> Message-ID: <5637F9B0.3010201@canterbury.ac.nz> Terry Reedy wrote: > '' is not very informative ;-). > I suspect the full path added in the past, but it is not now. > import os; os.getcwd() works better You misunderstand. The point is *not* to find the cwd, it's to find the directory containing the main script, so you can load resources related to it. There are probably other and possibly better ways to go about that, but it works and is sometimes used. The proposed change would break it. -- Greg From python at mrabarnett.plus.com Mon Nov 2 20:22:29 2015 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 3 Nov 2015 01:22:29 +0000 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: <5637F9B0.3010201@canterbury.ac.nz> References: <201510291559.t9TFxi4i003479@fido.openend.se> <20151101060629.GE10946@ando.pearwood.info> <56369EBF.2000806@canterbury.ac.nz> <5637F9B0.3010201@canterbury.ac.nz> Message-ID: <56380C55.4020405@mrabarnett.plus.com> On 2015-11-03 00:02, Greg Ewing wrote: > Terry Reedy wrote: >> '' is not very informative ;-). >> I suspect the full path added in the past, but it is not now. >> import os; os.getcwd() works better > > You misunderstand. The point is *not* to find the > cwd, it's to find the directory containing the main > script, so you can load resources related to it. > > There are probably other and possibly better ways > to go about that, but it works and is sometimes > used. The proposed change would break it. > I use os.path.dirname(__file__) in the main script. From agtoever at hotmail.com Tue Nov 3 10:46:41 2015 From: agtoever at hotmail.com (Albert ten Oever) Date: Tue, 3 Nov 2015 15:46:41 +0000 Subject: [Python-ideas] Include partitioning in itertools In-Reply-To: References: Message-ID: > [Sven R. Kunze] > > ... > > Not quite sure if I understand the implementation. Reading this > > http://mathworld.wolfram.com/BellNumber.html for the given example of 6, it > > should have 203 possible partitions. What am I missing? > > [Tim Peters] > The ActiveState recipe has nothing to do with set partitions. Instead > it returns all ways of breaking a sequence into a sequence of > subsequences whose sum (catenation) is the original sequence. If > there are N elements in the original sequence, there are 2**(N-1) ways > to do that. That's why there are 32 results in the recipe's example > output (2**(len("abcdef")-1) == 32). There would be 203 "set > partitions" of a 6-element set. @Tim: you are completely right. Sorry for the confusion referencing this implementation. Forget it. > On Sun, Nov 1, 2015 at 6:37 PM, Brendan Barnwell wrote: > > On 2015-11-01 17:58, David Mertz wrote: > >> Oh, yeah... I see what you are asking for isn't quite the same thing as > >> power set. But still, it's not something you can do in finite time with > >> infinite iterators (nor in tractable time with *large* iterators). So > >> `itertools` isn't where it belongs. > >> > > > > I don't think that reasoning is sound. Itertools already includes > > several combinatoric generators like permutations and combinations, which > > aren't computable for infinite iterators and which take intractable time > > for large finite iterators. > > @Brendan: exactly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Tue Nov 3 11:58:31 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 3 Nov 2015 17:58:31 +0100 Subject: [Python-ideas] Include partitioning in itertools In-Reply-To: <56365A31.6020803@wh2.tu-dresden.de> References: <5636486C.6080401@mail.de> <56365A31.6020803@wh2.tu-dresden.de> Message-ID: <5638E7B7.2060608@mail.de> Hey Lukas, I agree both version (+ the n-part-partition-version of it) might be very useful to have. But where do they go into? itertools might be the right place for the iter_partition; but where to with the set_partition? Best, Sven On 01.11.2015 19:30, Lukas Juhrich wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Hi Sven, > > The issue here is that such an implementation does not allow ?gaps? > and implicitly enforces that the elements of such a partition only > contain elements directly following each other, thus making it not a > complete implementation of producing a set partition. To give a > specific counterexample, `['acd', 'bef']` will never be produced. > > The question to ask is whether you want to implement exact > mathematical set partition, or if you implement something with the > above constraint, and call it something different like ?iterable > partition?. If choosing the latter, however, it should be made clear > by the function name and in the documentation that this is *not* > comparable to set partition. > > Perhaps it will be useful to implement both concepts, since such an > ?iterable partition? seems to be easier to implement and will perhaps > be faster (which, however, is a blind guess and depends on the > implementation of producing a set partition). > > - -- > Sincerely, > Lukas > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2 > > iQIcBAEBCAAGBQJWNlotAAoJENBP4KTXGFXeSJ8P/33IfUqVe+LqKjdjghOVrb2A > 2wcj85GJHV5hu8UIYNfchVwvBML7w0LpgvHsopD7NkXLYepOy1AuSDTlePYZGu2g > TvVFLd861HZVAQFtFwuuY1bAkNlCDDtNOUZYrortkc7F05TFBaGiTu7Z7zrHG9PN > 5eLZ7dpr5y9Ok8nCaerUVnxAn81+A+mr0O2KPAAOfTLA0C0bo4uWJsp1WJMcLF1N > o0RpzrZ68Qy1vzfnRaf5T/MpcxIFaPxTYee8wxg/pGse5gPJidvK83G5MV/ylHbK > INqG7iPTcD8DJ0OpvR/462QaxBf225fP3Dg+xUtKvpBw2SMrSfc8X9VXOE7EFMJ+ > m9Wajx5d1+wVvlTzdEa5Nu5A7I5DGDO+zW7jeuZ2Ri6q6aeKHRt1JLsi/5VIJ+/B > 0xb7/a5JKcjskrZuQrlCzVKlvOUq2Wf1SZKlPLq6jsB3xkT1qJhF2QssoR3s2aNn > 998gwRl3sJzrdVh9+QxCXjoQ92aqQCLRsvHll+uZ9bjJEhD3SXlyFhqmhLf396fq > BMWpxFrSuf4ltqke2WQrzrvpJnxd1MH7yH+2tVQej8/kUHJyCWdBLZeQJeFljiJa > 3zYh6BQ8KjwtjNcSHzmhANO/0FSwzVDPSvP2ocEbjYjv5WMXXk0/sNdRgsb3gkQ1 > 5zuBS6j2n5n9SeUZ0IcM > =Ki11 > -----END PGP SIGNATURE----- > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From steve at pearwood.info Tue Nov 3 19:15:54 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 4 Nov 2015 11:15:54 +1100 Subject: [Python-ideas] Include partitioning in itertools In-Reply-To: <5638E7B7.2060608@mail.de> References: <5636486C.6080401@mail.de> <56365A31.6020803@wh2.tu-dresden.de> <5638E7B7.2060608@mail.de> Message-ID: <20151104001553.GM10946@ando.pearwood.info> On Tue, Nov 03, 2015 at 05:58:31PM +0100, Sven R. Kunze wrote: > Hey Lukas, > > I agree both version (+ the n-part-partition-version of it) might be > very useful to have. > > But where do they go into? > > itertools might be the right place for the iter_partition; but where to > with the set_partition? I think the right place for both of them is your own personal Python toolbox. -- Steve From abarnert at yahoo.com Tue Nov 3 20:06:14 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 3 Nov 2015 17:06:14 -0800 Subject: [Python-ideas] Include partitioning in itertools In-Reply-To: <5638E7B7.2060608@mail.de> References: <5636486C.6080401@mail.de> <56365A31.6020803@wh2.tu-dresden.de> <5638E7B7.2060608@mail.de> Message-ID: On Nov 3, 2015, at 08:58, Sven R. Kunze wrote: > > Hey Lukas, > > I agree both version (+ the n-part-partition-version of it) might be very useful to have. > > But where do they go into? > > itertools might be the right place for the iter_partition; but where to with the set_partition? One more consideration: if you look at what's in itertools today, it isn't really everything you might want to do with and/or to produce iterators. Instead, it's (mostly) a collection of inner-loop building blocks that are accelerated with C code, and can be assembled together in pure Python to do almost everything you might want. The recipes give extensive examples of such things that are useful but don't need to be in the module itself. So, the question isn't just whether iter_partitions is like permutations in what it does, but whether it's like permutations in not having any more minimal building blocks?if there are such building blocks, it might be better to add those to itertools, and iter_partitions to the recipes. Meanwhile, if you think there's a wider need for these functions than can be served by a blog post or ActiveState recipe, surely they should be on PyPI (whether or not they're in the stdlib?after all, if they're that useful, people will want them in 3.5 and 2.7). That also gives you a way to prove their usefulness?look at all these open source projects that use the library, and these comments from users saying "this really should be in the stdlib", and so on. At least for iter_partition, you might want to consider submitting it as an addition to more-itertools instead of posting your own library. He's already got a bunch of other stuff that's itertools-ish that may not belong in the stdlib (including all the recipes from the stdlib docs) but may still be useful enough for people to install the package. From sjoerdjob at sjec.nl Wed Nov 4 17:02:41 2015 From: sjoerdjob at sjec.nl (sjoerdjob at sjec.nl) Date: Wed, 4 Nov 2015 23:02:41 +0100 Subject: [Python-ideas] Use `isinstance` check during exception handling Message-ID: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> Hi all, TL;DR: Change exception checking logic from err.__class__ in exc.__mro__ to isinstance(err, exc) Because it is more powerful, more consistent and allows for cleaner code. The reason I am proposing this change is that quite often, one has to deal with libraries behaving 'badly' with regard to exception granularity. I will take the Python base library before PEP3151 [1] as an example. It is not hard to imagine there are a lot of other cases of libraries that behave with little granularity in the exception classes. try: some(operation) except OSError as err: if err.errno == errno.ENOENT: # handle file-not-exists elif err.errno == errno.EACCES: # handle permission stuff. else: raise After PEP3151 one can now write: try: some(operation) except FileNotFoundError: # handle file-not-exists except PermissionError: # handle permission stuff. Now, PEP3151 took over a year to be resolved, judging by the dates. If we have to go through changes like this more often, one will have to wait quite a while. Also, consider backwards compatibility issues. Code using FileNotFoundError will only run on Python3.3+. Thus I propose to have the exception handling code use the normal `isinstance` machinery, as it is more powerful, and allows one to use virtual base classes to check if the instance of the exception is relevant to catch. An example: import errno # Fake `FileNotFoundError` in Python3.2 class ENOENTCheck(type): def __instancecheck__(cls, inst): return isinstance(inst, EnvironmentError) and \ inst.errno == errno.ENOENT class FileNotFoundError(EnvironmentError, metaclass=ENOENTCheck): pass try: open("no-such-file") except Exception as e: print(isinstance(e, FileNotFoundError)) This would print `True`. But, using `FileNotFoundError` as a class to catch will not work, because the `__mro__` is checked instead. I proposed the change on the Python issue tracker [2] already, but it was (rightfully, I think) decided that it needs some more discussion because it will bring a (potentially huge) semantic change. As an opening, here are some pros and cons I can think of: Cons: - *Code already written might break*: Highly unlikely. If you have written code that overrides `isinstance` logic, you'd more than likely not write code that depends on there being a difference in matching logic between the exception handling and normal `isinstance` calls. - *Performance will go down*: Yes, instance-checking is probably a lot more expensive then subclass-checking. I haven't made the change and benchmarked it yet, but based on Martin Panter's comment [3], it might only cost a lot when actually checking against a virtual base class. If cost is limited to those cases, I think the cost is negligible. - *Exception handling like this is just magic*: Yes, anything regarding the metaclasses might be considered as magic. But metaclasses also allow you to build great abstractions, like for instance the ORM included with Django. I think this is one of the cases where the benefits outweigh the cost. Pros: - *Cleaner code*: The same benefit PEP 3151 brought for the base library and OS errors can be extended to all the Python libraries that lack granularity. - *Consistency*: Who would ever expect that you can not always catch an exception `foo` using class `Bar` when `isinstance(foo, Bar)`. Even though I can currently think of more downsides than upsides, I do think that the downsides are really minor, while the upsides are huge: clean code and consistency. Also, I set up a proof-of-concept PEP3151 compatibility layer for Python2.7 that sort-of fakes this feature. It fakes it by checking the instance `sys.exc_info()[1]` in the `__subclasscheck__`. For the proof of concept see [4]. I only found out during porting to Python3 that it does not actually work, because of the direct checking of the `__mro__`, but that's hopefully going to be fixed soon-ish [5]. I'd be interested to hear your thoughts about this. Any more pros and cons would be welcome, as well as your judgement as to whether the breaking of backwards compatibility is potentially a bad idea. Kind regards, Sjoerd Job Postmus [1] PEP 3151 -- Reworking the OS and IO exception hierarchy https://www.python.org/dev/peps/pep-3151/ [2] Call `isinstance` instead of `issubclass` during exception handling http://bugs.python.org/issue25537 [3] http://bugs.python.org/issue25537#msg253963 [4] https://github.com/sjoerdjob/exhacktion/ [5] Catching virtual subclasses in except clauses http://bugs.python.org/issue12029 From tjreedy at udel.edu Wed Nov 4 20:39:40 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 4 Nov 2015 20:39:40 -0500 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> Message-ID: On 11/4/2015 5:02 PM, sjoerdjob at sjec.nl wrote: > TL;DR: Change exception checking logic from > err.__class__ in exc.__mro__ This is backwards. err.__class__ must be a (true) subclass, not a superclass, of exc, where err = exception instance, exc = exception class. There are two checks in the exception process. First, for 'raise err', is that err is a instance of StopIteration, which is to say, err.__class__ is a subclass of StopIteration. To avoid the fakery of __instancecheck__ and __subclasscheck__, I presume this is implemented as something like one of StopIteration in err.__class__.__mro__ StopIteration in type(err).__mro__ (There was a recent pydev discussion about when and why these may diverge.) The second, for 'except exc', is that exc is a superclass (other than object) of type(err). I presume this is implemented as something like exc in type(err)[:-1] # exclude object class This assume that err has already passed the first check above. The `try` doc says "An object is compatible with an exception if it is the class or a base class of the exception object". This text fails to exclude the object class as a compatible object. CPython code rejects 'except object:' with "TypeError: catching classes that do not inherit from BaseException is not allowed" > to > isinstance(err, exc) > Because it is more powerful, ?? This predicate is True more often than the current one because it a) does not exclude exc = object >>> isinstance(Exception(), object) True and b) accesses exc.__instancecheck__ (which is part of your point). This makes it a weaker predicate. I don't have strong opinions at the moment about the proposal itself, but semantic changes in core python syntax need a PEP (once you think there is *some* support). -- Terry Jan Reedy From mike at selik.org Wed Nov 4 21:42:58 2015 From: mike at selik.org (Michael Selik) Date: Thu, 05 Nov 2015 02:42:58 +0000 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> Message-ID: Hi Sjoerd, I'm not clear on what problem this suggestion is solving. PEP 3151 was solving the problem of existing code failing to be specific with IOError and OSError error numbers. Can you provide some less abstract snippets as examples, including surrounding context or a link to a project that has the problem? If I saw `except Exception as e` followed by `isinstance(e, SomeError)` I would be very confused as to why `except SomeError` was not better. Is this simply for backporting FileNotFoundError? Michael On Wed, Nov 4, 2015 at 8:39 PM Terry Reedy wrote: > > > On 11/4/2015 5:02 PM, sjoerdjob at sjec.nl wrote: > > TL;DR: Change exception checking logic from > > err.__class__ in exc.__mro__ > > This is backwards. err.__class__ must be a (true) subclass, not a > superclass, of exc, where err = exception instance, exc = exception class. > > There are two checks in the exception process. First, for 'raise err', > is that err is a instance of StopIteration, which is to say, > err.__class__ is a subclass of StopIteration. To avoid the fakery of > __instancecheck__ and __subclasscheck__, I presume this is implemented > as something like one of > StopIteration in err.__class__.__mro__ > StopIteration in type(err).__mro__ > (There was a recent pydev discussion about when and why these may diverge.) > > The second, for 'except exc', is that exc is a superclass (other than > object) of type(err). I presume this is implemented as something like > exc in type(err)[:-1] # exclude object class > This assume that err has already passed the first check above. > > The `try` doc says "An object is compatible with an exception if it is > the class or a base class of the exception object". This text fails to > exclude the object class as a compatible object. CPython code rejects > 'except object:' with "TypeError: catching classes that do not inherit > from BaseException is not allowed" > > > to > > isinstance(err, exc) > > Because it is more powerful, > > ?? This predicate is True more often than the current one because it a) > does not exclude exc = object > > >>> isinstance(Exception(), object) > True > > and b) accesses exc.__instancecheck__ (which is part of your point). > This makes it a weaker predicate. > > I don't have strong opinions at the moment about the proposal itself, > but semantic changes in core python syntax need a PEP (once you think > there is *some* support). > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Wed Nov 4 22:01:30 2015 From: mike at selik.org (Michael Selik) Date: Thu, 05 Nov 2015 03:01:30 +0000 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> Message-ID: After reading https://github.com/sjoerdjob/exhacktion/ I have a better understanding of what you want to accomplish. However, I'm still not clear on where the pain is coming from. If you find yourself writing an if in an except many times as you described: try: os.chown("README.md", 100, 100) except EnvironmentError as e: if e.errno == errno.ENOENT: pass else: raise Would using a context manager solve your problem? class IgnoreNotFound: def __enter__(self): return self def __exit__(self, etyp, e, etb): if issubclass(etyp, EnvironmentError) and e.message == errno.ENOENT: return True with IgnoreNotFound: os.chown("README.md", 100, 100) On Wed, Nov 4, 2015 at 9:42 PM Michael Selik wrote: > Hi Sjoerd, > > I'm not clear on what problem this suggestion is solving. PEP 3151 was > solving the problem of existing code failing to be specific with IOError > and OSError error numbers. Can you provide some less abstract snippets as > examples, including surrounding context or a link to a project that has the > problem? > > If I saw `except Exception as e` followed by `isinstance(e, SomeError)` I > would be very confused as to why `except SomeError` was not better. Is this > simply for backporting FileNotFoundError? > > Michael > > On Wed, Nov 4, 2015 at 8:39 PM Terry Reedy wrote: > >> >> >> On 11/4/2015 5:02 PM, sjoerdjob at sjec.nl wrote: >> > TL;DR: Change exception checking logic from >> > err.__class__ in exc.__mro__ >> >> This is backwards. err.__class__ must be a (true) subclass, not a >> superclass, of exc, where err = exception instance, exc = exception class. >> >> There are two checks in the exception process. First, for 'raise err', >> is that err is a instance of StopIteration, which is to say, >> err.__class__ is a subclass of StopIteration. To avoid the fakery of >> __instancecheck__ and __subclasscheck__, I presume this is implemented >> as something like one of >> StopIteration in err.__class__.__mro__ >> StopIteration in type(err).__mro__ >> (There was a recent pydev discussion about when and why these may >> diverge.) >> >> The second, for 'except exc', is that exc is a superclass (other than >> object) of type(err). I presume this is implemented as something like >> exc in type(err)[:-1] # exclude object class >> This assume that err has already passed the first check above. >> >> The `try` doc says "An object is compatible with an exception if it is >> the class or a base class of the exception object". This text fails to >> exclude the object class as a compatible object. CPython code rejects >> 'except object:' with "TypeError: catching classes that do not inherit >> from BaseException is not allowed" >> >> > to >> > isinstance(err, exc) >> > Because it is more powerful, >> >> ?? This predicate is True more often than the current one because it a) >> does not exclude exc = object >> >> >>> isinstance(Exception(), object) >> True >> >> and b) accesses exc.__instancecheck__ (which is part of your point). >> This makes it a weaker predicate. >> >> I don't have strong opinions at the moment about the proposal itself, >> but semantic changes in core python syntax need a PEP (once you think >> there is *some* support). >> >> -- >> Terry Jan Reedy >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Wed Nov 4 22:35:56 2015 From: mike at selik.org (Michael Selik) Date: Thu, 05 Nov 2015 03:35:56 +0000 Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen Message-ID: Placing current working directory at the beginning of sys.path or the end of sys.path is a tradeoff between two debugging tasks given the error message, `AttributeError: 'module' object has no attribute 'blah'`. The traceback tells you the name of the module. Which would you prefer: (1) needing to notice that you made a module of the same name, or (2) needing to notice that somewhere in the cryptic locations on sys.path there's a module of the same name? I'll pick option 1. Sure, it's tricky the first time it happens, but I've seen complete beginners figure it out after a few minutes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Wed Nov 4 22:45:01 2015 From: mike at selik.org (Michael Selik) Date: Thu, 05 Nov 2015 03:45:01 +0000 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence Message-ID: On 02.10.2015 05:49, Steven D'Aprano wrote: > At most, some have suggested that we don't have a good > word for those iterables which are not iterators. Isn't that a "sequence"? The collections module defines a Sequence as having a __getitem__ and __len__. Is a "non-iterator iterable" something different? -------------- next part -------------- An HTML attachment was scrubbed... URL: From sjoerdjob at sjec.nl Thu Nov 5 01:00:12 2015 From: sjoerdjob at sjec.nl (sjoerdjob at sjec.nl) Date: Thu, 5 Nov 2015 07:00:12 +0100 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> Message-ID: <744456a7ee014b3774f4bccbf673cacd.squirrel@mail.sjec.nl> > On 11/4/2015 5:02 PM, sjoerdjob at sjec.nl wrote: >> TL;DR: Change exception checking logic from >> err.__class__ in exc.__mro__ > > This is backwards. err.__class__ must be a (true) subclass, not a > superclass, of exc, where err = exception instance, exc = exception class. Ah, a minor glitch when writing the e-mail. My apologies. It should be exc in err.__class__.__mro__ > There are two checks in the exception process. First, for 'raise err', > is that err is a instance of StopIteration, which is to say, > err.__class__ is a subclass of StopIteration. To avoid the fakery of > __instancecheck__ and __subclasscheck__, I presume this is implemented > as something like one of > StopIteration in err.__class__.__mro__ > StopIteration in type(err).__mro__ > (There was a recent pydev discussion about when and why these may > diverge.) I can't actually find the logic which handles the `StopIteration`, unless you are talking specifically about the exception handling in for-loops and everything else that relates to iterables. (The code I'm looking at is Python/errors.c:PyErr_GivenExceptionMatches). > The second, for 'except exc', is that exc is a superclass (other than > object) of type(err). I presume this is implemented as something like > exc in type(err)[:-1] # exclude object class > This assume that err has already passed the first check above. >From what I can find (again: errors.c), the current logic is: (note: `err` is what is raised, `exc` is what's matched against) * if `exc` is a tuple, recurse over the elements of the tuple. * if the `err` is an instance, replace it with its class * if both `err` and `exc` are subclasses of exception: check PyType_IsSubtype(err, exc) which basically is the same as (see Objects/typeobject.c) exc in err.__mro__ note: not err.__mro__[:-1]. > The `try` doc says "An object is compatible with an exception if it is > the class or a base class of the exception object". This text fails to > exclude the object class as a compatible object. CPython code rejects > 'except object:' with "TypeError: catching classes that do not inherit > from BaseException is not allowed" > >> to >> isinstance(err, exc) >> Because it is more powerful, > > ?? This predicate is True more often than the current one because it a) > does not exclude exc = object My idea was basically to still demand that both `err` and `exc` must sub-class `BaseException`, but that only the check for `PyType_IsSubtype` would be replaced with `PyObject_IsInstance`. > >>> isinstance(Exception(), object) > True > > and b) accesses exc.__instancecheck__ (which is part of your point). > This makes it a weaker predicate. Yes, the predicate is weaker. And that gives us more power. > I don't have strong opinions at the moment about the proposal itself, > but semantic changes in core python syntax need a PEP (once you think > there is *some* support). It's not a syntax-change, but a semantics-change. Still I would not be surprised if it indeed does need a PEP, but there will only be a point in doing that when there is *some* support. Before writing that PEP, I was checking here to see if I could find some support, or if it the pros don't outweigh the cons for the 'masses'. > -- > Terry Jan Reedy Regards, Sjoerd Job Postmus From storchaka at gmail.com Thu Nov 5 03:05:55 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 5 Nov 2015 10:05:55 +0200 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> Message-ID: On 05.11.15 00:02, sjoerdjob at sjec.nl wrote: > Now, PEP3151 took over a year to be resolved, judging by the dates. If > we have to go through changes like this more often, one will have to > wait quite a while. Also, consider backwards compatibility issues. Code > using FileNotFoundError will only run on Python3.3+. > > Thus I propose to have the exception handling code use the normal > `isinstance` machinery, as it is more powerful, and allows one to use > virtual base classes to check if the instance of the exception is > relevant to catch. An example: > > import errno > > # Fake `FileNotFoundError` in Python3.2 Do you propose to add a new feature in 3.2? From sjoerdjob at sjec.nl Thu Nov 5 03:13:49 2015 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Thu, 5 Nov 2015 09:13:49 +0100 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> Message-ID: <7D22F391-69AF-46C1-9FCD-DFE3D1B01D13@sjec.nl> > On 5 Nov 2015, at 09:05, Serhiy Storchaka wrote: > >> On 05.11.15 00:02, sjoerdjob at sjec.nl wrote: >> Now, PEP3151 took over a year to be resolved, judging by the dates. If >> we have to go through changes like this more often, one will have to >> wait quite a while. Also, consider backwards compatibility issues. Code >> using FileNotFoundError will only run on Python3.3+. >> >> Thus I propose to have the exception handling code use the normal >> `isinstance` machinery, as it is more powerful, and allows one to use >> virtual base classes to check if the instance of the exception is >> relevant to catch. An example: >> >> import errno >> >> # Fake `FileNotFoundError` in Python3.2 > > Do you propose to add a new feature in 3.2? Definitely not! PEP3151 was just meant as an example of an improvement of exception granularity taking a long time. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From mike at selik.org Thu Nov 5 03:18:12 2015 From: mike at selik.org (Michael Selik) Date: Thu, 05 Nov 2015 08:18:12 +0000 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: <7D22F391-69AF-46C1-9FCD-DFE3D1B01D13@sjec.nl> References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> <7D22F391-69AF-46C1-9FCD-DFE3D1B01D13@sjec.nl> Message-ID: I think with good reason. It's impossible to enumerate all possible ways a program can error. Eventually one must fall back to inspecting various error payload information and perhaps even other state. Increasing error granularity is a slippery slope. The shift should be driven by evidence of annoying boilerplate code written in the standard library and major community projects. On Thu, Nov 5, 2015 at 3:13 AM Sjoerd Job Postmus wrote: > Definitely not! PEP3151 was just meant as an example of an improvement of > exception granularity taking a long time. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Nov 5 03:21:17 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 Nov 2015 19:21:17 +1100 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> Message-ID: On Thu, Nov 5, 2015 at 7:05 PM, Serhiy Storchaka wrote: > On 05.11.15 00:02, sjoerdjob at sjec.nl wrote: >> >> Now, PEP3151 took over a year to be resolved, judging by the dates. If >> we have to go through changes like this more often, one will have to >> wait quite a while. Also, consider backwards compatibility issues. Code >> using FileNotFoundError will only run on Python3.3+. >> >> Thus I propose to have the exception handling code use the normal >> `isinstance` machinery, as it is more powerful, and allows one to use >> virtual base classes to check if the instance of the exception is >> relevant to catch. An example: >> >> import errno >> >> # Fake `FileNotFoundError` in Python3.2 > > > Do you propose to add a new feature in 3.2? I think the point is to future-proof Python against further such incidents, and to allow some measure of control even over third-party libraries. For instance, psycopg2 dumps ProgrammingError on you for lots of all SQL errors, but has a few special cases like IntegrityError; if there's some special case that isn't handled by psycopg2 itself, you have no recourse but to catch something too broad and then manually filter. I do like the idea of being able to refactor exception specialization code: try: ... except BroadError as e: if is_narrower_error(e): # handle NarrowerError else: raise but I don't know whether it's worth complicating the core exception handling system with such a rare case. Also, what happens if you have a bug in your __instancecheck__ method? It's going to be *extremely* hard to track down. How often does this sort of except-level specialization come up? ChrisA From abarnert at yahoo.com Thu Nov 5 03:43:43 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 5 Nov 2015 00:43:43 -0800 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> Message-ID: On Nov 4, 2015, at 14:02, sjoerdjob at sjec.nl wrote: > > Hi all, > > TL;DR: Change exception checking logic from > err.__class__ in exc.__mro__ > to > isinstance(err, exc) > Because it is more powerful, more consistent and allows for cleaner > code. Correct me if I'm wrong, but doesn't Python allow you to raise a class and handle it without ever creating an instance? If an instance is needed (e.g., because the exception is caught by a handler with an as clause), one is created by calling exc(), but if it's not needed, that doesn't happen. Assuming I'm right, your proposal would mean the instance is _always_ needed, so if you raise a type, exc() is always called. So, for example: class MyException(Exception): def __init__(self, thingy, *a, **kw): super().__init__(*a, **kw) self.thingy = thingy try: raise MyException except MyException: print('whoops') Instead of printing "whoops", it would now (I think) abort with a TypeError about trying to call MyException with the wrong number of arguments. Plus, there's almost certainly code where someone skips creating the instance as an optimization?probably it's usually a useless premature one, but maybe someone has tested and it makes a difference in a real program. One obvious possibility is: if exc is itself a subclass of BaseException, you use the existing subclass check; if not, instead of using the subclass check on its type, you use the new instance check. But you'd have to think that through to make sure it makes sense. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Nov 5 04:17:24 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 5 Nov 2015 01:17:24 -0800 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: References: Message-ID: <510DA2AB-4BB6-4B36-9ADD-2FC48B9CE1EC@yahoo.com> On Nov 4, 2015, at 19:45, Michael Selik wrote: > > On 02.10.2015 05:49, Steven D'Aprano wrote: > > At most, some have suggested that we don't have a good > > word for those iterables which are not iterators. > > Isn't that a "sequence"? The collections module defines a Sequence as having a __getitem__ and __len__. Is a "non-iterator iterable" something different? A mapping isn't a sequence (even though it has __getitem__ and __len__, and also has the same supertypes, which add additional restrictions that you forgot like __contains__). Neither is a set, or any of the kinds of mapping views, or any intentionally-not-quite-sequence (like range before 3.2, or, IIRC, blist.sortedlist), or an iterable that, each time it's iterated, gives you 20 new values in [0, 1) that add up to 1. But they're all non-iterator iterables. In fact, there are a number of concepts that largely overlap, but don't completely, including: * Iterables that aren't iterators * Iterables that don't return self from __iter__ * Iterables that always return a new iterator * Iterables that can repeatably produce a "correct" sequence of values (as long as they aren't mutated in between iterations) * Iterables that can repeatably produce equal values (as long as they aren't mutated) * Iterables that can repeatedly produce identical values (as long as they aren't mutated) There are also variations of the last three that represent some data that can be "externally mutated" (e.g., a list of all broadcasting SMB servers on the local subnet will always produce equal values unless a server goes up or down between iterations?or, more simply, any view can change values if the thing it's a view of is mutated, without the view itself being mutated). Anyway, all of the above are true for sequences, mappings, sets, mapping views, and most not-quite-sequences, but not for the random example, or the SMB example. I don't think we need official names for all of these things, just one?as long as it's clear which one, and exactly what it means, I think that will cover well over 90% of all reasonable uses. I believe that earlier in the thread, Terry suggested "collection" to mean the last definition, which seems fine at first glance, so let's assume that. If we added that to the documentation (and fixed all the places that sloppily talk about sequences when they don't really mean sequences, etc.), and ideally added an ABC and a static type to match, then code could very easily explain that it produces or requires a collection. Most code that needs two passes over an iterable can just say it requires a collection. The fact that it might also work with some sort-of-repeatable-but-not-quite-collection types is fine (and we don't have to say that such types are "defective" or anything, as long as the documentation makes it clear that there are some very rare iterables that are neither iterators nor collections, and what that means). Of course Collection defined this way can't be an implicit ABC, because it doesn't have any interface beyond that of Iterator. But then Sequence and Mapping can't be implicit either, because they have the same interface as each other. In fact, that's the main reason we need ABCs in the first place?for cases where structural/duck typing can't cover the intended semantics. From vgr255 at live.ca Thu Nov 5 08:08:07 2015 From: vgr255 at live.ca (Emanuel Barry) Date: Thu, 5 Nov 2015 08:08:07 -0500 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>, Message-ID: I think the main concern here is not "Would that benefit the corner cases?" - it surely would. What we should be worrying is, "Will this impede the common cases?". I think that it should be possible, for all the (very) common cases, to use what CPython already uses internally. If it matches, I don't see why we'd even bother going through the __subclasscheck__ and __instancecheck__ hooks. IMO, it should only use those (whichever ends up being the prefered choice - I have no strong preference for now either way) if the standard exception lookup fails, and possibly including the fact that BaseException doesn't need to be a direct subclass (e.g. virtual subclasses). Just my 2 cents. -Emanuel -------------- next part -------------- An HTML attachment was scrubbed... URL: From sjoerdjob at sjec.nl Thu Nov 5 10:00:34 2015 From: sjoerdjob at sjec.nl (sjoerdjob at sjec.nl) Date: Thu, 5 Nov 2015 16:00:34 +0100 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> Message-ID: > On Nov 4, 2015, at 14:02, sjoerdjob at sjec.nl wrote: >> >> Hi all, >> >> TL;DR: Change exception checking logic from >> err.__class__ in exc.__mro__ >> to >> isinstance(err, exc) >> Because it is more powerful, more consistent and allows for cleaner >> code. > > Correct me if I'm wrong, but doesn't Python allow you to raise a class and > handle it without ever creating an instance? If an instance is needed > (e.g., because the exception is caught by a handler with an as clause), > one is created by calling exc(), but if it's not needed, that doesn't > happen. > > Assuming I'm right, your proposal would mean the instance is _always_ > needed, so if you raise a type, exc() is always called. So, for example: > > class MyException(Exception): > def __init__(self, thingy, *a, **kw): > super().__init__(*a, **kw) > self.thingy = thingy > > try: > raise MyException > except MyException: > print('whoops') > > Instead of printing "whoops", it would now (I think) abort with a > TypeError about trying to call MyException with the wrong number of > arguments. Yes, the exception is instantiated as soon as it is raised (not when it's matched). This happens in Python/ceval.c. So you actually have to catch `TypeError` in this case. > Plus, there's almost certainly code where someone skips creating the > instance as an optimization???probably it's usually a useless premature > one, but maybe someone has tested and it makes a difference in a real > program. It will get instantiated anyway. > One obvious possibility is: if exc is itself a subclass of BaseException, > you use the existing subclass check; if not, instead of using the subclass > check on its type, you use the new instance check. But you'd have to think > that through to make sure it makes sense. From mike at selik.org Thu Nov 5 10:39:33 2015 From: mike at selik.org (Michael Selik) Date: Thu, 05 Nov 2015 15:39:33 +0000 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <510DA2AB-4BB6-4B36-9ADD-2FC48B9CE1EC@yahoo.com> References: <510DA2AB-4BB6-4B36-9ADD-2FC48B9CE1EC@yahoo.com> Message-ID: I'd like to see several examples where a new term would improve the reading of the documentation. So far I agree with Akira Li and Brandon Barnwell. The current usage of "iterable" provides the correct level of specificity. I do not think `list(finite_iterable)` improves the function tooltip. I prefer `list(iterable)`. There's no need to cover all edge cases in the tooltip. Two tasks that require viewing more than one element simultaneously are sorting and finding a median. The functions `sorted(iterable)` and `statistics.median(data)` have reasonable documentation without mentioning the fact that inputting an infinite iterator would be a bad choice. On Thu, Nov 5, 2015 at 4:17 AM Andrew Barnert wrote: > On Nov 4, 2015, at 19:45, Michael Selik wrote: > > > > On 02.10.2015 05:49, Steven D'Aprano wrote: > > > At most, some have suggested that we don't have a good > > > word for those iterables which are not iterators. > > > > Isn't that a "sequence"? The collections module defines a Sequence as > having a __getitem__ and __len__. Is a "non-iterator iterable" something > different? > > A mapping isn't a sequence (even though it has __getitem__ and __len__, > and also has the same supertypes, which add additional restrictions that > you forgot like __contains__). Neither is a set, or any of the kinds of > mapping views, or any intentionally-not-quite-sequence (like range before > 3.2, or, IIRC, blist.sortedlist), or an iterable that, each time it's > iterated, gives you 20 new values in [0, 1) that add up to 1. But they're > all non-iterator iterables. > > In fact, there are a number of concepts that largely overlap, but don't > completely, including: > > * Iterables that aren't iterators > * Iterables that don't return self from __iter__ > * Iterables that always return a new iterator > * Iterables that can repeatably produce a "correct" sequence of values (as > long as they aren't mutated in between iterations) > * Iterables that can repeatably produce equal values (as long as they > aren't mutated) > * Iterables that can repeatedly produce identical values (as long as they > aren't mutated) > > There are also variations of the last three that represent some data that > can be "externally mutated" (e.g., a list of all broadcasting SMB servers > on the local subnet will always produce equal values unless a server goes > up or down between iterations?or, more simply, any view can change values > if the thing it's a view of is mutated, without the view itself being > mutated). > > Anyway, all of the above are true for sequences, mappings, sets, mapping > views, and most not-quite-sequences, but not for the random example, or the > SMB example. > > I don't think we need official names for all of these things, just one?as > long as it's clear which one, and exactly what it means, I think that will > cover well over 90% of all reasonable uses. > > I believe that earlier in the thread, Terry suggested "collection" to mean > the last definition, which seems fine at first glance, so let's assume > that. If we added that to the documentation (and fixed all the places that > sloppily talk about sequences when they don't really mean sequences, etc.), > and ideally added an ABC and a static type to match, then code could very > easily explain that it produces or requires a collection. Most code that > needs two passes over an iterable can just say it requires a collection. > The fact that it might also work with some > sort-of-repeatable-but-not-quite-collection types is fine (and we don't > have to say that such types are "defective" or anything, as long as the > documentation makes it clear that there are some very rare iterables that > are neither iterators nor collections, and what that means). > > Of course Collection defined this way can't be an implicit ABC, because it > doesn't have any interface beyond that of Iterator. But then Sequence and > Mapping can't be implicit either, because they have the same interface as > each other. In fact, that's the main reason we need ABCs in the first > place?for cases where structural/duck typing can't cover the intended > semantics. -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Thu Nov 5 10:45:32 2015 From: random832 at fastmail.com (Random832) Date: Thu, 05 Nov 2015 10:45:32 -0500 Subject: [Python-ideas] Use `isinstance` check during exception handling References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> Message-ID: <87lhactppf.fsf@fastmail.com> Michael Selik writes: > After reading https://github.com/sjoerdjob/exhacktion/ I have a better > understanding of what you want to accomplish. However, I'm still not > clear on where the pain is coming from. If you find yourself writing > an if in an except many times as you described: > > try: > os.chown("README.md", 100, 100) > except EnvironmentError as e: > if e.errno == errno.ENOENT: > pass > else: > raise You know, it occurs to me, there are some languages that support this as a high-level feature. I.e. in Visual Basic Try [Statements] Catch ex As Exception When [Boolean-valued expression] [Statements] End Try This is in fact the *only* way that MS Windows "Structured Exception Handling" exceptions work in C: __try { ... } __except([BOOL-valued expression]) { ... } Both of these have the effect that the exception "was never caught" if it returns false. From random832 at fastmail.com Thu Nov 5 10:58:55 2015 From: random832 at fastmail.com (Random832) Date: Thu, 05 Nov 2015 10:58:55 -0500 Subject: [Python-ideas] Use `isinstance` check during exception handling References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> Message-ID: <87h9l0tp34.fsf@fastmail.com> Chris Angelico writes: > I do like the idea of being able to refactor exception specialization code: > > try: > ... > except BroadError as e: > if is_narrower_error(e): > # handle NarrowerError > else: > raise > > but I don't know whether it's worth complicating the core exception > handling system with such a rare case. The syntax could be this: except BroadError as e if is_narrower_error(e): I think the bytecode *already* evaluates a boolean expression which today happens to always be a comparison of the exception's type. Notably, the exception type itself can apparently be an arbitrary expression, which (I was mildly surprised to discover) has access to sys.exc_info. class NeverThrown(Exception): pass def exfilter(): etype, e = sys.exc_info()[:2] if is_narrower_error(e): return etype else: return NeverThrown try: ... except exfilter() as e: ... This is, of course, a *profoundly* ugly way to do this. From sjoerdjob at sjec.nl Thu Nov 5 11:08:51 2015 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Thu, 5 Nov 2015 17:08:51 +0100 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: <87h9l0tp34.fsf@fastmail.com> References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> <87h9l0tp34.fsf@fastmail.com> Message-ID: <81F19DD4-F43A-4C8F-9056-58124F327ACE@sjec.nl> Yes. This way is really ugly. Also, wrt to the `if` statement: that only works nicely if you don't really want support for the tuple-case. How would you model different conditions for different elements of the tuple? > On 5 Nov 2015, at 16:58, Random832 wrote: > > Chris Angelico writes: >> I do like the idea of being able to refactor exception specialization code: >> >> try: >> ... >> except BroadError as e: >> if is_narrower_error(e): >> # handle NarrowerError >> else: >> raise >> >> but I don't know whether it's worth complicating the core exception >> handling system with such a rare case. > > The syntax could be this: > except BroadError as e if is_narrower_error(e): > > I think the bytecode *already* evaluates a boolean expression > which today happens to always be a comparison of the exception's > type. Notably, the exception type itself can apparently be an > arbitrary expression, which (I was mildly surprised to discover) > has access to sys.exc_info. > > class NeverThrown(Exception): pass > > def exfilter(): > etype, e = sys.exc_info()[:2] > if is_narrower_error(e): return etype > else: return NeverThrown > > try: > ... > except exfilter() as e: > ... > > This is, of course, a *profoundly* ugly way to do this. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ian.g.kelly at gmail.com Thu Nov 5 11:41:44 2015 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 5 Nov 2015 09:41:44 -0700 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: <87h9l0tp34.fsf@fastmail.com> References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> <87h9l0tp34.fsf@fastmail.com> Message-ID: On Thu, Nov 5, 2015 at 8:58 AM, Random832 wrote: > The syntax could be this: > except BroadError as e if is_narrower_error(e): > > I think the bytecode *already* evaluates a boolean expression > which today happens to always be a comparison of the exception's > type. Notably, the exception type itself can apparently be an > arbitrary expression, which (I was mildly surprised to discover) > has access to sys.exc_info. > > class NeverThrown(Exception): pass > > def exfilter(): > etype, e = sys.exc_info()[:2] > if is_narrower_error(e): return etype > else: return NeverThrown > > try: > ... > except exfilter() as e: > ... > > This is, of course, a *profoundly* ugly way to do this. That's pretty neat. You could parameterize the condition and allow a library function to handle the ugly part. def conditional(cls, cond): exc_type, exc = sys.exc_info()[:2] if issubclass(exc_type, cls) and cond(exc): return exc_type return NeverThrown try: ... except conditional(ValueError, lambda e: e.args[0] == 42): ... From sjoerdjob at sjec.nl Thu Nov 5 11:53:24 2015 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Thu, 5 Nov 2015 17:53:24 +0100 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> <87h9l0tp34.fsf@fastmail.com> Message-ID: This does sound like a nice approach. Neat! However: it *does* depend on only being resolved during the exception being matched. So you can't pre-define it. Still, I like it. > On 5 Nov 2015, at 17:41, Ian Kelly wrote: > >> On Thu, Nov 5, 2015 at 8:58 AM, Random832 wrote: >> The syntax could be this: >> except BroadError as e if is_narrower_error(e): >> >> I think the bytecode *already* evaluates a boolean expression >> which today happens to always be a comparison of the exception's >> type. Notably, the exception type itself can apparently be an >> arbitrary expression, which (I was mildly surprised to discover) >> has access to sys.exc_info. >> >> class NeverThrown(Exception): pass >> >> def exfilter(): >> etype, e = sys.exc_info()[:2] >> if is_narrower_error(e): return etype >> else: return NeverThrown >> >> try: >> ... >> except exfilter() as e: >> ... >> >> This is, of course, a *profoundly* ugly way to do this. > > That's pretty neat. You could parameterize the condition and allow a > library function to handle the ugly part. > > def conditional(cls, cond): > exc_type, exc = sys.exc_info()[:2] > if issubclass(exc_type, cls) and cond(exc): > return exc_type > return NeverThrown > > try: > ... > except conditional(ValueError, lambda e: e.args[0] == 42): > ... > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Thu Nov 5 15:06:40 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 5 Nov 2015 12:06:40 -0800 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> Message-ID: <83547380-3947-42BA-B8B6-D97AF52D296D@yahoo.com> On Nov 5, 2015, at 07:00, sjoerdjob at sjec.nl wrote: >>> On Nov 4, 2015, at 14:02, sjoerdjob at sjec.nl wrote: >>> >>> Hi all, >>> >>> TL;DR: Change exception checking logic from >>> err.__class__ in exc.__mro__ >>> to >>> isinstance(err, exc) >>> Because it is more powerful, more consistent and allows for cleaner >>> code. >> >> Correct me if I'm wrong, but doesn't Python allow you to raise a class and >> handle it without ever creating an instance? If an instance is needed >> (e.g., because the exception is caught by a handler with an as clause), >> one is created by calling exc(), but if it's not needed, that doesn't >> happen. >> >> Assuming I'm right, your proposal would mean the instance is _always_ >> needed, so if you raise a type, exc() is always called. So, for example: >> >> class MyException(Exception): >> def __init__(self, thingy, *a, **kw): >> super().__init__(*a, **kw) >> self.thingy = thingy >> >> try: >> raise MyException >> except MyException: >> print('whoops') >> >> Instead of printing "whoops", it would now (I think) abort with a >> TypeError about trying to call MyException with the wrong number of >> arguments. > > Yes, the exception is instantiated as soon as it is raised (not when it's > matched). This happens in Python/ceval.c. So you actually have to catch > `TypeError` in this case. Then what's the point of the code at Python/errors.c#186 that deals with the fact that the given exception (err) can be either an instance or a type? If it must always be an instance, surely those two lines are unnecessary and misleading. In fact, looking into it a bit more: the normal way to check for an exception from C is with PyErr_ExceptionMatches, which calls PyErr_GivenExceptionMatches with the type, not the value, of the exception. So, even if an instance has been created, you don't usually have it inside PyErr_GivenExceptionMatches anyway, so the proposed change wouldn't work without further changes. At the very least you'd need to change PyErr_ExceptionMatches to pass the value instead of the type. But presumably there's a reason it passes the type in the first place (maybe in CPython it's not possible to raise a type without an instance from pure Python code, but it is from C API code?), which means there's probably other code that has to change as well. And that's assuming there's no other code than calls PyErr_GivenExceptionMatches with the value of PyErr_Occurred() (or tstate->curexc_type or any other way of getting an exception type) anywhere in CPython or in any extension module anyone's ever written. Since there's nothing about the docs that implies that this is invalid, and in fact the docs for PyErr_ExceptionMatches and the source code for PyErr_GivenExceptionMatches both strongly imply that passing a type is the normal way to check, I doubt that assumption is true. >> Plus, there's almost certainly code where someone skips creating the >> instance as an optimization?probably it's usually a useless premature >> one, but maybe someone has tested and it makes a difference in a real >> program. > > It will get instantiated anyway. > >> One obvious possibility is: if exc is itself a subclass of BaseException, >> you use the existing subclass check; if not, instead of using the subclass >> check on its type, you use the new instance check. But you'd have to think >> that through to make sure it makes sense. > From sjoerdjob at sjec.nl Thu Nov 5 15:18:22 2015 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Thu, 5 Nov 2015 21:18:22 +0100 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: <83547380-3947-42BA-B8B6-D97AF52D296D@yahoo.com> References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> <83547380-3947-42BA-B8B6-D97AF52D296D@yahoo.com> Message-ID: <3540DB5E-C07D-4240-82EE-30966B45B2D8@sjec.nl> > On 5 Nov 2015, at 21:06, Andrew Barnert wrote: > > On Nov 5, 2015, at 07:00, sjoerdjob at sjec.nl wrote: > >>>> On Nov 4, 2015, at 14:02, sjoerdjob at sjec.nl wrote: >>>> >>>> Hi all, >>>> >>>> TL;DR: Change exception checking logic from >>>> err.__class__ in exc.__mro__ >>>> to >>>> isinstance(err, exc) >>>> Because it is more powerful, more consistent and allows for cleaner >>>> code. >>> >>> Correct me if I'm wrong, but doesn't Python allow you to raise a class and >>> handle it without ever creating an instance? If an instance is needed >>> (e.g., because the exception is caught by a handler with an as clause), >>> one is created by calling exc(), but if it's not needed, that doesn't >>> happen. >>> >>> Assuming I'm right, your proposal would mean the instance is _always_ >>> needed, so if you raise a type, exc() is always called. So, for example: >>> >>> class MyException(Exception): >>> def __init__(self, thingy, *a, **kw): >>> super().__init__(*a, **kw) >>> self.thingy = thingy >>> >>> try: >>> raise MyException >>> except MyException: >>> print('whoops') >>> >>> Instead of printing "whoops", it would now (I think) abort with a >>> TypeError about trying to call MyException with the wrong number of >>> arguments. >> >> Yes, the exception is instantiated as soon as it is raised (not when it's >> matched). This happens in Python/ceval.c. So you actually have to catch >> `TypeError` in this case. > > Then what's the point of the code at Python/errors.c#186 that deals with the fact that the given exception (err) can be either an instance or a type? If it must always be an instance, surely those two lines are unnecessary and misleading. > > In fact, looking into it a bit more: the normal way to check for an exception from C is with PyErr_ExceptionMatches, which calls PyErr_GivenExceptionMatches with the type, not the value, of the exception. So, even if an instance has been created, you don't usually have it inside PyErr_GivenExceptionMatches anyway, so the proposed change wouldn't work without further changes. At the very least you'd need to change PyErr_ExceptionMatches to pass the value instead of the type. But presumably there's a reason it passes the type in the first place (maybe in CPython it's not possible to raise a type without an instance from pure Python code, but it is from C API code?), which means there's probably other code that has to change as well. > > And that's assuming there's no other code than calls PyErr_GivenExceptionMatches with the value of PyErr_Occurred() (or tstate->curexc_type or any other way of getting an exception type) anywhere in CPython or in any extension module anyone's ever written. Since there's nothing about the docs that implies that this is invalid, and in fact the docs for PyErr_ExceptionMatches and the source code for PyErr_GivenExceptionMatches both strongly imply that passing a type is the normal way to check, I doubt that assumption is true. I added an assert before the except, asserting that no instance is passed, and surprisingly that assert triggers. Also: in some cases, tstate->cur_value is actually a PyString object instead of an exception subclass (for instance raised by PyErr_FormatV (or something like that, don't have the code in front of me right now). So yes, it does appear that a lot of code might need changing. But it also seems odd to me that exceptions get raised with strings for values instead of exception instances. It might seem troublesome to change these locations to raise actual instances instead of strings, but those might actually just be hidden bugs just the same. Do you think anyone would object if I'd go over the code looking for places where type(tstate->cur_value) != tstate->cur_type and propose patches for these? >>> Plus, there's almost certainly code where someone skips creating the >>> instance as an optimization?probably it's usually a useless premature >>> one, but maybe someone has tested and it makes a difference in a real >>> program. >> >> It will get instantiated anyway. >> >>> One obvious possibility is: if exc is itself a subclass of BaseException, >>> you use the existing subclass check; if not, instead of using the subclass >>> check on its type, you use the new instance check. But you'd have to think >>> that through to make sure it makes sense. >> From abarnert at yahoo.com Thu Nov 5 17:03:14 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 5 Nov 2015 14:03:14 -0800 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: <3540DB5E-C07D-4240-82EE-30966B45B2D8@sjec.nl> References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> <83547380-3947-42BA-B8B6-D97AF52D296D@yahoo.com> <3540DB5E-C07D-4240-82EE-30966B45B2D8@sjec.nl> Message-ID: On Nov 5, 2015, at 12:18, Sjoerd Job Postmus wrote: > >> On 5 Nov 2015, at 21:06, Andrew Barnert wrote: >> >> On Nov 5, 2015, at 07:00, sjoerdjob at sjec.nl wrote: >> >>>>> On Nov 4, 2015, at 14:02, sjoerdjob at sjec.nl wrote: >>>>> >>>>> Hi all, >>>>> >>>>> TL;DR: Change exception checking logic from >>>>> err.__class__ in exc.__mro__ >>>>> to >>>>> isinstance(err, exc) >>>>> Because it is more powerful, more consistent and allows for cleaner >>>>> code. >>>> >>>> Correct me if I'm wrong, but doesn't Python allow you to raise a class and >>>> handle it without ever creating an instance? If an instance is needed >>>> (e.g., because the exception is caught by a handler with an as clause), >>>> one is created by calling exc(), but if it's not needed, that doesn't >>>> happen. >>>> >>>> Assuming I'm right, your proposal would mean the instance is _always_ >>>> needed, so if you raise a type, exc() is always called. So, for example: >>>> >>>> class MyException(Exception): >>>> def __init__(self, thingy, *a, **kw): >>>> super().__init__(*a, **kw) >>>> self.thingy = thingy >>>> >>>> try: >>>> raise MyException >>>> except MyException: >>>> print('whoops') >>>> >>>> Instead of printing "whoops", it would now (I think) abort with a >>>> TypeError about trying to call MyException with the wrong number of >>>> arguments. >>> >>> Yes, the exception is instantiated as soon as it is raised (not when it's >>> matched). This happens in Python/ceval.c. So you actually have to catch >>> `TypeError` in this case. >> >> Then what's the point of the code at Python/errors.c#186 that deals with the fact that the given exception (err) can be either an instance or a type? If it must always be an instance, surely those two lines are unnecessary and misleading. >> >> In fact, looking into it a bit more: the normal way to check for an exception from C is with PyErr_ExceptionMatches, which calls PyErr_GivenExceptionMatches with the type, not the value, of the exception. So, even if an instance has been created, you don't usually have it inside PyErr_GivenExceptionMatches anyway, so the proposed change wouldn't work without further changes. At the very least you'd need to change PyErr_ExceptionMatches to pass the value instead of the type. But presumably there's a reason it passes the type in the first place (maybe in CPython it's not possible to raise a type without an instance from pure Python code, but it is from C API code?), which means there's probably other code that has to change as well. >> >> And that's assuming there's no other code than calls PyErr_GivenExceptionMatches with the value of PyErr_Occurred() (or tstate->curexc_type or any other way of getting an exception type) anywhere in CPython or in any extension module anyone's ever written. Since there's nothing about the docs that implies that this is invalid, and in fact the docs for PyErr_ExceptionMatches and the source code for PyErr_GivenExceptionMatches both strongly imply that passing a type is the normal way to check, I doubt that assumption is true. > > I added an assert before the except, asserting that no instance is passed, and surprisingly that assert triggers. > > Also: in some cases, tstate->cur_value is actually a PyString object instead of an exception subclass (for instance raised by PyErr_FormatV (or something like that, don't have the code in front of me right now). > > So yes, it does appear that a lot of code might need changing. But it also seems odd to me that exceptions get raised with strings for values instead of exception instances. > > It might seem troublesome to change these locations to raise actual instances instead of strings, but those might actually just be hidden bugs just the same. > > Do you think anyone would object if I'd go over the code looking for places where type(tstate->cur_value) != tstate->cur_type and propose patches for these? It would be really nice if someone remembered _why_ the code is this way. It may be a performance issue?if the exception is raised in C code and handled in C code and never displayed, the exception object never gets used, so why waste time (and complicate the reference graph) creating one? Or it may just be a holdover from Python 1.x's string exceptions. Or there may be some other reason neither of us can guess. Without knowing which it is, it's hard to know what to do about it. But meanwhile, even if you patch all of CPython, that doesn't change the fact that there are probably extension libraries out there setting exceptions with a null or string value, or checking the exception by using PyErr_ExceptionOccurred or other means that ignore the value (without going through PyErr_GivenExceptionMatches or anything else you could change), which are perfectly valid things to do according to the docs, and have worked for decades. How are you going to fix all of those? Also, even if it's true that pure Python code can't create an exception without a value, only C code can, the way the language reference is written, that seems to be something specific to CPython, not guaranteed by the language. So, other implementations might be doing something different, and you'd have to fix all of them as well. (That may not be a big deal?are there any other major 3.x implementations besides PyPy, which is still on 3.2?) The best thing I can think of is to add a new API around exception values instead of types (a replacement for ExceptionOccurred that returns the value instead of the type, a replacement for ExceptionMatches that requires an instance instead of taking an instance or type, etc.), change GivenExceptionMatches (and the except clause implementation) to use that API, change SetNone and friends to always create a type, deprecate ExceptionOccurred and friends, change the documentation to make it clear than raising a type in Python is the same as raising Type(), wait a couple versions for the C API deprecation, and only then can you assume that ExceptionMatches always gets an instance (and decide what you do if not), so you can make your proposed change. From rosuav at gmail.com Thu Nov 5 17:12:28 2015 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 6 Nov 2015 09:12:28 +1100 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> <83547380-3947-42BA-B8B6-D97AF52D296D@yahoo.com> <3540DB5E-C07D-4240-82EE-30966B45B2D8@sjec.nl> Message-ID: On Fri, Nov 6, 2015 at 9:03 AM, Andrew Barnert via Python-ideas wrote: > It would be really nice if someone remembered _why_ the code is this way. It may be a performance issue?if the exception is raised in C code and handled in C code and never displayed, the exception object never gets used, so why waste time (and complicate the reference graph) creating one? > I have some recollection that it's most commonly hit when you're working with StopIteration - an internal iterator raises it, a 'for' loop catches it, and no instance is ever constructed. ChrisA From stephen at xemacs.org Fri Nov 6 00:33:24 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 6 Nov 2015 14:33:24 +0900 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> <83547380-3947-42BA-B8B6-D97AF52D296D@yahoo.com> <3540DB5E-C07D-4240-82EE-30966B45B2D8@sjec.nl> Message-ID: <22076.15268.375643.440588@turnbull.sk.tsukuba.ac.jp> Andrew Barnert via Python-ideas writes: > Sjoerd Job Postmus wrote: > > It might seem troublesome to change these locations to raise > > actual instances instead of strings, but those might actually > > just be hidden bugs just the same. No, I'm sure mostly they're not. They're working code that has been there since the early days, perhaps a few from the 2.x days in modules. Python is generally very conservative about changing working code. > > Do you think anyone would object if I'd go over the code looking > > for places where type(tstate->cur_value) != tstate->cur_type and > > propose patches for these? I think, yes, though I don't know enough about it to object myself. Code churn of this kind is generally frowned upon, unless you can show that the risk of hidden bugs of the kind you refer to above is substantially greater than the risk that *you* will introduce bugs with your patches. > It would be really nice if someone remembered _why_ the code is > this way. It may be a performance issue? Or it could be bootstrapping -- it may be that "something" can be raised before the exception machinery is fully initialized. (On reflection, that seems unlikely, if raise is available, BaseException probably is too. But you never know. :) I would guess it was discussed, decided, and pronounced in one of the early Python 3 PEPs. (I'm sure that for Python 2 it was purely an issue of backward compatibility.) From sjoerdjob at sjec.nl Fri Nov 6 00:57:22 2015 From: sjoerdjob at sjec.nl (sjoerdjob at sjec.nl) Date: Fri, 6 Nov 2015 06:57:22 +0100 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: <22076.15268.375643.440588@turnbull.sk.tsukuba.ac.jp> References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl> <83547380-3947-42BA-B8B6-D97AF52D296D@yahoo.com> <3540DB5E-C07D-4240-82EE-30966B45B2D8@sjec.nl> <22076.15268.375643.440588@turnbull.sk.tsukuba.ac.jp> Message-ID: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl> > Andrew Barnert via Python-ideas writes: > > Sjoerd Job Postmus wrote: > > > > It might seem troublesome to change these locations to raise > > > actual instances instead of strings, but those might actually > > > just be hidden bugs just the same. > > No, I'm sure mostly they're not. They're working code that has been > there since the early days, perhaps a few from the 2.x days in > modules. Python is generally very conservative about changing working > code. > > > > Do you think anyone would object if I'd go over the code looking > > > for places where type(tstate->cur_value) != tstate->cur_type and > > > propose patches for these? > > I think, yes, though I don't know enough about it to object myself. > Code churn of this kind is generally frowned upon, unless you can show > that the risk of hidden bugs of the kind you refer to above is > substantially greater than the risk that *you* will introduce bugs > with your patches. After digging more into the code, I concur. It's quite easy to make some changes that break some stuff in this logic. > > It would be really nice if someone remembered _why_ the code is > > this way. It may be a performance issue??? > > Or it could be bootstrapping -- it may be that "something" can be > raised before the exception machinery is fully initialized. (On > reflection, that seems unlikely, if raise is available, BaseException > probably is too. But you never know. :) > > I would guess it was discussed, decided, and pronounced in one of the > early Python 3 PEPs. (I'm sure that for Python 2 it was purely an > issue of backward compatibility.) To all: thank you for at least considering my suggestion, and helping me consider the pros and cons. Even though I myself still think it would be a major benefit to change from `issubclass` (or `exc in err.__mro__`) to `isinstance`, I do now see that the changes needed to the Python codebase are quite substantial. The consideration about third-party C-modules is also something to further think about. As I understand it now, the changes needed---and the possibility of introduced bugs---outweigh the added benefit of such a change, at least for now. (Still, it send me along a nice cursory browse of the Python code base, including the bytecode compiler and the evaluator. The compilation of the bytecode for exception handling is nicely documented. To the author(s) of that code: great job, it's very understandable!) From abarnert at yahoo.com Fri Nov 6 02:49:07 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 6 Nov 2015 07:49:07 +0000 (UTC) Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl> References: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl> Message-ID: <1277750399.598478.1446796147897.JavaMail.yahoo@mail.yahoo.com> On Thursday, November 5, 2015 9:57 PM, "sjoerdjob at sjec.nl" wrote: >> Andrew Barnert via Python-ideas writes: >> > Sjoerd Job Postmus wrote: >> >> > > It might seem troublesome to change these locations to raise >> > > actual instances instead of strings, but those might actually >> > > just be hidden bugs just the same. >> >> No, I'm sure mostly they're not. They're working code that has > been >> there since the early days, perhaps a few from the 2.x days in >> modules. Python is generally very conservative about changing working >> code. >> >> > > Do you think anyone would object if I'd go over the code > looking >> > > for places where type(tstate->cur_value) != > tstate->cur_type and >> > > propose patches for these? >> >> I think, yes, though I don't know enough about it to object myself. >> Code churn of this kind is generally frowned upon, unless you can show >> that the risk of hidden bugs of the kind you refer to above is >> substantially greater than the risk that *you* will introduce bugs >> with your patches. > > After digging more into the code, I concur. It's quite easy to make some > changes that break some stuff in this logic. Looking at this a bit further, I think there may be something you can do that gets everything you actually wanted, without changes all over the place and serious risks. Basically, it's just two pretty small changes in errors.c: * Change PyErr_GivenExceptionMatches so that if given a value that's an instance of an exception type (PyExceptionInstance_Check(err) is true), instead of just getting its class and then ignoring the value, it does a standard isinstance check on the value. * Change PyErr_ExceptionMatches so that if there is an exception value, and its type matches the exception type, it is passed to PyErr_GivenExceptionMatches; otherwise, the exception type is passed. So, all Python code that raises and handles exceptions will get the benefit of isinstance checks. Any new C code that wants that benefit can get it very easily (only raise with PyErr_SetObject or something that uses it like PyErr_SetFromErrno; only handle with PyErr_GivenExceptionMatches). Any existing C code that skips creating an instance will continue to do so, and to get the same fast-path check used today, except for one extra call to PyExceptionInstance_Check(tstate->curexc_value). Of course it's not just that small patch to errors.c. You also need tests to exercise the new functionality. And a docs patch for the Exceptions section of the C API. And also a docs patch to the Exceptions section in the Reference. But no other code patches. And you'd definitely need to performance-test two things: First, that the extra PyExceptionInstance_Check doesn't slow down fast-path handling (e.g., in an empty for loops). Second, whatever performance tests your original proposal would have needed, to make sure the isinstance check doesn't noticeably slow down pure-Python code that handles a lot of exceptions. I think the only C API docs change needed is to PyErr_ExceptionMatches. However, a note explaining how to make sure to get and/or to bypass the isinstance check as desired might be useful. For the reference docs, I think just this note at the end of the section: > Note: Exceptions raised and handled by Python code use the same mechanism as the isinstance function for determining whether an exception matches. However, exceptions raised or handled by the implementation (or implementation-level extensions) may bypass that and check the type directly. > New in 3.6: In previous versions of Python, exception handling always bypassed the isinstance function and checked the type directly. From amir at rachum.com Fri Nov 6 17:11:14 2015 From: amir at rachum.com (Amir Rachum) Date: Sat, 7 Nov 2015 00:11:14 +0200 Subject: [Python-ideas] Adding collections.abc.Ordered Message-ID: I am suggesting the addition of a collections abstract base class called "Ordered". Its meaning is that a collection's iteration order is part of its API. The bulk of this mail describes a use case for this. The reason I believe that such abstract base class is required is that there is no way to test this behavior in a given class. An ordered collection has the exact same interface as an unordered collection (e.g, dict and OrderedDict), other than a _promise_ of the API that the order in which this collection will be iterated has some sort of meaning (In OrderedDict, it is the order in which keys were added to it.) As examples, set, frozenset, dict and defaultdict should *not* be considered as ordered. list, OrderedDict, deque and tuple should be considered ordered. Before I dive into the use case I am presenting, I would like to point out that a similar discussion was already done on the subject (suggesting at first that OrderedDict would abstract-subclass from Sequence) at the following thread* - http://code.activestate.com/lists/python-ideas/29532/. That thread was abandoned largely from a lack of a clear use case, which I hope will be more clear in this suggestion. I am working on package called basicstruct ( https://pypi.python.org/pypi/basicstruct). The way it works is that you define an object that inherits from basicstruct.BasicStruct, define __slots__ , and automatically get behaviors such as a nice __init__, __str__, comparison functions, attribute access, dict-like access, pickleability, etc. It is similar to namedtuple, except that it is mutable. In the Python documentation, the following is said regarding __slots__: Any non-string iterable may be assigned to __slots__. Mappings may also be used; however, in the future, special meaning may be assigned to the values corresponding to each key. Here's how the current__init__ method of BasicStruct looks like: class BasicStruct(object): """Class for holding struct-like objects.""" __slots__ = () # should be extended by deriving classes def __init__(self, *args, **kwargs): arg_pairs = zip(self.__slots__, args) for key, value in chain(arg_pairs, six.iteritems(kwargs)): setattr(self, key, value) for key in self.__slots__: if not hasattr(self, key): setattr(self, key, None) Notice the following line: arg_pairs = zip(self.__slots__, args) It assumes that __slots__ defines attributes in a certain order. So as a use I would expect that the following code class MyStruct(BasicStruct): __slots__ = ('x', 'y') MyStruct(0, 1) ... will create a struct in which x is 0 and y is 1. However, if I define __slots__ as a set, or dict, the result is undefined. class MyStruct(BasicStruct): __slots__ = {'x', 'y'} # No order is defined here MyStruct(0, 1) # Which is which? So, In BasicStruct's __init__ method it is required to test whether __slots__ was defined with a collection that has a meaningful order. If it was _not_, using non-keyword arguments in __init__ should be forbidden, since the order of __slots__ is arbitrary. Here is how I wish the code would look like: class BasicStruct(object): """Class for holding struct-like objects.""" __slots__ = () # should be extended by deriving classes def __init__(self, *args, **kwargs): ordered = isinstance(self.__slots__, Ordered) if args and not ordered: raise ValueError("Can't pass non-keyword arguments to {}, since " "__slots__ was declared with an unordered " "iterable.".format(self.__class__.__name__)) arg_pairs = zip(self.__slots__, args) for key, value in chain(arg_pairs, six.iteritems(kwargs)): setattr(self, key, value) for key in self.__slots__: if not hasattr(self, key): setattr(self, key, None) Thanks, Amir Rachum * The author of the original thread is Ram Rachum. To avoid some confusion - yes, we are related - Ram is my big brother. It's just so happens that I was looking around for this issue and found that he did so in the past as well. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Nov 6 23:47:41 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 6 Nov 2015 20:47:41 -0800 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: References: Message-ID: <2B772FD7-9B1A-45E1-A38C-415F194DA503@yahoo.com> As I understand it, the only reason you want this ABC is so that if someone uses your struct-sequence class and defines his attributes with no inherent order, you want to be able to raise an error if he uses positional initialization. Has anyone ever done that? Has anyone complained "I created a struct type with attributes a and b in arbitrary order, and then when I do Spam(1, 2), sometimes a gets 1 and sometimes it gets 2"? It seems to me that this is obviously the same as, say, a user explicitly zipping two sets together. Sure, it's a stupid thing to do?but it's an _obvious_ stupid thing to do, so zip doesn't have to prevent them from doing it or go out of its way to warn them. And it's the same here. The order of the attributes is the order of the slots, so if you specify them in arbitrary order, that will be an arbitrary order, obviously. Maybe there's something about this use case that causes people to not notice they've screwed up like this? If so, I think that's what you have to make a case for. (That's why I asked whether anyone has actually complained about it.) One more thing: what's stopping you from defining Ordered in your module and registering all the relevant builtin types there? In fact, you pretty much have to do this even if your proposal is accepted, unless you want your module to require 3.6+ (which seems unlikely, given that you're using six). So what would you gain by having it in the stdlib? > On Nov 6, 2015, at 14:11, Amir Rachum wrote: > > I am suggesting the addition of a collections abstract base class called "Ordered". Its meaning is that a collection's iteration order is part of its API. The bulk of this mail describes a use case for this. The reason I believe that such abstract base class is required is that there is no way to test this behavior in a given class. An ordered collection has the exact same interface as an unordered collection (e.g, dict and OrderedDict), other than a _promise_ of the API that the order in which this collection will be iterated has some sort of meaning (In OrderedDict, it is the order in which keys were added to it.) > > As examples, set, frozenset, dict and defaultdict should *not* be considered as ordered. list, OrderedDict, deque and tuple should be considered ordered. > > Before I dive into the use case I am presenting, I would like to point out that a similar discussion was already done on the subject (suggesting at first that OrderedDict would abstract-subclass from Sequence) at the following thread* - http://code.activestate.com/lists/python-ideas/29532/. That thread was abandoned largely from a lack of a clear use case, which I hope will be more clear in this suggestion. > > I am working on package called basicstruct (https://pypi.python.org/pypi/basicstruct). The way it works is that you define an object that inherits from basicstruct.BasicStruct, define __slots__ , and automatically get behaviors such as a nice __init__, __str__, comparison functions, attribute access, dict-like access, pickleability, etc. It is similar to namedtuple, except that it is mutable. > > In the Python documentation, the following is said regarding __slots__: > > Any non-string iterable may be assigned to __slots__. Mappings may also be used; however, in the future, special meaning may be assigned to the values corresponding to each key. > > Here's how the current__init__ method of BasicStruct looks like: > > class BasicStruct(object): > """Class for holding struct-like objects.""" > > __slots__ = () # should be extended by deriving classes > > def __init__(self, *args, **kwargs): > arg_pairs = zip(self.__slots__, args) > for key, value in chain(arg_pairs, six.iteritems(kwargs)): > setattr(self, key, value) > > for key in self.__slots__: > if not hasattr(self, key): > setattr(self, key, None) > > Notice the following line: > > arg_pairs = zip(self.__slots__, args) > > It assumes that __slots__ defines attributes in a certain order. So as a use I would expect that the following code > > class MyStruct(BasicStruct): > __slots__ = ('x', 'y') > > MyStruct(0, 1) > > ... will create a struct in which x is 0 and y is 1. > > However, if I define __slots__ as a set, or dict, the result is undefined. > > class MyStruct(BasicStruct): > __slots__ = {'x', 'y'} # No order is defined here > > MyStruct(0, 1) # Which is which? > > So, In BasicStruct's __init__ method it is required to test whether __slots__ was defined with a collection that has a meaningful order. If it was _not_, using non-keyword arguments in __init__ should be forbidden, since the order of __slots__ is arbitrary. Here is how I wish the code would look like: > > class BasicStruct(object): > """Class for holding struct-like objects.""" > > __slots__ = () # should be extended by deriving classes > > def __init__(self, *args, **kwargs): > ordered = isinstance(self.__slots__, Ordered) > > if args and not ordered: > raise ValueError("Can't pass non-keyword arguments to {}, since " > "__slots__ was declared with an unordered " > "iterable.".format(self.__class__.__name__)) > > arg_pairs = zip(self.__slots__, args) > for key, value in chain(arg_pairs, six.iteritems(kwargs)): > setattr(self, key, value) > > for key in self.__slots__: > if not hasattr(self, key): > setattr(self, key, None) > > Thanks, > Amir Rachum > > > > * The author of the original thread is Ram Rachum. To avoid some confusion - yes, we are related - Ram is my big brother. It's just so happens that I was looking around for this issue and found that he did so in the past as well. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sat Nov 7 00:00:49 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 7 Nov 2015 14:00:49 +0900 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: References: Message-ID: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> Amir Rachum writes: > An ordered collection has the exact same interface as an unordered > collection (e.g, dict and OrderedDict), other than a _promise_ of > the API that the order in which this collection will be iterated > has some sort of meaning (In OrderedDict, it is the order in which > keys were added to it.) I think this is underspecified. One of the main reasons why ordered versions of initially unordered types, especially dict, take so long to standardize is that people (even the Dutch!) disagree about the One Obvious Order to impose. After repeated refusals, only the factions that continue clamoring- and preferably, only one -need be considered. This ABC would be a tar pit of recriminations between class authors and class clients. At the very least, the author's intended comparison function should be introspectable so that client subclasses can include validation functions. (I guess you could just use "lambda slot: list(self.__slots__).index(slot)" as the key function, but that doesn't say much about the conceptual semantics.) Then the presence of the ordering function would provide a way to duck-type abc.Ordered. I haven't thought carefully about it, but a method with semantics of "insert next slot at end according to natural order" (named "natural_append"? "ordered_append"? or maybe the existing "append" used by sequences could be given this semantic as well? Nevertheless, I don't think this shold be added to Python yet. I don't think your use case is a valid one for supporting it. I have two reasons: (1) In principle, record types should not be initialized with positional arguments. I experienced cogniztive dissonance as soon as I saw your __init__(): > Here's how the current__init__ method of BasicStruct looks like: > class BasicStruct(object): > """Class for holding struct-like objects.""" "with naturally-ordered slots" should be appended. > __slots__ = () # should be extended by deriving classes > def __init__(self, *args, **kwargs): Yes, we do it in C all the time, and it works fine[1] there. But the semantics are mismatched at the conceptual level, and it is error-prone in practice if the type has more than about 3 slots of differing types. If a concrete BasicStruct "wants" to provide a positional constructor, it can use a factory function. Unfortunately this would be impossible to do automatically in the exactly the cases of interest -- you need the *source code* order of slots, but that's not available for introspection at runtime. So it would violate DRY. I don't consider that an argument for the proposal at the moment, but feel free to try to convince the list (and maybe me too) otherwise.) (In fact I consider it the reverse: for Sequences, The Order You See Is The Order You Get, and the DRY violation occurs *because* you are using an inappropriate container for your purpose (ordered slots). (2) BasicStruct isn't the relevant use case: you can always just document that __slots__ is restricted to ordered finite iterables except string-like ones which use sequences of length 1 to represent individual members, and perhaps trap some of the more obvious mistakes (set and dict and maybe their subclasses). The use case you need to present is one where a non-Sequence (eg, a set or dict) is the obvious representation for __slots__ in a BasicStruct. Note that __slots__ allows nearly arbitrary finite iterables of strings only because there's no reason not to, and it simplifies the implementation. In Python practice, that is indeed an invitation to experiment with concrete classes that use unordered __slots__ for some reason. But there's no sense that abstract classes that use __slots__ in an abstract way (as yours does) must preserve that generality, and therefore no reason (yet) for Python to help you preserve that generality. Footnotes: [1] According to the usual definition of "works fine in C": is no more than twice as error-prone as other features of C. Given the existence of pointers.... From abarnert at yahoo.com Sat Nov 7 04:54:56 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 7 Nov 2015 01:54:56 -0800 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> Message-ID: <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> On Nov 6, 2015, at 21:00, Stephen J. Turnbull wrote: > > Amir Rachum writes: > >> An ordered collection has the exact same interface as an unordered >> collection (e.g, dict and OrderedDict), other than a _promise_ of >> the API that the order in which this collection will be iterated >> has some sort of meaning (In OrderedDict, it is the order in which >> keys were added to it.) > > I think this is underspecified. One of the main reasons why ordered > versions of initially unordered types, especially dict, take so long > to standardize is that people (even the Dutch!) disagree about the One > Obvious Order to impose. After repeated refusals, only the factions > that continue clamoring- and preferably, only one -need be considered. > > This ABC would be a tar pit of recriminations between class authors > and class clients. I think you've missed the point of his proposal. It doesn't matter _what_ the order is?insertion order, sorted by __lt__ (possibly with a key function), looked up by a function that maps keys to indices, whatever?only that there is some meaningful order (and that iteration follows that order). If so, your type can declare that it implements Ordered. Of course the user has to know what that order is and what it means in order to do anything with it. In his example, if you use, say, a blist.sortedlist instead of a list for your slots, then you have to pass positional parameters to the initializer in sorted order instead of in source code order. But if that's what makes sense for your code, then great. The reason I'm not sure it's helpful is that "ordered in some way that makes sense to me that you don't know about" isn't really that useful of a thing for you to check for me. After all, I can use a blist.sortedlist('bac') and then pass the initializer values in b-a-c order, or use list('bac') and pass them in a-b-c order. Each of those is at least as easy to screw up, and just as wrong, as using a set('bac'), and his code can't possibly check for the first two, so what's the point of checking for the third? But, whether the proposal is useful or not, it is sufficiently specified. > At the very least, the author's intended > comparison function should be introspectable so that client subclasses > can include validation functions. (I guess you could just use > "lambda slot: list(self.__slots__).index(slot)" as the key function, > but that doesn't say much about the conceptual semantics.) Then the > presence of the ordering function would provide a way to duck-type > abc.Ordered. How would you define the comparison function for OrderedDict in a way that makes any semantic sense? Or, for that matter, list? The order is the result of a dynamic sequence of operations that nobody's kept track of and that in general can't be recovered. But it doesn't matter why the values are in some particular order, only that they are. Surely OrderedDict and list are perfectly reasonable types for __slots__, despite the fact that their comparison function doesn't exist. > I haven't thought carefully about it, but a method with semantics of > "insert next slot at end according to natural order" (named > "natural_append"? "ordered_append"? or maybe the existing "append" > used by sequences could be given this semantic as well? You're talking about the API for a mutable sorted collection here. That's an interesting question (and I've written a blog post about it, and had long conversations with the authors of various different sorted collection libs on PyPI), but it's not relevant here. If we wanted to add sorted collection ABCs to the stdlib, that still wouldn't help the OP at all, because things like list and OrderedDict are Ordered in his sense but not sorted in any useful sense. (If you're curious: I think the best answer is to use add. A sorted collection is more like a multiset, at least from the mutation side, than like a list. Append, insert, and __setitem__ are all nonsense; add makes perfect sense. But not everyone agrees with me. If you want to discuss this, you should probably take it off-list, or start a new thread on adding sorted collections ABCs, because they have very little to do with adding an Ordered ABC.) > Nevertheless, I don't think this shold be added to Python yet. I > don't think your use case is a valid one for supporting it. I have > two reasons: > > (1) In principle, record types should not be initialized with > positional arguments. I experienced cogniztive dissonance as soon > as I saw your __init__(): > >> Here's how the current__init__ method of BasicStruct looks like: > >> class BasicStruct(object): >> """Class for holding struct-like objects.""" > > "with naturally-ordered slots" should be appended. > >> __slots__ = () # should be extended by deriving classes > >> def __init__(self, *args, **kwargs): > > Yes, we do it in C all the time, and it works fine[1] there. But > the semantics are mismatched at the conceptual level, and it is > error-prone in practice if the type has more than about 3 slots of > differing types It's worth noting that the C committee agrees with you, which is why they added designated initializers to the language, which are often considered among the best upgrades from C89. Unless you need C89 support or (portable) C++ compatibility (sadly, you usually do need one or the other?), unless you're initializing something with only a few attributes with an obvious-to-the-reader order (like a point, where it's pretty clear that { 1, 2, 3 } is initializing x, y, and z), you use a designated initializer like { .spam=1, .eggs=2, .cheese=3 }. Which looks sort of like Python, but with extra line noise. > If a concrete BasicStruct "wants" to provide a > positional constructor, it can use a factory function. > > Unfortunately this would be impossible to do automatically in the > exactly the cases of interest -- you need the *source code* order > of slots, but that's not available for introspection at runtime. > So it would violate DRY. I don't consider that an argument for > the proposal at the moment, but feel free to try to convince the > list (and maybe me too) otherwise.) (In fact I consider it > the reverse: for Sequences, The Order You See Is The Order You > Get, and the DRY violation occurs *because* you are using an > inappropriate container for your purpose (ordered slots). > > (2) BasicStruct isn't the relevant use case: you can always just > document that __slots__ is restricted to ordered finite iterables It had better be a repeatable one too; after all, iter(['a', 'b']) is an ordered finite iterable, but not a very good candidate for slots. But that's a whole other thread. :) > except string-like ones which use sequences of length 1 to > represent individual members, and perhaps trap some of the more > obvious mistakes (set and dict and maybe their subclasses). > > The use case you need to present is one where a non-Sequence (eg, > a set or dict) is the obvious representation for __slots__ in a > BasicStruct. Note that __slots__ allows nearly arbitrary finite > iterables of strings only because there's no reason not to, and it > simplifies the implementation. In Python practice, that is indeed > an invitation to experiment with concrete classes that use > unordered __slots__ for some reason. But there's no sense that > abstract classes that use __slots__ in an abstract way (as yours > does) must preserve that generality, and therefore no reason (yet) > for Python to help you preserve that generality. > > > Footnotes: > [1] According to the usual definition of "works fine in C": is no > more than twice as error-prone as other features of C. Given the > existence of pointers.... > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From stephen at xemacs.org Sat Nov 7 09:41:39 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 7 Nov 2015 23:41:39 +0900 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> Message-ID: <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> Andrew Barnert writes: > On Nov 6, 2015, at 21:00, Stephen J. Turnbull wrote: > > This ABC would be a tar pit of recriminations between class authors > > and class clients. > I think you've missed the point of his proposal. It doesn't matter > _what_ the order is That's precisely what I see as problematic, unless the author of the subclass is the same as the author of the code using the subclass. > But, whether the proposal is useful or not, it is sufficiently > specified. I wrote inaccurately, I guess. My contention is that it's more dangerous than useful as currently specified but that could be reversed with additional restrictions. > How would you define the comparison function for OrderedDict in a > way that makes any semantic sense? Or, for that matter, list? Again, bad nomenclature, sorry.[1] I should have used "index". For this application, I have in mind class Ordered: def index(self, key): return list(z for z in self).index(key) def compare(self, key1, key2): return self.index(key1) < self.index(key2) as the default implementation (except that for Amir's application it's .index() that's interesting, not .compare() -- .compare() could be omitted I guess, fn. [1] applies). The problem with list would be that it can have multiple entries of the same value that are not adjacent, of course, but then it wouldn't work for Amir's application anyway. This is part of what I have in mind when I say "underspecified". I don't think thought about how this collections.abc.Ordered should interact with objects that could be considered to represent multisets. Do you? > Surely OrderedDict and list are perfectly reasonable types for > __slots__, despite the fact that their comparison function doesn't > exist. If given the above renaming you still say it doesn't exist, I don't understand what you're trying to say. Footnotes: [1] As an excuse for the poor nomenclature, in my field I have to worry about (pre-) ordered structures for which an index function does not exist, thus "comparison function". From mistersheik at gmail.com Sat Nov 7 10:35:41 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 7 Nov 2015 07:35:41 -0800 (PST) Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> Message-ID: <61c756e9-ffbf-45bd-8eb4-024a6329dded@googlegroups.com> On Saturday, November 7, 2015 at 9:42:19 AM UTC-5, Stephen J. Turnbull wrote: > > Andrew Barnert writes: > > On Nov 6, 2015, at 21:00, Stephen J. Turnbull > wrote: > > > > This ABC would be a tar pit of recriminations between class authors > > > and class clients. > > > I think you've missed the point of his proposal. It doesn't matter > > _what_ the order is > > That's precisely what I see as problematic, unless the author of the > subclass is the same as the author of the code using the subclass. > > > But, whether the proposal is useful or not, it is sufficiently > > specified. > > I wrote inaccurately, I guess. My contention is that it's more > dangerous than useful as currently specified but that could be > reversed with additional restrictions. > It's not "dangerous"!! Talk about hyperbole. It's just a question of utility. Personally I would like to see a much richer collections.abc for tests like this. > > > How would you define the comparison function for OrderedDict in a > > way that makes any semantic sense? Or, for that matter, list? > > Again, bad nomenclature, sorry.[1] I should have used "index". For > this application, I have in mind > > class Ordered: > > def index(self, key): > return list(z for z in self).index(key) > > def compare(self, key1, key2): > return self.index(key1) < self.index(key2) > > as the default implementation (except that for Amir's application it's > .index() that's interesting, not .compare() -- .compare() could be > omitted I guess, fn. [1] applies). The problem with list would be > that it can have multiple entries of the same value that are not > adjacent, of course, but then it wouldn't work for Amir's application > anyway. This is part of what I have in mind when I say > "underspecified". > > I don't think thought about how this collections.abc.Ordered should > interact with objects that could be considered to represent multisets. > Do you? > Did you read the original post? All you need is a consistent order. It doesn't matter if the same element turns up twice in order to be Ordered. He can check uniqueness after the fact. > > > Surely OrderedDict and list are perfectly reasonable types for > > __slots__, despite the fact that their comparison function doesn't > > exist. > > If given the above renaming you still say it doesn't exist, I don't > understand what you're trying to say. > > > > Footnotes: > [1] As an excuse for the poor nomenclature, in my field I have to > worry about (pre-) ordered structures for which an index function does > not exist, thus "comparison function". > > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sat Nov 7 12:06:03 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 7 Nov 2015 19:06:03 +0200 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: References: Message-ID: On 07.11.15 00:11, Amir Rachum wrote: > I am suggesting the addition of a collections abstract base class called > "Ordered". Its meaning is that a collection's iteration order is part of > its API. The bulk of this mail describes a use case for this. The reason > I believe that such abstract base class is required is that there is no > way to test this behavior in a given class. An ordered collection has > the exact same interface as an unordered collection (e.g, dict and > OrderedDict), other than a _promise_ of the API that the order in which > this collection will be iterated has some sort of meaning (In > OrderedDict, it is the order in which keys were added to it.) > > > As examples, set, frozenset, dict and defaultdict should *not* be > considered as ordered. list, OrderedDict, deque and tuple should be > considered ordered. I just wanted to offer this idea. I have two use cases: 1. Human-readable output. If the collection is not ordered, it is preferable to sort it before output. pprint and testing frameworks will benefit from this. 2. Serialization. If the mapping is not ordered, we can serialize its content as dict (that can be more efficient), otherwise we have to serialize it as a sequence of key-value pairs. Actually we need only two functions: test if the collection is ordered, and a way to register the class as ordered (or unordered). From mistersheik at gmail.com Sat Nov 7 12:19:57 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 07 Nov 2015 17:19:57 +0000 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: References: Message-ID: FYI: 1. might not work since the elements in a dict may not be sortable. On Sat, Nov 7, 2015 at 12:06 PM Serhiy Storchaka wrote: > On 07.11.15 00:11, Amir Rachum wrote: > > I am suggesting the addition of a collections abstract base class called > > "Ordered". Its meaning is that a collection's iteration order is part of > > its API. The bulk of this mail describes a use case for this. The reason > > I believe that such abstract base class is required is that there is no > > way to test this behavior in a given class. An ordered collection has > > the exact same interface as an unordered collection (e.g, dict and > > OrderedDict), other than a _promise_ of the API that the order in which > > this collection will be iterated has some sort of meaning (In > > OrderedDict, it is the order in which keys were added to it.) > > > > > > As examples, set, frozenset, dict and defaultdict should *not* be > > considered as ordered. list, OrderedDict, deque and tuple should be > > considered ordered. > > I just wanted to offer this idea. I have two use cases: > > 1. Human-readable output. If the collection is not ordered, it is > preferable to sort it before output. pprint and testing frameworks will > benefit from this. > > 2. Serialization. If the mapping is not ordered, we can serialize its > content as dict (that can be more efficient), otherwise we have to > serialize it as a sequence of key-value pairs. > > Actually we need only two functions: test if the collection is ordered, > and a way to register the class as ordered (or unordered). > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/B1Pt76OtJi8/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Sat Nov 7 14:18:29 2015 From: mike at selik.org (Michael Selik) Date: Sat, 07 Nov 2015 19:18:29 +0000 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: <1277750399.598478.1446796147897.JavaMail.yahoo@mail.yahoo.com> References: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl> <1277750399.598478.1446796147897.JavaMail.yahoo@mail.yahoo.com> Message-ID: Sjoerd, glad we agree, happy to help. In case someone is still interested in making a patch, I'd like to clarify why it makes me nervous. Though Sjoerd led with a TL;DR that summarized his suggestion as a swap of isinstance for issubclass, a better summary would have been: "I prefer more specific exception types". The proposed solution was to enable custom exception types that conditionally insert themselves into the class hierarchy depending on data stored on a raised exception instance. Unfortunately, that solution requires changing an obscure error handling feature. As Sjoerd noted when he sent the first message in this thread, it is possible this would introduce a backwards-incompatibility. Perhaps no one in the standard library or major public projects would write such weird code as to get tripped up by the proposed change, but it's always possible that someone out there is using this feature for an important project. I think we've learned the lesson from the long slog to Python 3 adoption (and how Perl 6 harmed that community, and countless other examples) that backwards compatibility is of utmost importance. The current code pattern for a single try/except is to catch the general exception and have an if-condition in the except block.That's a frequent and readable practice. try: cursor.execute(query) except sqlite3.OperationalError as str(e): if 'syntax error' in str(e): print('Your SQL is bad') If that code or similar code appears many times, a good way to refactor with the current features of Python is to use a context manager. class SqlSyntaxErrorHandler: def __enter__(self): return self def __exit__(self, etyp, einstance, etraceback): if (issubclass(etyp, sqlite3.OperationalError) and 'syntax error' in str(einstance)): print('Your SQL is bad') with SqlSyntaxErrorHandler: cursor.execute(first_query) with SqlSyntaxErrorHandler: cursor.execute(second_query) While context managers may not be a beginner topic, they are typically easier to read and debug than using metaclasses for conditional runtime inheritance. Even if there would be some benefit to the metaclass style, it's not worth the risk of changing current error handling behavior. -- Michael On Fri, Nov 6, 2015, 2:49 AM Andrew Barnert via Python-ideas < python-ideas at python.org> wrote: > On Thursday, November 5, 2015 9:57 PM, "sjoerdjob at sjec.nl" < > sjoerdjob at sjec.nl> wrote: > > >> Andrew Barnert via Python-ideas writes: > >> > Sjoerd Job Postmus wrote: > >> > >> > > It might seem troublesome to change these locations to raise > >> > > actual instances instead of strings, but those might actually > >> > > just be hidden bugs just the same. > >> > >> No, I'm sure mostly they're not. They're working code that has > > been > >> there since the early days, perhaps a few from the 2.x days in > >> modules. Python is generally very conservative about changing working > >> code. > >> > >> > > Do you think anyone would object if I'd go over the code > > looking > >> > > for places where type(tstate->cur_value) != > > tstate->cur_type and > >> > > propose patches for these? > >> > >> I think, yes, though I don't know enough about it to object myself. > >> Code churn of this kind is generally frowned upon, unless you can show > >> that the risk of hidden bugs of the kind you refer to above is > >> substantially greater than the risk that *you* will introduce bugs > >> with your patches. > > > > After digging more into the code, I concur. It's quite easy to make some > > changes that break some stuff in this logic. > > Looking at this a bit further, I think there may be something you can do > that gets everything you actually wanted, without changes all over the > place and serious risks. > > Basically, it's just two pretty small changes in errors.c: > > * Change PyErr_GivenExceptionMatches so that if given a value that's an > instance of an exception type (PyExceptionInstance_Check(err) is true), > instead of just getting its class and then ignoring the value, it does a > standard isinstance check on the value. > > * Change PyErr_ExceptionMatches so that if there is an exception value, > and its type matches the exception type, it is passed to > PyErr_GivenExceptionMatches; otherwise, the exception type is passed. > > > So, all Python code that raises and handles exceptions will get the > benefit of isinstance checks. Any new C code that wants that benefit can > get it very easily (only raise with PyErr_SetObject or something that uses > it like PyErr_SetFromErrno; only handle with PyErr_GivenExceptionMatches). > Any existing C code that skips creating an instance will continue to do so, > and to get the same fast-path check used today, except for one extra call > to PyExceptionInstance_Check(tstate->curexc_value). > > Of course it's not just that small patch to errors.c. You also need tests > to exercise the new functionality. And a docs patch for the Exceptions > section of the C API. And also a docs patch to the Exceptions section in > the Reference. But no other code patches. > > And you'd definitely need to performance-test two things: First, that the > extra PyExceptionInstance_Check doesn't slow down fast-path handling (e.g., > in an empty for loops). Second, whatever performance tests your original > proposal would have needed, to make sure the isinstance check doesn't > noticeably slow down pure-Python code that handles a lot of exceptions. > > > I think the only C API docs change needed is to PyErr_ExceptionMatches. > However, a note explaining how to make sure to get and/or to bypass the > isinstance check as desired might be useful. > > For the reference docs, I think just this note at the end of the section: > > > Note: Exceptions raised and handled by Python code use the same > mechanism as the isinstance function for determining whether an exception > matches. However, exceptions raised or handled by the implementation (or > implementation-level extensions) may bypass that and check the type > directly. > > > > New in 3.6: In previous versions of Python, exception handling always > bypassed the isinstance function and checked the type directly. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Nov 7 16:29:04 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 7 Nov 2015 16:29:04 -0500 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: References: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl> <1277750399.598478.1446796147897.JavaMail.yahoo@mail.yahoo.com> Message-ID: On 11/7/2015 2:18 PM, Michael Selik wrote: > The current code pattern for a single try/except is to catch the general > exception and have an if-condition in the except block.That's a frequent > and readable practice. > > try: > cursor.execute(query) > except sqlite3.OperationalError as str(e): > if 'syntax error' in str(e): > print('Your SQL is bad') > > If that code or similar code appears many times, a good way to refactor > with the current features of Python is to use a context manager. > > class SqlSyntaxErrorHandler: > def __enter__(self): > return self > def __exit__(self, etyp, einstance, etraceback): > if (issubclass(etyp, sqlite3.OperationalError) > and 'syntax error' in str(einstance)): > print('Your SQL is bad') I like this example of how to use the error handling ability of __exit__ to set up an 'error handling context. This is much clearer than the examples in the pep. However, I checked and issubclass(None, whatever) does not work. True should be returned when the exception is handled. def __exit__(self, etyp, einstance, etraceback): if (etyp and issubclass(etyp, sqlite3.OperationalError) and 'syntax error' in str(einstance)): print('Your SQL is bad') return True > with SqlSyntaxErrorHandler: > cursor.execute(first_query) > > with SqlSyntaxErrorHandler: > cursor.execute(second_query) -- Terry Jan Reedy From abarnert at yahoo.com Sat Nov 7 16:41:09 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 7 Nov 2015 13:41:09 -0800 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> Message-ID: <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> On Nov 7, 2015, at 06:41, Stephen J. Turnbull wrote: > > Andrew Barnert writes: >> On Nov 6, 2015, at 21:00, Stephen J. Turnbull wrote: > >>> This ABC would be a tar pit of recriminations between class authors >>> and class clients. > >> I think you've missed the point of his proposal. It doesn't matter >> _what_ the order is > > That's precisely what I see as problematic, unless the author of the > subclass is the same as the author of the code using the subclass. > >> But, whether the proposal is useful or not, it is sufficiently >> specified. > > I wrote inaccurately, I guess. My contention is that it's more > dangerous than useful as currently specified but that could be > reversed with additional restrictions. Not very useful, sure?I argued that myself?but how is it dangerous? The only risk here is the risk that comes from adding more code, docs, and complexity to the stdlib. >> How would you define the comparison function for OrderedDict in a >> way that makes any semantic sense? Or, for that matter, list? > > Again, bad nomenclature, sorry.[1] I should have used "index". For > this application, I have in mind > > class Ordered: > > def index(self, key): > return list(z for z in self).index(key) > > def compare(self, key1, key2): > return self.index(key1) < self.index(key2) For, How is that at all useful? What code can you imagine that needs to compare elements from an arbitrary iterable, and would rather have a comparison that's linear in the length of the iterable than none at all? Second, why list(z for z in self) instead of just list(self) or [z for z in self]? It seems like you're deliberately trying to find the slowest and least straightforward way of implementing it? But, most importantly, if you want an index method, why aren't you just requiring Sequence, which has one? The notion of index implies the notion of random accessibility?which is exactly what Sequence adds on top of Sized Iterable Container. Of course there are types for which compare might exist but index might not (e.g., a range of reals, or a set that's only partially ordered), but they're not iterables. (Well, a partially ordered set could Iterate in partially-arbitrary order, but then it's not Ordered in the OP's sense.) His proposal really does specify exactly what he wants, it's dead simple, and it does all he asks of it. The only question is whether anyone needs it (and, even if they do, whether it needs to be in the stdlib). > as the default implementation (except that for Amir's application it's > .index() that's interesting, not .compare() -- .compare() could be > omitted I guess, fn. [1] applies). The problem with list would be > that it can have multiple entries of the same value that are not > adjacent, of course, but then it wouldn't work for Amir's application > anyway. This is part of what I have in mind when I say > "underspecified". Well, yes, what he _really_ requires here is an ordered, _unique_, finite, repeatable iterable. Adding a test for Ordered is clearly insufficient for that, and doesn't seem particularly useful. But again, that doesn't mean that Ordered is underspecified in any way; it just means he doesn't have a good use case for it. > I don't think thought about how this collections.abc.Ordered should > interact with objects that could be considered to represent multisets. > Do you? I'm not sure what you're asking here. But clearly list, OrderedCounter, and a SortedMultiSet type are all ordered, and can all be used to represent multisets in rather obvious ways. And the only way his proposed Ordered has to interact with them is that they all inherit from or get registered with the Ordered ABC (and aren't lying about it). Of course your proposal for some kind of sorting-related ABC that generates the sort order from indexes would have a problem with multisets, and especially with list as you mentioned above, but that has nothing to do with his proposal. >> Surely OrderedDict and list are perfectly reasonable types for >> __slots__, despite the fact that their comparison function doesn't >> exist. > > If given the above renaming you still say it doesn't exist, I don't > understand what you're trying to say. OK, they're perfectly reasonable types for __slots__, despite the fact that no useful, acceptably efficient, or in any way desirable comparison function exists. Better? > Footnotes: > [1] As an excuse for the poor nomenclature, in my field I have to > worry about (pre-) ordered structures for which an index function does > not exist, thus "comparison function". From mike at selik.org Sat Nov 7 17:28:49 2015 From: mike at selik.org (Michael Selik) Date: Sat, 07 Nov 2015 22:28:49 +0000 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: References: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl> <1277750399.598478.1446796147897.JavaMail.yahoo@mail.yahoo.com> Message-ID: Sorry about that. I also forgot to instantiate the class. Here's functioning code... class SqlSyntaxErrorHandler: def __enter__(self): return self def __exit__(self, etyp, einstance, etraceback): if (etyp is not None and issubclass(etyp, sqlite3.OperationalError) and 'syntax error' in str(einstance)): print('Your SQL is bad') return True with SqlSyntaxErrorHandler(): cursor.execute(query) On Sat, Nov 7, 2015 at 4:29 PM Terry Reedy wrote: > On 11/7/2015 2:18 PM, Michael Selik wrote: > > > The current code pattern for a single try/except is to catch the general > > exception and have an if-condition in the except block.That's a frequent > > and readable practice. > > > > try: > > cursor.execute(query) > > except sqlite3.OperationalError as str(e): > > if 'syntax error' in str(e): > > print('Your SQL is bad') > > > > If that code or similar code appears many times, a good way to refactor > > with the current features of Python is to use a context manager. > > > > class SqlSyntaxErrorHandler: > > def __enter__(self): > > return self > > def __exit__(self, etyp, einstance, etraceback): > > if (issubclass(etyp, sqlite3.OperationalError) > > and 'syntax error' in str(einstance)): > > print('Your SQL is bad') > > I like this example of how to use the error handling ability of __exit__ > to set up an 'error handling context. This is much clearer than the > examples in the pep. However, I checked and > issubclass(None, whatever) does not work. True should be returned when > the exception is handled. > > def __exit__(self, etyp, einstance, etraceback): > if (etyp and issubclass(etyp, sqlite3.OperationalError) > and 'syntax error' in str(einstance)): > print('Your SQL is bad') > return True > > > with SqlSyntaxErrorHandler: > > cursor.execute(first_query) > > > > with SqlSyntaxErrorHandler: > > cursor.execute(second_query) > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Sat Nov 7 17:44:08 2015 From: mike at selik.org (Michael Selik) Date: Sat, 07 Nov 2015 22:44:08 +0000 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> Message-ID: Hi Amir, Could you (or someone else) help me understand the use case for this base class? Rather than describing the use case, could you point me to somewhere in the standard library or a public project where it would clarify the code? I currently share Andrew's concern -- knowing a thing is ordered does not help us understand *how* it's ordered. Therefore, despite an `Ordered` base class, we would still need to rely on error messages, tests, and documentation. As he said, calling this "dangerous" is not hyperbole but a proper description of how any change to the standard library will increase the maintenance burden, may reduce usability/learnability, and may have unpredictable consequences. -- Michael On Sat, Nov 7, 2015 at 4:41 PM Andrew Barnert via Python-ideas < python-ideas at python.org> wrote: > On Nov 7, 2015, at 06:41, Stephen J. Turnbull wrote: > > > > Andrew Barnert writes: > >> On Nov 6, 2015, at 21:00, Stephen J. Turnbull > wrote: > > > >>> This ABC would be a tar pit of recriminations between class authors > >>> and class clients. > > > >> I think you've missed the point of his proposal. It doesn't matter > >> _what_ the order is > > > > That's precisely what I see as problematic, unless the author of the > > subclass is the same as the author of the code using the subclass. > > > >> But, whether the proposal is useful or not, it is sufficiently > >> specified. > > > > I wrote inaccurately, I guess. My contention is that it's more > > dangerous than useful as currently specified but that could be > > reversed with additional restrictions. > > Not very useful, sure?I argued that myself?but how is it dangerous? The > only risk here is the risk that comes from adding more code, docs, and > complexity to the stdlib. > > >> How would you define the comparison function for OrderedDict in a > >> way that makes any semantic sense? Or, for that matter, list? > > > > Again, bad nomenclature, sorry.[1] I should have used "index". For > > this application, I have in mind > > > > class Ordered: > > > > def index(self, key): > > return list(z for z in self).index(key) > > > > def compare(self, key1, key2): > > return self.index(key1) < self.index(key2) > > For, How is that at all useful? What code can you imagine that needs to > compare elements from an arbitrary iterable, and would rather have a > comparison that's linear in the length of the iterable than none at all? > > Second, why list(z for z in self) instead of just list(self) or [z for z > in self]? It seems like you're deliberately trying to find the slowest and > least straightforward way of implementing it? > > But, most importantly, if you want an index method, why aren't you just > requiring Sequence, which has one? The notion of index implies the notion > of random accessibility?which is exactly what Sequence adds on top of Sized > Iterable Container. > > Of course there are types for which compare might exist but index might > not (e.g., a range of reals, or a set that's only partially ordered), but > they're not iterables. (Well, a partially ordered set could Iterate in > partially-arbitrary order, but then it's not Ordered in the OP's sense.) > > His proposal really does specify exactly what he wants, it's dead simple, > and it does all he asks of it. The only question is whether anyone needs it > (and, even if they do, whether it needs to be in the stdlib). > > > as the default implementation (except that for Amir's application it's > > .index() that's interesting, not .compare() -- .compare() could be > > omitted I guess, fn. [1] applies). The problem with list would be > > that it can have multiple entries of the same value that are not > > adjacent, of course, but then it wouldn't work for Amir's application > > anyway. This is part of what I have in mind when I say > > "underspecified". > > Well, yes, what he _really_ requires here is an ordered, _unique_, finite, > repeatable iterable. Adding a test for Ordered is clearly insufficient for > that, and doesn't seem particularly useful. But again, that doesn't mean > that Ordered is underspecified in any way; it just means he doesn't have a > good use case for it. > > > I don't think thought about how this collections.abc.Ordered should > > interact with objects that could be considered to represent multisets. > > Do you? > > I'm not sure what you're asking here. But clearly list, OrderedCounter, > and a SortedMultiSet type are all ordered, and can all be used to represent > multisets in rather obvious ways. And the only way his proposed Ordered has > to interact with them is that they all inherit from or get registered with > the Ordered ABC (and aren't lying about it). > > Of course your proposal for some kind of sorting-related ABC that > generates the sort order from indexes would have a problem with multisets, > and especially with list as you mentioned above, but that has nothing to do > with his proposal. > > >> Surely OrderedDict and list are perfectly reasonable types for > >> __slots__, despite the fact that their comparison function doesn't > >> exist. > > > > If given the above renaming you still say it doesn't exist, I don't > > understand what you're trying to say. > > OK, they're perfectly reasonable types for __slots__, despite the fact > that no useful, acceptably efficient, or in any way desirable comparison > function exists. Better? > > > Footnotes: > > [1] As an excuse for the poor nomenclature, in my field I have to > > worry about (pre-) ordered structures for which an index function does > > not exist, thus "comparison function". > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sun Nov 8 07:34:37 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 8 Nov 2015 21:34:37 +0900 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> Message-ID: <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp> Andrew Barnert writes: > Not very useful, sure?I argued that myself?but how is it > dangerous? The only risk here is the risk that comes from adding > more code, docs, and complexity to the stdlib. This proposal requires that users not only read the docs in the stdlib, but also the docs of each class, perhaps each object, using the functionality. Asking users to read docs thoroughly is a recipe for bugs, especially since the order used is likely to be documented as "the natural order", and people will have different ideas about that. This is the "complication" the Zen warns about, as well as being "implicit". You may not consider that dangerous, but I do. And on top of that, not only doesn't it provide any guarantees, you've pretty much convinced me it's impossible to do so. If that's not an attractive nuisance, what is? > But, most importantly, if you want an index method, why aren't you > just requiring Sequence, which has one? Because "ordered sets" and "ordered dicts" aren't Sequences. But don't ask me, ask the OP. As I wrote earlier, no use case for set- or dict-valued __slots__ has been given. I have nothing invested in anything I wrote except that. I'm happy to concede that everything else I wrote was boneheaded, retract it, and apologize for wasting your time. From guido at python.org Sun Nov 8 13:45:10 2015 From: guido at python.org (Guido van Rossum) Date: Sun, 8 Nov 2015 10:45:10 -0800 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp> References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp> Message-ID: I'm warming up slightly to the idea of this ABC. I've re-read Amir's post, and if I found myself in his situation I would have used a combination of documenting that __slots__ needs to be ordered and at runtime only checking for a few common definitely-bad cases. E.g. for exactly set or dict (not subclasses, since OrderedDict inherits from dict) would cover most of the common mistakes: most people will use a literal in their __slots__ definition so we just need to watch out for the common literals. Those users who are sophisticated enough to use some advanced mapping type in their __slots__ should just be expected to deal with the consequences. But the Ordered ABC, while not essential (unless you're a perfectionist, in which case you're in the wrong language community anyways :-) still fills a niche. The Ordered ABC should have no additional methods, and no default implementations. I think it should apply to collections but not to iterators. It should apply at the level of the read-only interface. Sequence is always Ordered. Mapping and Set are not by default, but can have it added. OrderedDict is the prime (maybe only) example -- it's a MutableMapping and Ordered. We might eventually get an OrderedSet. A sorted set or mapping (e.g. one implemented using some kind of tree) should also be considered Ordered, even though otherwise this is a totally different topic -- while some other languages or branches of math use "order" to refer to sorting (e.g. "partial ordering"), in Python we make a distinction: on the one hand there's sorted() and list.sort(), and on the other hand there's OrderedDict. So, I think that the Ordered ABC proposed here is totally well-defined and mildly useful. It may be somewhat confusing (because many people when they first encounter the term "ordered" they think it's about sorting -- witness some posts in this thread). The use cases are not very important. I guess I'm -0 on adding it -- if better use cases than Amir's are developed I might change to +0. (I don't care much about Serhiy's use cases.) Sorry for the rambling. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From amir at rachum.com Sun Nov 8 14:06:03 2015 From: amir at rachum.com (Amir Rachum) Date: Sun, 8 Nov 2015 21:06:03 +0200 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp> Message-ID: I would like to expand on my original use case, which I hope will provide a more persuading argument. As part of BasicStruct I intend to allow the use of mapping types as __slots__, with the semantics of default values. So it'll look something like this: class Point(BasicStruct): __slots__ = {'x': 5, 'y': 7} My fear is that someone will just use a dict literal as stated above and try to create an object with positional arguments: Point(0) The problem is that iteration order over a dict is not guaranteed, so this might initialize Point(x=0, y=7) like the use intends, or Point(x=5, y=0). So in this case I want to disable positional arguments. However, positional arguments for initializing struct are pretty convenient, so I would like to make that available as far as possible. An implementation using Ordered might look like this: class BasicStruct(object): """Class for holding struct-like objects.""" __slots__ = () # should be extended by deriving classes def __init__(self, *args, **kwargs): default_values = isinstance(self.__slots__, Mapping) ordered = isinstance(self.__slots__, Ordered) if args and not ordered: raise ValueError("Can't pass non-keyword arguments to {}, since " "__slots__ was declared with an unordered " "iterable.".format(self.__class__.__name__)) arg_pairs = zip(self.__slots__, args) for key, value in chain(arg_pairs, six.iteritems(kwargs)): setattr(self, key, value) for key in self.__slots__: if not hasattr(self, key): default_value = None if default_values: default_value = self.__slots__[key] setattr(self, key, default_value) This will allow users who are interested in positional argument initialization to use an Ordered mapping (such as OrderedDict, or a custom collection). I would like to emphasize again that the most compelling reason for me that this should be part of the stdlib is that there is no workaround for this that also allows users to use ordered custom collections. There is no way to check for "promised ordering" in any functional manner. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Sun Nov 8 14:10:01 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Sun, 08 Nov 2015 19:10:01 +0000 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp> Message-ID: On Sun, Nov 8, 2015 at 2:06 PM Amir Rachum wrote: > I would like to expand on my original use case, which I hope will provide > a more persuading argument. > As part of BasicStruct I intend to allow the use of mapping types as > __slots__, with the semantics of default values. > So it'll look something like this: > > class Point(BasicStruct): > __slots__ = {'x': 5, 'y': 7} > > My fear is that someone will just use a dict literal as stated above and > try to create an object with positional arguments: > > Point(0) > > The problem is that iteration order over a dict is not guaranteed, so this > might initialize Point(x=0, y=7) like the use intends, or Point(x=5, y=0). > So in this case I want to disable positional arguments. However, positional > arguments for initializing struct are pretty convenient, so I would like to > make that available as far as possible. An implementation using Ordered > might look like this: > > class BasicStruct(object): > """Class for holding struct-like objects.""" > > __slots__ = () # should be extended by deriving classes > > def __init__(self, *args, **kwargs): > default_values = isinstance(self.__slots__, Mapping) > ordered = isinstance(self.__slots__, Ordered) > > if args and not ordered: > raise ValueError("Can't pass non-keyword arguments to {}, > since " > "__slots__ was declared with an unordered > " > > "iterable.".format(self.__class__.__name__)) > > arg_pairs = zip(self.__slots__, args) > for key, value in chain(arg_pairs, six.iteritems(kwargs)): > setattr(self, key, value) > > for key in self.__slots__: > if not hasattr(self, key): > default_value = None > if default_values: > default_value = self.__slots__[key] > setattr(self, key, default_value) > > > This will allow users who are interested in positional argument > initialization to use an Ordered mapping (such as OrderedDict, or a custom > collection). > I would like to emphasize again that the most compelling reason for me > that this should be part of the stdlib is that there is no workaround for > this that also allows users to use ordered custom collections. There is no > way to check for "promised ordering" in any functional manner. > Did you know that you can register your custom collection with an abc? > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/B1Pt76OtJi8/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/B1Pt76OtJi8/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sjoerdjob at sjec.nl Sun Nov 8 16:12:31 2015 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Sun, 8 Nov 2015 22:12:31 +0100 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp> Message-ID: > On 8 Nov 2015, at 20:06, Amir Rachum wrote: > > I would like to expand on my original use case, which I hope will provide a more persuading argument. > As part of BasicStruct I intend to allow the use of mapping types as __slots__, with the semantics of default values. > So it'll look something like this: > > class Point(BasicStruct): > __slots__ = {'x': 5, 'y': 7} So instead they'll write __slots__ = OrderedDict({'x': 5, 'y': 7}) Causing the same issues? > > My fear is that someone will just use a dict literal as stated above and try to create an object with positional arguments: > > Point(0) > > The problem is that iteration order over a dict is not guaranteed, so this might initialize Point(x=0, y=7) like the use intends, or Point(x=5, y=0). So in this case I want to disable positional arguments. However, positional arguments for initializing struct are pretty convenient, so I would like to make that available as far as possible. An implementation using Ordered might look like this: > > class BasicStruct(object): > """Class for holding struct-like objects.""" > > __slots__ = () # should be extended by deriving classes > > def __init__(self, *args, **kwargs): > default_values = isinstance(self.__slots__, Mapping) > ordered = isinstance(self.__slots__, Ordered) > > if args and not ordered: > raise ValueError("Can't pass non-keyword arguments to {}, since " > "__slots__ was declared with an unordered " > "iterable.".format(self.__class__.__name__)) > > arg_pairs = zip(self.__slots__, args) > for key, value in chain(arg_pairs, six.iteritems(kwargs)): > setattr(self, key, value) > > for key in self.__slots__: > if not hasattr(self, key): > default_value = None > if default_values: > default_value = self.__slots__[key] > setattr(self, key, default_value) > > > This will allow users who are interested in positional argument initialization to use an Ordered mapping (such as OrderedDict, or a custom collection). > I would like to emphasize again that the most compelling reason for me that this should be part of the stdlib is that there is no workaround for this that also allows users to use ordered custom collections. There is no way to check for "promised ordering" in any functional manner. Besides the fact that you can create that base-class yourself, and register OrderedDict as an instance of it already, I think there's a better solution to your problem (not quite tested, but should work): 1. Create a Field class which maintains a creation-counter to be able to query ordering. 2. In the __init__ check against the creation ordering. Added bonus: you can choose to have some attributes have a default value, and others not having a default value. If you're willing to do some metaclass trickery, you could even end up with a syntax like: class Point(Struct): x = Field(5) y = Field(7) z = Field() # we force the user to supply the z value, as it has no default. And generate the __slots__ from that. Maybe look at Django's ModelBase implementation, and how it uses metaclasses for that? Even more added bonus would be defining special Field sub-classes which also check the type/value of the supplied values (NumericField raising ValueError when you pass it the string "foo"). Not really sure what the added benefit of an Ordered base-class would be here. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sun Nov 8 17:10:06 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 9 Nov 2015 00:10:06 +0200 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp> Message-ID: On 08.11.15 23:12, Sjoerd Job Postmus wrote: > On 8 Nov 2015, at 20:06, Amir Rachum > wrote: >> As part of BasicStruct I intend to allow the use of mapping types as >> __slots__, with the semantics of default values.. >> So it'll look something like this: >> >> class Point(BasicStruct): >> __slots__ = {'x': 5, 'y': 7} > > So instead they'll write > __slots__ = OrderedDict({'x': 5, 'y': 7}) > Causing the same issues? Perhaps OrderedDict should reject unordered sources. Hey, here is yet one use case! From abarnert at yahoo.com Sun Nov 8 19:42:05 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 8 Nov 2015 16:42:05 -0800 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp> Message-ID: <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com> On Nov 8, 2015, at 14:10, Serhiy Storchaka wrote: > >> On 08.11.15 23:12, Sjoerd Job Postmus wrote: >> On 8 Nov 2015, at 20:06, Amir Rachum > > wrote: >>> As part of BasicStruct I intend to allow the use of mapping types as >>> __slots__, with the semantics of default values.. >>> So it'll look something like this: >>> >>> class Point(BasicStruct): >>> __slots__ = {'x': 5, 'y': 7} >> >> So instead they'll write >> __slots__ = OrderedDict({'x': 5, 'y': 7}) >> Causing the same issues? > > Perhaps OrderedDict should reject unordered sources. Hey, here is yet one use case! I've maintained code that does this: self.headers = OrderedDict(headers) self.origheaders = len(headers) ? so it can later do this: altheaders = list(self.headers.items())[self.origheaders:] Not a great design, but one that exists in the wild, and would be broken by OrderedDict not allowing a dict as an argument. Also, this wouldn't allow creating an OrderedDict from an empty dict (which seems far less stupid, but I didn't lead with it because I can't remember seeing it in real code). From guido at python.org Sun Nov 8 23:28:31 2015 From: guido at python.org (Guido van Rossum) Date: Sun, 8 Nov 2015 20:28:31 -0800 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com> References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp> <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com> Message-ID: So if OrderedDict had always rejected construction from a dict, how would you have written this? On Sunday, November 8, 2015, Andrew Barnert via Python-ideas < python-ideas at python.org> wrote: > On Nov 8, 2015, at 14:10, Serhiy Storchaka > wrote: > > > >> On 08.11.15 23:12, Sjoerd Job Postmus wrote: > >> On 8 Nov 2015, at 20:06, Amir Rachum > >> >> wrote: > >>> As part of BasicStruct I intend to allow the use of mapping types as > >>> __slots__, with the semantics of default values.. > >>> So it'll look something like this: > >>> > >>> class Point(BasicStruct): > >>> __slots__ = {'x': 5, 'y': 7} > >> > >> So instead they'll write > >> __slots__ = OrderedDict({'x': 5, 'y': 7}) > >> Causing the same issues? > > > > Perhaps OrderedDict should reject unordered sources. Hey, here is yet > one use case! > > I've maintained code that does this: > > self.headers = OrderedDict(headers) > self.origheaders = len(headers) > > ? so it can later do this: > > altheaders = list(self.headers.items())[self.origheaders:] > > Not a great design, but one that exists in the wild, and would be broken > by OrderedDict not allowing a dict as an argument. > > Also, this wouldn't allow creating an OrderedDict from an empty dict > (which seems far less stupid, but I didn't lead with it because I can't > remember seeing it in real code). > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From sjoerdjob at sjec.nl Mon Nov 9 00:30:51 2015 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Mon, 9 Nov 2015 06:30:51 +0100 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: References: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl> <1277750399.598478.1446796147897.JavaMail.yahoo@mail.yahoo.com> Message-ID: Hi again everybody, So I've had a couple of days extra to think about it, and yes: it might still break some things, but I expect extremely little. After the discussions here, I came to think it would require a lot of changes to the codebase, and decided to let it rest. Last night, it dawned to me that there might be a fix with very little code-changes needed: * add an extra operator `PyCmp_EXC_VAL_MATCH` * in the compiler, change `DUP_TOP` to `DUP_TOP_TWO` just above the generation of the compare. I don't have the exact location, as I'm typing on my phone right now. * create an extra function PyErr_GivenExceptionValueMatches which calls PyObject_isInstance * call that from ceval for the `PyCmp_EXC_VAL_MATCH. * most important: update the docs. A little bit of code duplication, but not that much. As for the scary part. * existing code: Because this would only introduce changes for Python code, existing C-extensions would not be affected. Python code would, but I think the fallout would be very little, as the only noticeable changes are when a class overrides its __instancecheck__, which I believe is too negligible a corner-case. * performance: with the changes already proposed for the moving towards __subclasscheck__ from the current behaviour, there are some optimizations included which should also benefit this proposal. Exact measurements are of course needed. One of the changes I do see needed in Python-code is having the unit-testing libraries change their 'assertRaises' code to reflect the new behavior. I agree this might cause a backward-compatibility issue, however after some thinking about it, I don't think it will affect more than 1 percent of already existing code, probably even way less. I don't have any numbers to support that feeling, though. As far as an important project: if you're depending on obscure features, you should probably be ready to expect changes there. Yes, the current behavior is precisely documented, but in nearly all of the cases the actual implementation and the proposed change agree, unless one did magic. I made the changes I listed above and ran `make; make test`, and got only 1 error due to socket.socket running out of memory or something. I'll re-run the tests on a clean checkout as well, but for now I think it's unrelated. Regards, Sjoerd Job Postmus > On 7 Nov 2015, at 20:18, Michael Selik wrote: > > Sjoerd, glad we agree, happy to help. > > In case someone is still interested in making a patch, I'd like to clarify why it makes me nervous. > > Though Sjoerd led with a TL;DR that summarized his suggestion as a swap of isinstance for issubclass, a better summary would have been: "I prefer more specific exception types". The proposed solution was to enable custom exception types that conditionally insert themselves into the class hierarchy depending on data stored on a raised exception instance. > > Unfortunately, that solution requires changing an obscure error handling feature. As Sjoerd noted when he sent the first message in this thread, it is possible this would introduce a backwards-incompatibility. Perhaps no one in the standard library or major public projects would write such weird code as to get tripped up by the proposed change, but it's always possible that someone out there is using this feature for an important project. I think we've learned the lesson from the long slog to Python 3 adoption (and how Perl 6 harmed that community, and countless other examples) that backwards compatibility is of utmost importance. > > The current code pattern for a single try/except is to catch the general exception and have an if-condition in the except block.That's a frequent and readable practice. > > try: > cursor.execute(query) > except sqlite3.OperationalError as str(e): > if 'syntax error' in str(e): > print('Your SQL is bad') > > If that code or similar code appears many times, a good way to refactor with the current features of Python is to use a context manager. > > class SqlSyntaxErrorHandler: > def __enter__(self): > return self > def __exit__(self, etyp, einstance, etraceback): > if (issubclass(etyp, sqlite3.OperationalError) > and 'syntax error' in str(einstance)): > print('Your SQL is bad') > > with SqlSyntaxErrorHandler: > cursor.execute(first_query) > > with SqlSyntaxErrorHandler: > cursor.execute(second_query) > > While context managers may not be a beginner topic, they are typically easier to read and debug than using metaclasses for conditional runtime inheritance. Even if there would be some benefit to the metaclass style, it's not worth the risk of changing current error handling behavior. > > -- Michael > > >> On Fri, Nov 6, 2015, 2:49 AM Andrew Barnert via Python-ideas wrote: >> On Thursday, November 5, 2015 9:57 PM, "sjoerdjob at sjec.nl" wrote: >> >> >> Andrew Barnert via Python-ideas writes: >> >> > Sjoerd Job Postmus wrote: >> >> >> >> > > It might seem troublesome to change these locations to raise >> >> > > actual instances instead of strings, but those might actually >> >> > > just be hidden bugs just the same. >> >> >> >> No, I'm sure mostly they're not. They're working code that has >> > been >> >> there since the early days, perhaps a few from the 2.x days in >> >> modules. Python is generally very conservative about changing working >> >> code. >> >> >> >> > > Do you think anyone would object if I'd go over the code >> > looking >> >> > > for places where type(tstate->cur_value) != >> > tstate->cur_type and >> >> > > propose patches for these? >> >> >> >> I think, yes, though I don't know enough about it to object myself. >> >> Code churn of this kind is generally frowned upon, unless you can show >> >> that the risk of hidden bugs of the kind you refer to above is >> >> substantially greater than the risk that *you* will introduce bugs >> >> with your patches. >> > >> > After digging more into the code, I concur. It's quite easy to make some >> > changes that break some stuff in this logic. >> >> Looking at this a bit further, I think there may be something you can do that gets everything you actually wanted, without changes all over the place and serious risks. >> >> Basically, it's just two pretty small changes in errors.c: >> >> * Change PyErr_GivenExceptionMatches so that if given a value that's an instance of an exception type (PyExceptionInstance_Check(err) is true), instead of just getting its class and then ignoring the value, it does a standard isinstance check on the value. >> >> * Change PyErr_ExceptionMatches so that if there is an exception value, and its type matches the exception type, it is passed to PyErr_GivenExceptionMatches; otherwise, the exception type is passed. >> >> >> So, all Python code that raises and handles exceptions will get the benefit of isinstance checks. Any new C code that wants that benefit can get it very easily (only raise with PyErr_SetObject or something that uses it like PyErr_SetFromErrno; only handle with PyErr_GivenExceptionMatches). Any existing C code that skips creating an instance will continue to do so, and to get the same fast-path check used today, except for one extra call to PyExceptionInstance_Check(tstate->curexc_value). >> >> Of course it's not just that small patch to errors.c. You also need tests to exercise the new functionality. And a docs patch for the Exceptions section of the C API. And also a docs patch to the Exceptions section in the Reference. But no other code patches. >> >> And you'd definitely need to performance-test two things: First, that the extra PyExceptionInstance_Check doesn't slow down fast-path handling (e.g., in an empty for loops). Second, whatever performance tests your original proposal would have needed, to make sure the isinstance check doesn't noticeably slow down pure-Python code that handles a lot of exceptions. >> >> >> I think the only C API docs change needed is to PyErr_ExceptionMatches. However, a note explaining how to make sure to get and/or to bypass the isinstance check as desired might be useful. >> >> For the reference docs, I think just this note at the end of the section: >> >> > Note: Exceptions raised and handled by Python code use the same mechanism as the isinstance function for determining whether an exception matches. However, exceptions raised or handled by the implementation (or implementation-level extensions) may bypass that and check the type directly. >> >> >> > New in 3.6: In previous versions of Python, exception handling always bypassed the isinstance function and checked the type directly. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sjoerdjob at sjec.nl Mon Nov 9 00:41:25 2015 From: sjoerdjob at sjec.nl (sjoerdjob at sjec.nl) Date: Mon, 9 Nov 2015 06:41:25 +0100 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: References: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl> <1277750399.598478.1446796147897.JavaMail.yahoo@mail.yahoo.com> Message-ID: <6f22ce0f98ae5b4f7a290343bad92564.squirrel@mail.sjec.nl> > I made the changes I listed above and ran `make; make test`, and got only > 1 error due to socket.socket running out of memory or something. I'll > re-run the tests on a clean checkout as well, but for now I think it's > unrelated. After running the tests against a clean checkout (without my changes): I get the same errors from test_socket. So it's unrelated. > Regards, > Sjoerd Job Postmus > >> On 7 Nov 2015, at 20:18, Michael Selik wrote: >> >> Sjoerd, glad we agree, happy to help. >> >> In case someone is still interested in making a patch, I'd like to >> clarify why it makes me nervous. >> >> Though Sjoerd led with a TL;DR that summarized his suggestion as a swap >> of isinstance for issubclass, a better summary would have been: "I >> prefer more specific exception types". The proposed solution was to >> enable custom exception types that conditionally insert themselves into >> the class hierarchy depending on data stored on a raised exception >> instance. >> >> Unfortunately, that solution requires changing an obscure error handling >> feature. As Sjoerd noted when he sent the first message in this thread, >> it is possible this would introduce a backwards-incompatibility. Perhaps >> no one in the standard library or major public projects would write such >> weird code as to get tripped up by the proposed change, but it's always >> possible that someone out there is using this feature for an important >> project. I think we've learned the lesson from the long slog to Python 3 >> adoption (and how Perl 6 harmed that community, and countless other >> examples) that backwards compatibility is of utmost importance. >> >> The current code pattern for a single try/except is to catch the general >> exception and have an if-condition in the except block.That's a frequent >> and readable practice. >> >> try: >> cursor.execute(query) >> except sqlite3.OperationalError as str(e): >> if 'syntax error' in str(e): >> print('Your SQL is bad') >> >> If that code or similar code appears many times, a good way to refactor >> with the current features of Python is to use a context manager. >> >> class SqlSyntaxErrorHandler: >> def __enter__(self): >> return self >> def __exit__(self, etyp, einstance, etraceback): >> if (issubclass(etyp, sqlite3.OperationalError) >> and 'syntax error' in str(einstance)): >> print('Your SQL is bad') >> >> with SqlSyntaxErrorHandler: >> cursor.execute(first_query) >> >> with SqlSyntaxErrorHandler: >> cursor.execute(second_query) >> >> While context managers may not be a beginner topic, they are typically >> easier to read and debug than using metaclasses for conditional runtime >> inheritance. Even if there would be some benefit to the metaclass style, >> it's not worth the risk of changing current error handling behavior. >> >> -- Michael >> >> >>> On Fri, Nov 6, 2015, 2:49 AM Andrew Barnert via Python-ideas >>> wrote: >>> On Thursday, November 5, 2015 9:57 PM, "sjoerdjob at sjec.nl" >>> wrote: >>> >>> >> Andrew Barnert via Python-ideas writes: >>> >> > Sjoerd Job Postmus wrote: >>> >> >>> >> > > It might seem troublesome to change these locations to raise >>> >> > > actual instances instead of strings, but those might actually >>> >> > > just be hidden bugs just the same. >>> >> >>> >> No, I'm sure mostly they're not. They're working code that has >>> > been >>> >> there since the early days, perhaps a few from the 2.x days in >>> >> modules. Python is generally very conservative about changing >>> working >>> >> code. >>> >> >>> >> > > Do you think anyone would object if I'd go over the code >>> > looking >>> >> > > for places where type(tstate->cur_value) != >>> > tstate->cur_type and >>> >> > > propose patches for these? >>> >> >>> >> I think, yes, though I don't know enough about it to object myself. >>> >> Code churn of this kind is generally frowned upon, unless you can >>> show >>> >> that the risk of hidden bugs of the kind you refer to above is >>> >> substantially greater than the risk that *you* will introduce bugs >>> >> with your patches. >>> > >>> > After digging more into the code, I concur. It's quite easy to make >>> some >>> > changes that break some stuff in this logic. >>> >>> Looking at this a bit further, I think there may be something you can >>> do that gets everything you actually wanted, without changes all over >>> the place and serious risks. >>> >>> Basically, it's just two pretty small changes in errors.c: >>> >>> * Change PyErr_GivenExceptionMatches so that if given a value that's an >>> instance of an exception type (PyExceptionInstance_Check(err) is true), >>> instead of just getting its class and then ignoring the value, it does >>> a standard isinstance check on the value. >>> >>> * Change PyErr_ExceptionMatches so that if there is an exception value, >>> and its type matches the exception type, it is passed to >>> PyErr_GivenExceptionMatches; otherwise, the exception type is passed. >>> >>> >>> So, all Python code that raises and handles exceptions will get the >>> benefit of isinstance checks. Any new C code that wants that benefit >>> can get it very easily (only raise with PyErr_SetObject or something >>> that uses it like PyErr_SetFromErrno; only handle with >>> PyErr_GivenExceptionMatches). Any existing C code that skips creating >>> an instance will continue to do so, and to get the same fast-path check >>> used today, except for one extra call to >>> PyExceptionInstance_Check(tstate->curexc_value). >>> >>> Of course it's not just that small patch to errors.c. You also need >>> tests to exercise the new functionality. And a docs patch for the >>> Exceptions section of the C API. And also a docs patch to the >>> Exceptions section in the Reference. But no other code patches. >>> >>> And you'd definitely need to performance-test two things: First, that >>> the extra PyExceptionInstance_Check doesn't slow down fast-path >>> handling (e.g., in an empty for loops). Second, whatever performance >>> tests your original proposal would have needed, to make sure the >>> isinstance check doesn't noticeably slow down pure-Python code that >>> handles a lot of exceptions. >>> >>> >>> I think the only C API docs change needed is to PyErr_ExceptionMatches. >>> However, a note explaining how to make sure to get and/or to bypass the >>> isinstance check as desired might be useful. >>> >>> For the reference docs, I think just this note at the end of the >>> section: >>> >>> > Note: Exceptions raised and handled by Python code use the same >>> mechanism as the isinstance function for determining whether an >>> exception matches. However, exceptions raised or handled by the >>> implementation (or implementation-level extensions) may bypass that >>> and check the type directly. >>> >>> >>> > New in 3.6: In previous versions of Python, exception handling always >>> bypassed the isinstance function and checked the type directly. >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ram at rachum.com Mon Nov 9 03:03:18 2015 From: ram at rachum.com (Ram Rachum) Date: Mon, 9 Nov 2015 10:03:18 +0200 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp> <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com> Message-ID: I'm not Andrew, but I'm guessing simply writing `OrderedDict(headers.items())`. On Mon, Nov 9, 2015 at 6:28 AM, Guido van Rossum wrote: > So if OrderedDict had always rejected construction from a dict, how would > you have written this? > > > On Sunday, November 8, 2015, Andrew Barnert via Python-ideas < > python-ideas at python.org> wrote: > >> On Nov 8, 2015, at 14:10, Serhiy Storchaka wrote: >> > >> >> On 08.11.15 23:12, Sjoerd Job Postmus wrote: >> >> On 8 Nov 2015, at 20:06, Amir Rachum > >> > wrote: >> >>> As part of BasicStruct I intend to allow the use of mapping types as >> >>> __slots__, with the semantics of default values.. >> >>> So it'll look something like this: >> >>> >> >>> class Point(BasicStruct): >> >>> __slots__ = {'x': 5, 'y': 7} >> >> >> >> So instead they'll write >> >> __slots__ = OrderedDict({'x': 5, 'y': 7}) >> >> Causing the same issues? >> > >> > Perhaps OrderedDict should reject unordered sources. Hey, here is yet >> one use case! >> >> I've maintained code that does this: >> >> self.headers = OrderedDict(headers) >> self.origheaders = len(headers) >> >> ? so it can later do this: >> >> altheaders = list(self.headers.items())[self.origheaders:] >> >> Not a great design, but one that exists in the wild, and would be broken >> by OrderedDict not allowing a dict as an argument. >> >> Also, this wouldn't allow creating an OrderedDict from an empty dict >> (which seems far less stupid, but I didn't lead with it because I can't >> remember seeing it in real code). >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido (mobile) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 9 11:29:21 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 9 Nov 2015 08:29:21 -0800 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp> <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com> Message-ID: Well, that would still defeat the purpose, wouldn't it? The items are no more ordered than the headers dict itself. Also, items() doesn't return a sequence -- it's an ItemsView (which inherits from Set) and presumably it's not Ordered. I guess my question is not so much how to prevent getting an exception -- I'm trying to tease out what the right order for the headers would be. Or perhaps I'm just trying to understand what the code is doing (the snippet shown mostly looks like bad code to me). On Mon, Nov 9, 2015 at 12:03 AM, Ram Rachum wrote: > I'm not Andrew, but I'm guessing simply writing > `OrderedDict(headers.items())`. > > On Mon, Nov 9, 2015 at 6:28 AM, Guido van Rossum wrote: > >> So if OrderedDict had always rejected construction from a dict, how would >> you have written this? >> >> >> On Sunday, November 8, 2015, Andrew Barnert via Python-ideas < >> python-ideas at python.org> wrote: >> >>> On Nov 8, 2015, at 14:10, Serhiy Storchaka wrote: >>> > >>> >> On 08.11.15 23:12, Sjoerd Job Postmus wrote: >>> >> On 8 Nov 2015, at 20:06, Amir Rachum >> >> > wrote: >>> >>> As part of BasicStruct I intend to allow the use of mapping types as >>> >>> __slots__, with the semantics of default values.. >>> >>> So it'll look something like this: >>> >>> >>> >>> class Point(BasicStruct): >>> >>> __slots__ = {'x': 5, 'y': 7} >>> >> >>> >> So instead they'll write >>> >> __slots__ = OrderedDict({'x': 5, 'y': 7}) >>> >> Causing the same issues? >>> > >>> > Perhaps OrderedDict should reject unordered sources. Hey, here is yet >>> one use case! >>> >>> I've maintained code that does this: >>> >>> self.headers = OrderedDict(headers) >>> self.origheaders = len(headers) >>> >>> ? so it can later do this: >>> >>> altheaders = list(self.headers.items())[self.origheaders:] >>> >>> Not a great design, but one that exists in the wild, and would be broken >>> by OrderedDict not allowing a dict as an argument. >>> >>> Also, this wouldn't allow creating an OrderedDict from an empty dict >>> (which seems far less stupid, but I didn't lead with it because I can't >>> remember seeing it in real code). >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> >> -- >> --Guido (mobile) >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Mon Nov 9 19:20:51 2015 From: mike at selik.org (Michael Selik) Date: Mon, 9 Nov 2015 19:20:51 -0500 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp> <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com> Message-ID: I found a use case, out in the wild! I've been searching through GitHub for "ordered" and similar phrases. Much of the usage is OrderedDict and OrderedSet, which really don't benefit from an essentially redundant inheritance from a hypothetical collections.Ordered. There are some tools that enable UI reordering, and a bunch of tree- and graph-traversal ordering functions. Again, not much benefit from a collections.Ordered. However, Django has a class factory that creates an attribute `can_order`: def formset_factory(form, formset=BaseFormSet, extra=1, can_order=False, can_delete=False, max_num=None, validate_max=False, min_num=None, validate_min=False): """Return a FormSet for the given form class.""" if min_num is None: min_num = DEFAULT_MIN_NUM if max_num is None: max_num = DEFAULT_MAX_NUM # hard limit on forms instantiated, to prevent memory-exhaustion attacks # limit is simply max_num + DEFAULT_MAX_NUM (which is 2*DEFAULT_MAX_NUM # if max_num is None in the first place) absolute_max = max_num + DEFAULT_MAX_NUM attrs = {'form': form, 'extra': extra, 'can_order': can_order, 'can_delete': can_delete, 'min_num': min_num, 'max_num': max_num, 'absolute_max': absolute_max, 'validate_min': validate_min, 'validate_max': validate_max} return type(form.__name__ + str('FormSet'), (formset,), attrs) This attribute gets set in several places, but checked only twice in the Django codebase. Unfortunately, I think switching to the proposed inheritance mechanism would make the code worse, not better: `self.can_order` would become `isinstance(self, collections.Ordered)`. The readability of the Django internal code would not be much different, but users would lose consistency of introspectability as the other features like `can_delete` are simple class attributes. On Mon, Nov 9, 2015 at 11:29 AM, Guido van Rossum wrote: > Well, that would still defeat the purpose, wouldn't it? The items are no > more ordered than the headers dict itself. Also, items() doesn't return a > sequence -- it's an ItemsView (which inherits from Set) and presumably it's > not Ordered. > > I guess my question is not so much how to prevent getting an exception -- > I'm trying to tease out what the right order for the headers would be. Or > perhaps I'm just trying to understand what the code is doing (the snippet > shown mostly looks like bad code to me). > > On Mon, Nov 9, 2015 at 12:03 AM, Ram Rachum wrote: >> >> I'm not Andrew, but I'm guessing simply writing >> `OrderedDict(headers.items())`. >> >> On Mon, Nov 9, 2015 at 6:28 AM, Guido van Rossum wrote: >>> >>> So if OrderedDict had always rejected construction from a dict, how would >>> you have written this? >>> >>> >>> On Sunday, November 8, 2015, Andrew Barnert via Python-ideas >>> wrote: >>>> >>>> On Nov 8, 2015, at 14:10, Serhiy Storchaka wrote: >>>> > >>>> >> On 08.11.15 23:12, Sjoerd Job Postmus wrote: >>>> >> On 8 Nov 2015, at 20:06, Amir Rachum >>> >> > wrote: >>>> >>> As part of BasicStruct I intend to allow the use of mapping types as >>>> >>> __slots__, with the semantics of default values.. >>>> >>> So it'll look something like this: >>>> >>> >>>> >>> class Point(BasicStruct): >>>> >>> __slots__ = {'x': 5, 'y': 7} >>>> >> >>>> >> So instead they'll write >>>> >> __slots__ = OrderedDict({'x': 5, 'y': 7}) >>>> >> Causing the same issues? >>>> > >>>> > Perhaps OrderedDict should reject unordered sources. Hey, here is yet >>>> > one use case! >>>> >>>> I've maintained code that does this: >>>> >>>> self.headers = OrderedDict(headers) >>>> self.origheaders = len(headers) >>>> >>>> ? so it can later do this: >>>> >>>> altheaders = list(self.headers.items())[self.origheaders:] >>>> >>>> Not a great design, but one that exists in the wild, and would be broken >>>> by OrderedDict not allowing a dict as an argument. >>>> >>>> Also, this wouldn't allow creating an OrderedDict from an empty dict >>>> (which seems far less stupid, but I didn't lead with it because I can't >>>> remember seeing it in real code). >>>> >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> >>> >>> -- >>> --Guido (mobile) >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From guido at python.org Mon Nov 9 19:47:08 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 9 Nov 2015 16:47:08 -0800 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp> <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com> Message-ID: Hm, this just indicates whether some UI appears by which the user can reorder the forms in a list? Is it even appropriate to move this from the instance to the class? On Mon, Nov 9, 2015 at 4:20 PM, Michael Selik wrote: > I found a use case, out in the wild! > > I've been searching through GitHub for "ordered" and similar phrases. > Much of the usage is OrderedDict and OrderedSet, which really don't > benefit from an essentially redundant inheritance from a hypothetical > collections.Ordered. There are some tools that enable UI reordering, > and a bunch of tree- and graph-traversal ordering functions. Again, > not much benefit from a collections.Ordered. > > However, Django has a class factory that creates an attribute `can_order`: > > def formset_factory(form, formset=BaseFormSet, extra=1, can_order=False, > can_delete=False, max_num=None, validate_max=False, > min_num=None, validate_min=False): > """Return a FormSet for the given form class.""" > if min_num is None: > min_num = DEFAULT_MIN_NUM > if max_num is None: > max_num = DEFAULT_MAX_NUM > # hard limit on forms instantiated, to prevent memory-exhaustion > attacks > # limit is simply max_num + DEFAULT_MAX_NUM (which is 2*DEFAULT_MAX_NUM > # if max_num is None in the first place) > absolute_max = max_num + DEFAULT_MAX_NUM > attrs = {'form': form, 'extra': extra, > 'can_order': can_order, 'can_delete': can_delete, > 'min_num': min_num, 'max_num': max_num, > 'absolute_max': absolute_max, 'validate_min': validate_min, > 'validate_max': validate_max} > return type(form.__name__ + str('FormSet'), (formset,), attrs) > > > > This attribute gets set in several places, but checked only twice in > the Django codebase. Unfortunately, I think switching to the proposed > inheritance mechanism would make the code worse, not better: > `self.can_order` would become `isinstance(self, collections.Ordered)`. > The readability of the Django internal code would not be much > different, but users would lose consistency of introspectability as > the other features like `can_delete` are simple class attributes. > > > > On Mon, Nov 9, 2015 at 11:29 AM, Guido van Rossum > wrote: > > Well, that would still defeat the purpose, wouldn't it? The items are no > > more ordered than the headers dict itself. Also, items() doesn't return a > > sequence -- it's an ItemsView (which inherits from Set) and presumably > it's > > not Ordered. > > > > I guess my question is not so much how to prevent getting an exception -- > > I'm trying to tease out what the right order for the headers would be. Or > > perhaps I'm just trying to understand what the code is doing (the snippet > > shown mostly looks like bad code to me). > > > > On Mon, Nov 9, 2015 at 12:03 AM, Ram Rachum wrote: > >> > >> I'm not Andrew, but I'm guessing simply writing > >> `OrderedDict(headers.items())`. > >> > >> On Mon, Nov 9, 2015 at 6:28 AM, Guido van Rossum > wrote: > >>> > >>> So if OrderedDict had always rejected construction from a dict, how > would > >>> you have written this? > >>> > >>> > >>> On Sunday, November 8, 2015, Andrew Barnert via Python-ideas > >>> wrote: > >>>> > >>>> On Nov 8, 2015, at 14:10, Serhiy Storchaka > wrote: > >>>> > > >>>> >> On 08.11.15 23:12, Sjoerd Job Postmus wrote: > >>>> >> On 8 Nov 2015, at 20:06, Amir Rachum >>>> >> > wrote: > >>>> >>> As part of BasicStruct I intend to allow the use of mapping types > as > >>>> >>> __slots__, with the semantics of default values.. > >>>> >>> So it'll look something like this: > >>>> >>> > >>>> >>> class Point(BasicStruct): > >>>> >>> __slots__ = {'x': 5, 'y': 7} > >>>> >> > >>>> >> So instead they'll write > >>>> >> __slots__ = OrderedDict({'x': 5, 'y': 7}) > >>>> >> Causing the same issues? > >>>> > > >>>> > Perhaps OrderedDict should reject unordered sources. Hey, here is > yet > >>>> > one use case! > >>>> > >>>> I've maintained code that does this: > >>>> > >>>> self.headers = OrderedDict(headers) > >>>> self.origheaders = len(headers) > >>>> > >>>> ? so it can later do this: > >>>> > >>>> altheaders = list(self.headers.items())[self.origheaders:] > >>>> > >>>> Not a great design, but one that exists in the wild, and would be > broken > >>>> by OrderedDict not allowing a dict as an argument. > >>>> > >>>> Also, this wouldn't allow creating an OrderedDict from an empty dict > >>>> (which seems far less stupid, but I didn't lead with it because I > can't > >>>> remember seeing it in real code). > >>>> > >>>> _______________________________________________ > >>>> Python-ideas mailing list > >>>> Python-ideas at python.org > >>>> https://mail.python.org/mailman/listinfo/python-ideas > >>>> Code of Conduct: http://python.org/psf/codeofconduct/ > >>> > >>> > >>> > >>> -- > >>> --Guido (mobile) > >>> > >>> _______________________________________________ > >>> Python-ideas mailing list > >>> Python-ideas at python.org > >>> https://mail.python.org/mailman/listinfo/python-ideas > >>> Code of Conduct: http://python.org/psf/codeofconduct/ > >> > >> > > > > > > > > -- > > --Guido van Rossum (python.org/~guido) > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Mon Nov 9 20:03:11 2015 From: mike at selik.org (Michael Selik) Date: Tue, 10 Nov 2015 01:03:11 +0000 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp> <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com> Message-ID: If my thoughts got lost in the mess of text: No, I don't think it is appropriate. On Mon, Nov 9, 2015 at 7:47 PM Guido van Rossum wrote: > Hm, this just indicates whether some UI appears by which the user can > reorder the forms in a list? Is it even appropriate to move this from the > instance to the class? > > On Mon, Nov 9, 2015 at 4:20 PM, Michael Selik wrote: > >> I found a use case, out in the wild! >> >> I've been searching through GitHub for "ordered" and similar phrases. >> Much of the usage is OrderedDict and OrderedSet, which really don't >> benefit from an essentially redundant inheritance from a hypothetical >> collections.Ordered. There are some tools that enable UI reordering, >> and a bunch of tree- and graph-traversal ordering functions. Again, >> not much benefit from a collections.Ordered. >> >> However, Django has a class factory that creates an attribute `can_order`: >> >> def formset_factory(form, formset=BaseFormSet, extra=1, can_order=False, >> can_delete=False, max_num=None, validate_max=False, >> min_num=None, validate_min=False): >> """Return a FormSet for the given form class.""" >> if min_num is None: >> min_num = DEFAULT_MIN_NUM >> if max_num is None: >> max_num = DEFAULT_MAX_NUM >> # hard limit on forms instantiated, to prevent memory-exhaustion >> attacks >> # limit is simply max_num + DEFAULT_MAX_NUM (which is >> 2*DEFAULT_MAX_NUM >> # if max_num is None in the first place) >> absolute_max = max_num + DEFAULT_MAX_NUM >> attrs = {'form': form, 'extra': extra, >> 'can_order': can_order, 'can_delete': can_delete, >> 'min_num': min_num, 'max_num': max_num, >> 'absolute_max': absolute_max, 'validate_min': validate_min, >> 'validate_max': validate_max} >> return type(form.__name__ + str('FormSet'), (formset,), attrs) >> >> >> >> This attribute gets set in several places, but checked only twice in >> the Django codebase. Unfortunately, I think switching to the proposed >> inheritance mechanism would make the code worse, not better: >> `self.can_order` would become `isinstance(self, collections.Ordered)`. >> The readability of the Django internal code would not be much >> different, but users would lose consistency of introspectability as >> the other features like `can_delete` are simple class attributes. >> >> >> >> On Mon, Nov 9, 2015 at 11:29 AM, Guido van Rossum >> wrote: >> > Well, that would still defeat the purpose, wouldn't it? The items are no >> > more ordered than the headers dict itself. Also, items() doesn't return >> a >> > sequence -- it's an ItemsView (which inherits from Set) and presumably >> it's >> > not Ordered. >> > >> > I guess my question is not so much how to prevent getting an exception >> -- >> > I'm trying to tease out what the right order for the headers would be. >> Or >> > perhaps I'm just trying to understand what the code is doing (the >> snippet >> > shown mostly looks like bad code to me). >> > >> > On Mon, Nov 9, 2015 at 12:03 AM, Ram Rachum wrote: >> >> >> >> I'm not Andrew, but I'm guessing simply writing >> >> `OrderedDict(headers.items())`. >> >> >> >> On Mon, Nov 9, 2015 at 6:28 AM, Guido van Rossum >> wrote: >> >>> >> >>> So if OrderedDict had always rejected construction from a dict, how >> would >> >>> you have written this? >> >>> >> >>> >> >>> On Sunday, November 8, 2015, Andrew Barnert via Python-ideas >> >>> wrote: >> >>>> >> >>>> On Nov 8, 2015, at 14:10, Serhiy Storchaka >> wrote: >> >>>> > >> >>>> >> On 08.11.15 23:12, Sjoerd Job Postmus wrote: >> >>>> >> On 8 Nov 2015, at 20:06, Amir Rachum > >>>> >> > wrote: >> >>>> >>> As part of BasicStruct I intend to allow the use of mapping >> types as >> >>>> >>> __slots__, with the semantics of default values.. >> >>>> >>> So it'll look something like this: >> >>>> >>> >> >>>> >>> class Point(BasicStruct): >> >>>> >>> __slots__ = {'x': 5, 'y': 7} >> >>>> >> >> >>>> >> So instead they'll write >> >>>> >> __slots__ = OrderedDict({'x': 5, 'y': 7}) >> >>>> >> Causing the same issues? >> >>>> > >> >>>> > Perhaps OrderedDict should reject unordered sources. Hey, here is >> yet >> >>>> > one use case! >> >>>> >> >>>> I've maintained code that does this: >> >>>> >> >>>> self.headers = OrderedDict(headers) >> >>>> self.origheaders = len(headers) >> >>>> >> >>>> ? so it can later do this: >> >>>> >> >>>> altheaders = list(self.headers.items())[self.origheaders:] >> >>>> >> >>>> Not a great design, but one that exists in the wild, and would be >> broken >> >>>> by OrderedDict not allowing a dict as an argument. >> >>>> >> >>>> Also, this wouldn't allow creating an OrderedDict from an empty dict >> >>>> (which seems far less stupid, but I didn't lead with it because I >> can't >> >>>> remember seeing it in real code). >> >>>> >> >>>> _______________________________________________ >> >>>> Python-ideas mailing list >> >>>> Python-ideas at python.org >> >>>> https://mail.python.org/mailman/listinfo/python-ideas >> >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >> >>> >> >>> >> >>> >> >>> -- >> >>> --Guido (mobile) >> >>> >> >>> _______________________________________________ >> >>> Python-ideas mailing list >> >>> Python-ideas at python.org >> >>> https://mail.python.org/mailman/listinfo/python-ideas >> >>> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> >> >> > >> > >> > >> > -- >> > --Guido van Rossum (python.org/~guido) >> > >> > _______________________________________________ >> > Python-ideas mailing list >> > Python-ideas at python.org >> > https://mail.python.org/mailman/listinfo/python-ideas >> > Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Mon Nov 9 20:14:05 2015 From: mike at selik.org (Michael Selik) Date: Tue, 10 Nov 2015 01:14:05 +0000 Subject: [Python-ideas] Use `isinstance` check during exception handling In-Reply-To: <6f22ce0f98ae5b4f7a290343bad92564.squirrel@mail.sjec.nl> References: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl> <1277750399.598478.1446796147897.JavaMail.yahoo@mail.yahoo.com> <6f22ce0f98ae5b4f7a290343bad92564.squirrel@mail.sjec.nl> Message-ID: I searched cpython/Lib/*.py for except blocks that have an if-statement as the first line. r'except \w+ as\s+\w+:\n\s+if .*:' Not the world's most thorough regex, but it seemed to do the trick. There are very few repeated except/if blocks that would be appropriate for refactoring with any technique. The majority were in two modules -- mailbox.py and pathlib.py -- checking if the errorno was ENOINT and/or ENOTDIR. I suppose those could be improved with FileNotFound and NotADirectoryError, but it doesn't look like we need anything more potent. Is there a different place we should look for refactoring candidates? On Mon, Nov 9, 2015, 12:41 AM wrote: > > I made the changes I listed above and ran `make; make test`, and got only > > 1 error due to socket.socket running out of memory or something. I'll > > re-run the tests on a clean checkout as well, but for now I think it's > > unrelated. > > After running the tests against a clean checkout (without my changes): I > get the same errors from test_socket. So it's unrelated. > > > Regards, > > Sjoerd Job Postmus > > > >> On 7 Nov 2015, at 20:18, Michael Selik wrote: > >> > >> Sjoerd, glad we agree, happy to help. > >> > >> In case someone is still interested in making a patch, I'd like to > >> clarify why it makes me nervous. > >> > >> Though Sjoerd led with a TL;DR that summarized his suggestion as a swap > >> of isinstance for issubclass, a better summary would have been: "I > >> prefer more specific exception types". The proposed solution was to > >> enable custom exception types that conditionally insert themselves into > >> the class hierarchy depending on data stored on a raised exception > >> instance. > >> > >> Unfortunately, that solution requires changing an obscure error handling > >> feature. As Sjoerd noted when he sent the first message in this thread, > >> it is possible this would introduce a backwards-incompatibility. Perhaps > >> no one in the standard library or major public projects would write such > >> weird code as to get tripped up by the proposed change, but it's always > >> possible that someone out there is using this feature for an important > >> project. I think we've learned the lesson from the long slog to Python 3 > >> adoption (and how Perl 6 harmed that community, and countless other > >> examples) that backwards compatibility is of utmost importance. > >> > >> The current code pattern for a single try/except is to catch the general > >> exception and have an if-condition in the except block.That's a frequent > >> and readable practice. > >> > >> try: > >> cursor.execute(query) > >> except sqlite3.OperationalError as str(e): > >> if 'syntax error' in str(e): > >> print('Your SQL is bad') > >> > >> If that code or similar code appears many times, a good way to refactor > >> with the current features of Python is to use a context manager. > >> > >> class SqlSyntaxErrorHandler: > >> def __enter__(self): > >> return self > >> def __exit__(self, etyp, einstance, etraceback): > >> if (issubclass(etyp, sqlite3.OperationalError) > >> and 'syntax error' in str(einstance)): > >> print('Your SQL is bad') > >> > >> with SqlSyntaxErrorHandler: > >> cursor.execute(first_query) > >> > >> with SqlSyntaxErrorHandler: > >> cursor.execute(second_query) > >> > >> While context managers may not be a beginner topic, they are typically > >> easier to read and debug than using metaclasses for conditional runtime > >> inheritance. Even if there would be some benefit to the metaclass style, > >> it's not worth the risk of changing current error handling behavior. > >> > >> -- Michael > >> > >> > >>> On Fri, Nov 6, 2015, 2:49 AM Andrew Barnert via Python-ideas > >>> wrote: > >>> On Thursday, November 5, 2015 9:57 PM, "sjoerdjob at sjec.nl" > >>> wrote: > >>> > >>> >> Andrew Barnert via Python-ideas writes: > >>> >> > Sjoerd Job Postmus wrote: > >>> >> > >>> >> > > It might seem troublesome to change these locations to raise > >>> >> > > actual instances instead of strings, but those might actually > >>> >> > > just be hidden bugs just the same. > >>> >> > >>> >> No, I'm sure mostly they're not. They're working code that has > >>> > been > >>> >> there since the early days, perhaps a few from the 2.x days in > >>> >> modules. Python is generally very conservative about changing > >>> working > >>> >> code. > >>> >> > >>> >> > > Do you think anyone would object if I'd go over the code > >>> > looking > >>> >> > > for places where type(tstate->cur_value) != > >>> > tstate->cur_type and > >>> >> > > propose patches for these? > >>> >> > >>> >> I think, yes, though I don't know enough about it to object myself. > >>> >> Code churn of this kind is generally frowned upon, unless you can > >>> show > >>> >> that the risk of hidden bugs of the kind you refer to above is > >>> >> substantially greater than the risk that *you* will introduce bugs > >>> >> with your patches. > >>> > > >>> > After digging more into the code, I concur. It's quite easy to make > >>> some > >>> > changes that break some stuff in this logic. > >>> > >>> Looking at this a bit further, I think there may be something you can > >>> do that gets everything you actually wanted, without changes all over > >>> the place and serious risks. > >>> > >>> Basically, it's just two pretty small changes in errors.c: > >>> > >>> * Change PyErr_GivenExceptionMatches so that if given a value that's an > >>> instance of an exception type (PyExceptionInstance_Check(err) is true), > >>> instead of just getting its class and then ignoring the value, it does > >>> a standard isinstance check on the value. > >>> > >>> * Change PyErr_ExceptionMatches so that if there is an exception value, > >>> and its type matches the exception type, it is passed to > >>> PyErr_GivenExceptionMatches; otherwise, the exception type is passed. > >>> > >>> > >>> So, all Python code that raises and handles exceptions will get the > >>> benefit of isinstance checks. Any new C code that wants that benefit > >>> can get it very easily (only raise with PyErr_SetObject or something > >>> that uses it like PyErr_SetFromErrno; only handle with > >>> PyErr_GivenExceptionMatches). Any existing C code that skips creating > >>> an instance will continue to do so, and to get the same fast-path check > >>> used today, except for one extra call to > >>> PyExceptionInstance_Check(tstate->curexc_value). > >>> > >>> Of course it's not just that small patch to errors.c. You also need > >>> tests to exercise the new functionality. And a docs patch for the > >>> Exceptions section of the C API. And also a docs patch to the > >>> Exceptions section in the Reference. But no other code patches. > >>> > >>> And you'd definitely need to performance-test two things: First, that > >>> the extra PyExceptionInstance_Check doesn't slow down fast-path > >>> handling (e.g., in an empty for loops). Second, whatever performance > >>> tests your original proposal would have needed, to make sure the > >>> isinstance check doesn't noticeably slow down pure-Python code that > >>> handles a lot of exceptions. > >>> > >>> > >>> I think the only C API docs change needed is to PyErr_ExceptionMatches. > >>> However, a note explaining how to make sure to get and/or to bypass the > >>> isinstance check as desired might be useful. > >>> > >>> For the reference docs, I think just this note at the end of the > >>> section: > >>> > >>> > Note: Exceptions raised and handled by Python code use the same > >>> mechanism as the isinstance function for determining whether an > >>> exception matches. However, exceptions raised or handled by the > >>> implementation (or implementation-level extensions) may bypass that > >>> and check the type directly. > >>> > >>> > >>> > New in 3.6: In previous versions of Python, exception handling always > >>> bypassed the isinstance function and checked the type directly. > >>> _______________________________________________ > >>> Python-ideas mailing list > >>> Python-ideas at python.org > >>> https://mail.python.org/mailman/listinfo/python-ideas > >>> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Mon Nov 9 22:52:55 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 10 Nov 2015 12:52:55 +0900 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp> <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com> Message-ID: <22081.27159.81864.553297@turnbull.sk.tsukuba.ac.jp> Guido van Rossum writes: > I guess my question is not so much how to prevent getting an > exception -- I'm trying to tease out what the right order for the > headers would be. If they are email headers, then each MTA is supposed to *prepend* its trace fields (Received and friends) to the existing header block. Those new fields would correspond to Andrew's "alternate" headers. Perhaps you are thinking of the fact that MTAs are also supposed to preserve the existing order of headers. However, the mail system as a whole is in practice not reliable because of intermediating agents like mailing lists and user forwarding. These sometimes insert application fields (such as List-* and Resent-*) before the trace fields block rather than prepend their fields to the existing header. As an indication of this lack of reliability, DKIM does not prescribe a set or block of fields to sign, but rather the fields to sign are specified as a sequence in the DKIM-Signature header, and verifiers must extract and append the fields to the content to be signed in that order regardless of physical order. Python's email package provides a header abstraction which is an ordered mapping. The point is to be able to reproduce the entire message with full accuracy by Message.flatten() (ie, without affecting signatures). So the relevant order is order found in the input message (ie, FIFO). However, this order is not stable in the sense required by the OP of this thread: existing fields may be deleted and new ones inserted at arbitrary positions. Nor does it refer to any externally-defined collation order for fields. BTW, I do understand the difference between sorting and ordering. What I missed is that that Amir (the OP) didn't ask for his BasicStruct to be Ordered (although I guess it would be, considered as a container for its slots); he wants to check that the initializer for __slots__ is Ordered. But that is insufficient in this use case (an alphabetically-ordered instance would be a problematic initializer: consider a derived BasicStruct that adds members to its parent BasicStruct). I still question the general usefulness of this ABC on the one hand, and ask how far to proliferate its more useful subclasses on the other. From abarnert at yahoo.com Mon Nov 9 05:32:06 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 9 Nov 2015 02:32:06 -0800 Subject: [Python-ideas] Adding collections.abc.Ordered In-Reply-To: References: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp> <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com> <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp> <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com> <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp> <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com> Message-ID: On Nov 8, 2015, at 20:28, Guido van Rossum wrote: > > So if OrderedDict had always rejected construction from a dict, how would you have written this? Well, I wouldn't have written it this way in the first place. The obvious way to do it is to store the original headers and new headers separately. ChainMapping two dicts together is much more obvious than cramming them into one OrderedDict and keeping track of where the dividing index is. So, I'd do this: self.original_headers = copy.copy(headers) self.alt_headers = OrderedDict() self.headers = ChainMap(self.alt_headers, self.original_headers) Now origheaders is just len(self.original_headers), altheaders is self.alt_headers.items(). I didn't make this change to the code in question because it worked, and had no edits for over 2 years, and it only took me a couple minutes to figure out what it was doing, so it made more sense just to leave it alone. If Python 3.7 forced me to change it by not allowing OrderedDict to be constructed from a dict, I'd have to decide whether to make this change, or the obvious mechanical fix of just calling OrderedDict on iter(headers.items()) instead of headers to trick the Ordered check. By the way, the fact that this simple mechanical workaround works, and provides no additional semantics to tell the reader what's going on, seems like a minor strike against the proposal. (And if you ban iterators too, I'd just change the iter(?) to list(?).) All Ordered can really check is that the type provides some way to preserve a meaningful order, not that the data really are meaningfully ordered, after all. >> On Sunday, November 8, 2015, Andrew Barnert via Python-ideas wrote: >> On Nov 8, 2015, at 14:10, Serhiy Storchaka wrote: >> > >> >> On 08.11.15 23:12, Sjoerd Job Postmus wrote: >> >> On 8 Nov 2015, at 20:06, Amir Rachum > >> > wrote: >> >>> As part of BasicStruct I intend to allow the use of mapping types as >> >>> __slots__, with the semantics of default values.. >> >>> So it'll look something like this: >> >>> >> >>> class Point(BasicStruct): >> >>> __slots__ = {'x': 5, 'y': 7} >> >> >> >> So instead they'll write >> >> __slots__ = OrderedDict({'x': 5, 'y': 7}) >> >> Causing the same issues? >> > >> > Perhaps OrderedDict should reject unordered sources. Hey, here is yet one use case! >> >> I've maintained code that does this: >> >> self.headers = OrderedDict(headers) >> self.origheaders = len(headers) >> >> ? so it can later do this: >> >> altheaders = list(self.headers.items())[self.origheaders:] >> >> Not a great design, but one that exists in the wild, and would be broken by OrderedDict not allowing a dict as an argument. >> >> Also, this wouldn't allow creating an OrderedDict from an empty dict (which seems far less stupid, but I didn't lead with it because I can't remember seeing it in real code). >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > -- > --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Fri Nov 13 23:46:44 2015 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 13 Nov 2015 22:46:44 -0600 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> Message-ID: On Fri, Oct 16, 2015 at 9:08 PM, Andrew Barnert via Python-ideas < python-ideas at python.org> wrote: > Actually, forget all that; it's even simpler. > > At least in recent 3.x, the only thing wrong with inheriting from both > types, assuming you put OrderedDict first, is the __init__ signature. So: > > class OrderedDefaultDict(OrderedDict, defaultdict): > def __init__(self, default_factory=None, *a, **kw): > OrderedDict.__init__(self, *a, **kw) > self.default_factory = default_factory > > More importantly, because __missing__ support is built into dict, despite > the confusing docs for defaultdict, you don't really need defaultdict at > all here: > > class OrderedDefaultDict(OrderedDict): > def __init__(self, default_factory=None, *a, **kw): > OrderedDict.__init__(self, *a, **kw) > self.default_factory = default_factory > def __missing__(self, key): > self[key] = value = default_factory() > return value > > And either of these should work with 2.5+ (according to > https://docs.python.org/2/library/stdtypes.html#dict that's when > dict.__missing__ was added). > > Thanks! - [ ] Could/should maybe either of these make it into the standard library, that would save a fair amount of copying. .. Great in combination w/ dict views: https://docs.python.org/2/library/stdtypes.html#dictionary-view-objects > Sent from my iPhone > > On Oct 16, 2015, at 18:57, Andrew Barnert wrote: > > Sorry, sent too soon? > > Sent from my iPhone > > On Oct 16, 2015, at 18:39, Andrew Barnert via Python-ideas < > python-ideas at python.org> wrote: > > On Oct 16, 2015, at 03:36, Ian Foote wrote: > > I recently wanted to use an OrderedCounter and an OrderedDefaultDict. > After a bit of googling I discovered that OrderedCounter is quite easy to > implement: > > > This is already in the docs for OrderedDict, so it shouldn't have taken > any googling. > > from collections import Counter, OrderedDict > > class OrderedCounter(Counter, OrderedDict): > 'Counter that remembers the order elements are first seen' > def __repr__(self): > return '%s(%r)' % (self.__class__.__name__, > OrderedDict(self)) > def __reduce__(self): > return self.__class__, (OrderedDict(self),) > > > from https://rhettinger.wordpress.com/2011/05/26/super-considered-super/ > > Unfortunately an OrderedDefaultDict did not seem so easy. I did find > http://stackoverflow.com/questions/6190331/can-i-do-an-ordered-default-dict-in-python > which suggests: > > > Most of the trickiness here is handling None as a factory, which is only > useful for subclasses that implement a custom __missing__ > > > ? or that requires two-stage initialization, where it can initialize the > base with None and then assign the factory later. Still not all that > common, but it is a separate reason for the extra complexity besides a > custom __missing__ that doesn't need the factory. > > To make something that works like you'd expect except for that part is a > lot simpler. > > > Off the top of my head (untested, since I'm on my phone): > > class DefaultOrderedDict(OrderedDict): > def __init__(self, default_factory, *a, **kw): > super().__init__(*a, **kw) > self.default_factory = default_factory > def __getitem__(self, key): > try: > return super().__getitem__(key) > except KeyError: > return self.__missing__(key) > def __missing__(self, key): > self[key] = value = self.default_factory() > return value > def __repr__(self): > return '{}({}, {})'.format(type(self).__name__, > self.default_factory, dict.__repr__(self)) > > In fact, if you aren't worried about subclassing, or about being able to > directly call or pass around a bound__missing__, you can make this even > simpler. > > I may be wrong about this being picklable/copyable without any extra help; > if so, it still doesn't need the whole complicated "treat a None factory as > if it doesn't exist" part. > > Also, I think the SO recipe works in Python 2.4+, or whichever first added > OrderedDict, while this requires 3.1 (although it should be obvious how to > backport it, make sure that OrderedDict isn't an old-style class?). > > Anyway, other than those issues, it seems pretty obvious, and I don't > think there's any part of that functionality that's easy to get wrong. > Handling a None factory in __missing__ and pickle/copy is easy to get > wrong, but if you aren't going to subclass this and therefore aren't going > to try to build that part, you're not going to build it wrong. > > And I'm not sure a docs recipe that's mostly code that's unnecessary for > most uses, hiding the important and simple part, would be a great idea. > > from collections import OrderedDict, Callable > class DefaultOrderedDict(OrderedDict): > # Source: http://stackoverflow.com/a/6190500/562769 > def __init__(self, default_factory=None, *a, **kw): > if (default_factory is not None and > not isinstance(default_factory, Callable)): > raise TypeError('first argument must be callable') > OrderedDict.__init__(self, *a, **kw) > self.default_factory = default_factory > > def __getitem__(self, key): > try: > return OrderedDict.__getitem__(self, key) > except KeyError: > return self.__missing__(key) > > def __missing__(self, key): > if self.default_factory is None: > raise KeyError(key) > self[key] = value = self.default_factory() > return value > > def __reduce__(self): > if self.default_factory is None: > args = tuple() > else: > args = self.default_factory, > return type(self), args, None, None, self.items() > > def copy(self): > return self.__copy__() > > def __copy__(self): > return type(self)(self.default_factory, self) > > def __deepcopy__(self, memo): > import copy > return type(self)(self.default_factory, > copy.deepcopy(self.items())) > > def __repr__(self): > return 'OrderedDefaultDict(%s, %s)' % (self.default_factory, > OrderedDict.__repr__(self)) > > > This seems to me both easy to get wrong and hard to discover. It also > risks getting out of sync with updates to collections. > I'm wondering if this is sufficient justification to add OrderedCounter > and OrderedDict to collections, either directly or as recipes. > > > Since one of them is already a recipe and the other one would probably not > be useful as one, I don't think that's a great idea. > > Adding OrderedDefaultDict to the module itself might be useful. Or, > alternatively, put it in PyPI or ActiveState and add a link to it from the > docs? (I'm pretty sure there are already implementations in both places, > actually. Also, it would be nice to be able to point at a third-party > implementation that works in 2.6+/3.2+ or whatever even if it only appears > in the 3.6 docs.) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Nov 14 00:17:20 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 13 Nov 2015 21:17:20 -0800 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> Message-ID: On Nov 13, 2015, at 20:46, Wes Turner wrote: > >> On Fri, Oct 16, 2015 at 9:08 PM, Andrew Barnert via Python-ideas wrote: >> Actually, forget all that; it's even simpler. >> >> At least in recent 3.x, the only thing wrong with inheriting from both types, assuming you put OrderedDict first, is the __init__ signature. So: >> >> class OrderedDefaultDict(OrderedDict, defaultdict): >> def __init__(self, default_factory=None, *a, **kw): >> OrderedDict.__init__(self, *a, **kw) >> self.default_factory = default_factory >> >> More importantly, because __missing__ support is built into dict, despite the confusing docs for defaultdict, you don't really need defaultdict at all here: >> >> class OrderedDefaultDict(OrderedDict): >> def __init__(self, default_factory=None, *a, **kw): >> OrderedDict.__init__(self, *a, **kw) >> self.default_factory = default_factory >> def __missing__(self, key): >> self[key] = value = default_factory() >> return value >> >> And either of these should work with 2.5+ (according to https://docs.python.org/2/library/stdtypes.html#dict that's when dict.__missing__ was added). > > Thanks! > > - [ ] Could/should maybe either of these make it into the standard library, > that would save a fair amount of copying. You could say the same about everything in the recipes, every one-liner like "def identity(x): return x", and so on. But most such things aren't used that often, and there's a cost to putting them in the stdlib?more for people to learn and remember, more code to be maintained, bigger downloads, etc. So just saying "maybe it would save some copying" isn't an argument for adding it to the stdlib. Adding OrderedDefaultDict as a docs recipe (as OrderedCounter already is) might be worth doing. Reorganizing the docs a bit to make it more obvious (by better highlighting __missing__, and making it clear that it's a method of dict rather than something special about defaultdict) seems even more likely to be worth it. If someone has an idea of how it should read, just file a docs bug and submit a patch and see if whoever's in charge of that area says it needs to come back here. > .. Great in combination w/ dict views: > https://docs.python.org/2/library/stdtypes.html#dictionary-view-objects Of course, and that works out of the box. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at whoosh.ca Mon Nov 16 01:17:37 2015 From: matt at whoosh.ca (Matt Chaput) Date: Mon, 16 Nov 2015 01:17:37 -0500 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) Message-ID: Since PyCharm added support for pep484 I've been working on adding type hints to the Whoosh code base. It makes it a lot more enjoyable to work with the code in the IDE. As I add more and more hints, it's already revealed bugs where arguments were accidentally swapped, or interfaces weren't updated everywhere. I feel much more confident working in the (quite large) codebase with typing. Basically, I'm loving the idea of PEP484. For me, it's a huge improvement in the development experience. However, I've already run into a few issues. I'm not sure if these were discussed during the development of PEP484. 1. The biggest problem, most perplexing problem for me is circular imports. That type hints must actually be imported at the top level is a MASSIVE limitation, at least in a big codebase. I started off importing the types I needed, then switched to the "import the module, specify the type in a string" trick, but even that has failed. In a large/complex codebase, the more things you have to import at the top level (to use for typing), the more potential import circles you create. Now I've hit an impasse where every step in the circle is using the module/string trick, but it doesn't help because I need to subclass something in the circle. For now, I'm going to have to workaround this, either by trying to move "interface" base classes out to their own side package, or just not type hinting some things. Neither is appetizing at all. I think a good solution might be if the typing system could have its own special "imports" that are only used by the typing system, e.g. something like this at the top of a file: # "real" import from foo import bar # imports just for type checking __typeimports__ = """ from baz import Qux """ 2. It would be really nice if we could have "type aliasing" (or whatever it's called). I have a lot of types that are just something like "Tuple[int, int]", so type checking doesn't help much. It would be much more useful if I have a value that Python sees as (e.g.) an int, but have the type system track it as something more specific. Something like this: DocId = typing.TypeAlias(int) DocLength = typing.TypeAlias(int) def id_and_length() -> Tuple[DocId, Length]: docid = 5 # type: DocId length = 10 # type: Length return docid, length 3. I can't tell from the PEP484 documentation... If I have a fully hinted base class, and subclass it with the same signature for every method, am I still supposed to hint all the arguments and returns in the subclass? That's what I've been doing, but it's pretty tedious. Cheers, From vito.detullio at gmail.com Mon Nov 16 01:51:02 2015 From: vito.detullio at gmail.com (Vito De Tullio) Date: Mon, 16 Nov 2015 07:51:02 +0100 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) References: Message-ID: Matt Chaput wrote: > 2. It would be really nice if we could have "type aliasing" (or whatever > it's called). I have a lot of types that are just something like > "Tuple[int, int]", so type checking doesn't help much. It would be much > more useful if I have a value that Python sees as (e.g.) an int, but have > the type system track it as something more specific. Something like this: > > DocId = typing.TypeAlias(int) > DocLength = typing.TypeAlias(int) > > def id_and_length() -> Tuple[DocId, Length]: > docid = 5 # type: DocId > length = 10 # type: Length > return docid, length like... type aliases? https://www.python.org/dev/peps/pep-0484/#type-aliases -- By ZeD From spinenkoia at gmail.com Mon Nov 16 10:03:32 2015 From: spinenkoia at gmail.com (=?UTF-8?B?0JjQstCw0L0g0KHQv9C40L3QtdC90LrQvg==?=) Date: Mon, 16 Nov 2015 20:03:32 +0500 Subject: [Python-ideas] The prohibition overrides the class method Message-ID: Hello, I want to apologize for my English. I propose to make an analogue of the final specifier in C ++. In class abc.ABCMeta has decorator abstractmethod, you can add decorator finalmethod. I have a sketch of implementation: def finalmethod(funcobj): funcobj.__isfinalmethod__ = True return funcobj class Override(ABCMeta): def __init__(cls, name, bases, namespace): super(Override, cls).__init__(name, bases, namespace) finals = {name for name, value in namespace.items() if getattr(value, "__isfinalmethod__", False) } for base in bases: for name in getattr(base, "__finalmethods__", set()): value = getattr(cls, name, None) if getattr(value, "__isfinalmethod__", False): finals.add(name) else: raise TypeError("function '" + str(value.__name__) + "' in class '" + str(cls.__name__) + "' is final") cls.__finalmethods__ = frozenset(finals) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 16 10:44:45 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 17 Nov 2015 02:44:45 +1100 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: References: Message-ID: On Tue, Nov 17, 2015 at 2:03 AM, ???? ???????? wrote: > Hello, I want to apologize for my English. It's fine - and a lot better than my Russian! (Or Ukrainian, or any of the other languages that use a Cyrillic script. See, I can't even tell one from another - all I know is which script you've used.) > I propose to make an analogue of the final specifier in C ++. In class > abc.ABCMeta has decorator abstractmethod, you can add decorator finalmethod. It's not common in Python to actually enforce this kind of thing. What's the advantage of stopping people from overriding some methods? In Java, as I understand it, the compiler optimizes calls to final methods; but in Python, you can even add methods to individual instances of a class: >>> class X: ... def method(self): ... print("I am the regular method.") ... >>> x = X() >>> x.method() I am the regular method. >>> x.method = lambda: print("I am the custom method.") >>> x.method() I am the custom method. Can you show an example of how this decorator would help your project? If so, it might make a useful recipe, rather than actually a part of the language or standard library - it can go into a collection of "here's some neat tricks you can do with function decorators" tips. ChrisA From matt at whoosh.ca Mon Nov 16 10:48:54 2015 From: matt at whoosh.ca (Matt Chaput) Date: Mon, 16 Nov 2015 10:48:54 -0500 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: References: Message-ID: <0142D3D9-B73C-4D7C-990D-8C3D191B461A@whoosh.ca> > like... type aliases? https://www.python.org/dev/peps/pep-0484/#type-aliases As I read it, that's just a way to avoid having to spell out the same type definition more than once. It's not the same as being able to define a new type for the type system which Python sees as a plain old primitive type. I'm probably not using the right programming language theory word for this concept :) Matt From rosuav at gmail.com Mon Nov 16 10:53:31 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 17 Nov 2015 02:53:31 +1100 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: <0142D3D9-B73C-4D7C-990D-8C3D191B461A@whoosh.ca> References: <0142D3D9-B73C-4D7C-990D-8C3D191B461A@whoosh.ca> Message-ID: On Tue, Nov 17, 2015 at 2:48 AM, Matt Chaput wrote: > As I read it, that's just a way to avoid having to spell out the same type definition more than once. It's not the same as being able to define a new type for the type system which Python sees as a plain old primitive type. > > I'm probably not using the right programming language theory word for this concept :) I'm not entirely sure, but I think you might be able to subclass something to create a special form. A length might be a subclass of integer, and a docid could be a different subclass of integer. Lengths and document IDs could then be counted as distinct from each other, although any function that expects an integer would happily accept either. (That might not be appropriate for docid, but it would for length (eg range() should accept them), so you might want to do docid some other way.) ChrisA From ian.g.kelly at gmail.com Mon Nov 16 10:55:24 2015 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 16 Nov 2015 08:55:24 -0700 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: References: Message-ID: On Mon, Nov 16, 2015 at 8:44 AM, Chris Angelico wrote: > On Tue, Nov 17, 2015 at 2:03 AM, ???? ???????? wrote: >> Hello, I want to apologize for my English. > > It's fine - and a lot better than my Russian! (Or Ukrainian, or any of > the other languages that use a Cyrillic script. See, I can't even tell > one from another - all I know is which script you've used.) > >> I propose to make an analogue of the final specifier in C ++. In class >> abc.ABCMeta has decorator abstractmethod, you can add decorator finalmethod. > > It's not common in Python to actually enforce this kind of thing. > What's the advantage of stopping people from overriding some methods? I would add the point that while abstractmethod helps the programmer by warning when they've forgotten to implement a method, finalmethod is more likely to just get in the way when the programmer needs to override a method, but the class author has arbitrarily decided that they can't. Python usually tries to give the programmer what they need while staying out of their way as much as possible. From srkunze at mail.de Mon Nov 16 11:33:53 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 16 Nov 2015 17:33:53 +0100 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: References: Message-ID: <564A0571.1010107@mail.de> Or like structural typing - there has already been some discussion on this list: https://github.com/ambv/typehinting/issues/11#issuecomment-138133867 On 16.11.2015 07:51, Vito De Tullio wrote: > Matt Chaput wrote: > >> 2. It would be really nice if we could have "type aliasing" (or whatever >> it's called). I have a lot of types that are just something like >> "Tuple[int, int]", so type checking doesn't help much. It would be much >> more useful if I have a value that Python sees as (e.g.) an int, but have >> the type system track it as something more specific. Something like this: >> >> DocId = typing.TypeAlias(int) >> DocLength = typing.TypeAlias(int) >> >> def id_and_length() -> Tuple[DocId, Length]: >> docid = 5 # type: DocId >> length = 10 # type: Length >> return docid, length > like... type aliases? https://www.python.org/dev/peps/pep-0484/#type-aliases > From srkunze at mail.de Mon Nov 16 11:27:20 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 16 Nov 2015 17:27:20 +0100 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: References: Message-ID: <564A03E8.2050709@mail.de> It's a well-known feature from other languages ("sealed"). Though, I tend to agree here that Python should not create barriers. What would a real-world example look like? I cannot imagine where people can decide *properly***when to place that modifier. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Nov 16 12:07:43 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 16 Nov 2015 09:07:43 -0800 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: References: Message-ID: <23E8DA7D-CBCB-410E-8A0A-595C0AFCADCF@yahoo.com> On Nov 16, 2015, at 07:03, ???? ???????? wrote: > > Hello, I want to apologize for my English. > I propose to make an analogue of the final specifier in C ++. In class abc.ABCMeta has decorator abstractmethod, you can add decorator finalmethod. Since the C++ standard is (mostly) written based on public proposals on open-std.org, presumably there's some discussion of the motivation behind adding this feature to C++11. It would be worth knowing what that motivation is, to see whether it applies to Python. For example, if it was proposed just to provide optimization opportunities, or to be closer to C++/CLI, then it probably isn't worth adding to Python (unless you have Python-specific use cases to offer, and it's just a coincidence that C++ and Python could use very similar features). On the other hand, if it solves a real problem with making some hierarchies easier to read, or removing opportunities for hard-to-diagnose errors, then it may be (assuming those problems have equivalents in Python). Sometimes there are also blog posts presenting the idea to C++ users instead of the C++ committee, which often get the motivation across better (e.g., explaining how rvalue references solve the forwarding and move problems that boost ran into with the previous version of C++). So, if you want to suggest a feature copied from C++11, it's probably worth searching for those references and providing links and summaries. I suspect that it's primarily for optimization. In C++, by default, methods can't be overridden, so if the compiler sees spam.cook() it can just compile in a direct call to Spam::cook. But if cook() is declared virtual in either Spam or any of its superclasses, it instead has to compile an indirect call to spam->vtable[1], which also means it will usually keep Spam::vtable in L2 cache whenever a Spam object is around, and so on, which can make a big difference. So, when the compiler can prove that spam really is a Spam, and not some subtype (e.g., because the function constructed it locally, or accepted it by value), it will optimize out the virtual call. What final adds is that if cook() is declared final in Spam or any of its superclasses, the compiler can optimize out the virtual call even if the value could actually be a subtype (e.g., because the function accepted it by reference), because it would be illegal for any such subtype to have overridden the method. So, that can be a huge benefit. Obviously, none of this applies in Python. Methods are always looked up by name in the value's dict, then the class and each of its bases at runtime; there's no concept of a variable's type to start the search at, and no way to record something that can be directly called at compile time (especially since type, function, and method objects are themselves constructed at runtime). > I have a sketch of implementation: > def finalmethod(funcobj): > funcobj.__isfinalmethod__ = True > return funcobj > > class Override(ABCMeta): > def __init__(cls, name, bases, namespace): > super(Override, cls).__init__(name, bases, namespace) > finals = {name > for name, value in namespace.items() > if getattr(value, "__isfinalmethod__", False) > } > > for base in bases: > for name in getattr(base, "__finalmethods__", set()): > value = getattr(cls, name, None) > if getattr(value, "__isfinalmethod__", False): > finals.add(name) > else: > raise TypeError("function '" + str(value.__name__) + "' in class '" + str(cls.__name__) + "' is final") > cls.__finalmethods__ = frozenset(finals) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Nov 16 12:19:14 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 16 Nov 2015 09:19:14 -0800 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: References: Message-ID: On Nov 15, 2015, at 22:17, Matt Chaput wrote: > > 2. It would be really nice if we could have "type aliasing" (or whatever it's called). I have a lot of types that are just something like "Tuple[int, int]", so type checking doesn't help much. It would be much more useful if I have a value that Python sees as (e.g.) an int, but have the type system track it as something more specific. Something like this: > > DocId = typing.TypeAlias(int) > DocLength = typing.TypeAlias(int) > > def id_and_length() -> Tuple[DocId, Length]: > docid = 5 # type: DocId > length = 10 # type: Length > return docid, length It sounds like you want a subtype that adds no new semantics or other runtime effects. Like this: class DocId(int): pass class DocLength(int): pass def id_and_length() -> Tuple[DocId, DocLength]: return DocId(5), DocLength(10) These will behave exactly like int objects, except that you can (dynamically or statically) type check them. It does mean that if someone uses "type(spam[0]) == int" it will fail, but I think if you care either way, you'd actually want it to fail. Meanwhile, "isinstance(spam[0], int)" or "spam[0] + eggs" or even using it in a function that requires something usable as a C long will work as expected. The object will also be the same size as an int in memory (although it will pickle a little bigger). It can't be optimized into a constant at compile time, but I doubt that's ever an issue. And it makes your intentions perfectly clear. From rosuav at gmail.com Mon Nov 16 12:29:00 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 17 Nov 2015 04:29:00 +1100 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: References: Message-ID: On Tue, Nov 17, 2015 at 4:19 AM, Andrew Barnert via Python-ideas wrote: > The object will also be the same size as an int in memory (although it will pickle a little bigger). To make that true, you have to use slots: >>> import sys >>> sys.getsizeof(12341234123412341234) 36 >>> class X(int): pass ... >>> sys.getsizeof(X(12341234123412341234)) 60 >>> class X(int): __slots__=() ... >>> sys.getsizeof(X(12341234123412341234)) 36 ChrisA From guido at python.org Mon Nov 16 12:31:39 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 16 Nov 2015 09:31:39 -0800 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: References: Message-ID: If you really want DocId to be as small as int you should add `__slots__ = ()` to the class def. It's still a bit slower (some code special-cases exactly int) and it also has some other imperfections -- e.g. DocId(2) + DocId(2) == int(4), not DocId(4). But that's probably okay for a document ID (and the solution is way too messy to use). But in general this is a pretty verbose option. If the main purpose is for documentation maybe type aliases, only used on annotations and other types are enough (`DocId = int`). But yeah, the type checker won't track it. (That's a possible future feature though.) On Mon, Nov 16, 2015 at 9:19 AM, Andrew Barnert via Python-ideas wrote: > On Nov 15, 2015, at 22:17, Matt Chaput wrote: >> >> 2. It would be really nice if we could have "type aliasing" (or whatever it's called). I have a lot of types that are just something like "Tuple[int, int]", so type checking doesn't help much. It would be much more useful if I have a value that Python sees as (e.g.) an int, but have the type system track it as something more specific. Something like this: >> >> DocId = typing.TypeAlias(int) >> DocLength = typing.TypeAlias(int) >> >> def id_and_length() -> Tuple[DocId, Length]: >> docid = 5 # type: DocId >> length = 10 # type: Length >> return docid, length > > It sounds like you want a subtype that adds no new semantics or other runtime effects. Like this: > > class DocId(int): pass > class DocLength(int): pass > > def id_and_length() -> Tuple[DocId, DocLength]: > return DocId(5), DocLength(10) > > These will behave exactly like int objects, except that you can (dynamically or statically) type check them. It does mean that if someone uses "type(spam[0]) == int" it will fail, but I think if you care either way, you'd actually want it to fail. Meanwhile, "isinstance(spam[0], int)" or "spam[0] + eggs" or even using it in a function that requires something usable as a C long will work as expected. The object will also be the same size as an int in memory (although it will pickle a little bigger). It can't be optimized into a constant at compile time, but I doubt that's ever an issue. And it makes your intentions perfectly clear. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From spinenkoia at gmail.com Mon Nov 16 13:15:55 2015 From: spinenkoia at gmail.com (=?UTF-8?B?0JjQstCw0L0g0KHQv9C40L3QtdC90LrQvg==?=) Date: Mon, 16 Nov 2015 23:15:55 +0500 Subject: [Python-ideas] The prohibition overrides the class method Message-ID: Thank you for the constructive responses, perhaps, I agree with you that this is not in the spirit of Python. As for the real example of the use, sometimes there to make the implementation of the method in the base class, but that none of the rest of the development team could not accidentally or unknowingly override this method (perhaps due to the need to be unsuccessful architecture). -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon Nov 16 13:24:51 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 16 Nov 2015 12:24:51 -0600 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: References: Message-ID: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com> You could always put this in the docstring: DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE. Short, the-the-point, and evokes fear. ... Seriously, though, I personally think this might go well with the whole PEP 484 type-annotation thing. I mean, if there are already generics and abstract methods. If we're going for static typing, adding final would fit well. Something like: @finalmethod def whatever(self): ... The type checker (in this case, Mypy) would check it, but there would be no runtime effect. On November 16, 2015 12:15:55 PM CST, "???? ????????" wrote: >Thank you for the constructive responses, perhaps, I agree with you >that >this is not in the spirit of Python. >As for the real example of the use, sometimes there to make the >implementation of the method in the base class, but that none of the >rest >of the development team could not accidentally or unknowingly override >this >method (perhaps due to the need to be unsuccessful architecture). > > >------------------------------------------------------------------------ > >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From joejev at gmail.com Mon Nov 16 14:19:28 2015 From: joejev at gmail.com (Joseph Jevnik) Date: Mon, 16 Nov 2015 14:19:28 -0500 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com> References: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com> Message-ID: def finalmethod(f): f.__doc__ += 'DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE!' return f On Mon, Nov 16, 2015 at 1:24 PM, Ryan Gonzalez wrote: > You could always put this in the docstring: > > DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE. > > > Short, the-the-point, and evokes fear. > > ... > > Seriously, though, I personally think this might go well with the whole > PEP 484 type-annotation thing. I mean, if there are already generics and > abstract methods. If we're going for static typing, adding final would fit > well. Something like: > > @finalmethod > def whatever(self): ... > > The type checker (in this case, Mypy) would check it, but there would be > no runtime effect. > > On November 16, 2015 12:15:55 PM CST, "???? ????????" < > spinenkoia at gmail.com> wrote: >> >> Thank you for the constructive responses, perhaps, I agree with you that >> this is not in the spirit of Python. >> As for the real example of the use, sometimes there to make the >> implementation of the method in the base class, but that none of the rest >> of the development team could not accidentally or unknowingly override this >> method (perhaps due to the need to be unsuccessful architecture). >> >> ------------------------------ >> >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > -- > Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Nov 16 16:59:20 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 16 Nov 2015 13:59:20 -0800 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com> References: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com> Message-ID: On Nov 16, 2015, at 10:24, Ryan Gonzalez wrote: > > You could always put this in the docstring: > > DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE. > > > Short, the-the-point, and evokes fear. Sure, until the first time someone accidentally violates it and doesn't die. Then you will have trained him to ignore your docstrings, comments, type annotations, and warnings completely. (Of course you could take the OP's code and add some stuff that causes the hard drive casing and spindle to break, firing the platter at the developer's skull like one of those discs from the Dolph Lundgren movie I Come in Peace, but that would break compatibility with computers that only have SSDs, and I don't think that's feasible in 2015.) > > ... > > Seriously, though, I personally think this might go well with the whole PEP 484 type-annotation thing. I mean, if there are already generics and abstract methods. If we're going for static typing, adding final would fit well. Something like: > > @finalmethod > def whatever(self): ... > > The type checker (in this case, Mypy) would check it, but there would be no runtime effect. Yes, that makes sense. If the feature isn't supposed to have any runtime effect, and is modeled after something that's definitely compile-time-centric in C++ and friends, signaling that by importing it from typing seems to be a pretty clear way to get that across. Although I still don't know why we need it. See below: >> On November 16, 2015 12:15:55 PM CST, "???? ????????" wrote: >> Thank you for the constructive responses, perhaps, I agree with you that this is not in the spirit of Python. >> As for the real example of the use, sometimes there to make the implementation of the method in the base class, but that none of the rest of the development team could not accidentally or unknowingly override this method (perhaps due to the need to be unsuccessful architecture). If it's just part of the implementation of the base class, just double-underscore it so that it can't be accidentally overridden (or called). It's only when something is also part of the interface that this is a problem. And it's kind of strange to have something be part of the interface of a type, but not be refinable by subtypes. When would you want that? (And as for accidental overriding, if a class claims doesn't even know what the interface of some supertype is, it probably shouldn't be inheriting?either it should be a mixin for someone who does know the supertype to compose, or it should be encapsulating rather than inheriting.) Do you have a concrete example of some code where you wanted finalmethod, and would use it to prevent the rest of your team from overriding it? -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Nov 16 17:16:38 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 16 Nov 2015 14:16:38 -0800 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: References: Message-ID: <670B43D6-2BB2-4CBB-AE74-26E4C57B7782@yahoo.com> On Nov 16, 2015, at 09:31, Guido van Rossum wrote: > > If you really want DocId to be as small as int you should add > `__slots__ = ()` to the class def. It's still a bit slower (some code > special-cases exactly int) and it also has some other imperfections -- > e.g. DocId(2) + DocId(2) == int(4), not DocId(4). But that's probably > okay for a document ID (and the solution is way too messy to use). Good points. But I don't think any of those things matter for the OP's use case. And surely being able to runtime-check the type and get the same results as compile-time checks, and not requiring any new language features, are advantages? I'm sure there are cases where the performance matters, but are there enough cases where the performance matters, runtime typing (e.g., in logs and debugger) doesn't matter, and static typing does matter? As a side note, most of the C++ family have something akin to "typedef" and/or "using" type aliases that are explicitly _not_ treated as different types by the static typing algorithm: if your function wants a DocId and I pass an int, you get neither an error nor a warning, because they are literally just different names for the same type. So, if Python really does need a feature that's stricter than what C++ and friends do, using a name that evokes comparisons to their feature is probably not a good idea. The way C++ does what the OP is asking for is exactly what I suggested: an empty class that publicly inherits the type. (Of course it only works with class types, not simple types like int, in C++. But Python only has class types, so that wouldn't be a problem.) > But in general this is a pretty verbose option. Even with __slots__, it's fewer characters, and the same number of concepts, as the OP's proposed solution. I think it's also more obvious to read: you're saying that DocId is a brand-new type that's a subtype of int (that adds nothing), which by definition means it can be used anywhere an int can be used, but not vice-versa. You don't have to know anything about MyPy or isinstance or anything else to figure out how they'll handle it (except the basic knowledge that Python's type system is generally a sensible OO system). > If the main purpose is > for documentation maybe type aliases, only used on annotations and > other types are enough (`DocId = int`). But yeah, the type checker > won't track it. (That's a possible future feature though.) > > On Mon, Nov 16, 2015 at 9:19 AM, Andrew Barnert via Python-ideas > wrote: >> On Nov 15, 2015, at 22:17, Matt Chaput wrote: >>> >>> 2. It would be really nice if we could have "type aliasing" (or whatever it's called). I have a lot of types that are just something like "Tuple[int, int]", so type checking doesn't help much. It would be much more useful if I have a value that Python sees as (e.g.) an int, but have the type system track it as something more specific. Something like this: >>> >>> DocId = typing.TypeAlias(int) >>> DocLength = typing.TypeAlias(int) >>> >>> def id_and_length() -> Tuple[DocId, Length]: >>> docid = 5 # type: DocId >>> length = 10 # type: Length >>> return docid, length >> >> It sounds like you want a subtype that adds no new semantics or other runtime effects. Like this: >> >> class DocId(int): pass >> class DocLength(int): pass >> >> def id_and_length() -> Tuple[DocId, DocLength]: >> return DocId(5), DocLength(10) >> >> These will behave exactly like int objects, except that you can (dynamically or statically) type check them. It does mean that if someone uses "type(spam[0]) == int" it will fail, but I think if you care either way, you'd actually want it to fail. Meanwhile, "isinstance(spam[0], int)" or "spam[0] + eggs" or even using it in a function that requires something usable as a C long will work as expected. The object will also be the same size as an int in memory (although it will pickle a little bigger). It can't be optimized into a constant at compile time, but I doubt that's ever an issue. And it makes your intentions perfectly clear. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido van Rossum (python.org/~guido) From guido at python.org Mon Nov 16 17:32:42 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 16 Nov 2015 14:32:42 -0800 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: <670B43D6-2BB2-4CBB-AE74-26E4C57B7782@yahoo.com> References: <670B43D6-2BB2-4CBB-AE74-26E4C57B7782@yahoo.com> Message-ID: On Mon, Nov 16, 2015 at 2:16 PM, Andrew Barnert wrote: > On Nov 16, 2015, at 09:31, Guido van Rossum wrote: >> >> If you really want DocId to be as small as int you should add >> `__slots__ = ()` to the class def. It's still a bit slower (some code >> special-cases exactly int) and it also has some other imperfections -- >> e.g. DocId(2) + DocId(2) == int(4), not DocId(4). But that's probably >> okay for a document ID (and the solution is way too messy to use). > > Good points. But I don't think any of those things matter for the OP's use case. Well, *you* claimed there was no size difference. > And surely being able to runtime-check the type and get the same results as compile-time checks, and not requiring any new language features, are advantages? Now you're changing the subject -- Matt very specifically asked for something that at runtime was just an int but was tracked more specifically by the type checker. We currently don't have that: there's either some runtime overhead (your solution) or the type checker doesn't track it (type aliasing as PEP 484 defines it). Anyway, lots of things PEP 484 tracks cannot be checked at runtime (e.g. anything involving TypeVar). > I'm sure there are cases where the performance matters, but are there enough cases where the performance matters, runtime typing (e.g., in logs and debugger) doesn't matter, and static typing does matter? A key requirement for PEP 484 is that there's no runtime overhead (apart from modules loading a tiny bit slower because the annotations are evaluated). Otherwise people will be afraid of using it. Having zillions of subclasses of builtin types at runtime just so the type checker can track them separately is in direct contradiction to this requirement. (Note that the BDFL-delegate specifically insisted that we remove isinstance() support from PEP-484's types.) > As a side note, most of the C++ family have something akin to "typedef" and/or "using" type aliases that are explicitly _not_ treated as different types by the static typing algorithm: if your function wants a DocId and I pass an int, you get neither an error nor a warning, because they are literally just different names for the same type. Yes, that's what PEP 484 type aliases do too. > So, if Python really does need a feature that's stricter than what C++ and friends do, using a name that evokes comparisons to their feature is probably not a good idea. The way C++ does what the OP is asking for is exactly what I suggested: an empty class that publicly inherits the type. (Of course it only works with class types, not simple types like int, in C++. But Python only has class types, so that wouldn't be a problem.) But C++ classes like that have no runtime overhead. The equivalent Python syntax does, alas. >> But in general this is a pretty verbose option. > > Even with __slots__, it's fewer characters, and the same number of concepts, as the OP's proposed solution. But such a hack -- everytime someone sees that they'll wonder why the `__slots__ = ()`. And AFAICT Matt didn't propose any solution -- he just showed some example code indicating what he wanted to be taken care of automatically. > I think it's also more obvious to read: you're saying that DocId is a brand-new type that's a subtype of int (that adds nothing), which by definition means it can be used anywhere an int can be used, but not vice-versa. You don't have to know anything about MyPy or isinstance or anything else to figure out how they'll handle it (except the basic knowledge that Python's type system is generally a sensible OO system). Yeah, but I'd still prefer a solution that is only read by the type checker. And defining subclasses of int is pretty uncommon, so it'll confuse the heck out of a lot of readers -- much more so than other parts of type annotations (which are ignorable). Anyway, let's stop bickering until Matt has had the time to read the thread and respond. -- --Guido van Rossum (python.org/~guido) From rymg19 at gmail.com Mon Nov 16 17:45:34 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 16 Nov 2015 16:45:34 -0600 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: References: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com> Message-ID: On Mon, Nov 16, 2015 at 3:59 PM, Andrew Barnert wrote: > On Nov 16, 2015, at 10:24, Ryan Gonzalez wrote: > > You could always put this in the docstring: > > DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE. > > > > Short, the-the-point, and evokes fear. > > > Sure, until the first time someone accidentally violates it and doesn't > die. Then you will have trained him to ignore your docstrings, comments, > type annotations, and warnings completely. (Of course you could take the > OP's code and add some stuff that causes the hard drive casing and spindle > to break, firing the platter at the developer's skull like one of those > discs from the Dolph Lundgren movie I Come in Peace, but that would break > compatibility with computers that only have SSDs, and I don't think that's > feasible in 2015.) > Well, it's just as easy to ignore type annotations and call `int(None)`. (You should definitely patent that hard drive stuff, though! ;) > > ... > > Seriously, though, I personally think this might go well with the whole > PEP 484 type-annotation thing. I mean, if there are already generics and > abstract methods. If we're going for static typing, adding final would fit > well. Something like: > > @finalmethod > def whatever(self): ... > > The type checker (in this case, Mypy) would check it, but there would be > no runtime effect. > > > Yes, that makes sense. If the feature isn't supposed to have any runtime > effect, and is modeled after something that's definitely > compile-time-centric in C++ and friends, signaling that by importing it > from typing seems to be a pretty clear way to get that across. > > Although I still don't know why we need it. See below: > > On November 16, 2015 12:15:55 PM CST, "???? ????????" < > spinenkoia at gmail.com> wrote: >> >> Thank you for the constructive responses, perhaps, I agree with you that >> this is not in the spirit of Python. >> As for the real example of the use, sometimes there to make the >> implementation of the method in the base class, but that none of the rest >> of the development team could not accidentally or unknowingly override this >> method (perhaps due to the need to be unsuccessful architecture). >> > > If it's just part of the implementation of the base class, just > double-underscore it so that it can't be accidentally overridden (or > called). > > It's only when something is also part of the interface that this is a > problem. And it's kind of strange to have something be part of the > interface of a type, but not be refinable by subtypes. When would you want > that? (And as for accidental overriding, if a class claims doesn't even > know what the interface of some supertype is, it probably shouldn't be > inheriting?either it should be a mixin for someone who does know the > supertype to compose, or it should be encapsulating rather than inheriting.) > > Do you have a concrete example of some code where you wanted finalmethod, > and would use it to prevent the rest of your team from overriding it? > What about http://programmers.stackexchange.com/a/288974/118754? -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Mon Nov 16 18:15:11 2015 From: mike at selik.org (Michael Selik) Date: Mon, 16 Nov 2015 23:15:11 +0000 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: References: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com> Message-ID: For that envelopes example on StackExchange, I'd probably use name mangling to reduce chances of subclassing mistakes and have prepare() call stuff(), seal(), stamp() in the correct order or whatever those were. I'm not sold in the need for "final". Btw, why does my iPad not have a back tick on the keyboard? My Android can do it... On Mon, Nov 16, 2015 at 5:53 PM Ryan Gonzalez wrote: > On Mon, Nov 16, 2015 at 3:59 PM, Andrew Barnert > wrote: > >> On Nov 16, 2015, at 10:24, Ryan Gonzalez wrote: >> >> You could always put this in the docstring: >> >> DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE. >> >> >> >> Short, the-the-point, and evokes fear. >> >> >> Sure, until the first time someone accidentally violates it and doesn't >> die. Then you will have trained him to ignore your docstrings, comments, >> type annotations, and warnings completely. (Of course you could take the >> OP's code and add some stuff that causes the hard drive casing and spindle >> to break, firing the platter at the developer's skull like one of those >> discs from the Dolph Lundgren movie I Come in Peace, but that would break >> compatibility with computers that only have SSDs, and I don't think that's >> feasible in 2015.) >> > > Well, it's just as easy to ignore type annotations and call `int(None)`. > > (You should definitely patent that hard drive stuff, though! ;) > > >> >> ... >> >> Seriously, though, I personally think this might go well with the whole >> PEP 484 type-annotation thing. I mean, if there are already generics and >> abstract methods. If we're going for static typing, adding final would fit >> well. Something like: >> >> @finalmethod >> def whatever(self): ... >> >> The type checker (in this case, Mypy) would check it, but there would be >> no runtime effect. >> >> >> Yes, that makes sense. If the feature isn't supposed to have any runtime >> effect, and is modeled after something that's definitely >> compile-time-centric in C++ and friends, signaling that by importing it >> from typing seems to be a pretty clear way to get that across. >> >> Although I still don't know why we need it. See below: >> >> On November 16, 2015 12:15:55 PM CST, "???? ????????" < >> spinenkoia at gmail.com> wrote: >>> >>> Thank you for the constructive responses, perhaps, I agree with you that >>> this is not in the spirit of Python. >>> As for the real example of the use, sometimes there to make the >>> implementation of the method in the base class, but that none of the rest >>> of the development team could not accidentally or unknowingly override this >>> method (perhaps due to the need to be unsuccessful architecture). >>> >> >> If it's just part of the implementation of the base class, just >> double-underscore it so that it can't be accidentally overridden (or >> called). >> >> It's only when something is also part of the interface that this is a >> problem. And it's kind of strange to have something be part of the >> interface of a type, but not be refinable by subtypes. When would you want >> that? (And as for accidental overriding, if a class claims doesn't even >> know what the interface of some supertype is, it probably shouldn't be >> inheriting?either it should be a mixin for someone who does know the >> supertype to compose, or it should be encapsulating rather than inheriting.) >> >> Do you have a concrete example of some code where you wanted finalmethod, >> and would use it to prevent the rest of your team from overriding it? >> > > What about http://programmers.stackexchange.com/a/288974/118754? > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your > program. Something?s wrong. > http://kirbyfan64.github.io/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Mon Nov 16 18:24:55 2015 From: mike at selik.org (Michael Selik) Date: Mon, 16 Nov 2015 23:24:55 +0000 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: References: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com> Message-ID: Ryan, are you saying that def helper(self): pass __helper = helper Would not be sufficient protection against clumsy subclassing? On Mon, Nov 16, 2015 at 6:22 PM Ryan Gonzalez wrote: > On Mon, Nov 16, 2015 at 5:15 PM, Michael Selik wrote: > >> For that envelopes example on StackExchange, I'd probably use name >> mangling to reduce chances of subclassing mistakes and have prepare() call >> stuff(), seal(), stamp() in the correct order or whatever those were. I'm >> not sold in the need for "final". >> >> > Maybe the subclasses should be able to access the functions. They just > shouldn't override it. > > >> Btw, why does my iPad not have a back tick on the keyboard? My Android >> can do it... >> >> > Android > iPad. > > *ducks from incoming bullets* > > >> On Mon, Nov 16, 2015 at 5:53 PM Ryan Gonzalez wrote: >> >>> On Mon, Nov 16, 2015 at 3:59 PM, Andrew Barnert >>> wrote: >>> >>>> On Nov 16, 2015, at 10:24, Ryan Gonzalez wrote: >>>> >>>> You could always put this in the docstring: >>>> >>>> DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE. >>>> >>>> >>>> >>>> Short, the-the-point, and evokes fear. >>>> >>>> >>>> Sure, until the first time someone accidentally violates it and doesn't >>>> die. Then you will have trained him to ignore your docstrings, comments, >>>> type annotations, and warnings completely. (Of course you could take >>>> the OP's code and add some stuff that causes the hard drive casing and >>>> spindle to break, firing the platter at the developer's skull like one of >>>> those discs from the Dolph Lundgren movie I Come in Peace, but that would >>>> break compatibility with computers that only have SSDs, and I don't think >>>> that's feasible in 2015.) >>>> >>> >>> Well, it's just as easy to ignore type annotations and call `int(None)`. >>> >>> (You should definitely patent that hard drive stuff, though! ;) >>> >>> >>>> >>>> ... >>>> >>>> Seriously, though, I personally think this might go well with the whole >>>> PEP 484 type-annotation thing. I mean, if there are already generics and >>>> abstract methods. If we're going for static typing, adding final would fit >>>> well. Something like: >>>> >>>> @finalmethod >>>> def whatever(self): ... >>>> >>>> The type checker (in this case, Mypy) would check it, but there would >>>> be no runtime effect. >>>> >>>> >>>> Yes, that makes sense. If the feature isn't supposed to have any >>>> runtime effect, and is modeled after something that's definitely >>>> compile-time-centric in C++ and friends, signaling that by importing it >>>> from typing seems to be a pretty clear way to get that across. >>>> >>>> Although I still don't know why we need it. See below: >>>> >>>> On November 16, 2015 12:15:55 PM CST, "???? ????????" < >>>> spinenkoia at gmail.com> wrote: >>>>> >>>>> Thank you for the constructive responses, perhaps, I agree with you >>>>> that this is not in the spirit of Python. >>>>> As for the real example of the use, sometimes there to make the >>>>> implementation of the method in the base class, but that none of the rest >>>>> of the development team could not accidentally or unknowingly override this >>>>> method (perhaps due to the need to be unsuccessful architecture). >>>>> >>>> >>>> If it's just part of the implementation of the base class, just >>>> double-underscore it so that it can't be accidentally overridden (or >>>> called). >>>> >>>> It's only when something is also part of the interface that this is a >>>> problem. And it's kind of strange to have something be part of the >>>> interface of a type, but not be refinable by subtypes. When would you want >>>> that? (And as for accidental overriding, if a class claims doesn't even >>>> know what the interface of some supertype is, it probably shouldn't be >>>> inheriting?either it should be a mixin for someone who does know the >>>> supertype to compose, or it should be encapsulating rather than inheriting.) >>>> >>>> Do you have a concrete example of some code where you wanted >>>> finalmethod, and would use it to prevent the rest of your team from >>>> overriding it? >>>> >>> >>> What about http://programmers.stackexchange.com/a/288974/118754? >>> >>> -- >>> Ryan >>> [ERROR]: Your autotools build scripts are 200 lines longer than your >>> program. Something?s wrong. >>> http://kirbyfan64.github.io/ >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your > program. Something?s wrong. > http://kirbyfan64.github.io/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Mon Nov 16 18:39:27 2015 From: random832 at fastmail.com (Random832) Date: Mon, 16 Nov 2015 18:39:27 -0500 Subject: [Python-ideas] The prohibition overrides the class method References: Message-ID: I'm not sure how this thread has gone on this long without anyone mentioning __methods. If you prefix a method with two underscores, it will transform the name of the method in order to make it difficult to accidentally override. So in other methods inside the base class that need to call your method, you can call the underscored version and subclasses won't override it (I think it's possible for them to e.g. class A: def __test(self): print("123") def test(self): return self.__internal_test() def other_function(self): ... self.__test() ... class B(A): def __test(self): print("456") >>> B().test() 123 B can still override test if it wants to, but it won't affect any calls to the __test method from A's other methods. If it *really* wants to, it can override _A__test(), but someone's not likely to do that by mistake. From ian.g.kelly at gmail.com Mon Nov 16 18:40:26 2015 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 16 Nov 2015 16:40:26 -0700 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: References: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com> Message-ID: On Nov 16, 2015 4:15 PM, "Michael Selik" wrote: > > For that envelopes example on StackExchange, I'd probably use name mangling to reduce chances of subclassing mistakes and have prepare() call stuff(), seal(), stamp() in the correct order or whatever those were. I'm not sold in the need for "final". Name mangling would not work in that example. The prepare method is defined by a base class and overridden in a subclass; it is the subclass that wishes to declare the method final. Name mangling would obstruct overriding the method at all, which would be a problem for the base class that invites the method to be overridden. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Nov 16 17:09:59 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 17 Nov 2015 11:09:59 +1300 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com> References: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com> Message-ID: <564A5437.7000704@canterbury.ac.nz> Ryan Gonzalez wrote: > You could always put this in the docstring: > > DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE. And don't turn on the headlight, either. :-) -- Greg From mike at selik.org Mon Nov 16 19:12:58 2015 From: mike at selik.org (Michael Selik) Date: Tue, 17 Nov 2015 00:12:58 +0000 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: References: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com> Message-ID: Thanks, Ian, I missed that detail. While my initial thought was that name mangling would achieve the desired level of subclassing-clumsiness defense. As below... # using 2 steps instead of 3, for brevity class Envelope: def prepare(self): print('omnibus!') class Child(Envelope): def stuff(self): print('step 1') __stuff = stuff def seal(self): print('step 2') __seal = seal def prepare(self): self.__stuff() self.__seal() The post on StackExchange added the extra wrinkle that both classes have C++ virtual functions, kind-of like this... from abc import ABC, abstractmethod class Envelope(ABC): def something_useful(self): ''' I'd normally avoid tricky inheritance but there must be some really useful code in here that I don't want to refactor ''' @abstractmethod def prepare(self): pass class Child(Envelope): @abstractmethod def stuff(self): print('step 1') @abstractmethod def seal(self): print('step 2') def prepare(self): 'should be FINAL, Override at your OWN PERIL' self.stuff() self.seal() This contrived scenario seems better resolved by refactoring the `Envelope` class. Moreover, I don't understand why the `Child` class would exist at all. Why override `prepare` only to create more abstract methods? On Mon, Nov 16, 2015 at 6:40 PM Ian Kelly wrote: > On Nov 16, 2015 4:15 PM, "Michael Selik" wrote: > > > > For that envelopes example on StackExchange, I'd probably use name > mangling to reduce chances of subclassing mistakes and have prepare() call > stuff(), seal(), stamp() in the correct order or whatever those were. I'm > not sold in the need for "final". > > Name mangling would not work in that example. The prepare method is > defined by a base class and overridden in a subclass; it is the subclass > that wishes to declare the method final. Name mangling would obstruct > overriding the method at all, which would be a problem for the base class > that invites the method to be overridden. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Nov 16 20:27:03 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 16 Nov 2015 17:27:03 -0800 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: References: <670B43D6-2BB2-4CBB-AE74-26E4C57B7782@yahoo.com> Message-ID: <3007695256009891190@unknownmsgid> >> As a side note, most of the C++ family have something akin to "typedef" and/or "using" type aliases that are explicitly _not_ treated as different types by the static typing algorithm: ... > Yes, that's what PEP 484 type aliases do too. Darn -- C typedefs have their uses, but improving type safety is not one of them. PEP 484 is all about type checking/safety -- it seems it would be a lot more useful if aliases were treated as different types. Oh well. CHB From matt at whoosh.ca Mon Nov 16 20:35:04 2015 From: matt at whoosh.ca (Matt Chaput) Date: Mon, 16 Nov 2015 20:35:04 -0500 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: References: <670B43D6-2BB2-4CBB-AE74-26E4C57B7782@yahoo.com> Message-ID: > Yeah, but I'd still prefer a solution that is only read by the type > checker. And defining subclasses of int is pretty uncommon, so it'll > confuse the heck out of a lot of readers -- much more so than other > parts of type annotations (which are ignorable). I think a solution that only affected the type system would be much better, but it's cool that subclassing int exists as a workaround (didn't know that would work :). I'll have to decide if it's worth the trouble for me. Thanks all of you for learning me a thing! Matt From guido at python.org Mon Nov 16 20:35:42 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 16 Nov 2015 17:35:42 -0800 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: <3007695256009891190@unknownmsgid> References: <670B43D6-2BB2-4CBB-AE74-26E4C57B7782@yahoo.com> <3007695256009891190@unknownmsgid> Message-ID: No, PEP 484 is about a pragmatic compromise that doesn't require adding new syntax and yet allows a reasonable amount of checking. Type checking religion has no place here. On Mon, Nov 16, 2015 at 5:27 PM, Chris Barker - NOAA Federal wrote: >>> As a side note, most of the C++ family have something akin to "typedef" and/or "using" type aliases that are explicitly _not_ treated as different types by the static typing algorithm: > > ... >> Yes, that's what PEP 484 type aliases do too. > > Darn -- C typedefs have their uses, but improving type safety is not > one of them. > > PEP 484 is all about type checking/safety -- it seems it would be a > lot more useful if aliases were treated as different types. > > Oh well. > > CHB -- --Guido van Rossum (python.org/~guido) From rymg19 at gmail.com Mon Nov 16 18:21:57 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 16 Nov 2015 17:21:57 -0600 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: References: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com> Message-ID: On Mon, Nov 16, 2015 at 5:15 PM, Michael Selik wrote: > For that envelopes example on StackExchange, I'd probably use name > mangling to reduce chances of subclassing mistakes and have prepare() call > stuff(), seal(), stamp() in the correct order or whatever those were. I'm > not sold in the need for "final". > > Maybe the subclasses should be able to access the functions. They just shouldn't override it. > Btw, why does my iPad not have a back tick on the keyboard? My Android can > do it... > > Android > iPad. *ducks from incoming bullets* > On Mon, Nov 16, 2015 at 5:53 PM Ryan Gonzalez wrote: > >> On Mon, Nov 16, 2015 at 3:59 PM, Andrew Barnert >> wrote: >> >>> On Nov 16, 2015, at 10:24, Ryan Gonzalez wrote: >>> >>> You could always put this in the docstring: >>> >>> DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE. >>> >>> >>> >>> Short, the-the-point, and evokes fear. >>> >>> >>> Sure, until the first time someone accidentally violates it and doesn't >>> die. Then you will have trained him to ignore your docstrings, comments, >>> type annotations, and warnings completely. (Of course you could take >>> the OP's code and add some stuff that causes the hard drive casing and >>> spindle to break, firing the platter at the developer's skull like one of >>> those discs from the Dolph Lundgren movie I Come in Peace, but that would >>> break compatibility with computers that only have SSDs, and I don't think >>> that's feasible in 2015.) >>> >> >> Well, it's just as easy to ignore type annotations and call `int(None)`. >> >> (You should definitely patent that hard drive stuff, though! ;) >> >> >>> >>> ... >>> >>> Seriously, though, I personally think this might go well with the whole >>> PEP 484 type-annotation thing. I mean, if there are already generics and >>> abstract methods. If we're going for static typing, adding final would fit >>> well. Something like: >>> >>> @finalmethod >>> def whatever(self): ... >>> >>> The type checker (in this case, Mypy) would check it, but there would be >>> no runtime effect. >>> >>> >>> Yes, that makes sense. If the feature isn't supposed to have any runtime >>> effect, and is modeled after something that's definitely >>> compile-time-centric in C++ and friends, signaling that by importing it >>> from typing seems to be a pretty clear way to get that across. >>> >>> Although I still don't know why we need it. See below: >>> >>> On November 16, 2015 12:15:55 PM CST, "???? ????????" < >>> spinenkoia at gmail.com> wrote: >>>> >>>> Thank you for the constructive responses, perhaps, I agree with you >>>> that this is not in the spirit of Python. >>>> As for the real example of the use, sometimes there to make the >>>> implementation of the method in the base class, but that none of the rest >>>> of the development team could not accidentally or unknowingly override this >>>> method (perhaps due to the need to be unsuccessful architecture). >>>> >>> >>> If it's just part of the implementation of the base class, just >>> double-underscore it so that it can't be accidentally overridden (or >>> called). >>> >>> It's only when something is also part of the interface that this is a >>> problem. And it's kind of strange to have something be part of the >>> interface of a type, but not be refinable by subtypes. When would you want >>> that? (And as for accidental overriding, if a class claims doesn't even >>> know what the interface of some supertype is, it probably shouldn't be >>> inheriting?either it should be a mixin for someone who does know the >>> supertype to compose, or it should be encapsulating rather than inheriting.) >>> >>> Do you have a concrete example of some code where you wanted >>> finalmethod, and would use it to prevent the rest of your team from >>> overriding it? >>> >> >> What about http://programmers.stackexchange.com/a/288974/118754? >> >> -- >> Ryan >> [ERROR]: Your autotools build scripts are 200 lines longer than your >> program. Something?s wrong. >> http://kirbyfan64.github.io/ >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Nov 16 21:21:28 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 16 Nov 2015 18:21:28 -0800 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: References: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com> Message-ID: <422CD3EE-03A0-4EDF-B2FF-D71A0DF965A3@yahoo.com> On Nov 16, 2015, at 16:12, Michael Selik wrote: > > Thanks, Ian, I missed that detail. > > While my initial thought was that name mangling would achieve the desired level of subclassing-clumsiness defense. As below... > > # using 2 steps instead of 3, for brevity > > class Envelope: > def prepare(self): > print('omnibus!') > > class Child(Envelope): > > def stuff(self): > print('step 1') > __stuff = stuff > > def seal(self): > print('step 2') > __seal = seal > > def prepare(self): > self.__stuff() > self.__seal() > > > The post on StackExchange added the extra wrinkle that both classes have C++ virtual functions, kind-of like this... > > from abc import ABC, abstractmethod > > class Envelope(ABC): > > def something_useful(self): > ''' I'd normally avoid tricky inheritance > but there must be some really useful code > in here that I don't want to refactor > ''' > > @abstractmethod > def prepare(self): > pass > > > class Child(Envelope): > > @abstractmethod > def stuff(self): > print('step 1') > > @abstractmethod > def seal(self): > print('step 2') > > def prepare(self): > 'should be FINAL, Override at your OWN PERIL' > self.stuff() > self.seal() > > This contrived scenario seems better resolved by refactoring the `Envelope` class. Moreover, I don't understand why the `Child` class would exist at all. Why override `prepare` only to create more abstract methods? Apparently, because they want to use the strategy pattern, twice, but in a way that gets none of the advantages of that pattern. To actually get the benefits, those new abstract methods would not be in Child; instead, you'd create a new EnvelopePrepaper interface that Child uses. Why can't you just merge the EnvelopePreparer into Child itself? Because then you lose all the decoupling benefits that the strategy pattern is meant to provide, and you add new problems in that a single class now has multiple separate interfaces that can accidentally step on each other's toes. In fact, what they clearly wanted was for Envelope to use a preparer strategy, and for Child to _be_ such a preparer strategy that itself uses a 2-stage-preparer strategy, and for other people to implement 2-stage-preparer strategies. But then they've merged both interfaces into the class that uses those strategies instead for no good reason. If you fix either of those problems, or just use a simpler design instead of misusing strategy, or even if you only misuse strategy once instead of multiple times, the problem wouldn't arise. In other words, it's a subset of exactly the problem I suggested before seeing the example: If you want to inherit from a class, you have to know that class's interface, because you're promising to implement it. This design forces users to inherit from a class whose interface they don't know, and then hopes for a new language feature to hide one of the symptoms of that problem instead of fixing the problem. >> On Mon, Nov 16, 2015 at 6:40 PM Ian Kelly wrote: >> On Nov 16, 2015 4:15 PM, "Michael Selik" wrote: >> > >> > For that envelopes example on StackExchange, I'd probably use name mangling to reduce chances of subclassing mistakes and have prepare() call stuff(), seal(), stamp() in the correct order or whatever those were. I'm not sold in the need for "final". >> >> Name mangling would not work in that example. The prepare method is defined by a base class and overridden in a subclass; it is the subclass that wishes to declare the method final. Name mangling would obstruct overriding the method at all, which would be a problem for the base class that invites the method to be overridden. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at whoosh.ca Mon Nov 16 21:41:45 2015 From: matt at whoosh.ca (Matt Chaput) Date: Mon, 16 Nov 2015 21:41:45 -0500 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: References: Message-ID: I forgot to say, there's been a couple of places where I haven't been able to add hinting that hopefully can be improved: 1. Standard library types that should be public but aren't (regular expression object) 2. Specifying an argument/return that's a class (that should be a subclass of some class). There might be a way to do 2 that I don't know about :) Cheers, Matt From guido at python.org Mon Nov 16 21:51:38 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 16 Nov 2015 18:51:38 -0800 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: References: Message-ID: #1: typing.re defines the re types. For the others, the typeshed project takes contributions. #2: You can use def f() -> type: ... to specify a class as return type; but we currently don't have a way to contrain that class. On Mon, Nov 16, 2015 at 6:41 PM, Matt Chaput wrote: > I forgot to say, there's been a couple of places where I haven't been able to add hinting that hopefully can be improved: > > 1. Standard library types that should be public but aren't (regular expression object) > 2. Specifying an argument/return that's a class (that should be a subclass of some class). > > There might be a way to do 2 that I don't know about :) > > Cheers, > > Matt > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From rymg19 at gmail.com Mon Nov 16 22:36:22 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 16 Nov 2015 21:36:22 -0600 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: References: Message-ID: On November 16, 2015 8:51:38 PM CST, Guido van Rossum wrote: >#1: typing.re defines the re types. For the others, the typeshed >project takes contributions. > >#2: You can use def f() -> type: ... to specify a class as return >type; but we currently don't have a way to contrain that class. > What about a typing.Class or typing.Type generic? Like: def f() -> Type[MyBaseClass]: ... >On Mon, Nov 16, 2015 at 6:41 PM, Matt Chaput wrote: >> I forgot to say, there's been a couple of places where I haven't been >able to add hinting that hopefully can be improved: >> >> 1. Standard library types that should be public but aren't (regular >expression object) >> 2. Specifying an argument/return that's a class (that should be a >subclass of some class). >> >> There might be a way to do 2 that I don't know about :) >> >> Cheers, >> >> Matt >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. From guido at python.org Mon Nov 16 22:39:33 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 16 Nov 2015 19:39:33 -0800 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: References: Message-ID: There's an existing (but postponed) proposal to use Type[X]: https://github.com/ambv/typehinting/issues/107 On Mon, Nov 16, 2015 at 7:36 PM, Ryan Gonzalez wrote: > > > On November 16, 2015 8:51:38 PM CST, Guido van Rossum wrote: >>#1: typing.re defines the re types. For the others, the typeshed >>project takes contributions. >> >>#2: You can use def f() -> type: ... to specify a class as return >>type; but we currently don't have a way to contrain that class. >> > > What about a typing.Class or typing.Type generic? Like: > > def f() -> Type[MyBaseClass]: ... > >>On Mon, Nov 16, 2015 at 6:41 PM, Matt Chaput wrote: >>> I forgot to say, there's been a couple of places where I haven't been >>able to add hinting that hopefully can be improved: >>> >>> 1. Standard library types that should be public but aren't (regular >>expression object) >>> 2. Specifying an argument/return that's a class (that should be a >>subclass of some class). >>> >>> There might be a way to do 2 that I don't know about :) >>> >>> Cheers, >>> >>> Matt >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. -- --Guido van Rossum (python.org/~guido) From spinenkoia at gmail.com Tue Nov 17 00:30:01 2015 From: spinenkoia at gmail.com (=?UTF-8?B?0JjQstCw0L0g0KHQv9C40L3QtdC90LrQvg==?=) Date: Tue, 17 Nov 2015 10:30:01 +0500 Subject: [Python-ideas] The prohibition overrides the class method Message-ID: Thank you all very much. Perhaps I'll leave that idea, Python too flexible language for such things. -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Nov 17 06:17:42 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 17 Nov 2015 12:17:42 +0100 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: References: Message-ID: 2015-11-16 16:03 GMT+01:00 ???? ???????? : > I propose to make an analogue of the final specifier in C ++. In class > abc.ABCMeta has decorator abstractmethod, you can add decorator finalmethod. Would it be allowed to override the method in the class where the method is defined using unittest.mock? Mocking looks popular and importang in the Python community to write unit tests. Victor From matt at whoosh.ca Tue Nov 17 11:25:18 2015 From: matt at whoosh.ca (Matt Chaput) Date: Tue, 17 Nov 2015 11:25:18 -0500 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: References: Message-ID: > #1: typing.re defines the re types. For the others, the typeshed > project takes contributions. Me: Really? I don't remember seeing... Me: (Searches pep484 document) Me: (Embarrassed) Me: (Crawls in hole) Thanks! :) Matt From rymg19 at gmail.com Tue Nov 17 12:40:09 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 17 Nov 2015 11:40:09 -0600 Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484) In-Reply-To: References: Message-ID: <111A0DAD-8B32-494E-BD65-5507806E867F@gmail.com> On November 17, 2015 10:25:18 AM CST, Matt Chaput wrote: >> #1: typing.re defines the re types. For the others, the typeshed >> project takes contributions. > >Me: Really? I don't remember seeing... >Me: (Searches pep484 document) >Me: (Embarrassed) >Me: (Crawls in hole) > >Thanks! :) > Don't worry; I do stuff like that *all the time*. Literally. >Matt > > >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. From mike at selik.org Tue Nov 17 17:48:37 2015 From: mike at selik.org (Michael Selik) Date: Tue, 17 Nov 2015 22:48:37 +0000 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: References: Message-ID: Cython has a `final` for preventing a class from being overridden ( http://docs.cython.org/src/userguide/extension_types.html#subclassing). I don't know if Cython supports `final` methods outside of final classes. They might be more interested in implementing that feature. On Tue, Nov 17, 2015, 12:30 AM ???? ???????? wrote: > Thank you all very much. Perhaps I'll leave that idea, Python too flexible > language for such things. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Nov 17 21:41:29 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 18 Nov 2015 12:41:29 +1000 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: References: Message-ID: On 18 November 2015 at 08:48, Michael Selik wrote: > Cython has a `final` for preventing a class from being overridden > (http://docs.cython.org/src/userguide/extension_types.html#subclassing). I > don't know if Cython supports `final` methods outside of final classes. They > might be more interested in implementing that feature. +1, as a lot of Cython's power comes from allowing developers to deliberately (and selectively) *opt out* of Python's dynamic features. They even provide an annotation mode that shows how far you've progressed in rendering a section of code entirely static: http://docs.cython.org/src/quickstart/cythonize.html#determining-where-to-add-types Regards, Nick. P.S. For folks interested in learning more about that, I'd recommend Caleb Hattingh's PyCon Australia talk about using Cython to speed up and parallelise code hotspots: http://docs.cython.org/src/quickstart/cythonize.html#determining-where-to-add-types -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Nov 17 22:18:17 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 18 Nov 2015 13:18:17 +1000 Subject: [Python-ideas] The prohibition overrides the class method In-Reply-To: References: Message-ID: On 18 November 2015 at 12:41, Nick Coghlan wrote: > P.S. For folks interested in learning more about that, I'd recommend > Caleb Hattingh's PyCon Australia talk about using Cython to speed up > and parallelise code hotspots: > http://docs.cython.org/src/quickstart/cythonize.html#determining-where-to-add-types Oops, the actual video link: https://www.youtube.com/watch?v=NfnMJMkhDoQ Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From me at jeltef.nl Mon Nov 23 14:09:17 2015 From: me at jeltef.nl (Jelte Fennema) Date: Mon, 23 Nov 2015 20:09:17 +0100 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators Message-ID: Hi, After reading PEP0465 about the dedicated matrix multiplication I started wondering if the same solution couldn't be applied to boolean operators as well. There currently are a lot of high profile libraries that have their own functions for boolean operators, like Numpy, Pandas or SQLAlchemy. They do this because the current boolean operators can't be overloaded. PEP0335 was created to solve this problem (and makes the problem more clear), but was rejected because it needed changes to the byte code for the boolean operators, which would make them slower. Currently some of these libraries resort to the bitwise operators (at least Pandas), but those don't bind as strong as comparison operators, which means you have to do comparisons like this: (series1 == 2) & (series2 == 3) That is why I propose to create new operators just like for matrix multiplication which can be used in libraries that need one. I'm not sure what the operators should look like, but my first guess would be &&, || and ! for and, or and not respectively. Is this an idea that sounds reasonable? Jelte -------------- next part -------------- An HTML attachment was scrubbed... URL: From vgr255 at live.ca Mon Nov 23 15:22:53 2015 From: vgr255 at live.ca (Emanuel Barry) Date: Mon, 23 Nov 2015 15:22:53 -0500 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: Could you provide some links to where this could be useful, and how code could be rewritten? I can see the desire for such a feature, I myself would have liked such an operator or keyword. If you get general approval on this list, you can then move on to write a PEP (as that's what's needed if you wish to add a new keyword and/or operator to the language). I'm +0 for now, and may change once you provide us with use cases in the wild.-Emanuel From: me at jeltef.nl Date: Mon, 23 Nov 2015 20:09:17 +0100 To: python-ideas at python.org Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators Hi, After reading PEP0465 about the dedicated matrix multiplication I started wondering if the same solution couldn't be applied to boolean operators as well. There currently are a lot of high profile libraries that have their own functions for boolean operators, like Numpy, Pandas or SQLAlchemy. They do this because the current boolean operators can't be overloaded. PEP0335 was created to solve this problem (and makes the problem more clear), but was rejected because it needed changes to the byte code for the boolean operators, which would make them slower. Currently some of these libraries resort to the bitwise operators (at least Pandas), but those don't bind as strong as comparison operators, which means you have to do comparisons like this: (series1 == 2) & (series2 == 3) That is why I propose to create new operators just like for matrix multiplication which can be used in libraries that need one. I'm not sure what the operators should look like, but my first guess would be &&, || and ! for and, or and not respectively. Is this an idea that sounds reasonable? Jelte _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at jeltef.nl Mon Nov 23 18:49:30 2015 From: me at jeltef.nl (Jelte Fennema) Date: Mon, 23 Nov 2015 15:49:30 -0800 (PST) Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: Some examples of current practice for SQLAlchemy (an SQL ORM) can be found here: http://docs.sqlalchemy.org/en/rel_1_0/orm/tutorial.html#common-filter-operators An slightly adapted example is this: from sqlalchemy import or_, and_ query.filter(or_(User.name == 'ed', and_(User.name == 'wendy', User.age > 20))) With the new operators this could simply be rewritten to: query.filter(User.name == 'ed' || (User.name == 'wendy' && User.age > 20)) This is much clearer in my opinion. Pandas overloads the binary and and or operators, which causes the small issue that it needs an extra pair of braces around expressions, see https://stackoverflow.com/questions/24775648/element-wise-logcial-or-in-pandas or http://stackoverflow.com/a/19581644/2570866 This means that this (which selects the rows that either are lower than three or equal to five): df[(df < 3) | (df == 5)] Can be rewritten to this: df[df < 3 || df == 5] This is clearly little advantage, but it also means that there is no need to override the binary or and and. That way they can be used for their original purpose. For Numpy the case is again like with SQLAlchemy see : https://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_and.html and https://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_or.html I hope this made the use for the overloadable logical and and or operators clear. The not operator might be a bit less useful and I'm not sure it's needed as much. Currently SQLAlchemy and Pandas overload the "~" (invert) operator and Numpy has a function again: https://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_not.html Lastly, for SQLAlchemy an in operator that does not return a boolean could also be useful. I can't think of use cases for the others though and I also can't directly think of an operator that would be as clear as the &&, || and ! operators. As for the PEP, I have no problem writing one if this is accepted as a useful addition. Also any suggestions and critiques are very welcome of course. On Monday, 23 November 2015 21:24:23 UTC+1, Emanuel Barry wrote: > > Could you provide some links to where this could be useful, and how code > could be rewritten? I can see the desire for such a feature, I myself would > have liked such an operator or keyword. If you get general approval on this > list, you can then move on to write a PEP (as that's what's needed if you > wish to add a new keyword and/or operator to the language). > > I'm +0 for now, and may change once you provide us with use cases in the > wild. > -Emanuel > > ------------------------------ > From: m... at jeltef.nl > Date: Mon, 23 Nov 2015 20:09:17 +0100 > To: python... at python.org > Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean > operators > > Hi, > > After reading PEP0465 about > the dedicated matrix multiplication I started wondering if the same > solution couldn't be applied to boolean operators as well. There currently > are a lot of high profile libraries that have their own functions for > boolean operators, like Numpy, Pandas or SQLAlchemy. They do this because > the current boolean operators can't be overloaded. PEP0335 > was created to solve this > problem (and makes the problem more clear), but was rejected because it > needed changes to the byte code for the boolean operators, which would make > them slower. Currently some of these libraries resort to the bitwise > operators (at least Pandas), but those don't bind as strong as comparison > operators, which means you have to do comparisons like this: (series1 == 2) > & (series2 == 3) > > That is why I propose to create new operators just like for matrix > multiplication which can be used in libraries that need one. I'm not sure > what the operators should look like, but my first guess would be &&, || and > ! for and, or and not respectively. > > Is this an idea that sounds reasonable? > > Jelte > > _______________________________________________ Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: > http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at jeltef.nl Mon Nov 23 18:49:30 2015 From: me at jeltef.nl (Jelte Fennema) Date: Mon, 23 Nov 2015 15:49:30 -0800 (PST) Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: Some examples of current practice for SQLAlchemy (an SQL ORM) can be found here: http://docs.sqlalchemy.org/en/rel_1_0/orm/tutorial.html#common-filter-operators An slightly adapted example is this: from sqlalchemy import or_, and_ query.filter(or_(User.name == 'ed', and_(User.name == 'wendy', User.age > 20))) With the new operators this could simply be rewritten to: query.filter(User.name == 'ed' || (User.name == 'wendy' && User.age > 20)) This is much clearer in my opinion. Pandas overloads the binary and and or operators, which causes the small issue that it needs an extra pair of braces around expressions, see https://stackoverflow.com/questions/24775648/element-wise-logcial-or-in-pandas or http://stackoverflow.com/a/19581644/2570866 This means that this (which selects the rows that either are lower than three or equal to five): df[(df < 3) | (df == 5)] Can be rewritten to this: df[df < 3 || df == 5] This is clearly little advantage, but it also means that there is no need to override the binary or and and. That way they can be used for their original purpose. For Numpy the case is again like with SQLAlchemy see : https://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_and.html and https://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_or.html I hope this made the use for the overloadable logical and and or operators clear. The not operator might be a bit less useful and I'm not sure it's needed as much. Currently SQLAlchemy and Pandas overload the "~" (invert) operator and Numpy has a function again: https://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_not.html Lastly, for SQLAlchemy an in operator that does not return a boolean could also be useful. I can't think of use cases for the others though and I also can't directly think of an operator that would be as clear as the &&, || and ! operators. As for the PEP, I have no problem writing one if this is accepted as a useful addition. Also any suggestions and critiques are very welcome of course. On Monday, 23 November 2015 21:24:23 UTC+1, Emanuel Barry wrote: > > Could you provide some links to where this could be useful, and how code > could be rewritten? I can see the desire for such a feature, I myself would > have liked such an operator or keyword. If you get general approval on this > list, you can then move on to write a PEP (as that's what's needed if you > wish to add a new keyword and/or operator to the language). > > I'm +0 for now, and may change once you provide us with use cases in the > wild. > -Emanuel > > ------------------------------ > From: m... at jeltef.nl > Date: Mon, 23 Nov 2015 20:09:17 +0100 > To: python... at python.org > Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean > operators > > Hi, > > After reading PEP0465 about > the dedicated matrix multiplication I started wondering if the same > solution couldn't be applied to boolean operators as well. There currently > are a lot of high profile libraries that have their own functions for > boolean operators, like Numpy, Pandas or SQLAlchemy. They do this because > the current boolean operators can't be overloaded. PEP0335 > was created to solve this > problem (and makes the problem more clear), but was rejected because it > needed changes to the byte code for the boolean operators, which would make > them slower. Currently some of these libraries resort to the bitwise > operators (at least Pandas), but those don't bind as strong as comparison > operators, which means you have to do comparisons like this: (series1 == 2) > & (series2 == 3) > > That is why I propose to create new operators just like for matrix > multiplication which can be used in libraries that need one. I'm not sure > what the operators should look like, but my first guess would be &&, || and > ! for and, or and not respectively. > > Is this an idea that sounds reasonable? > > Jelte > > _______________________________________________ Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: > http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 23 19:08:25 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 Nov 2015 11:08:25 +1100 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: On Tue, Nov 24, 2015 at 10:49 AM, Jelte Fennema wrote: > As for the PEP, I have no problem writing one if this is accepted as a > useful addition. Also any suggestions and critiques are very welcome of > course. I think it's reasonable, except for the potential confusion of having *three* "and" operators. The one with the word is never going to change - its semantics demand that it not be overridable. When should you use & and when &&? Judging by how @ has gone, I think the answer will be simple: "Always use &, unless the docs for some third-party library say to use &&", in which case I think it should be okay. ChrisA From guido at python.org Mon Nov 23 19:13:42 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Nov 2015 16:13:42 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: I honestly think the added confusion makes it a non-starter. It's also confusing that in other languages that have && and ||, they are shortcut operators, but the proposed operators here won't be. And the real question isn't "when to use & vs. &&", it's "when to use 'and' vs. &&". On Mon, Nov 23, 2015 at 4:08 PM, Chris Angelico wrote: > On Tue, Nov 24, 2015 at 10:49 AM, Jelte Fennema wrote: >> As for the PEP, I have no problem writing one if this is accepted as a >> useful addition. Also any suggestions and critiques are very welcome of >> course. > > I think it's reasonable, except for the potential confusion of having > *three* "and" operators. The one with the word is never going to > change - its semantics demand that it not be overridable. When should > you use & and when &&? Judging by how @ has gone, I think the answer > will be simple: "Always use &, unless the docs for some third-party > library say to use &&", in which case I think it should be okay. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From me at jeltef.nl Mon Nov 23 19:17:58 2015 From: me at jeltef.nl (Jelte Fennema) Date: Tue, 24 Nov 2015 01:17:58 +0100 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: This confusion could quite simply be solved by just not implementing the operations on the standard types, even though they would be trivial to implement. This just leaves the possibility for library developers to do something useful with the operators, like with the new @ operator. On 24 November 2015 at 01:13, Guido van Rossum wrote: > I honestly think the added confusion makes it a non-starter. It's also > confusing that in other languages that have && and ||, they are > shortcut operators, but the proposed operators here won't be. And the > real question isn't "when to use & vs. &&", it's "when to use 'and' > vs. &&". > > On Mon, Nov 23, 2015 at 4:08 PM, Chris Angelico wrote: > > On Tue, Nov 24, 2015 at 10:49 AM, Jelte Fennema wrote: > >> As for the PEP, I have no problem writing one if this is accepted as a > >> useful addition. Also any suggestions and critiques are very welcome of > >> course. > > > > I think it's reasonable, except for the potential confusion of having > > *three* "and" operators. The one with the word is never going to > > change - its semantics demand that it not be overridable. When should > > you use & and when &&? Judging by how @ has gone, I think the answer > > will be simple: "Always use &, unless the docs for some third-party > > library say to use &&", in which case I think it should be okay. > > > > ChrisA > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruce at leban.us Mon Nov 23 19:26:18 2015 From: bruce at leban.us (Bruce Leban) Date: Mon, 23 Nov 2015 16:26:18 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: On Mon, Nov 23, 2015 at 4:08 PM, Chris Angelico wrote: > I think it's reasonable, except for the potential confusion of having > *three* "and" operators. > I think using && and || would be an attractive nuisance for people switching from another programming language. Right now if I accidentally right && in Python or "and" in another language, I get an immediate syntax error. With this proposal, I get unexpected results. If this idea were to fly, a better name would be something that doesn't have that problem, e.g., .and. .or. .not. I don't want to bikeshed the exact syntax**, but I think it should be clear that with something like this: (1) no one is going to accidentally type them and (2) they are pretty clearly some variation of the standard and/or/not. **Lots of other possibilities that are syntax errors right now: @and, (and), etc. I like .and. because it's less visual clutter and it's easy to type. --- Bruce Check out my puzzle book and get it free here: http://J.mp/ingToConclusionsFree (available on iOS) -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at jeltef.nl Mon Nov 23 20:11:47 2015 From: me at jeltef.nl (Jelte Fennema) Date: Tue, 24 Nov 2015 02:11:47 +0100 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: On 24 November 2015 at 01:26, Bruce Leban wrote: > If this idea were to fly, a better name would be something that doesn't > have that problem, e.g., .and. .or. .not. I don't want to bikeshed the > exact syntax**, but I think it should be clear that with something like > this: (1) no one is going to accidentally type them and (2) they are pretty > clearly some variation of the standard and/or/not. I think a naming scheme like that is indeed a good way to solve the confusion issues, since it is also immediately clear that these would be a special version of the normal operators. Another advantage is that it could also be extended to the in operator, if that one is to be included. I'm not sure I like the dots version very much though, but like you said there are lots of syntax error options to choose from. On 24 November 2015 at 01:26, Bruce Leban wrote: > > On Mon, Nov 23, 2015 at 4:08 PM, Chris Angelico wrote: > >> I think it's reasonable, except for the potential confusion of having >> *three* "and" operators. >> > > I think using && and || would be an attractive nuisance for people > switching from another programming language. Right now if I accidentally > right && in Python or "and" in another language, I get an immediate syntax > error. With this proposal, I get unexpected results. > > If this idea were to fly, a better name would be something that doesn't > have that problem, e.g., .and. .or. .not. I don't want to bikeshed the > exact syntax**, but I think it should be clear that with something like > this: (1) no one is going to accidentally type them and (2) they are pretty > clearly some variation of the standard and/or/not. > > **Lots of other possibilities that are syntax errors right now: @and, (and), > etc. I like .and. because it's less visual clutter and it's easy to type. > > --- Bruce > Check out my puzzle book and get it free here: > http://J.mp/ingToConclusionsFree (available on iOS) > > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Nov 23 20:38:38 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 23 Nov 2015 17:38:38 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: <-334649383431326154@unknownmsgid> > I honestly think the added confusion makes it a non-starter. Coming from my experience from the Numpy world, the fact that you get "rich comparisons" for most what seem like Boolean operators, but not for and and or is very confusing to newbies. Much of the time, you can use the bitwise operators, as you often have done a comparison first : (A < x) & (A > y) But it's kind of a coincidence that it works, so it only makes thing more confusing for newbies. Bitwise operators really are kind of obscure these days. Explaining that you use ".and." Instead of " and" would be a much lighter lift than getting into the whole explanation for why we can't overload "and". -CHB I think first we decide it is or isn't a good idea, and then decide how to spell it, but .and. and .or. kind of appeal to me. > It's also > confusing that in other languages that have && and ||, they are > shortcut operators, but the proposed operators here won't be. And the > real question isn't "when to use & vs. &&", it's "when to use 'and' > vs. &&". > >> On Mon, Nov 23, 2015 at 4:08 PM, Chris Angelico wrote: >>> On Tue, Nov 24, 2015 at 10:49 AM, Jelte Fennema wrote: >>> As for the PEP, I have no problem writing one if this is accepted as a >>> useful addition. Also any suggestions and critiques are very welcome of >>> course. >> >> I think it's reasonable, except for the potential confusion of having >> *three* "and" operators. The one with the word is never going to >> change - its semantics demand that it not be overridable. When should >> you use & and when &&? Judging by how @ has gone, I think the answer >> will be simple: "Always use &, unless the docs for some third-party >> library say to use &&", in which case I think it should be okay. >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From mike at selik.org Mon Nov 23 20:44:03 2015 From: mike at selik.org (Michael Selik) Date: Tue, 24 Nov 2015 01:44:03 +0000 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: Why hasn't SQLAlchemy gone the route of NumPy with overloaded operators? Perhaps whatever reason it is would prevent using any new operators as well. With NumPy I make that mistake constantly: A == a & B == b rather than (A == a) & (B == b) I'd put that in the category of parentheses tax along with the print function and old style % string interpolation. Annoying, but it's inappropriate to use a gun to swat a fly. (In case my metaphor is unclear, creating a new operator is the gun -- risking collateral damage and all that) As Guido said, the real usability problem is that the ``and`` operator is a new Python programmer's first instinct. Adding yet another operator would make Python harder to learn and read. Even if you advertise a new operator, many libraries will be slow to change and we'll have 3 different techniques to teach. Let's weigh the benefits against the negative consequences. On Mon, Nov 23, 2015 at 8:12 PM Jelte Fennema wrote: > On 24 November 2015 at 01:26, Bruce Leban wrote: > >> If this idea were to fly, a better name would be something that doesn't >> have that problem, e.g., .and. .or. .not. I don't want to bikeshed the >> exact syntax**, but I think it should be clear that with something like >> this: (1) no one is going to accidentally type them and (2) they are pretty >> clearly some variation of the standard and/or/not. > > > I think a naming scheme like that is indeed a good way to solve the > confusion issues, since it is also immediately clear that these would be a > special version of the normal operators. Another advantage is that it could > also be extended to the in operator, if that one is to be included. > > I'm not sure I like the dots version very much though, but like you said > there are lots of syntax error options to choose from. > > On 24 November 2015 at 01:26, Bruce Leban wrote: > >> >> On Mon, Nov 23, 2015 at 4:08 PM, Chris Angelico wrote: >> >>> I think it's reasonable, except for the potential confusion of having >>> *three* "and" operators. >>> >> >> I think using && and || would be an attractive nuisance for people >> switching from another programming language. Right now if I accidentally >> right && in Python or "and" in another language, I get an immediate syntax >> error. With this proposal, I get unexpected results. >> >> If this idea were to fly, a better name would be something that doesn't >> have that problem, e.g., .and. .or. .not. I don't want to bikeshed the >> exact syntax**, but I think it should be clear that with something like >> this: (1) no one is going to accidentally type them and (2) they are pretty >> clearly some variation of the standard and/or/not. >> >> **Lots of other possibilities that are syntax errors right now: @and, (and), >> etc. I like .and. because it's less visual clutter and it's easy to type. >> >> --- Bruce >> Check out my puzzle book and get it free here: >> http://J.mp/ingToConclusionsFree (available on iOS) >> >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> >> _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at jeltef.nl Mon Nov 23 21:09:27 2015 From: me at jeltef.nl (Jelte Fennema) Date: Tue, 24 Nov 2015 03:09:27 +0100 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: On 24 November 2015 at 02:44, Michael Selik wrote: > Why hasn't SQLAlchemy gone the route of NumPy with overloaded operators? > It seems I was wrong about that, they apparently do: http://stackoverflow.com/a/14185275/2570866 I'd put that in the category of parentheses tax along with the print > function and old style % string interpolation. This seems like a bit of a weird argument since the parentheses for the print function are put there for a reason (see PEP3105) and the old style % string interpolation will be replaced by the new format string literal. Adding yet another operator would make Python harder to learn and read. > Even if you advertise a new operator, many libraries will be slow to change > and we'll have 3 different techniques to teach. I don't think that much confusion will arise, since the normal way is to to use is the short `and` version. Only in libraries where it would be explicitly told the new operator would be used. It would also do away with the confusion about why and cannot be overridden and why the precedence of the & operator is "wrong". On 24 November 2015 at 02:44, Michael Selik wrote: > Why hasn't SQLAlchemy gone the route of NumPy with overloaded operators? > Perhaps whatever reason it is would prevent using any new operators as > well. > > With NumPy I make that mistake constantly: > A == a & B == b > rather than > (A == a) & (B == b) > > I'd put that in the category of parentheses tax along with the print > function and old style % string interpolation. Annoying, but it's > inappropriate to use a gun to swat a fly. (In case my metaphor is unclear, > creating a new operator is the gun -- risking collateral damage and all > that) > > As Guido said, the real usability problem is that the ``and`` operator is > a new Python programmer's first instinct. Adding yet another operator would > make Python harder to learn and read. Even if you advertise a new operator, > many libraries will be slow to change and we'll have 3 different techniques > to teach. Let's weigh the benefits against the negative consequences. > > On Mon, Nov 23, 2015 at 8:12 PM Jelte Fennema wrote: > >> On 24 November 2015 at 01:26, Bruce Leban wrote: >> >>> If this idea were to fly, a better name would be something that doesn't >>> have that problem, e.g., .and. .or. .not. I don't want to bikeshed the >>> exact syntax**, but I think it should be clear that with something like >>> this: (1) no one is going to accidentally type them and (2) they are pretty >>> clearly some variation of the standard and/or/not. >> >> >> I think a naming scheme like that is indeed a good way to solve the >> confusion issues, since it is also immediately clear that these would be a >> special version of the normal operators. Another advantage is that it could >> also be extended to the in operator, if that one is to be included. >> >> I'm not sure I like the dots version very much though, but like you said >> there are lots of syntax error options to choose from. >> >> On 24 November 2015 at 01:26, Bruce Leban wrote: >> >>> >>> On Mon, Nov 23, 2015 at 4:08 PM, Chris Angelico >>> wrote: >>> >>>> I think it's reasonable, except for the potential confusion of having >>>> *three* "and" operators. >>>> >>> >>> I think using && and || would be an attractive nuisance for people >>> switching from another programming language. Right now if I accidentally >>> right && in Python or "and" in another language, I get an immediate syntax >>> error. With this proposal, I get unexpected results. >>> >>> If this idea were to fly, a better name would be something that doesn't >>> have that problem, e.g., .and. .or. .not. I don't want to bikeshed the >>> exact syntax**, but I think it should be clear that with something like >>> this: (1) no one is going to accidentally type them and (2) they are pretty >>> clearly some variation of the standard and/or/not. >>> >>> **Lots of other possibilities that are syntax errors right now: @and, (and), >>> etc. I like .and. because it's less visual clutter and it's easy to >>> type. >>> >>> --- Bruce >>> Check out my puzzle book and get it free here: >>> http://J.mp/ingToConclusionsFree (available on iOS) >>> >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "python-ideas" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> python-ideas+unsubscribe at googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "python-ideas" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> python-ideas+unsubscribe at googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Nov 23 21:38:28 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 23 Nov 2015 18:38:28 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: On Mon, Nov 23, 2015 at 5:44 PM, Michael Selik wrote: > With NumPy I make that mistake constantly: > A == a & B == b > rather than > (A == a) & (B == b) > > I'd put that in the category of parentheses tax > It's not just the operator precedence that's an issue here -- this also means something different than what people expect. As it happens a bitwise and works "just like" and when the values are boolean (like above), but that isn't always the case -- confusion waiting to happen. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Nov 23 23:55:02 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 24 Nov 2015 17:55:02 +1300 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: <5653EDA6.5020903@canterbury.ac.nz> Bruce Leban wrote: > If this idea were to fly, a better name would be something that doesn't > have that problem, e.g., .and. .or. .not. Dots are a bit on the ugly side. Some other possibilities: AND, OR, NOT And, Or, Not en, of, nlet -- Greg From abarnert at yahoo.com Tue Nov 24 00:35:52 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 23 Nov 2015 21:35:52 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <5653EDA6.5020903@canterbury.ac.nz> References: <5653EDA6.5020903@canterbury.ac.nz> Message-ID: <7CEF161C-557F-4AF1-ADF8-FD83C301ED2D@yahoo.com> (Top-posting because this is really a reply to a combination of three earlier posts, not to Greg Ewing's post, except at the very end.) -1 on "&&" and "||". To anyone familiar with C and friends, it seems like they ought to be like C (short-circuiting, and more generally the same thing we already spell "and" and "or"); to anyone else, it would make no sense for them to have different precedence than "&" and "|". -0.5 on ".and." and ".or." for multiple reasons, most of which apply just as well to anything similar: * That looks like a general custom-infix syntax. Anyone coming to Python (including Python 2 users) will expect that they can just as easily define ".spam." (and then be disappointed that they can't...), or will be worried that 3.7 will add a bunch of new dot operators they'll have to learn. Not a _huge_ negative, but already enough to turn me off the idea. * What would you call the dunder methods (and operator module functions), and how would you deal with the fact that any novice/transplant is going to assume "__and__" means ".and." rather than "&"? * Operators starting with "." are ambiguous with float literals and/or attribute access ("spam.and.eggs" looks like a member of a member of spam), at least to humans, if not to the parser. * Operators starting and/or ending with "." are hard to talk about because of natural-language punctuation (and even harder on a mobile keyboard, or an overly clever text editor or word processor). -0 on actually adding a general custom-infix syntax, come to think of it. Any solution to the problems above should work just as well here. And it means that in the future, libraries don't have to cram things into inappropriate symbols or be stuck with prefix or dot-method notation. And I don't think it would be any harder to learn than a few special cases. And I think ".between." would be just as useful in an ORM as ".in.". More generally, the whole point of "@" was that everyone agreed that it was the only new operator anyone would need (except maybe "@@") for a decade or two; if that's not true, infinity is a better number than 4. +1 on the idea if someone can come up with a good spelling that avoids all the above problems and reads as naturally as "@". > On Nov 23, 2015, at 20:55, Greg Ewing wrote: > > Bruce Leban wrote: >> If this idea were to fly, a better name would be something that doesn't have that problem, e.g., .and. .or. .not. > > Dots are a bit on the ugly side. > > Some other possibilities: > > AND, OR, NOT > > And, Or, Not > > en, of, nlet Is that last one supposed to be "niet"? When I try "nlet", Google assumes it's a typo for "net", which I guess could be a unary boolean identity operator, but I don't think we need that. :) Anyway, that pattern is a bit hard to extend to an overloadable "in" operator, because Dutch for "in" is "in". From ncoghlan at gmail.com Tue Nov 24 00:40:53 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Nov 2015 15:40:53 +1000 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <-334649383431326154@unknownmsgid> References: <-334649383431326154@unknownmsgid> Message-ID: On 24 November 2015 at 11:38, Chris Barker - NOAA Federal wrote: > I think first we decide it is or isn't a good idea, and then decide > how to spell it, but .and. and .or. kind of appeal to me. I think it's reasonably clear that rich logical operators with appropriate precedence and non-shortcircuiting behaviour would be a nice feature to have, the question is whether or not they can be introduced without making things even more confusing than they already are. Using placeholder syntax, let's consider the three operations: A rich_and B A rich_or B rich_not A To explain this fully will require explaining how they differ from: A and B -> A if not bool(A) else B A or B -> A if bool(A) else B not A -> not bool(A) and: A & B -> operator.and_(A, B) A | B -> operator.or_(A, B) ~A -> operator.not_(A) Depending on a user's background, it will also potentially require explaining how they differ from these operations in C and other languages: A && B A || B !A Casting any new forms specifically as "matrix" operators (like the new matmul operator Jelte referenced in the opening message of the thread) also runs into problems, since "@" is a genuinely distinct operation only applicable to matrices, while the goal here is instead to broadcast an existing operation over the array elements, which matrix objects are already able to do implicitly for most binary operators. Something that *could* potentially be comprehensible is the idea of allowing "elementwise" logical operators, with a suitable syntactic spelling. There'd still be a slight niggle to explain why and/or/not have explicitly elementwise variants when other binary operations don't, but that would likely still be less confusing than explaining the use of the bitwise operators. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From random832 at fastmail.com Tue Nov 24 01:34:20 2015 From: random832 at fastmail.com (Random832) Date: Tue, 24 Nov 2015 06:34:20 +0000 (UTC) Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators References: Message-ID: On 2015-11-24, Chris Angelico wrote: > I think it's reasonable, except for the potential confusion of having > *three* "and" operators. The one with the word is never going to > change - its semantics demand that it not be overridable. How? Are you referring to the short-circuit? C# allows overloading the short-circuit operators by doing it in two parts - in pseudocode: a and b: a if a.__false__() else a & b a or b: a if a.__true__() else a | b There's no fundamental reason short-circuiting "demands" that it not be overridable, just because C++ can't do it. From rosuav at gmail.com Tue Nov 24 01:52:13 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 Nov 2015 17:52:13 +1100 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: On Tue, Nov 24, 2015 at 5:34 PM, Random832 wrote: > On 2015-11-24, Chris Angelico wrote: >> I think it's reasonable, except for the potential confusion of having >> *three* "and" operators. The one with the word is never going to >> change - its semantics demand that it not be overridable. > > How? Are you referring to the short-circuit? C# allows > overloading the short-circuit operators by doing it in two > parts - in pseudocode: > > a and b: a if a.__false__() else a & b > a or b: a if a.__true__() else a | b > > There's no fundamental reason short-circuiting "demands" that it > not be overridable, just because C++ can't do it. The Python semantics are defined more tightly than that, though - the "else" clause in each case would simply be "b". Changing that is not something you can do with operator overloading, so it would mean a fundamental change in the operator's semantics. And once you do that, you end up with a completely different operator. So, yes, the semantics of Python's short-circuiting 'and' and 'or' operators precludes any form of overriding. Yes, it's possible to have overridable short-circuiting operators, but Python's ones are not those. (Also, I think I prefer the simpler semantics. But that's a matter of personal choice.) ChrisA From abarnert at yahoo.com Tue Nov 24 02:15:16 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 23 Nov 2015 23:15:16 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: <8FAFB343-875C-4365-A918-D7146D338FF2@yahoo.com> On Nov 23, 2015, at 22:34, Random832 wrote: > >> On 2015-11-24, Chris Angelico wrote: >> I think it's reasonable, except for the potential confusion of having >> *three* "and" operators. The one with the word is never going to >> change - its semantics demand that it not be overridable. > > How? Are you referring to the short-circuit? C# allows > overloading the short-circuit operators by doing it in two > parts - in pseudocode: > > a and b: a if a.__false__() else a & b > a or b: a if a.__true__() else a | b > > There's no fundamental reason short-circuiting "demands" that it > not be overridable, just because C++ can't do it. Actually, it _is_ possible in C++. You can pretty much do anything in C++, as long as you're an expert, you don't care about anyone reading your code, and you don't mind 1970s-style build times, and this is no exception. You just write an expression template library that uses non-short-circuiting-at-compile-time operators to generate functions that are short-circuiting at runtime, and then add the boilerplate to wrap any constants or normal variables up into your expression template types, and you're set. A simpler solution is to use lambda lifting, like Swift: "a and b" just means "a.__and__(lambda: b)". You could also split the operator into two method calls, without combining with bool conversion a la C#. So "a and b" becomes "tmp = a.__and1__()", then "tmp.__and2__(b) if tmp2 else tmp2". That way, "and" can return a non-boolean value, just as it does when not overloaded. I can't think of any language that does this off the top of my head, but I'm sure they exist. Or, in languages with macros or equivalent, you just use a macro instead of a function. Or, in languages with lazy evaluation by default, it's even simpler--if you don't use the result of "b" anywhere, it never gets evaluated. From greg.ewing at canterbury.ac.nz Tue Nov 24 01:05:09 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 24 Nov 2015 19:05:09 +1300 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: <-334649383431326154@unknownmsgid> Message-ID: <5653FE15.8080402@canterbury.ac.nz> Nick Coghlan wrote: > Something that *could* potentially be comprehensible is the idea of > allowing "elementwise" logical operators, with a suitable syntactic > spelling. -1 on tying any of this explicity to a notion of elementwise operations. The range of potential uses is wider than that. -- Greg From sjoerdjob at sjec.nl Tue Nov 24 03:03:50 2015 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Tue, 24 Nov 2015 09:03:50 +0100 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: <20151124080350.GA30034@sjoerdjob.com> Regarding the short-circuiting `and` and `or`, I think there's a way we can have our overloading-cake and only eat the pieces we need too. Even though it might not be totally intuitive, it should be possible to make the overloaded method get a 0-argument function instead of a value. In that case, `a and b` would become `a.__land__(lambda: b)`. As far as performance goes, there probably would have to be some special casing that first checks if `a` has an overloaded `and`, and if not use the default behaviour. The default implementation: class object: def __land__(self, other_f): if not self: return self else: return other_f() def __lor__(self, other_f): if self: return self else: return other_f() And for some python-expressions-to-AST nodeclass: class BaseExpression: def __land__(self, other_f): return AndExpression(self, other_f()) def __lor__(self, other_f): return OrExpression(self, other_f()) Again, the fact that the second argument to `__land__`/`__or__` is a function might be a bit confusing, but that's probably going to be the only way to make short-circuiting work for an overloaded and/or without going the route of lazy evaluation. On Mon, Nov 23, 2015 at 04:13:42PM -0800, Guido van Rossum wrote: > I honestly think the added confusion makes it a non-starter. It's also > confusing that in other languages that have && and ||, they are > shortcut operators, but the proposed operators here won't be. And the > real question isn't "when to use & vs. &&", it's "when to use 'and' > vs. &&". > > On Mon, Nov 23, 2015 at 4:08 PM, Chris Angelico wrote: > > On Tue, Nov 24, 2015 at 10:49 AM, Jelte Fennema wrote: > >> As for the PEP, I have no problem writing one if this is accepted as a > >> useful addition. Also any suggestions and critiques are very welcome of > >> course. > > > > I think it's reasonable, except for the potential confusion of having > > *three* "and" operators. The one with the word is never going to > > change - its semantics demand that it not be overridable. When should > > you use & and when &&? Judging by how @ has gone, I think the answer > > will be simple: "Always use &, unless the docs for some third-party > > library say to use &&", in which case I think it should be okay. > > > > ChrisA > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From njs at pobox.com Tue Nov 24 03:30:57 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 24 Nov 2015 00:30:57 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: On Nov 23, 2015 22:34, "Random832" wrote: > > On 2015-11-24, Chris Angelico wrote: > > I think it's reasonable, except for the potential confusion of having > > *three* "and" operators. The one with the word is never going to > > change - its semantics demand that it not be overridable. > > How? Are you referring to the short-circuit? C# allows > overloading the short-circuit operators by doing it in two > parts - in pseudocode: > > a and b: a if a.__false__() else a & b > a or b: a if a.__true__() else a | b > > There's no fundamental reason short-circuiting "demands" that it > not be overridable, just because C++ can't do it. No, I disagree -- for us short circuiting makes overriding extremely difficult, probably impossible, because in python in general and in the use cases being discussed here in particular, the right-hand side argument should have a chance to overload. It's going to be brutal on newbies and general code comprehensibility if True and array_of_bools or True and sqlalchemy_expression act totally differently than the mirrored versions, but this is an unavoidable if you have both short circuiting and overloading. (Example of where you'd see code like this: if restrict_user_id is not None: user_id_constraint = (MyClass.user_id == restrict_user_id) else: user_id_constraint = True if restrict_month is not None: month_constraint = (MyClass.month == restrict_month) else: month_constraint = True ... return query(MyClass).filter(user_id_constraint and month_constraint and ...) Yes, this could be written differently, but this pattern comes up fairly often IME. And it's otherwise very reliable; in numpy in particular 'True op array' and 'array(True) op array' act identically for every overloaded binary operator.) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Tue Nov 24 04:14:58 2015 From: mike at selik.org (Michael Selik) Date: Tue, 24 Nov 2015 09:14:58 +0000 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: As I think of people's reactions to seeing & and | for the first time, the typical response is, "What do you mean by bitwise?" Not, "Why are you using bitwise operators for non-bitwise operations?" Interestingly, no one in this thread seems to have a problem with ``&`` and ``|`` for set intersection and union. The primary complaint is that NumPy users instinctively reach for ``and``/``or`` and then forget the operator precedence of ``&``/``|``. On Mon, Nov 23, 2015 at 9:09 PM Jelte Fennema wrote: > On 24 November 2015 at 02:44, Michael Selik wrote: > >> Why hasn't SQLAlchemy gone the route of NumPy with overloaded operators? >> > It seems I was wrong about that, they apparently do: > http://stackoverflow.com/a/14185275/2570866 > Using pipe and ampersand looks readable. The use of bitwise operators for overloading and/or/not seems standard for many objects, in standard library and major projects. In fact, I use those operators as logical far more often than as bitwise. I'd bet a great number of NumPy users in the science community are completely unaware of their bitwise effects. I'd put that in the category of parentheses tax along with the print >> function and old style % string interpolation. > > This seems like a bit of a weird argument since the parentheses for the > print function are put there for a reason (see PEP3105) and the old style % > string interpolation will be replaced by the new format string literal. > I picked ``print`` as an example to show that requiring parens is fine. I picked string interpolation as an example because the ``.format`` solution also accepted parens as necessary. The rationale for f-string syntax (as written in the PEP) does not complain about parens but instead mentions that they will be necessary inside the string in several circumstances, like lambdas. Adding yet another operator would make Python harder to learn and read. >> Even if you advertise a new operator, many libraries will be slow to change >> and we'll have 3 different techniques to teach. > > I don't think that much confusion will arise, since the normal way is to > to use is the short `and` version. Only in libraries where it would be > explicitly told the new operator would be used. It would also do away with > the confusion about why and cannot be overridden and why the precedence of > the & operator is "wrong". > That precedence issue doesn't cause problems when teaching, in my experience. People accept readily that there's issues with operator precedence. On the other hand, they have trouble reading code that uses an older style (perhaps using ``&``) instead of the newer style that they learned (the hypothetical ``&&``). Even the switch between ``%`` interpolation and ``.format`` is still causing problems in large organizations. -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at jeltef.nl Tue Nov 24 04:59:44 2015 From: me at jeltef.nl (Jelte Fennema) Date: Tue, 24 Nov 2015 10:59:44 +0100 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: On 24 November 2015 at 10:14, Michael Selik wrote: > As I think of people's reactions to seeing & and | for the first time, the > typical response is, "What do you mean by bitwise?" Not, "Why are you using > bitwise operators for non-bitwise operations?" Interestingly, no one in > this thread seems to have a problem with ``&`` and ``|`` for set > intersection and union. The primary complaint is that NumPy users > instinctively reach for ``and``/``or`` and then forget the operator > precedence of ``&``/``|``. > Using pipe and ampersand looks readable. The use of bitwise operators for > overloading and/or/not seems standard for many objects, in standard library > and major projects. In fact, I use those operators as logical far more > often than as bitwise. I'd bet a great number of NumPy users in the science > community are completely unaware of their bitwise effects. > > I picked ``print`` as an example to show that requiring parens is fine. I > picked string interpolation as an example because the ``.format`` solution > also accepted parens as necessary. The rationale for f-string syntax (as > written in the PEP) does not complain about parens but instead mentions > that they will be necessary inside the string in several circumstances, > like lambdas. > > That precedence issue doesn't cause problems when teaching, in my > experience. People accept readily that there's issues with operator > precedence. On the other hand, they have trouble reading code that uses an > older style (perhaps using ``&``) instead of the newer style that they > learned (the hypothetical ``&&``). Even the switch between ``%`` > interpolation and ``.format`` is still causing problems in large > organizations. > > I get your points that it is bad to confuse people with even more and/or operators they have to learn. But I think it's weird that you are saying that people accept the wrong operator precedence readily, even though before you said you do it wrong yourself constantly. This is also why I think the set intersection and union are not a problem is because the operator precedence is correct there. Your point that the problem is mostly that the old code will still be usin the ``&`` operator, which could confuse people is true. But I also think that would eventually disappear, which would make Python better in the future. Another thing is that it seems some people are worried about the form of the new operators. Some options (which don't use the C style &&, because that would indeed cause confusion): .and. *and* +and+ %and% @and@ ?and? ^and^ {and} (and) [and] |and| :and: _and_ (could currently be a variablename) en (the Dutch version) Some of these could also be used with just one special character, like @and, but I think the surrounded ones look more visually pleasing. These are just a couple of examples and some of them seem fine to me. I do think that it is important though to not focus on the form already. It seems better to first figure out if the new operators would not confuse the newcomers to much in whatever form they come. On 24 November 2015 at 10:14, Michael Selik wrote: > As I think of people's reactions to seeing & and | for the first time, the > typical response is, "What do you mean by bitwise?" Not, "Why are you using > bitwise operators for non-bitwise operations?" Interestingly, no one in > this thread seems to have a problem with ``&`` and ``|`` for set > intersection and union. The primary complaint is that NumPy users > instinctively reach for ``and``/``or`` and then forget the operator > precedence of ``&``/``|``. > > > On Mon, Nov 23, 2015 at 9:09 PM Jelte Fennema wrote: > >> On 24 November 2015 at 02:44, Michael Selik wrote: >> >>> Why hasn't SQLAlchemy gone the route of NumPy with overloaded operators? >>> >> It seems I was wrong about that, they apparently do: >> http://stackoverflow.com/a/14185275/2570866 >> > > Using pipe and ampersand looks readable. The use of bitwise operators for > overloading and/or/not seems standard for many objects, in standard library > and major projects. In fact, I use those operators as logical far more > often than as bitwise. I'd bet a great number of NumPy users in the science > community are completely unaware of their bitwise effects. > > > I'd put that in the category of parentheses tax along with the print >>> function and old style % string interpolation. >> >> This seems like a bit of a weird argument since the parentheses for the >> print function are put there for a reason (see PEP3105) and the old style % >> string interpolation will be replaced by the new format string literal. >> > > I picked ``print`` as an example to show that requiring parens is fine. I > picked string interpolation as an example because the ``.format`` solution > also accepted parens as necessary. The rationale for f-string syntax (as > written in the PEP) does not complain about parens but instead mentions > that they will be necessary inside the string in several circumstances, > like lambdas. > > Adding yet another operator would make Python harder to learn and read. >>> Even if you advertise a new operator, many libraries will be slow to change >>> and we'll have 3 different techniques to teach. >> >> I don't think that much confusion will arise, since the normal way is to >> to use is the short `and` version. Only in libraries where it would be >> explicitly told the new operator would be used. It would also do away with >> the confusion about why and cannot be overridden and why the precedence of >> the & operator is "wrong". >> > > That precedence issue doesn't cause problems when teaching, in my > experience. People accept readily that there's issues with operator > precedence. On the other hand, they have trouble reading code that uses an > older style (perhaps using ``&``) instead of the newer style that they > learned (the hypothetical ``&&``). Even the switch between ``%`` > interpolation and ``.format`` is still causing problems in large > organizations. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at jeltef.nl Tue Nov 24 05:10:54 2015 From: me at jeltef.nl (Jelte Fennema) Date: Tue, 24 Nov 2015 11:10:54 +0100 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <7CEF161C-557F-4AF1-ADF8-FD83C301ED2D@yahoo.com> References: <5653EDA6.5020903@canterbury.ac.nz> <7CEF161C-557F-4AF1-ADF8-FD83C301ED2D@yahoo.com> Message-ID: On 24 November 2015 at 06:35, Andrew Barnert via Python-ideas < python-ideas at python.org> wrote: > -0 on actually adding a general custom-infix syntax, come to think of it. > Any solution to the problems above should work just as well here. And it > means that in the future, libraries don't have to cram things into > inappropriate symbols or be stuck with prefix or dot-method notation. And I > don't think it would be any harder to learn than a few special cases. And I > think ".between." would be just as useful in an ORM as ".in.". More > generally, the whole point of "@" was that everyone agreed that it was the > only new operator anyone would need (except maybe "@@") for a decade or > two; if that's not true, infinity is a better number than 4. This seems like an interesting idea, but I think it would be hard finding a notation that does not conflict with current operations. For the and/or almost any symbol would work, since they are keywords and the operators have no meaning on them, but if any name can be used operations on variables will become unclear. So another symbol would probably have to be used that isn't used yet, like the $ symbol. Also, I think there are probably some other issues that I haven't thought of, since this would add a pretty big language feature. PS. I didn't include the dollar convention in my last options list for the new and operator, but it is ofcourse also a possibility: $and$, or maybe $and. On 24 November 2015 at 06:35, Andrew Barnert via Python-ideas < python-ideas at python.org> wrote: > (Top-posting because this is really a reply to a combination of three > earlier posts, not to Greg Ewing's post, except at the very end.) > > -1 on "&&" and "||". To anyone familiar with C and friends, it seems like > they ought to be like C (short-circuiting, and more generally the same > thing we already spell "and" and "or"); to anyone else, it would make no > sense for them to have different precedence than "&" and "|". > > -0.5 on ".and." and ".or." for multiple reasons, most of which apply just > as well to anything similar: > > * That looks like a general custom-infix syntax. Anyone coming to Python > (including Python 2 users) will expect that they can just as easily define > ".spam." (and then be disappointed that they can't...), or will be worried > that 3.7 will add a bunch of new dot operators they'll have to learn. Not a > _huge_ negative, but already enough to turn me off the idea. > > * What would you call the dunder methods (and operator module functions), > and how would you deal with the fact that any novice/transplant is going to > assume "__and__" means ".and." rather than "&"? > > * Operators starting with "." are ambiguous with float literals and/or > attribute access ("spam.and.eggs" looks like a member of a member of spam), > at least to humans, if not to the parser. > > * Operators starting and/or ending with "." are hard to talk about because > of natural-language punctuation (and even harder on a mobile keyboard, or > an overly clever text editor or word processor). > > -0 on actually adding a general custom-infix syntax, come to think of it. > Any solution to the problems above should work just as well here. And it > means that in the future, libraries don't have to cram things into > inappropriate symbols or be stuck with prefix or dot-method notation. And I > don't think it would be any harder to learn than a few special cases. And I > think ".between." would be just as useful in an ORM as ".in.". More > generally, the whole point of "@" was that everyone agreed that it was the > only new operator anyone would need (except maybe "@@") for a decade or > two; if that's not true, infinity is a better number than 4. > > +1 on the idea if someone can come up with a good spelling that avoids all > the above problems and reads as naturally as "@". > > > On Nov 23, 2015, at 20:55, Greg Ewing > wrote: > > > > Bruce Leban wrote: > >> If this idea were to fly, a better name would be something that doesn't > have that problem, e.g., .and. .or. .not. > > > > Dots are a bit on the ugly side. > > > > Some other possibilities: > > > > AND, OR, NOT > > > > And, Or, Not > > > > en, of, nlet > > Is that last one supposed to be "niet"? When I try "nlet", Google assumes > it's a typo for "net", which I guess could be a unary boolean identity > operator, but I don't think we need that. :) > > Anyway, that pattern is a bit hard to extend to an overloadable "in" > operator, because Dutch for "in" is "in". > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Tue Nov 24 05:36:42 2015 From: mike at selik.org (Michael Selik) Date: Tue, 24 Nov 2015 10:36:42 +0000 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: On Tue, Nov 24, 2015 at 5:00 AM Jelte Fennema wrote: > On 24 November 2015 at 10:14, Michael Selik wrote: > >> As I think of people's reactions to seeing & and | for the first time, >> the typical response is, "What do you mean by bitwise?" Not, "Why are you >> using bitwise operators for non-bitwise operations?" Interestingly, no one >> in this thread seems to have a problem with ``&`` and ``|`` for set >> intersection and union. The primary complaint is that NumPy users >> instinctively reach for ``and``/``or`` and then forget the operator >> precedence of ``&``/``|``. >> > > I get your points that it is bad to confuse people with even more and/or > operators they have to learn. But I think it's weird that you are saying > that people accept the wrong operator precedence readily, even though > before you said you do it wrong yourself constantly. > I make that mistake in an interactive environment and fix it moments later, so it's not a big thing for me. I also occasionally forget to put a colon at the end of my for-loops, etc. ;-) > This is also why I think the set intersection and union are not a problem > is because the operator precedence is correct there. > > Your point that the problem is mostly that the old code will still be usin > the ``&`` operator, which could confuse people is true. But I also think > that would eventually disappear, which would make Python better in the > future. > What's the half-life of deprecated code? That stuff is like nuclear waste. Or more like a bacteria you're spot-treating with antibiotic. It keeps replicating while you apply ointment and might evolve a resistance. Ok, ok, I'm getting a little too colorful there. -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Tue Nov 24 10:11:45 2015 From: random832 at fastmail.com (Random832) Date: Tue, 24 Nov 2015 15:11:45 +0000 (UTC) Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators References: Message-ID: On 2015-11-24, Chris Angelico wrote: > The Python semantics are defined more tightly than that, though - the > "else" clause in each case would simply be "b". Changing that is not > something you can do with operator overloading, so it would mean a > fundamental change in the operator's semantics. And once you do that, > you end up with a completely different operator. I don't see that as "fundamental". Certainly it can't _actually_ be a & b, but it could certainly be "a __foo__ b" where the default implementation of the __foo__ method (on object) simply returns b. If that's "fundamental" then adding overloading to _any_ operator that didn't support it in Python 1.0 violates those operators' "fundamental" behavior of raising a TypeError when applied to types they did not support in Python 1.0. This would also allow b.__rfoo__, to address the objection Nathaniel Smith raised, of the right-hand-side not being able to override. It can't override "does it short-circuit", since that'd defeat the point of short-circuiting, but it can override the value in the case where it doesn't, and a well-behaved 'elementwise-"and"able' type would work either way, only short-circuiting if every element of the left side is false (or true for "or"). From chris.barker at noaa.gov Tue Nov 24 12:31:30 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 24 Nov 2015 09:31:30 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: > > As I think of people's reactions to seeing & and | for the first time, the >>> typical response is, "What do you mean by bitwise?" Not, "Why are you using >>> bitwise operators for non-bitwise operations?" >>> >> agreed -- but unfortunately, numpy may be one place where people really do want bitwise operations sometimes -- and it's way too late now to re-define them anyway. > Interestingly, no one in this thread seems to have a problem with ``&`` >>> and ``|`` for set intersection and union. >>> >> I'd say that's because for sets, it would have been a very rare case to do bitwise operations -- I'm sure I lack imagination, but I can't think of even one case where it would make sense. So why not re-use them? The primary complaint is that NumPy users instinctively reach for >>> ``and``/``or`` and then forget the operator precedence of ``&``/``|``. >>> >> well, that is an issue, though I think the fact that you can only use & | when you want "and" "or" when it happens to make sense -- i.e. for boolean arrays, is the bigger problem. Also, newbies will either not think to use them, or will see them in code and think they ARE another way to spell and and or. It also doesn't allow element-wise short circuiting behavior. you get: In [*2*]: np.array([0,1,2,3]) & np.array([3,2,1,0]) Out[*2*]: array([0, 0, 0, 0]) when you want: In [*3*]: [ a and b for a,b in zip([0,1,2,3], [3,2,1,0])] Out[*3*]: [0, 2, 1, 0] oh, and this: In [*4*]: np.array([0,1,2,3]) | np.array([3,2,1,0]) Out[*4*]: array([3, 3, 3, 3]) would surely confuse people that don't "get" what bitwise means, even though it results in teh same thing in a boolean context: In [*5*]: (np.array([0,1,2,3]) | np.array([3,2,1,0])).astype(np.bool) Out[*5*]: array([ True, True, True, True], dtype=bool) Honestly, I'm not sure element-wise short circuiting is desired, but it would be nice to be consistent about that. As for precedence, it's a annoying, but at least in my experience, the very first test case fails, and then I remember to add the parens. As for the more and more operators issue -- for the most part, this would be used for special purposes -- most people wouldn't even notice there were there, and numpy users might well think it' s special numpy thing (same for SQLAlchemy folks, or...) Unless folks wanted to add support for element-wise and and or to the built-in sequence types. I don't think anyone has proposed that. Your point that the problem is mostly that the old code will still be using >> the ``&`` operator, which could confuse people is true. >> > well, yes - but code using & would continue to work the same way -- and anyone confused would, in fact, have been mis-interpreting what & meant already :-) As for element-wise everything -- THAT discussion has happened multiple times in the past, and been rejected. Time to go dig into the archives -- not sure if there is rejected PEP for it. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From brenbarn at brenbarn.net Tue Nov 24 13:12:43 2015 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Tue, 24 Nov 2015 10:12:43 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: <5654A89B.40607@brenbarn.net> On 2015-11-23 22:34, Random832 wrote: > On 2015-11-24, Chris Angelico wrote: >> >I think it's reasonable, except for the potential confusion of having >> >*three* "and" operators. The one with the word is never going to >> >change - its semantics demand that it not be overridable. > How? Are you referring to the short-circuit? C# allows > overloading the short-circuit operators by doing it in two > parts - in pseudocode: > > a and b: a if a.__false__() else a & b > a or b: a if a.__true__() else a | b > > There's no fundamental reason short-circuiting "demands" that it > not be overridable, just because C++ can't do it. The problem is that this kind of overriding doesn't handle the main use case, which is elementwise and-ing/or-ing. If a is some numpy-like array of [1, 1], then it may be boolean true, but you still want "a magic_or b" to do the elementwise operation, not just return "a" by itself. (I say "numpy-like" because a real numpy array will fail here precisely because its boolean truth value is undefined, so it raises an error.) The problem is that, with the existing and/or, the issue of the boolean truth values of the operands is entangled with the actual and/or operation. For an elementwise operation, you *always* want to do it elementwise, regardless of whether the operands "count" as boolean false in some other situation. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From guido at python.org Tue Nov 24 13:20:09 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 24 Nov 2015 10:20:09 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <5654A89B.40607@brenbarn.net> References: <5654A89B.40607@brenbarn.net> Message-ID: To everyone claiming that you can't overload and/or because they are shortcut operators, please re-read PEP 335. It provides a clean solution -- it was rejected because it adds an extra byte code to all code using those operators (the majority of which don't need it). -- --Guido van Rossum (python.org/~guido) From abarnert at yahoo.com Tue Nov 24 13:15:43 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 24 Nov 2015 10:15:43 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: <5653EDA6.5020903@canterbury.ac.nz> <7CEF161C-557F-4AF1-ADF8-FD83C301ED2D@yahoo.com> Message-ID: On Nov 24, 2015, at 02:10, Jelte Fennema wrote: > >> On 24 November 2015 at 06:35, Andrew Barnert via Python-ideas wrote: >> -0 on actually adding a general custom-infix syntax, come to think of it. Any solution to the problems above should work just as well here. And it means that in the future, libraries don't have to cram things into inappropriate symbols or be stuck with prefix or dot-method notation. And I don't think it would be any harder to learn than a few special cases. And I think ".between." would be just as useful in an ORM as ".in.". More generally, the whole point of "@" was that everyone agreed that it was the only new operator anyone would need (except maybe "@@") for a decade or two; if that's not true, infinity is a better number than 4. > > This seems like an interesting idea, but I think it would be hard finding a notation that does not conflict with current operations. The obvious notation is "a `spam` b", as used in Haskell and various other languages. It doesn't conflict with current operations. It doesn't look like a pair of operators, but like a matched bracketing or quoting, which is exactly what we'd want. It has just about the right screen density (. can be easy to miss, while $ is so heavy that it makes the "in" in $in$ harder to read). It doesn't conflict with a different meaning elsewhere, like $(in), which can look like part of string template to a human reader. Also, experience with those other languages shows that it works. In particular, in Haskell, I could always just define any new string of symbols to call spam on its operands, and with any precedence and associativity I want--but it's often much more readable to just use `spam` instead. Its biggest problem is that Guido hates backticks, and is on record as promising that they will never be reused in Python now that they no longer mean repr. Since I don't think he'd accept the idea anyway (IIRC, last time it came up, he said it wasn't even worth writing a PEP to categorically dismiss), that makes this the perfect syntax for it. :) More seriously, my point that infinity is a better number than 4 is more an argument against adding custom and, or, not , and in operators than an argument for adding custom arbitrary operators, which means the spelling isn't that important anyway. I don't think you're going to do much better than `and`, and I don't think that's good enough for Python. > For the and/or almost any symbol would work, since they are keywords and the operators have no meaning on them, No, because code has to be readable and parseable by humans, not just by compilers. Even if the compiler can tell that "spam.and.eggs" or "2.in.e" aren't using dots for member access or float purposes, that isn't clear to a human reader until you look at it carefully. (Of course you can always use `and` or some other syntax that wouldn't be confusing even if and weren't a keyword, but at that point you're back to "no better than custom infixes".) Also, bear in mind that what you're competing with is a.spam(b), so anything that looks too heavyweight or too weird, like a $(spam) b, isn't going to be much improvement except in very long expressions where closing the parens gets troublesome (which often aren't going to be readable anyway, and maybe it's not so terrible to encourage people to break them up and name subparts). > but if any name can be used operations on variables will become unclear. So another symbol would probably have to be used that isn't used yet, like the $ symbol. Also, I think there are probably some other issues that I haven't thought of, since this would add a pretty big language feature. Actually, last time I looked into it (around 3.4), it didn't have any other big issues or complex interactions with other features (and I don't think static typing, await, or any other changes will affect that, but I'd have to look more carefully). And it's pretty simple to implement, and pretty easy to describe. If you're interested, I may have written up a blog post, and if I didn't, I could write one now. (I think I also have a hack that implements it with a retokenizing import hook, if you want to play with it.) While we're at it, every time custom infix operators come up, someone points out that you can already fake it pretty well. For example, "a &And& b" just needs a global named "And" whose __rand__(self, a) returns an object whose __and__(self, b) does what you want. And this allows you to use whatever precedence you want by picking the surrounding operators (much like Swift's custom infix operators get their precedence from the first symbol in the name), and lets you implement operator sectioning (you can make __rand__(self, a).__call__ the same as its __and__, and then you can pass around "a &And" as a partial function), and so on. There are a few glitches, but it works well with the kind of code you'd write with an ORM. And again, this is just as true for a proposal to add four custom operators as for a proposal to add a general feature. So, you have to think about why sqlanywhere, numpy, etc. have decided not to use this trick so far, when it's been well known since around the days of Python 2.4, and why they'd be better off with custom operators that don't look that different from the fake ones. > PS. I didn't include the dollar convention in my last options list for the new and operator, but it is ofcourse also a possibility: $and$, or maybe $and. > >> On 24 November 2015 at 06:35, Andrew Barnert via Python-ideas wrote: >> (Top-posting because this is really a reply to a combination of three earlier posts, not to Greg Ewing's post, except at the very end.) >> >> -1 on "&&" and "||". To anyone familiar with C and friends, it seems like they ought to be like C (short-circuiting, and more generally the same thing we already spell "and" and "or"); to anyone else, it would make no sense for them to have different precedence than "&" and "|". >> >> -0.5 on ".and." and ".or." for multiple reasons, most of which apply just as well to anything similar: >> >> * That looks like a general custom-infix syntax. Anyone coming to Python (including Python 2 users) will expect that they can just as easily define ".spam." (and then be disappointed that they can't...), or will be worried that 3.7 will add a bunch of new dot operators they'll have to learn. Not a _huge_ negative, but already enough to turn me off the idea. >> >> * What would you call the dunder methods (and operator module functions), and how would you deal with the fact that any novice/transplant is going to assume "__and__" means ".and." rather than "&"? >> >> * Operators starting with "." are ambiguous with float literals and/or attribute access ("spam.and.eggs" looks like a member of a member of spam), at least to humans, if not to the parser. >> >> * Operators starting and/or ending with "." are hard to talk about because of natural-language punctuation (and even harder on a mobile keyboard, or an overly clever text editor or word processor). >> >> -0 on actually adding a general custom-infix syntax, come to think of it. Any solution to the problems above should work just as well here. And it means that in the future, libraries don't have to cram things into inappropriate symbols or be stuck with prefix or dot-method notation. And I don't think it would be any harder to learn than a few special cases. And I think ".between." would be just as useful in an ORM as ".in.". More generally, the whole point of "@" was that everyone agreed that it was the only new operator anyone would need (except maybe "@@") for a decade or two; if that's not true, infinity is a better number than 4. >> >> +1 on the idea if someone can come up with a good spelling that avoids all the above problems and reads as naturally as "@". >> >> > On Nov 23, 2015, at 20:55, Greg Ewing wrote: >> > >> > Bruce Leban wrote: >> >> If this idea were to fly, a better name would be something that doesn't have that problem, e.g., .and. .or. .not. >> > >> > Dots are a bit on the ugly side. >> > >> > Some other possibilities: >> > >> > AND, OR, NOT >> > >> > And, Or, Not >> > >> > en, of, nlet >> >> Is that last one supposed to be "niet"? When I try "nlet", Google assumes it's a typo for "net", which I guess could be a unary boolean identity operator, but I don't think we need that. :) >> >> Anyway, that pattern is a bit hard to extend to an overloadable "in" operator, because Dutch for "in" is "in". >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From brenbarn at brenbarn.net Tue Nov 24 01:41:45 2015 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Mon, 23 Nov 2015 22:41:45 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: Message-ID: <565406A9.90605@brenbarn.net> On 2015-11-23 22:34, Random832 wrote: > On 2015-11-24, Chris Angelico wrote: >> I think it's reasonable, except for the potential confusion of having >> *three* "and" operators. The one with the word is never going to >> change - its semantics demand that it not be overridable. > > How? Are you referring to the short-circuit? C# allows > overloading the short-circuit operators by doing it in two > parts - in pseudocode: > > a and b: a if a.__false__() else a & b > a or b: a if a.__true__() else a | b > > There's no fundamental reason short-circuiting "demands" that it > not be overridable, just because C++ can't do it. The problem is that this kind of overriding doesn't handle the main use case, which is elementwise and-ing/or-ing. If a is some numpy-like array of [1, 1], then it may be boolean true, but you still want "a magic_or b" to do the elementwise operation, not just return "a" by itself. (I say "numpy-like" because a real numpy array will fail here precisely because its boolean truth value is undefined, so it raises an error.) The problem is that, with the existing and/or, the issue of the boolean truth values of the operands is entangled with the actual and/or operation. For an elementwise operation, you *always* want to do it elementwise, regardless of whether the operands "count" as boolean false in some other situation. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From random832 at fastmail.com Tue Nov 24 14:07:35 2015 From: random832 at fastmail.com (Random832) Date: Tue, 24 Nov 2015 19:07:35 +0000 (UTC) Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators References: <5654A89B.40607@brenbarn.net> Message-ID: On 2015-11-24, Brendan Barnwell wrote: > The problem is that this kind of overriding doesn't handle the main use > case, which is elementwise and-ing/or-ing. If a is some numpy-like > array of [1, 1], then it may be boolean true, but you still want "a > magic_or b" to do the elementwise operation, not just return "a" by > itself. Why? I am asking specifically because *all* elements of a are true, so all elements of the result will also be true, and taken from a. Clearly [0, 1] should *not* return true for a.__true__(), which is why I didn't use __bool__. But the elementwise operation [1, 1] magic_or [0, 2] returns [1, 1]. From brenbarn at brenbarn.net Tue Nov 24 14:27:51 2015 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Tue, 24 Nov 2015 11:27:51 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: <5654A89B.40607@brenbarn.net> Message-ID: <5654BA37.8080208@brenbarn.net> On 2015-11-24 11:07, Random832 wrote: > On 2015-11-24, Brendan Barnwell wrote: >> The problem is that this kind of overriding doesn't handle the main use >> case, which is elementwise and-ing/or-ing. If a is some numpy-like >> array of [1, 1], then it may be boolean true, but you still want "a >> magic_or b" to do the elementwise operation, not just return "a" by >> itself. > > Why? I am asking specifically because *all* elements of a are true, so all > elements of the result will also be true, and taken from a. Clearly [0, 1] > should *not* return true for a.__true__(), which is why I didn't use __bool__. > But the elementwise operation [1, 1] magic_or [0, 2] returns [1, 1]. I guess I'm not understanding what you mean by __true__ then. What is this __true__ for which ([0, 1]).__true__() is not true? -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From njs at pobox.com Tue Nov 24 14:37:41 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 24 Nov 2015 11:37:41 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: <5654A89B.40607@brenbarn.net> Message-ID: On Nov 24, 2015 10:21 AM, "Guido van Rossum" wrote: > > To everyone claiming that you can't overload and/or because they are > shortcut operators, please re-read PEP 335. It provides a clean > solution -- it was rejected because it adds an extra byte code to all > code using those operators (the majority of which don't need it). The semantic objection that I raised -- short circuiting means that you can't correctly overload 'True and numpy_array', because unlike all other binops the overload must be defined on the left hand argument -- does apply to PEP 335 AFAICT. This problem is IMHO serious enough that even if PEP 335 were accepted today I'm not entirely sure that numpy would actually implement the overloads due to the headaches it would cause for teaching and code review -- we'd have to have some debate about it at least. (Possibly useful analogy: Having to always double check that the array argument appears on the left rather than the right is kinda like how the old 'a and b or c' idiom forces you to constantly keep an extra special rule in the back of your head and always double check that b cannot be falsey whenever you read or write it, which IIUC was one of the major reasons why it was considered an insufficient substitute for a real ternary operator.) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Nov 24 14:51:46 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 24 Nov 2015 11:51:46 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: <5654A89B.40607@brenbarn.net> Message-ID: On Tue, Nov 24, 2015 at 11:37 AM, Nathaniel Smith wrote: > On Nov 24, 2015 10:21 AM, "Guido van Rossum" wrote: >> >> To everyone claiming that you can't overload and/or because they are >> shortcut operators, please re-read PEP 335. It provides a clean >> solution -- it was rejected because it adds an extra byte code to all >> code using those operators (the majority of which don't need it). > > The semantic objection that I raised -- short circuiting means that you > can't correctly overload 'True and numpy_array', because unlike all other > binops the overload must be defined on the left hand argument -- does apply > to PEP 335 AFAICT. This problem is IMHO serious enough that even if PEP 335 > were accepted today I'm not entirely sure that numpy would actually > implement the overloads due to the headaches it would cause for teaching and > code review -- we'd have to have some debate about it at least. OK, that's useful feedback. Is NumPy interested in coming up with an alternative that works, or are you fine with the status quo? -- --Guido van Rossum (python.org/~guido) From greg.ewing at canterbury.ac.nz Tue Nov 24 16:24:07 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 25 Nov 2015 10:24:07 +1300 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <565406A9.90605@brenbarn.net> References: <565406A9.90605@brenbarn.net> Message-ID: <5654D577.6060601@canterbury.ac.nz> Brendan Barnwell wrote: > On 2015-11-23 22:34, Random832 wrote: > >> C# allows overloading the short-circuit operators >> >> a and b: a if a.__false__() else a & b >> a or b: a if a.__true__() else a | b >> > The problem is that this kind of overriding doesn't handle the main > use case, which is elementwise and-ing/or-ing. Maybe we can hook into __bool__ somehow, though? Suppose 'a and b' were treated as: try: result = bool(a) except IDoNotShortCircuit: result = a.__logical_and___(b) else: if not result: result = b Since a __bool__ call is required anyway, this shouldn't slow down the case where there is no overriding. -- Greg From mike at selik.org Tue Nov 24 16:42:24 2015 From: mike at selik.org (Michael Selik) Date: Tue, 24 Nov 2015 21:42:24 +0000 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <5654D577.6060601@canterbury.ac.nz> References: <565406A9.90605@brenbarn.net> <5654D577.6060601@canterbury.ac.nz> Message-ID: Interesting. To allow rich logical operation in this manner, we would need to relax the requirement that ``__bool__`` returns a ``bool``. What negative effects would that have? On Tue, Nov 24, 2015 at 4:24 PM Greg Ewing wrote: > Brendan Barnwell wrote: > > On 2015-11-23 22:34, Random832 wrote: > > > >> C# allows overloading the short-circuit operators > >> > >> a and b: a if a.__false__() else a & b > >> a or b: a if a.__true__() else a | b > >> > > The problem is that this kind of overriding doesn't handle the main > > use case, which is elementwise and-ing/or-ing. > > Maybe we can hook into __bool__ somehow, though? > > Suppose 'a and b' were treated as: > > try: > result = bool(a) > except IDoNotShortCircuit: > result = a.__logical_and___(b) > else: > if not result: > result = b > > Since a __bool__ call is required anyway, this > shouldn't slow down the case where there is no > overriding. > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Tue Nov 24 16:47:07 2015 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 24 Nov 2015 21:47:07 +0000 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <5654D577.6060601@canterbury.ac.nz> References: <565406A9.90605@brenbarn.net> <5654D577.6060601@canterbury.ac.nz> Message-ID: <5654DADB.7010405@mrabarnett.plus.com> On 2015-11-24 21:24, Greg Ewing wrote: > Brendan Barnwell wrote: >> On 2015-11-23 22:34, Random832 wrote: >> > >> C# allows overloading the short-circuit operators >>> >>> a and b: a if a.__false__() else a & b >>> a or b: a if a.__true__() else a | b >>> >> The problem is that this kind of overriding doesn't handle the main >> use case, which is elementwise and-ing/or-ing. > > Maybe we can hook into __bool__ somehow, though? > > Suppose 'a and b' were treated as: > > try: > result = bool(a) > except IDoNotShortCircuit: > result = a.__logical_and___(b) > else: > if not result: > result = b > > Since a __bool__ call is required anyway, this > shouldn't slow down the case where there is no > overriding. > That still doesn't deal with the issue of what should happen if the order is reversed, e.g. "numpy_array and simple_bool" vs "simple_bool and numpy_array", where "numpy_array" has the non-shortcircuiting behaviour but "simple_bool" hasn't. From greg.ewing at canterbury.ac.nz Tue Nov 24 17:29:57 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 25 Nov 2015 11:29:57 +1300 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: <5654A89B.40607@brenbarn.net> Message-ID: <5654E4E5.6070206@canterbury.ac.nz> Guido van Rossum wrote: > To everyone claiming that you can't overload and/or because they are > shortcut operators, please re-read PEP 335. It provides a clean > solution -- it was rejected because it adds an extra byte code to all > code using those operators (the majority of which don't need it). Was that the *only* reason for rejection? If that problem could be solved somehow, would you reconsider? -- Greg From greg.ewing at canterbury.ac.nz Tue Nov 24 17:45:45 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 25 Nov 2015 11:45:45 +1300 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <5654DADB.7010405@mrabarnett.plus.com> References: <565406A9.90605@brenbarn.net> <5654D577.6060601@canterbury.ac.nz> <5654DADB.7010405@mrabarnett.plus.com> Message-ID: <5654E899.9090502@canterbury.ac.nz> MRAB wrote: > That still doesn't deal with the issue of what should happen if the > order is reversed, e.g. "numpy_array and simple_bool" vs "simple_bool > and numpy_array", where "numpy_array" has the non-shortcircuiting > behaviour but "simple_bool" hasn't. That's true. I guess the only solution that really works properly is to have a second set of operators, or some way of flagging that you're using non-short-circuiting semantics. There's one language I've seen -- I think it was Eiffel, or maybe Ada -- that had two sets of boolean operators. But it was kind of the other way around: 'and' and 'or' were non-short-circuiting, and to get short-circuiting you had to say 'and then' or 'or else'. -- Greg From rymg19 at gmail.com Tue Nov 24 18:00:34 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 24 Nov 2015 17:00:34 -0600 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <5654E899.9090502@canterbury.ac.nz> References: <565406A9.90605@brenbarn.net> <5654D577.6060601@canterbury.ac.nz> <5654DADB.7010405@mrabarnett.plus.com> <5654E899.9090502@canterbury.ac.nz> Message-ID: On November 24, 2015 4:45:45 PM CST, Greg Ewing wrote: >MRAB wrote: > >> That still doesn't deal with the issue of what should happen if the >> order is reversed, e.g. "numpy_array and simple_bool" vs "simple_bool >> and numpy_array", where "numpy_array" has the non-shortcircuiting >> behaviour but "simple_bool" hasn't. > >That's true. > >I guess the only solution that really works properly is >to have a second set of operators, or some way of flagging >that you're using non-short-circuiting semantics. > >There's one language I've seen -- I think it was Eiffel, >or maybe Ada -- that had two sets of boolean operators. >But it was kind of the other way around: 'and' and 'or' >were non-short-circuiting, and to get short-circuiting >you had to say 'and then' or 'or else'. That was a lot of languages: - Ada: and then, or else - Algol: ANDTHEN, ORELSE - Erlang: andalso, orelse - Extended Pascal: and_then, or_else - GNU Pascal: and then, or else - Oz: andthen, orelse - Standard ML: andalso, orelse - Visual Basic: AndAlso, OrElse Interestingly, Visual Basic allows you to overload non-short-circuiting And. To overload AndAlso, you need to overload And and IsFalse. It seems that AndAlso was roughly equivalent to And + IsFalse: A AndAlso B = If Not (IsFalse A) Then A And B Else A Endif -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. From random832 at fastmail.com Tue Nov 24 18:53:07 2015 From: random832 at fastmail.com (Random832) Date: Tue, 24 Nov 2015 23:53:07 +0000 (UTC) Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators References: <5654A89B.40607@brenbarn.net> <5654BA37.8080208@brenbarn.net> Message-ID: On 2015-11-24, Brendan Barnwell wrote: > I guess I'm not understanding what you mean by __true__ then. What is > this __true__ for which ([0, 1]).__true__() is not true? It'd be a new operator / magic-function, so whatever the person writing the class wants it to be. I specifically didn't use bool(). def __true__(self): # return True ony if you want "or" to short-circuit pass From random832 at fastmail.com Tue Nov 24 19:00:07 2015 From: random832 at fastmail.com (Random832) Date: Wed, 25 Nov 2015 00:00:07 +0000 (UTC) Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators References: <5654A89B.40607@brenbarn.net> Message-ID: On 2015-11-24, Nathaniel Smith wrote: > The semantic objection that I raised -- short circuiting means that you > can't correctly overload 'True and numpy_array', Well, you mean "True or" or "False and", but anyway... I've got to confess, I don't really understand the semantic paradigm under which it's appropriate to do this all, but it's not appropriate for it to return True. If you can pass in True here, that implies that the True you'll get out of this operation is an appropriate value for using in further operations on similarly-shaped arrays. What does it _matter_ that you get True (or 1, etc) instead of [True, True, True, True, True, True]? From ncoghlan at gmail.com Wed Nov 25 02:06:04 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Nov 2015 17:06:04 +1000 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <5653FE15.8080402@canterbury.ac.nz> References: <-334649383431326154@unknownmsgid> <5653FE15.8080402@canterbury.ac.nz> Message-ID: On 24 November 2015 at 16:05, Greg Ewing wrote: > Nick Coghlan wrote: >> >> Something that *could* potentially be comprehensible is the idea of >> allowing "elementwise" logical operators, with a suitable syntactic >> spelling. > > -1 on tying any of this explicity to a notion of > elementwise operations. The range of potential uses > is wider than that. While I'm still largely of the view that introducing additional operators would make things more confusing rather than less, I'm also convinced that if anything like this is going to be pursued without being incredibly confusing for beginners there needs to be a fairly concise answer to "What are these operators for?". Take the "bitwise operators", for example. The notion of a "bitwise operator" is conceptually dense for folks that have never worked with binary numbers before. Despite that, if someone asks "What do the & and | operators do in Python?" the semantics can still be conveyed relatively quickly using some truth table examples like: >>> bin(0b101 & 0b110) '0b100' >>> bin(0b101 & 0b101) '0b101' >>> bin(0b101 | 0b110) '0b111' >>> bin(0b101 | 0b101) '0b101' Explaining "~" fully is a bit trickier (since you would need to explain why two's complement representations of binary numbers are useful), but it's possible to avoid that explanation by using the alternative arithmetic formulation for "~" given in https://wiki.python.org/moin/BitwiseOperators : "~x == -x -1" Matrix multiplication is another example of something that isn't particularly easy to explain to folks that aren't already familiar with the relevant domain, but also conveys clearly that you can ignore it if you're not working with matrices. It's then also useful to remember that the answers to "What is this for?" and "How is this used?" for a language construct can diverge over time. The original "What is this for?" use cases are the ones that guide the design decisions towards concrete answers that define how the construct works, and provide the underlying rationale for the way the construct behaves. The "How is this used?" cases then arise later when folks say "Yes, those existing semantics are suitable for my current use case, so I can reuse the syntax". Some specific examples: "+" is used not only for addition, but also sequence concatenation. "&" is not only "bitwise and", but also set intersection "/" is not only division, but also pathlib path joining NumPy repurposes most of the binary operators (including the bitwise ones) as element-wise matrix operations. SQL Alchemy repurposes a number of them for SQL query operations. SymPy changes them from arithmetic operations to symbolic ones. Those use cases don't change the answers to "What are these operators for?" from a language design perspective, they only change the answers to "How are these operators used?" from a practical perspective. Getting back to the specific topic of this thread, this could actually make an interesting usability study for a language design theorist, by looking at the kinds of mistakes folks make trying to learn elementwise logic operations in NumPy, and then seeing whether the introduction of overridable elementwise logical operators reduces the learning curve. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.barker at noaa.gov Wed Nov 25 11:00:37 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Wed, 25 Nov 2015 08:00:37 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: <5654A89B.40607@brenbarn.net> Message-ID: <-1350841247926098108@unknownmsgid> > What does it _matter_ that you get True (or 1, etc) instead of > [True, True, True, True, True, True]? Sometimes you want to know that everything in an array is True ( which is common enough that numpy.alltrue() exists) But other times you want to know which items in an array a true, and which are not -- most often to use as a Boolean mask -- in which case, an array that happens to have all true values is a different beast altogether from a single True. And remember that thee could be arrays of any number of dimensions. Also: Python Truthyness rules define an empty container as False, and any non-empty container as True -- it's probably better not to make arrays unique in that regard. -CHB > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From chris.barker at noaa.gov Wed Nov 25 18:58:46 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Wed, 25 Nov 2015 15:58:46 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: <-334649383431326154@unknownmsgid> <5653FE15.8080402@canterbury.ac.nz> Message-ID: <3518005034548011247@unknownmsgid> > While I'm still largely of the view that introducing additional > operators would make things more confusing rather than less, Well, almost by definition, more stuff to understand is more confusing for beginners. > there needs to be a fairly > concise answer to "What are these operators for?". I don't think "they are for doing logical operations on each of the elements in a sequence, rather than the sequence as a whole", along with an example or two is particularly challenging. In fact, much less so than the Bitwise operators, or matrix multiplication, which require a bit of domain knowledge, as you say. But those aren't a big problem either: "if you don't know what it means, you probably don't need it" But as for general element-wise operators: IIRC, this was discussed a lot back in the day -- and was driven by experience with e.g. Matlab, where the regular math operators do linear algebra by default, and there are alternative "element wise" operators. Numpy, on the other hand, does element-wise by default, so we wanted another set for linear algebra. However, we came to realize that the only one really needed was matrix multiply -- and thus the new @ operator. This all worked because Numpy could overload the math operators to be element wise, and once rich comparisons were implemented, that covered almost everything. So all that's left is and-or. Add the fact that use the Bitwise & and | in their place in most cases, and we've done fine so far. All that being said -- two more operators for "rich and" and "rich or". Would nicely complete the picture. I was just introducing my intro Python class to the magic methods last night -- there are a LOT of them! Two more is pretty trivial Addison of complexity. As long as we can find a way to spell them that is not too confusing or ugly -- I think it's a win-win. Note: having worked with array-oriented languages/libraries for a long time, I'd like element-wise operators that worked with all the built-in types. But I suspect that Python is never going to go there. So we only need these two. -CHB From ncoghlan at gmail.com Thu Nov 26 00:44:17 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Nov 2015 15:44:17 +1000 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <3518005034548011247@unknownmsgid> References: <-334649383431326154@unknownmsgid> <5653FE15.8080402@canterbury.ac.nz> <3518005034548011247@unknownmsgid> Message-ID: On 26 November 2015 at 09:58, Chris Barker - NOAA Federal wrote: >> While I'm still largely of the view that introducing additional >> operators would make things more confusing rather than less, > > Well, almost by definition, more stuff to understand is more confusing > for beginners. > >> there needs to be a fairly >> concise answer to "What are these operators for?". > > I don't think "they are for doing logical operations on each of the > elements in a sequence, rather than the sequence as a whole", along > with an example or two is particularly challenging. > > In fact, much less so than the Bitwise operators, or matrix > multiplication, which require a bit of domain knowledge, as you say. > But those aren't a big problem either: "if you don't know what it > means, you probably don't need it" Right, that's why I think "elementwise logical operators (which may potentially be useful for other things)" is an idea that has some hope of avoiding creating new barriers to learning: * if "elementwise" doesn't mean anything to you, and no library you're using mentions them in its documentation, you can ignore them * for folks that do know what it means, "elementwise" is evocative of the relevant semantics for at least the data analysis use case > But as for general element-wise operators: I wasn't suggesting those - just element-wise logical operators. In terms of scoping the use case: bitwise and/or/not work fine for manipulating existing boolean masks, and converting a single matrix to a boolean mask involves apply an elementwise function, not elementwise logical operations. So elementwise logical operators would presumably be aimed at *data merging* problems - creating a combined matrix where some values are taken from matrix A and others from matrix B, based on the truthiness of those values. I'm not enough of a data analyst to know how common that problem is, or whether it might be better served by a higher level "replace_elements" operation that accepts a base array, a replacement array, and a boolean mask saying which values to replace. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From abarnert at yahoo.com Thu Nov 26 01:27:19 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 25 Nov 2015 22:27:19 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: <-334649383431326154@unknownmsgid> <5653FE15.8080402@canterbury.ac.nz> <3518005034548011247@unknownmsgid> Message-ID: <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com> On Nov 25, 2015, at 21:44, Nick Coghlan wrote: > > On 26 November 2015 at 09:58, Chris Barker - NOAA Federal > wrote: >>> While I'm still largely of the view that introducing additional >>> operators would make things more confusing rather than less, >> >> Well, almost by definition, more stuff to understand is more confusing >> for beginners. >> >>> there needs to be a fairly >>> concise answer to "What are these operators for?". >> >> I don't think "they are for doing logical operations on each of the >> elements in a sequence, rather than the sequence as a whole", along >> with an example or two is particularly challenging. >> >> In fact, much less so than the Bitwise operators, or matrix >> multiplication, which require a bit of domain knowledge, as you say. >> But those aren't a big problem either: "if you don't know what it >> means, you probably don't need it" > > Right, that's why I think "elementwise logical operators (which may > potentially be useful for other things)" is an idea that has some hope > of avoiding creating new barriers to learning: > > * if "elementwise" doesn't mean anything to you, and no library you're > using mentions them in its documentation, you can ignore them But "elementwise" isn't what people doing symbolic computation or most other uses of DSL/expression-tree libraries are doing. Even for ORMs and other query-based libraries like AppScript, where arguably it is what they're doing, they probably aren't thinking of it that way, and wouldn't recognize that it should mean something to them, much less that it's what they're looking for. So I think this is effectively less general/useless than a solution that just allows overloading boolean operators somehow, without adding a distinction between elementwise (and overloadable) and objectwise (and not). > * for folks that do know what it means, "elementwise" is evocative of > the relevant semantics for at least the data analysis use case > >> But as for general element-wise operators: > > I wasn't suggesting those - just element-wise logical operators. In > terms of scoping the use case: bitwise and/or/not work fine for > manipulating existing boolean masks, and converting a single matrix to > a boolean mask involves apply an elementwise function, not elementwise > logical operations. > > So elementwise logical operators would presumably be aimed at *data > merging* problems - creating a combined matrix where some values are > taken from matrix A and others from matrix B, based on the truthiness > of those values. > I'm not enough of a data analyst to know how common > that problem is, or whether it might be better served by a higher > level "replace_elements" operation that accepts a base array, a > replacement array, and a boolean mask saying which values to replace. When I use NumPy, sometimes I'm doing GPU-ish stream operations, which need things like compacting select, which aren't obviously expressible in boolean terms, so I end up looking for methods for everything rather than operators even when they might make sense. But otherwise, when I'm doing more typical NumPy stuff (or at least what I think is more typical, but I could easily be wrong), I look for elementwise operators all over the place, including abusing the bitwise operators when it makes sense, so I probably would use real boolean operators if it were more obvious/readable. From njs at pobox.com Thu Nov 26 01:42:01 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 25 Nov 2015 22:42:01 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: <5654A89B.40607@brenbarn.net> Message-ID: On Nov 24, 2015 11:53 AM, "Guido van Rossum" wrote: > > On Tue, Nov 24, 2015 at 11:37 AM, Nathaniel Smith wrote: > > On Nov 24, 2015 10:21 AM, "Guido van Rossum" wrote: > >> > >> To everyone claiming that you can't overload and/or because they are > >> shortcut operators, please re-read PEP 335. It provides a clean > >> solution -- it was rejected because it adds an extra byte code to all > >> code using those operators (the majority of which don't need it). > > > > The semantic objection that I raised -- short circuiting means that you > > can't correctly overload 'True and numpy_array', because unlike all other > > binops the overload must be defined on the left hand argument -- does apply > > to PEP 335 AFAICT. This problem is IMHO serious enough that even if PEP 335 > > were accepted today I'm not entirely sure that numpy would actually > > implement the overloads due to the headaches it would cause for teaching and > > code review -- we'd have to have some debate about it at least. > > OK, that's useful feedback. Is NumPy interested in coming up with an > alternative that works, or are you fine with the status quo? We'd certainly love it if there were a better alternative, but -- speaking just for myself here -- I've hesitated to wade in because I don't have any brilliant ideas to contribute :-). The right-hand-side overload problem seems like an inevitable consequence of short-circuiting, and we certainly aren't going to switch 'and'/'or' to become eagerly evaluating. OTOH none of the alternative proposals mooted so far have struck me as very compelling or pythonic, if only because all the proposed spellings are ugly, but, who knows, sometimes something awesome appears deep in these threads. Two thoughts on places where it might be easier to make some progress... - the 'not' overloading proposed in PEP 335 doesn't seem to create any horrible problems - it'd be a minor thing, but maybe it's worth pulling out as a standalone change? - the worst code expansion created by lack of overloading isn't a == 1 and b == 2 becoming (a == 1) & (b == 2) but rather 0 < complex expression < 1 becoming tmp = complex expression (0 < tmp) & (tmp < 1) That is, the implicit 'and' inside chained comparisons is the biggest pain point. And this issue is orthogonal to the proposals that involve adding new operators, b/c they only help with explicit 'and'/'or', not implicit 'and'. Which is why I was sounding you out about making chained comparisons eagerly evaluated at the bar at pycon this year ;-). I have the suspicion that the short-circuiting semantics of chained comparisons are more surprising and confusing than they are useful and we should just make them eagerly evaluated, and I know you have the opposite intuition, so I think the next step here would be to collect some data (run a survey, scan some code, ...?) to figure out which of us is right :-). The latter idea in particular has been on my todo list for at least 6 months and still has not bubbled up near the top, so if anyone is interested in pushing it forward then please feel free :-). -n From stephen at xemacs.org Thu Nov 26 02:55:48 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 26 Nov 2015 16:55:48 +0900 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: <-334649383431326154@unknownmsgid> <5653FE15.8080402@canterbury.ac.nz> <3518005034548011247@unknownmsgid> Message-ID: <22102.47876.529469.808030@turnbull.sk.tsukuba.ac.jp> Nick Coghlan writes: > So elementwise logical operators would presumably be aimed at *data > merging* problems - creating a combined matrix where some values are > taken from matrix A and others from matrix B, based on the truthiness > of those values. I'm not enough of a data analyst to know how > common Well, for the social science data analysis I do, that would be inappropriate. Variables from different sources are different variables, you wouldn't just "or" them into a single column of a data frame. You would want your data model to account for the fact that even if they purport to measure the same factor, they're actually different indicators. But for a completely different kind of data, images, that sounds a lot like (Duff's?) compositing operations. But those are a lot more flexible than just "and" and "or": color images are "fuzzy" logic, and so admit many more logical operations (eg, "clamped sum", "proportional combination", etc. I'm not sure how that would fit here, since Python has only a limited number of operator symbols, fewer than there are compositing operations IIRC. Steve From oscar.j.benjamin at gmail.com Thu Nov 26 06:56:36 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 26 Nov 2015 11:56:36 +0000 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: <5654A89B.40607@brenbarn.net> Message-ID: On 26 November 2015 at 06:42, Nathaniel Smith wrote: > > - the worst code expansion created by lack of overloading isn't > a == 1 and b == 2 > becoming > (a == 1) & (b == 2) > but rather > 0 < complex expression < 1 > becoming > tmp = complex expression > (0 < tmp) & (tmp < 1) > That is, the implicit 'and' inside chained comparisons is the biggest > pain point. And this issue is orthogonal to the proposals that involve > adding new operators, b/c they only help with explicit 'and'/'or', not > implicit 'and'. Which is why I was sounding you out about making > chained comparisons eagerly evaluated at the bar at pycon this year > ;-). I have the suspicion that the short-circuiting semantics of > chained comparisons are more surprising and confusing than they are > useful and we should just make them eagerly evaluated, and I know you > have the opposite intuition, so I think the next step here would be to > collect some data (run a survey, scan some code, ...?) to figure out > which of us is right :-). Regardless of which is more useful it would be a very subtle backwards compatibility break. I imagine that code that relies on the short-circuit here is rare. I'm confident that it is much rarer than numpy-ish code that works around this in the way you showed above or just by evaluating the complex expression twice as in (0 < complex expression) & (complex expression < 1) which is what I normally write when the expression isn't too long. But there's guaranteed to be some breakage and a good migration path to mitigate that is unclear. You could add a __future__ import and a warning mode to detect when short-circuiting happens in chained comparisons. Unfortunately the naive implementation of the warning mode would simply trigger on 50% of chained comparisons (every time the left hand relation is False). I would definitely use chained comparisons for numpy arrays if it were possible but at the same time I don't think the status quo on this is that bad. It's a bit of a gotcha for new numpy users to learn but numpy is good at giving the appropriate error messages: >>> from numpy import array >>> 1 < array([1, 2, 3]) < 2 Traceback (most recent call last): File "", line 1, in ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() >>> 1 < array([1, 2, 3]) and 3 Traceback (most recent call last): File "", line 1, in ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() If you google that error message there's lots of SO posts etc. that can explain in more detail. It's also worth noting since we're comparing with Matlab that Matlab actually has both short-circuit logical && and element-wise logical & equivalent to Python's and/& respectively. And Matlab doesn't have chained comparisons or rather they don't do anything nearly as useful as Python's chained comparisons i.e. in Matlab -2 < -1 < 0 is evaluated as: -2 < -1 < 0 (-2 < -1) < 0 1 < 0 0 (i.e. False) which is basically useless and doesn't even give a decent error message like numpy does. -- Oscar From greg.ewing at canterbury.ac.nz Thu Nov 26 15:54:44 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 27 Nov 2015 09:54:44 +1300 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com> References: <-334649383431326154@unknownmsgid> <5653FE15.8080402@canterbury.ac.nz> <3518005034548011247@unknownmsgid> <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com> Message-ID: <56577194.5000003@canterbury.ac.nz> Andrew Barnert via Python-ideas wrote: > But "elementwise" isn't what people doing symbolic computation or most other > uses of DSL/expression-tree libraries are doing. Right. I think just describing them as "overloadable versions of the boolean operators" would be best. What they mean is up to the types concerned, just as with all the oher operators. -- Greg From python at lucidity.plus.com Thu Nov 26 18:22:25 2015 From: python at lucidity.plus.com (Erik) Date: Thu, 26 Nov 2015 23:22:25 +0000 Subject: [Python-ideas] PEP471 - (os.scandir()) Message-ID: <56579431.4010907@lucidity.plus.com> PEP471 introduces a faster way of doing low-level directory traversal which is then used to implement and speed up the higher-level API os.walk() - which for me at least is the "go to API" for most directory scanning code I write. However, when using os.walk() the first thing that one tends to do with the results is to analyse them in some way (look at file sizes, datestamps and other things that stat() returns) which is exactly the information that os.scandir() is caching and speeding up but which is then thrown away in order to emulate os.walk()'s original name-based API (well, name and type as the directory/file distinction is also there). So, I'd like to suggest an os.walk()-like API that returns the os.scandir() DirEntry structures rather than names (*). I have my own local version that's just a copy of os.walk() that appends "entry" rather than "entry.name" to the returned lists, but that's a nasty way of achieving this. How to do it - os.walk() "direntries=True" keyword? os.walkentries() function? Something else better than those? Regards, E. (*) I have studied the PEP, followed a lot of the references and looked at the 3.5.0 implementation. I can't see that I've missed such a thing already existing, but it's possible. If so, perhaps this is instead a request to make that thing more obvious somehow! From guido at python.org Thu Nov 26 19:47:15 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 26 Nov 2015 16:47:15 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: <5654A89B.40607@brenbarn.net> Message-ID: On Wed, Nov 25, 2015 at 10:42 PM, Nathaniel Smith wrote: > On Nov 24, 2015 11:53 AM, "Guido van Rossum" wrote: > > > > On Tue, Nov 24, 2015 at 11:37 AM, Nathaniel Smith wrote: > > > On Nov 24, 2015 10:21 AM, "Guido van Rossum" wrote: > > >> > > >> To everyone claiming that you can't overload and/or because they are > > >> shortcut operators, please re-read PEP 335. It provides a clean > > >> solution -- it was rejected because it adds an extra byte code to all > > >> code using those operators (the majority of which don't need it). > > > > > > The semantic objection that I raised -- short circuiting means that you > > > can't correctly overload 'True and numpy_array', because unlike all > other > > > binops the overload must be defined on the left hand argument -- does > apply > > > to PEP 335 AFAICT. This problem is IMHO serious enough that even if > PEP 335 > > > were accepted today I'm not entirely sure that numpy would actually > > > implement the overloads due to the headaches it would cause for > teaching and > > > code review -- we'd have to have some debate about it at least. > > > > OK, that's useful feedback. Is NumPy interested in coming up with an > > alternative that works, or are you fine with the status quo? > > We'd certainly love it if there were a better alternative, but -- > speaking just for myself here -- I've hesitated to wade in because I > don't have any brilliant ideas to contribute :-). The right-hand-side > overload problem seems like an inevitable consequence of > short-circuiting, and we certainly aren't going to switch 'and'/'or' > to become eagerly evaluating. OTOH none of the alternative proposals > mooted so far have struck me as very compelling or pythonic, if only > because all the proposed spellings are ugly, but, who knows, sometimes > something awesome appears deep in these threads. > > Two thoughts on places where it might be easier to make some progress... > > - the 'not' overloading proposed in PEP 335 doesn't seem to create any > horrible problems - it'd be a minor thing, but maybe it's worth > pulling out as a standalone change? > This seems pretty harmless, and mostly orthogonal to the rest -- except that overloadable 'not' is not very attractive or useful by itself if we decide not to address the others. > - the worst code expansion created by lack of overloading isn't > a == 1 and b == 2 > becoming > (a == 1) & (b == 2) > but rather > 0 < complex expression < 1 > becoming > tmp = complex expression > (0 < tmp) & (tmp < 1) > That is, the implicit 'and' inside chained comparisons is the biggest > pain point. And this issue is orthogonal to the proposals that involve > adding new operators, b/c they only help with explicit 'and'/'or', not > implicit 'and'. Which is why I was sounding you out about making > chained comparisons eagerly evaluated at the bar at pycon this year > ;-). I have the suspicion that the short-circuiting semantics of > chained comparisons are more surprising and confusing than they are > useful and we should just make them eagerly evaluated, and I know you > have the opposite intuition, so I think the next step here would be to > collect some data (run a survey, scan some code, ...?) to figure out > which of us is right :-). > > The latter idea in particular has been on my todo list for at least 6 > months and still has not bubbled up near the top, so if anyone is > interested in pushing it forward then please feel free :-). What to do with chaining comparisons is a very good question, but changing them to be non-short-circuiting would cause a world of backwards incompatible pain. For example, I've definitely written code where I carefully arranged the comparisons so that the most expensive one comes last (in hopes of sometimes avoiding the time spent on it if the outcome is already determined). This is particularly easy with chained ==, but you can sometimes also change a < b < c into c > b > a, if a happens to be the expensive one. However, note that PEP 335 *does* address the problem of chained comparisons head-on -- in a < b < c, if a < b returns a numpy array (or some other special object that's not falsey), it will then proceed to compute b < c and combine the two using the overloading of the default 'and'; because the first result is not a simple bool, the problem you described with `True and numpy_array` does not apply. So, maybe you and the numpy community can ponder PEP 335 some more? Honestly, from the general language design POV (i.e., mine :-), PEP 335 feels more acceptable than introducing new non-short-circuit and/or operators. How common would the `True and numpy_array` problem really be? I suppose any real occurrences would not use the literal True; `True and x` is just a wordy way to spell x, and doing this element-wise would just return the array x unchanged. (Or would it cast the elements to bool? That still feels like a unary operator to me that deserves a more direct spelling.) I don't see a use case for a literal left operator in symbolic algebra or SQL either. But let's assume we have some scalar expression that evaluates to a bool. Even then, `x and numpy_array` feels like a clumsy way to spell `numpy_array if x else `. But I suppose you've thought about this more than I have. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Nov 26 21:11:32 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 27 Nov 2015 12:11:32 +1000 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <56577194.5000003@canterbury.ac.nz> References: <-334649383431326154@unknownmsgid> <5653FE15.8080402@canterbury.ac.nz> <3518005034548011247@unknownmsgid> <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com> <56577194.5000003@canterbury.ac.nz> Message-ID: On 27 November 2015 at 06:54, Greg Ewing wrote: > Andrew Barnert via Python-ideas wrote: >> >> But "elementwise" isn't what people doing symbolic computation or most >> other >> uses of DSL/expression-tree libraries are doing. > > Right. I think just describing them as "overloadable versions > of the boolean operators" would be best. What they mean is > up to the types concerned, just as with all the oher operators. Except that *isn't* what we do with the other operators. "&" is the bitwise and operator, for example - that's why the special method is called "__and__". The fact you can use it for an elementwise bitwise and operation on NumPy arrays, or for set intersection, isn't part of the core design. If there isn't *at least one* specific motivating use case, then "it might be useful for something" isn't a good reason to add new syntax. However, looking again at PEP 335, I'm not sure I see any reason it needs to noticeably slower in the standard case than the status quo (I'm not saying it would be *easy* to retain the speed, but the complexity would be in the eval loop implementation and the code generation process, not user code). We also have the richer benchmark suite these days to actually quantify the impact of checking for the new __and1__/__or1__ slots before falling back to __bool__, and tracing JIT's would still be able to generate appropriate code for the fast path at runtime. So perhaps it might be worth dusting off that original idea and seeing what the impact is on the performance benchmarks? Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ericfahlgren at gmail.com Fri Nov 27 08:49:55 2015 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Fri, 27 Nov 2015 05:49:55 -0800 Subject: [Python-ideas] PEP471 - (os.scandir()) In-Reply-To: <56579431.4010907@lucidity.plus.com> References: <56579431.4010907@lucidity.plus.com> Message-ID: <037b01d1291a$80123360$80369a20$@gmail.com> > -----Original Message----- > From: Erik [mailto:python at lucidity.plus.com] > Sent: Thursday, November 26, 2015 15:22 > To: python-ideas > Subject: [Python-ideas] PEP471 - (os.scandir()) > > PEP471 introduces a faster way of doing low-level directory traversal which is then used to implement and speed up the higher-level API > os.walk() - which for me at least is the "go to API" for most directory scanning code I write. > > However, when using os.walk() the first thing that one tends to do with the results is to analyse them in some way (look at file sizes, datestamps and other things that stat() returns) which is exactly the information that os.scandir() is caching and speeding up but which is then thrown away in order to emulate os.walk()'s original name-based API (well, name and type as the directory/file distinction is also there). > > So, I'd like to suggest an os.walk()-like API that returns the > os.scandir() DirEntry structures rather than names (*). I have my own local version that's just a copy of os.walk() that appends "entry" > rather than "entry.name" to the returned lists, but that's a nasty way of achieving this. > > How to do it - > > os.walk() "direntries=True" keyword? > os.walkentries() function? > Something else better than those? "walk" + "scandir" = "walkdir"??? I'm definitely +1 on this, as it is fresh on my mind, too. I just converted our build tools over to use a homebrew walk as you did, and now use DirEntry instead of path names almost exclusively. EricF From abarnert at yahoo.com Fri Nov 27 12:28:10 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 27 Nov 2015 09:28:10 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: <-334649383431326154@unknownmsgid> <5653FE15.8080402@canterbury.ac.nz> <3518005034548011247@unknownmsgid> <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com> <56577194.5000003@canterbury.ac.nz> Message-ID: <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com> On Nov 26, 2015, at 18:11, Nick Coghlan wrote: > >> On 27 November 2015 at 06:54, Greg Ewing wrote: >> Andrew Barnert via Python-ideas wrote: >>> >>> But "elementwise" isn't what people doing symbolic computation or most >>> other >>> uses of DSL/expression-tree libraries are doing. >> >> Right. I think just describing them as "overloadable versions >> of the boolean operators" would be best. What they mean is >> up to the types concerned, just as with all the oher operators. > > Except that *isn't* what we do with the other operators. "&" is the > bitwise and operator, for example - that's why the special method is > called "__and__". The fact you can use it for an elementwise bitwise > and operation on NumPy arrays, or for set intersection, isn't part of > the core design. If there isn't *at least one* specific motivating use > case, then "it might be useful for something" isn't a good reason to > add new syntax. Sure. But the sqlanywhere case was the very first motivating use the OP mentioned, before NumPy, not something that may come up in the future that we haven't imagined yet. Also, the distinguishing thing about this new magic method vs. the existing __and__ isn't that it's elementwise, but that it's boolean/logical rather than bitwise/arithmetic, even for NumPy users. So, calling it "elementwise and", or giving it a name that implies elementwise, will confuse anyone who hasn't read this whole thread. And finally, NumPy is one of the uses that doesn't require short circuiting, and the same is almost certainly true for other elementwise uses, and yet we seem to all be agreed that the new overload has to be short-circuitable. > However, looking again at PEP 335, I'm not sure I see any reason it > needs to noticeably slower in the standard case than the status quo > (I'm not saying it would be *easy* to retain the speed, but the > complexity would be in the eval loop implementation and the code > generation process, not user code). We also have the richer benchmark > suite these days to actually quantify the impact of checking for the > new __and1__/__or1__ slots before falling back to __bool__, and > tracing JIT's would still be able to generate appropriate code for the > fast path at runtime. > > So perhaps it might be worth dusting off that original idea and seeing > what the impact is on the performance benchmarks? > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From wjun77 at gmail.com Fri Nov 27 12:49:32 2015 From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=) Date: Sat, 28 Nov 2015 01:49:32 +0800 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism Message-ID: Hello everyone: I'm suggesting a modification to the AttributeError/__getattr__ mechanism, see issue25634: http://bugs.python.org/issue25634 I used __getattr__ sometimes, and descriptor especially property is so widely used. I wonder whether someone had encountered the same problem with me. However, this is a complicated problem including performance issues, and backward compatibility. Thanks for your attention, Jun Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Fri Nov 27 13:20:01 2015 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 27 Nov 2015 18:20:01 +0000 Subject: [Python-ideas] PEP471 - (os.scandir()) In-Reply-To: <037b01d1291a$80123360$80369a20$@gmail.com> References: <56579431.4010907@lucidity.plus.com> <037b01d1291a$80123360$80369a20$@gmail.com> Message-ID: <56589ED1.6030805@mrabarnett.plus.com> On 2015-11-27 13:49, Eric Fahlgren wrote: >> -----Original Message----- >> From: Erik [mailto:python at lucidity.plus.com] >> Sent: Thursday, November 26, 2015 15:22 >> To: python-ideas >> Subject: [Python-ideas] PEP471 - (os.scandir()) >> >> PEP471 introduces a faster way of doing low-level directory traversal which is then used to implement and speed up the higher-level API >> os.walk() - which for me at least is the "go to API" for most directory scanning code I write. >> >> However, when using os.walk() the first thing that one tends to do with the results is to analyse them in some way (look at file sizes, datestamps and other things that stat() returns) which is exactly the information that os.scandir() is caching and speeding up but which is then thrown away in order to emulate os.walk()'s original name-based API (well, name and type as the directory/file distinction is also there). >> >> So, I'd like to suggest an os.walk()-like API that returns the >> os.scandir() DirEntry structures rather than names (*). I have my own local version that's just a copy of os.walk() that appends "entry" >> rather than "entry.name" to the returned lists, but that's a nasty way of achieving this. >> >> How to do it - >> >> os.walk() "direntries=True" keyword? >> os.walkentries() function? >> Something else better than those? > > "walk" + "scandir" = "walkdir"??? > > I'm definitely +1 on this, as it is fresh on my mind, too. I just converted our build tools over to use a homebrew walk as you did, and now use DirEntry instead of path names almost exclusively. > There's nothing in that name that suggests "scandir" rather that "listdir". You could just as easily say "walk" + "listdir" = "walkdir". From p.f.moore at gmail.com Fri Nov 27 13:20:35 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 27 Nov 2015 18:20:35 +0000 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: References: Message-ID: On 27 November 2015 at 17:49, ?? wrote: > I used __getattr__ sometimes, and descriptor especially property is so > widely used. I wonder whether someone had encountered the same problem with > me. > > However, this is a complicated problem including performance issues, and > backward compatibility. I understand the issue you are reporting (and thanks for stripping it down to a simple example in the issue) but could you give a real world example of how this might actually be an issue in practice? I don't believe I've ever heard anyone report this as an issue before, which implies to me that it's not a particularly common problem in real code. I'm not 100% clear on how bad any backward compatibility or performance issues might be, but it's certainly true that we'd need real-world use cases to justify the cost of the change. Paul From p.f.moore at gmail.com Fri Nov 27 13:32:09 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 27 Nov 2015 18:32:09 +0000 Subject: [Python-ideas] PEP471 - (os.scandir()) In-Reply-To: <56579431.4010907@lucidity.plus.com> References: <56579431.4010907@lucidity.plus.com> Message-ID: On 26 November 2015 at 23:22, Erik wrote: > I have studied the PEP, followed a lot of the references and looked at the > 3.5.0 implementation. I can't see that I've missed such a thing already > existing, but it's possible. If so, perhaps this is instead a request to > make that thing more obvious somehow! Does pathlib use scandir? If so, then maybe you get the caching benefits by using pathlib? And if pathlib doesn't use scandir, maybe it should? [I just checked, it looks like pathlib doesn't use scandir :-(] Paul From abarnert at yahoo.com Fri Nov 27 13:34:24 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 27 Nov 2015 10:34:24 -0800 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: References: Message-ID: It seems like you can get some of the benefits of this proposal without backward compat issues. Instead of changing things so AttributeError from __getattribute__ or a descriptor no longer calls __getattr__, just add a new subclass that doesn't, and change the Descriptor HOWTO to suggest using that subclass (with a bit of discussion that says you can use AttributeError if you want to trigger __getattr__, but usually you won't). That wouldn't fix any current code, but it also wouldn't break any code that intentionally uses the features as documented. And it would make it easy to write correct new code. One more thing: > Without descriptor, unexpected AttributeError could only come from overriding __getattribute__, which is a rare case, although still an imperfection. Is that really an imperfection? It says right there in the docs for __getattribute__ that you can delegate to __getattr__ by raising AttributeError, so if someone does that, presumably it's intentional. It's not like the case with descriptors, where you have to think through the interaction of multiple features to figure out that raising an AttributeError will call __getattr__, and therefore many such uses are probably bugs. Sent from my iPhone > On Nov 27, 2015, at 09:49, ?? wrote: > > Hello everyone: > > I'm suggesting a modification to the AttributeError/__getattr__ mechanism, see issue25634: > http://bugs.python.org/issue25634 > > I used __getattr__ sometimes, and descriptor especially property is so widely used. I wonder whether someone had encountered the same problem with me. > > However, this is a complicated problem including performance issues, and backward compatibility. > > Thanks for your attention, > Jun Wang > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Nov 27 13:34:45 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 27 Nov 2015 18:34:45 +0000 Subject: [Python-ideas] PEP471 - (os.scandir()) In-Reply-To: References: <56579431.4010907@lucidity.plus.com> Message-ID: On 27 November 2015 at 18:32, Paul Moore wrote: > On 26 November 2015 at 23:22, Erik wrote: >> I have studied the PEP, followed a lot of the references and looked at the >> 3.5.0 implementation. I can't see that I've missed such a thing already >> existing, but it's possible. If so, perhaps this is instead a request to >> make that thing more obvious somehow! > > Does pathlib use scandir? If so, then maybe you get the caching > benefits by using pathlib? And if pathlib doesn't use scandir, maybe > it should? [I just checked, it looks like pathlib doesn't use scandir > :-(] Never mind - see https://www.python.org/dev/peps/pep-0471/#return-values-being-pathlib-path-objects Pathlib objects must not cache the results of stat calls, so they cannot use scandir. Paul From abarnert at yahoo.com Fri Nov 27 13:42:31 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 27 Nov 2015 10:42:31 -0800 Subject: [Python-ideas] PEP471 - (os.scandir()) In-Reply-To: References: <56579431.4010907@lucidity.plus.com> Message-ID: <3E188123-D4E5-4913-AA56-53A9CEF85BDD@yahoo.com> On Nov 27, 2015, at 10:32, Paul Moore wrote: > >> On 26 November 2015 at 23:22, Erik wrote: >> I have studied the PEP, followed a lot of the references and looked at the >> 3.5.0 implementation. I can't see that I've missed such a thing already >> existing, but it's possible. If so, perhaps this is instead a request to >> make that thing more obvious somehow! > > Does pathlib use scandir? If so, then maybe you get the caching > benefits by using pathlib? And if pathlib doesn't use scandir, maybe > it should? [I just checked, it looks like pathlib doesn't use scandir > :-(] Does pathlib even have a walk equivalent? (I know it has glob('**'), but that's not the same thing.) Or are you suggesting that people should use path.iterdir with explicit recursion (or an explicit stack), and therefore just changing iterdir to use scandir (and prefill as many cached attribs as possible in each result) is what we want? From cuthbert at mit.edu Fri Nov 27 14:29:36 2015 From: cuthbert at mit.edu (Michael Scott Cuthbert) Date: Fri, 27 Nov 2015 19:29:36 +0000 Subject: [Python-ideas] add a single __future__ for py3? Message-ID: <0540E031-580B-4ED7-9E9D-7E37C84959E4@mit.edu> (an old list topic that I accidentally sent to the Google Groups list) One smaller request is to remove 'barry_as_FLUFL' from "__future__.all_feature_names". It's a great practical joke, but it bit me hard on a project where I check that, for Py2/Py3 consistency, each module imports all relevant __future__ names, and ensures that doctests on Py2 are run with them on. I figured that I could iterate through .all_feature_names and turn them all on except unicode_literals and be safe, but suddenly I was getting Syntax errors on all my != tests, and there was not anywhere to turn to for an explanation. Barry might be pleased. In any case it seems better to treat it like "braces" which isn?t listed in "all_feature_names" Best, Myke Cuthbert (long-time-reader-first-time-writer intro: switched from Perl to Python in 2006, wrote music21 (music analysis toolkit) in Python, got tenure at MIT for it. Hugely thankful to the amazing community) From brenbarn at brenbarn.net Fri Nov 27 14:43:14 2015 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Fri, 27 Nov 2015 11:43:14 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com> References: <-334649383431326154@unknownmsgid> <5653FE15.8080402@canterbury.ac.nz> <3518005034548011247@unknownmsgid> <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com> <56577194.5000003@canterbury.ac.nz> <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com> Message-ID: <5658B252.1060902@brenbarn.net> On 2015-11-27 09:28, Andrew Barnert via Python-ideas wrote: > And finally, NumPy is one of the uses that doesn't require short > circuiting, and the same is almost certainly true for other > elementwise uses, and yet we seem to all be agreed that the new > overload has to be short-circuitable. Do we? I don't. I agree that if we add a way to overload the existing and/or to support these new usages, then that has to be short-circuitable. because and/or currently are short-circuitable and we can't get rid of that. But to me one of the attractive aspects of this new proposal is that the new operators need not be short-circuitable, which would avoid the various contortions required in a scheme like PEP 335 and thus greatly simplify the overloading. In other words the whole point of these new operators would be to do and-like and/or or-like operations that definitely do want both of their arguments all the time (such as elementwise operations or combining abstract query objects like in these SQL cases). -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From guido at python.org Fri Nov 27 15:15:42 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 27 Nov 2015 12:15:42 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <5658B252.1060902@brenbarn.net> References: <-334649383431326154@unknownmsgid> <5653FE15.8080402@canterbury.ac.nz> <3518005034548011247@unknownmsgid> <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com> <56577194.5000003@canterbury.ac.nz> <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com> <5658B252.1060902@brenbarn.net> Message-ID: On Fri, Nov 27, 2015 at 11:43 AM, Brendan Barnwell wrote: > On 2015-11-27 09:28, Andrew Barnert via Python-ideas wrote: > >> And finally, NumPy is one of the uses that doesn't require short >> circuiting, and the same is almost certainly true for other >> elementwise uses, and yet we seem to all be agreed that the new >> overload has to be short-circuitable. >> > > Do we? I don't. I agree that if we add a way to overload the > existing and/or to support these new usages, then that has to be > short-circuitable. because and/or currently are short-circuitable and we > can't get rid of that. But to me one of the attractive aspects of this new > proposal is that the new operators need not be short-circuitable, which > would avoid the various contortions required in a scheme like PEP 335 and > thus greatly simplify the overloading. In other words the whole point of > these new operators would be to do and-like and/or or-like operations that > definitely do want both of their arguments all the time (such as > elementwise operations or combining abstract query objects like in these > SQL cases). > Agreed, that final clause from Andrew seems a non-sequitur. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From wjun77 at gmail.com Fri Nov 27 15:18:39 2015 From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=) Date: Sat, 28 Nov 2015 04:18:39 +0800 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: References: Message-ID: I think the real world use case, as *property* is so common, is just when __getattr__ needed. So anyone using __getattr__ in practice? In fact I'm new tweaking these advanced features of python. Any suggestions are welcome and appreciated. In my case, I delegate failed attribute lookups to one member of the instance, as the given simple example in issue25634. For example, there's a class Point, class Point(): def __init__(self, x, y): self.x = x self.y = y and class Circle class Circle(): def __init__(self, center, radius): self.center = center self.radius = radius I don't think Circle should inherit Point. But I want to write Circle().x instead of Circle().center.x, so I write something like def __getattr__(self, name): try: return getattr(self.center, name) except AttributeError: raise AttributeError("'{}' object has no attribute '{}'".format(self.__class__.__name__, name)) from None Another case is when I try to implement the design pattern of state. That is, when calling window.rightClick(), the behavior differs according to the state of the window, or window.__state. So class ActiveState(State): @staticmethod def rightClick(self): print('right clicked') class InactiveState(State): @staticmethod def rightClick(self): pass and class Window(): def __init__(self): self.__state = ActiveState def __getattr__(self, name): try: return partial(getattr(self.__state, name), self) except AttributeError: raise AttributeError("'{}' object has no attribute '{}'".format(self.__class__.__name__, name)) from None (The real situation is more complicated. In fact I've written a state module, and I think I can publish it in a week if anyone is interested.) > Is that really an imperfection? It says right there in the docs for __getattribute__ that you can delegate to __getattr__ by raising AttributeError Just like raising *StopIteration* in generator, as described in PEP479. Although I'm not sure *StopIteration* has ever been documented as one way to exit generator. And I admit that an unintentional AttributeError in __getattribute__ is a rare case if ever. 2015-11-28 2:34 GMT+08:00 Andrew Barnert : > It seems like you can get some of the benefits of this proposal without > backward compat issues. > > Instead of changing things so AttributeError from __getattribute__ or a > descriptor no longer calls __getattr__, just add a new subclass that > doesn't, and change the Descriptor HOWTO to suggest using that subclass > (with a bit of discussion that says you can use AttributeError if you want > to trigger __getattr__, but usually you won't). > > That wouldn't fix any current code, but it also wouldn't break any code > that intentionally uses the features as documented. And it would make it > easy to write correct new code. > > One more thing: > > Without descriptor, unexpected AttributeError could only come from overriding __getattribute__, which is a rare case, although still an imperfection. > > Is that really an imperfection? It says right there in the docs for > __getattribute__ that you can delegate to __getattr__ by raising > AttributeError, so if someone does that, presumably it's intentional. It's > not like the case with descriptors, where you have to think through the > interaction of multiple features to figure out that raising an > AttributeError will call __getattr__, and therefore many such uses are > probably bugs. > > Sent from my iPhone > > On Nov 27, 2015, at 09:49, ?? wrote: > > Hello everyone: > > I'm suggesting a modification to the AttributeError/__getattr__ mechanism, > see issue25634: > http://bugs.python.org/issue25634 > > I used __getattr__ sometimes, and descriptor especially property is so > widely used. I wonder whether someone had encountered the same problem with > me. > > However, this is a complicated problem including performance issues, and > backward compatibility. > > Thanks for your attention, > Jun Wang > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Nov 27 15:20:52 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 27 Nov 2015 12:20:52 -0800 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: References: Message-ID: I'm confused about how the proposal would work. How would you prevent self.x.t from also raising AttributeMissError (thereby defeating its purpose)? On Fri, Nov 27, 2015 at 12:18 PM, ?? wrote: > I think the real world use case, as *property* is so common, is just when > __getattr__ needed. So anyone using __getattr__ in practice? In fact I'm > new tweaking these advanced features of python. Any suggestions are welcome > and appreciated. > > > In my case, I delegate failed attribute lookups to one member of the > instance, as the given simple example in issue25634. For example, there's a > class Point, > > class Point(): > def __init__(self, x, y): > self.x = x > self.y = y > > and class Circle > > class Circle(): > def __init__(self, center, radius): > self.center = center > self.radius = radius > > I don't think Circle should inherit Point. But I want to write Circle().x > instead of Circle().center.x, so I write something like > def __getattr__(self, name): > try: > return getattr(self.center, name) > except AttributeError: > raise AttributeError("'{}' object has no attribute > '{}'".format(self.__class__.__name__, name)) from None > > > Another case is when I try to implement the design pattern of state. That > is, when calling window.rightClick(), the behavior differs according to the > state of the window, or window.__state. So > > class ActiveState(State): > @staticmethod > def rightClick(self): > print('right clicked') > class InactiveState(State): > @staticmethod > def rightClick(self): > pass > > and > > class Window(): > def __init__(self): > self.__state = ActiveState > def __getattr__(self, name): > try: > return partial(getattr(self.__state, name), self) > except AttributeError: > raise AttributeError("'{}' object has no attribute > '{}'".format(self.__class__.__name__, name)) from None > > (The real situation is more complicated. In fact I've written a state > module, and I think I can publish it in a week if anyone is interested.) > > > > Is that really an imperfection? It says right there in the docs for > __getattribute__ that you can delegate to __getattr__ by raising > AttributeError > Just like raising *StopIteration* in generator, as described in PEP479. > Although I'm not sure *StopIteration* has ever been documented as one way > to exit generator. And I admit that an unintentional AttributeError in > __getattribute__ is a rare case if ever. > > 2015-11-28 2:34 GMT+08:00 Andrew Barnert : > >> It seems like you can get some of the benefits of this proposal without >> backward compat issues. >> >> Instead of changing things so AttributeError from __getattribute__ or a >> descriptor no longer calls __getattr__, just add a new subclass that >> doesn't, and change the Descriptor HOWTO to suggest using that subclass >> (with a bit of discussion that says you can use AttributeError if you want >> to trigger __getattr__, but usually you won't). >> >> That wouldn't fix any current code, but it also wouldn't break any code >> that intentionally uses the features as documented. And it would make it >> easy to write correct new code. >> >> One more thing: >> >> Without descriptor, unexpected AttributeError could only come from overriding __getattribute__, which is a rare case, although still an imperfection. >> >> Is that really an imperfection? It says right there in the docs for >> __getattribute__ that you can delegate to __getattr__ by raising >> AttributeError, so if someone does that, presumably it's intentional. It's >> not like the case with descriptors, where you have to think through the >> interaction of multiple features to figure out that raising an >> AttributeError will call __getattr__, and therefore many such uses are >> probably bugs. >> >> Sent from my iPhone >> >> On Nov 27, 2015, at 09:49, ?? wrote: >> >> Hello everyone: >> >> I'm suggesting a modification to the AttributeError/__getattr__ >> mechanism, see issue25634: >> http://bugs.python.org/issue25634 >> >> I used __getattr__ sometimes, and descriptor especially property is so >> widely used. I wonder whether someone had encountered the same problem with >> me. >> >> However, this is a complicated problem including performance issues, and >> backward compatibility. >> >> Thanks for your attention, >> Jun Wang >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From wjun77 at gmail.com Fri Nov 27 15:32:18 2015 From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=) Date: Sat, 28 Nov 2015 04:32:18 +0800 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: References: Message-ID: At first I think of adding AttributeMissError. In that case, object.__getattr__ (here self.x.__getattr__) could turn AttributeMissError to AttributeError. But after some consideration I think there's no need to add an extra Exception. Just ignore the title of that issue. 2015-11-28 4:20 GMT+08:00 Guido van Rossum : > I'm confused about how the proposal would work. How would you prevent > self.x.t from also raising AttributeMissError (thereby defeating its > purpose)? > > On Fri, Nov 27, 2015 at 12:18 PM, ?? wrote: > >> I think the real world use case, as *property* is so common, is just >> when __getattr__ needed. So anyone using __getattr__ in practice? In fact >> I'm new tweaking these advanced features of python. Any suggestions are >> welcome and appreciated. >> >> >> In my case, I delegate failed attribute lookups to one member of the >> instance, as the given simple example in issue25634. For example, there's a >> class Point, >> >> class Point(): >> def __init__(self, x, y): >> self.x = x >> self.y = y >> >> and class Circle >> >> class Circle(): >> def __init__(self, center, radius): >> self.center = center >> self.radius = radius >> >> I don't think Circle should inherit Point. But I want to write Circle().x >> instead of Circle().center.x, so I write something like >> def __getattr__(self, name): >> try: >> return getattr(self.center, name) >> except AttributeError: >> raise AttributeError("'{}' object has no attribute >> '{}'".format(self.__class__.__name__, name)) from None >> >> >> Another case is when I try to implement the design pattern of state. That >> is, when calling window.rightClick(), the behavior differs according to the >> state of the window, or window.__state. So >> >> class ActiveState(State): >> @staticmethod >> def rightClick(self): >> print('right clicked') >> class InactiveState(State): >> @staticmethod >> def rightClick(self): >> pass >> >> and >> >> class Window(): >> def __init__(self): >> self.__state = ActiveState >> def __getattr__(self, name): >> try: >> return partial(getattr(self.__state, name), self) >> except AttributeError: >> raise AttributeError("'{}' object has no attribute >> '{}'".format(self.__class__.__name__, name)) from None >> >> (The real situation is more complicated. In fact I've written a state >> module, and I think I can publish it in a week if anyone is interested.) >> >> >> > Is that really an imperfection? It says right there in the docs for >> __getattribute__ that you can delegate to __getattr__ by raising >> AttributeError >> Just like raising *StopIteration* in generator, as described in PEP479. >> Although I'm not sure *StopIteration* has ever been documented as one >> way to exit generator. And I admit that an unintentional AttributeError in >> __getattribute__ is a rare case if ever. >> >> 2015-11-28 2:34 GMT+08:00 Andrew Barnert : >> >>> It seems like you can get some of the benefits of this proposal without >>> backward compat issues. >>> >>> Instead of changing things so AttributeError from __getattribute__ or a >>> descriptor no longer calls __getattr__, just add a new subclass that >>> doesn't, and change the Descriptor HOWTO to suggest using that subclass >>> (with a bit of discussion that says you can use AttributeError if you want >>> to trigger __getattr__, but usually you won't). >>> >>> That wouldn't fix any current code, but it also wouldn't break any code >>> that intentionally uses the features as documented. And it would make it >>> easy to write correct new code. >>> >>> One more thing: >>> >>> Without descriptor, unexpected AttributeError could only come from overriding __getattribute__, which is a rare case, although still an imperfection. >>> >>> Is that really an imperfection? It says right there in the docs for >>> __getattribute__ that you can delegate to __getattr__ by raising >>> AttributeError, so if someone does that, presumably it's intentional. It's >>> not like the case with descriptors, where you have to think through the >>> interaction of multiple features to figure out that raising an >>> AttributeError will call __getattr__, and therefore many such uses are >>> probably bugs. >>> >>> Sent from my iPhone >>> >>> On Nov 27, 2015, at 09:49, ?? wrote: >>> >>> Hello everyone: >>> >>> I'm suggesting a modification to the AttributeError/__getattr__ >>> mechanism, see issue25634: >>> http://bugs.python.org/issue25634 >>> >>> I used __getattr__ sometimes, and descriptor especially property is so >>> widely used. I wonder whether someone had encountered the same problem with >>> me. >>> >>> However, this is a complicated problem including performance issues, and >>> backward compatibility. >>> >>> Thanks for your attention, >>> Jun Wang >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Nov 27 16:23:08 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 28 Nov 2015 08:23:08 +1100 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: References: Message-ID: On Sat, Nov 28, 2015 at 7:32 AM, ?? wrote: > At first I think of adding AttributeMissError. In that case, > object.__getattr__ (here self.x.__getattr__) could turn AttributeMissError > to AttributeError. > But after some consideration I think there's no need to add an extra > Exception. Just ignore the title of that issue. So... if I'm understanding the issue correctly, it's about the interaction of @property and __getattr__? You're talking about this as a boundary over which AttributeError changes. Borrowing your example from the tracker issue: class property(property): def __get__(self, *a): try: return super().__get__(*a) except AttributeError as e: raise RuntimeError("Property raised AttributeError") from e class A(): def __init__(self, x=None): self.x = x @property def t(self): return self.x.t def __getattr__(self, name): return 'default' print(A().t) Traceback (most recent call last): File "attrerr.py", line 4, in __get__ return super().__get__(*a) File "attrerr.py", line 14, in t return self.x.t AttributeError: 'NoneType' object has no attribute 't' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "attrerr.py", line 19, in print(A().t) File "attrerr.py", line 6, in __get__ raise RuntimeError("Property raised AttributeError") from e RuntimeError: Property raised AttributeError This would create a boundary, same as PEP 479 does for StopIteration, across which AttributeError becomes RuntimeError. This could be incorporated into the built-in property if desired, or kept on a per-module basis with the above. ChrisA From abarnert at yahoo.com Fri Nov 27 16:37:51 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 27 Nov 2015 13:37:51 -0800 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: References: Message-ID: On Nov 27, 2015, at 12:18, ?? wrote: > > I think the real world use case, as property is so common, is just when __getattr__ needed. So anyone using __getattr__ in practice? Your own examples seem like reasonable uses of __getattr__ to me. But they don't demonstrate your problem, because you don't have any properties, other custom descriptors, or __getattribute__. Do you have any examples that actually do demonstrate the problem to be solved? > > Is that really an imperfection? It says right there in the docs for __getattribute__ that you can delegate to __getattr__ by raising AttributeError > Just like raising StopIteration in generator, as described in PEP479. But the docs didn't recommend raising StopIteration to exit a generator. The docs implied it by omission, or at least we're ambiguous about what would happen, and you could test it and see that it did, which meant some people took advantage of it to do things like using a StopIteration-throwing function in a comprehension filter clause (with caveats about listcomp vs. genexpr and 2.x vs. 3.x and most readers having to guess why it works). But surely breaking that isn't the same as breaking code that's been explicitly stated to work, and used as sample code, for decades. > Although I'm not sure StopIteration has ever been documented as one way to exit generator. And I admit that an unintentional AttributeError in __getattribute__ is a rare case if ever. That's my point: "fixing" __getattribute__ to eliminate a "rare case if ever" bug, while also introducing a possibly less-rare backward compatibility problem, seems like a terrible idea. Of course that isn't your intention; it's just a side effect of trying to eliminate a more common bug, with probably rarer intentional uses, in descriptors. But to me, that implies that any solution that can fix descriptors without also "fixing" __getattribute__ is a lot better. And, as you imply in the bug report, this could be done by having either the descriptor mechanism itself, or the object.__getattribute__ implementation, handle AttributeError from descriptor lookup by reraising it as something else. But still, this is all a minor side issue about choosing between your different variations. The big problem is that I think all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so. Maybe you don't want to do so very often, but is it really so rare that we can justify making it impossible? (Especially considering the backward-compat issues for any code that's been doing that for years...) You didn't comment on the alternative I suggested; would it not satisfy your needs, or have some other problem that makes it unacceptable? > 2015-11-28 2:34 GMT+08:00 Andrew Barnert : >> It seems like you can get some of the benefits of this proposal without backward compat issues. >> >> Instead of changing things so AttributeError from __getattribute__ or a descriptor no longer calls __getattr__, just add a new subclass that doesn't, and change the Descriptor HOWTO to suggest using that subclass (with a bit of discussion that says you can use AttributeError if you want to trigger __getattr__, but usually you won't). >> >> That wouldn't fix any current code, but it also wouldn't break any code that intentionally uses the features as documented. And it would make it easy to write correct new code. >> >> One more thing: >>> Without descriptor, unexpected AttributeError could only come from overriding __getattribute__, which is a rare case, although still an imperfection. >> >> Is that really an imperfection? It says right there in the docs for __getattribute__ that you can delegate to __getattr__ by raising AttributeError, so if someone does that, presumably it's intentional. It's not like the case with descriptors, where you have to think through the interaction of multiple features to figure out that raising an AttributeError will call __getattr__, and therefore many such uses are probably bugs. >> >> Sent from my iPhone >> >>> On Nov 27, 2015, at 09:49, ?? wrote: >>> >>> Hello everyone: >>> >>> I'm suggesting a modification to the AttributeError/__getattr__ mechanism, see issue25634: >>> http://bugs.python.org/issue25634 >>> >>> I used __getattr__ sometimes, and descriptor especially property is so widely used. I wonder whether someone had encountered the same problem with me. >>> >>> However, this is a complicated problem including performance issues, and backward compatibility. >>> >>> Thanks for your attention, >>> Jun Wang >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wjun77 at gmail.com Fri Nov 27 18:23:13 2015 From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=) Date: Sat, 28 Nov 2015 07:23:13 +0800 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: References: Message-ID: > This would create a boundary, same as PEP 479 does for StopIteration, > across which AttributeError becomes RuntimeError. Although my problem is about property as I don't use descriptor here, I think descriptor other than property should also be considered, such as classmethod/staticmethod, and normal descriptor. Raising "RuntimeError: descriptor raised AttributeError" is acceptable for me, but maybe some c code need modification, I don't know. > Do you have any examples that actually do demonstrate the problem to be solved? So you want more details about AttributeError in property? I thought property is widely used, and AttributeError occurs at all times. Maybe I've used property too heavy. In the window example, a simplified demonstration: class Window(): @property def backgroundImg(self): if self._backgroundImg is None: #need update, while the number of items changes self.set_backgroundImg() return self._backgroundImg def set_backgroundImg(self): self._backgroundImg = loadImg('white.bmp') for widget in self.widgets: widget.show(self._backgroundImg) Class Widget(): def show(self, img): img.draw(self.item.img, self.pos) However, widget.item may be None, while e.g. there are four widgets but only three items in total. In this case I should fill the area with white. But in this version of *show*, I just FORGET item can be None. So the traceback infomation: 'Window' object has no attribute 'backgroundImg'. In fact it takes a while before I find the cause is the AttributeError/__getattr__ mechanism. > But surely breaking that isn't the same as breaking code that's been explicitly stated to work, and used as sample code, for decades. I don't know this is such a severe problem. I used to think raising AttributeError in __getattribute__ to trigger __getattr__ is rare. > any solution that can fix descriptors without also "fixing" __getattribute__ is a lot better In practice I don't concern __getattribute__. But in my opinion it's better to 'fix' this in python4. > all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so This is a big problem, OK. > You didn't comment on the alternative I suggested; would it not satisfy your needs, or have some other problem that makes it unacceptable? I don't quite understand, you mean adding a subclass of *object* with the only difference of this behavior? 2015-11-28 5:37 GMT+08:00 Andrew Barnert : > On Nov 27, 2015, at 12:18, ?? wrote: > > I think the real world use case, as *property* is so common, is just when > __getattr__ needed. So anyone using __getattr__ in practice? > > > Your own examples seem like reasonable uses of __getattr__ to me. But they > don't demonstrate your problem, because you don't have any properties, > other custom descriptors, or __getattribute__. Do you have any examples > that actually do demonstrate the problem to be solved? > > > Is that really an imperfection? It says right there in the docs for > __getattribute__ that you can delegate to __getattr__ by raising > AttributeError > Just like raising *StopIteration* in generator, as described in PEP479. > > > But the docs didn't recommend raising StopIteration to exit a generator. > The docs implied it by omission, or at least we're ambiguous about what > would happen, and you could test it and see that it did, which meant some > people took advantage of it to do things like using a > StopIteration-throwing function in a comprehension filter clause (with > caveats about listcomp vs. genexpr and 2.x vs. 3.x and most readers having > to guess why it works). But surely breaking that isn't the same as breaking > code that's been explicitly stated to work, and used as sample code, for > decades. > > Although I'm not sure *StopIteration* has ever been documented as one way > to exit generator. And I admit that an unintentional AttributeError in > __getattribute__ is a rare case if ever. > > > That's my point: "fixing" __getattribute__ to eliminate a "rare case if > ever" bug, while also introducing a possibly less-rare backward > compatibility problem, seems like a terrible idea. Of course that isn't > your intention; it's just a side effect of trying to eliminate a more > common bug, with probably rarer intentional uses, in descriptors. But to > me, that implies that any solution that can fix descriptors without also > "fixing" __getattribute__ is a lot better. And, as you imply in the bug > report, this could be done by having either the descriptor mechanism > itself, or the object.__getattribute__ implementation, handle > AttributeError from descriptor lookup by reraising it as something else. > > But still, this is all a minor side issue about choosing between your > different variations. The big problem is that I think all of your solutions > make it too hard to trigger __getattr__ from a descriptor when you really > _do_ want to do so. Maybe you don't want to do so very often, but is it > really so rare that we can justify making it impossible? (Especially > considering the backward-compat issues for any code that's been doing that > for years...) > > You didn't comment on the alternative I suggested; would it not satisfy > your needs, or have some other problem that makes it unacceptable? > > 2015-11-28 2:34 GMT+08:00 Andrew Barnert : > >> It seems like you can get some of the benefits of this proposal without >> backward compat issues. >> >> Instead of changing things so AttributeError from __getattribute__ or a >> descriptor no longer calls __getattr__, just add a new subclass that >> doesn't, and change the Descriptor HOWTO to suggest using that subclass >> (with a bit of discussion that says you can use AttributeError if you want >> to trigger __getattr__, but usually you won't). >> >> That wouldn't fix any current code, but it also wouldn't break any code >> that intentionally uses the features as documented. And it would make it >> easy to write correct new code. >> >> One more thing: >> >> Without descriptor, unexpected AttributeError could only come from overriding __getattribute__, which is a rare case, although still an imperfection. >> >> Is that really an imperfection? It says right there in the docs for >> __getattribute__ that you can delegate to __getattr__ by raising >> AttributeError, so if someone does that, presumably it's intentional. It's >> not like the case with descriptors, where you have to think through the >> interaction of multiple features to figure out that raising an >> AttributeError will call __getattr__, and therefore many such uses are >> probably bugs. >> >> Sent from my iPhone >> >> On Nov 27, 2015, at 09:49, ?? wrote: >> >> Hello everyone: >> >> I'm suggesting a modification to the AttributeError/__getattr__ >> mechanism, see issue25634: >> http://bugs.python.org/issue25634 >> >> I used __getattr__ sometimes, and descriptor especially property is so >> widely used. I wonder whether someone had encountered the same problem with >> me. >> >> However, this is a complicated problem including performance issues, and >> backward compatibility. >> >> Thanks for your attention, >> Jun Wang >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Nov 27 18:27:57 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 28 Nov 2015 10:27:57 +1100 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: References: Message-ID: On Sat, Nov 28, 2015 at 10:23 AM, ?? wrote: > However, widget.item may be None, while e.g. there are four widgets but only > three items in total. In this case I should fill the area with white. But in > this version of show, I just FORGET item can be None. So the traceback > infomation: 'Window' object has no attribute 'backgroundImg'. In fact it > takes a while before I find the cause is the AttributeError/__getattr__ > mechanism. Hmm. A possibly more general solution is to declare that a function mustn't ever raise a certain exception. from functools import wraps def dontraise(exc): def wrapper(f): @wraps(f) def inner(*a,**kw): try: f(*a,**kw) except exc as e: raise RuntimeError from e return inner return wrapper class A(): def __init__(self, x=None): self.x = x @property @dontraise(AttributeError) def t(self): return self.x.t def __getattr__(self, name): return 'default' print(A().t) By guarding your function with dontraise(AttributeError), you declare that any AttributeError it raises must not leak out. Same as the property change, but not bound to the property class itself. ChrisA From steve at pearwood.info Fri Nov 27 22:46:09 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 28 Nov 2015 14:46:09 +1100 Subject: [Python-ideas] add a single __future__ for py3? In-Reply-To: <0540E031-580B-4ED7-9E9D-7E37C84959E4@mit.edu> References: <0540E031-580B-4ED7-9E9D-7E37C84959E4@mit.edu> Message-ID: <20151128034609.GU3821@ando.pearwood.info> On Fri, Nov 27, 2015 at 07:29:36PM +0000, Michael Scott Cuthbert wrote: > (an old list topic that I accidentally sent to the Google Groups list) I'm confused -- your subject line talks about a single __future__ for Python 3, but you don't talk about that in the body of the post. I'm not sure what the comment about Google Groups is supposed to mean. If you posted something there, most of us aren't going to see it, you'll need to re-post it here, not just tell us you posted it on Google Groups. As for your secondary point: > One smaller request is to remove 'barry_as_FLUFL' from > "__future__.all_feature_names". It's a great practical joke, but it > bit me hard on a project where I check that, for Py2/Py3 consistency, > each module imports all relevant __future__ names, and ensures that > doctests on Py2 are run with them on. I figured that I could iterate > through .all_feature_names and turn them all on except > unicode_literals and be safe, Hmmm, well, no, that's not really safe (as you've found out), particularly if you intend your code to be forward-compatible. Barry_as_FLUFL is a joke, but it demonstrates a real problem with your approach. Should some future version of Python add a new __future__ feature, and you run your code under that new version, you will unintentionally turn the feature on when your code is not expecting it, and likely break. To put it another way... suppose Guido had a change of heart about != and decided that <> really was better, and so Barry_as_FLUFL was spelled "proper_ne" and was intended as a real feature, not a joke. Your code would still have broken, but removing "proper_ne" from all_feature_names would certainly not be an option. The problem isn't that Barry_as_FLUFL is listed, but that you blindly applied all features without knowing what they are. Don't do that, it isn't safe. -- Steve From abarnert at yahoo.com Fri Nov 27 22:52:33 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 28 Nov 2015 03:52:33 +0000 (UTC) Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: References: Message-ID: <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com> On Friday, November 27, 2015 3:23 PM, ?? wrote: >> Do you have any examples that actually do demonstrate the problem to be solved? >So you want more details about AttributeError in property? No. I'm assuming Paul wanted an example that demonstrates the problem (a @property or other descriptor that raises or passes AttributeError, and a __getattr__ that blindly returns a value for anything). Just like your toy example on the bug tracker does, but realistic code rather than a toy example. What you've provided is an example that doesn't demonstrate the problem at all. The code below doesn't even have a __getattr__ in it, and it does exactly what you should expect it to do (assuming you fill in the missing bits in any reasonable way), so it can't possibly demonstrate why interactions with __getattr__ are a problem. > I thought property is widely used, and AttributeError occurs at all times. Maybe I've used property too heavy.>In the window example, a simplified demonstration: > >class Window(): > @property > def backgroundImg(self): > if self._backgroundImg is None: #need update, while the number of items changes > self.set_backgroundImg() > return self._backgroundImg > > def set_backgroundImg(self): > self._backgroundImg = loadImg('white.bmp') > for widget in self.widgets: > widget.show(self._backgroundImg) > >Class Widget(): > def show(self, img): > img.draw(self.item.img, self.pos) > >However, widget.item may be None, while e.g. there are four widgets but only three items in total. In this case I should fill the area with white. But in this version of show, I just FORGET item can be None. So the traceback infomation: 'Window' object has no attribute 'backgroundImg'. No, that can't possibly be your problem. If that were the case, the AttributeError will say that 'NoneType' object has no attribute 'img'. And the traceback would run from the Widget.show method, where the self.item.img is, back up the chain through your @property method. I'm guessing your actual problem is that you forgot to set self._backgroundImg = None somewhere (e.g., in the __init__ method). In that case, you would get an error that looks more like the one you're claiming to get (but the attribute mentioned is '_backgroundImg', not 'backgroundImg'), with only one level of traceback and everything. Or maybe there's a typo in your actual code, and you really don't have a 'backgroundImg' at all on Window objects; that would give exactly the error you're describing. No matter which case it is, the problem has nothing to do with @property or descriptors in general, or with __getattr__ (obviously, since there is no __getattr__ in the code), much less with the interaction between them, so any fix to that interaction couldn't possibly help this example. > In fact it takes a while before I find the cause is the AttributeError/__getattr__ mechanism. Since that isn't the cause, it would be bad if Python pointed you to look in that direction sooner... >> But surely breaking that isn't the same as breaking code that's been explicitly stated to work, and used as sample code, for decades. >I don't know this is such a severe problem. I used to think raising AttributeError in __getattribute__ to trigger __getattr__ is rare. > >> any solution that can fix descriptors without also "fixing" __getattribute__ is a lot better >In practice I don't concern __getattribute__. But in my opinion it's better to 'fix' this in python4. Why? If you think that erroneous uses are rare or nonexistent, while intentional uses are rare but not nonexistent, "fixing" it means breaking code gratuitously for no benefit. >> all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so >This is a big problem, OK. > >> You didn't comment on the alternative I suggested; would it not satisfy your needs, or have some other problem that makes it unacceptable? >I don't quite understand, you mean adding a subclass of object with the only difference of this behavior? No, adding a subclass of AttributeError, much like the one you mentioned in the bug report, but with the opposite meaning: the existing AttributeError continues to trigger __getattr__, but the new subclass doesn't. This makes it trivial to write new code that doesn't accidentally trigger __getattr__, without breaking old code (or rare new code) that wants to trigger __getattr__. The code currently does something like this pseudocode: try: val = obj.__getattribute__(name) except AttributeError: __getattr__ = getattr(type(obj), '__getattr__', None) if __getattr__: return __getattr__(name) I'm cheating a bit, but you get the idea. The problem is that we have no idea whether __getattribute__ failed to find anything (in which case we definitely want __getattr__ called), or found a descriptor whose __get__ raised an AttributeError (in which case we may not--e.g., a write-only attribute should not all through to __getattr__). My suggestion is to change it like this: try: val = obj.__getattribute__(name) except AttributeDynamicError: raise except AttributeError: __getattr__ = getattr(type(obj), '__getattr__', None) if __getattr__: return __getattr__(name) Now, if __getattribute__ found a descriptor whose __get__ raised an AttributeDynamicError, that passes on to the user code. (And, since it's a subclass of AttributeError, the user code should have no problem handling it.) And the Descriptor HOWTO will be changed to suggest raising AttributeDynamicError, except when you explicitly want it to call __getattr__, which you usually don't. And examples like simulating a write-only attribute will raise AttributeDynamicError. And maybe @property will automatically convert any AttributeError to AttributeDynamicError (not sure about that part). So, new code can easily be written to act the way you want, but existing code using descriptors that intentionally raise or pass an AttributeError continues to work the same way it always has, and new code that does the same can also be written easily. Obviously, the downside of any backward-compat-friendly change is that someone who has old code with a hidden bug they didn't know about will still have that same bug in Python 3.7; they have to change their code to take advantage of the fix. But I don't think that's a serious problem. (Especially if we decide @property is buggy and should be changed--most people who are writing actual custom descriptors, not just using the ones in the stdlib, probably understand this stuff.) From abarnert at yahoo.com Fri Nov 27 23:03:25 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 28 Nov 2015 04:03:25 +0000 (UTC) Subject: [Python-ideas] add a single __future__ for py3? In-Reply-To: <20151128034609.GU3821@ando.pearwood.info> References: <20151128034609.GU3821@ando.pearwood.info> Message-ID: <992758577.9555647.1448683405221.JavaMail.yahoo@mail.yahoo.com> On Friday, November 27, 2015 7:46 PM, Steven D'Aprano wrote: > > On Fri, Nov 27, 2015 at 07:29:36PM +0000, Michael Scott Cuthbert wrote: > >> (an old list topic that I accidentally sent to the Google Groups list) > > I'm confused -- your subject line talks about a single __future__ for > Python 3, but you don't talk about that in the body of the post. I'm not > > sure what the comment about Google Groups is supposed to mean. If you > posted something there, most of us aren't going to see it, you'll need > to re-post it here, not just tell us you posted it on Google Groups. > > > As for your secondary point: > >> One smaller request is to remove 'barry_as_FLUFL' from >> "__future__.all_feature_names". It's a great practical joke, > but it >> bit me hard on a project where I check that, for Py2/Py3 consistency, >> each module imports all relevant __future__ names, and ensures that >> doctests on Py2 are run with them on. I figured that I could iterate >> through .all_feature_names and turn them all on except >> unicode_literals and be safe, > > Hmmm, well, no, that's not really safe (as you've found out), > particularly if you intend your code to be forward-compatible. > Barry_as_FLUFL is a joke, but it demonstrates a real problem with your > approach. Should some future version of Python add a new __future__ > feature, and you run your code under that new version, you will > unintentionally turn the feature on when your code is not expecting it, > and likely break. > > To put it another way... suppose Guido had a change of heart about != > and decided that <> really was better, and so Barry_as_FLUFL was spelled > "proper_ne" and was intended as a real feature, not a joke. Your code > would still have broken, but removing "proper_ne" from > all_feature_names > would certainly not be an option. > > The problem isn't that Barry_as_FLUFL is listed, but that you blindly > applied all features without knowing what they are. Don't do that, it > isn't safe. Assuming he's doing this in a test suite, rather than in real code, I think it is safe, and maybe even a good idea. His test suite passes in 3.6, but in 3.7, it fails because some of his code doesn't work correctly with "from futures import thing_nobody_expects". That implies that his code may fail in 3.8, so he'd better take a look at thing_nobody_expects. And it's good that he finds this out now, instead of 18 months from now when his first users upgrade to 3.8 and his code breaks. That being said, whatever test system he uses had better let him mark a future as disabled or known-bad or whatever. Otherwise, he has to put off any commits at all until he fixes the thing_nobody_expects stuff. And realistically, he probably has a lot of higher-priority bugs to take care of in the intervening 18 months. And if he can do that temporarily for thing_nobody_expects, I don't see why he can't do that permanently for Barry_as_FLUFL. So, I don't think Python needs to change; his test tool does. (In fact, Barry_as_FLUFL serves as a good test for test tools' future testing.:)) Also, the fact that he's already got code to "turn them all on except unicode_literals" implies that he doesn't really have a problem in the first place; he just needs to change that code to turn them all on except unicode_literals and Barry_as_FLUFL. From wjun77 at gmail.com Fri Nov 27 23:25:44 2015 From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=) Date: Sat, 28 Nov 2015 12:25:44 +0800 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com> References: <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com> Message-ID: I don't have realistic code that blindly returns a value now, but showing unhelpful and confusing traceback messages. > The code below doesn't even have a __getattr__ in it >> In *the* window example I've already post the __getattr__-related code when discussing the use case of __getattr__, so I omitted those code here. Sorry for the misleading code. Here's the complete version: class ActiveState(State): @staticmethod def rightClick(self): print('right clicked') class InactiveState(State): @staticmethod def rightClick(self): pass class Window(): def __init__(self): self.__state = ActiveState def __getattr__(self, name): try: return partial(getattr(self.__state, name), self) except AttributeError: raise AttributeError("'{}' object has no attribute '{}'".format(self.__class__.__name__, name)) from None @property def backgroundImg(self): if self._backgroundImg is None: #need update, while the number of items changes self.set_backgroundImg() return self._backgroundImg def set_backgroundImg(self): self._backgroundImg = loadImg('white.bmp') for widget in self.widgets: widget.show(self._backgroundImg) Class Widget(): def show(self, img): img.draw(self.item.img, self.pos) > Why? If you think that erroneous uses are rare or nonexistent, while intentional uses are rare but not nonexistent, "fixing" it means breaking code gratuitously for no benefit. I mean if we don't consider any backward compatibility, maybe when creating a new language other than python, I think it's a better choice to 'fix' it. Only my personal opinion. I have something urgent to do now, so I'll read the rest part of your post carefully later. Anyway thanks for your attention. 2015-11-28 11:52 GMT+08:00 Andrew Barnert : > On Friday, November 27, 2015 3:23 PM, ?? wrote: > > >> Do you have any examples that actually do demonstrate the problem to be > solved? > > >So you want more details about AttributeError in property? > > No. I'm assuming Paul wanted an example that demonstrates the problem (a > @property or other descriptor that raises or passes AttributeError, and a > __getattr__ that blindly returns a value for anything). Just like your toy > example on the bug tracker does, but realistic code rather than a toy > example. > > What you've provided is an example that doesn't demonstrate the problem at > all. The code below doesn't even have a __getattr__ in it, and it does > exactly what you should expect it to do (assuming you fill in the missing > bits in any reasonable way), so it can't possibly demonstrate why > interactions with __getattr__ are a problem. > > > I thought property is widely used, and AttributeError occurs at all > times. Maybe I've used property too heavy.>In the window example, a > simplified demonstration: > > > >class Window(): > > @property > > def backgroundImg(self): > > if self._backgroundImg is None: #need update, while the number of > items changes > > self.set_backgroundImg() > > return self._backgroundImg > > > > def set_backgroundImg(self): > > self._backgroundImg = loadImg('white.bmp') > > for widget in self.widgets: > > widget.show(self._backgroundImg) > > > >Class Widget(): > > def show(self, img): > > img.draw(self.item.img, self.pos) > > > > >However, widget.item may be None, while e.g. there are four widgets but > only three items in total. In this case I should fill the area with white. > But in this version of show, I just FORGET item can be None. So the > traceback infomation: 'Window' object has no attribute 'backgroundImg'. > > No, that can't possibly be your problem. If that were the case, the > AttributeError will say that 'NoneType' object has no attribute 'img'. And > the traceback would run from the Widget.show method, where the > self.item.img is, back up the chain through your @property method. > > I'm guessing your actual problem is that you forgot to set > self._backgroundImg = None somewhere (e.g., in the __init__ method). In > that case, you would get an error that looks more like the one you're > claiming to get (but the attribute mentioned is '_backgroundImg', not > 'backgroundImg'), with only one level of traceback and everything. > > > Or maybe there's a typo in your actual code, and you really don't have a > 'backgroundImg' at all on Window objects; that would give exactly the error > you're describing. > > No matter which case it is, the problem has nothing to do with @property > or descriptors in general, or with __getattr__ (obviously, since there is > no __getattr__ in the code), much less with the interaction between them, > so any fix to that interaction couldn't possibly help this example. > > > > In fact it takes a while before I find the cause is the > AttributeError/__getattr__ mechanism. > > > Since that isn't the cause, it would be bad if Python pointed you to look > in that direction sooner... > >> But surely breaking that isn't the same as breaking code that's been > explicitly stated to work, and used as sample code, for decades. > >I don't know this is such a severe problem. I used to think raising > AttributeError in __getattribute__ to trigger __getattr__ is rare. > > > >> any solution that can fix descriptors without also "fixing" > __getattribute__ is a lot better > > >In practice I don't concern __getattribute__. But in my opinion it's > better to 'fix' this in python4. > > Why? If you think that erroneous uses are rare or nonexistent, while > intentional uses are rare but not nonexistent, "fixing" it means breaking > code gratuitously for no benefit. > >> all of your solutions make it too hard to trigger __getattr__ from a > descriptor when you really _do_ want to do so > >This is a big problem, OK. > > > >> You didn't comment on the alternative I suggested; would it not satisfy > your needs, or have some other problem that makes it unacceptable? > >I don't quite understand, you mean adding a subclass of object with the > only difference of this behavior? > > > No, adding a subclass of AttributeError, much like the one you mentioned > in the bug report, but with the opposite meaning: the existing > AttributeError continues to trigger __getattr__, but the new subclass > doesn't. This makes it trivial to write new code that doesn't accidentally > trigger __getattr__, without breaking old code (or rare new code) that > wants to trigger __getattr__. > > The code currently does something like this pseudocode: > > try: > val = obj.__getattribute__(name) > except AttributeError: > __getattr__ = getattr(type(obj), '__getattr__', None) > if __getattr__: return __getattr__(name) > > I'm cheating a bit, but you get the idea. The problem is that we have no > idea whether __getattribute__ failed to find anything (in which case we > definitely want __getattr__ called), or found a descriptor whose __get__ > raised an AttributeError (in which case we may not--e.g., a write-only > attribute should not all through to __getattr__). > > My suggestion is to change it like this: > > > try: > val = obj.__getattribute__(name) > except AttributeDynamicError: > raise > except AttributeError: > __getattr__ = getattr(type(obj), '__getattr__', None) > if __getattr__: return __getattr__(name) > > > Now, if __getattribute__ found a descriptor whose __get__ raised an > AttributeDynamicError, that passes on to the user code. (And, since it's a > subclass of AttributeError, the user code should have no problem handling > it.) And the Descriptor HOWTO will be changed to suggest raising > AttributeDynamicError, except when you explicitly want it to call > __getattr__, which you usually don't. And examples like simulating a > write-only attribute will raise AttributeDynamicError. And maybe @property > will automatically convert any AttributeError to AttributeDynamicError (not > sure about that part). > > So, new code can easily be written to act the way you want, but existing > code using descriptors that intentionally raise or pass an AttributeError > continues to work the same way it always has, and new code that does the > same can also be written easily. > > Obviously, the downside of any backward-compat-friendly change is that > someone who has old code with a hidden bug they didn't know about will > still have that same bug in Python 3.7; they have to change their code to > take advantage of the fix. But I don't think that's a serious problem. > (Especially if we decide @property is buggy and should be changed--most > people who are writing actual custom descriptors, not just using the ones > in the stdlib, probably understand this stuff.) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Nov 28 01:13:12 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 27 Nov 2015 22:13:12 -0800 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: References: <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com> Message-ID: <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com> On Nov 27, 2015, at 20:25, ?? wrote: > > I don't have realistic code that blindly returns a value now, but showing unhelpful and confusing traceback messages. Well, yes. If you catch an AttributeError and raise a different one that hides all the relevant information, it will be unhelpful and confusing. But just don't do that, and you don't have that problem. (That being said, the way you've written it, I think the new AttributeError will contain the old one, so you should still be able to debug it--unless you're using Python 2, but in that case, obviously you need to migrating to Python 3... But anyway, it would be simpler to just not write code that makes debugging harder and provides no benefits.) As a side note: why are you writing @staticmethods that take a self parameter? The whole point of static methods is that they don't get passed self. If you need to pass in the "owner", calling it "self" is misleading; give it a name that makes it clear what's being passed. But it looks like you don't even need that, since you never use it. Is this code by chance an attempt to directly port some Java code to Python? > > The code below doesn't even have a __getattr__ in it > >> In the window example > I've already post the __getattr__-related code when discussing the use case of __getattr__, so I omitted those code here. > Sorry for the misleading code. Here's the complete version: > > class ActiveState(State): > @staticmethod > def rightClick(self): > print('right clicked') > class InactiveState(State): > @staticmethod > def rightClick(self): > pass > > class Window(): > def __init__(self): > self.__state = ActiveState > def __getattr__(self, name): > try: > return partial(getattr(self.__state, name), self) > except AttributeError: > raise AttributeError("'{}' object has no attribute '{}'".format(self.__class__.__name__, name)) from None > > @property > def backgroundImg(self): > if self._backgroundImg is None: #need update, while the number of items changes > self.set_backgroundImg() > return self._backgroundImg > > def set_backgroundImg(self): > self._backgroundImg = loadImg('white.bmp') > for widget in self.widgets: > widget.show(self._backgroundImg) > > Class Widget(): > def show(self, img): > img.draw(self.item.img, self.pos) > > > > Why? If you think that erroneous uses are rare or nonexistent, while intentional uses are rare but not nonexistent, "fixing" it means breaking code gratuitously for no benefit. > I mean if we don't consider any backward compatibility, maybe when creating a new language other than python, I think it's a better choice to 'fix' it. Only my personal opinion. > > I have something urgent to do now, so I'll read the rest part of your post carefully later. Anyway thanks for your attention. > > 2015-11-28 11:52 GMT+08:00 Andrew Barnert : >> On Friday, November 27, 2015 3:23 PM, ?? wrote: >> >> >> Do you have any examples that actually do demonstrate the problem to be solved? >> >> >So you want more details about AttributeError in property? >> >> No. I'm assuming Paul wanted an example that demonstrates the problem (a @property or other descriptor that raises or passes AttributeError, and a __getattr__ that blindly returns a value for anything). Just like your toy example on the bug tracker does, but realistic code rather than a toy example. >> >> What you've provided is an example that doesn't demonstrate the problem at all. The code below doesn't even have a __getattr__ in it, and it does exactly what you should expect it to do (assuming you fill in the missing bits in any reasonable way), so it can't possibly demonstrate why interactions with __getattr__ are a problem. >> >> > I thought property is widely used, and AttributeError occurs at all times. Maybe I've used property too heavy.>In the window example, a simplified demonstration: >> > >> >class Window(): >> > @property >> > def backgroundImg(self): >> > if self._backgroundImg is None: #need update, while the number of items changes >> > self.set_backgroundImg() >> > return self._backgroundImg >> > >> > def set_backgroundImg(self): >> > self._backgroundImg = loadImg('white.bmp') >> > for widget in self.widgets: >> > widget.show(self._backgroundImg) >> > >> >Class Widget(): >> > def show(self, img): >> > img.draw(self.item.img, self.pos) >> > >> >> >However, widget.item may be None, while e.g. there are four widgets but only three items in total. In this case I should fill the area with white. But in this version of show, I just FORGET item can be None. So the traceback infomation: 'Window' object has no attribute 'backgroundImg'. >> >> No, that can't possibly be your problem. If that were the case, the AttributeError will say that 'NoneType' object has no attribute 'img'. And the traceback would run from the Widget.show method, where the self.item.img is, back up the chain through your @property method. >> >> I'm guessing your actual problem is that you forgot to set self._backgroundImg = None somewhere (e.g., in the __init__ method). In that case, you would get an error that looks more like the one you're claiming to get (but the attribute mentioned is '_backgroundImg', not 'backgroundImg'), with only one level of traceback and everything. >> >> >> Or maybe there's a typo in your actual code, and you really don't have a 'backgroundImg' at all on Window objects; that would give exactly the error you're describing. >> >> No matter which case it is, the problem has nothing to do with @property or descriptors in general, or with __getattr__ (obviously, since there is no __getattr__ in the code), much less with the interaction between them, so any fix to that interaction couldn't possibly help this example. >> >> >> > In fact it takes a while before I find the cause is the AttributeError/__getattr__ mechanism. >> >> >> Since that isn't the cause, it would be bad if Python pointed you to look in that direction sooner... >> >> But surely breaking that isn't the same as breaking code that's been explicitly stated to work, and used as sample code, for decades. >> >I don't know this is such a severe problem. I used to think raising AttributeError in __getattribute__ to trigger __getattr__ is rare. >> > >> >> any solution that can fix descriptors without also "fixing" __getattribute__ is a lot better >> >> >In practice I don't concern __getattribute__. But in my opinion it's better to 'fix' this in python4. >> >> Why? If you think that erroneous uses are rare or nonexistent, while intentional uses are rare but not nonexistent, "fixing" it means breaking code gratuitously for no benefit. >> >> all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so >> >This is a big problem, OK. >> > >> >> You didn't comment on the alternative I suggested; would it not satisfy your needs, or have some other problem that makes it unacceptable? >> >I don't quite understand, you mean adding a subclass of object with the only difference of this behavior? >> >> >> No, adding a subclass of AttributeError, much like the one you mentioned in the bug report, but with the opposite meaning: the existing AttributeError continues to trigger __getattr__, but the new subclass doesn't. This makes it trivial to write new code that doesn't accidentally trigger __getattr__, without breaking old code (or rare new code) that wants to trigger __getattr__. >> >> The code currently does something like this pseudocode: >> >> try: >> val = obj.__getattribute__(name) >> except AttributeError: >> __getattr__ = getattr(type(obj), '__getattr__', None) >> if __getattr__: return __getattr__(name) >> >> I'm cheating a bit, but you get the idea. The problem is that we have no idea whether __getattribute__ failed to find anything (in which case we definitely want __getattr__ called), or found a descriptor whose __get__ raised an AttributeError (in which case we may not--e.g., a write-only attribute should not all through to __getattr__). >> >> My suggestion is to change it like this: >> >> >> try: >> val = obj.__getattribute__(name) >> except AttributeDynamicError: >> raise >> except AttributeError: >> __getattr__ = getattr(type(obj), '__getattr__', None) >> if __getattr__: return __getattr__(name) >> >> >> Now, if __getattribute__ found a descriptor whose __get__ raised an AttributeDynamicError, that passes on to the user code. (And, since it's a subclass of AttributeError, the user code should have no problem handling it.) And the Descriptor HOWTO will be changed to suggest raising AttributeDynamicError, except when you explicitly want it to call __getattr__, which you usually don't. And examples like simulating a write-only attribute will raise AttributeDynamicError. And maybe @property will automatically convert any AttributeError to AttributeDynamicError (not sure about that part). >> >> So, new code can easily be written to act the way you want, but existing code using descriptors that intentionally raise or pass an AttributeError continues to work the same way it always has, and new code that does the same can also be written easily. >> >> Obviously, the downside of any backward-compat-friendly change is that someone who has old code with a hidden bug they didn't know about will still have that same bug in Python 3.7; they have to change their code to take advantage of the fix. But I don't think that's a serious problem. (Especially if we decide @property is buggy and should be changed--most people who are writing actual custom descriptors, not just using the ones in the stdlib, probably understand this stuff.) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wjun77 at gmail.com Sun Nov 29 09:31:06 2015 From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=) Date: Sun, 29 Nov 2015 22:31:06 +0800 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com> References: <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com> <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com> Message-ID: > If you catch an AttributeError and raise a different one that hides all the relevant information, it will be unhelpful and confusing. What matters is whether __getattr__ which hides all the relevant information of any AttributeError is defined, not a default value returned or any other behavior in __getattr__. > all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so If we do want to trigger __getattr__ from a descriptor, the AttributeMissError solution seems feasible. Raising AttributeMissError in descriptor triggers __getattr__. (By the way, I used to think controlling the program flow by Exception is a bad idea, and now I understand that it do be practical sometimes. But I insist that AttributeError is too general to be used here.) > And maybe @property will automatically convert any AttributeError to AttributeDynamicError (not sure about that part). Then raising AttributeError in property intentionally 'when you really _do_ want to do so' to trigger __getattr__ will fail, right? @property (or descriptor) converting AttributeError to AttributeDynamicError, or __getattr__ converting AttributeMissError to AttributeError. The former has the advantage of keeping __getattr__ triggered by raising AttributeError in __getattribute__ as documented for decades. But I don't like this idea, because it's conceptually ugly. If you don't agree or understand, my reply can only be 'we have different aesthetic'. I personally prefer no change to this. > why are you writing @staticmethods that take a self parameter? 'self' is more familiar to type, and I can change the class of the method between the Window class and the ActiveState class without modifying the name between 'self' and 'owner'. In real code there's no need to write @staticmethod explicitly; the metaclass of State will automatically change any normal method to staticmethod. And ActiveState is defined in Window. Hmm, it seems that no one feels necessary to make change other than myself. I guess it's because no one uses __getattr__ in practice at all. Anyway, now that I know there is a 'pit' there, it won't bother me too much in the future, either by using the *dontraise* decorator by Chris or any other means. 2015-11-28 14:13 GMT+08:00 Andrew Barnert : > On Nov 27, 2015, at 20:25, ?? wrote: > > I don't have realistic code that blindly returns a value now, but showing > unhelpful and confusing traceback messages. > > > Well, yes. If you catch an AttributeError and raise a different one that > hides all the relevant information, it will be unhelpful and confusing. But > just don't do that, and you don't have that problem. > > (That being said, the way you've written it, I think the new > AttributeError will contain the old one, so you should still be able to > debug it--unless you're using Python 2, but in that case, obviously you > need to migrating to Python 3... But anyway, it would be simpler to just > not write code that makes debugging harder and provides no benefits.) > > As a side note: why are you writing @staticmethods that take a self > parameter? The whole point of static methods is that they don't get passed > self. If you need to pass in the "owner", calling it "self" is misleading; > give it a name that makes it clear what's being passed. But it looks like > you don't even need that, since you never use it. Is this code by chance an > attempt to directly port some Java code to Python? > > > The code below doesn't even have a __getattr__ in it > >> In *the* window example > I've already post the __getattr__-related code when discussing the use > case of __getattr__, so I omitted those code here. > > Sorry for the misleading code. Here's the complete version: > > class ActiveState(State): > @staticmethod > def rightClick(self): > print('right clicked') > class InactiveState(State): > @staticmethod > def rightClick(self): > pass > > class Window(): > def __init__(self): > self.__state = ActiveState > def __getattr__(self, name): > try: > return partial(getattr(self.__state, name), self) > except AttributeError: > raise AttributeError("'{}' object has no attribute > '{}'".format(self.__class__.__name__, name)) from None > > @property > def backgroundImg(self): > if self._backgroundImg is None: #need update, while the number of > items changes > self.set_backgroundImg() > return self._backgroundImg > > def set_backgroundImg(self): > self._backgroundImg = loadImg('white.bmp') > for widget in self.widgets: > widget.show(self._backgroundImg) > > Class Widget(): > def show(self, img): > img.draw(self.item.img, self.pos) > > > > Why? If you think that erroneous uses are rare or nonexistent, while > intentional uses are rare but not nonexistent, "fixing" it means breaking > code gratuitously for no benefit. > I mean if we don't consider any backward compatibility, maybe when > creating a new language other than python, I think it's a better choice to > 'fix' it. Only my personal opinion. > > I have something urgent to do now, so I'll read the rest part of your post > carefully later. Anyway thanks for your attention. > > 2015-11-28 11:52 GMT+08:00 Andrew Barnert : > >> On Friday, November 27, 2015 3:23 PM, ?? wrote: >> >> >> Do you have any examples that actually do demonstrate the problem to >> be solved? >> >> >So you want more details about AttributeError in property? >> >> No. I'm assuming Paul wanted an example that demonstrates the problem (a >> @property or other descriptor that raises or passes AttributeError, and a >> __getattr__ that blindly returns a value for anything). Just like your toy >> example on the bug tracker does, but realistic code rather than a toy >> example. >> >> What you've provided is an example that doesn't demonstrate the problem >> at all. The code below doesn't even have a __getattr__ in it, and it does >> exactly what you should expect it to do (assuming you fill in the missing >> bits in any reasonable way), so it can't possibly demonstrate why >> interactions with __getattr__ are a problem. >> >> > I thought property is widely used, and AttributeError occurs at all >> times. Maybe I've used property too heavy.>In the window example, a >> simplified demonstration: >> > >> >class Window(): >> > @property >> > def backgroundImg(self): >> > if self._backgroundImg is None: #need update, while the number >> of items changes >> > self.set_backgroundImg() >> > return self._backgroundImg >> > >> > def set_backgroundImg(self): >> > self._backgroundImg = loadImg('white.bmp') >> > for widget in self.widgets: >> > widget.show(self._backgroundImg) >> > >> >Class Widget(): >> > def show(self, img): >> > img.draw(self.item.img, self.pos) >> > >> >> >However, widget.item may be None, while e.g. there are four widgets but >> only three items in total. In this case I should fill the area with white. >> But in this version of show, I just FORGET item can be None. So the >> traceback infomation: 'Window' object has no attribute 'backgroundImg'. >> >> No, that can't possibly be your problem. If that were the case, the >> AttributeError will say that 'NoneType' object has no attribute 'img'. And >> the traceback would run from the Widget.show method, where the >> self.item.img is, back up the chain through your @property method. >> >> I'm guessing your actual problem is that you forgot to set >> self._backgroundImg = None somewhere (e.g., in the __init__ method). In >> that case, you would get an error that looks more like the one you're >> claiming to get (but the attribute mentioned is '_backgroundImg', not >> 'backgroundImg'), with only one level of traceback and everything. >> >> >> Or maybe there's a typo in your actual code, and you really don't have a >> 'backgroundImg' at all on Window objects; that would give exactly the error >> you're describing. >> >> No matter which case it is, the problem has nothing to do with @property >> or descriptors in general, or with __getattr__ (obviously, since there is >> no __getattr__ in the code), much less with the interaction between them, >> so any fix to that interaction couldn't possibly help this example. >> >> >> > In fact it takes a while before I find the cause is the >> AttributeError/__getattr__ mechanism. >> >> >> Since that isn't the cause, it would be bad if Python pointed you to look >> in that direction sooner... >> >> But surely breaking that isn't the same as breaking code that's been >> explicitly stated to work, and used as sample code, for decades. >> >I don't know this is such a severe problem. I used to think raising >> AttributeError in __getattribute__ to trigger __getattr__ is rare. >> > >> >> any solution that can fix descriptors without also "fixing" >> __getattribute__ is a lot better >> >> >In practice I don't concern __getattribute__. But in my opinion it's >> better to 'fix' this in python4. >> >> Why? If you think that erroneous uses are rare or nonexistent, while >> intentional uses are rare but not nonexistent, "fixing" it means breaking >> code gratuitously for no benefit. >> >> all of your solutions make it too hard to trigger __getattr__ from a >> descriptor when you really _do_ want to do so >> >This is a big problem, OK. >> > >> >> You didn't comment on the alternative I suggested; would it not >> satisfy your needs, or have some other problem that makes it unacceptable? >> >I don't quite understand, you mean adding a subclass of object with the >> only difference of this behavior? >> >> >> No, adding a subclass of AttributeError, much like the one you mentioned >> in the bug report, but with the opposite meaning: the existing >> AttributeError continues to trigger __getattr__, but the new subclass >> doesn't. This makes it trivial to write new code that doesn't accidentally >> trigger __getattr__, without breaking old code (or rare new code) that >> wants to trigger __getattr__. >> >> The code currently does something like this pseudocode: >> >> try: >> val = obj.__getattribute__(name) >> except AttributeError: >> __getattr__ = getattr(type(obj), '__getattr__', None) >> if __getattr__: return __getattr__(name) >> >> I'm cheating a bit, but you get the idea. The problem is that we have no >> idea whether __getattribute__ failed to find anything (in which case we >> definitely want __getattr__ called), or found a descriptor whose __get__ >> raised an AttributeError (in which case we may not--e.g., a write-only >> attribute should not all through to __getattr__). >> >> My suggestion is to change it like this: >> >> >> try: >> val = obj.__getattribute__(name) >> except AttributeDynamicError: >> raise >> except AttributeError: >> __getattr__ = getattr(type(obj), '__getattr__', None) >> if __getattr__: return __getattr__(name) >> >> >> Now, if __getattribute__ found a descriptor whose __get__ raised an >> AttributeDynamicError, that passes on to the user code. (And, since it's a >> subclass of AttributeError, the user code should have no problem handling >> it.) And the Descriptor HOWTO will be changed to suggest raising >> AttributeDynamicError, except when you explicitly want it to call >> __getattr__, which you usually don't. And examples like simulating a >> write-only attribute will raise AttributeDynamicError. And maybe @property >> will automatically convert any AttributeError to AttributeDynamicError (not >> sure about that part). >> >> So, new code can easily be written to act the way you want, but existing >> code using descriptors that intentionally raise or pass an AttributeError >> continues to work the same way it always has, and new code that does the >> same can also be written easily. >> >> Obviously, the downside of any backward-compat-friendly change is that >> someone who has old code with a hidden bug they didn't know about will >> still have that same bug in Python 3.7; they have to change their code to >> take advantage of the fix. But I don't think that's a serious problem. >> (Especially if we decide @property is buggy and should be changed--most >> people who are writing actual custom descriptors, not just using the ones >> in the stdlib, probably understand this stuff.) >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sun Nov 29 11:07:22 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 29 Nov 2015 08:07:22 -0800 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: References: <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com> <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com> Message-ID: <84C3BBEA-3D17-4B50-9571-6405EABE03F3@yahoo.com> On Nov 29, 2015, at 06:31, ?? wrote: > > > If you catch an AttributeError and raise a different one that hides all the relevant information, it will be unhelpful and confusing. > What matters is whether __getattr__ which hides all the relevant information of any AttributeError is defined, not a default value returned or any other behavior in __getattr__. Your problem is that it took you hours to hunt down where the AttributeError came from, because it gave useless information instead of useful information. The only reason that's happening is that you're handling any AttributeError by swallowing it and raising a new one with useless information. Just take that out, and your hours of debugging go away. > > all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so > If we do want to trigger __getattr__ from a descriptor, the AttributeMissError solution seems feasible. Raising AttributeMissError in descriptor triggers __getattr__. The difference, as I've already explained, is that your solution breaks backward compatibility, requiring anyone using such code to change it, while my variation leaves working code alone, only requiring new code to be written differently. All else being equal, the latter is obvious less disruptive a change and therefore better. So, if you think your more-disruptive solution is better, that must mean all else is not equal. But you have to explain how. What's wrong with my version? > (By the way, I used to think controlling the program flow by Exception is a bad idea, and now I understand that it do be practical sometimes. But I insist that AttributeError is too general to be used here.) All the places exceptions are used for flow control in Python and its stdlib and pythonic code in general use general exceptions. Occasionally (as with generators raising StopIteration where they shouldn't be), this can lead to specific trouble, and that occasional specific trouble gets fixed. Maybe this is one of those cases. That would be two times in as many decades. That's not a good reason to change the general principle. > > And maybe @property will automatically convert any AttributeError to AttributeDynamicError (not sure about that part). > Then raising AttributeError in property intentionally 'when you really _do_ want to do so' to trigger __getattr__ will fail, right? The point is that @property is a high-level, simplified way to write a specific kind of descriptor. It may be perfectly reasonable that properties should never trigger __getattr__; after all, you can always fall back to writing your own descriptor (or even your own property-like descriptor factory--it's only a few lines of code) if you want to. (Again, I'm not sure one way or the other about this.) > @property (or descriptor) converting Nobody is suggestion descriptors do anything different here. The code would be explicit code in the implementation of @property (and in the Descriptor HOWTO section that shows how @property works). > AttributeError to AttributeDynamicError, or __getattr__ converting AttributeMissError to AttributeError. The former has the advantage of keeping __getattr__ triggered by raising AttributeError in __getattribute__ as documented for decades. But I don't like this idea, because it's conceptually ugly. If you don't agree or understand, my reply can only be 'we have different aesthetic'. I personally prefer no change to this. > > > why are you writing @staticmethods that take a self parameter? > 'self' is more familiar to type, and I can change the class of the method between the Window class and the ActiveState class without modifying the name between 'self' and 'owner'. In real code there's no need to write @staticmethod explicitly; the metaclass of State will automatically change any normal method to staticmethod. And ActiveState is defined in Window. "More familiar to type" is a bad reason to use something where it has the wrong meaning. You wouldn't try to write exponentiation as "*" because a single asterisk is more familiar than a double, so why would you use "self" instead of the right parameter name? The fact that in this case it only misleads human readers, instead of also misleading the interpreter, doesn't make it any less of a problem. (And having a metaclass that automatically changes methods to @staticmethod doesn't make a difference. Do you call the first parameter of a __new__ method "self" because you don't have to decorate it?) > Hmm, it seems that no one feels necessary to make change other than myself. I guess it's because no one uses __getattr__ in practice at all. First, I'm sure lots of people use __getattr__ all over the place. (Personally, I consider the ability to write simple dynamic proxy classes easier than any language except Smalltalk to be a major selling point for Python, and it's been a deciding factor over ObjC in at least one project.) More importantly: you've had at least two people saying "I see the problem in principle, but I'd like to see real-life code where you're mixing properties that raise AttributeError and __getattr__", and another person trying to dig out details of your proposal and discuss alternatives. How do you interpret that as a lack of interest? If nobody else saw any point to your proposal, nobody would be responding at all. > Anyway, now that I know there is a 'pit' there, it won't bother me too much in the future, either by using the dontraise decorator by Chris or any other means. > > 2015-11-28 14:13 GMT+08:00 Andrew Barnert : >>> On Nov 27, 2015, at 20:25, ?? wrote: >>> >>> I don't have realistic code that blindly returns a value now, but showing unhelpful and confusing traceback messages. >> >> Well, yes. If you catch an AttributeError and raise a different one that hides all the relevant information, it will be unhelpful and confusing. But just don't do that, and you don't have that problem. >> >> (That being said, the way you've written it, I think the new AttributeError will contain the old one, so you should still be able to debug it--unless you're using Python 2, but in that case, obviously you need to migrating to Python 3... But anyway, it would be simpler to just not write code that makes debugging harder and provides no benefits.) >> >> As a side note: why are you writing @staticmethods that take a self parameter? The whole point of static methods is that they don't get passed self. If you need to pass in the "owner", calling it "self" is misleading; give it a name that makes it clear what's being passed. But it looks like you don't even need that, since you never use it. Is this code by chance an attempt to directly port some Java code to Python? >> >>> > The code below doesn't even have a __getattr__ in it >>> >> In the window example >>> I've already post the __getattr__-related code when discussing the use case of __getattr__, so I omitted those code here. >>> Sorry for the misleading code. Here's the complete version: >>> >>> class ActiveState(State): >>> @staticmethod >>> def rightClick(self): >>> print('right clicked') >>> class InactiveState(State): >>> @staticmethod >>> def rightClick(self): >>> pass >>> >>> class Window(): >>> def __init__(self): >>> self.__state = ActiveState >>> def __getattr__(self, name): >>> try: >>> return partial(getattr(self.__state, name), self) >>> except AttributeError: >>> raise AttributeError("'{}' object has no attribute '{}'".format(self.__class__.__name__, name)) from None >>> >>> @property >>> def backgroundImg(self): >>> if self._backgroundImg is None: #need update, while the number of items changes >>> self.set_backgroundImg() >>> return self._backgroundImg >>> >>> def set_backgroundImg(self): >>> self._backgroundImg = loadImg('white.bmp') >>> for widget in self.widgets: >>> widget.show(self._backgroundImg) >>> >>> Class Widget(): >>> def show(self, img): >>> img.draw(self.item.img, self.pos) >>> >>> >>> > Why? If you think that erroneous uses are rare or nonexistent, while intentional uses are rare but not nonexistent, "fixing" it means breaking code gratuitously for no benefit. >>> I mean if we don't consider any backward compatibility, maybe when creating a new language other than python, I think it's a better choice to 'fix' it. Only my personal opinion. >>> >>> I have something urgent to do now, so I'll read the rest part of your post carefully later. Anyway thanks for your attention. >>> >>> 2015-11-28 11:52 GMT+08:00 Andrew Barnert : >>>> On Friday, November 27, 2015 3:23 PM, ?? wrote: >>>> >>>> >> Do you have any examples that actually do demonstrate the problem to be solved? >>>> >>>> >So you want more details about AttributeError in property? >>>> >>>> No. I'm assuming Paul wanted an example that demonstrates the problem (a @property or other descriptor that raises or passes AttributeError, and a __getattr__ that blindly returns a value for anything). Just like your toy example on the bug tracker does, but realistic code rather than a toy example. >>>> >>>> What you've provided is an example that doesn't demonstrate the problem at all. The code below doesn't even have a __getattr__ in it, and it does exactly what you should expect it to do (assuming you fill in the missing bits in any reasonable way), so it can't possibly demonstrate why interactions with __getattr__ are a problem. >>>> >>>> > I thought property is widely used, and AttributeError occurs at all times. Maybe I've used property too heavy.>In the window example, a simplified demonstration: >>>> > >>>> >class Window(): >>>> > @property >>>> > def backgroundImg(self): >>>> > if self._backgroundImg is None: #need update, while the number of items changes >>>> > self.set_backgroundImg() >>>> > return self._backgroundImg >>>> > >>>> > def set_backgroundImg(self): >>>> > self._backgroundImg = loadImg('white.bmp') >>>> > for widget in self.widgets: >>>> > widget.show(self._backgroundImg) >>>> > >>>> >Class Widget(): >>>> > def show(self, img): >>>> > img.draw(self.item.img, self.pos) >>>> > >>>> >>>> >However, widget.item may be None, while e.g. there are four widgets but only three items in total. In this case I should fill the area with white. But in this version of show, I just FORGET item can be None. So the traceback infomation: 'Window' object has no attribute 'backgroundImg'. >>>> >>>> No, that can't possibly be your problem. If that were the case, the AttributeError will say that 'NoneType' object has no attribute 'img'. And the traceback would run from the Widget.show method, where the self.item.img is, back up the chain through your @property method. >>>> >>>> I'm guessing your actual problem is that you forgot to set self._backgroundImg = None somewhere (e.g., in the __init__ method). In that case, you would get an error that looks more like the one you're claiming to get (but the attribute mentioned is '_backgroundImg', not 'backgroundImg'), with only one level of traceback and everything. >>>> >>>> >>>> Or maybe there's a typo in your actual code, and you really don't have a 'backgroundImg' at all on Window objects; that would give exactly the error you're describing. >>>> >>>> No matter which case it is, the problem has nothing to do with @property or descriptors in general, or with __getattr__ (obviously, since there is no __getattr__ in the code), much less with the interaction between them, so any fix to that interaction couldn't possibly help this example. >>>> >>>> >>>> > In fact it takes a while before I find the cause is the AttributeError/__getattr__ mechanism. >>>> >>>> >>>> Since that isn't the cause, it would be bad if Python pointed you to look in that direction sooner... >>>> >> But surely breaking that isn't the same as breaking code that's been explicitly stated to work, and used as sample code, for decades. >>>> >I don't know this is such a severe problem. I used to think raising AttributeError in __getattribute__ to trigger __getattr__ is rare. >>>> > >>>> >> any solution that can fix descriptors without also "fixing" __getattribute__ is a lot better >>>> >>>> >In practice I don't concern __getattribute__. But in my opinion it's better to 'fix' this in python4. >>>> >>>> Why? If you think that erroneous uses are rare or nonexistent, while intentional uses are rare but not nonexistent, "fixing" it means breaking code gratuitously for no benefit. >>>> >> all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so >>>> >This is a big problem, OK. >>>> > >>>> >> You didn't comment on the alternative I suggested; would it not satisfy your needs, or have some other problem that makes it unacceptable? >>>> >I don't quite understand, you mean adding a subclass of object with the only difference of this behavior? >>>> >>>> >>>> No, adding a subclass of AttributeError, much like the one you mentioned in the bug report, but with the opposite meaning: the existing AttributeError continues to trigger __getattr__, but the new subclass doesn't. This makes it trivial to write new code that doesn't accidentally trigger __getattr__, without breaking old code (or rare new code) that wants to trigger __getattr__. >>>> >>>> The code currently does something like this pseudocode: >>>> >>>> try: >>>> val = obj.__getattribute__(name) >>>> except AttributeError: >>>> __getattr__ = getattr(type(obj), '__getattr__', None) >>>> if __getattr__: return __getattr__(name) >>>> >>>> I'm cheating a bit, but you get the idea. The problem is that we have no idea whether __getattribute__ failed to find anything (in which case we definitely want __getattr__ called), or found a descriptor whose __get__ raised an AttributeError (in which case we may not--e.g., a write-only attribute should not all through to __getattr__). >>>> >>>> My suggestion is to change it like this: >>>> >>>> >>>> try: >>>> val = obj.__getattribute__(name) >>>> except AttributeDynamicError: >>>> raise >>>> except AttributeError: >>>> __getattr__ = getattr(type(obj), '__getattr__', None) >>>> if __getattr__: return __getattr__(name) >>>> >>>> >>>> Now, if __getattribute__ found a descriptor whose __get__ raised an AttributeDynamicError, that passes on to the user code. (And, since it's a subclass of AttributeError, the user code should have no problem handling it.) And the Descriptor HOWTO will be changed to suggest raising AttributeDynamicError, except when you explicitly want it to call __getattr__, which you usually don't. And examples like simulating a write-only attribute will raise AttributeDynamicError. And maybe @property will automatically convert any AttributeError to AttributeDynamicError (not sure about that part). >>>> >>>> So, new code can easily be written to act the way you want, but existing code using descriptors that intentionally raise or pass an AttributeError continues to work the same way it always has, and new code that does the same can also be written easily. >>>> >>>> Obviously, the downside of any backward-compat-friendly change is that someone who has old code with a hidden bug they didn't know about will still have that same bug in Python 3.7; they have to change their code to take advantage of the fix. But I don't think that's a serious problem. (Especially if we decide @property is buggy and should be changed--most people who are writing actual custom descriptors, not just using the ones in the stdlib, probably understand this stuff.) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wjun77 at gmail.com Sun Nov 29 13:10:12 2015 From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=) Date: Mon, 30 Nov 2015 02:10:12 +0800 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: References: <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com> <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com> <84C3BBEA-3D17-4B50-9571-6405EABE03F3@yahoo.com> Message-ID: > A dedicated subclass of *Attr*ibuteError to trigger __get*attr*__ is more natural. I mean it's unnatural to convert AttributeError to AttributeDynamicError by property or descriptor. Why not use RuntimeError as that of PEP479? And there's no need to write custom property-like descriptor factory when you want to trigger __getattr__, although it may be only a few lines of code. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Nov 29 13:34:19 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 30 Nov 2015 05:34:19 +1100 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: References: <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com> <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com> <84C3BBEA-3D17-4B50-9571-6405EABE03F3@yahoo.com> Message-ID: On Mon, Nov 30, 2015 at 5:10 AM, ?? wrote: >> A dedicated subclass of AttributeError to trigger __getattr__ is more >> natural. > I mean it's unnatural to convert AttributeError to AttributeDynamicError by > property or descriptor. Why not use RuntimeError as that of PEP479? > Earlier I posted some example code that would do exactly that. If an AttributeError would escape a property function, it becomes a RuntimeError. Does that do what you're looking for? ChrisA From abarnert at yahoo.com Sun Nov 29 14:31:14 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 29 Nov 2015 11:31:14 -0800 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: References: <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com> <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com> <84C3BBEA-3D17-4B50-9571-6405EABE03F3@yahoo.com> Message-ID: <6EA6FD1E-AB22-431C-9A68-88C47975B667@yahoo.com> On Nov 29, 2015, at 10:10, ?? wrote: > > > A dedicated subclass of AttributeError to trigger __getattr__ is more natural. > I mean it's unnatural to convert AttributeError to AttributeDynamicError by property or descriptor. Again, nobody has suggested that the descriptor protocol should do such a thing, so I don't know why you keep arguing about it. I did suggest that maybe the property descriptor should do so. Or maybe not. If your only problem with my alternate proposal is something that was included as a "maybe" optional bit in parentheses, then you don't seem to have an actual problem with my alternate proposal. Which brings us back to the point: My alternate proposal is less disruptive and more backward-compatible. Is there some actual problem with it, or some other reason to prefer your original version? > Why not use RuntimeError as that of PEP479? Three reasons: 1. There are thousands of lines of working code that handle AttributeError; forcing them to change to handle RuntimeError would be gratuitous backward incompatibility. 2. RuntimeError suggests that the code did something unexpected wrong, rather than a specific thing, possibly intentionally, that you may want to trap and handle. In PEP 479, this makes sense--it was decided that generators should not raise StopIteration instead of returning, and therefore it is a serious error to do so, not just a normal exception. But raising AttributeError from a property or other descriptor is a perfectly reasonable thing to do. It comes naturally from build-time proxies. It's recommended in the descriptor HOWTO. And so on. It makes perfect sense--and there's no other way to accomplish what it does, either. (How would you implement a write-only property that makes "print(spam.eggs)" raise an AttributeError except by raising it in the property function?) 3. In the vast majority of uses of properties and other descriptors, there is no __getattr__, and AttributeError is exactly what people want there. The only problem to be solved here is the rare but unexpected and annoying interaction of such dynamic AttributeErrors and __getattr__. Preventing descriptors from raising AttributeError even in the common cases to block that rare case is like removing a smudge from your table by disintegrating the table. Yes, the smudge is gone, but you haven't come out ahead. > And there's no need to write custom property-like descriptor factory when you want to trigger __getattr__, although it may be only a few lines of code. Your original suggestion would make it _impossible_ for a descriptor to trigger __getattr__. And so would your suggestion here of converting AttributeError to RuntimeError. That doesn't remove the need; it leaves the need but makes it impossible to satisfy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wjun77 at gmail.com Mon Nov 30 00:14:55 2015 From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=) Date: Mon, 30 Nov 2015 13:14:55 +0800 Subject: [Python-ideas] The AttributeError/__getattr__ mechanism In-Reply-To: <6EA6FD1E-AB22-431C-9A68-88C47975B667@yahoo.com> References: <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com> <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com> <84C3BBEA-3D17-4B50-9571-6405EABE03F3@yahoo.com> <6EA6FD1E-AB22-431C-9A68-88C47975B667@yahoo.com> Message-ID: > > Earlier I posted some example code that would do exactly that. If an > AttributeError would escape a property function, it becomes a > RuntimeError. Does that do what you're looking for? > Yes, while I need to decorate every property with *dontraise*. In python 3.5 or older it do be a handy tool and thanks for your suggestion. However, similar scheme could be used to solve the problem of PEP479, but a change in the interpreter core is made. I would like similar change in interpreter core or stdlib in python 3.6, so no one will encounter the same problem with future versions of python, knowing this discussion or not. > Again, nobody has suggested that the descriptor protocol should do such a thing, so I don't know why you keep arguing about it. I myself suggested it as an option if nobody else, that's all. > Is there some actual problem with it, or some other reason to prefer your original version? > Your original suggestion would make it _impossible_ for a descriptor to trigger __getattr__. I proposed 3 versions: AttributeMissError, RuntimeError, or 'object.__getattribute__ calls __getattr__ explicitly; __getattr__ not triggered by AttributeError anymore' in the issue tracker. In fact I think I am not qualified to talk about the detailed implementation because I know little about the related CPython code. I said I personally prefer no change to your version in aesthetic, which means that I don't have any persuasive reasons. I think expressing opinion is more important than persuading others. > Three reasons: These looks reasonable to me and thanks for your explanation. Now I think your version is better than RuntimeError. So if compatibility do count, your version seems the only choice other than no change. 2015-11-30 3:31 GMT+08:00 Andrew Barnert : > On Nov 29, 2015, at 10:10, ?? wrote: > > > A dedicated subclass of *Attr*ibuteError to trigger __get*attr*__ is > more natural. > I mean it's unnatural to convert AttributeError to AttributeDynamicError > by property or descriptor. > > > Again, nobody has suggested that the descriptor protocol should do such a > thing, so I don't know why you keep arguing about it. > > I did suggest that maybe the property descriptor should do so. Or maybe > not. If your only problem with my alternate proposal is something that was > included as a "maybe" optional bit in parentheses, then you don't seem to > have an actual problem with my alternate proposal. > > Which brings us back to the point: My alternate proposal is less > disruptive and more backward-compatible. Is there some actual problem with > it, or some other reason to prefer your original version? > > Why not use RuntimeError as that of PEP479? > > > Three reasons: > > 1. There are thousands of lines of working code that handle > AttributeError; forcing them to change to handle RuntimeError would be > gratuitous backward incompatibility. > > 2. RuntimeError suggests that the code did something unexpected wrong, > rather than a specific thing, possibly intentionally, that you may want to > trap and handle. In PEP 479, this makes sense--it was decided that > generators should not raise StopIteration instead of returning, and > therefore it is a serious error to do so, not just a normal exception. But > raising AttributeError from a property or other descriptor is a perfectly > reasonable thing to do. It comes naturally from build-time proxies. It's > recommended in the descriptor HOWTO. And so on. It makes perfect sense--and > there's no other way to accomplish what it does, either. (How would you > implement a write-only property that makes "print(spam.eggs)" raise an > AttributeError except by raising it in the property function?) > > 3. In the vast majority of uses of properties and other descriptors, there > is no __getattr__, and AttributeError is exactly what people want there. > The only problem to be solved here is the rare but unexpected and annoying > interaction of such dynamic AttributeErrors and __getattr__. Preventing > descriptors from raising AttributeError even in the common cases to block > that rare case is like removing a smudge from your table by disintegrating > the table. Yes, the smudge is gone, but you haven't come out ahead. > > And there's no need to write custom property-like descriptor factory when > you want to trigger __getattr__, although it may be only a few lines of > code. > > > Your original suggestion would make it _impossible_ for a descriptor to > trigger __getattr__. And so would your suggestion here of converting > AttributeError to RuntimeError. That doesn't remove the need; it leaves the > need but makes it impossible to satisfy. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Nov 30 01:45:25 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 30 Nov 2015 16:45:25 +1000 Subject: [Python-ideas] PEP471 - (os.scandir()) In-Reply-To: <3E188123-D4E5-4913-AA56-53A9CEF85BDD@yahoo.com> References: <56579431.4010907@lucidity.plus.com> <3E188123-D4E5-4913-AA56-53A9CEF85BDD@yahoo.com> Message-ID: On 28 November 2015 at 04:42, Andrew Barnert via Python-ideas wrote: > On Nov 27, 2015, at 10:32, Paul Moore wrote: >> >>> On 26 November 2015 at 23:22, Erik wrote: >>> I have studied the PEP, followed a lot of the references and looked at the >>> 3.5.0 implementation. I can't see that I've missed such a thing already >>> existing, but it's possible. If so, perhaps this is instead a request to >>> make that thing more obvious somehow! >> >> Does pathlib use scandir? If so, then maybe you get the caching >> benefits by using pathlib? And if pathlib doesn't use scandir, maybe >> it should? [I just checked, it looks like pathlib doesn't use scandir >> :-(] > > Does pathlib even have a walk equivalent? (I know it has glob('**'), but that's not the same thing.) > > Or are you suggesting that people should use path.iterdir with explicit recursion (or an explicit stack), and therefore just changing iterdir to use scandir (and prefill as many cached attribs as possible in each result) is what we want? The main problem with having pathlib do any caching at all is that caching the results of stat calls implicitly in any context is a recipe for significant confusion, since you're at the mercy of race conditions as the filesystem changes out from underneath you. There also isn't an obviously "right" answer in the general case for cache invalidation, as in some cases you're interested in the file as it was when you originally opened it, and don't care if it got swapped out from underneath you, while in others you're interested in the file path, and want the filesystem details for right now, not the details from a few seconds ago. For os.scandir(), we just delegate the behaviour to the underlying filesystem APIs - how readdir() and FindNextFile react to the directory contents changing during iteration is OS defined, and Python will inherit that variation (and may miss newly added files as a result). The current os.walk() implementation constrains the scope of the scandir() filesystem state caching, since it doesn't let the DirEntry objects escape outside the generator - there's no need to ask yourself "What's the risk of stale filesystem data here?", since you're not getting access to the cached info in the first place, and hence always need to go query the filesystem directly. This is a fairly universal pattern: for a given *application* you can likely figure out what to cache and when to invalidate it, even though those are unanswerable questions in the general case. Another example of that would be the stat caches in the current implementation of the import system, together with the corresponding need to call importlib.invalidate_caches() if you want to make sure the import system can see a module that was only just written to disk. That's not to say that a general purpose directory walking utility producing DirEntry objects isn't an interesting prospect. Rather, it's an attempt to highlight that this is an area where there may be a significant gulf between "works for my use case" and "is a suitable addition to the standard library", particularly since this can now be a pure Python recipe atop os.scandir. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From abarnert at yahoo.com Mon Nov 30 04:12:23 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 30 Nov 2015 01:12:23 -0800 Subject: [Python-ideas] PEP471 - (os.scandir()) In-Reply-To: References: <56579431.4010907@lucidity.plus.com> <3E188123-D4E5-4913-AA56-53A9CEF85BDD@yahoo.com> Message-ID: <9B8778E8-0133-42A9-A116-2A0E931B6ACE@yahoo.com> On Nov 29, 2015, at 22:45, Nick Coghlan wrote: > > On 28 November 2015 at 04:42, Andrew Barnert via Python-ideas > wrote: >> On Nov 27, 2015, at 10:32, Paul Moore wrote: >>> >>>> On 26 November 2015 at 23:22, Erik wrote: >>>> I have studied the PEP, followed a lot of the references and looked at the >>>> 3.5.0 implementation. I can't see that I've missed such a thing already >>>> existing, but it's possible. If so, perhaps this is instead a request to >>>> make that thing more obvious somehow! >>> >>> Does pathlib use scandir? If so, then maybe you get the caching >>> benefits by using pathlib? And if pathlib doesn't use scandir, maybe >>> it should? [I just checked, it looks like pathlib doesn't use scandir >>> :-(] >> >> Does pathlib even have a walk equivalent? (I know it has glob('**'), but that's not the same thing.) >> >> Or are you suggesting that people should use path.iterdir with explicit recursion (or an explicit stack), and therefore just changing iterdir to use scandir (and prefill as many cached attribs as possible in each result) is what we want? > > The main problem with having pathlib do any caching at all is that > caching the results of stat calls implicitly in any context is a > recipe for significant confusion, since you're at the mercy of race > conditions as the filesystem changes out from underneath you. There > also isn't an obviously "right" answer in the general case for cache > invalidation, as in some cases you're interested in the file as it was > when you originally opened it, and don't care if it got swapped out > from underneath you, while in others you're interested in the file > path, and want the filesystem details for right now, not the details > from a few seconds ago. > > For os.scandir(), we just delegate the behaviour to the underlying > filesystem APIs - how readdir() and FindNextFile react to the > directory contents changing during iteration is OS defined, and Python > will inherit that variation (and may miss newly added files as a > result). > > The current os.walk() implementation constrains the scope of the > scandir() filesystem state caching, since it doesn't let the DirEntry > objects escape outside the generator That's a good point. The fts functions that both BSD and GNU use to replace the ftw and the various other old *nix filesystem walk functions deal with this carefully; the short version is that any information you want to keep around from the current file after looking at the next file, you have to explicitly copy it, which makes it hard to confuse yourself about how up-to-date it is. (You can also go a directory at a time, more like os.walk, but with the same basic restriction: once you go to the next directory, the previous list of file entries is invalid.) > - there's no need to ask yourself > "What's the risk of stale filesystem data here?", since you're not > getting access to the cached info in the first place, and hence always > need to go query the filesystem directly. > > This is a fairly universal pattern: for a given *application* you can > likely figure out what to cache and when to invalidate it, even though > those are unanswerable questions in the general case. Another example > of that would be the stat caches in the current implementation of the > import system, together with the corresponding need to call > importlib.invalidate_caches() if you want to make sure the import > system can see a module that was only just written to disk. > > That's not to say that a general purpose directory walking utility > producing DirEntry objects isn't an interesting prospect. Rather, it's > an attempt to highlight that this is an area where there may be a > significant gulf between "works for my use case" and "is a suitable > addition to the standard library", particularly since this can now be > a pure Python recipe atop os.scandir. I still think providing an fts-like API instead of os.walk would be the clearest way to provide cached data (especially since people could look up nice generic documentation on fts). I don't think it would be too hard to emulate it (or a large enough subset--you can ask fts to return anything from just names to full stat structs, with well-defined performance characteristics for each combination of flags, and that probably can't be exactly the same) on top of scandir (or directly on Windows FindFirst/etc.), but then I said that when the PEP was being discussed and then never had time to actually try it... The bigger problem is that "you have to copy it, which is painful enough that you can't confuse yourself" is a much lower barrier to confusion in Python than in C, so it might not be as effective. From p.f.moore at gmail.com Mon Nov 30 04:40:21 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 30 Nov 2015 09:40:21 +0000 Subject: [Python-ideas] PEP471 - (os.scandir()) In-Reply-To: <9B8778E8-0133-42A9-A116-2A0E931B6ACE@yahoo.com> References: <56579431.4010907@lucidity.plus.com> <3E188123-D4E5-4913-AA56-53A9CEF85BDD@yahoo.com> <9B8778E8-0133-42A9-A116-2A0E931B6ACE@yahoo.com> Message-ID: On 30 November 2015 at 09:12, Andrew Barnert wrote: > I still think providing an fts-like API instead of os.walk would be the clearest way to provide cached data (especially since people could look up nice generic documentation on fts). As a Windows user I'm not familiar with fts (and Google didn't come up with anything obvious). So I'm not sure how true "people could look up generic docuimentation" would be in practice. But from your description it may be useful - I presume it's something that could be built as a 3rd party library based on os.scandir, at least as an initial proof of concept? Paul From steve at pearwood.info Mon Nov 30 05:32:54 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 30 Nov 2015 21:32:54 +1100 Subject: [Python-ideas] Write-only property [was Re: The AttributeError/__getattr__ mechanism] In-Reply-To: <6EA6FD1E-AB22-431C-9A68-88C47975B667@yahoo.com> References: <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com> <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com> <84C3BBEA-3D17-4B50-9571-6405EABE03F3@yahoo.com> <6EA6FD1E-AB22-431C-9A68-88C47975B667@yahoo.com> Message-ID: <20151130103254.GW3821@ando.pearwood.info> On Sun, Nov 29, 2015 at 11:31:14AM -0800, Andrew Barnert via Python-ideas wrote: > (How would you implement a write-only property > that makes "print(spam.eggs)" raise an AttributeError except by > raising it in the property function?) eggs = property(None, setter, None, "") makes eggs a write-only property. print(spam.eggs) gives AttributeError: unreadable attribute -- Steve From abarnert at yahoo.com Mon Nov 30 11:07:14 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 30 Nov 2015 08:07:14 -0800 Subject: [Python-ideas] Write-only property [was Re: The AttributeError/__getattr__ mechanism] In-Reply-To: <20151130103254.GW3821@ando.pearwood.info> References: <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com> <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com> <84C3BBEA-3D17-4B50-9571-6405EABE03F3@yahoo.com> <6EA6FD1E-AB22-431C-9A68-88C47975B667@yahoo.com> <20151130103254.GW3821@ando.pearwood.info> Message-ID: On Nov 30, 2015, at 02:32, Steven D'Aprano wrote: > >> On Sun, Nov 29, 2015 at 11:31:14AM -0800, Andrew Barnert via Python-ideas wrote: >> >> (How would you implement a write-only property >> that makes "print(spam.eggs)" raise an AttributeError except by >> raising it in the property function?) > > eggs = property(None, setter, None, "") > > makes eggs a write-only property. print(spam.eggs) gives > > AttributeError: unreadable attribute And how do you think property implements that? The Descriptor HOWTO shows a pure-Python equivalent to property, which shows exactly how it works: by raising it in the property __get__ function, of course. Also notice that it looks exactly the same as if you'd provided an fget argument to property and that function raised (except that way you can control the text of the error). I think this is a good design. And I think that, except for often-unwanted (but usually-irrelevant) interaction with __getattr__, it's exactly what users would want and expect. But even if that weren't true, it's what Python had done for decades. Which implies that if we are going to gratuitously break it, we need to make it relatively cheap for people whose code depends on the existing behavior to restore it. def __get__(self, obj, objtype=None): if obj is None: return self if self.fget is None: raise AttributeError("unreadable attribute") return self.fget(obj) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Nov 30 11:18:25 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 30 Nov 2015 08:18:25 -0800 Subject: [Python-ideas] PEP471 - (os.scandir()) In-Reply-To: References: <56579431.4010907@lucidity.plus.com> <3E188123-D4E5-4913-AA56-53A9CEF85BDD@yahoo.com> <9B8778E8-0133-42A9-A116-2A0E931B6ACE@yahoo.com> Message-ID: <85EF61C2-3448-4A5B-B315-CB63E0B93E1F@yahoo.com> On Nov 30, 2015, at 01:40, Paul Moore wrote: > >> On 30 November 2015 at 09:12, Andrew Barnert wrote: >> I still think providing an fts-like API instead of os.walk would be the clearest way to provide cached data (especially since people could look up nice generic documentation on fts). > > As a Windows user I'm not familiar with fts (and Google didn't come up > with anything obvious). The perils of acronym-based naming; it's very easy to go from one of two meaningful search results to way down the list just because UrbanDictionary popularized some txt speak slang and wikipedia started covering every government agency in the world with its name translated to English... So you're right, that benefit no longer applies. You can still find "man fts" very easily, but that isn't what people would be looking for, and doesn't find any of the user-friendly tutorials, just the manpage. > So I'm not sure how true "people could look up > generic docuimentation" would be in practice. But from your > description it may be useful - I presume it's something that could be > built as a 3rd party library based on os.scandir, at least as an > initial proof of concept? A complete implementation that supported all of the flags and maintained the appropriate performance guarantees might be hard. But a partial implementation that supports just the most common flags and falls back to "stat everything, sometimes twice" as a proof of concept should be doable, once I get some free time. From henryschreineriii at gmail.com Mon Nov 30 11:32:49 2015 From: henryschreineriii at gmail.com (Henry Schreiner) Date: Mon, 30 Nov 2015 10:32:49 -0600 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators Message-ID: <38974CAF-ECC7-499F-819E-61D46C713FF3@gmail.com> It seems to me there are two ideas being discussed. One is adding 2 new operators, something like && and ||, and providing them for some sort of "element wise" or other boolean context, and giving them a better order of evaluations than bitwise ops. This would have the advantage of being usable on the built in python lists/tuples if needed, but otherwise doesn't seem to solve all the issues above, including expressions like -1 < mat < 1, which work for python variables but not for things like Numpy arrays. (for example, selecting values from an array works like this: mat[(-1 From henryschreineriii at gmail.com Mon Nov 30 11:22:31 2015 From: henryschreineriii at gmail.com (Henry Schreiner) Date: Mon, 30 Nov 2015 08:22:31 -0800 (PST) Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <23d4f7d4-2cf0-49d5-9c18-71b1edce216f@googlegroups.com> References: <-334649383431326154@unknownmsgid> <5653FE15.8080402@canterbury.ac.nz> <3518005034548011247@unknownmsgid> <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com> <56577194.5000003@canterbury.ac.nz> <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com> <5658B252.1060902@brenbarn.net> <23d4f7d4-2cf0-49d5-9c18-71b1edce216f@googlegroups.com> Message-ID: <67d34a7d-d634-4a0f-bdf9-5a79015523f6@googlegroups.com> Sorry, the order was incorrect in that last message. It should be: When x and y is written, if x has __booland__, then x.__booland__(y) is returned. If not, then return x if x==False else if x==True, then if y has y.__rbooland__(x), that is returned. If not, then return y. On Monday, November 30, 2015 at 10:11:03 AM UTC-6, Henry Schreiner wrote: > > It seems to me there are two ideas being discussed. One is adding 2 new > operators, something like && and ||, and providing them for some sort of > "element wise" or other boolean context, and giving them a better order of > evaluations than bitwise ops. This would have the advantage of being usable > on the built in python lists/tuples if needed, but otherwise doesn't seem > to solve all the issues above, including expressions like -1 < mat < 1, > which work for python variables but not for things like Numpy arrays. (for > example, selecting values from an array works like this: mat[(-1 (mat<1)] but not like this: mat[-1 < mat < 1] or this: mat[-1 mat<1] ). > > The common "general purpose infix notation" pops up here, of course, but > I'd consider it a variant of the first idea. > > The other idea is that overloads could be added to the existing and and or > operators. This, I believe, would solve all the main problems, and would > not add more to the language for a newcomer to learn. (In fact, most people > could forget about the current bitwise and and or, since that's not what > most people need, at least those using databases and Numpy). The downside > is that the current and and or operators are short-circuiting, and adding > a way to overload a short-circuiting operator is tricky/ugly/expensive > (though the last one might not be that bad anymore). But, as I think Andrew > was mentioning, what about a mixed solution? One that provides a > non-short-circuiting overload, but the original short-circuiting behavior > is default? It would look like this: > > Two new magic methods would be added, for now called __booland__ and > __boolor__, along with __rbooland__ and __rboolor__. > > > > Evaluation would work like this, using and for an example: > When x and y is written, if x has __booland__, then y.__booland__(y) is > called. If it does not, then if y exists and has y.__rbooland__(x), that > is called. If neither, then x and y is performed in a normal, > short circuiting manner, calling bool(x) then calling bool(y) only if x > == True. (The second line here might be the hard part due to the way > short circuiting works) > > > > This would allow object that want custom and/or behavior to implement > these magic functions. They would lose (explicit) short-circuiting, but > they would gain custom behavior. Most of these special objects do not need > or expect the short circuit behavior (Only if an array was all True or > False would it even make sense in numpy , and this would allow > expressions like -1 < mat < 1 to work as expected in Python. This would > be easy to teach to newcomers, too, since only objects that have a definite > True/False should short circuit, others have the special methods. The > old behavior could be forced on overriding objects by wrapping them in > bool(). > > > Would something like that be possible and better than PEP 335? > > PS: > Either type of solution would also be useful for the Plumbum library, by > the way, as it would be useful to add and-ing and or-ing to pipelines, but > the bitwise operators have a different meaning for Plumbum. Due to the lazy > evaluation of Plumbum, even the above solution (or any solution) would > still allow short-circuiting in the pipeline. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Nov 30 12:08:12 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 1 Dec 2015 04:08:12 +1100 Subject: [Python-ideas] Write-only property [was Re: The AttributeError/__getattr__ mechanism] In-Reply-To: References: <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com> <84C3BBEA-3D17-4B50-9571-6405EABE03F3@yahoo.com> <6EA6FD1E-AB22-431C-9A68-88C47975B667@yahoo.com> <20151130103254.GW3821@ando.pearwood.info> Message-ID: <20151130170807.GY3821@ando.pearwood.info> On Mon, Nov 30, 2015 at 08:07:14AM -0800, Andrew Barnert wrote: > On Nov 30, 2015, at 02:32, Steven D'Aprano wrote: > > > >> On Sun, Nov 29, 2015 at 11:31:14AM -0800, Andrew Barnert via Python-ideas wrote: > >> > >> (How would you implement a write-only property > >> that makes "print(spam.eggs)" raise an AttributeError except by > >> raising it in the property function?) > > > > eggs = property(None, setter, None, "") > > > > makes eggs a write-only property. print(spam.eggs) gives > > > > AttributeError: unreadable attribute > > And how do you think property implements that? Sorry, I misunderstood your (rhetorical?) question -- I read it as implying one should explicitly write a getter which raised AttributeError. -- Steve From guido at python.org Mon Nov 30 12:23:52 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 30 Nov 2015 09:23:52 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <67d34a7d-d634-4a0f-bdf9-5a79015523f6@googlegroups.com> References: <-334649383431326154@unknownmsgid> <5653FE15.8080402@canterbury.ac.nz> <3518005034548011247@unknownmsgid> <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com> <56577194.5000003@canterbury.ac.nz> <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com> <5658B252.1060902@brenbarn.net> <23d4f7d4-2cf0-49d5-9c18-71b1edce216f@googlegroups.com> <67d34a7d-d634-4a0f-bdf9-5a79015523f6@googlegroups.com> Message-ID: On Mon, Nov 30, 2015 at 8:22 AM, Henry Schreiner < henryschreineriii at gmail.com> wrote: > Sorry, the order was incorrect in that last message. It should be: > > When x and y is written, if x has __booland__, then x.__booland__(y) is > returned. If not, then return x if x==False else if x==True, then if y > has y.__rbooland__(x), that is returned. If not, then return y > This proposal isn't significantly different from PEP 335. In PEP 335, first you ask the left operand whether to overload 'and' or not (by calling its __and1__); if it says yes, you evaluate the second arg and ask the first to evaluate it. In your proposal, the first question is answered by inspecting whether x defines __booland__. I don't think this makes much practical difference. The second part of your proposal is just a renaming of __and2__/__rand2__ to __booland__/__rbooland__. I'm not sure I like this much; the return type of these methods is almost never going to be an actual bool, so the 'bool' in the name is somewhat misleading. (The names chosen in PEP 335 are intentionally free of semantic hints, since the intuition of someone who encounters these for the first time isn't likely to be useful.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From henryschreineriii at gmail.com Mon Nov 30 13:08:24 2015 From: henryschreineriii at gmail.com (Henry Schreiner) Date: Mon, 30 Nov 2015 10:08:24 -0800 (PST) Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: <-334649383431326154@unknownmsgid> <5653FE15.8080402@canterbury.ac.nz> <3518005034548011247@unknownmsgid> <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com> <56577194.5000003@canterbury.ac.nz> <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com> <5658B252.1060902@brenbarn.net> <23d4f7d4-2cf0-49d5-9c18-71b1edce216f@googlegroups.com> <67d34a7d-d634-4a0f-bdf9-5a79015523f6@googlegroups.com> Message-ID: <7df3290f-c021-4e8b-b71b-f49be3f1554b@googlegroups.com> I didn't realize the similarity of the proposals. I do think that testing for existence is better than running a method (that might not exist), at least from a high level perspective. So PEP 335 might be dusted off again and run through benchmarks? There seems to be at least a test implementation for Python 2.3, are there benchmarks documented somewhere? It sound like the version that adds new byte codes instead of an additional byte code per op was never included. I do like the cleanliness of not adding extra operators with nearly identical meanings. On Monday, November 30, 2015 at 11:24:12 AM UTC-6, Guido van Rossum wrote: > > On Mon, Nov 30, 2015 at 8:22 AM, Henry Schreiner > wrote: > >> Sorry, the order was incorrect in that last message. It should be: >> >> When x and y is written, if x has __booland__, then x.__booland__(y) is >> returned. If not, then return x if x==False else if x==True, then if y >> has y.__rbooland__(x), that is returned. If not, then return y >> > > This proposal isn't significantly different from PEP 335. In PEP 335, > first you ask the left operand whether to overload 'and' or not (by calling > its __and1__); if it says yes, you evaluate the second arg and ask the > first to evaluate it. In your proposal, the first question is answered by > inspecting whether x defines __booland__. I don't think this makes much > practical difference. The second part of your proposal is just a renaming > of __and2__/__rand2__ to __booland__/__rbooland__. I'm not sure I like this > much; the return type of these methods is almost never going to be an > actual bool, so the 'bool' in the name is somewhat misleading. (The names > chosen in PEP 335 are intentionally free of semantic hints, since the > intuition of someone who encounters these for the first time isn't likely > to be useful.) > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 30 13:23:26 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 30 Nov 2015 10:23:26 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <7df3290f-c021-4e8b-b71b-f49be3f1554b@googlegroups.com> References: <-334649383431326154@unknownmsgid> <5653FE15.8080402@canterbury.ac.nz> <3518005034548011247@unknownmsgid> <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com> <56577194.5000003@canterbury.ac.nz> <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com> <5658B252.1060902@brenbarn.net> <23d4f7d4-2cf0-49d5-9c18-71b1edce216f@googlegroups.com> <67d34a7d-d634-4a0f-bdf9-5a79015523f6@googlegroups.com> <7df3290f-c021-4e8b-b71b-f49be3f1554b@googlegroups.com> Message-ID: Well, there is still the issue of whether numpy would adopt PEP 335 even if it was implemented. I'm still waiting for a response to my message questioning how common `bool_value and/or numpy_array` really is. Also, there may still be a significant difference between PEP 335 and your proposal -- IIUC in your proposal if the value of the first operand requires the second operand to be evaluated (e.g. `True or y`) then in your proposal if y defined the 'r' special method it will be used; I don't think PEP 335 does that. I like your version better. On Mon, Nov 30, 2015 at 10:08 AM, Henry Schreiner < henryschreineriii at gmail.com> wrote: > I didn't realize the similarity of the proposals. I do think that testing > for existence is better than running a method (that might not exist), at > least from a high level perspective. > > So PEP 335 might be dusted off again and run through benchmarks? There > seems to be at least a test implementation for Python 2.3, are there > benchmarks documented somewhere? It sound like the version that adds new > byte codes instead of an additional byte code per op was never included. > > I do like the cleanliness of not adding extra operators with nearly > identical meanings. > > On Monday, November 30, 2015 at 11:24:12 AM UTC-6, Guido van Rossum wrote: >> >> On Mon, Nov 30, 2015 at 8:22 AM, Henry Schreiner >> wrote: >> >>> Sorry, the order was incorrect in that last message. It should be: >>> >>> When x and y is written, if x has __booland__, then x.__booland__(y) is >>> returned. If not, then return x if x==False else if x==True, then if y >>> has y.__rbooland__(x), that is returned. If not, then return y >>> >> >> This proposal isn't significantly different from PEP 335. In PEP 335, >> first you ask the left operand whether to overload 'and' or not (by calling >> its __and1__); if it says yes, you evaluate the second arg and ask the >> first to evaluate it. In your proposal, the first question is answered by >> inspecting whether x defines __booland__. I don't think this makes much >> practical difference. The second part of your proposal is just a renaming >> of __and2__/__rand2__ to __booland__/__rbooland__. I'm not sure I like this >> much; the return type of these methods is almost never going to be an >> actual bool, so the 'bool' in the name is somewhat misleading. (The names >> chosen in PEP 335 are intentionally free of semantic hints, since the >> intuition of someone who encounters these for the first time isn't likely >> to be useful.) >> >> -- >> --Guido van Rossum (python.org/~guido) >> > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From henryschreineriii at gmail.com Mon Nov 30 15:56:49 2015 From: henryschreineriii at gmail.com (Henry Schreiner) Date: Mon, 30 Nov 2015 14:56:49 -0600 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: <712322608.10180212.1448915321340.JavaMail.yahoo@mail.yahoo.com> References: <712322608.10180212.1448915321340.JavaMail.yahoo@mail.yahoo.com> Message-ID: My proposal, I believe, is exactly the same as the ?Reduced Special Methods set? of PEP 335; that?s the phase two only part. I believe the __r*__ methods there are the same as my proposal. A portion of the complexity of PEP 335 is used in the ?phase 1? special methods, which are only helpful if you want to overload the shortcutting behavior itself on a per object basis, which I don?t think there is much need for. Assuming your methods are part of the class, as Python usually requires anyway, this would allow a class to either be short circuiting, boolean only (not containing __and2__ or __sand__ or __xand__ or whatever it would be called), or would have them and therefore would have customized non-short-circuiting behavior. (But, due to the fact that the class controls how much calculation happens, this generally can still limit computation, as in my Plumbum example; the only requirement is that the variable exist so it can be passed to the function). If it looks like this would be acceptable to the NumPy folks, then maybe PEP 335 could be changed to make the Reduced Special Methods the main suggestion, with extending to the two phase method as a side note? Henry Schreiner > On Nov 30, 2015, at 2:28 PM, Andrew Barnert wrote: > > PEP 335 uses the standard rules for 'r' methods, so y.__rand2__ will only be used if issubclass(type(y), type(x)) or if x.__and2__ doesn't exist. Wouldn't having different rules for 'r' methods in this case than for all other operators be confusing? (Maybe not; the comparison operators are the closest analogs, and they have special rules too...) > > Meanwhile, looking at PEP 335 again, it seems like we could very easily add just the "phase-2" part today (which would allow non-short-circuiting overloads, while preserving short-circuiting for types that don't overload), and then add the "phase-1" part (which would add optional short-circuiting overloads) later if needed, or just never add it if not. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vgr255 at live.ca Mon Nov 30 20:07:29 2015 From: vgr255 at live.ca (Emanuel Barry) Date: Mon, 30 Nov 2015 20:07:29 -0500 Subject: [Python-ideas] Multiple arguments for decorators Message-ID: An idea that I had a couple of times was to have a syntactic way to define multiple arguments for a @decorator. This would be good if you want to, for example, define a property with a getter and a setter. I had a few ideas, but none of them seem to be "the obvious way to do it". I'm including them anyway, but I'd rather see if people are interested in the idea or if it's just something that's not worth it. I'll gladly write a PEP if it gains a lot of interest. Method 1: Add parens around all arguments class Foo: def __init__(self): self._x = 42 @property( def x(self): # getter return self._x def x(self, value): # setter self._x = value def x(self): # deleter del self._x ) This one was my first idea, but with long methods you might very well miss the closing paren, or even wonder "how does the code even run?" Good for small properties like this, bad with longer methods. -1 from me Method 2: Specify how many arguments in the decorator class Foo: def __init__(self): self._x = 42 @3:property def x(self): # getter return self._x def x(self, value): # setter self._x = value def x(self): # deleter del self._x This one would get all three following methods defined (or should it get any value defined -- for example, a None in place of the setter there, would set the setter to None). Implementation-wise, I believe the order is kept while the class is first evaluated, and then often thrown away when it's inserted into the class' __dict__ (I could be wrong, though). This has the advantage of not breaking code already written, as it cannot possibly conflict with an existing name. I'm neutral here. Method 3: Specify arguments using the parameters' names class Foo: def __init__(self): self._x = 42 @property def x:fget(self): return self._x def x:fset(self, value): self._x = value def x:fdel(self): del self._x This has the advantage of being explicit (and the arguments can be swapped since they're named), but the disadvantage that some builtins don't accept kwargs (this would be a good occasion to fix that, though, but that is besides my point). I've got mixed feelings on this one. I'm neutral as well here. What do you guys think? Is this a good idea, or is it not? Thank you for your time!-Emanuel -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 30 20:43:28 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 1 Dec 2015 12:43:28 +1100 Subject: [Python-ideas] Multiple arguments for decorators In-Reply-To: References: Message-ID: On Tue, Dec 1, 2015 at 12:07 PM, Emanuel Barry wrote: > An idea that I had a couple of times was to have a syntactic way to define > multiple arguments for a @decorator. This would be good if you want to, for > example, define a property with a getter and a setter. I had a few ideas, > but none of them seem to be "the obvious way to do it". I'm including them > anyway, but I'd rather see if people are interested in the idea or if it's > just something that's not worth it. I'll gladly write a PEP if it gains a > lot of interest. Other than @property, are there any use-cases you know of? > Method 1: Add parens around all arguments Definitely -1 on this syntax - it'd make incomplete lines of code hard to diagnose. > Method 2: Specify how many arguments in the decorator > > class Foo: > def __init__(self): > self._x = 42 > @3:property > def x(self): # getter > return self._x > def x(self, value): # setter > self._x = value > def x(self): # deleter > del self._x -0.5 on this syntax. It's too much action-at-a-distance; if the three functions are really trivial, then it wouldn't be too bad, but otherwise how do you know that "def x(self):" is making the deleter? > Method 3: Specify arguments using the parameters' names > > class Foo: > def __init__(self): > self._x = 42 > @property > def x:fget(self): > return self._x > def x:fset(self, value): > self._x = value > def x:fdel(self): > del self._x > > This has the advantage of being explicit (and the arguments can be swapped > since they're named), but the disadvantage that some builtins don't accept > kwargs (this would be a good occasion to fix that, though, but that is > besides my point). This is very close to the existing syntax for @property, and it better adorns the individual functions. +0.5. What happens if you rename the property, though? How about this: class Foo: def __init__(self): self._x = 42 @property def x(self): # First positional parameter return self._x def :fset(self, value): # Named parameter self._x = value def :fdel(self): # Another named parameter del self._x Remove the repetition of the name "x", and then there's no chance of getting it wrong. But it's sounding here more like you're creating a block of code. And that, to my mind, suggests that it should be indented. Something like: class Foo: def __init__(self): self._x = 42 with @property as x: def fget(self): return self._x def fset(self, value): self._x = value def fdel(self): del self._x This groups the three functions, and their names would be available to use as keyword arguments. It would be rather different from current with-block semantics, though. Effectively, it'd be something like this: 1) Evaluate the decorator itself (in this case, the simple name 'property'), but don't call it. 2) Create a new scope, nested inside the current scope. (Similar to a list comp.) 3) Execute the indented block in that scope. 4) Call the decorator, passing all names bound in this scope as keyword arguments. 5) Bind the return value of the decorator to the given name. Thoughts? ChrisA From cfkaran2 at gmail.com Mon Nov 30 20:54:45 2015 From: cfkaran2 at gmail.com (Cem Karan) Date: Mon, 30 Nov 2015 20:54:45 -0500 Subject: [Python-ideas] Multiple arguments for decorators In-Reply-To: References: Message-ID: <48DFBAB2-974D-44EE-BB0A-A4F673B0A908@gmail.com> On Nov 30, 2015, at 8:43 PM, Chris Angelico wrote: > On Tue, Dec 1, 2015 at 12:07 PM, Emanuel Barry wrote: <> > But it's sounding here more like you're creating a block of code. And > that, to my mind, suggests that it should be indented. Something like: > > class Foo: > def __init__(self): > self._x = 42 > with @property as x: > def fget(self): > return self._x > def fset(self, value): > self._x = value > def fdel(self): > del self._x > > This groups the three functions, and their names would be available to > use as keyword arguments. It would be rather different from current > with-block semantics, though. Effectively, it'd be something like > this: > > 1) Evaluate the decorator itself (in this case, the simple name > 'property'), but don't call it. > 2) Create a new scope, nested inside the current scope. (Similar to a > list comp.) > 3) Execute the indented block in that scope. > 4) Call the decorator, passing all names bound in this scope as > keyword arguments. > 5) Bind the return value of the decorator to the given name. I like this; it really SHOULD be indented. But I also agree that 'with' is probably not the best keyword here, it makes it a little difficult to quickly read and see what's going on. Since this is for decorators, could we just drop 'with' altogether? E.g.: class Foo: def __init__(self): self._x = 42 @property as x: def fget(self): return self._x def fset(self, value): self._x = value def fdel(self): del self._x Thanks, Cem Karan From vgr255 at live.ca Mon Nov 30 21:01:39 2015 From: vgr255 at live.ca (Emanuel Barry) Date: Mon, 30 Nov 2015 21:01:39 -0500 Subject: [Python-ideas] Multiple arguments for decorators In-Reply-To: References: , Message-ID: > Date: Tue, 1 Dec 2015 12:43:28 +1100 > From: rosuav at gmail.com > CC: python-ideas at python.org > Subject: Re: [Python-ideas] Multiple arguments for decorators > > > Other than @property, are there any use-cases you know of? Not in the builtins/stdlib, but I do have decorators that I'd probably rewrite to support the new syntax. In one of my projects I work around this by some other means, but it looks ugly. > > Method 1: Add parens around all arguments > > Definitely -1 on this syntax - it'd make incomplete lines of code hard > to diagnose. Yep, not surprised -- I only included it because that was the first idea I had, but I myself don't like it :) > > Method 2: Specify how many arguments in the decorator > > -0.5 on this syntax. It's too much action-at-a-distance; if the three > functions are really trivial, then it wouldn't be too bad, but > otherwise how do you know that "def x(self):" is making the deleter? I guess it does make it a bit of a hassle to maintain, maybe if it was combined with another one it would make sense. We'll see. > > Method 3: Specify arguments using the parameters' names > > This is very close to the existing syntax for @property, and it better > adorns the individual functions. +0.5. What happens if you rename the > property, though? How about this: > > class Foo: > def __init__(self): > self._x = 42 > @property > def x(self): # First positional parameter > return self._x > def :fset(self, value): # Named parameter > self._x = value > def :fdel(self): # Another named parameter > del self._x > > Remove the repetition of the name "x", and then there's no chance of > getting it wrong. I personally don't like this, the colon there looks weird to me. And you're just ignoring the first method's name and passing it positional, unlike the others which are named. I like the general idea you're bringing though, but it could use a tweak or three imo. > But it's sounding here more like you're creating a block of code. And > that, to my mind, suggests that it should be indented. Something like: > > class Foo: > def __init__(self): > self._x = 42 > with @property as x: > def fget(self): > return self._x > def fset(self, value): > self._x = value > def fdel(self): > del self._x > > This groups the three functions, and their names would be available to > use as keyword arguments. It would be rather different from current > with-block semantics, though. Effectively, it'd be something like > this: > > 1) Evaluate the decorator itself (in this case, the simple name > 'property'), but don't call it. > 2) Create a new scope, nested inside the current scope. (Similar to a > list comp.) > 3) Execute the indented block in that scope. > 4) Call the decorator, passing all names bound in this scope as > keyword arguments. > 5) Bind the return value of the decorator to the given name. > > Thoughts? Glancing at this, it seems to me that "property" is having a unary @ operator applied to it, but I guess that since the possibility to introduce a unary @ operator shrank down to exactly 0 when the decorator syntax was added, that's not really an issue. I'm also not sure about overloading the semantics of the 'with' statement. Nevertheless, I like this approach. I wonder if something similar (using a with statement) can be achieved right now. Probably, with the use of vars and sys._getframe (I never said it would be clean!) > ChrisA Thanks for your input! -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 30 21:03:52 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 1 Dec 2015 13:03:52 +1100 Subject: [Python-ideas] Multiple arguments for decorators In-Reply-To: <48DFBAB2-974D-44EE-BB0A-A4F673B0A908@gmail.com> References: <48DFBAB2-974D-44EE-BB0A-A4F673B0A908@gmail.com> Message-ID: On Tue, Dec 1, 2015 at 12:54 PM, Cem Karan wrote: > I like this; it really SHOULD be indented. But I also agree that 'with' is probably not the best keyword here, it makes it a little difficult to quickly read and see what's going on. Since this is for decorators, could we just drop 'with' altogether? E.g.: > > class Foo: > def __init__(self): > self._x = 42 > @property as x: > def fget(self): > return self._x > def fset(self, value): > self._x = value > def fdel(self): > del self._x > > Thanks, > Cem Karan Either way works for me. My first try was some kind of hack that actually used a context manager as a context manager, but without compiler support, it wouldn't really work properly. So the syntax can be varied away from 'with'. ChrisA From rosuav at gmail.com Mon Nov 30 21:14:48 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 1 Dec 2015 13:14:48 +1100 Subject: [Python-ideas] Multiple arguments for decorators In-Reply-To: References: Message-ID: On Tue, Dec 1, 2015 at 1:01 PM, Emanuel Barry wrote: >> Date: Tue, 1 Dec 2015 12:43:28 +1100 >> From: rosuav at gmail.com >> CC: python-ideas at python.org >> Subject: Re: [Python-ideas] Multiple arguments for decorators >> >> >> Other than @property, are there any use-cases you know of? > > Not in the builtins/stdlib, but I do have decorators that I'd probably > rewrite to support the new syntax. In one of my projects I work around this > by some other means, but it looks ugly. The proposal would be strengthened by more examples. Currently, @property can do something very similar to what your proposal offers, so this is only a small improvement. >> This is very close to the existing syntax for @property, and it better >> adorns the individual functions. +0.5. What happens if you rename the >> property, though? How about this: >> >> class Foo: >> def __init__(self): >> self._x = 42 >> @property >> def x(self): # First positional parameter >> return self._x >> def :fset(self, value): # Named parameter >> self._x = value >> def :fdel(self): # Another named parameter >> del self._x >> >> Remove the repetition of the name "x", and then there's no chance of >> getting it wrong. > > I personally don't like this, the colon there looks weird to me. And you're > just ignoring the first method's name and passing it positional, unlike the > others which are named. I like the general idea you're bringing though, but > it could use a tweak or three imo. Agreed that the colon looks weird, but I don't know of any better way to spell it. This was just a half-way house to the thought that followed, though. >> But it's sounding here more like you're creating a block of code. And >> that, to my mind, suggests that it should be indented. Something like: >> >> class Foo: >> def __init__(self): >> self._x = 42 >> with @property as x: >> def fget(self): >> return self._x >> def fset(self, value): >> self._x = value >> def fdel(self): >> del self._x >> >> This groups the three functions, and their names would be available to >> use as keyword arguments. It would be rather different from current >> with-block semantics, though. Effectively, it'd be something like >> this: >> >> 1) Evaluate the decorator itself (in this case, the simple name >> 'property'), but don't call it. >> 2) Create a new scope, nested inside the current scope. (Similar to a >> list comp.) >> 3) Execute the indented block in that scope. >> 4) Call the decorator, passing all names bound in this scope as >> keyword arguments. >> 5) Bind the return value of the decorator to the given name. >> >> Thoughts? > > Glancing at this, it seems to me that "property" is having a unary @ > operator applied to it, but I guess that since the possibility to introduce > a unary @ operator shrank down to exactly 0 when the decorator syntax was > added, that's not really an issue. I'm also not sure about overloading the > semantics of the 'with' statement. > > Nevertheless, I like this approach. I wonder if something similar (using a > with statement) can be achieved right now. Probably, with the use of vars > and sys._getframe (I never said it would be clean!) I don't think it can, because there's no way for a context manager to say "tell me about all name bindings in this block". The nearest I can think of is a nested class definition, which I can make work, but it's definitely ugly: def call(func): def inner(cls): return func(**{k:v for k,v in cls.__dict__.items() if not k.startswith('_')}) return inner class Foo: def __init__(self): self._x = 42 @call(property) class x: def fget(self): return self._x def fset(self, value): self._x = value def fdel(self): del self._x foo = Foo() print(foo.x) foo.x = 28 print(foo._x) ChrisA From mertz at gnosis.cx Mon Nov 30 21:41:15 2015 From: mertz at gnosis.cx (David Mertz) Date: Mon, 30 Nov 2015 18:41:15 -0800 Subject: [Python-ideas] Multiple arguments for decorators In-Reply-To: References: Message-ID: On Mon, Nov 30, 2015 at 6:14 PM, Chris Angelico wrote: > def call(func): > def inner(cls): > return func(**{k:v for k,v in cls.__dict__.items() if not > k.startswith('_')}) > return inner > > class Foo: > def __init__(self): > self._x = 42 > @call(property) > class x: > def fget(self): > return self._x > def fset(self, value): > self._x = value > def fdel(self): > del self._x > I think this looks perfectly nice, actually. I was just trying to work out almost the same thing but using a `def x()` rather than `class f` as the nesting construct. I think Chris' is better though. I think I might want to define something like: make_property = call(property) class Foo: def __init__(self): self._x = 42 @make_property class x: def fget(self): return self._x def fset(self, value): self._x = value def fdel(self): del self._x -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vgr255 at live.ca Mon Nov 30 21:43:14 2015 From: vgr255 at live.ca (Emanuel Barry) Date: Mon, 30 Nov 2015 21:43:14 -0500 Subject: [Python-ideas] Multiple arguments for decorators In-Reply-To: References: , , , Message-ID: > The proposal would be strengthened by more examples. Currently, > @property can do something very similar to what your proposal offers, > so this is only a small improvement. While searching for more examples, it struck me that I wouldn't want to allow only methods to be passed, but I would want any value (for the name and/or docstring, mainly). For example, consider the following code: class Foo: @attribute def bar(self): return 42 bar.__doc__ = "The meaning of life, the universe, and everything." I would benefit from being able to pass in the doc at the same time, as well. That is a poor example though, here is a better one: @total_decorate(handler=custom_handler, name="special_cased")class Foo: <...> Or, as I do sometimes: @total_decorate()class Foo: <...> This would benefit from being able to make that a single call (given some changes to the code) and avoid the ugly parens, that I see quite often. This example could be extended to reprlib.recursive_repr too. I must admit that I currently lack in examples, I'll try to provide more tomorrow. I took property because it's simple and everyone knows about it (pretty much). > > I personally don't like this, the colon there looks weird to me. And you're > > just ignoring the first method's name and passing it positional, unlike the > > others which are named. I like the general idea you're bringing though, but > > it could use a tweak or three imo. > > Agreed that the colon looks weird, but I don't know of any better way > to spell it. This was just a half-way house to the thought that > followed, though. I'm not particularly attached to any syntax, all my suggestions are only that - suggestions. > I don't think it can, because there's no way for a context manager to > say "tell me about all name bindings in this block". The nearest I can > think of is a nested class definition, which I can make work, but it's > definitely ugly: Well, you store all the names in the previous frame in __enter__, and then do your stuff in __exit__. It *is* hacky though. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Mon Nov 30 21:49:48 2015 From: mertz at gnosis.cx (David Mertz) Date: Mon, 30 Nov 2015 18:49:48 -0800 Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators In-Reply-To: References: <-334649383431326154@unknownmsgid> <5653FE15.8080402@canterbury.ac.nz> <3518005034548011247@unknownmsgid> <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com> <56577194.5000003@canterbury.ac.nz> <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com> <5658B252.1060902@brenbarn.net> <23d4f7d4-2cf0-49d5-9c18-71b1edce216f@googlegroups.com> <67d34a7d-d634-4a0f-bdf9-5a79015523f6@googlegroups.com> <7df3290f-c021-4e8b-b71b-f49be3f1554b@googlegroups.com> Message-ID: I find the need to use np.logical_or quite often in NumPy. I'm sure the NumPy community would much prefer to be able to write: new = arr[arr<10 or arr>100] Instead of the current: new = arr[np.logical_or(arr<10, arr>100)] Likewise for np.logical_and(). On Mon, Nov 30, 2015 at 10:23 AM, Guido van Rossum wrote: > Well, there is still the issue of whether numpy would adopt PEP 335 even > if it was implemented. I'm still waiting for a response to my message > questioning how common `bool_value and/or numpy_array` really is. > > Also, there may still be a significant difference between PEP 335 and your > proposal -- IIUC in your proposal if the value of the first operand > requires the second operand to be evaluated (e.g. `True or y`) then in your > proposal if y defined the 'r' special method it will be used; I don't think > PEP 335 does that. I like your version better. > > On Mon, Nov 30, 2015 at 10:08 AM, Henry Schreiner < > henryschreineriii at gmail.com> wrote: > >> I didn't realize the similarity of the proposals. I do think that testing >> for existence is better than running a method (that might not exist), at >> least from a high level perspective. >> >> So PEP 335 might be dusted off again and run through benchmarks? There >> seems to be at least a test implementation for Python 2.3, are there >> benchmarks documented somewhere? It sound like the version that adds new >> byte codes instead of an additional byte code per op was never included. >> >> I do like the cleanliness of not adding extra operators with nearly >> identical meanings. >> >> On Monday, November 30, 2015 at 11:24:12 AM UTC-6, Guido van Rossum wrote: >>> >>> On Mon, Nov 30, 2015 at 8:22 AM, Henry Schreiner >>> wrote: >>> >>>> Sorry, the order was incorrect in that last message. It should be: >>>> >>>> When x and y is written, if x has __booland__, then x.__booland__(y) is >>>> returned. If not, then return x if x==False else if x==True, then if y >>>> has y.__rbooland__(x), that is returned. If not, then return y >>>> >>> >>> This proposal isn't significantly different from PEP 335. In PEP 335, >>> first you ask the left operand whether to overload 'and' or not (by calling >>> its __and1__); if it says yes, you evaluate the second arg and ask the >>> first to evaluate it. In your proposal, the first question is answered by >>> inspecting whether x defines __booland__. I don't think this makes much >>> practical difference. The second part of your proposal is just a renaming >>> of __and2__/__rand2__ to __booland__/__rbooland__. I'm not sure I like this >>> much; the return type of these methods is almost never going to be an >>> actual bool, so the 'bool' in the name is somewhat misleading. (The names >>> chosen in PEP 335 are intentionally free of semantic hints, since the >>> intuition of someone who encounters these for the first time isn't likely >>> to be useful.) >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> >> > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Nov 30 21:52:11 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 1 Dec 2015 02:52:11 +0000 (UTC) Subject: [Python-ideas] Multiple arguments for decorators In-Reply-To: References: Message-ID: <1745702014.10551218.1448938331185.JavaMail.yahoo@mail.yahoo.com> On Monday, November 30, 2015 6:14 PM, Chris Angelico wrote: > On Tue, Dec 1, 2015 at 1:01 PM, Emanuel Barry wrote: > The proposal would be strengthened by more examples. Currently, > @property can do something very similar to what your proposal offers, > so this is only a small improvement. Agreed. And it's not clear why other decorators can't just do something similar to what @property does. >>> But it's sounding here more like you're creating a block of > code. And >>> that, to my mind, suggests that it should be indented. Agreed. The way to group things in Python is with an indented suite; trying to read three tiny things at the same level as being controlled by something above all of them isn't too terrible, but if there are three big things, or eight tiny things? >>> class Foo: >>> def __init__(self): >>> self._x = 42 >>> with @property as x: >>> def fget(self): >>> return self._x >>> def fset(self, value): >>> self._x = value >>> def fdel(self): >>> del self._x >>> >>> This groups the three functions, and their names would be available to >>> use as keyword arguments. It would be rather different from current >>> with-block semantics, though. I think this is a bad idea from the start. Only functions and classes have scopes; normal suite do not. If you change that to add "... except the suite of a with statement whose context manager is a decorator", that's no longer a simple rule you can hold in your head. > The nearest I can > think of is a nested class definition, which I can make work, but it's > definitely ugly: It seems a lot cleaner to just pass a class to the decorator: class Property: def __init__(self, cls): self.fget = getattr(cls, 'fget', None) self.fset = getattr(cls, 'fset', None) self.fdel = getattr(cls, 'fdel', None) self.doc = getattr(cls, '__doc__', None) # everything below this point is exactly the same as the # existing implementation in the descriptor HOWTO (or # the C implementation in descrobject.c). class Foo: def __init__(self): self._x = 42 @Property class x: def fget(self): return self._x def fset(self, value): self._x = value def fdel(self): del self._x Sure, @call slightly simplifies those 4 lines of boilerplate at the start of Property.__init__, and does the same for every other decorator that you define to be used with @call--but at the cost of making every _use_ of every such decorator uglier, and making things more complex to think through. Is it really worth it? Actually, that's not a rhetorical question; it's hard to guess without seeing more examples beyond @property... From rosuav at gmail.com Mon Nov 30 21:58:37 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 1 Dec 2015 13:58:37 +1100 Subject: [Python-ideas] Multiple arguments for decorators In-Reply-To: <1745702014.10551218.1448938331185.JavaMail.yahoo@mail.yahoo.com> References: <1745702014.10551218.1448938331185.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Tue, Dec 1, 2015 at 1:52 PM, Andrew Barnert wrote: > I think this is a bad idea from the start. Only functions and classes have scopes; normal suite do not. If you change that to add "... except the suite of a with statement whose context manager is a decorator", that's no longer a simple rule you can hold in your head. > And list comprehensions. They have a new nested scope, even though you can't see the function call. Everywhere else that you see a 'for' loop, it's exactly the same semantics as any other assignment - but if it's inside square brackets, it's a special form of local name that doesn't extend past the brackets. This would be the same - a hidden function call that creates a nested scope. ChrisA From stephen at xemacs.org Mon Nov 30 22:16:30 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 1 Dec 2015 12:16:30 +0900 Subject: [Python-ideas] Multiple arguments for decorators In-Reply-To: References: Message-ID: <22109.4366.757509.63946@turnbull.sk.tsukuba.ac.jp> David Mertz writes: > On Mon, Nov 30, 2015 at 6:14 PM, Chris Angelico wrote: >> def call(func): >> def inner(cls): >> return func(**{k:v for k,v in cls.__dict__.items() if not >> k.startswith('_')}) >> return inner >> >> class Foo: >> def __init__(self): >> self._x = 42 >> @call(property) >> class x: >> def fget(self): >> return self._x >> def fset(self, value): >> self._x = value >> def fdel(self): >> del self._x >> > I think this looks perfectly nice, actually. That was my first thought too. The use of the "class" keyword was a bit of a shock at first, but it grew on me, especially in this context of @property (ie, if I think "@property defines a class with appropriate behavior and installs instances on objects of the enclosing class"). I'm having trouble interpreting the function name "call", though. Am I missing something obvious ("Angelico" doesn't sound Dutch to me, though :-), or maybe there's a better (longer) name for it? Also, can call be expected to DTRT for anything *but* property? From kmod at dropbox.com Mon Nov 30 22:21:57 2015 From: kmod at dropbox.com (Kevin Modzelewski) Date: Mon, 30 Nov 2015 19:21:57 -0800 Subject: [Python-ideas] Multiple arguments for decorators In-Reply-To: References: Message-ID: Class scopes definitely feel like a good match -- they are a way of saying "evaluate all of these expression, pass the resulting locals to a custom function, and bind the result of that function to the classname". Usually the function is type(), which constructs a new class, but by setting a custom metaclass we can avoid creating a class just to wrap the scope: class PropertyMetaclass(type): def __new__(cls, name, bases, attrs): return property(attrs.get('get'), attrs.get('set'), attrs.get('del'), attrs.get('__doc__')) class Foo(object): class myprop(metaclass=PropertyMetaclass): def get(self): return 1 def set(self, v): pass __doc__ = 1 f = Foo() print(f.myprop) The "class myprop(metaclass=PropertyClass)" line is pretty ugly though. On Mon, Nov 30, 2015 at 6:41 PM, David Mertz wrote: > On Mon, Nov 30, 2015 at 6:14 PM, Chris Angelico wrote: > >> def call(func): >> def inner(cls): >> return func(**{k:v for k,v in cls.__dict__.items() if not >> k.startswith('_')}) >> return inner >> >> class Foo: >> def __init__(self): >> self._x = 42 >> @call(property) >> class x: >> def fget(self): >> return self._x >> def fset(self, value): >> self._x = value >> def fdel(self): >> del self._x >> > > I think this looks perfectly nice, actually. I was just trying to work > out almost the same thing but using a `def x()` rather than `class f` as > the nesting construct. I think Chris' is better though. I think I might > want to define something like: > > make_property = call(property) > > class Foo: > def __init__(self): > self._x = 42 > @make_property > class x: > def fget(self): > return self._x > def fset(self, value): > self._x = value > def fdel(self): > del self._x > > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 30 22:32:13 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 1 Dec 2015 14:32:13 +1100 Subject: [Python-ideas] Multiple arguments for decorators In-Reply-To: <22109.4366.757509.63946@turnbull.sk.tsukuba.ac.jp> References: <22109.4366.757509.63946@turnbull.sk.tsukuba.ac.jp> Message-ID: On Tue, Dec 1, 2015 at 2:16 PM, Stephen J. Turnbull wrote: > I'm having trouble interpreting the function name "call", though. Am > I missing something obvious ("Angelico" doesn't sound Dutch to me, > though :-), or maybe there's a better (longer) name for it? > > Also, can call be expected to DTRT for anything *but* property? No, it's not Dutch... but my maternal grandparents were Dutch, if that helps? (My parents were both born here in Australia, but my grandparents are split two-and-two Dutch and Sicilian. So I get to eat garlic like nobody's business, and then pretend to understand Python. Maybe.) The semantics of the function I've given are simple: Pass it a callable and a block of assignments (implemented as a class body), and it calls the callable with the assignments as keyword arguments. So it'll work for anything that accepts keyword arguments - which in Python is nearly anything. It doesn't necessarily have to carry callables: import json @call(json.loads) class config: with open("config.json") as _f: # Leading underscore for non-arguments s = _f.read() encoding = "UTF-8" def object_hook(d): if "imag" in d and "real" in d and len(d)==2: return complex(d["real"], d["imag"]) return d parse_int = float import pprint pprint.pprint(config) Incidentally, this works in Python 2 as well as Python 3. I found out by typoing the command and seeing a bunch of u'...' strings in the pprint output - no other difference. ChrisA From abarnert at yahoo.com Mon Nov 30 23:30:04 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 30 Nov 2015 20:30:04 -0800 Subject: [Python-ideas] Multiple arguments for decorators In-Reply-To: References: <1745702014.10551218.1448938331185.JavaMail.yahoo@mail.yahoo.com> Message-ID: <650A2120-A263-4568-8C4D-D1B4241B9382@yahoo.com> On Nov 30, 2015, at 18:58, Chris Angelico wrote: > >> On Tue, Dec 1, 2015 at 1:52 PM, Andrew Barnert wrote: >> I think this is a bad idea from the start. Only functions and classes have scopes; normal suite do not. If you change that to add "... except the suite of a with statement whose context manager is a decorator", that's no longer a simple rule you can hold in your head. > > And list comprehensions. Comprehensions define functions, so it's the same rule; you just have to know that functions can be defined three ways (def statement, lambdas expression, or comprehension) rather than just two. Sure, that's not _ideally_ simple, but that hardly seems a reason to make it even _less_ simple. Also, comprehensions don't have a suite--you can't define arbitrary new variables or use global/nonlocal statements or anything else that would make you have to think carefully about scoping. Most importantly, comprehensions don't have a suite that looks exactly the same as another kind of suite introduced by the same keyword that doesn't define a scope, except on this one special case where it does. From abarnert at yahoo.com Mon Nov 30 23:37:17 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 30 Nov 2015 20:37:17 -0800 Subject: [Python-ideas] Multiple arguments for decorators In-Reply-To: References: Message-ID: On Nov 30, 2015, at 19:21, Kevin Modzelewski via Python-ideas wrote: > > Class scopes definitely feel like a good match -- they are a way of saying "evaluate all of these expression, pass the resulting locals to a custom function, and bind the result of that function to the classname". Usually the function is type(), which constructs a new class, but by setting a custom metaclass we can avoid creating a class just to wrap the scope: Is there really a harm in creating a class? A property is a type, and the obvious way to simulate it in Python rather than C (as shown by the sample code in the HOWTO) is with a class statement. Besides, if you're creating many thousands of properties in a loop, the time and space cost of property creation is probably the least of your worries. Again, maybe that isn't true for other types of decorators this feature might be useful for, but without having any examples to think about, it's hard to guess... > class PropertyMetaclass(type): > def __new__(cls, name, bases, attrs): > return property(attrs.get('get'), attrs.get('set'), attrs.get('del'), attrs.get('__doc__')) I still don't get the benefit of having a metaclass or constructor function or wrapper function or anything else, instead of just making property take a class instead of four functions. The latter is significantly nicer on the user side, and only a tiny bit more verbose in the implementation of property, and easier to understand. Unless there are other decorators where they wouldn't be true, or so many potentially useful one-shot decorators that defining them all a little more succinctly is worth the cost, why add the extra layer? From rosuav at gmail.com Mon Nov 30 23:38:26 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 1 Dec 2015 15:38:26 +1100 Subject: [Python-ideas] Multiple arguments for decorators In-Reply-To: <650A2120-A263-4568-8C4D-D1B4241B9382@yahoo.com> References: <1745702014.10551218.1448938331185.JavaMail.yahoo@mail.yahoo.com> <650A2120-A263-4568-8C4D-D1B4241B9382@yahoo.com> Message-ID: On Tue, Dec 1, 2015 at 3:30 PM, Andrew Barnert wrote: > Comprehensions define functions, so it's the same rule; you just have to know that functions can be defined three ways (def statement, lambdas expression, or comprehension) rather than just two. Sure, that's not _ideally_ simple, but that hardly seems a reason to make it even _less_ simple. > Which I don't really like. It makes sense for generator expressions, but list comprehensions don't *look* like they introduce a new sub-scope. A 'with' block does look like it creates a new scope, but its binding leaks out (usually giving you a closed file object or something). Sure, I can understand why things are the way they are, but it's not intuitive. You have to dig into things a bit to grok it. > Most importantly, comprehensions don't have a suite that looks exactly the same as another kind of suite introduced by the same keyword that doesn't define a scope, except on this one special case where it does. > Agreed. And while I called the class-based system "ugly" to start with, I'm coming around to it more and more - especially since it works in current versions of Python, rather than demanding core interpreter changes. It's not the most intuitive use of syntax either (the 'class' block isn't really creating a class at all - it creates a function parameter list), but it isn't as bad as I thought it was. ChrisA From abarnert at yahoo.com Mon Nov 30 23:47:09 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 30 Nov 2015 20:47:09 -0800 Subject: [Python-ideas] Multiple arguments for decorators In-Reply-To: References: Message-ID: Tangent here: Is this potentially a good use case for Nick Coghlan's PEP 403 (@in clauses) and/or PEP 3150 (statement-local namespaces with "given:")? At first glance, the obvious way to use PEP 3150 doesn't look beautiful: class Spam: x = property(**?.__dict__) given: """x""" def fget(self): return self._x def fset(self, value): self._x = value def fdel(self): del self._x But maybe there's a way to make this nicer? > On Nov 30, 2015, at 20:37, Andrew Barnert via Python-ideas wrote: > >> On Nov 30, 2015, at 19:21, Kevin Modzelewski via Python-ideas wrote: >> >> Class scopes definitely feel like a good match -- they are a way of saying "evaluate all of these expression, pass the resulting locals to a custom function, and bind the result of that function to the classname". Usually the function is type(), which constructs a new class, but by setting a custom metaclass we can avoid creating a class just to wrap the scope: > > Is there really a harm in creating a class? > > A property is a type, and the obvious way to simulate it in Python rather than C (as shown by the sample code in the HOWTO) is with a class statement. > > Besides, if you're creating many thousands of properties in a loop, the time and space cost of property creation is probably the least of your worries. > > Again, maybe that isn't true for other types of decorators this feature might be useful for, but without having any examples to think about, it's hard to guess... > >> class PropertyMetaclass(type): >> def __new__(cls, name, bases, attrs): >> return property(attrs.get('get'), attrs.get('set'), attrs.get('del'), attrs.get('__doc__')) > > I still don't get the benefit of having a metaclass or constructor function or wrapper function or anything else, instead of just making property take a class instead of four functions. The latter is significantly nicer on the user side, and only a tiny bit more verbose in the implementation of property, and easier to understand. Unless there are other decorators where they wouldn't be true, or so many potentially useful one-shot decorators that defining them all a little more succinctly is worth the cost, why add the extra layer? > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/