From steve at pearwood.info Sun Nov 1 01:06:30 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 1 Nov 2015 17:06:30 +1100 Subject: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen In-Reply-To: References: <201510291559.t9TFxi4i003479@fido.openend.se> Message-ID: <20151101060629.GE10946@ando.pearwood.info> CC'ing Python-Ideas. Follow-ups to Python-Ideas please. On Thu, Oct 29, 2015 at 09:22:15PM -0400, Terry Reedy wrote: > Leaving IDLE aside, the reason '' is added to sys.path is so that people > can import their own modules. This is very useful. Shadowing is the > result of putting it at the front. I have long thought this a dubious > choice. If '' were instead appended, people could still import modules > that did not duplicate stdlib names. Anyone who wanted shadowing could > move '' to the front. But then shadowing would be intentional, not an > accident. Terry is right. Shadowing should be possible, and it should require a deliberate decision on the part of the programmer. Consider the shell, say, bash or similar. My understanding is that the shell PATH deliberately excludes the current directory because of the possibility of malicious software shadowing usual commands in /bin etc. If you want to run an executable in the current directory, you have to explicitly provide the path to it: ./myscript rather than just myscript. Now Python isn't exactly the shell, and so I'm not proposing that Python does the same thing. But surely we can agree on the following? - Shadowing explicitly installed packages, including the stdlib, is *occasionally* useful. - But when shadowing occurs, it is *nearly always* accidental. - Such accidental shadowing often causes problems. - And further more, debugging shadowing problems is sometimes tricky even for experienced coders, and almost impossible for beginners. (It's not until you've been burned once or thrice by shadowing that you recognise the symptoms, at which point it is then usually easy to debug.) - Hence, we should put the onus on those who want to shadow installed packages) to do so *explicitly*, or at least make it easier to avoid accidental shadowing. I propose the following two changes: (1) Beginning with Python 3.6, the default is that the current directory is put at the end of sys.path rather than the beginning. Instead of: >>> print(sys.path) ['', '/this', '/that', '/another'] we will have this instead: >>> print(sys.path) ['/this', '/that', '/another', ''] Those who don't shadow installed packages won't notice any difference. Scripts which deliberately or unintentionally shadow installed packages will break from this change. I don't have a problem with this. You can't fix harmful behaviour without breaking code that depends on that harmful behaviour. Additionally, I expect that those who rely on the current behaviour will be in a small minority, much fewer than those who will be bitten by accidental shadowing into the indefinite future. And if you want the old behaviour back, it is easy to do so, by changing the path before doing your imports: import sys if sys.path[-1] == "": sys.path = [""] + sys.path[:-1] or equivalent. I do not belive that it is onerous for those who want shadowing to have to take steps to do so explicitly. That can be added to your scripts on a case-by-case basis, or your PYTHONSTARTUP file, by modifying your site.py, or (I think) by putting the code into the sitecustomize or usercustomize modules. (2) IDLE doesn't need to wait for Python 3.6 to make this change. I believe that IDLE is permitted to make backwards incompatible changes in minor releases, so there is no reason why it can't change the path effective immediately. That's a simpler fix than scanning the entire path, raising warnings (which beginners won't understand and will either ignore or panic over) or other complex solutions. It may not prevent *every* shadowing incident, but it will improve the situation immeasurably. Thoughts? -- Steve From storchaka at gmail.com Sun Nov 1 01:29:32 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 1 Nov 2015 08:29:32 +0200 Subject: [Python-Dev] Unable to submit a patch to the tracker Message-ID: I'm unable to submit any file to any issue, neither via web-form nor via e-mail. Checked with different browsers from different computers. Meta-tracker doesn't work too. http://psf.upfronthosting.co.za/roundup/meta/issue575 From lac at openend.se Sun Nov 1 02:40:36 2015 From: lac at openend.se (Laura Creighton) Date: Sun, 01 Nov 2015 08:40:36 +0100 Subject: [Python-Dev] Unable to submit a patch to the tracker In-Reply-To: References: Message-ID: <201511010740.tA17easo024848@fido.openend.se> In a message of Sun, 01 Nov 2015 08:29:32 +0200, Serhiy Storchaka writes: >I'm unable to submit any file to any issue, neither via web-form nor via >e-mail. Checked with different browsers from different computers. >Meta-tracker doesn't work too. > >http://psf.upfronthosting.co.za/roundup/meta/issue575 It's working for me. Laura From rosuav at gmail.com Sun Nov 1 05:30:40 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 1 Nov 2015 21:30:40 +1100 Subject: [Python-Dev] Python 3.5.1 plans Message-ID: PEP 478 [1] doesn't currently have any info on a planned 3.5.1 release (and actually, it has 3.5.0 Final listed as a future release). About when is it likely to happen? The one thing I'm hanging out for is an installer patch on Windows that detects XP and immediately aborts with a convenient error; if the number of emails to python-list is indicative, there are a lot of people out there getting confused. ChrisA [1] https://www.python.org/dev/peps/pep-0478/ From phil at riverbankcomputing.com Sun Nov 1 05:55:43 2015 From: phil at riverbankcomputing.com (Phil Thompson) Date: Sun, 1 Nov 2015 10:55:43 +0000 Subject: [Python-Dev] Python 3.5.1 plans In-Reply-To: References: Message-ID: On 1 Nov 2015, at 10:30 a.m., Chris Angelico wrote: > > PEP 478 [1] doesn't currently have any info on a planned 3.5.1 release > (and actually, it has 3.5.0 Final listed as a future release). About > when is it likely to happen? The one thing I'm hanging out for is an > installer patch on Windows that detects XP and immediately aborts with > a convenient error; if the number of emails to python-list is > indicative, there are a lot of people out there getting confused. That doesn't need a new version of Python, just a new installer. Phil From rosuav at gmail.com Sun Nov 1 06:01:54 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 1 Nov 2015 22:01:54 +1100 Subject: [Python-Dev] Python 3.5.1 plans In-Reply-To: References: Message-ID: On Sun, Nov 1, 2015 at 9:55 PM, Phil Thompson wrote: > On 1 Nov 2015, at 10:30 a.m., Chris Angelico wrote: >> >> PEP 478 [1] doesn't currently have any info on a planned 3.5.1 release >> (and actually, it has 3.5.0 Final listed as a future release). About >> when is it likely to happen? The one thing I'm hanging out for is an >> installer patch on Windows that detects XP and immediately aborts with >> a convenient error; if the number of emails to python-list is >> indicative, there are a lot of people out there getting confused. > > That doesn't need a new version of Python, just a new installer. True, it doesn't, but it needs that installer to have as much checking as a new point release will get. My understanding was that it was going to be done as part of 3.5.1. ChrisA From steve.dower at python.org Sun Nov 1 08:52:40 2015 From: steve.dower at python.org (Steve Dower) Date: Sun, 1 Nov 2015 05:52:40 -0800 Subject: [Python-Dev] Python 3.5.1 plans In-Reply-To: References: Message-ID: The installer and the contained contents are currently tied together, making it fairly difficult to mix and match versions. When 3.5.1 happens is up to Larry, but I'm feeling like the initial rush of bug reports has died down and we should be considering the remaining open ones in the context of blocking that release. (The XP notification, as well as the more confusing Vista and Win7 w/o service packs notifications, have been added already. Python-list will probably keep asking though, just in case it can get something for free...) Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Chris Angelico" Sent: ?11/?1/?2015 3:03 To: "Phil Thompson" Cc: "python-dev" Subject: Re: [Python-Dev] Python 3.5.1 plans On Sun, Nov 1, 2015 at 9:55 PM, Phil Thompson wrote: > On 1 Nov 2015, at 10:30 a.m., Chris Angelico wrote: >> >> PEP 478 [1] doesn't currently have any info on a planned 3.5.1 release >> (and actually, it has 3.5.0 Final listed as a future release). About >> when is it likely to happen? The one thing I'm hanging out for is an >> installer patch on Windows that detects XP and immediately aborts with >> a convenient error; if the number of emails to python-list is >> indicative, there are a lot of people out there getting confused. > > That doesn't need a new version of Python, just a new installer. True, it doesn't, but it needs that installer to have as much checking as a new point release will get. My understanding was that it was going to be done as part of 3.5.1. ChrisA _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From lac at openend.se Sun Nov 1 09:21:12 2015 From: lac at openend.se (Laura Creighton) Date: Sun, 01 Nov 2015 15:21:12 +0100 Subject: [Python-Dev] Python 3.5.1 plans In-Reply-To: References: Message-ID: <201511011421.tA1ELCTq021979@fido.openend.se> In a message of Sun, 01 Nov 2015 05:52:40 -0800, Steve Dower writes: >The installer and the contained contents are currently tied together, making it fairly difficult to mix and match versions. > >When 3.5.1 happens is up to Larry, but I'm feeling like the initial rush of bug reports has died down and we should be considering the remaining open ones in the context of blocking that release. (The XP notification, as well as the more confusing Vista and Win7 w/o service packs notifications, have been added already. Python-list will probably keep asking though, just in case it can get something for free...) > >Cheers, >Steve Webmaster here. The initial rush of bug reports I see has not gone down. It's been in steady increase. This is not a data-blip due to last weekend's whole outage of mail.python.org. Getting the 'XP -- you are out of luck' message and the api-ms-win-crt-runtime-l1-1-0-dll is Microsofts Universal CRT. You don't have one. You need to install it. Get it here: http://www.microsoft.com/en-us/download/details.aspx?id=48234 message out of a failed install -- as soon as possible -- would have a serious positive impact around here. Most of the replies I make are of that sort. I have 2 templates to make it quick to do, but still, be nice if I could save my time for others. And I don't report the number of people I talk to with this problem to the bug tracker. This could possibly have mislead you. It is not like the: The 3.5 Windows installer fails with "The TARGETDIR variable must be provided when invoking this installer" bug https://bugs.python.org/issue25144 where I report when I hear about them if I think there is a good chance that having me report it will cause the person who submitted the bug to me to go track it -- because maybe asking him or her to give you more information about their system will help you resolve it. For the first 2 sorts, well, I won't spam you with the knowledge that windows xp users are complaining to webmaster a whole lot about 3.5 not working. But from my end -- sooner python.org serves up the improved installer, the better. Even if you made a release-candidate-right-now to do nothing but that. Laura From storchaka at gmail.com Sun Nov 1 10:50:51 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 1 Nov 2015 17:50:51 +0200 Subject: [Python-Dev] Unable to submit a patch to the tracker In-Reply-To: References: Message-ID: On 01.11.15 08:29, Serhiy Storchaka wrote: > I'm unable to submit any file to any issue, neither via web-form nor via > e-mail. Checked with different browsers from different computers. > Meta-tracker doesn't work too. > > http://psf.upfronthosting.co.za/roundup/meta/issue575 Sorry for the noise. The cause was in my mistake. From steve.dower at python.org Sun Nov 1 11:23:48 2015 From: steve.dower at python.org (Steve Dower) Date: Sun, 1 Nov 2015 08:23:48 -0800 Subject: [Python-Dev] Python 3.5.1 plans In-Reply-To: <201511011421.tA1ELCTq021979@fido.openend.se> References: <201511011421.tA1ELCTq021979@fido.openend.se> Message-ID: "The initial rush of bug reports I see has not gone down. It's been in steady increase." To clarify, I meant unique bugs. When my job is to fix them, more reports of the same bug are not important to me, especially once they've been fixed but not released. I do understand that someone has to deal with every report, new or not (for my work project, that's me), and you are appreciated for it especially since it's not your role. You should probably just have one template redirecting people to python-list though. Top-posted from my Windows Phone -----Original Message----- From: "Laura Creighton" Sent: ?11/?1/?2015 6:22 To: "Steve Dower" Cc: "Chris Angelico" ; "Phil Thompson" ; "python-dev" ; "lac at openend.se" Subject: Re: [Python-Dev] Python 3.5.1 plans In a message of Sun, 01 Nov 2015 05:52:40 -0800, Steve Dower writes: >The installer and the contained contents are currently tied together, making it fairly difficult to mix and match versions. > >When 3.5.1 happens is up to Larry, but I'm feeling like the initial rush of bug reports has died down and we should be considering the remaining open ones in the context of blocking that release. (The XP notification, as well as the more confusing Vista and Win7 w/o service packs notifications, have been added already. Python-list will probably keep asking though, just in case it can get something for free...) > >Cheers, >Steve Webmaster here. The initial rush of bug reports I see has not gone down. It's been in steady increase. This is not a data-blip due to last weekend's whole outage of mail.python.org. Getting the 'XP -- you are out of luck' message and the api-ms-win-crt-runtime-l1-1-0-dll is Microsofts Universal CRT. You don't have one. You need to install it. Get it here: http://www.microsoft.com/en-us/download/details.aspx?id=48234 message out of a failed install -- as soon as possible -- would have a serious positive impact around here. Most of the replies I make are of that sort. I have 2 templates to make it quick to do, but still, be nice if I could save my time for others. And I don't report the number of people I talk to with this problem to the bug tracker. This could possibly have mislead you. It is not like the: The 3.5 Windows installer fails with "The TARGETDIR variable must be provided when invoking this installer" bug https://bugs.python.org/issue25144 where I report when I hear about them if I think there is a good chance that having me report it will cause the person who submitted the bug to me to go track it -- because maybe asking him or her to give you more information about their system will help you resolve it. For the first 2 sorts, well, I won't spam you with the knowledge that windows xp users are complaining to webmaster a whole lot about 3.5 not working. But from my end -- sooner python.org serves up the improved installer, the better. Even if you made a release-candidate-right-now to do nothing but that. Laura -------------- next part -------------- An HTML attachment was scrubbed... URL: From lac at openend.se Sun Nov 1 12:10:30 2015 From: lac at openend.se (Laura Creighton) Date: Sun, 01 Nov 2015 18:10:30 +0100 Subject: [Python-Dev] Python 3.5.1 plans In-Reply-To: References: <201511011421.tA1ELCTq021979@fido.openend.se> Message-ID: <201511011710.tA1HAULq002365@fido.openend.se> In a message of Sun, 01 Nov 2015 08:23:48 -0800, Steve Dower writes: >"The initial rush of bug reports I see has not gone down. It's been in steady increase." > >To clarify, I meant unique bugs. When my job is to fix them, more reports of the same bug are not important to me, especially once they've been fixed but not released. > >I do understand that someone has to deal with every report, new or not (for my work project, that's me), and you are appreciated for it especially since it's not your role. You should probably just have one template redirecting people to python-list though. > >Top-posted from my Windows Phone sending them to python-list is not a good idea. people get bored at the same old bugs and by now bored python-listers will reply something sarcastic at people with the same old bugs. Especialy if they post there not saying what windows version they have. There is only so much --'be kinder, it may be the 35th time for us, it is still that person's first bug report with the same old same old' I can do. The chief reason I am subscribed to python-list is so that newbies are not greeted with such sarcasm alone. A better experience from the download page should not wait for Larry's other concerns. If the new one is ready to go, and tells XP users to get a new OS and universal-crt missers to go get one here Put that on python.org as soon as possible. even if you need to bump the python 3.5 numbering. you get 3.5.1 for this, and this alone and everybody else gets to wait for 3.5.2 and when Larry feels this is a good idea. Laura From larry at hastings.org Sun Nov 1 13:51:05 2015 From: larry at hastings.org (Larry Hastings) Date: Sun, 1 Nov 2015 10:51:05 -0800 Subject: [Python-Dev] Python 3.5.1 plans In-Reply-To: <201511011710.tA1HAULq002365@fido.openend.se> References: <201511011421.tA1ELCTq021979@fido.openend.se> <201511011710.tA1HAULq002365@fido.openend.se> Message-ID: <56365F19.1060008@hastings.org> On 11/01/2015 09:10 AM, Laura Creighton wrote: > Put that on python.org as soon as possible. > even if you need to bump the python 3.5 numbering. > you get 3.5.1 for this, and this alone and everybody else gets to > wait for 3.5.2 and when Larry feels this is a good idea. Well, let me say this. I haven't been added to any apocalyptic 3.5 bugs recently. If Steve says he's not seeing novel bugs, and Laura says she's seeing a steady trickle of people reporting the same bugs that have already been fixed, then it seems like a good time to cut 3.5.1. 3.5 does have a single issue on the tracker marked as a "release blocker": smtplib.py AUTH LOGIN code messed up sending login and password data since 3.5 http://bugs.python.org/issue25446 R. David Murray set it to "release blocker", and I trust his judgment, so 3.5.1 can't go out until this is fixed. I'll engage with him on the issue to see about when we can get a fix. If I didn't have any release blockers, I'd schedule 3.5.1 for sometime around three weeks from now. And FYI I want to release 3.4.4 about two weeks after that. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From lac at openend.se Sun Nov 1 16:28:52 2015 From: lac at openend.se (Laura Creighton) Date: Sun, 01 Nov 2015 22:28:52 +0100 Subject: [Python-Dev] Python 3.5.1 plans In-Reply-To: <56365F19.1060008@hastings.org> References: <201511011421.tA1ELCTq021979@fido.openend.se> <201511011710.tA1HAULq002365@fido.openend.se> <56365F19.1060008@hastings.org> Message-ID: <201511012128.tA1LSqD0021706@fido.openend.se> In a message of Sun, 01 Nov 2015 10:51:05 -0800, Larry Hastings writes: > > >On 11/01/2015 09:10 AM, Laura Creighton wrote: >> Put that on python.org as soon as possible. >> even if you need to bump the python 3.5 numbering. >> you get 3.5.1 for this, and this alone and everybody else gets to >> wait for 3.5.2 and when Larry feels this is a good idea. > >Well, let me say this. I haven't been added to any apocalyptic 3.5 bugs >recently. If Steve says he's not seeing novel bugs, and Laura says >she's seeing a steady trickle of people reporting the same bugs that >have already been fixed, then it seems like a good time to cut 3.5.1. > >3.5 does have a single issue on the tracker marked as a "release blocker": > > smtplib.py AUTH LOGIN code messed up sending login and password data > since 3.5 > > http://bugs.python.org/issue25446 > >R. David Murray set it to "release blocker", and I trust his judgment, >so 3.5.1 can't go out until this is fixed. I'll engage with him on the >issue to see about when we can get a fix. > >If I didn't have any release blockers, I'd schedule 3.5.1 for sometime >around three weeks from now. And FYI I want to release 3.4.4 about two >weeks after that. > > >//arry/ Sounds wonderful to me. Thank you Larry. Laura From rosuav at gmail.com Sun Nov 1 16:46:43 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 Nov 2015 08:46:43 +1100 Subject: [Python-Dev] Python 3.5.1 plans In-Reply-To: <56365F19.1060008@hastings.org> References: <201511011421.tA1ELCTq021979@fido.openend.se> <201511011710.tA1HAULq002365@fido.openend.se> <56365F19.1060008@hastings.org> Message-ID: On Mon, Nov 2, 2015 at 5:51 AM, Larry Hastings wrote: > If I didn't have any release blockers, I'd schedule 3.5.1 for sometime > around three weeks from now. And FYI I want to release 3.4.4 about two > weeks after that. > Thanks Larry! That'd be awesome. If there's testing needed on that ticket, or anything, I'll help out. ChrisA From meadori at gmail.com Sun Nov 1 17:00:49 2015 From: meadori at gmail.com (Meador Inge) Date: Sun, 1 Nov 2015 16:00:49 -0600 Subject: [Python-Dev] Generated Bytecode ... In-Reply-To: References: Message-ID: On Fri, Oct 23, 2015 at 3:23 AM, Victor Stinner wrote: > Note: I propose "noopt" because we already have "optimization level 0" > which still uses optimizations, it's the default mode. It's different > than gcc -O0 which really disables all optimizations. I already prefix > the "noopt" suffix for .pyc files than "opt--1" proposed by Brett in > http://bugs.python.org/issue2506 Strictly speaking, this isn't completely true. Most (all?) C/C++ compilers do some level of optimization when they are "disabled". GCC/Clang, for example, both will do some form of constant folding with -O0. The compilers are just *much* less aggressive with the primary goal of making the debugging experience much more enjoyable. The situation here sounds similar -- we want an option to provide a better debugging and tracing experience, which seems like a very reasonable idea to me. -- Meador From benjamin at python.org Sun Nov 1 17:42:56 2015 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 01 Nov 2015 14:42:56 -0800 Subject: [Python-Dev] 2.7.11 plans Message-ID: <1446417776.2942029.426137689.4B30F1D4@webmail.messagingengine.com> Happy November, everyone. It?s nearly time for the next semi-annual instalment of the 2.7 series. I?m planning to release a 2.7.11 release candidate on November 21st and 2.7.11 final on December 5. More than half-done releasing 2.7-ly yours, Benjamin From rymg19 at gmail.com Sun Nov 1 18:26:27 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Sun, 1 Nov 2015 17:26:27 -0600 Subject: [Python-Dev] 2.7.11 plans In-Reply-To: <1446417776.2942029.426137689.4B30F1D4@webmail.messagingengine.com> References: <1446417776.2942029.426137689.4B30F1D4@webmail.messagingengine.com> Message-ID: On Sun, Nov 1, 2015 at 4:42 PM, Benjamin Peterson wrote: > Happy November, everyone. It?s nearly time for the next semi-annual > instalment of the 2.7 series. I?m planning to release a 2.7.11 release > candidate on November 21st and 2.7.11 final on December 5. > > More than half-done releasing 2.7-ly yours, > This made me laugh! > Benjamin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Nov 4 03:50:33 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 4 Nov 2015 09:50:33 +0100 Subject: [Python-Dev] Second milestone of FAT Python Message-ID: Hi, I'm writing a new "FAT Python" project to try to implement optimizations in CPython (inlining, constant folding, move invariants out of loops, etc.) using a "static" optimizer (not a JIT). For the background, see the thread on python-ideas: https://mail.python.org/pipermail/python-ideas/2015-October/036908.html See also the documentation: https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst I implemented the most basic optimization to test my code: replace calls to builtin functions (with constant arguments) with the result. For example, len("abc") is replaced with 3. I reached the second milestone: it's now possible to run the full Python test suite with these optimizations enabled. It confirms that the optimizations don't break the Python semantic. Example: --- >>> def func(): ... return len("abc") ... >>> import dis >>> dis.dis(func) 2 0 LOAD_GLOBAL 0 (len) 3 LOAD_CONST 1 ('abc') 6 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 9 RETURN_VALUE >>> len(func.get_specialized()) 1 >>> specialized=func.get_specialized()[0] >>> dis.dis(specialized['code']) 2 0 LOAD_CONST 1 (3) 3 RETURN_VALUE >>> len(specialized['guards']) 2 >>> func() 3 >>> len=lambda obj: "mock" >>> func() 'mock' >>> func.get_specialized() [] --- The function func() has specialized bytecode which returns directly 3 instead of calling len("abc"). The specialized bytecode has two guards dictionary keys: builtins.__dict__['len'] and globals()['len']. If one of these keys is modified, the specialized bytecode is simply removed (when the function is called) and the original bytecode is executed. You cannot expect any speedup at this milestone, it's just to validate the implementation. You can only get speedup if you implement *manually* optimizations. See for example posixpath.isabs() which inlines manually the call to the _get_sep() function. More optimizations will be implemented in the third milestone. I don't know yet if I will be able to implement constant folding, function inlining and/or moving invariants out of loops. Download, compile and test FAT Python with: hg clone http://hg.python.org/sandbox/fatpython ./configure && make && ./python -m test test_astoptimizer test_fat Currently, only 24 functions are specialized in the standard library. Calling a builtin function with constant arguments in not common (it was expected, it's only the first step for my optimizer). But 161 functions are specialized in tests. To be honest, I had to modify some tests to make them pass in FAT mode. But most changes are related to the .pyc filename, or to the exact size in bytes of dictionary objects. FAT Python is still experimental. Currently, the main bug is that the AST optimizer can optimize a call to a function which is not the expected builtin function. I already started to implement code to understand namespaces (detect global and local variables), but it's not enough yet to detect when a builtin is overriden. See TODO.rst for known bugs and limitations. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Wed Nov 4 04:07:41 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 4 Nov 2015 09:07:41 +0000 Subject: [Python-Dev] Second milestone of FAT Python In-Reply-To: References: Message-ID: How do you check that someone did not e.g. bind something different to "len"? On Wed, Nov 4, 2015 at 8:50 AM, Victor Stinner wrote: > Hi, > > I'm writing a new "FAT Python" project to try to implement optimizations in > CPython (inlining, constant folding, move invariants out of loops, etc.) > using a "static" optimizer (not a JIT). For the background, see the thread > on python-ideas: > https://mail.python.org/pipermail/python-ideas/2015-October/036908.html > > See also the documentation: > https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst > https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst > > I implemented the most basic optimization to test my code: replace calls to > builtin functions (with constant arguments) with the result. For example, > len("abc") is replaced with 3. I reached the second milestone: it's now > possible to run the full Python test suite with these optimizations enabled. > It confirms that the optimizations don't break the Python semantic. > > Example: > --- >>>> def func(): > ... return len("abc") > ... >>>> import dis >>>> dis.dis(func) > 2 0 LOAD_GLOBAL 0 (len) > 3 LOAD_CONST 1 ('abc') > 6 CALL_FUNCTION 1 (1 positional, 0 keyword pair) > 9 RETURN_VALUE > >>>> len(func.get_specialized()) > 1 >>>> specialized=func.get_specialized()[0] >>>> dis.dis(specialized['code']) > 2 0 LOAD_CONST 1 (3) > 3 RETURN_VALUE >>>> len(specialized['guards']) > 2 > >>>> func() > 3 > >>>> len=lambda obj: "mock" >>>> func() > 'mock' >>>> func.get_specialized() > [] > --- > > The function func() has specialized bytecode which returns directly 3 > instead of calling len("abc"). The specialized bytecode has two guards > dictionary keys: builtins.__dict__['len'] and globals()['len']. If one of > these keys is modified, the specialized bytecode is simply removed (when the > function is called) and the original bytecode is executed. > > > You cannot expect any speedup at this milestone, it's just to validate the > implementation. You can only get speedup if you implement *manually* > optimizations. See for example posixpath.isabs() which inlines manually the > call to the _get_sep() function. More optimizations will be implemented in > the third milestone. I don't know yet if I will be able to implement > constant folding, function inlining and/or moving invariants out of loops. > > > Download, compile and test FAT Python with: > > hg clone http://hg.python.org/sandbox/fatpython > ./configure && make && ./python -m test test_astoptimizer test_fat > > > Currently, only 24 functions are specialized in the standard library. > Calling a builtin function with constant arguments in not common (it was > expected, it's only the first step for my optimizer). But 161 functions are > specialized in tests. > > > To be honest, I had to modify some tests to make them pass in FAT mode. But > most changes are related to the .pyc filename, or to the exact size in bytes > of dictionary objects. > > FAT Python is still experimental. Currently, the main bug is that the AST > optimizer can optimize a call to a function which is not the expected > builtin function. I already started to implement code to understand > namespaces (detect global and local variables), but it's not enough yet to > detect when a builtin is overriden. See TODO.rst for known bugs and > limitations. > > Victor > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > From fijall at gmail.com Wed Nov 4 04:08:17 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 4 Nov 2015 09:08:17 +0000 Subject: [Python-Dev] Second milestone of FAT Python In-Reply-To: References: Message-ID: Uh, sorry, misread your full mail, scratch that On Wed, Nov 4, 2015 at 9:07 AM, Maciej Fijalkowski wrote: > How do you check that someone did not e.g. bind something different to "len"? > > On Wed, Nov 4, 2015 at 8:50 AM, Victor Stinner wrote: >> Hi, >> >> I'm writing a new "FAT Python" project to try to implement optimizations in >> CPython (inlining, constant folding, move invariants out of loops, etc.) >> using a "static" optimizer (not a JIT). For the background, see the thread >> on python-ideas: >> https://mail.python.org/pipermail/python-ideas/2015-October/036908.html >> >> See also the documentation: >> https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst >> https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst >> >> I implemented the most basic optimization to test my code: replace calls to >> builtin functions (with constant arguments) with the result. For example, >> len("abc") is replaced with 3. I reached the second milestone: it's now >> possible to run the full Python test suite with these optimizations enabled. >> It confirms that the optimizations don't break the Python semantic. >> >> Example: >> --- >>>>> def func(): >> ... return len("abc") >> ... >>>>> import dis >>>>> dis.dis(func) >> 2 0 LOAD_GLOBAL 0 (len) >> 3 LOAD_CONST 1 ('abc') >> 6 CALL_FUNCTION 1 (1 positional, 0 keyword pair) >> 9 RETURN_VALUE >> >>>>> len(func.get_specialized()) >> 1 >>>>> specialized=func.get_specialized()[0] >>>>> dis.dis(specialized['code']) >> 2 0 LOAD_CONST 1 (3) >> 3 RETURN_VALUE >>>>> len(specialized['guards']) >> 2 >> >>>>> func() >> 3 >> >>>>> len=lambda obj: "mock" >>>>> func() >> 'mock' >>>>> func.get_specialized() >> [] >> --- >> >> The function func() has specialized bytecode which returns directly 3 >> instead of calling len("abc"). The specialized bytecode has two guards >> dictionary keys: builtins.__dict__['len'] and globals()['len']. If one of >> these keys is modified, the specialized bytecode is simply removed (when the >> function is called) and the original bytecode is executed. >> >> >> You cannot expect any speedup at this milestone, it's just to validate the >> implementation. You can only get speedup if you implement *manually* >> optimizations. See for example posixpath.isabs() which inlines manually the >> call to the _get_sep() function. More optimizations will be implemented in >> the third milestone. I don't know yet if I will be able to implement >> constant folding, function inlining and/or moving invariants out of loops. >> >> >> Download, compile and test FAT Python with: >> >> hg clone http://hg.python.org/sandbox/fatpython >> ./configure && make && ./python -m test test_astoptimizer test_fat >> >> >> Currently, only 24 functions are specialized in the standard library. >> Calling a builtin function with constant arguments in not common (it was >> expected, it's only the first step for my optimizer). But 161 functions are >> specialized in tests. >> >> >> To be honest, I had to modify some tests to make them pass in FAT mode. But >> most changes are related to the .pyc filename, or to the exact size in bytes >> of dictionary objects. >> >> FAT Python is still experimental. Currently, the main bug is that the AST >> optimizer can optimize a call to a function which is not the expected >> builtin function. I already started to implement code to understand >> namespaces (detect global and local variables), but it's not enough yet to >> detect when a builtin is overriden. See TODO.rst for known bugs and >> limitations. >> >> Victor >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com >> From steve at pearwood.info Wed Nov 4 05:16:15 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 4 Nov 2015 21:16:15 +1100 Subject: [Python-Dev] Second milestone of FAT Python In-Reply-To: References: Message-ID: <20151104101615.GO10946@ando.pearwood.info> On Wed, Nov 04, 2015 at 09:50:33AM +0100, Victor Stinner wrote: > Hi, > > I'm writing a new "FAT Python" project to try to implement optimizations in > CPython (inlining, constant folding, move invariants out of loops, etc.) > using a "static" optimizer (not a JIT). For the background, see the thread > on python-ideas: > https://mail.python.org/pipermail/python-ideas/2015-October/036908.html Nice work Victor, and thank you for your efforts! -- Steve From srkunze at mail.de Wed Nov 4 15:14:37 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 4 Nov 2015 21:14:37 +0100 Subject: [Python-Dev] Second milestone of FAT Python In-Reply-To: References: Message-ID: <563A672D.6080803@mail.de> Hi Victor, great to hear. I think everybody here appreciates your efforts. Do you think there will be any change of merging this back into CPython? Best, Sven On 04.11.2015 09:50, Victor Stinner wrote: > Hi, > > I'm writing a new "FAT Python" project to try to implement > optimizations in CPython (inlining, constant folding, move invariants > out of loops, etc.) using a "static" optimizer (not a JIT). For the > background, see the thread on python-ideas: > https://mail.python.org/pipermail/python-ideas/2015-October/036908.html > > See also the documentation: > https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst > https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst > > I implemented the most basic optimization to test my code: replace > calls to builtin functions (with constant arguments) with the result. > For example, len("abc") is replaced with 3. I reached the second > milestone: it's now possible to run the full Python test suite with > these optimizations enabled. It confirms that the optimizations don't > break the Python semantic. > > Example: > --- > >>> def func(): > ... return len("abc") > ... > >>> import dis > >>> dis.dis(func) > 2 0 LOAD_GLOBAL 0 (len) > 3 LOAD_CONST 1 ('abc') > 6 CALL_FUNCTION 1 (1 positional, 0 keyword pair) > 9 RETURN_VALUE > > >>> len(func.get_specialized()) > 1 > >>> specialized=func.get_specialized()[0] > >>> dis.dis(specialized['code']) > 2 0 LOAD_CONST 1 (3) > 3 RETURN_VALUE > >>> len(specialized['guards']) > 2 > > >>> func() > 3 > > >>> len=lambda obj: "mock" > >>> func() > 'mock' > >>> func.get_specialized() > [] > --- > > The function func() has specialized bytecode which returns directly 3 > instead of calling len("abc"). The specialized bytecode has two guards > dictionary keys: builtins.__dict__['len'] and globals()['len']. If one > of these keys is modified, the specialized bytecode is simply removed > (when the function is called) and the original bytecode is executed. > > > You cannot expect any speedup at this milestone, it's just to validate > the implementation. You can only get speedup if you implement > *manually* optimizations. See for example posixpath.isabs() which > inlines manually the call to the _get_sep() function. More > optimizations will be implemented in the third milestone. I don't know > yet if I will be able to implement constant folding, function inlining > and/or moving invariants out of loops. > > > Download, compile and test FAT Python with: > > hg clone http://hg.python.org/sandbox/fatpython > ./configure && make && ./python -m test test_astoptimizer test_fat > > > Currently, only 24 functions are specialized in the standard library. > Calling a builtin function with constant arguments in not common (it > was expected, it's only the first step for my optimizer). But 161 > functions are specialized in tests. > > > To be honest, I had to modify some tests to make them pass in FAT > mode. But most changes are related to the .pyc filename, or to the > exact size in bytes of dictionary objects. > > FAT Python is still experimental. Currently, the main bug is that the > AST optimizer can optimize a call to a function which is not the > expected builtin function. I already started to implement code to > understand namespaces (detect global and local variables), but it's > not enough yet to detect when a builtin is overriden. See TODO.rst for > known bugs and limitations. > > Victor > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed Nov 4 15:34:24 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 4 Nov 2015 21:34:24 +0100 Subject: [Python-Dev] Second milestone of FAT Python In-Reply-To: <563A672D.6080803@mail.de> References: <563A672D.6080803@mail.de> Message-ID: <563A6BD0.805@mail.de> typo: "chance" instead of "change" On 04.11.2015 21:14, Sven R. Kunze wrote: > Hi Victor, > > great to hear. I think everybody here appreciates your efforts. > > Do you think there will be any change of merging this back into CPython? > > Best, > Sven > > On 04.11.2015 09:50, Victor Stinner wrote: >> Hi, >> >> I'm writing a new "FAT Python" project to try to implement >> optimizations in CPython (inlining, constant folding, move invariants >> out of loops, etc.) using a "static" optimizer (not a JIT). For the >> background, see the thread on python-ideas: >> https://mail.python.org/pipermail/python-ideas/2015-October/036908.html >> >> See also the documentation: >> https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst >> https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst >> >> I implemented the most basic optimization to test my code: replace >> calls to builtin functions (with constant arguments) with the result. >> For example, len("abc") is replaced with 3. I reached the second >> milestone: it's now possible to run the full Python test suite with >> these optimizations enabled. It confirms that the optimizations don't >> break the Python semantic. >> >> Example: >> --- >> >>> def func(): >> ... return len("abc") >> ... >> >>> import dis >> >>> dis.dis(func) >> 2 0 LOAD_GLOBAL 0 (len) >> 3 LOAD_CONST 1 ('abc') >> 6 CALL_FUNCTION 1 (1 positional, 0 keyword pair) >> 9 RETURN_VALUE >> >> >>> len(func.get_specialized()) >> 1 >> >>> specialized=func.get_specialized()[0] >> >>> dis.dis(specialized['code']) >> 2 0 LOAD_CONST 1 (3) >> 3 RETURN_VALUE >> >>> len(specialized['guards']) >> 2 >> >> >>> func() >> 3 >> >> >>> len=lambda obj: "mock" >> >>> func() >> 'mock' >> >>> func.get_specialized() >> [] >> --- >> >> The function func() has specialized bytecode which returns directly 3 >> instead of calling len("abc"). The specialized bytecode has two >> guards dictionary keys: builtins.__dict__['len'] and >> globals()['len']. If one of these keys is modified, the specialized >> bytecode is simply removed (when the function is called) and the >> original bytecode is executed. >> >> >> You cannot expect any speedup at this milestone, it's just to >> validate the implementation. You can only get speedup if you >> implement *manually* optimizations. See for example posixpath.isabs() >> which inlines manually the call to the _get_sep() function. More >> optimizations will be implemented in the third milestone. I don't >> know yet if I will be able to implement constant folding, function >> inlining and/or moving invariants out of loops. >> >> >> Download, compile and test FAT Python with: >> >> hg clone http://hg.python.org/sandbox/fatpython >> ./configure && make && ./python -m test test_astoptimizer test_fat >> >> >> Currently, only 24 functions are specialized in the standard library. >> Calling a builtin function with constant arguments in not common (it >> was expected, it's only the first step for my optimizer). But 161 >> functions are specialized in tests. >> >> >> To be honest, I had to modify some tests to make them pass in FAT >> mode. But most changes are related to the .pyc filename, or to the >> exact size in bytes of dictionary objects. >> >> FAT Python is still experimental. Currently, the main bug is that the >> AST optimizer can optimize a call to a function which is not the >> expected builtin function. I already started to implement code to >> understand namespaces (detect global and local variables), but it's >> not enough yet to detect when a builtin is overriden. See TODO.rst >> for known bugs and limitations. >> >> Victor >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe:https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Nov 4 17:20:49 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 4 Nov 2015 17:20:49 -0500 Subject: [Python-Dev] Second milestone of FAT Python In-Reply-To: References: Message-ID: On 11/4/2015 3:50 AM, Victor Stinner wrote: > Hi, > > I'm writing a new "FAT Python" project to try to implement optimizations > in CPython (inlining, constant folding, move invariants out of loops, > etc.) using a "static" optimizer (not a JIT). For the background, see > the thread on python-ideas: > https://mail.python.org/pipermail/python-ideas/2015-October/036908.html > > See also the documentation: > https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst > https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst > > I implemented the most basic optimization to test my code: replace calls > to builtin functions (with constant arguments) with the result. For > example, len("abc") is replaced with 3. I reached the second milestone: > it's now possible to run the full Python test suite with these > optimizations enabled. It confirms that the optimizations don't break > the Python semantic. Is the test suite complete enough to say this? (see below) > Example: > --- > >>> def func(): > ... return len("abc") > ... > >>> import dis > >>> dis.dis(func) > 2 0 LOAD_GLOBAL 0 (len) > 3 LOAD_CONST 1 ('abc') > 6 CALL_FUNCTION 1 (1 positional, 0 keyword pair) > 9 RETURN_VALUE > > >>> len(func.get_specialized()) > 1 > >>> specialized=func.get_specialized()[0] > >>> dis.dis(specialized['code']) > 2 0 LOAD_CONST 1 (3) > 3 RETURN_VALUE > >>> len(specialized['guards']) > 2 > > >>> func() > 3 > > >>> len=lambda obj: "mock" > >>> func() > 'mock' In particular, does the test suite have tests like this, to verify that over-riding builtins works? > >>> func.get_specialized() > [] > --- -- Terry Jan Reedy From victor.stinner at gmail.com Wed Nov 4 17:23:34 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 4 Nov 2015 23:23:34 +0100 Subject: [Python-Dev] Second milestone of FAT Python In-Reply-To: <563A672D.6080803@mail.de> References: <563A672D.6080803@mail.de> Message-ID: 2015-11-04 21:14 GMT+01:00 Sven R. Kunze : > Do you think there will be any change of merging this back into CPython? If I understood correctly the first feedback on python-ideas, there were mostly positive, so I believe that it would be possible to merge the FAT mode into CPython. The changes required to implement the FAT mode are restricted to a few files and are small. But I would prefer to keep the whole "FAT mode" optional, to keep exactly the same performances than before for the "standard" mode. I will write a PEP for that, but first I have to ensure that FAT Python works as expected (it's really faster), and that it's generic enough to be used by projects like Numba, Cython, pythran, etc. (but not JIT-based interpreters like PyPy or pyston which already have their own efficient guards.) Victor From mike at selik.org Wed Nov 4 22:59:05 2015 From: mike at selik.org (Michael Selik) Date: Thu, 05 Nov 2015 03:59:05 +0000 Subject: [Python-Dev] Rationale behind lazy map/filter Message-ID: > I'm not suggesting restarting at the top (I've elsewhere suggested that > many such methods would be better as an *iterable* that can be restarted > at the top by calling iter() multiple times, but that's not the same > thing). I'm suggesting raising an exception other than StopIteration, so > that this situation can be detected. If you are writing code that tries > to resume iterating after the iterator has been exhausted, I have to > ask: why? The most obvious case for me would be tailing a file. Loop over the lines in the file, sleep, then do it again. There are many tasks analogous to that scenario -- anything querying a shared resource. -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Wed Nov 4 23:06:32 2015 From: larry at hastings.org (Larry Hastings) Date: Wed, 4 Nov 2015 20:06:32 -0800 Subject: [Python-Dev] Python 3.5.1 release schedule Message-ID: <563AD5C8.7040600@hastings.org> At the request of the platform experts, 3.5.1 is now scheduled to happen simultaneously with 2.7.11. That means: Saturday November 21, 2015 tag 3.5.1rc1 Sunday November 22, 2015 release 3.5.1rc1 Saturday December 5, 2015 tag 3.5.1 final Sunday December 6, 2015 release 3.5.1 final I'll update the release schedule soon, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Nov 5 05:33:56 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 5 Nov 2015 11:33:56 +0100 Subject: [Python-Dev] PYC filename and optimization Message-ID: Hi, While working on the FAT Python project to try to optimize CPython, I have an issue with the cached .pyc files. I would like to have one .pyc without optimization when the FAT mode is disabled (default) and one .pyc with optimization when the FAT mode is enabled (-F command line option). Currently, I'm using this filenames: - Lib/__pycache__/os.cpython-36.pyc: default mode - Lib/__pycache__/os.cpython-36.fat-0.pyc: FAT mode With -O: - Lib/__pycache__/os.cpython-36.opt-1.pyc: default mode - Lib/__pycache__/os.cpython-36.fat-1.pyc: FAT mode With -OO: - Lib/__pycache__/os.cpython-36.opt-2.pyc: default mode - Lib/__pycache__/os.cpython-36.fat-2.pyc: FAT mode Is it the "correct" way to name .pyc files? I had to modify unit tests which expected an exact filename. An alternative would be to add a new dot: - Lib/__pycache__/os.cpython-36.fat.pyc - Lib/__pycache__/os.cpython-36.fat.opt-1.pyc - Lib/__pycache__/os.cpython-36.fat.opt-2.pyc Or should I add a flag to the cache tag ("cpython-36")? - Lib/__pycache__/os.cpython-36f.pyc - Lib/__pycache__/os.cpython-36f.opt-1.pyc - Lib/__pycache__/os.cpython-36f.opt-2.pyc CPython adds the "d" suffix to the cache tag in debug build, but in my case the flag would be added dynamically at Python startup. The FAT mode is enabled by the -F command line option, not by a compilation flag. Victor From victor.stinner at gmail.com Thu Nov 5 09:33:40 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 5 Nov 2015 15:33:40 +0100 Subject: [Python-Dev] Second milestone of FAT Python In-Reply-To: References: Message-ID: Hi, 2015-11-04 23:20 GMT+01:00 Terry Reedy : > On 11/4/2015 3:50 AM, Victor Stinner wrote: >> (...) I reached the second milestone: >> it's now possible to run the full Python test suite with these >> optimizations enabled. It confirms that the optimizations don't break >> the Python semantic. > > Is the test suite complete enough to say this? (see below) > (...) > In particular, does the test suite have tests like this, to verify that > over-riding builtins works? I modified the AST optimizer to not emit guards (remove all guards), tests still pass. So: no, the optimized code doesn't override builtin functions. But I have dedicated unit tests for that in test_fat ;-) Victor From barry at python.org Thu Nov 5 10:40:40 2015 From: barry at python.org (Barry Warsaw) Date: Thu, 5 Nov 2015 09:40:40 -0600 Subject: [Python-Dev] PYC filename and optimization In-Reply-To: References: Message-ID: <20151105094040.6062a7b9@anarchist> On Nov 05, 2015, at 11:33 AM, Victor Stinner wrote: >- Lib/__pycache__/os.cpython-36.pyc: default mode >- Lib/__pycache__/os.cpython-36.fat-0.pyc: FAT mode > >With -O: > >- Lib/__pycache__/os.cpython-36.opt-1.pyc: default mode >- Lib/__pycache__/os.cpython-36.fat-1.pyc: FAT mode > > >With -OO: > >- Lib/__pycache__/os.cpython-36.opt-2.pyc: default mode >- Lib/__pycache__/os.cpython-36.fat-2.pyc: FAT mode > >Is it the "correct" way to name .pyc files? I had to modify unit tests >which expected an exact filename. I think this is the logical extension of PEP 488 for your use case. Cheers, -Barry From brett at python.org Thu Nov 5 13:10:41 2015 From: brett at python.org (Brett Cannon) Date: Thu, 05 Nov 2015 18:10:41 +0000 Subject: [Python-Dev] PYC filename and optimization In-Reply-To: <20151105094040.6062a7b9@anarchist> References: <20151105094040.6062a7b9@anarchist> Message-ID: On Thu, 5 Nov 2015 at 07:41 Barry Warsaw wrote: > On Nov 05, 2015, at 11:33 AM, Victor Stinner wrote: > > >- Lib/__pycache__/os.cpython-36.pyc: default mode > >- Lib/__pycache__/os.cpython-36.fat-0.pyc: FAT mode > > > >With -O: > > > >- Lib/__pycache__/os.cpython-36.opt-1.pyc: default mode > >- Lib/__pycache__/os.cpython-36.fat-1.pyc: FAT mode > > > > > >With -OO: > > > >- Lib/__pycache__/os.cpython-36.opt-2.pyc: default mode > >- Lib/__pycache__/os.cpython-36.fat-2.pyc: FAT mode > > > >Is it the "correct" way to name .pyc files? I had to modify unit tests > >which expected an exact filename. > > I think this is the logical extension of PEP 488 for your use case. > It will require tweaking importlib.util.cache_from_source() to no longer restriction to alphanumeric-only for the optimization level, but that can be tweaked to use `opt-` when given an alphanumeric and then to blindly use another value otherwise. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Thu Nov 5 13:24:07 2015 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 05 Nov 2015 13:24:07 -0500 Subject: [Python-Dev] Rationale behind lazy map/filter In-Reply-To: References: Message-ID: <20151105182408.47A59B1408B@webabinitio.net> On Thu, 05 Nov 2015 03:59:05 +0000, Michael Selik wrote: > > I'm not suggesting restarting at the top (I've elsewhere suggested that > > many such methods would be better as an *iterable* that can be restarted > > at the top by calling iter() multiple times, but that's not the same > > thing). I'm suggesting raising an exception other than StopIteration, so > > that this situation can be detected. If you are writing code that tries > > to resume iterating after the iterator has been exhausted, I have to > > ask: why? > > The most obvious case for me would be tailing a file. Loop over the lines > in the file, sleep, then do it again. There are many tasks analogous to > that scenario -- anything querying a shared resource. The 'file' iterator actually breaks the rules of iterators: it does *not* continue to raise StopIteration once it has been exhausted, if more input becomes available. Given that it is one of the most commonly used iterators (and I would not be surprised if other special-purpose iterators copied its design), this pattern does seem like a blocker for the proposal. --David From status at bugs.python.org Fri Nov 6 12:08:33 2015 From: status at bugs.python.org (Python tracker) Date: Fri, 6 Nov 2015 18:08:33 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20151106170833.6216C56764@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2015-10-30 - 2015-11-06) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5218 (+14) closed 32108 (+35) total 37326 (+49) Open issues with patches: 2301 Issues opened (33) ================== #25520: unittest load_tests protocol not working as documented http://bugs.python.org/issue25520 opened by stefan #25521: optparse module does not emit DeprecationWarning http://bugs.python.org/issue25521 opened by John Hagen #25522: IDLE: warn if save-as name matches stdlib name http://bugs.python.org/issue25522 opened by terry.reedy #25524: Add PyMemoryView_FromObjectWithFlags() http://bugs.python.org/issue25524 opened by skrah #25527: Invalid (... and confusing) warning raised by 2to3 regarding r http://bugs.python.org/issue25527 opened by Antony.Lee #25528: Attempt to further increase test coverage of calendar module http://bugs.python.org/issue25528 opened by Rohit Mediratta #25529: Provide access to the validated certificate chain in ssl modul http://bugs.python.org/issue25529 opened by Lukasa #25530: ssl: OP_NO_SSLv3 should always be set unless a user specifical http://bugs.python.org/issue25530 opened by alex #25531: greenlet header file is missing inside virtualenv http://bugs.python.org/issue25531 opened by kracekumar #25532: infinite loop when running inspect.unwrap over unittest.mock.c http://bugs.python.org/issue25532 opened by cjw296 #25533: Make pkgutil.iter_modules() yield built-in modules http://bugs.python.org/issue25533 opened by martin.panter #25534: SimpleHTTPServer throwed an exception due to negtive st_mtime http://bugs.python.org/issue25534 opened by Sean.Wang #25538: Traceback from __exit__ method is misleading http://bugs.python.org/issue25538 opened by r.david.murray #25541: Wrong usage of sockaddr_un struct for abstract namespace unix http://bugs.python.org/issue25541 opened by soutys #25543: locale.atof keep '.' even if not part of the localeconv http://bugs.python.org/issue25543 opened by ced #25544: cleanup temporary files in distutils.has_function http://bugs.python.org/issue25544 opened by minrk #25545: email parsing docs need to be clear that only ASCII strings ar http://bugs.python.org/issue25545 opened by tanzer at swing.co.at #25546: python 3.5 installation problem; Error 0x80240017: Failed to e http://bugs.python.org/issue25546 opened by rfrank1234 #25547: Improve repr for files to show whether the file is open or clo http://bugs.python.org/issue25547 opened by rhettinger #25548: Show the address in the repr for class objects http://bugs.python.org/issue25548 opened by rhettinger #25550: RecursionError in re with '(' * 500 http://bugs.python.org/issue25550 opened by The Compiler #25551: Event's test_reset_internal_locks too fragile http://bugs.python.org/issue25551 opened by nirs #25553: smtpd strips final carraige return from received message body http://bugs.python.org/issue25553 opened by Deli Zhang #25555: Fix parser and AST: fill lineno and col_offset when compiling http://bugs.python.org/issue25555 opened by haypo #25557: Optimize LOAD_NAME bytecode http://bugs.python.org/issue25557 opened by haypo #25558: Use static asserts in C code http://bugs.python.org/issue25558 opened by serhiy.storchaka #25559: signal.siginterrupt description has typo http://bugs.python.org/issue25559 opened by Tom Meagher #25560: Unhandled warning in test_unicode_file http://bugs.python.org/issue25560 opened by ppperry #25562: Python 2 & 3 don't allow the user to disable ctypes SEH in win http://bugs.python.org/issue25562 opened by tzickel #25563: Windows 10 IDLE / Tkinter startup problem http://bugs.python.org/issue25563 opened by veensl #25564: IDLE behaves differently than python on `del __builtins__` http://bugs.python.org/issue25564 opened by ppperry #25566: asyncio reference cycles after ConnectionResetError http://bugs.python.org/issue25566 opened by joente #25567: shlex.quote doesn't work on bytestrings http://bugs.python.org/issue25567 opened by Jonas Thiem Most recent 15 issues with no replies (15) ========================================== #25566: asyncio reference cycles after ConnectionResetError http://bugs.python.org/issue25566 #25560: Unhandled warning in test_unicode_file http://bugs.python.org/issue25560 #25551: Event's test_reset_internal_locks too fragile http://bugs.python.org/issue25551 #25544: cleanup temporary files in distutils.has_function http://bugs.python.org/issue25544 #25529: Provide access to the validated certificate chain in ssl modul http://bugs.python.org/issue25529 #25528: Attempt to further increase test coverage of calendar module http://bugs.python.org/issue25528 #25527: Invalid (... and confusing) warning raised by 2to3 regarding r http://bugs.python.org/issue25527 #25524: Add PyMemoryView_FromObjectWithFlags() http://bugs.python.org/issue25524 #25520: unittest load_tests protocol not working as documented http://bugs.python.org/issue25520 #25518: Investigate implementation of PyOS_CheckStack on OSX http://bugs.python.org/issue25518 #25511: multiprocessing pool blocks SIGTERM from being handled http://bugs.python.org/issue25511 #25497: Rewrite test_robotparser http://bugs.python.org/issue25497 #25491: ftplib.sendcmd only accepts string http://bugs.python.org/issue25491 #25486: Resurrect inspect.getargspec() in 3.6 http://bugs.python.org/issue25486 #25479: Increase unit test coverage for abc.py http://bugs.python.org/issue25479 Most recent 15 issues waiting for review (15) ============================================= #25558: Use static asserts in C code http://bugs.python.org/issue25558 #25557: Optimize LOAD_NAME bytecode http://bugs.python.org/issue25557 #25555: Fix parser and AST: fill lineno and col_offset when compiling http://bugs.python.org/issue25555 #25551: Event's test_reset_internal_locks too fragile http://bugs.python.org/issue25551 #25544: cleanup temporary files in distutils.has_function http://bugs.python.org/issue25544 #25533: Make pkgutil.iter_modules() yield built-in modules http://bugs.python.org/issue25533 #25530: ssl: OP_NO_SSLv3 should always be set unless a user specifical http://bugs.python.org/issue25530 #25528: Attempt to further increase test coverage of calendar module http://bugs.python.org/issue25528 #25521: optparse module does not emit DeprecationWarning http://bugs.python.org/issue25521 #25516: threading.Condition._is_owned() is wrong when using threading. http://bugs.python.org/issue25516 #25498: Python 3.4.3 core dump with simple sample code http://bugs.python.org/issue25498 #25497: Rewrite test_robotparser http://bugs.python.org/issue25497 #25495: binascii documentation incorrect http://bugs.python.org/issue25495 #25489: sys.exit() caught in exception handler http://bugs.python.org/issue25489 #25485: Add a context manager to telnetlib.Telnet http://bugs.python.org/issue25485 Top 10 most discussed issues (10) ================================= #25521: optparse module does not emit DeprecationWarning http://bugs.python.org/issue25521 18 msgs #25495: binascii documentation incorrect http://bugs.python.org/issue25495 15 msgs #25531: greenlet header file is missing inside virtualenv http://bugs.python.org/issue25531 13 msgs #25558: Use static asserts in C code http://bugs.python.org/issue25558 12 msgs #25545: email parsing docs need to be clear that only ASCII strings ar http://bugs.python.org/issue25545 9 msgs #25498: Python 3.4.3 core dump with simple sample code http://bugs.python.org/issue25498 7 msgs #23496: Steps for Android Native Build of Python 3.4.2 http://bugs.python.org/issue23496 6 msgs #24726: OrderedDict has strange behaviour when dict.__setitem__ is use http://bugs.python.org/issue24726 6 msgs #25532: infinite loop when running inspect.unwrap over unittest.mock.c http://bugs.python.org/issue25532 6 msgs #18010: pydoc search chokes on import errors http://bugs.python.org/issue18010 5 msgs Issues closed (33) ================== #5550: [urllib.request]: Comparison of HTTP headers should be insensi http://bugs.python.org/issue5550 closed by martin.panter #12300: Document pydoc.help http://bugs.python.org/issue12300 closed by terry.reedy #18973: Use argparse in the calendar module http://bugs.python.org/issue18973 closed by serhiy.storchaka #24803: PyNumber_Long Buffer Over-read.patch http://bugs.python.org/issue24803 closed by serhiy.storchaka #25395: SIGSEGV using json.tool: highly nested OrderedDict http://bugs.python.org/issue25395 closed by serhiy.storchaka #25426: Deprecate the regex_compile benchmark http://bugs.python.org/issue25426 closed by brett.cannon #25443: Add a count of how many benchmarks are left to run http://bugs.python.org/issue25443 closed by brett.cannon #25449: Test OrderedDict subclass http://bugs.python.org/issue25449 closed by serhiy.storchaka #25450: Python 3.5 starts in C:\Windows\system32 as current directory http://bugs.python.org/issue25450 closed by steve.dower #25454: operator.methodcaller should accept additional arguments durin http://bugs.python.org/issue25454 closed by rhettinger #25474: Weird behavior when setting f_trace in a context manager http://bugs.python.org/issue25474 closed by Fred Gansevles #25483: Improve f-string implementation: FORMAT_VALUE opcode http://bugs.python.org/issue25483 closed by eric.smith #25487: imp module DeprecationWarning in test suite http://bugs.python.org/issue25487 closed by brett.cannon #25493: warnings.warn: wrong stacklevel causes import of local file "s http://bugs.python.org/issue25493 closed by brett.cannon #25505: undefined name 'window' in Tools/scripts/fixdiv.py http://bugs.python.org/issue25505 closed by terry.reedy #25510: fileinput.FileInput.readline() always returns str object at th http://bugs.python.org/issue25510 closed by serhiy.storchaka #25519: Minor difflib documentation bug http://bugs.python.org/issue25519 closed by terry.reedy #25523: Correct "a" article to "an" article http://bugs.python.org/issue25523 closed by martin.panter #25525: Deallocation scheme for memoryview is unsafe http://bugs.python.org/issue25525 closed by skrah #25526: (python2.7.10)ImportError: No module named _ssl http://bugs.python.org/issue25526 closed by zach.ware #25535: collections.Counter methods return Counter objects instead of http://bugs.python.org/issue25535 closed by rhettinger #25536: use sys.platform instead of os.name in asyncio docs consistent http://bugs.python.org/issue25536 closed by gvanrossum #25537: Call `isinstance` instead of `issubclass` during exception han http://bugs.python.org/issue25537 closed by r.david.murray #25539: python3 fail on parsing http header http://bugs.python.org/issue25539 closed by ????????? #25540: virtual conf file encoding error on windows http://bugs.python.org/issue25540 closed by serhiy.storchaka #25542: tuple unpacking on assignment should be lazy for unpacked gene http://bugs.python.org/issue25542 closed by Mario Wenzel #25549: call sum on list of timedelta throws TypeError http://bugs.python.org/issue25549 closed by r.david.murray #25552: python turtle page does not run http://bugs.python.org/issue25552 closed by r.david.murray #25554: memory leak (reference cycles) using re http://bugs.python.org/issue25554 closed by serhiy.storchaka #25556: Fixes for PyObject_GetItem() http://bugs.python.org/issue25556 closed by haypo #25561: unstable result of time.monotonic http://bugs.python.org/issue25561 closed by eric.smith #25565: subprocess.Popen creates inheritable file descriptors on Windo http://bugs.python.org/issue25565 closed by eryksun #25568: typing backport: AttributeError: 'module' object has no attrib http://bugs.python.org/issue25568 closed by nicolas33 From mark at markroseman.com Fri Nov 6 12:08:24 2015 From: mark at markroseman.com (Mark Roseman) Date: Fri, 6 Nov 2015 09:08:24 -0800 Subject: [Python-Dev] modernizing IDLE Message-ID: Just wanted to share some documentation I?ve put together (and a snapshot of the code) on the effort to improve the appearance and behaviour of IDLE: http://www.tkdocs.com/tutorial/idle.html (There?s also currently a post on Hacker News about this). While Terry can better speak to plans and timelines about getting these changes integrated in, it seems a good time to get some other people to have a closer look and share any feedback. Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmkleffner at gmail.com Sat Nov 7 10:55:33 2015 From: cmkleffner at gmail.com (Carl Kleffner) Date: Sat, 7 Nov 2015 16:55:33 +0100 Subject: [Python-Dev] modernizing IDLE In-Reply-To: References: Message-ID: there is also http://idlex.sourceforge.net/about.html to mention. 2015-11-06 18:08 GMT+01:00 Mark Roseman : > Just wanted to share some documentation I?ve put together (and a snapshot > of the code) on the effort to improve the appearance and behaviour of IDLE: > > http://www.tkdocs.com/tutorial/idle.html > > (There?s also currently a post on Hacker News about this). > > While Terry can better speak to plans and timelines about getting these > changes integrated in, it seems a good time to get some other people to > have a closer look and share any feedback. > > Mark > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pmiscml at gmail.com Sat Nov 7 18:00:59 2015 From: pmiscml at gmail.com (Paul Sokolovsky) Date: Sun, 8 Nov 2015 02:00:59 +0300 Subject: [Python-Dev] [ANN] MicroPython 1.5 Message-ID: <20151108020059.54a36344@x230> Hello, MicroPython is a lean and efficient Python implementation for microcontrollers, embedded, and mobile systems (which also runs just as fine on desktops of course). https://github.com/micropython/micropython Recent 1.5 release is an important milestone for the project, major changes including: 1. Advanced REPL support with smart auto-indent and auto-completion for comfortable interactive typing pf Python code, easily switchable to paste mode to copy pre-made code snippets. 2. Support for client SSL connections. 3. upip, MicroPython builtin package manager, is now fully standalone thanks to SSL support. 4. There's new, elaborated API to access hardware features ("machine" module). 5. There're now 2 well-supported microcontroller boards for MicroPython, and dozen(s) community-supported ones. 6. MicroPython was selected as one of the languages supported for BBC micro:bit initiative: http://ntoll.org/article/story-micropython-on-microbit 7. There's work on native efficient JNI bridge, inspired by Kivy's PyJNIus module, included into growing Android port. More detailed changelog for this version is at https://github.com/micropython/micropython/releases/tag/v1.5 MicroPython supports growing subset of Python3 standard library (including simplified asyncio package) - not included by default with an executable, but easily installable per-module using builtin "upip" package manager: https://github.com/micropython/micropython-lib -- Best regards, Paul mailto:pmiscml at gmail.com From ncoghlan at gmail.com Tue Nov 10 07:45:04 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Nov 2015 22:45:04 +1000 Subject: [Python-Dev] modernizing IDLE In-Reply-To: References: Message-ID: On 7 November 2015 at 03:08, Mark Roseman wrote: > Just wanted to share some documentation I?ve put together (and a snapshot of > the code) on the effort to improve the appearance and behaviour of IDLE: > > http://www.tkdocs.com/tutorial/idle.html > > (There?s also currently a post on Hacker News about this). > > While Terry can better speak to plans and timelines about getting these > changes integrated in, it seems a good time to get some other people to have > a closer look and share any feedback. Very cool! Thanks to all involved for these modernisation efforts :) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From a.r.kaija at gmail.com Tue Nov 10 02:19:24 2015 From: a.r.kaija at gmail.com (Alec Kaija) Date: Tue, 10 Nov 2015 02:19:24 -0500 Subject: [Python-Dev] This isn't a question Message-ID: Sorry to bother anyone, but I just wanted to thank the Python community for being awesome and for all of the ways you have all inspired/supported me (mostly without even knowing it). Thanks! a -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Nov 10 13:09:37 2015 From: brett at python.org (Brett Cannon) Date: Tue, 10 Nov 2015 18:09:37 +0000 Subject: [Python-Dev] This isn't a question In-Reply-To: References: Message-ID: You're quite welcome! And thanks for taking the time to say "thanks". :) On Tue, 10 Nov 2015, 11:35 Alec Kaija wrote: > Sorry to bother anyone, but I just wanted to thank the Python community > for being awesome and for all of the ways you have all inspired/supported > me (mostly without even knowing it). > > Thanks! > > > a > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vgr255 at live.ca Tue Nov 10 17:00:04 2015 From: vgr255 at live.ca (Emanuel Barry) Date: Tue, 10 Nov 2015 17:00:04 -0500 Subject: [Python-Dev] This isn't a question In-Reply-To: References: , Message-ID: While I'm a part of the community, that wasn't always the case and it's very large, so I'll take this opportunity to come in and say a big thank you as well! From: brett at python.org Date: Tue, 10 Nov 2015 18:09:37 +0000 To: a.r.kaija at gmail.com; python-dev at python.org Subject: Re: [Python-Dev] This isn't a question You're quite welcome! And thanks for taking the time to say "thanks". :) On Tue, 10 Nov 2015, 11:35 Alec Kaija wrote: Sorry to bother anyone, but I just wanted to thank the Python community for being awesome and for all of the ways you have all inspired/supported me (mostly without even knowing it). Thanks! a _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vgr255%40live.ca -------------- next part -------------- An HTML attachment was scrubbed... URL: From motoom at xs4all.nl Tue Nov 10 18:02:59 2015 From: motoom at xs4all.nl (Michiel Overtoom) Date: Wed, 11 Nov 2015 00:02:59 +0100 Subject: [Python-Dev] modernizing IDLE In-Reply-To: References: Message-ID: <25544A86-7073-47F6-9189-F734D5007B01@xs4all.nl> > On 06 Nov 2015, at 18:08, Mark Roseman wrote: > > (There?s also currently a post on Hacker News about this). You have a link for that HN item? I've looked at the first five pages but couldn't find it. Greetings, From rymg19 at gmail.com Tue Nov 10 18:23:15 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 10 Nov 2015 17:23:15 -0600 Subject: [Python-Dev] modernizing IDLE In-Reply-To: <25544A86-7073-47F6-9189-F734D5007B01@xs4all.nl> References: <25544A86-7073-47F6-9189-F734D5007B01@xs4all.nl> Message-ID: On Tue, Nov 10, 2015 at 5:02 PM, Michiel Overtoom wrote: > > > On 06 Nov 2015, at 18:08, Mark Roseman wrote: > > > > (There?s also currently a post on Hacker News about this). > > You have a link for that HN item? I've looked at the first five pages but > couldn't find it. > > https://news.ycombinator.com/item?id=10519785 I used the big "Search" box at the bottom of the main Hacker News page. ;) > Greetings, > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Nov 10 19:06:57 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 11 Nov 2015 10:06:57 +1000 Subject: [Python-Dev] This isn't a question In-Reply-To: References: Message-ID: On 11 November 2015 at 04:09, Brett Cannon wrote: > You're quite welcome! And thanks for taking the time to say "thanks". :) Indeed! Folks saying "Thank you" has a surprisingly big impact, since our natural tendency is to spend time dwelling on all the things that still need improvement :) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Nov 10 19:47:53 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 11 Nov 2015 10:47:53 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) Message-ID: Hi folks, I have a confession to make - I dropped the ball on the HTTPS verification backport proposals in PEP 493, and let the upstream and downstream approval processes get out of sequence. As a result, the RHEL 7.2 beta released back in September incorporates the HTTPS verification feature backport based on the current PEP 493 draft, even though that hasn't formally been pronounced as an Active recommendation by python-dev yet. Accordingly, I'm belatedly submitting it for pronouncement now: https://www.python.org/dev/peps/pep-0493/ There's currently no BDFL-Delegate assigned, so if Guido doesn't want to handle it, we'll need to address that question first. Our last discussion back in July seemed to show that folks either didn't care about the question (because they're using unmodified upstream versions so the PEP didn't affect them), or else thought the approach described in the PEP was reasonable, so I'm hoping the consequences of my mistake won't be too severe. Regards, Nick. P.S. I'm aware that this looks like presenting a fait accompli at a point where it's too late to realistically say "No", but the truth is that preparation for the Python in Education miniconf at PyCon Australia ramped up immediately after the July discussion, and then I personally got confused as to the scope of what was being included in 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493 being deferred to a later release, but it's actually the whole package of 466+476+493). That's my fault for trying to keep track of too many things at once (and thus failing at some of them), not anyone else's. ================================ PEP: 493 Title: HTTPS verification recommendations for Python 2.7 redistributors Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan , Robert Kuska , Marc-Andr? Lemburg Status: Draft Type: Informational Content-Type: text/x-rst Created: 10-May-2015 Post-History: 06-Jul-2015 Abstract ======== PEP 476 updated Python's default handling of HTTPS certificates to be appropriate for communication over the public internet. The Python 2.7 long term maintenance series was judged to be in scope for this change, with the new behaviour introduced in the Python 2.7.9 maintenance release. This PEP provides recommendations to downstream redistributors wishing to provide a smoother migration experience when helping their users to manage this change in Python's default behaviour. Rationale ========= PEP 476 changed Python's default behaviour to better match the needs and expectations of developers operating over the public internet, a category which appears to include most new Python developers. It is the position of the authors of this PEP that this was a correct decision. However, it is also the case that this change *does* cause problems for infrastructure administrators operating private intranets that rely on self-signed certificates, or otherwise encounter problems with the new default certificate verification settings. The long term answer for such environments is to update their internal certificate management to at least match the standards set by the public internet, but in the meantime, it is desirable to offer these administrators a way to continue receiving maintenance updates to the Python 2.7 series, without having to gate that on upgrades to their certificate management infrastructure. PEP 476 did attempt to address this question, by covering how to revert the new settings process wide by monkeypatching the ``ssl`` module to restore the old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed to allow system administrators to disable the feature by default in their Standard Operating Environment definition has been determined to be insufficient in at least some cases. The specific case of interest to the authors of this PEP is the one where a Linux distributor aims to provide their users with a `smoother migration path `__ than the standard one provided by consuming upstream CPython 2.7 releases directly, but other potential challenges have also been pointed out with updating embedded Python runtimes and other user level installations of Python. Rather than allowing a plethora of mutually incompatibile migration techniques to bloom, this PEP proposes two alternative approaches that redistributors may take when addressing these problems. Redistributors may choose to implement one, both, or neither of these approaches based on their assessment of the needs of their particular userbase. These designs are being proposed as a recommendation for redistributors, rather than as new upstream features, as they are needed purely to support legacy environments migrating from older versions of Python 2.7. Neither approach is being proposed as an upstream Python 2.7 feature, nor as a feature in any version of Python 3 (whether published directly by the Python Software Foundation or by a redistributor). Requirements for capability detection ===================================== As these recommendations are intended to cover backports to earlier Python versions, the Python version number cannot be used as a reliable means for detecting them. Instead, the recommendations are defined to allow the presence or absence of the feature to be determined using the following technique:: python -c "import ssl; ssl._relevant_attribute" This will fail with `AttributeError` (and hence a non-zero return code) if the relevant capability is not available. The marker attributes are prefixed with an underscore to indicate the implementation dependent nature of these capabilities - not all Python distributions will offer them, only those that are providing a multi-stage migration process from the legacy HTTPS handling to the new default behaviour. Recommendation for an environment variable based security downgrade =================================================================== Some redistributors may wish to provide a per-application option to disable certificate verification in selected applications that run on or embed CPython without needing to modify the application itself. In these cases, a configuration mechanism is needed that provides: * an opt-out model that allows certificate verification to be selectively turned off for particular applications after upgrading to a version of Python that verifies certificates by default * the ability for all users to configure this setting on a per-application basis, rather than on a per-system, or per-Python-installation basis This approach may be used for any redistributor provided version of Python 2.7, including those that advertise themselves as providing Python 2.7.9 or later. Required marker attribute ------------------------- The required marker attribute on the ``ssl`` module when implementing this recommendation is:: _https_verify_envvar = 'PYTHONHTTPSVERIFY' This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant environment variable name. Recommended modifications to the Python standard library -------------------------------------------------------- The recommended approach to providing a per-application configuration setting for HTTPS certificate verification that doesn't require modifications to the application itself is to: * modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first imported into a Python process * set the ``ssl._create_default_https_context`` function to be an alias for ``ssl._create_unverified_context`` if this environment variable is present and set to ``'0'`` * otherwise, set the ``ssl._create_default_https_context`` function to be an alias for ``ssl.create_default_context`` as usual Example implementation ---------------------- :: _https_verify_envvar = 'PYTHONHTTPSVERIFY' def _get_https_context_factory(): config_setting = os.environ.get(_https_verify_envvar) if config_setting == '0': return _create_unverified_context return create_default_context _create_default_https_context = _get_https_context_factory() Security Considerations ----------------------- Relative to an unmodified version of CPython 2.7.9 or later, this approach does introduce a new downgrade attack against the default security settings that potentially allows a sufficiently determined attacker to revert Python to the vulnerable configuration used in CPython 2.7.8 and earlier releases. However, such an attack requires the ability to modify the execution environment of a Python process prior to the import of the ``ssl`` module, and any attacker with such access would already be able to modify the behaviour of the underlying OpenSSL implementation. Recommendation for backporting to earlier Python versions ========================================================= Some redistributors, most notably Linux distributions, may choose to backport the PEP 476 HTTPS verification changes to modified Python versions based on earlier Python 2 maintenance releases. In these cases, a configuration mechanism is needed that provides: * an opt-in model that allows the decision to enable HTTPS certificate verification to be made independently of the decision to upgrade to the Python version where the feature was first backported * the ability for system administrators to set the default behaviour of Python applications and scripts run directly in the system Python installation * the ability for the redistributor to consider changing the default behaviour of *new* installations at some point in the future without impacting existing installations that have been explicitly configured to skip verifying HTTPS certificates by default This approach should not be used for any Python installation that advertises itself as providing Python 2.7.9 or later, as most Python users will have the reasonable expectation that all such environments will validate HTTPS certificates by default. Required marker attribute ------------------------- The required marker attribute on the ``ssl`` module when implementing this recommendation is:: _cert_verification_config = '' This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant configuration file name. Recommended modifications to the Python standard library -------------------------------------------------------- The recommended approach to backporting the PEP 476 modifications to an earlier point release is to implement the following changes relative to the default PEP 476 behaviour implemented in Python 2.7.9+: * modify the ``ssl`` module to read a system wide configuration file when the module is first imported into a Python process * define a platform default behaviour (either verifying or not verifying HTTPS certificates) to be used if this configuration file is not present * support selection between the following three modes of operation: * ensure HTTPS certificate verification is enabled * ensure HTTPS certificate verification is disabled * delegate the decision to the redistributor providing this Python version * set the ``ssl._create_default_https_context`` function to be an alias for either ``ssl.create_default_context`` or ``ssl._create_unverified_context`` based on the given configuration setting. Recommended file location ------------------------- This approach is currently only defined for \*nix system Python installations. The recommended configuration file name is ``/etc/python/cert-verification.cfg``. The ``.cfg`` filename extension is recommended for consistency with the ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library. Recommended file format ----------------------- The configuration file should use a ConfigParser ini-style format with a single section named ``[https]`` containing one required setting ``verify``. Permitted values for ``verify`` are: * ``enable``: ensure HTTPS certificate verification is enabled by default * ``disable``: ensure HTTPS certificate verification is disabled by default * ``platform_default``: delegate the decision to the redistributor providing this particular Python version If the ``[https]`` section or the ``verify`` setting are missing, or if the ``verify`` setting is set to an unknown value, it should be treated as if the configuration file is not present. Example implementation ---------------------- :: _cert_verification_config = '/etc/python/cert-verification.cfg' def _get_https_context_factory(): # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory) _create_default_https_context = _get_https_context_factory() Security Considerations ----------------------- The specific recommendations for the backporting case are designed to work for privileged, security sensitive processes, even those being run in the following locked down configuration: * run from a locked down administrator controlled directory rather than a normal user directory (preventing ``sys.path[0]`` based privilege escalation attacks) * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based privilege escalation attacks) * run using the ``-s`` switch (preventing user site directory based privilege escalation attacks) * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege escalation attacks) The intent is that the *only* reason HTTPS verification should be getting turned off system wide when using this approach is because: * an end user is running a redistributor provided version of CPython rather than running upstream CPython directly * that redistributor has decided to provide a smoother migration path to verifying HTTPS certificates by default than that being provided by the upstream project * either the redistributor or the local infrastructure administrator has determined that it is appropriate to override the default upstream behaviour (at least for the time being) Using an administrator controlled configuration file rather than an environment variable has the essential feature of providing a smoother migration path, even for applications being run with the ``-E`` switch. Combining the recommendations ============================= If a redistributor chooses to implement both recommendations, then the environment variable should take precedence over the system-wide configuration setting. This allows the setting to be changed for a given user, virtual environment or application, regardless of the system-wide default behaviour. In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and set to anything *other* than ``'0'``, then HTTPS certificate verification should be enabled. Example implementation ---------------------- :: _https_verify_envvar = 'PYTHONHTTPSVERIFY' _cert_verification_config = '/etc/python/cert-verification.cfg' def _get_https_context_factory(): # Check for am environmental override of the default behaviour config_setting = os.environ.get(_https_verify_envvar) if config_setting is not None: if config_setting == '0': return _create_unverified_context return create_default_context # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory) _create_default_https_context = _get_https_context_factory() Copyright ========= This document has been placed into the public domain. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From motoom at xs4all.nl Tue Nov 10 21:07:28 2015 From: motoom at xs4all.nl (Michiel Overtoom) Date: Wed, 11 Nov 2015 03:07:28 +0100 Subject: [Python-Dev] modernizing IDLE In-Reply-To: References: <25544A86-7073-47F6-9189-F734D5007B01@xs4all.nl> Message-ID: > On 11 Nov 2015, at 00:23, Ryan Gonzalez wrote: > > https://news.ycombinator.com/item?id=10519785 Thanks for the URL. I consider myself a reasonably seasoned IDE user, having used Visual Studio, emacs with SLIME, Borland C++Builder, Xcode, Delphi, Eclipse, LightTable and PyCharm, and probably I forgot some. However, I still use IDLE regularly on Windows and Unix systems (not on OSX). I don't care much about advanced IDE features. Basically I want a programmers' text editor with minimal syntax highlighting and a shortcut key like F9 or CMD+R to run my program. When developing webapps, I rely on the 'save' function of the editor and having the web framework reload the sources automatically while in developer mode. For debugging, I use Python's 'pdb' or the more fancier 'pudb' where appropriate. I really like to see IDLE stay as part of the Python installation. And I'm sure, with all the attention it gets nowadays, that it'll stay a minimal but usable basic IDE for developing Python programs. Greetings, From projetmbc at gmail.com Wed Nov 11 10:13:03 2015 From: projetmbc at gmail.com (Christophe Bal) Date: Wed, 11 Nov 2015 16:13:03 +0100 Subject: [Python-Dev] Translate Python language Message-ID: Hello. I'm a french teacher and I would like to use Python with young child but I've a big problem. All the keyword are in english. So I would like to know if there is a way to hack the AST tools such as to use french keywords and then translate a list of french keywords to their english regular version. Where should I start ? My idea is not to build a new language but simply translate words using an hacked version of the AST tools. Hoping to be clear and I do not pollute the message in this list. *Christophe BAL* *Enseignant de math?matiques en Lyc?e **et d?veloppeur Python amateur* *---* *French math teacher in a "Lyc?e" **and **Python **amateur developer* -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Nov 11 11:17:33 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 11 Nov 2015 16:17:33 +0000 Subject: [Python-Dev] Translate Python language In-Reply-To: References: Message-ID: On 11 November 2015 at 15:13, Christophe Bal wrote: > Hello. > > I'm a french teacher and I would like to use Python with young child but > I've a big problem. All the keyword are in english. So I would like to know > if there is a way to hack the AST tools such as to use french keywords and > then translate a list of french keywords to their english regular version. > > Where should I start ? My idea is not to build a new language but simply > translate words using an hacked version of the AST tools. > > Hoping to be clear and I do not pollute the message in this list. Do you really just mean keywords? Most of the English words used in Python (for example "open") aren't actually keywords but are names of functions defined in the builtins and/or standard library. Those are not exposed as distinct objects in the AST, but are simply name lookups. If you do mean just keywords, then you could probably relatively simply hack the Python interpreter code to use different names for the keywords, and build your own translated copy of Python. I don't know the details, though. But in practice this wouldn't really help your student, as there would still be many English words they would encounter and need to understand (at least in the context of the code they are writing). My understanding is that most non-English speaking teachers tend to accept that their students simply need to treat the keywords and function names used in typical programming languages (Python included) as things they just have to learn. But I suspect most students are typically older than your young child. Sorry I can't be more help. Paul From donald at stufft.io Wed Nov 11 11:20:04 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 11 Nov 2015 11:20:04 -0500 Subject: [Python-Dev] Translate Python language In-Reply-To: References: Message-ID: On November 11, 2015 at 11:19:07 AM, Paul Moore (p.f.moore at gmail.com) wrote: > On 11 November 2015 at 15:13, Christophe Bal wrote: > > Hello. > > > > I'm a french teacher and I would like to use Python with young child but > > I've a big problem. All the keyword are in english. So I would like to know > > if there is a way to hack the AST tools such as to use french keywords and > > then translate a list of french keywords to their english regular version. > > > > Where should I start ? My idea is not to build a new language but simply > > translate words using an hacked version of the AST tools. > > > > Hoping to be clear and I do not pollute the message in this list. > > Do you really just mean keywords? Most of the English words used in > Python (for example "open") aren't actually keywords but are names of > functions defined in the builtins and/or standard library. Those are > not exposed as distinct objects in the AST, but are simply name > lookups. > You might be able to use an encoding to do this. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From brett at python.org Wed Nov 11 14:31:52 2015 From: brett at python.org (Brett Cannon) Date: Wed, 11 Nov 2015 19:31:52 +0000 Subject: [Python-Dev] Translate Python language In-Reply-To: References: Message-ID: On Wed, 11 Nov 2015, 10:55 Christophe Bal wrote: Hello. I'm a french teacher and I would like to use Python with young child but I've a big problem. All the keyword are in english. So I would like to know if there is a way to hack the AST tools such as to use french keywords and then translate a list of french keywords to their english regular version. Where should I start ? My idea is not to build a new language but simply translate words using an hacked version of the AST tools. You actually need a new grammar for the parser as the AST comes from that. Then you may be able to build the AST. -brett Hoping to be clear and I do not pollute the message in this list. *Christophe BAL* *Enseignant** de **math?matiques* *en** Lyc?e **et **d?veloppeur** Python amateur* *---* *French math teacher in a "Lyc?e" **and **Python **amateur developer* _______________________________________________ Python-Dev mailing list Python-Dev at python.org https :// mail.python.org /mailman/ listinfo / python-dev Unsubscribe: https :// mail.python.org /mailman/options/ python-dev / brett %40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From nirsof at gmail.com Wed Nov 11 14:47:11 2015 From: nirsof at gmail.com (Nir Soffer) Date: Wed, 11 Nov 2015 21:47:11 +0200 Subject: [Python-Dev] Translate Python language In-Reply-To: References: Message-ID: On Wed, Nov 11, 2015 at 5:13 PM, Christophe Bal wrote: > Hello. > > I'm a french teacher and I would like to use Python with young child but > I've a big problem. All the keyword are in english. So I would like to know > if there is a way to hack the AST tools such as to use french keywords and > then translate a list of french keywords to their english regular version. > > Where should I start ? My idea is not to build a new language but simply > translate words using an hacked version of the AST tools. > > Hoping to be clear and I do not pollute the message in this list. > I did this in 2006 for Hebrew - we had Hebrew Python dialect, with Hebrew turtle module and interactive shell. You can try to reuse this: https://github.com/nirs/hpy Nir > > > *Christophe BAL* > *Enseignant de math?matiques en Lyc?e **et d?veloppeur Python amateur* > *---* > *French math teacher in a "Lyc?e" **and **Python **amateur developer* > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/nirsof%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Wed Nov 11 14:55:23 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 11 Nov 2015 20:55:23 +0100 Subject: [Python-Dev] Translate Python language In-Reply-To: References: Message-ID: <56439D2B.5060707@egenix.com> On 11.11.2015 17:20, Donald Stufft wrote: > On November 11, 2015 at 11:19:07 AM, Paul Moore (p.f.moore at gmail.com) wrote: >> On 11 November 2015 at 15:13, Christophe Bal wrote: >>> Hello. >>> >>> I'm a french teacher and I would like to use Python with young child but >>> I've a big problem. All the keyword are in english. So I would like to know >>> if there is a way to hack the AST tools such as to use french keywords and >>> then translate a list of french keywords to their english regular version. >>> >>> Where should I start ? My idea is not to build a new language but simply >>> translate words using an hacked version of the AST tools. >>> >>> Hoping to be clear and I do not pollute the message in this list. >> >> Do you really just mean keywords? Most of the English words used in >> Python (for example "open") aren't actually keywords but are names of >> functions defined in the builtins and/or standard library. Those are >> not exposed as distinct objects in the AST, but are simply name >> lookups. >> > > You might be able to use an encoding to do this. Or use Andrew Dalke's fun project LOLPython as basis: http://www.dalkescientific.com/writings/diary/archive/2007/06/01/lolpython.html It uses a parser which then translates the source code into regular Python source code, which it then runs. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 11 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ 2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From benjamin at python.org Thu Nov 12 01:50:05 2015 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 11 Nov 2015 22:50:05 -0800 Subject: [Python-Dev] [Python-checkins] cpython (2.7): Backport early-out 91259f061cfb to reduce the cost of bb1a2944bcb6 In-Reply-To: <20151007031207.18390.20414@psf.io> References: <20151007031207.18390.20414@psf.io> Message-ID: <1447311005.2650788.436807905.05A186FE@webmail.messagingengine.com> On Tue, Oct 6, 2015, at 19:12, raymond.hettinger wrote: > https://hg.python.org/cpython/rev/37aee118e1a3 > changeset: 98578:37aee118e1a3 > branch: 2.7 > parent: 98572:60c44a09c5fc > user: Raymond Hettinger > date: Tue Oct 06 23:12:02 2015 -0400 > summary: > Backport early-out 91259f061cfb to reduce the cost of bb1a2944bcb6 > > files: > Modules/_collectionsmodule.c | 6 +++++- > 1 files changed, 5 insertions(+), 1 deletions(-) > > > diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c > --- a/Modules/_collectionsmodule.c > +++ b/Modules/_collectionsmodule.c > @@ -651,6 +651,9 @@ > Py_ssize_t n; > PyObject *item; > > + if (Py_SIZE(deque) == 0) > + return; > + dequeue is not varsized in Python 2.7, so using Py_SIZE() is incorrect. From python at rcn.com Thu Nov 12 10:20:29 2015 From: python at rcn.com (Raymond Hettinger) Date: Thu, 12 Nov 2015 07:20:29 -0800 Subject: [Python-Dev] [Python-checkins] cpython (2.7): Backport early-out 91259f061cfb to reduce the cost of bb1a2944bcb6 In-Reply-To: <1447311005.2650788.436807905.05A186FE@webmail.messagingengine.com> References: <20151007031207.18390.20414@psf.io> <1447311005.2650788.436807905.05A186FE@webmail.messagingengine.com> Message-ID: <305A353F-82F7-4A6C-A9E5-235808CBA966@rcn.com> > On Nov 11, 2015, at 10:50 PM, Benjamin Peterson wrote: > >> + if (Py_SIZE(deque) == 0) >> + return; >> + > > dequeue is not varsized in Python 2.7, so using Py_SIZE() is incorrect. Fixed in a2a518b6ded4. - if (Py_SIZE(deque) == 0) + if (deque->len == 0) Raymond From benjamin at python.org Thu Nov 12 15:13:10 2015 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 12 Nov 2015 12:13:10 -0800 Subject: [Python-Dev] [Python-checkins] cpython (2.7): Backport early-out 91259f061cfb to reduce the cost of bb1a2944bcb6 In-Reply-To: <305A353F-82F7-4A6C-A9E5-235808CBA966@rcn.com> References: <20151007031207.18390.20414@psf.io> <1447311005.2650788.436807905.05A186FE@webmail.messagingengine.com> <305A353F-82F7-4A6C-A9E5-235808CBA966@rcn.com> Message-ID: <1447359190.2841964.438218129.0466D7F0@webmail.messagingengine.com> There's two instances in that file. On Thu, Nov 12, 2015, at 07:20, Raymond Hettinger wrote: > > > On Nov 11, 2015, at 10:50 PM, Benjamin Peterson wrote: > > > >> + if (Py_SIZE(deque) == 0) > >> + return; > >> + > > > > dequeue is not varsized in Python 2.7, so using Py_SIZE() is incorrect. > > Fixed in a2a518b6ded4. > > - if (Py_SIZE(deque) == 0) > + if (deque->len == 0) > > > Raymond From ethan at stoneleaf.us Fri Nov 13 10:12:32 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 13 Nov 2015 07:12:32 -0800 Subject: [Python-Dev] doc tests failing Message-ID: <5645FDE0.9040304@stoneleaf.us> What am I doing wrong? I have tried: -------------------------------------------------------- hg update 3.5 # and hg update default make distclean && ./configure --with-pydebug && make -j2 cd Doc make doctest -------------------------------------------------------- and in both cases I get page after page of errors. I have tried installing python-sphinx and python3-sphinx; I have tried adding PYTHON=../python and PYTHON=python3 to the `make doctest` line -- all to no avail. Here's a random sample of the errors: ********************************************************************** File "library/shlex.rst", line ?, in default Failed example: remote_command Exception raised: Traceback (most recent call last): File "/usr/lib/python2.7/doctest.py", line 1315, in __run compileflags, 1) in test.globs File "", line 1, in remote_command NameError: name 'remote_command' is not defined ********************************************************************** File "howto/sorting.rst", line ?, in default Failed example: sorted([5, 2, 4, 1, 3], cmp=numeric_compare) Exception raised: Traceback (most recent call last): File "/usr/lib/python2.7/doctest.py", line 1315, in __run compileflags, 1) in test.globs File "", line 1, in sorted([5, 2, 4, 1, 3], cmp=numeric_compare) NameError: name 'numeric_compare' is not defined ********************************************************************** File "library/ipaddress.rst", line ?, in default Failed example: n2 = ip_network('192.0.2.1/32') Exception raised: Traceback (most recent call last): File "/usr/lib/python2.7/doctest.py", line 1315, in __run compileflags, 1) in test.globs File "", line 1, in n2 = ip_network('192.0.2.1/32') NameError: name 'ip_network' is not defined -- ~Ethan~ From g.brandl at gmx.net Fri Nov 13 10:40:16 2015 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 13 Nov 2015 16:40:16 +0100 Subject: [Python-Dev] doc tests failing In-Reply-To: <5645FDE0.9040304@stoneleaf.us> References: <5645FDE0.9040304@stoneleaf.us> Message-ID: On 11/13/2015 04:12 PM, Ethan Furman wrote: > What am I doing wrong? Running "make doctest" in the first place. The doctests in the core docs were not written to support that, and also never updated to do so. Georg > I have tried: > > -------------------------------------------------------- > hg update 3.5 # and hg update default > make distclean && ./configure --with-pydebug && make -j2 > cd Doc > make doctest > -------------------------------------------------------- > > and in both cases I get page after page of errors. I have tried > installing python-sphinx and python3-sphinx; I have tried adding > PYTHON=../python and PYTHON=python3 to the `make doctest` line -- all to > no avail. > > Here's a random sample of the errors: > > ********************************************************************** > File "library/shlex.rst", line ?, in default > Failed example: > remote_command > Exception raised: > Traceback (most recent call last): > File "/usr/lib/python2.7/doctest.py", line 1315, in __run > compileflags, 1) in test.globs > File "", line 1, in > remote_command > NameError: name 'remote_command' is not defined > > ********************************************************************** > File "howto/sorting.rst", line ?, in default > Failed example: > sorted([5, 2, 4, 1, 3], cmp=numeric_compare) > Exception raised: > Traceback (most recent call last): > File "/usr/lib/python2.7/doctest.py", line 1315, in __run > compileflags, 1) in test.globs > File "", line 1, in > sorted([5, 2, 4, 1, 3], cmp=numeric_compare) > NameError: name 'numeric_compare' is not defined > > ********************************************************************** > File "library/ipaddress.rst", line ?, in default > Failed example: > n2 = ip_network('192.0.2.1/32') > Exception raised: > Traceback (most recent call last): > File "/usr/lib/python2.7/doctest.py", line 1315, in __run > compileflags, 1) in test.globs > File "", line 1, in > n2 = ip_network('192.0.2.1/32') > NameError: name 'ip_network' is not defined > > -- > ~Ethan~ > From zachary.ware+pydev at gmail.com Fri Nov 13 10:42:41 2015 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Fri, 13 Nov 2015 09:42:41 -0600 Subject: [Python-Dev] doc tests failing In-Reply-To: <5645FDE0.9040304@stoneleaf.us> References: <5645FDE0.9040304@stoneleaf.us> Message-ID: On Fri, Nov 13, 2015 at 9:12 AM, Ethan Furman wrote: > What am I doing wrong? Expecting it to work :) `make doctest` is a Sphinx feature, not specific to our docs. Ideally someday it should work (and I'll add it to the Docs buildbot), but nobody has tried to make it work yet. Making it work might be a good large-ish project for newer contributors. -- Zach From rdmurray at bitdance.com Fri Nov 13 10:44:26 2015 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 13 Nov 2015 10:44:26 -0500 Subject: [Python-Dev] doc tests failing In-Reply-To: <5645FDE0.9040304@stoneleaf.us> References: <5645FDE0.9040304@stoneleaf.us> Message-ID: <20151113154427.07A63B18088@webabinitio.net> We don't have clean doctests for the docs. Patches welcome. At one point I had made the turtle doctests pass (it draws a bunch of stuff on the screen) because otherwise we don't have very many turtle tests, but I haven't checked it in a couple years. Hmm. We could list making the doc doctests pass as an activity for beginners in the devguide. On Fri, 13 Nov 2015 07:12:32 -0800, Ethan Furman wrote: > What am I doing wrong? > > I have tried: > > -------------------------------------------------------- > hg update 3.5 # and hg update default > make distclean && ./configure --with-pydebug && make -j2 > cd Doc > make doctest > -------------------------------------------------------- > > and in both cases I get page after page of errors. I have tried > installing python-sphinx and python3-sphinx; I have tried adding > PYTHON=../python and PYTHON=python3 to the `make doctest` line -- all to > no avail. > > Here's a random sample of the errors: > > ********************************************************************** > File "library/shlex.rst", line ?, in default > Failed example: > remote_command > Exception raised: > Traceback (most recent call last): > File "/usr/lib/python2.7/doctest.py", line 1315, in __run > compileflags, 1) in test.globs > File "", line 1, in > remote_command > NameError: name 'remote_command' is not defined > > ********************************************************************** > File "howto/sorting.rst", line ?, in default > Failed example: > sorted([5, 2, 4, 1, 3], cmp=numeric_compare) > Exception raised: > Traceback (most recent call last): > File "/usr/lib/python2.7/doctest.py", line 1315, in __run > compileflags, 1) in test.globs > File "", line 1, in > sorted([5, 2, 4, 1, 3], cmp=numeric_compare) > NameError: name 'numeric_compare' is not defined > > ********************************************************************** > File "library/ipaddress.rst", line ?, in default > Failed example: > n2 = ip_network('192.0.2.1/32') > Exception raised: > Traceback (most recent call last): > File "/usr/lib/python2.7/doctest.py", line 1315, in __run > compileflags, 1) in test.globs > File "", line 1, in > n2 = ip_network('192.0.2.1/32') > NameError: name 'ip_network' is not defined > > -- > ~Ethan~ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com From status at bugs.python.org Fri Nov 13 12:08:33 2015 From: status at bugs.python.org (Python tracker) Date: Fri, 13 Nov 2015 18:08:33 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20151113170833.E47ED56881@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2015-11-06 - 2015-11-13) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5232 (+14) closed 32143 (+35) total 37375 (+49) Open issues with patches: 2307 Issues opened (36) ================== #7759: mhlib fails on Btrfs filesystem (test_mhlib failure) http://bugs.python.org/issue7759 reopened by serhiy.storchaka #25570: urllib.request > Request.add_header("abcd","efgh") fails with http://bugs.python.org/issue25570 opened by crickert #25571: Improve the lltrace feature with the Py_Debug mode http://bugs.python.org/issue25571 opened by matrixise #25572: _ssl doesn't build on OSX 10.11 http://bugs.python.org/issue25572 opened by matrixise #25573: FrameSummary repr() does not support previously working uses o http://bugs.python.org/issue25573 opened by zopieux #25574: 2.7 incorrectly documents objects hash as equal to id http://bugs.python.org/issue25574 opened by wim.glenn #25576: Remove ???Content-Type: application/x-www-form-urlencoded; cha http://bugs.python.org/issue25576 opened by martin.panter #25578: Memory leak in SSLSocket.getpeercert() with 0-length AIA exten http://bugs.python.org/issue25578 opened by alex #25580: async and await missing from token list http://bugs.python.org/issue25580 opened by SilentGhost #25582: Fixed memory leaks in test_ctypes http://bugs.python.org/issue25582 opened by serhiy.storchaka #25583: os.makedirs with exist_ok=True raises PermissionError on Windo http://bugs.python.org/issue25583 opened by plachotich #25585: Bad path leads to: ImportError: DLL load failed: %1 is not a v http://bugs.python.org/issue25585 opened by lac #25586: socket.sendall broken when a socket has a timeout http://bugs.python.org/issue25586 opened by jstasiak #25588: Run test suite from IDLE idlelib.run subprocess http://bugs.python.org/issue25588 opened by terry.reedy #25590: tab-completition on instances repeatedly accesses attribute/de http://bugs.python.org/issue25590 opened by ezio.melotti #25591: improve test coverage for the imaplib http://bugs.python.org/issue25591 opened by maciej.szulik #25592: distutils docs: data_files always uses sys.prefix http://bugs.python.org/issue25592 opened by jdemeyer #25593: _sock_connect_cb can be called twice resulting in InvalidState http://bugs.python.org/issue25593 opened by thehesiod #25594: enum instance attribute access possible http://bugs.python.org/issue25594 opened by SilentGhost #25596: regular files handled as directories in the glob module http://bugs.python.org/issue25596 opened by xdegaye #25597: unittest.mock does not wrap dict objects correctly http://bugs.python.org/issue25597 opened by Darragh Bailey #25599: asyncio.iscoroutinefunction returns unexpected results when pr http://bugs.python.org/issue25599 opened by luhn #25601: test_cpickle failure on the ware-gentoo-x86 buildbot http://bugs.python.org/issue25601 opened by serhiy.storchaka #25602: Add support for EVFILT_USER kqueue filter http://bugs.python.org/issue25602 opened by jceel #25604: [Minor] bug in integer true division algorithm http://bugs.python.org/issue25604 opened by mark.dickinson #25606: asyncio doc: 'socket' transport extra info is not mandatory http://bugs.python.org/issue25606 opened by haypo #25608: ascynio readexactly() should raise ValueError if passed length http://bugs.python.org/issue25608 opened by mmarkk #25609: Add a ContextManager ABC and type http://bugs.python.org/issue25609 opened by brett.cannon #25610: Add typing.Awaitable http://bugs.python.org/issue25610 opened by brett.cannon #25611: test_deque failure on Gentoo and OpenIndiana buildbots on 2.7 http://bugs.python.org/issue25611 opened by serhiy.storchaka #25612: nested try..excepts don't work correctly for generators http://bugs.python.org/issue25612 opened by yselivanov #25613: fix ssl tests with sslv3 disabled http://bugs.python.org/issue25613 opened by doko #25614: Lib/code.py: InteractiveConsole.raw_input writes prompt to std http://bugs.python.org/issue25614 opened by Yclept.Nemo #25615: Document unsorted behaviour of glob.glob http://bugs.python.org/issue25615 opened by Dave Jones #25616: Extract OrderedDict tests into separate file http://bugs.python.org/issue25616 opened by serhiy.storchaka #25617: Installing local installation of Python http://bugs.python.org/issue25617 opened by koustavpal88 Most recent 15 issues with no replies (15) ========================================== #25617: Installing local installation of Python http://bugs.python.org/issue25617 #25616: Extract OrderedDict tests into separate file http://bugs.python.org/issue25616 #25615: Document unsorted behaviour of glob.glob http://bugs.python.org/issue25615 #25614: Lib/code.py: InteractiveConsole.raw_input writes prompt to std http://bugs.python.org/issue25614 #25611: test_deque failure on Gentoo and OpenIndiana buildbots on 2.7 http://bugs.python.org/issue25611 #25610: Add typing.Awaitable http://bugs.python.org/issue25610 #25602: Add support for EVFILT_USER kqueue filter http://bugs.python.org/issue25602 #25592: distutils docs: data_files always uses sys.prefix http://bugs.python.org/issue25592 #25591: improve test coverage for the imaplib http://bugs.python.org/issue25591 #25588: Run test suite from IDLE idlelib.run subprocess http://bugs.python.org/issue25588 #25578: Memory leak in SSLSocket.getpeercert() with 0-length AIA exten http://bugs.python.org/issue25578 #25574: 2.7 incorrectly documents objects hash as equal to id http://bugs.python.org/issue25574 #25566: asyncio reference cycles after ConnectionResetError http://bugs.python.org/issue25566 #25560: Unhandled warning in test_unicode_file http://bugs.python.org/issue25560 #25551: Event's test_reset_internal_locks too fragile http://bugs.python.org/issue25551 Most recent 15 issues waiting for review (15) ============================================= #25615: Document unsorted behaviour of glob.glob http://bugs.python.org/issue25615 #25612: nested try..excepts don't work correctly for generators http://bugs.python.org/issue25612 #25602: Add support for EVFILT_USER kqueue filter http://bugs.python.org/issue25602 #25596: regular files handled as directories in the glob module http://bugs.python.org/issue25596 #25594: enum instance attribute access possible http://bugs.python.org/issue25594 #25593: _sock_connect_cb can be called twice resulting in InvalidState http://bugs.python.org/issue25593 #25592: distutils docs: data_files always uses sys.prefix http://bugs.python.org/issue25592 #25591: improve test coverage for the imaplib http://bugs.python.org/issue25591 #25590: tab-completition on instances repeatedly accesses attribute/de http://bugs.python.org/issue25590 #25586: socket.sendall broken when a socket has a timeout http://bugs.python.org/issue25586 #25583: os.makedirs with exist_ok=True raises PermissionError on Windo http://bugs.python.org/issue25583 #25582: Fixed memory leaks in test_ctypes http://bugs.python.org/issue25582 #25580: async and await missing from token list http://bugs.python.org/issue25580 #25578: Memory leak in SSLSocket.getpeercert() with 0-length AIA exten http://bugs.python.org/issue25578 #25576: Remove ???Content-Type: application/x-www-form-urlencoded; cha http://bugs.python.org/issue25576 Top 10 most discussed issues (10) ================================= #25593: _sock_connect_cb can be called twice resulting in InvalidState http://bugs.python.org/issue25593 27 msgs #25570: urllib.request > Request.add_header("abcd","efgh") fails with http://bugs.python.org/issue25570 12 msgs #25583: os.makedirs with exist_ok=True raises PermissionError on Windo http://bugs.python.org/issue25583 12 msgs #25586: socket.sendall broken when a socket has a timeout http://bugs.python.org/issue25586 12 msgs #25585: Bad path leads to: ImportError: DLL load failed: %1 is not a v http://bugs.python.org/issue25585 8 msgs #7759: mhlib fails on Btrfs filesystem (test_mhlib failure) http://bugs.python.org/issue7759 7 msgs #24999: ICC compiler: ICC treats denormal floating point numbers as 0. http://bugs.python.org/issue24999 7 msgs #25590: tab-completition on instances repeatedly accesses attribute/de http://bugs.python.org/issue25590 7 msgs #23883: __all__ lists are incomplete http://bugs.python.org/issue23883 6 msgs #25179: PEP 498 f-strings need to be documented http://bugs.python.org/issue25179 6 msgs Issues closed (34) ================== #6598: calling email.utils.make_msgid frequently has a non-trivial pr http://bugs.python.org/issue6598 closed by serhiy.storchaka #7267: format method: c presentation type broken in 2.7 http://bugs.python.org/issue7267 closed by haypo #14350: Strange Exception from copying an iterator http://bugs.python.org/issue14350 closed by serhiy.storchaka #18010: pydoc search chokes on import errors http://bugs.python.org/issue18010 closed by martin.panter #21935: Implement AUTH command in smtpd. http://bugs.python.org/issue21935 closed by r.david.murray #24306: Backport py.exe to 3.4 http://bugs.python.org/issue24306 closed by steve.dower #24661: CGIHTTPServer: premature unescaping of query string http://bugs.python.org/issue24661 closed by martin.panter #24802: int and float constructing from non NUL-terminated buffer http://bugs.python.org/issue24802 closed by martin.panter #24805: Python installer having problem in installing Python for all u http://bugs.python.org/issue24805 closed by steve.dower #24848: Warts in UTF-7 error handling http://bugs.python.org/issue24848 closed by serhiy.storchaka #25064: Adjust tempfile documentation for bytes filename support http://bugs.python.org/issue25064 closed by martin.panter #25263: test_tkinter fails randomly on the buildbots "AMD64 Windows10" http://bugs.python.org/issue25263 closed by serhiy.storchaka #25313: IDLE: gracefully handle themes (or keysets, or ...) not presen http://bugs.python.org/issue25313 closed by terry.reedy #25446: smtplib.py AUTH LOGIN code messed up sending login and passwor http://bugs.python.org/issue25446 closed by r.david.murray #25462: Avoid repeated hash calculation in C implementation of Ordered http://bugs.python.org/issue25462 closed by serhiy.storchaka #25465: Pickle uses O(n) memory overhead http://bugs.python.org/issue25465 closed by serhiy.storchaka #25530: ssl: OP_NO_SSLv3 should always be set unless a user specifical http://bugs.python.org/issue25530 closed by python-dev #25555: Fix parser and AST: fill lineno and col_offset when compiling http://bugs.python.org/issue25555 closed by haypo #25558: Use static asserts in C code http://bugs.python.org/issue25558 closed by serhiy.storchaka #25569: Memory leak in SSLSocket.getpeercert() http://bugs.python.org/issue25569 closed by python-dev #25575: idle not working http://bugs.python.org/issue25575 closed by zach.ware #25577: os.path.dirname leave left quote and remove right one http://bugs.python.org/issue25577 closed by zach.ware #25579: def is not a keyword with tokenize.py http://bugs.python.org/issue25579 closed by SilentGhost #25581: Python -U raises error during site import http://bugs.python.org/issue25581 closed by r.david.murray #25584: a recursive glob pattern fails to list files in the current di http://bugs.python.org/issue25584 closed by serhiy.storchaka #25587: Useless addition in PyTuple_New http://bugs.python.org/issue25587 closed by haypo #25589: test_ascii_formatd fails on Mac when built with Intel compiler http://bugs.python.org/issue25589 closed by christopher.hogan #25595: test_deleted_cwd in test_importlib is failed on AIX http://bugs.python.org/issue25595 closed by serhiy.storchaka #25598: Fix memory_hex (#9951) for non-contiguous buffers http://bugs.python.org/issue25598 closed by skrah #25600: argparse, argument_default=argparse.SUPPRESS seems to have no http://bugs.python.org/issue25600 closed by r.david.murray #25603: spelling mistake - 26.1 typing http://bugs.python.org/issue25603 closed by python-dev #25605: fcntl doc: document exceptions raised on error for ioctl() and http://bugs.python.org/issue25605 closed by haypo #25607: test_distutils provokes unexpected output in test_shutil http://bugs.python.org/issue25607 closed by serhiy.storchaka #892902: problem with pickling newstyle class instances http://bugs.python.org/issue892902 closed by serhiy.storchaka From ethan at stoneleaf.us Fri Nov 13 13:01:40 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 13 Nov 2015 10:01:40 -0800 Subject: [Python-Dev] doc tests failing In-Reply-To: <5645FDE0.9040304@stoneleaf.us> References: <5645FDE0.9040304@stoneleaf.us> Message-ID: <56462584.1030100@stoneleaf.us> On 11/13/2015 07:12 AM, Ethan Furman wrote: > What am I doing wrong? On 11/13/2015 07:40 AM, Georg Brandl wrote: > Running "make doctest" in the first place. The doctests in the core > docs were not written to support that, and also never updated to do > so. On 11/13/2015 07:42 AM, Zachary Ware wrote: > Expecting it to work :) On 11/13/2015 07:44 AM, R. David Murray wrote: > We don't have clean doctests for the docs. Patches welcome. Ah, okay, that explains it. :) Which means I had tested 'enum.rst' manually back when I wrote it, and had forgotten. Thank you for the reminder! -- ~Ethan~ From andrew.svetlov at gmail.com Fri Nov 13 17:06:23 2015 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sat, 14 Nov 2015 00:06:23 +0200 Subject: [Python-Dev] Rietveld is broken Message-ID: When I try to press 'retry' on http://bugs.python.org/review/25074/ page and send a message server responds with 500 error. Going through the page for viewing diff for concrete file http://bugs.python.org/review/25074/diff/15535/Lib/test/test_asyncio/test_base_events.py with adding a comment works well. The behavior is very annoying -- I was forced to write missed message twice. -- Thanks, Andrew Svetlov From damien.p.george at gmail.com Fri Nov 13 17:32:07 2015 From: damien.p.george at gmail.com (Damien George) Date: Fri, 13 Nov 2015 22:32:07 +0000 Subject: [Python-Dev] Clarification of PEP 394 for scripts that run under Python 2 and 3 Message-ID: Hi python-dev, We have a Python script that runs correctly under Python 2.6, 2.7 and 3.3+. It is executed on a *nix system using the "python" executable (ie not python2 or python3 specifically). This works just fine for systems that have Python 2 installed, or 2 and 3, or just 3 and symlink "python" to "python3" (eg Arch Linux). But it fails for systems that have only Python 3 and do not create a "python" symlink, ie only "python3" exists as an executable. We thought that PEP 394 would come to the rescue here but it seems to be unclear on this point. In particular it says: - 4th point of the abstract: "so python should be used in the shebang line only for scripts that are source compatible with both Python 2 and 3" - 6th point of the recommendation section: "One exception to this is scripts that are deliberately written to be source compatible with both Python 2.x and 3.x. Such scripts may continue to use python on their shebang line without affecting their portability" - 8th point in the migration notes: "If these conventions are adhered to, it will become the case that the python command is only executed in an interactive manner as a user convenience, or to run scripts that are source compatible with both Python 2 and Python 3." Well, that's pretty clear to me: one can expect the "python" executable to be available to run scripts that are compatible with versions 2.x and 3.x. The confusion comes because there are systems that install Python 3 without creating a "python" symlink (hence breaking the above). And Guido said that "'python' should always be the same as 'python2'" (see https://mail.python.org/pipermail/python-dev/2014-September/136389.html). Further, Nick Coghlan seemed to agree that "when there's only python3 installed, there should be no /usr/bin/python" (see https://mail.python.org/pipermail/python-dev/2014-September/136527.html). My questions are: 1. What is the true intent of PEP 394 when only Python 3 is installed? Is "python" available or not to run scripts compatible with 2.x and 3.x? 2. Is it possible to write a shebang line that supports all variations of Python installations on *nix machines? 3. If the answer to 2 is no, then what is the recommended way to support all Python installations with one standalone script? Thanks! Regards, Damien. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Nov 13 17:57:27 2015 From: barry at python.org (Barry Warsaw) Date: Fri, 13 Nov 2015 17:57:27 -0500 Subject: [Python-Dev] Clarification of PEP 394 for scripts that run under Python 2 and 3 In-Reply-To: References: Message-ID: <20151113175727.338f192c@limelight.wooz.org> On Nov 13, 2015, at 10:32 PM, Damien George wrote: >1. What is the true intent of PEP 394 when only Python 3 is installed? Is >"python" available or not to run scripts compatible with 2.x and 3.x? > >2. Is it possible to write a shebang line that supports all variations of >Python installations on *nix machines? > >3. If the answer to 2 is no, then what is the recommended way to support >all Python installations with one standalone script? It's important to remember that PEP 394 is an informational PEP, still under active (if dormant) discussion. It is also just a recommendation, and can't force any downstream redistributors to do one thing or another. Still, the intent is to provide a set of guidelines for the majority of *nix distributors to (eventually) adopt. I'll also point you to the recently created linux-sig where this topic is being discussed. https://mail.python.org/pipermail/linux-sig/2015-October/000000.html As you've noticed, Arch took a particular approach, but speaking as part of the Debian/Ubuntu community, don't expect that ecosystem to go down the same path. It's highly unlikely /usr/bin/python will ever point to anything other than Python 2, even when only Python 3 is installed by default. That might change once Python 2.7 is actually EOL'd, and that won't happen for quite a long time. PEP 373 currently says there will be bug fix releases until 2020, and I'd expect security-only releases for some time after that. In general, the discussions on linux-sig and elsewhere is coalescing around a launcher-type approach, where you'd put something like /usr/bin/py in your shebang if your script really is bilingual. Then `py` can try to figure out what's available on your system, what your preference is, etc. and then execute your script using that version-specific binary. I would expect the launcher to eventually be provided by the upstream Python development community. AFAIK, there have been a few stabs at writing such a launcher, but nothing that's far enough to be usable. So for now, you don't have great options. Cheers, -Barry From storchaka at gmail.com Sat Nov 14 14:19:37 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 14 Nov 2015 21:19:37 +0200 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings Message-ID: For now UTF-16 and UTF-32 source encodings are not supported. There is a comment in Parser/tokenizer.c: /* Disable support for UTF-16 BOMs until a decision is made whether this needs to be supported. */ Can we make a decision whether this support will be added in foreseeable future (say in near 10 years), or no? Removing commented out and related code will help to refactor the tokenizer, and that can help to fix some existing bugs (e.g. issue14811, issue18961, issue20115 and may be others). Current tokenizing code is too tangled. If the support of UTF-16 and UTF-32 is planned, I'll take this to attention during refactoring. But in many places besides the tokenizer the ASCII compatible encoding of source files is expected. From victor.stinner at gmail.com Sat Nov 14 17:56:43 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 14 Nov 2015 23:56:43 +0100 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: Message-ID: These encodings are rarely used. I don't think that any text editor use them. Editors use ascii, latin1, utf8 and... all locale encoding. But I don't know any OS using UTF-16 as a locale encoding. UTF-32 wastes disk space. Ok, even if it exists, Python already accepts a very wide range of encoding. It is not worth to make the parser much more complex just to support encodings which are also never used (for .py files). Victor Le 14 nov. 2015 20:20, "Serhiy Storchaka" a ?crit : > For now UTF-16 and UTF-32 source encodings are not supported. There is a > comment in Parser/tokenizer.c: > > /* Disable support for UTF-16 BOMs until a decision > is made whether this needs to be supported. */ > > Can we make a decision whether this support will be added in foreseeable > future (say in near 10 years), or no? > > Removing commented out and related code will help to refactor the > tokenizer, and that can help to fix some existing bugs (e.g. issue14811, > issue18961, issue20115 and may be others). Current tokenizing code is too > tangled. > > If the support of UTF-16 and UTF-32 is planned, I'll take this to > attention during refactoring. But in many places besides the tokenizer the > ASCII compatible encoding of source files is expected. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sat Nov 14 17:57:58 2015 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 14 Nov 2015 14:57:58 -0800 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: Message-ID: <1447541878.551398.439918241.1E1CFA11@webmail.messagingengine.com> I agree that supporting UTF-16 doesn't seem terribly useful. Also, thank you for giving the tokenizer some love! On Sat, Nov 14, 2015, at 11:19, Serhiy Storchaka wrote: > For now UTF-16 and UTF-32 source encodings are not supported. There is a > comment in Parser/tokenizer.c: > > /* Disable support for UTF-16 BOMs until a decision > is made whether this needs to be supported. */ > > Can we make a decision whether this support will be added in foreseeable > future (say in near 10 years), or no? > > Removing commented out and related code will help to refactor the > tokenizer, and that can help to fix some existing bugs (e.g. issue14811, > issue18961, issue20115 and may be others). Current tokenizing code is > too tangled. > > If the support of UTF-16 and UTF-32 is planned, I'll take this to > attention during refactoring. But in many places besides the tokenizer > the ASCII compatible encoding of source files is expected. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org From storchaka at gmail.com Sat Nov 14 18:21:54 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 15 Nov 2015 01:21:54 +0200 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: Message-ID: On 15.11.15 00:56, Victor Stinner wrote: > These encodings are rarely used. I don't think that any text editor use > them. Editors use ascii, latin1, utf8 and... all locale encoding. But I > don't know any OS using UTF-16 as a locale encoding. UTF-32 wastes disk > space. AFAIK the standard Windows editor Notepad uses UTF-16. And I often encountered Windows resource files in UTF-16. UTF-16 was more popular than UTF-8 on Windows some time. If this horse is dead I'll throw it away. From v+python at g.nevcal.com Sat Nov 14 18:39:32 2015 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 14 Nov 2015 15:39:32 -0800 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: Message-ID: <5647C634.5030700@g.nevcal.com> On 11/14/2015 3:21 PM, Serhiy Storchaka wrote: > On 15.11.15 00:56, Victor Stinner wrote: >> These encodings are rarely used. I don't think that any text editor use >> them. Editors use ascii, latin1, utf8 and... all locale encoding. But I >> don't know any OS using UTF-16 as a locale encoding. UTF-32 wastes disk >> space. > > AFAIK the standard Windows editor Notepad uses UTF-16. And I often > encountered Windows resource files in UTF-16. UTF-16 was more popular > than UTF-8 on Windows some time. If this horse is dead I'll throw it away. Just use UTF-8, ignoring an optional leading BOM. If someone wants to use something else, they can write a "preprocessor" to convert it to UTF-8 for use by Python. -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Sat Nov 14 18:48:04 2015 From: random832 at fastmail.com (Random832) Date: Sat, 14 Nov 2015 18:48:04 -0500 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings References: Message-ID: Victor Stinner writes: > These encodings are rarely used. I don't think that any text editor > use them. MS Windows' Notepad can be made to use UTF-16. From steve.dower at python.org Sat Nov 14 20:06:47 2015 From: steve.dower at python.org (Steve Dower) Date: Sat, 14 Nov 2015 17:06:47 -0800 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: Message-ID: The native encoding on Windows has been UTF-16 since Windows NT. Obviously we've survived without Python tokenization support for a long time, but every API uses it. I've hit a few cases where it would have been handy for Python to be able to detect it, though nothing I couldn't work around. Saying it is rarely used is rather exposing your own unawareness though - it could arguably be the most commonly used encoding (depending on how you define "used"). Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Victor Stinner" Sent: ?11/?14/?2015 14:58 To: "Serhiy Storchaka" Cc: "python-dev at python.org" Subject: Re: [Python-Dev] Support of UTF-16 and UTF-32 source encodings These encodings are rarely used. I don't think that any text editor use them. Editors use ascii, latin1, utf8 and... all locale encoding. But I don't know any OS using UTF-16 as a locale encoding. UTF-32 wastes disk space. Ok, even if it exists, Python already accepts a very wide range of encoding. It is not worth to make the parser much more complex just to support encodings which are also never used (for .py files). Victor Le 14 nov. 2015 20:20, "Serhiy Storchaka" a ?crit : For now UTF-16 and UTF-32 source encodings are not supported. There is a comment in Parser/tokenizer.c: /* Disable support for UTF-16 BOMs until a decision is made whether this needs to be supported. */ Can we make a decision whether this support will be added in foreseeable future (say in near 10 years), or no? Removing commented out and related code will help to refactor the tokenizer, and that can help to fix some existing bugs (e.g. issue14811, issue18961, issue20115 and may be others). Current tokenizing code is too tangled. If the support of UTF-16 and UTF-32 is planned, I'll take this to attention during refactoring. But in many places besides the tokenizer the ASCII compatible encoding of source files is expected. _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Nov 14 20:15:28 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 15 Nov 2015 12:15:28 +1100 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: Message-ID: On Sun, Nov 15, 2015 at 12:06 PM, Steve Dower wrote: > The native encoding on Windows has been UTF-16 since Windows NT. Obviously > we've survived without Python tokenization support for a long time, but > every API uses it. > > I've hit a few cases where it would have been handy for Python to be able to > detect it, though nothing I couldn't work around. Saying it is rarely used > is rather exposing your own unawareness though - it could arguably be the > most commonly used encoding (depending on how you define "used"). What matters here is: How likely is it that an arbitrary Python script (or, say, "arbitrary text file") is encoded UTF-16 rather than something ASCII-compatible? I think even Notepad defaults to UTF-8 for files, now. The fact that it's sending text to the GUI subsystem in UTF-16 is immaterial here. Can the py.exe launcher handle a UTF-16 shebang? (I'm pretty sure Unix program loaders won't.) That alone might be a reason for strongly encouraging ASCII-compat encodings. ChrisA From v+python at g.nevcal.com Sat Nov 14 20:24:05 2015 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 14 Nov 2015 17:24:05 -0800 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: Message-ID: <5647DEB5.2000905@g.nevcal.com> On 11/14/2015 5:15 PM, Chris Angelico wrote: > Can the py.exe launcher handle a UTF-16 shebang? (I'm pretty sure Unix > program loaders won't.) That alone might be a reason for strongly > encouraging ASCII-compat encodings. That raises an interesting question about if py.exe can handle a leading UTF-8 BOM. I have my emacs-on-Windows configured to store UTF-8 without BOM, but Notepad would put a BOM when saving UTF-8, last I checked. -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Sat Nov 14 20:25:57 2015 From: random832 at fastmail.com (Random832) Date: Sat, 14 Nov 2015 20:25:57 -0500 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings References: Message-ID: Chris Angelico writes: > Can the py.exe launcher handle a UTF-16 shebang? (I'm pretty sure Unix > program loaders won't.) A lot of them can't handle UTF-8 with a BOM, either. > That alone might be a reason for strongly encouraging ASCII-compat > encodings. A "python" or "python3" or "env" executable in any particular location such as /usr/bin isn't technically guaranteed, either, it's just relied on as a "works 99% of the time" thing. There's a reasonable case to be made that transforming files in such a way as they get launched by python (which may require an encoding change to an ASCII-compatible encoding, or a wrapper script, or the python -x hack) is the responsibility of platform-specific installer code. From v+python at g.nevcal.com Sat Nov 14 20:27:06 2015 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 14 Nov 2015 17:27:06 -0800 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: Message-ID: <5647DF6A.7000900@g.nevcal.com> On 11/14/2015 5:15 PM, Chris Angelico wrote: > I think even Notepad defaults to UTF-8 for > files, now. Just installed Windows 10 on a new machine, and upgraded to the latest Windows 10 release, 1511. Notepad defaults to ANSI encoding, as I think it always has. UTF-8 is an option, and it does seem to try to notice the original encoding of the file, when editing old files, but when creating a new one.... ANSI. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Nov 14 20:37:54 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 15 Nov 2015 12:37:54 +1100 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: <5647DF6A.7000900@g.nevcal.com> References: <5647DF6A.7000900@g.nevcal.com> Message-ID: On Sun, Nov 15, 2015 at 12:27 PM, Glenn Linderman wrote: > Notepad defaults to ANSI encoding, as I think it always has. UTF-8 is an > option, and it does seem to try to notice the original encoding of the file, > when editing old files, but when creating a new one.... ANSI. Thanks. Is "ANSI" always an eight-bit ASCII-compatible encoding? ChrisA From v+python at g.nevcal.com Sat Nov 14 20:47:48 2015 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 14 Nov 2015 17:47:48 -0800 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: <5647DF6A.7000900@g.nevcal.com> Message-ID: <5647E444.3080701@g.nevcal.com> On 11/14/2015 5:37 PM, Chris Angelico wrote: > On Sun, Nov 15, 2015 at 12:27 PM, Glenn Linderman wrote: >> Notepad defaults to ANSI encoding, as I think it always has. UTF-8 is an >> option, and it does seem to try to notice the original encoding of the file, >> when editing old files, but when creating a new one.... ANSI. > Thanks. Is "ANSI" always an eight-bit ASCII-compatible encoding? I wouldn't trust an answer to this question that didn't come from someone that used Windows with Chinese, Japanese, or Korean, as their default language for the install. So I don't have a trustworthy answer to give. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Nov 14 20:59:32 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 15 Nov 2015 12:59:32 +1100 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: <5647E444.3080701@g.nevcal.com> References: <5647DF6A.7000900@g.nevcal.com> <5647E444.3080701@g.nevcal.com> Message-ID: On Sun, Nov 15, 2015 at 12:47 PM, Glenn Linderman wrote: > On 11/14/2015 5:37 PM, Chris Angelico wrote: > > On Sun, Nov 15, 2015 at 12:27 PM, Glenn Linderman > wrote: > > Notepad defaults to ANSI encoding, as I think it always has. UTF-8 is an > option, and it does seem to try to notice the original encoding of the file, > when editing old files, but when creating a new one.... ANSI. > > Thanks. Is "ANSI" always an eight-bit ASCII-compatible encoding? > > > I wouldn't trust an answer to this question that didn't come from someone > that used Windows with Chinese, Japanese, or Korean, as their default > language for the install. So I don't have a trustworthy answer to give. > Heh, yeah. But I'd trust an answer from Steve Dower :) ChrisA From random832 at fastmail.com Sat Nov 14 21:10:02 2015 From: random832 at fastmail.com (Random832) Date: Sat, 14 Nov 2015 21:10:02 -0500 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings References: <5647DF6A.7000900@g.nevcal.com> <5647E444.3080701@g.nevcal.com> Message-ID: Glenn Linderman writes: > On 11/14/2015 5:37 PM, Chris Angelico wrote: > > Thanks. Is "ANSI" always an eight-bit ASCII-compatible encoding? > > I wouldn't trust an answer to this question that didn't come from > someone that used Windows with Chinese, Japanese, or Korean, as their > default language for the install. So I don't have a trustworthy answer > to give. AFAIK (I haven't actually used it as a default language, but I do know some details of their encodings) There are two main "issues" with the windows CJK encodings regarding ASCII compatibility: - There is a different symbol (a currency symbol) at 0x5c. Sort of. Most unicode translations of it will treat it as a backslash, and users do expect it to work for things like \n, path separators, etc, but it displays as ? or ?. - Dual-byte characters can have ASCII bytes as their *second* byte. From steve at pearwood.info Sat Nov 14 21:19:44 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 15 Nov 2015 13:19:44 +1100 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: Message-ID: <20151115021941.GE2038@ando.pearwood.info> On Sat, Nov 14, 2015 at 09:19:37PM +0200, Serhiy Storchaka wrote: > If the support of UTF-16 and UTF-32 is planned, I'll take this to > attention during refactoring. But in many places besides the tokenizer > the ASCII compatible encoding of source files is expected. Perhaps another way of looking at this: Is it feasible to drop support for arbitrary encodings and just require UTF-8 (with or without a pseudo-BOM)? -- Steve From eryksun at gmail.com Sat Nov 14 21:57:51 2015 From: eryksun at gmail.com (eryksun) Date: Sat, 14 Nov 2015 20:57:51 -0600 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: Message-ID: On Sat, Nov 14, 2015 at 7:06 PM, Steve Dower wrote: > The native encoding on Windows has been UTF-16 since Windows NT. Obviously > we've survived without Python tokenization support for a long time, but > every API uses it. Windows 2000 was the first version to have broad support for UTF-16. Windows NT (1993) was released before UTF-16, so its Unicode support is limited to UCS-2. (Note that console windows still restrict each character cell to a single WCHAR character. So a non-BMP character encoded as a UTF-16 surrogate pair always appears as two box glyphs. Of course you can copy and paste from the console to a UTF-16 aware window just fine.) > I've hit a few cases where it would have been handy for Python to be able to > detect it, though nothing I couldn't work around. Can you elaborate some example cases? I can see using UTF-16 for the REPL in the Windows console, but a hypothetical WinConIO class could simply transcode to and from UTF-8. Drekin's win-unicode-console package works like this. From eryksun at gmail.com Sat Nov 14 21:59:01 2015 From: eryksun at gmail.com (eryksun) Date: Sat, 14 Nov 2015 20:59:01 -0600 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: Message-ID: On Sat, Nov 14, 2015 at 7:15 PM, Chris Angelico wrote: > Can the py.exe launcher handle a UTF-16 shebang? (I'm pretty sure Unix > program loaders won't.) That alone might be a reason for strongly > encouraging ASCII-compat encodings. The launcher supports shebangs encoded as UTF-8 (default), UTF-16 (LE/BE), and UTF-32 (LE/BE): https://hg.python.org/cpython/file/v3.5.0/PC/launcher.c#l1138 From stephen at xemacs.org Sun Nov 15 02:23:50 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 15 Nov 2015 16:23:50 +0900 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: Message-ID: <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp> Steve Dower writes: > Saying [UTF-16] is rarely used is rather exposing your own > unawareness though - it could arguably be the most commonly used > encoding (depending on how you define "used"). Because we're discussing the storage of .py files, the relevant definition is the one used by the Unicode Standard, of course: a text/plain stream intended to be manipulated by any conformant Unicode processor that claims to handle text/plain. File formats with in-band formatting codes and allowing embedded non-text content like Word, or operating system or stdlib APIs, don't count. Nor have I seen UTF-16 used in email or HTML since the unregretted days of Win2k betas[1] (but I don't frequent Windows- or Java-oriented sites, so I have to admit my experience is limited in a possibly relevant way). In Japan my impression is that modern versions of Windows have Memopad[sic] configured to emit UTF-8-with-signature by default for new files, and if not, the abomination known as Shift JIS (I'm not sure if that is a user or OEM option, though). Never a widechar encoding (after all, the whole point of Shift JIS was to use an 8-bit encoding for the katakana syllabary to save space or bandwidth). I think if anyone wants to use UTF-16 or UTF-32 for exchange of Python programs, they probably already know how to convert them to UTF-8. As somebody already suggested, this can be delegated to the py.exe launcher, if necessary, AFAICS. I don't see any good reason for allowing non-ASCII-compatible encodings in the reference CPython interpreter. However, having mentioned Windows and Java, I have to wonder about IronPython and Jython, respectively. Having never lived in either of those environments, I don't know what text encoding their users might prefer (or even occasionally encounter) in Python program source. Steve Footnotes: [1] The version of Outlook Express shipped with them would emit "HTML" mail with ASCII tags and UTF-8-encoded text (even if it was encodable in pure ASCII). No, it wasn't spam, either, so it probably really was Outlook Express as it claimed to be in one of the headers. From random832 at fastmail.com Sun Nov 15 06:51:12 2015 From: random832 at fastmail.com (Random832) Date: Sun, 15 Nov 2015 06:51:12 -0500 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings References: <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp> Message-ID: "Stephen J. Turnbull" writes: > I don't see any good reason for allowing non-ASCII-compatible > encodings in the reference CPython interpreter. There might be a case for having the tokenizer not care about encodings at all and just operate on a stream of unicode characters provided by a different layer. From p.f.moore at gmail.com Sun Nov 15 07:56:18 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 15 Nov 2015 12:56:18 +0000 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp> References: <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp> Message-ID: On 15 November 2015 at 07:23, Stephen J. Turnbull wrote: > I don't see any good reason for allowing non-ASCII-compatible > encodings in the reference CPython interpreter. >From PEP 263: Any encoding which allows processing the first two lines in the way indicated above is allowed as source code encoding, this includes ASCII compatible encodings as well as certain multi-byte encodings such as Shift_JIS. It does not include encodings which use two or more bytes for all characters like e.g. UTF-16. The reason for this is to keep the encoding detection algorithm in the tokenizer simple. So this pretty much confirms that double-byte encodings are not valid for Python source files. Paul From lac at openend.se Sun Nov 15 09:43:58 2015 From: lac at openend.se (Laura Creighton) Date: Sun, 15 Nov 2015 15:43:58 +0100 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp> Message-ID: <201511151443.tAFEhwfi029674@fido.openend.se> In a message of Sun, 15 Nov 2015 12:56:18 +0000, Paul Moore writes: >On 15 November 2015 at 07:23, Stephen J. Turnbull wrote: >> I don't see any good reason for allowing non-ASCII-compatible >> encodings in the reference CPython interpreter. > >>From PEP 263: > > Any encoding which allows processing the first two lines in the > way indicated above is allowed as source code encoding, this > includes ASCII compatible encodings as well as certain > multi-byte encodings such as Shift_JIS. It does not include > encodings which use two or more bytes for all characters like > e.g. UTF-16. The reason for this is to keep the encoding > detection algorithm in the tokenizer simple. > >So this pretty much confirms that double-byte encodings are not valid >for Python source files. > >Paul Steve Turnbull, who lives in Japan, and speaks and writes Japanese is saying that "he cannot see any reason for allowing non-ASCII compatible encodings in Cpython". This makes me wonder. Is this along the lines of 'even in Japan we do not want such things' or along the lines of 'when in Japan we want such things we want to so brutally do so much more, so keep the reference implementation simple, and don't try to help us with this seems-like-a-good-idea-but-isnt-in-practice' ideas like this one, or .... Laura From ja.py at farowl.co.uk Sun Nov 15 10:17:49 2015 From: ja.py at farowl.co.uk (Jeff Allen) Date: Sun, 15 Nov 2015 15:17:49 +0000 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp> References: <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp> Message-ID: <5648A21D.5050707@farowl.co.uk> I'm approaching this from the premise that we would like to avoid needless surprises for users not versed in text encoding. I did a simple experiment with notepad on Windows 7 as if a na?ve user. If I write the one-line program: print("Hello world.") # by Jeff It runs, no surprise. We may legitimately encounter Unicode in string literals and comments. If I write: print("j't'kif Ana?s!") # par Herv? and try to save it, notepad tells me this file "contains characters in Unicode format which will be lost if you save this as an ANSI encoded text file." To keep the Unicode information I should cancel and choose a Unicode option. In the "Save as" dialogue the default encoding is ANSI. The second option "Unicode" is clearly right as the warning said "Unicode" 3 times and I don't know what big-endian or UTF-8 mean. Good that worked. Closed and opened it looks exactly as I typed it. But the bytes I actually wrote on disk consist of a BOM and UTF-16-LE. And running it I get: File "bonjour.py", line 1 SyntaxError: Non-UTF-8 code starting with '\xff' in file bonjour.py on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details If I take the hint here and save as UTF-8, then it works, including printing the accent. Inspection of the bytes shows it starts with a UTF-8 BOM. In Jython I get the same results (choking on UTF-16), but saved as UTF-8, it works. I just have to make sure that's a Unicode constant if I want it to print correctly, as we're at 2.7. Jython has a checkered past with encodings, but tries to do exactly the same as CPython 2.7.x. Now, a fact I haven't mentioned is that my machine was localised to simplified Chinese (to diagnose some bug) during this test. If I re-localise to my usual English (UK), I do not get the guidance from notepad: instead it quietly saves as Latin-1 (cp1252), perhaps because I'm in Western Europe. Python baulks at this, at the first accented character. If I save from notepad as Unicode or UTF-8 the results are as before, including the BOM. In some circumstances, then, the natural result of using notepad and not sticking to ASCII may be UTF-16-LE with a BOM, or Latin-1 depending on localisation, it seems. The Python error message provides a clue what a user should do, but they would need some background, a helpful teacher, or the Internet to sort it out. Jeff Allen On 15/11/2015 07:23, Stephen J. Turnbull wrote: > Steve Dower writes: > > > Saying [UTF-16] is rarely used is rather exposing your own > > unawareness though - it could arguably be the most commonly used > > encoding (depending on how you define "used"). > > Because we're discussing the storage of .py files, the relevant > definition is the one used by the Unicode Standard, of course: a > text/plain stream intended to be manipulated by any conformant Unicode > processor that claims to handle text/plain. File formats with in-band > formatting codes and allowing embedded non-text content like Word, or > operating system or stdlib APIs, don't count. Nor have I seen UTF-16 > used in email or HTML since the unregretted days of Win2k betas[1] > (but I don't frequent Windows- or Java-oriented sites, so I have to > admit my experience is limited in a possibly relevant way). > > In Japan my impression is that modern versions of Windows have > Memopad[sic] configured to emit UTF-8-with-signature by default for > new files, and if not, the abomination known as Shift JIS (I'm not > sure if that is a user or OEM option, though). Never a widechar > encoding (after all, the whole point of Shift JIS was to use an 8-bit > encoding for the katakana syllabary to save space or bandwidth). > > I think if anyone wants to use UTF-16 or UTF-32 for exchange of Python > programs, they probably already know how to convert them to UTF-8. As > somebody already suggested, this can be delegated to the py.exe > launcher, if necessary, AFAICS. > > I don't see any good reason for allowing non-ASCII-compatible > encodings in the reference CPython interpreter. > > However, having mentioned Windows and Java, I have to wonder about > IronPython and Jython, respectively. Having never lived in either of > those environments, I don't know what text encoding their users might > prefer (or even occasionally encounter) in Python program source. > > Steve > > Footnotes: > [1] The version of Outlook Express shipped with them would emit > "HTML" mail with ASCII tags and UTF-8-encoded text (even if it was > encodable in pure ASCII). No, it wasn't spam, either, so it probably > really was Outlook Express as it claimed to be in one of the headers. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe:https://mail.python.org/mailman/options/python-dev/ja.py%40farowl.co.uk > From stephen at xemacs.org Sun Nov 15 11:40:09 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 16 Nov 2015 01:40:09 +0900 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: <201511151443.tAFEhwfi029674@fido.openend.se> References: <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp> <201511151443.tAFEhwfi029674@fido.openend.se> Message-ID: <22088.46441.171953.459003@turnbull.sk.tsukuba.ac.jp> Laura Creighton writes: > Steve Turnbull, who lives in Japan, and speaks and writes Japanese > is saying that "he cannot see any reason for allowing non-ASCII > compatible encodings in Cpython". > > This makes me wonder. > > Is this along the lines of 'even in Japan we do not want such > things' or along the lines of 'when in Japan we want such things > we want to so brutally do so much more, so keep the reference > implementation simple, and don't try to help us with this > seems-like-a-good-idea-but-isnt-in-practice' ideas like this one, > or > .... I'm saying that to my knowledge Japan is the most complicated place there is when it comes to encodings, and even so, nobody here seems to be using UTF-16 as the encoding for program sources (or any other text/* media). Of course as Steve Dower pointed out it's in heavy use as an internal text encoding, in OS APIs, in some languages' stdlib APIs (ie, Java and I suppose .NET), and I guess in single-application file formats (Word), but the programs that use those APIs are written in ASCII compatible-encodings (and Shift JIS and Big5). The Japanese don't need or want UTF-16 in text files, etc. Besides that, I can also say that PEP 263 didn't legislate the use of ASCII-compatible encodings. For one thing, Shift JIS and Big5 aren't 100% compatible because they uses 0x20-0x7f in multibyte characters. They're just close enough to ASCII compatible to mostly "just work", at least on Microsoft OSes provided by OEMs in the relevant countries. What PEP 263 did do was to specify that non-ASCII-compatible encodings are not supported by the PEP 263 mechanism for declaring the encoding of a Python source program. That's because it looks for a "magic number" which is the ASCII-encoded form of "coding:" in the first two lines. It doesn't rule out alternative mechanisms for encoding detection (specifically, use of the UTF-16 "BOM" signature); it just doesn't propose implementing them. IIRC nobody has ever asked for them, but I think the idea is absurd so I have to admit I may have seen a request and forgot it instantly. Bottom line: as long as Python (or the launcher) is able to transcode the source to the internal Unicode format (UTF-8 in Python 2, and widechar or PEP 393 in Python 3) before actually beginning parsing, any on-disk encoding is OK. But I just don't see a use case for UTF-16. If I'm wrong, I think that this feature should be added to launchers, not CPython, because it forces the decoder to know what formats other than ASCII are implemented and to try heuristics to guess, rather than just obeying the coding cookie. From stephen at xemacs.org Sun Nov 15 11:42:12 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 16 Nov 2015 01:42:12 +0900 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp> Message-ID: <22088.46564.623670.58229@turnbull.sk.tsukuba.ac.jp> Random832 writes: > "Stephen J. Turnbull" writes: > > I don't see any good reason for allowing non-ASCII-compatible > > encodings in the reference CPython interpreter. > > There might be a case for having the tokenizer not care about encodings > at all and just operate on a stream of unicode characters provided by a > different layer. That's exactly what the PEP 263 implementation does in Python 2 (with the caveat that Python 2 doesn't know anything about Unicode, it's a UTF-8 stream and the non-ASCII characters are treated as bytes of unknown semantics, so they can't be used in syntax). I don't know about Python 3, I haven't looked at the decoding of source programs. But I would assume it implements PEP 263 still, except that since str is now either widechars or PEP 393 encoding (ie, flexible widechars) that encoding is now used instead of UTF-8. I'm sure that there are plenty of ASCII-isms in the tokenizer in the sense that it assumes the ASCII *character* (not byte) repertoire. But I'm not sure why Serhiy thinks that the tokenizer cares about the representation on-disk. But as I say, I haven't looked at the code so he might be right. Steve From guido at python.org Sun Nov 15 12:34:50 2015 From: guido at python.org (Guido van Rossum) Date: Sun, 15 Nov 2015 09:34:50 -0800 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: <22088.46564.623670.58229@turnbull.sk.tsukuba.ac.jp> References: <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp> <22088.46564.623670.58229@turnbull.sk.tsukuba.ac.jp> Message-ID: Let me just unilaterally end this discussion. It's fine to disregard the future possibility of using UTF-16 or -32 for Python source code. Serhiy can happily rip out any comments or dead code dealing with that possibility. -- --Guido van Rossum (python.org/~guido) From p.f.moore at gmail.com Sun Nov 15 14:26:59 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 15 Nov 2015 19:26:59 +0000 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: <22088.46441.171953.459003@turnbull.sk.tsukuba.ac.jp> References: <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp> <201511151443.tAFEhwfi029674@fido.openend.se> <22088.46441.171953.459003@turnbull.sk.tsukuba.ac.jp> Message-ID: On 15 November 2015 at 16:40, Stephen J. Turnbull wrote: > What PEP 263 did do was to specify that non-ASCII-compatible encodings > are not supported by the PEP 263 mechanism for declaring the encoding > of a Python source program. That's because it looks for a "magic > number" which is the ASCII-encoded form of "coding:" in the first two > lines. It doesn't rule out alternative mechanisms for encoding > detection (specifically, use of the UTF-16 "BOM" signature); it just > doesn't propose implementing them. That was my initial thought. But combine this with the statement from the language docs that the default encoding when there is no PEP 263 encoding specification is UTF-8 (or ASCII in Python 2) and there's no valid way that I can see that a UTF-16 encoding could be valid (short of a formal language change). Anyway, Guido has spoken, so I'll leave it there. Paul From raymond.hettinger at gmail.com Sun Nov 15 14:57:39 2015 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 15 Nov 2015 11:57:39 -0800 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp> <22088.46564.623670.58229@turnbull.sk.tsukuba.ac.jp> Message-ID: <34F3CFE0-1F32-4BD2-A5DD-E77D2F6CC76B@gmail.com> > On Nov 15, 2015, at 9:34 AM, Guido van Rossum wrote: > > Let me just unilaterally end this discussion. It's fine to disregard > the future possibility of using UTF-16 or -32 for Python source code. > Serhiy can happily rip out any comments or dead code dealing with that > possibility. Thank you. Raymond From mal at egenix.com Sun Nov 15 15:38:45 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 15 Nov 2015 21:38:45 +0100 Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings In-Reply-To: References: Message-ID: <5648ED55.4020906@egenix.com> On 14.11.2015 23:56, Victor Stinner wrote: > These encodings are rarely used. I don't think that any text editor use > them. Editors use ascii, latin1, utf8 and... all locale encoding. But I > don't know any OS using UTF-16 as a locale encoding. UTF-32 wastes disk > space. UTF-16 is used a lot for Windows text files, e.g. Unicode CSV files (the save as "Unicode text file" option writes UTF-16). However, nowadays, all text editors also support UTF-8 and many of these recognize the UTF-8 BOM as identifier to detect Unicode text files. > Ok, even if it exists, Python already accepts a very wide range of > encoding. It is not worth to make the parser much more complex just to > support encodings which are also never used (for .py files). Agreed. In Python 2 we decided to only allow ASCII super-sets for Python source files, which out ruled multi-byte encodings such as UTF-16 and -32. I don't think we need to make the parser more complex just to support them. UTF-8 works fine as Python source code encoding. > Victor > Le 14 nov. 2015 20:20, "Serhiy Storchaka" a ?crit : > >> For now UTF-16 and UTF-32 source encodings are not supported. There is a >> comment in Parser/tokenizer.c: >> >> /* Disable support for UTF-16 BOMs until a decision >> is made whether this needs to be supported. */ >> >> Can we make a decision whether this support will be added in foreseeable >> future (say in near 10 years), or no? >> >> Removing commented out and related code will help to refactor the >> tokenizer, and that can help to fix some existing bugs (e.g. issue14811, >> issue18961, issue20115 and may be others). Current tokenizing code is too >> tangled. >> >> If the support of UTF-16 and UTF-32 is planned, I'll take this to >> attention during refactoring. But in many places besides the tokenizer the >> ASCII compatible encoding of source files is expected. >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com >> > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 15 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ 2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From ncoghlan at gmail.com Sun Nov 15 18:48:27 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 16 Nov 2015 09:48:27 +1000 Subject: [Python-Dev] Clarification of PEP 394 for scripts that run under Python 2 and 3 In-Reply-To: <20151113175727.338f192c@limelight.wooz.org> References: <20151113175727.338f192c@limelight.wooz.org> Message-ID: On 14 November 2015 at 08:57, Barry Warsaw wrote: > On Nov 13, 2015, at 10:32 PM, Damien George wrote: > >>1. What is the true intent of PEP 394 when only Python 3 is installed? Is >>"python" available or not to run scripts compatible with 2.x and 3.x? >> >>2. Is it possible to write a shebang line that supports all variations of >>Python installations on *nix machines? >> >>3. If the answer to 2 is no, then what is the recommended way to support >>all Python installations with one standalone script? > > It's important to remember that PEP 394 is an informational PEP, still under > active (if dormant) discussion. It is also just a recommendation, and can't > force any downstream redistributors to do one thing or another. Still, the > intent is to provide a set of guidelines for the majority of *nix distributors > to (eventually) adopt. > > I'll also point you to the recently created linux-sig where this topic is > being discussed. > > https://mail.python.org/pipermail/linux-sig/2015-October/000000.html > > As you've noticed, Arch took a particular approach, but speaking as part of > the Debian/Ubuntu community, don't expect that ecosystem to go down the same > path. It's highly unlikely /usr/bin/python will ever point to anything other > than Python 2, even when only Python 3 is installed by default. That might > change once Python 2.7 is actually EOL'd, and that won't happen for quite a > long time. PEP 373 currently says there will be bug fix releases until 2020, > and I'd expect security-only releases for some time after that. This is the same situation for the Fedora/RHEL/CentOS ecosystem - for the time being, we think failure -> "yum install /usr/bin/python" is the cleanest user experience we can readily offer for Python 3 only systems. However, we don't think it's an acceptable long term answer, and something *else* needs to be done with /usr/bin/python than just making it a symlink to Python 3 (since the latter will lead to cryptic error messages for Python 2 only scripts). > In general, the discussions on linux-sig and elsewhere is coalescing around a > launcher-type approach, where you'd put something like /usr/bin/py in your > shebang if your script really is bilingual. Then `py` can try to figure out > what's available on your system, what your preference is, etc. and then > execute your script using that version-specific binary. I would expect the > launcher to eventually be provided by the upstream Python development > community. A couple of other links folks may find worth reading for background: * LWN summary of the discussion at this year's language summit: https://lwn.net/Articles/640296/ * Geoffrey Thomas's launcher concept: https://ldpreload.com/blog/usr-bin-python-23 This thread prompted me to do an update pass over https://wiki.python.org/moin/Python3LinuxDistroPortingStatus, adding these background links to the Python symlink section, and updating the Fedora porting status link to refer to the recently published porting DB: http://portingdb-encukou.rhcloud.com/ Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brett at python.org Mon Nov 16 15:18:02 2015 From: brett at python.org (Brett Cannon) Date: Mon, 16 Nov 2015 20:18:02 +0000 Subject: [Python-Dev] Benchmark results across all major Python implementations Message-ID: I gave the opening keynote at PyCon CA and then gave the same talk at PyData NYC on the various interpreters of Python (Jupyter notebook of my presentation can be found at bit.ly/pycon-ca-keynote; no video yet). I figured people here might find the benchmark numbers interesting so I'm sharing the link here. I'm still hoping someday speed.python.org becomes a thing so I never have to spend so much time benchmarking so may Python implementations ever again and this sort of thing is just part of what we do to keep the implementation ecosystem healthy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Mon Nov 16 15:23:49 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 16 Nov 2015 21:23:49 +0100 Subject: [Python-Dev] Benchmark results across all major Python implementations In-Reply-To: References: Message-ID: Hi Brett Any thoughts on improving the benchmark set (I think all of {cpython,pypy,pyston} introduced new benchmarks to the set). "speed.python.org" becoming a thing is generally stopped on "noone cares enough to set it up". Cheers, fijal On Mon, Nov 16, 2015 at 9:18 PM, Brett Cannon wrote: > I gave the opening keynote at PyCon CA and then gave the same talk at PyData > NYC on the various interpreters of Python (Jupyter notebook of my > presentation can be found at bit.ly/pycon-ca-keynote; no video yet). I > figured people here might find the benchmark numbers interesting so I'm > sharing the link here. > > I'm still hoping someday speed.python.org becomes a thing so I never have to > spend so much time benchmarking so may Python implementations ever again and > this sort of thing is just part of what we do to keep the implementation > ecosystem healthy. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > From rdmurray at bitdance.com Mon Nov 16 15:47:49 2015 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 16 Nov 2015 15:47:49 -0500 Subject: [Python-Dev] Benchmark results across all major Python implementations In-Reply-To: References: Message-ID: <20151116204749.DBFE9B14091@webabinitio.net> On Mon, 16 Nov 2015 21:23:49 +0100, Maciej Fijalkowski wrote: > Any thoughts on improving the benchmark set (I think all of > {cpython,pypy,pyston} introduced new benchmarks to the set). > "speed.python.org" becoming a thing is generally stopped on "noone > cares enough to set it up". Actually, with some help from Intel, it is getting there. You can see the 'benchmarks' entry in the buildbot console: http://buildbot.python.org/all/console but it isn't quite working yet. We are also waiting on a review of the salt state: https://github.com/python/psf-salt/pull/74 (All work done by Zach Ware.) --David From brett at python.org Mon Nov 16 15:51:35 2015 From: brett at python.org (Brett Cannon) Date: Mon, 16 Nov 2015 20:51:35 +0000 Subject: [Python-Dev] Benchmark results across all major Python implementations In-Reply-To: References: Message-ID: On Mon, 16 Nov 2015 at 12:24 Maciej Fijalkowski wrote: > Hi Brett > > Any thoughts on improving the benchmark set (I think all of > {cpython,pypy,pyston} introduced new benchmarks to the set). > We should probably start a mailing list and finally hash out a common set of benchmarks that we all agree are reasonable for measuring performance. I think we all generally agree that high-level benchmarks are good and micro-benchmarks aren't that important for cross-implementation comparisons (obviously they have a usefulness when trying to work on a specific feature set, but that should be considered specific to an implementation and not to some globally accepted set of benchmarks). So we should identify benchmarks which somewhat represent real world workloads and try to have a balanced representation that doesn't lean one way or another (e.g., not all string manipulation or scientific computing, but both should obviously be represented). > "speed.python.org" becoming a thing is generally stopped on "noone > cares enough to set it up". > Oh, I know. I didn't say this could be considered wishful thinking since I know I have enough on my plate to prevent me from making it happen. -Brett > > Cheers, > fijal > > > On Mon, Nov 16, 2015 at 9:18 PM, Brett Cannon wrote: > > I gave the opening keynote at PyCon CA and then gave the same talk at > PyData > > NYC on the various interpreters of Python (Jupyter notebook of my > > presentation can be found at bit.ly/pycon-ca-keynote; no video yet). I > > figured people here might find the benchmark numbers interesting so I'm > > sharing the link here. > > > > I'm still hoping someday speed.python.org becomes a thing so I never > have to > > spend so much time benchmarking so may Python implementations ever again > and > > this sort of thing is just part of what we do to keep the implementation > > ecosystem healthy. > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Mon Nov 16 16:38:05 2015 From: brian at python.org (Brian Curtin) Date: Mon, 16 Nov 2015 16:38:05 -0500 Subject: [Python-Dev] Benchmark results across all major Python implementations In-Reply-To: References: Message-ID: On Monday, November 16, 2015, Brett Cannon > wrote: > > > On Mon, 16 Nov 2015 at 12:24 Maciej Fijalkowski wrote: > >> Hi Brett >> >> Any thoughts on improving the benchmark set (I think all of >> {cpython,pypy,pyston} introduced new benchmarks to the set). >> > > We should probably start a mailing list > There is/was a speed at python.org list. > "speed.python.org" becoming a thing is generally stopped on "noone >> cares enough to set it up". >> > > Oh, I know. I didn't say this could be considered wishful thinking since I > know I have enough on my plate to prevent me from making it happen. > There was a grant given years ago to improve some of this stuff but I don't believe the work ever saw the light of day. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Mon Nov 16 17:16:25 2015 From: brian at python.org (Brian Curtin) Date: Mon, 16 Nov 2015 17:16:25 -0500 Subject: [Python-Dev] Benchmark results across all major Python implementations In-Reply-To: References: Message-ID: On Monday, November 16, 2015, Brett Cannon > wrote: > > > On Mon, 16 Nov 2015 at 12:24 Maciej Fijalkowski wrote: > >> Hi Brett >> >> Any thoughts on improving the benchmark set (I think all of >> {cpython,pypy,pyston} introduced new benchmarks to the set). >> > > We should probably start a mailing list > There is/was a speed at python.org list. > "speed.python.org" becoming a thing is generally stopped on "noone >> cares enough to set it up". >> > > Oh, I know. I didn't say this could be considered wishful thinking since I > know I have enough on my plate to prevent me from making it happen. > There was a grant given years ago to improve some of this stuff but I don't believe the work ever saw the light of day. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.ware+pydev at gmail.com Mon Nov 16 17:23:51 2015 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Mon, 16 Nov 2015 16:23:51 -0600 Subject: [Python-Dev] Benchmark results across all major Python implementations In-Reply-To: References: Message-ID: On Mon, Nov 16, 2015 at 3:38 PM, Brian Curtin wrote: > On Monday, November 16, 2015, Brett Cannon wrote: >>> >>> Hi Brett >>> >>> Any thoughts on improving the benchmark set (I think all of >>> {cpython,pypy,pyston} introduced new benchmarks to the set). >> >> >> We should probably start a mailing list > > > There is/was a speed at python.org list. Is, but seems to be heavily moderated without active moderation. I sent an offer to speed-owner this morning to help with moderation, but as yet no response. I know I have a couple of messages waiting in the queue :) On Mon, 16 Nov 2015 at 12:24 Maciej Fijalkowski wrote: > "speed.python.org" becoming a thing is generally stopped on "noone > cares enough to set it up". Setup is done in my dev environment, it's now blocking on people more qualified than me to review and merge, then final tweaking of the buildbot setup. For those interested: The part in most need of review is the changes to the PSF Salt configuration to set up and run Codespeed on speed.python.org. The changes can be found in PR #74 on the psf-salt Github repo[0]. The Codespeed instance is housed in a Github repo owned by me[1] currently. There's one small patch to the codespeed code (which was merged upstream this morning), the rest of the changes in my fork are adapting a copy of the sample_project to be our own instance. I've been told that the grant proposal from long ago expected the use of codespeed2 instead of codespeed. I chose codespeed over codespeed2 because it appeared to be easier to get set up and running (which may or may not have actually been true), but also because I've not seen codespeed2 in action anywhere. The final piece that could use review is the changes to our benchmark repository, currently available in a sandbox repo on hg.python.org[2]. My original plan had been to use PyPy's benchmark suite, but as you can tell from the logs of the existing buildslave, cpython doesn't run that suite especially well, and the cpython suite has the advantage of working with Python 2 and 3 out of the box. Please have a look, give it a try if you can, and let me know what needs improvement! [0] https://github.com/python/psf-salt/pull/74 [1] https://github.com/zware/codespeed [2] https://hg.python.org/sandbox/zware-benchmarks -- Zach From mail at timgolden.me.uk Mon Nov 16 17:34:02 2015 From: mail at timgolden.me.uk (Tim Golden) Date: Mon, 16 Nov 2015 22:34:02 +0000 Subject: [Python-Dev] Benchmark results across all major Python implementations In-Reply-To: References: Message-ID: <564A59DA.2070704@timgolden.me.uk> On 16/11/2015 22:23, Zachary Ware wrote: > On Mon, Nov 16, 2015 at 3:38 PM, Brian Curtin wrote: >> On Monday, November 16, 2015, Brett Cannon wrote: > >>>> >>>> Hi Brett >>>> >>>> Any thoughts on improving the benchmark set (I think all of >>>> {cpython,pypy,pyston} introduced new benchmarks to the set). >>> >>> >>> We should probably start a mailing list >> >> >> There is/was a speed at python.org list. > > Is, but seems to be heavily moderated without active moderation. I > sent an offer to speed-owner this morning to help with moderation, but > as yet no response. I know I have a couple of messages waiting in the > queue :) Just to help things along, I've added myself as list owner and released a bunch of messages. TJG From jim.baker at python.org Mon Nov 16 17:59:54 2015 From: jim.baker at python.org (Jim Baker) Date: Mon, 16 Nov 2015 15:59:54 -0700 Subject: [Python-Dev] Benchmark results across all major Python implementations In-Reply-To: References: Message-ID: Brett, Very cool, I'm glad to see that Jython's performance was competitive under most of these benchmarks. I would also be interested in joining the proposed mailing list. re elementtree - I assume the benchmarking is usually done with cElementTree. However Jython currently lacks a Java equivalent, so importing cElementTree just uses the pure Python version. Hence the significant performance difference of approx. 40x for etree_parse and 16x for etree_iterparse. - Jim On Mon, Nov 16, 2015 at 1:18 PM, Brett Cannon wrote: > I gave the opening keynote at PyCon CA and then gave the same talk at > PyData NYC on the various interpreters of Python (Jupyter notebook of my > presentation can be found at bit.ly/pycon-ca-keynote; no video yet). I > figured people here might find the benchmark numbers interesting so I'm > sharing the link here. > > I'm still hoping someday speed.python.org becomes a thing so I never have > to spend so much time benchmarking so may Python implementations ever again > and this sort of thing is just part of what we do to keep the > implementation ecosystem healthy. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/jbaker%40zyasoft.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 16 18:07:24 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 16 Nov 2015 15:07:24 -0800 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: Message-ID: So I dropped the ball on this too -- I was going to either have a look or tell you quickly to find a BDFL-delegate, but ended up doing neither. Skimming it now I really don't think I'm the right person to review this. Maybe you could ask Alex Gaynor? On Tue, Nov 10, 2015 at 4:47 PM, Nick Coghlan wrote: > Hi folks, > > I have a confession to make - I dropped the ball on the HTTPS > verification backport proposals in PEP 493, and let the upstream and > downstream approval processes get out of sequence. > > As a result, the RHEL 7.2 beta released back in September incorporates > the HTTPS verification feature backport based on the current PEP 493 > draft, even though that hasn't formally been pronounced as an Active > recommendation by python-dev yet. > > Accordingly, I'm belatedly submitting it for pronouncement now: > https://www.python.org/dev/peps/pep-0493/ > > There's currently no BDFL-Delegate assigned, so if Guido doesn't want > to handle it, we'll need to address that question first. > > Our last discussion back in July seemed to show that folks either > didn't care about the question (because they're using unmodified > upstream versions so the PEP didn't affect them), or else thought the > approach described in the PEP was reasonable, so I'm hoping the > consequences of my mistake won't be too severe. > > Regards, > Nick. > > P.S. I'm aware that this looks like presenting a fait accompli at a > point where it's too late to realistically say "No", but the truth is > that preparation for the Python in Education miniconf at PyCon > Australia ramped up immediately after the July discussion, and then I > personally got confused as to the scope of what was being included in > 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493 > being deferred to a later release, but it's actually the whole package > of 466+476+493). That's my fault for trying to keep track of too many > things at once (and thus failing at some of them), not anyone else's. > > ================================ > > PEP: 493 > Title: HTTPS verification recommendations for Python 2.7 redistributors > Version: $Revision$ > Last-Modified: $Date$ > Author: Nick Coghlan , > Robert Kuska , > Marc-Andr? Lemburg > Status: Draft > Type: Informational > Content-Type: text/x-rst > Created: 10-May-2015 > Post-History: 06-Jul-2015 > > > Abstract > ======== > > PEP 476 updated Python's default handling of HTTPS certificates to be > appropriate for communication over the public internet. The Python 2.7 long > term maintenance series was judged to be in scope for this change, with the > new behaviour introduced in the Python 2.7.9 maintenance release. > > This PEP provides recommendations to downstream redistributors wishing to > provide a smoother migration experience when helping their users to manage > this change in Python's default behaviour. > > > Rationale > ========= > > PEP 476 changed Python's default behaviour to better match the needs and > expectations of developers operating over the public internet, a category > which appears to include most new Python developers. It is the position of > the authors of this PEP that this was a correct decision. > > However, it is also the case that this change *does* cause problems for > infrastructure administrators operating private intranets that rely on > self-signed certificates, or otherwise encounter problems with the new default > certificate verification settings. > > The long term answer for such environments is to update their internal > certificate management to at least match the standards set by the public > internet, but in the meantime, it is desirable to offer these administrators > a way to continue receiving maintenance updates to the Python 2.7 series, > without having to gate that on upgrades to their certificate management > infrastructure. > > PEP 476 did attempt to address this question, by covering how to revert the > new settings process wide by monkeypatching the ``ssl`` module to restore the > old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed > to allow system administrators to disable the feature by default in their > Standard Operating Environment definition has been determined to be > insufficient in at least some cases. The specific case of interest to the > authors of this PEP is the one where a Linux distributor aims to provide > their users with a > `smoother migration path > `__ > than the standard one provided by consuming upstream CPython 2.7 releases > directly, but other potential challenges have also been pointed out with > updating embedded Python runtimes and other user level installations of Python. > > Rather than allowing a plethora of mutually incompatibile migration techniques > to bloom, this PEP proposes two alternative approaches that redistributors > may take when addressing these problems. Redistributors may choose to implement > one, both, or neither of these approaches based on their assessment of the > needs of their particular userbase. > > These designs are being proposed as a recommendation for redistributors, rather > than as new upstream features, as they are needed purely to support legacy > environments migrating from older versions of Python 2.7. Neither approach > is being proposed as an upstream Python 2.7 feature, nor as a feature in any > version of Python 3 (whether published directly by the Python Software > Foundation or by a redistributor). > > > Requirements for capability detection > ===================================== > > As these recommendations are intended to cover backports to earlier Python > versions, the Python version number cannot be used as a reliable means for > detecting them. Instead, the recommendations are defined to allow the presence > or absence of the feature to be determined using the following technique:: > > python -c "import ssl; ssl._relevant_attribute" > > This will fail with `AttributeError` (and hence a non-zero return code) if the > relevant capability is not available. > > The marker attributes are prefixed with an underscore to indicate the > implementation dependent nature of these capabilities - not all Python > distributions will offer them, only those that are providing a multi-stage > migration process from the legacy HTTPS handling to the new default behaviour. > > > Recommendation for an environment variable based security downgrade > =================================================================== > > Some redistributors may wish to provide a per-application option to disable > certificate verification in selected applications that run on or embed CPython > without needing to modify the application itself. > > In these cases, a configuration mechanism is needed that provides: > > * an opt-out model that allows certificate verification to be selectively > turned off for particular applications after upgrading to a version of > Python that verifies certificates by default > * the ability for all users to configure this setting on a per-application > basis, rather than on a per-system, or per-Python-installation basis > > This approach may be used for any redistributor provided version of Python 2.7, > including those that advertise themselves as providing Python 2.7.9 or later. > > > Required marker attribute > ------------------------- > > The required marker attribute on the ``ssl`` module when implementing this > recommendation is:: > > _https_verify_envvar = 'PYTHONHTTPSVERIFY' > > This not only makes it straightforward to detect the presence (or absence) of > the capability, it also makes it possible to programmatically determine the > relevant environment variable name. > > > Recommended modifications to the Python standard library > -------------------------------------------------------- > > The recommended approach to providing a per-application configuration setting > for HTTPS certificate verification that doesn't require modifications to the > application itself is to: > > * modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment > variable when the module is first imported into a Python process > * set the ``ssl._create_default_https_context`` function to be an alias for > ``ssl._create_unverified_context`` if this environment variable is present > and set to ``'0'`` > * otherwise, set the ``ssl._create_default_https_context`` function to be an > alias for ``ssl.create_default_context`` as usual > > > Example implementation > ---------------------- > > :: > > _https_verify_envvar = 'PYTHONHTTPSVERIFY' > > def _get_https_context_factory(): > config_setting = os.environ.get(_https_verify_envvar) > if config_setting == '0': > return _create_unverified_context > return create_default_context > > _create_default_https_context = _get_https_context_factory() > > > Security Considerations > ----------------------- > > Relative to an unmodified version of CPython 2.7.9 or later, this approach > does introduce a new downgrade attack against the default security settings > that potentially allows a sufficiently determined attacker to revert Python > to the vulnerable configuration used in CPython 2.7.8 and earlier releases. > However, such an attack requires the ability to modify the execution > environment of a Python process prior to the import of the ``ssl`` module, > and any attacker with such access would already be able to modify the > behaviour of the underlying OpenSSL implementation. > > > Recommendation for backporting to earlier Python versions > ========================================================= > > Some redistributors, most notably Linux distributions, may choose to backport > the PEP 476 HTTPS verification changes to modified Python versions based on > earlier Python 2 maintenance releases. In these cases, a configuration > mechanism is needed that provides: > > * an opt-in model that allows the decision to enable HTTPS certificate > verification to be made independently of the decision to upgrade to the > Python version where the feature was first backported > * the ability for system administrators to set the default behaviour of Python > applications and scripts run directly in the system Python installation > * the ability for the redistributor to consider changing the default behaviour > of *new* installations at some point in the future without impacting existing > installations that have been explicitly configured to skip verifying HTTPS > certificates by default > > This approach should not be used for any Python installation that advertises > itself as providing Python 2.7.9 or later, as most Python users will have the > reasonable expectation that all such environments will validate HTTPS > certificates by default. > > > Required marker attribute > ------------------------- > > The required marker attribute on the ``ssl`` module when implementing this > recommendation is:: > > _cert_verification_config = '' > > This not only makes it straightforward to detect the presence (or absence) of > the capability, it also makes it possible to programmatically determine the > relevant configuration file name. > > > Recommended modifications to the Python standard library > -------------------------------------------------------- > > The recommended approach to backporting the PEP 476 modifications to an earlier > point release is to implement the following changes relative to the default > PEP 476 behaviour implemented in Python 2.7.9+: > > * modify the ``ssl`` module to read a system wide configuration file when the > module is first imported into a Python process > * define a platform default behaviour (either verifying or not verifying HTTPS > certificates) to be used if this configuration file is not present > * support selection between the following three modes of operation: > > * ensure HTTPS certificate verification is enabled > * ensure HTTPS certificate verification is disabled > * delegate the decision to the redistributor providing this Python version > > * set the ``ssl._create_default_https_context`` function to be an alias for > either ``ssl.create_default_context`` or ``ssl._create_unverified_context`` > based on the given configuration setting. > > > Recommended file location > ------------------------- > > This approach is currently only defined for \*nix system Python installations. > > The recommended configuration file name is > ``/etc/python/cert-verification.cfg``. > > The ``.cfg`` filename extension is recommended for consistency with the > ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library. > > > Recommended file format > ----------------------- > > The configuration file should use a ConfigParser ini-style format with a > single section named ``[https]`` containing one required setting ``verify``. > > Permitted values for ``verify`` are: > > * ``enable``: ensure HTTPS certificate verification is enabled by default > * ``disable``: ensure HTTPS certificate verification is disabled by default > * ``platform_default``: delegate the decision to the redistributor providing > this particular Python version > > If the ``[https]`` section or the ``verify`` setting are missing, or if the > ``verify`` setting is set to an unknown value, it should be treated as if the > configuration file is not present. > > > Example implementation > ---------------------- > > :: > > _cert_verification_config = '/etc/python/cert-verification.cfg' > > def _get_https_context_factory(): > # Check for a system-wide override of the default behaviour > context_factories = { > 'enable': create_default_context, > 'disable': _create_unverified_context, > 'platform_default': _create_unverified_context, # For now :) > } > import ConfigParser > config = ConfigParser.RawConfigParser() > config.read(_cert_verification_config) > try: > verify_mode = config.get('https', 'verify') > except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): > verify_mode = 'platform_default' > default_factory = context_factories.get('platform_default') > return context_factories.get(verify_mode, default_factory) > > _create_default_https_context = _get_https_context_factory() > > > Security Considerations > ----------------------- > > The specific recommendations for the backporting case are designed to work for > privileged, security sensitive processes, even those being run in the following > locked down configuration: > > * run from a locked down administrator controlled directory rather than a normal > user directory (preventing ``sys.path[0]`` based privilege escalation attacks) > * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based > privilege escalation attacks) > * run using the ``-s`` switch (preventing user site directory based privilege > escalation attacks) > * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege > escalation attacks) > > The intent is that the *only* reason HTTPS verification should be getting > turned off system wide when using this approach is because: > > * an end user is running a redistributor provided version of CPython rather > than running upstream CPython directly > * that redistributor has decided to provide a smoother migration path to > verifying HTTPS certificates by default than that being provided by the > upstream project > * either the redistributor or the local infrastructure administrator has > determined that it is appropriate to override the default upstream behaviour > (at least for the time being) > > Using an administrator controlled configuration file rather than an environment > variable has the essential feature of providing a smoother migration path, even > for applications being run with the ``-E`` switch. > > > Combining the recommendations > ============================= > > If a redistributor chooses to implement both recommendations, then the > environment variable should take precedence over the system-wide configuration > setting. This allows the setting to be changed for a given user, virtual > environment or application, regardless of the system-wide default behaviour. > > In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and > set to anything *other* than ``'0'``, then HTTPS certificate verification > should be enabled. > > Example implementation > ---------------------- > > :: > > _https_verify_envvar = 'PYTHONHTTPSVERIFY' > _cert_verification_config = '/etc/python/cert-verification.cfg' > > def _get_https_context_factory(): > # Check for am environmental override of the default behaviour > config_setting = os.environ.get(_https_verify_envvar) > if config_setting is not None: > if config_setting == '0': > return _create_unverified_context > return create_default_context > > # Check for a system-wide override of the default behaviour > context_factories = { > 'enable': create_default_context, > 'disable': _create_unverified_context, > 'platform_default': _create_unverified_context, # For now :) > } > import ConfigParser > config = ConfigParser.RawConfigParser() > config.read(_cert_verification_config) > try: > verify_mode = config.get('https', 'verify') > except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): > verify_mode = 'platform_default' > default_factory = context_factories.get('platform_default') > return context_factories.get(verify_mode, default_factory) > > _create_default_https_context = _get_https_context_factory() > > > Copyright > ========= > > This document has been placed into the public domain. > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From david.c.stewart at intel.com Mon Nov 16 18:37:06 2015 From: david.c.stewart at intel.com (Stewart, David C) Date: Mon, 16 Nov 2015 23:37:06 +0000 Subject: [Python-Dev] Benchmark results across all major Python implementations In-Reply-To: References: Message-ID: Last June we started publishing a daily performance report of the latest Python tip against the previous day's run and some established synch point. We mail these to the community to act as a "canary in the coal mine." I wrote about it at https://01.org/lp/blog/0-day-challenge-what-pulse-internet You can see our manager-style dashboard of a couple of key workloads at http://languagesperformance.intel.com/ (I have this running constantly on a dedicated screen in my office). Would love to get better workloads if we can. Dave From: Python-Dev on behalf of Brett Cannon Date: Monday, November 16, 2015 at 12:18 PM To: python-dev Subject: [Python-Dev] Benchmark results across all major Python implementations I gave the opening keynote at PyCon CA and then gave the same talk at PyData NYC on the various interpreters of Python (Jupyter notebook of my presentation can be found at bit.ly/pycon-ca-keynote; no video yet). I figured people here might find the benchmark numbers interesting so I'm sharing the link here. I'm still hoping someday speed.python.org becomes a thing so I never have to spend so much time benchmarking so may Python implementations ever again and this sort of thing is just part of what we do to keep the implementation ecosystem healthy. From ncoghlan at gmail.com Mon Nov 16 18:54:45 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 17 Nov 2015 09:54:45 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: Message-ID: On 17 November 2015 at 09:07, Guido van Rossum wrote: > So I dropped the ball on this too -- I was going to either have a look > or tell you quickly to find a BDFL-delegate, but ended up doing > neither. Skimming it now I really don't think I'm the right person to > review this. Maybe you could ask Alex Gaynor? If Alex is interested, that could definitely work. I've also asked Christian Heimes if he might be interested in handling it, as he worked on several of the SSL module changes that were backported, and he also works on FreeIPA [1] now, which will potentially be affected by the changes to RHEL & CentOS 7. Regards, Nick. [1] FreeIPA is an open source Identity Management & Authentication system: http://www.freeipa.org/ > > On Tue, Nov 10, 2015 at 4:47 PM, Nick Coghlan wrote: >> Hi folks, >> >> I have a confession to make - I dropped the ball on the HTTPS >> verification backport proposals in PEP 493, and let the upstream and >> downstream approval processes get out of sequence. >> >> As a result, the RHEL 7.2 beta released back in September incorporates >> the HTTPS verification feature backport based on the current PEP 493 >> draft, even though that hasn't formally been pronounced as an Active >> recommendation by python-dev yet. >> >> Accordingly, I'm belatedly submitting it for pronouncement now: >> https://www.python.org/dev/peps/pep-0493/ >> >> There's currently no BDFL-Delegate assigned, so if Guido doesn't want >> to handle it, we'll need to address that question first. >> >> Our last discussion back in July seemed to show that folks either >> didn't care about the question (because they're using unmodified >> upstream versions so the PEP didn't affect them), or else thought the >> approach described in the PEP was reasonable, so I'm hoping the >> consequences of my mistake won't be too severe. >> >> Regards, >> Nick. >> >> P.S. I'm aware that this looks like presenting a fait accompli at a >> point where it's too late to realistically say "No", but the truth is >> that preparation for the Python in Education miniconf at PyCon >> Australia ramped up immediately after the July discussion, and then I >> personally got confused as to the scope of what was being included in >> 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493 >> being deferred to a later release, but it's actually the whole package >> of 466+476+493). That's my fault for trying to keep track of too many >> things at once (and thus failing at some of them), not anyone else's. >> >> ================================ >> >> PEP: 493 >> Title: HTTPS verification recommendations for Python 2.7 redistributors >> Version: $Revision$ >> Last-Modified: $Date$ >> Author: Nick Coghlan , >> Robert Kuska , >> Marc-Andr? Lemburg >> Status: Draft >> Type: Informational >> Content-Type: text/x-rst >> Created: 10-May-2015 >> Post-History: 06-Jul-2015 >> >> >> Abstract >> ======== >> >> PEP 476 updated Python's default handling of HTTPS certificates to be >> appropriate for communication over the public internet. The Python 2.7 long >> term maintenance series was judged to be in scope for this change, with the >> new behaviour introduced in the Python 2.7.9 maintenance release. >> >> This PEP provides recommendations to downstream redistributors wishing to >> provide a smoother migration experience when helping their users to manage >> this change in Python's default behaviour. >> >> >> Rationale >> ========= >> >> PEP 476 changed Python's default behaviour to better match the needs and >> expectations of developers operating over the public internet, a category >> which appears to include most new Python developers. It is the position of >> the authors of this PEP that this was a correct decision. >> >> However, it is also the case that this change *does* cause problems for >> infrastructure administrators operating private intranets that rely on >> self-signed certificates, or otherwise encounter problems with the new default >> certificate verification settings. >> >> The long term answer for such environments is to update their internal >> certificate management to at least match the standards set by the public >> internet, but in the meantime, it is desirable to offer these administrators >> a way to continue receiving maintenance updates to the Python 2.7 series, >> without having to gate that on upgrades to their certificate management >> infrastructure. >> >> PEP 476 did attempt to address this question, by covering how to revert the >> new settings process wide by monkeypatching the ``ssl`` module to restore the >> old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed >> to allow system administrators to disable the feature by default in their >> Standard Operating Environment definition has been determined to be >> insufficient in at least some cases. The specific case of interest to the >> authors of this PEP is the one where a Linux distributor aims to provide >> their users with a >> `smoother migration path >> `__ >> than the standard one provided by consuming upstream CPython 2.7 releases >> directly, but other potential challenges have also been pointed out with >> updating embedded Python runtimes and other user level installations of Python. >> >> Rather than allowing a plethora of mutually incompatibile migration techniques >> to bloom, this PEP proposes two alternative approaches that redistributors >> may take when addressing these problems. Redistributors may choose to implement >> one, both, or neither of these approaches based on their assessment of the >> needs of their particular userbase. >> >> These designs are being proposed as a recommendation for redistributors, rather >> than as new upstream features, as they are needed purely to support legacy >> environments migrating from older versions of Python 2.7. Neither approach >> is being proposed as an upstream Python 2.7 feature, nor as a feature in any >> version of Python 3 (whether published directly by the Python Software >> Foundation or by a redistributor). >> >> >> Requirements for capability detection >> ===================================== >> >> As these recommendations are intended to cover backports to earlier Python >> versions, the Python version number cannot be used as a reliable means for >> detecting them. Instead, the recommendations are defined to allow the presence >> or absence of the feature to be determined using the following technique:: >> >> python -c "import ssl; ssl._relevant_attribute" >> >> This will fail with `AttributeError` (and hence a non-zero return code) if the >> relevant capability is not available. >> >> The marker attributes are prefixed with an underscore to indicate the >> implementation dependent nature of these capabilities - not all Python >> distributions will offer them, only those that are providing a multi-stage >> migration process from the legacy HTTPS handling to the new default behaviour. >> >> >> Recommendation for an environment variable based security downgrade >> =================================================================== >> >> Some redistributors may wish to provide a per-application option to disable >> certificate verification in selected applications that run on or embed CPython >> without needing to modify the application itself. >> >> In these cases, a configuration mechanism is needed that provides: >> >> * an opt-out model that allows certificate verification to be selectively >> turned off for particular applications after upgrading to a version of >> Python that verifies certificates by default >> * the ability for all users to configure this setting on a per-application >> basis, rather than on a per-system, or per-Python-installation basis >> >> This approach may be used for any redistributor provided version of Python 2.7, >> including those that advertise themselves as providing Python 2.7.9 or later. >> >> >> Required marker attribute >> ------------------------- >> >> The required marker attribute on the ``ssl`` module when implementing this >> recommendation is:: >> >> _https_verify_envvar = 'PYTHONHTTPSVERIFY' >> >> This not only makes it straightforward to detect the presence (or absence) of >> the capability, it also makes it possible to programmatically determine the >> relevant environment variable name. >> >> >> Recommended modifications to the Python standard library >> -------------------------------------------------------- >> >> The recommended approach to providing a per-application configuration setting >> for HTTPS certificate verification that doesn't require modifications to the >> application itself is to: >> >> * modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment >> variable when the module is first imported into a Python process >> * set the ``ssl._create_default_https_context`` function to be an alias for >> ``ssl._create_unverified_context`` if this environment variable is present >> and set to ``'0'`` >> * otherwise, set the ``ssl._create_default_https_context`` function to be an >> alias for ``ssl.create_default_context`` as usual >> >> >> Example implementation >> ---------------------- >> >> :: >> >> _https_verify_envvar = 'PYTHONHTTPSVERIFY' >> >> def _get_https_context_factory(): >> config_setting = os.environ.get(_https_verify_envvar) >> if config_setting == '0': >> return _create_unverified_context >> return create_default_context >> >> _create_default_https_context = _get_https_context_factory() >> >> >> Security Considerations >> ----------------------- >> >> Relative to an unmodified version of CPython 2.7.9 or later, this approach >> does introduce a new downgrade attack against the default security settings >> that potentially allows a sufficiently determined attacker to revert Python >> to the vulnerable configuration used in CPython 2.7.8 and earlier releases. >> However, such an attack requires the ability to modify the execution >> environment of a Python process prior to the import of the ``ssl`` module, >> and any attacker with such access would already be able to modify the >> behaviour of the underlying OpenSSL implementation. >> >> >> Recommendation for backporting to earlier Python versions >> ========================================================= >> >> Some redistributors, most notably Linux distributions, may choose to backport >> the PEP 476 HTTPS verification changes to modified Python versions based on >> earlier Python 2 maintenance releases. In these cases, a configuration >> mechanism is needed that provides: >> >> * an opt-in model that allows the decision to enable HTTPS certificate >> verification to be made independently of the decision to upgrade to the >> Python version where the feature was first backported >> * the ability for system administrators to set the default behaviour of Python >> applications and scripts run directly in the system Python installation >> * the ability for the redistributor to consider changing the default behaviour >> of *new* installations at some point in the future without impacting existing >> installations that have been explicitly configured to skip verifying HTTPS >> certificates by default >> >> This approach should not be used for any Python installation that advertises >> itself as providing Python 2.7.9 or later, as most Python users will have the >> reasonable expectation that all such environments will validate HTTPS >> certificates by default. >> >> >> Required marker attribute >> ------------------------- >> >> The required marker attribute on the ``ssl`` module when implementing this >> recommendation is:: >> >> _cert_verification_config = '' >> >> This not only makes it straightforward to detect the presence (or absence) of >> the capability, it also makes it possible to programmatically determine the >> relevant configuration file name. >> >> >> Recommended modifications to the Python standard library >> -------------------------------------------------------- >> >> The recommended approach to backporting the PEP 476 modifications to an earlier >> point release is to implement the following changes relative to the default >> PEP 476 behaviour implemented in Python 2.7.9+: >> >> * modify the ``ssl`` module to read a system wide configuration file when the >> module is first imported into a Python process >> * define a platform default behaviour (either verifying or not verifying HTTPS >> certificates) to be used if this configuration file is not present >> * support selection between the following three modes of operation: >> >> * ensure HTTPS certificate verification is enabled >> * ensure HTTPS certificate verification is disabled >> * delegate the decision to the redistributor providing this Python version >> >> * set the ``ssl._create_default_https_context`` function to be an alias for >> either ``ssl.create_default_context`` or ``ssl._create_unverified_context`` >> based on the given configuration setting. >> >> >> Recommended file location >> ------------------------- >> >> This approach is currently only defined for \*nix system Python installations. >> >> The recommended configuration file name is >> ``/etc/python/cert-verification.cfg``. >> >> The ``.cfg`` filename extension is recommended for consistency with the >> ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library. >> >> >> Recommended file format >> ----------------------- >> >> The configuration file should use a ConfigParser ini-style format with a >> single section named ``[https]`` containing one required setting ``verify``. >> >> Permitted values for ``verify`` are: >> >> * ``enable``: ensure HTTPS certificate verification is enabled by default >> * ``disable``: ensure HTTPS certificate verification is disabled by default >> * ``platform_default``: delegate the decision to the redistributor providing >> this particular Python version >> >> If the ``[https]`` section or the ``verify`` setting are missing, or if the >> ``verify`` setting is set to an unknown value, it should be treated as if the >> configuration file is not present. >> >> >> Example implementation >> ---------------------- >> >> :: >> >> _cert_verification_config = '/etc/python/cert-verification.cfg' >> >> def _get_https_context_factory(): >> # Check for a system-wide override of the default behaviour >> context_factories = { >> 'enable': create_default_context, >> 'disable': _create_unverified_context, >> 'platform_default': _create_unverified_context, # For now :) >> } >> import ConfigParser >> config = ConfigParser.RawConfigParser() >> config.read(_cert_verification_config) >> try: >> verify_mode = config.get('https', 'verify') >> except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): >> verify_mode = 'platform_default' >> default_factory = context_factories.get('platform_default') >> return context_factories.get(verify_mode, default_factory) >> >> _create_default_https_context = _get_https_context_factory() >> >> >> Security Considerations >> ----------------------- >> >> The specific recommendations for the backporting case are designed to work for >> privileged, security sensitive processes, even those being run in the following >> locked down configuration: >> >> * run from a locked down administrator controlled directory rather than a normal >> user directory (preventing ``sys.path[0]`` based privilege escalation attacks) >> * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based >> privilege escalation attacks) >> * run using the ``-s`` switch (preventing user site directory based privilege >> escalation attacks) >> * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege >> escalation attacks) >> >> The intent is that the *only* reason HTTPS verification should be getting >> turned off system wide when using this approach is because: >> >> * an end user is running a redistributor provided version of CPython rather >> than running upstream CPython directly >> * that redistributor has decided to provide a smoother migration path to >> verifying HTTPS certificates by default than that being provided by the >> upstream project >> * either the redistributor or the local infrastructure administrator has >> determined that it is appropriate to override the default upstream behaviour >> (at least for the time being) >> >> Using an administrator controlled configuration file rather than an environment >> variable has the essential feature of providing a smoother migration path, even >> for applications being run with the ``-E`` switch. >> >> >> Combining the recommendations >> ============================= >> >> If a redistributor chooses to implement both recommendations, then the >> environment variable should take precedence over the system-wide configuration >> setting. This allows the setting to be changed for a given user, virtual >> environment or application, regardless of the system-wide default behaviour. >> >> In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and >> set to anything *other* than ``'0'``, then HTTPS certificate verification >> should be enabled. >> >> Example implementation >> ---------------------- >> >> :: >> >> _https_verify_envvar = 'PYTHONHTTPSVERIFY' >> _cert_verification_config = '/etc/python/cert-verification.cfg' >> >> def _get_https_context_factory(): >> # Check for am environmental override of the default behaviour >> config_setting = os.environ.get(_https_verify_envvar) >> if config_setting is not None: >> if config_setting == '0': >> return _create_unverified_context >> return create_default_context >> >> # Check for a system-wide override of the default behaviour >> context_factories = { >> 'enable': create_default_context, >> 'disable': _create_unverified_context, >> 'platform_default': _create_unverified_context, # For now :) >> } >> import ConfigParser >> config = ConfigParser.RawConfigParser() >> config.read(_cert_verification_config) >> try: >> verify_mode = config.get('https', 'verify') >> except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): >> verify_mode = 'platform_default' >> default_factory = context_factories.get('platform_default') >> return context_factories.get(verify_mode, default_factory) >> >> _create_default_https_context = _get_https_context_factory() >> >> >> Copyright >> ========= >> >> This document has been placed into the public domain. >> >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Mon Nov 16 19:00:46 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 16 Nov 2015 16:00:46 -0800 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: Message-ID: Hm, making Christian the BDFL-delegate would mean two out of three authors *and* the BDFL-delegate all working for Red Hat, which clearly has a stake (and IIUC has already committed to this approach ahead of PEP approval). SO then it would look like this is just rubber-stamping Red Hat's internal decision process (if it's a process -- sounds more like an accident :-). So, Alex, do you want to approve this PEP? On Mon, Nov 16, 2015 at 3:54 PM, Nick Coghlan wrote: > On 17 November 2015 at 09:07, Guido van Rossum wrote: >> So I dropped the ball on this too -- I was going to either have a look >> or tell you quickly to find a BDFL-delegate, but ended up doing >> neither. Skimming it now I really don't think I'm the right person to >> review this. Maybe you could ask Alex Gaynor? > > If Alex is interested, that could definitely work. I've also asked > Christian Heimes if he might be interested in handling it, as he > worked on several of the SSL module changes that were backported, and > he also works on FreeIPA [1] now, which will potentially be affected > by the changes to RHEL & CentOS 7. > > Regards, > Nick. > > [1] FreeIPA is an open source Identity Management & Authentication > system: http://www.freeipa.org/ > >> >> On Tue, Nov 10, 2015 at 4:47 PM, Nick Coghlan wrote: >>> Hi folks, >>> >>> I have a confession to make - I dropped the ball on the HTTPS >>> verification backport proposals in PEP 493, and let the upstream and >>> downstream approval processes get out of sequence. >>> >>> As a result, the RHEL 7.2 beta released back in September incorporates >>> the HTTPS verification feature backport based on the current PEP 493 >>> draft, even though that hasn't formally been pronounced as an Active >>> recommendation by python-dev yet. >>> >>> Accordingly, I'm belatedly submitting it for pronouncement now: >>> https://www.python.org/dev/peps/pep-0493/ >>> >>> There's currently no BDFL-Delegate assigned, so if Guido doesn't want >>> to handle it, we'll need to address that question first. >>> >>> Our last discussion back in July seemed to show that folks either >>> didn't care about the question (because they're using unmodified >>> upstream versions so the PEP didn't affect them), or else thought the >>> approach described in the PEP was reasonable, so I'm hoping the >>> consequences of my mistake won't be too severe. >>> >>> Regards, >>> Nick. >>> >>> P.S. I'm aware that this looks like presenting a fait accompli at a >>> point where it's too late to realistically say "No", but the truth is >>> that preparation for the Python in Education miniconf at PyCon >>> Australia ramped up immediately after the July discussion, and then I >>> personally got confused as to the scope of what was being included in >>> 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493 >>> being deferred to a later release, but it's actually the whole package >>> of 466+476+493). That's my fault for trying to keep track of too many >>> things at once (and thus failing at some of them), not anyone else's. >>> >>> ================================ >>> >>> PEP: 493 >>> Title: HTTPS verification recommendations for Python 2.7 redistributors >>> Version: $Revision$ >>> Last-Modified: $Date$ >>> Author: Nick Coghlan , >>> Robert Kuska , >>> Marc-Andr? Lemburg >>> Status: Draft >>> Type: Informational >>> Content-Type: text/x-rst >>> Created: 10-May-2015 >>> Post-History: 06-Jul-2015 >>> >>> >>> Abstract >>> ======== >>> >>> PEP 476 updated Python's default handling of HTTPS certificates to be >>> appropriate for communication over the public internet. The Python 2.7 long >>> term maintenance series was judged to be in scope for this change, with the >>> new behaviour introduced in the Python 2.7.9 maintenance release. >>> >>> This PEP provides recommendations to downstream redistributors wishing to >>> provide a smoother migration experience when helping their users to manage >>> this change in Python's default behaviour. >>> >>> >>> Rationale >>> ========= >>> >>> PEP 476 changed Python's default behaviour to better match the needs and >>> expectations of developers operating over the public internet, a category >>> which appears to include most new Python developers. It is the position of >>> the authors of this PEP that this was a correct decision. >>> >>> However, it is also the case that this change *does* cause problems for >>> infrastructure administrators operating private intranets that rely on >>> self-signed certificates, or otherwise encounter problems with the new default >>> certificate verification settings. >>> >>> The long term answer for such environments is to update their internal >>> certificate management to at least match the standards set by the public >>> internet, but in the meantime, it is desirable to offer these administrators >>> a way to continue receiving maintenance updates to the Python 2.7 series, >>> without having to gate that on upgrades to their certificate management >>> infrastructure. >>> >>> PEP 476 did attempt to address this question, by covering how to revert the >>> new settings process wide by monkeypatching the ``ssl`` module to restore the >>> old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed >>> to allow system administrators to disable the feature by default in their >>> Standard Operating Environment definition has been determined to be >>> insufficient in at least some cases. The specific case of interest to the >>> authors of this PEP is the one where a Linux distributor aims to provide >>> their users with a >>> `smoother migration path >>> `__ >>> than the standard one provided by consuming upstream CPython 2.7 releases >>> directly, but other potential challenges have also been pointed out with >>> updating embedded Python runtimes and other user level installations of Python. >>> >>> Rather than allowing a plethora of mutually incompatibile migration techniques >>> to bloom, this PEP proposes two alternative approaches that redistributors >>> may take when addressing these problems. Redistributors may choose to implement >>> one, both, or neither of these approaches based on their assessment of the >>> needs of their particular userbase. >>> >>> These designs are being proposed as a recommendation for redistributors, rather >>> than as new upstream features, as they are needed purely to support legacy >>> environments migrating from older versions of Python 2.7. Neither approach >>> is being proposed as an upstream Python 2.7 feature, nor as a feature in any >>> version of Python 3 (whether published directly by the Python Software >>> Foundation or by a redistributor). >>> >>> >>> Requirements for capability detection >>> ===================================== >>> >>> As these recommendations are intended to cover backports to earlier Python >>> versions, the Python version number cannot be used as a reliable means for >>> detecting them. Instead, the recommendations are defined to allow the presence >>> or absence of the feature to be determined using the following technique:: >>> >>> python -c "import ssl; ssl._relevant_attribute" >>> >>> This will fail with `AttributeError` (and hence a non-zero return code) if the >>> relevant capability is not available. >>> >>> The marker attributes are prefixed with an underscore to indicate the >>> implementation dependent nature of these capabilities - not all Python >>> distributions will offer them, only those that are providing a multi-stage >>> migration process from the legacy HTTPS handling to the new default behaviour. >>> >>> >>> Recommendation for an environment variable based security downgrade >>> =================================================================== >>> >>> Some redistributors may wish to provide a per-application option to disable >>> certificate verification in selected applications that run on or embed CPython >>> without needing to modify the application itself. >>> >>> In these cases, a configuration mechanism is needed that provides: >>> >>> * an opt-out model that allows certificate verification to be selectively >>> turned off for particular applications after upgrading to a version of >>> Python that verifies certificates by default >>> * the ability for all users to configure this setting on a per-application >>> basis, rather than on a per-system, or per-Python-installation basis >>> >>> This approach may be used for any redistributor provided version of Python 2.7, >>> including those that advertise themselves as providing Python 2.7.9 or later. >>> >>> >>> Required marker attribute >>> ------------------------- >>> >>> The required marker attribute on the ``ssl`` module when implementing this >>> recommendation is:: >>> >>> _https_verify_envvar = 'PYTHONHTTPSVERIFY' >>> >>> This not only makes it straightforward to detect the presence (or absence) of >>> the capability, it also makes it possible to programmatically determine the >>> relevant environment variable name. >>> >>> >>> Recommended modifications to the Python standard library >>> -------------------------------------------------------- >>> >>> The recommended approach to providing a per-application configuration setting >>> for HTTPS certificate verification that doesn't require modifications to the >>> application itself is to: >>> >>> * modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment >>> variable when the module is first imported into a Python process >>> * set the ``ssl._create_default_https_context`` function to be an alias for >>> ``ssl._create_unverified_context`` if this environment variable is present >>> and set to ``'0'`` >>> * otherwise, set the ``ssl._create_default_https_context`` function to be an >>> alias for ``ssl.create_default_context`` as usual >>> >>> >>> Example implementation >>> ---------------------- >>> >>> :: >>> >>> _https_verify_envvar = 'PYTHONHTTPSVERIFY' >>> >>> def _get_https_context_factory(): >>> config_setting = os.environ.get(_https_verify_envvar) >>> if config_setting == '0': >>> return _create_unverified_context >>> return create_default_context >>> >>> _create_default_https_context = _get_https_context_factory() >>> >>> >>> Security Considerations >>> ----------------------- >>> >>> Relative to an unmodified version of CPython 2.7.9 or later, this approach >>> does introduce a new downgrade attack against the default security settings >>> that potentially allows a sufficiently determined attacker to revert Python >>> to the vulnerable configuration used in CPython 2.7.8 and earlier releases. >>> However, such an attack requires the ability to modify the execution >>> environment of a Python process prior to the import of the ``ssl`` module, >>> and any attacker with such access would already be able to modify the >>> behaviour of the underlying OpenSSL implementation. >>> >>> >>> Recommendation for backporting to earlier Python versions >>> ========================================================= >>> >>> Some redistributors, most notably Linux distributions, may choose to backport >>> the PEP 476 HTTPS verification changes to modified Python versions based on >>> earlier Python 2 maintenance releases. In these cases, a configuration >>> mechanism is needed that provides: >>> >>> * an opt-in model that allows the decision to enable HTTPS certificate >>> verification to be made independently of the decision to upgrade to the >>> Python version where the feature was first backported >>> * the ability for system administrators to set the default behaviour of Python >>> applications and scripts run directly in the system Python installation >>> * the ability for the redistributor to consider changing the default behaviour >>> of *new* installations at some point in the future without impacting existing >>> installations that have been explicitly configured to skip verifying HTTPS >>> certificates by default >>> >>> This approach should not be used for any Python installation that advertises >>> itself as providing Python 2.7.9 or later, as most Python users will have the >>> reasonable expectation that all such environments will validate HTTPS >>> certificates by default. >>> >>> >>> Required marker attribute >>> ------------------------- >>> >>> The required marker attribute on the ``ssl`` module when implementing this >>> recommendation is:: >>> >>> _cert_verification_config = '' >>> >>> This not only makes it straightforward to detect the presence (or absence) of >>> the capability, it also makes it possible to programmatically determine the >>> relevant configuration file name. >>> >>> >>> Recommended modifications to the Python standard library >>> -------------------------------------------------------- >>> >>> The recommended approach to backporting the PEP 476 modifications to an earlier >>> point release is to implement the following changes relative to the default >>> PEP 476 behaviour implemented in Python 2.7.9+: >>> >>> * modify the ``ssl`` module to read a system wide configuration file when the >>> module is first imported into a Python process >>> * define a platform default behaviour (either verifying or not verifying HTTPS >>> certificates) to be used if this configuration file is not present >>> * support selection between the following three modes of operation: >>> >>> * ensure HTTPS certificate verification is enabled >>> * ensure HTTPS certificate verification is disabled >>> * delegate the decision to the redistributor providing this Python version >>> >>> * set the ``ssl._create_default_https_context`` function to be an alias for >>> either ``ssl.create_default_context`` or ``ssl._create_unverified_context`` >>> based on the given configuration setting. >>> >>> >>> Recommended file location >>> ------------------------- >>> >>> This approach is currently only defined for \*nix system Python installations. >>> >>> The recommended configuration file name is >>> ``/etc/python/cert-verification.cfg``. >>> >>> The ``.cfg`` filename extension is recommended for consistency with the >>> ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library. >>> >>> >>> Recommended file format >>> ----------------------- >>> >>> The configuration file should use a ConfigParser ini-style format with a >>> single section named ``[https]`` containing one required setting ``verify``. >>> >>> Permitted values for ``verify`` are: >>> >>> * ``enable``: ensure HTTPS certificate verification is enabled by default >>> * ``disable``: ensure HTTPS certificate verification is disabled by default >>> * ``platform_default``: delegate the decision to the redistributor providing >>> this particular Python version >>> >>> If the ``[https]`` section or the ``verify`` setting are missing, or if the >>> ``verify`` setting is set to an unknown value, it should be treated as if the >>> configuration file is not present. >>> >>> >>> Example implementation >>> ---------------------- >>> >>> :: >>> >>> _cert_verification_config = '/etc/python/cert-verification.cfg' >>> >>> def _get_https_context_factory(): >>> # Check for a system-wide override of the default behaviour >>> context_factories = { >>> 'enable': create_default_context, >>> 'disable': _create_unverified_context, >>> 'platform_default': _create_unverified_context, # For now :) >>> } >>> import ConfigParser >>> config = ConfigParser.RawConfigParser() >>> config.read(_cert_verification_config) >>> try: >>> verify_mode = config.get('https', 'verify') >>> except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): >>> verify_mode = 'platform_default' >>> default_factory = context_factories.get('platform_default') >>> return context_factories.get(verify_mode, default_factory) >>> >>> _create_default_https_context = _get_https_context_factory() >>> >>> >>> Security Considerations >>> ----------------------- >>> >>> The specific recommendations for the backporting case are designed to work for >>> privileged, security sensitive processes, even those being run in the following >>> locked down configuration: >>> >>> * run from a locked down administrator controlled directory rather than a normal >>> user directory (preventing ``sys.path[0]`` based privilege escalation attacks) >>> * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based >>> privilege escalation attacks) >>> * run using the ``-s`` switch (preventing user site directory based privilege >>> escalation attacks) >>> * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege >>> escalation attacks) >>> >>> The intent is that the *only* reason HTTPS verification should be getting >>> turned off system wide when using this approach is because: >>> >>> * an end user is running a redistributor provided version of CPython rather >>> than running upstream CPython directly >>> * that redistributor has decided to provide a smoother migration path to >>> verifying HTTPS certificates by default than that being provided by the >>> upstream project >>> * either the redistributor or the local infrastructure administrator has >>> determined that it is appropriate to override the default upstream behaviour >>> (at least for the time being) >>> >>> Using an administrator controlled configuration file rather than an environment >>> variable has the essential feature of providing a smoother migration path, even >>> for applications being run with the ``-E`` switch. >>> >>> >>> Combining the recommendations >>> ============================= >>> >>> If a redistributor chooses to implement both recommendations, then the >>> environment variable should take precedence over the system-wide configuration >>> setting. This allows the setting to be changed for a given user, virtual >>> environment or application, regardless of the system-wide default behaviour. >>> >>> In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and >>> set to anything *other* than ``'0'``, then HTTPS certificate verification >>> should be enabled. >>> >>> Example implementation >>> ---------------------- >>> >>> :: >>> >>> _https_verify_envvar = 'PYTHONHTTPSVERIFY' >>> _cert_verification_config = '/etc/python/cert-verification.cfg' >>> >>> def _get_https_context_factory(): >>> # Check for am environmental override of the default behaviour >>> config_setting = os.environ.get(_https_verify_envvar) >>> if config_setting is not None: >>> if config_setting == '0': >>> return _create_unverified_context >>> return create_default_context >>> >>> # Check for a system-wide override of the default behaviour >>> context_factories = { >>> 'enable': create_default_context, >>> 'disable': _create_unverified_context, >>> 'platform_default': _create_unverified_context, # For now :) >>> } >>> import ConfigParser >>> config = ConfigParser.RawConfigParser() >>> config.read(_cert_verification_config) >>> try: >>> verify_mode = config.get('https', 'verify') >>> except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): >>> verify_mode = 'platform_default' >>> default_factory = context_factories.get('platform_default') >>> return context_factories.get(verify_mode, default_factory) >>> >>> _create_default_https_context = _get_https_context_factory() >>> >>> >>> Copyright >>> ========= >>> >>> This document has been placed into the public domain. >>> >>> >>> -- >>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) > > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -- --Guido van Rossum (python.org/~guido) From storchaka at gmail.com Mon Nov 16 20:53:32 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 17 Nov 2015 03:53:32 +0200 Subject: [Python-Dev] Reading Python source file Message-ID: I'm working on rewriting Python tokenizer (in particular the part that reads and decodes Python source file). The code is complicated. For now there are such cases: * Reading from the string in memory. * Interactive reading from the file. * Reading from the file: - Raw reading ignoring encoding in parser generator. - Raw reading UTF-8 encoded file. - Reading and recoding to UTF-8. The file is read by the line. It makes hard to check correctness of the first line if the encoding is specified in the second line. And it makes very hard problems with null bytes and with desynchronizing buffered C and Python files. All this problems can be easily solved if read all Python source file in memory and then parse it as string. This would allow to drop a large complex and buggy part of code. Are there disadvantages in this solution? As for memory consumption, the source text itself will consume only small part of the memory consumed by AST tree and other structures. As for performance, reading and decoding all file can be faster then by the line. [1] http://bugs.python.org/issue25643 From guido at python.org Mon Nov 16 22:00:39 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 16 Nov 2015 19:00:39 -0800 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: Message-ID: If you free the memory used for the source buffer before starting code generation you should be good. On Mon, Nov 16, 2015 at 5:53 PM, Serhiy Storchaka wrote: > I'm working on rewriting Python tokenizer (in particular the part that reads > and decodes Python source file). The code is complicated. For now there are > such cases: > > * Reading from the string in memory. > * Interactive reading from the file. > * Reading from the file: > - Raw reading ignoring encoding in parser generator. > - Raw reading UTF-8 encoded file. > - Reading and recoding to UTF-8. > > The file is read by the line. It makes hard to check correctness of the > first line if the encoding is specified in the second line. And it makes > very hard problems with null bytes and with desynchronizing buffered C and > Python files. All this problems can be easily solved if read all Python > source file in memory and then parse it as string. This would allow to drop > a large complex and buggy part of code. > > Are there disadvantages in this solution? As for memory consumption, the > source text itself will consume only small part of the memory consumed by > AST tree and other structures. As for performance, reading and decoding all > file can be faster then by the line. > > [1] http://bugs.python.org/issue25643 > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From python at mrabarnett.plus.com Mon Nov 16 22:05:33 2015 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 17 Nov 2015 03:05:33 +0000 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: Message-ID: <564A997D.2080706@mrabarnett.plus.com> On 2015-11-17 01:53, Serhiy Storchaka wrote: > I'm working on rewriting Python tokenizer (in particular the part that > reads and decodes Python source file). The code is complicated. For now > there are such cases: > > * Reading from the string in memory. > * Interactive reading from the file. > * Reading from the file: > - Raw reading ignoring encoding in parser generator. > - Raw reading UTF-8 encoded file. > - Reading and recoding to UTF-8. > > The file is read by the line. It makes hard to check correctness of the > first line if the encoding is specified in the second line. And it makes > very hard problems with null bytes and with desynchronizing buffered C > and Python files. All this problems can be easily solved if read all > Python source file in memory and then parse it as string. This would > allow to drop a large complex and buggy part of code. > > Are there disadvantages in this solution? As for memory consumption, the > source text itself will consume only small part of the memory consumed > by AST tree and other structures. As for performance, reading and > decoding all file can be faster then by the line. > > [1] http://bugs.python.org/issue25643 > As I understand it, *nix expects the shebang to be b'#!', which means that the first line should be ASCII-compatible (it's possible that the UTF-8 BOM might be present). This kind of suggests that encodings like UTF-16 would cause a problem on such systems. The encoding line also needs to be ASCII-compatible. I believe that the recent thread "Support of UTF-16 and UTF-32 source encodings" also concluded that UTF-16 and UTF-32 shouldn't be supported. This means that you could treat the first 2 lines as though they were some kind of extended ASCII (Latin-1?), the line ending being '\n' or '\r' or '\r\n'. Once you'd identify the encoding, you could decode everything (including the shebang line) using that encoding. (What should happen if the encoding line then decoded differently, i.e. encoding_line.decode(encoding) != encoding_line.decode('latin-1')?) From random832 at fastmail.com Mon Nov 16 23:09:51 2015 From: random832 at fastmail.com (Random832) Date: Mon, 16 Nov 2015 23:09:51 -0500 Subject: [Python-Dev] Reading Python source file References: <564A997D.2080706@mrabarnett.plus.com> Message-ID: MRAB writes: > As I understand it, *nix expects the shebang to be b'#!', > which means that the first line should be ASCII-compatible > (it's possible that the UTF-8 BOM might be present). The UTF-8 BOM interferes with it on Mac OSX and Linux, at least. From brett at snarky.ca Mon Nov 16 13:49:03 2015 From: brett at snarky.ca (Brett Cannon) Date: Mon, 16 Nov 2015 18:49:03 +0000 Subject: [Python-Dev] [Python-checkins] Daily reference leaks (97e2a6810f7f): sum=10 In-Reply-To: <20151116084238.37478.4533@psf.io> References: <20151116084238.37478.4533@psf.io> Message-ID: Just an FYI there seems to be a consistent, minor refcount leak found by test_capi that has been there for what seems like a couple of weeks. On Mon, 16 Nov 2015 at 00:42 wrote: > results for 97e2a6810f7f on branch "default" > -------------------------------------------- > > test_asyncio leaked [0, 0, 3] memory blocks, sum=3 > test_capi leaked [1, 1, 1] references, sum=3 > test_functools leaked [0, 2, 2] memory blocks, sum=4 > > > Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', > '3:3:/home/psf-users/antoine/refleaks/reflogBLsY2a', '--timeout', '7200'] > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Nov 17 04:59:06 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 17 Nov 2015 10:59:06 +0100 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: Message-ID: <564AFA6A.8010505@egenix.com> On 17.11.2015 02:53, Serhiy Storchaka wrote: > I'm working on rewriting Python tokenizer (in particular the part that reads and decodes Python > source file). The code is complicated. For now there are such cases: > > * Reading from the string in memory. > * Interactive reading from the file. > * Reading from the file: > - Raw reading ignoring encoding in parser generator. > - Raw reading UTF-8 encoded file. > - Reading and recoding to UTF-8. > > The file is read by the line. It makes hard to check correctness of the first line if the encoding > is specified in the second line. And it makes very hard problems with null bytes and with > desynchronizing buffered C and Python files. All this problems can be easily solved if read all > Python source file in memory and then parse it as string. This would allow to drop a large complex > and buggy part of code. > > Are there disadvantages in this solution? As for memory consumption, the source text itself will > consume only small part of the memory consumed by AST tree and other structures. As for performance, > reading and decoding all file can be faster then by the line. A problem with this approach is that you can no longer fail early and detect indentation errors et al. while parsing the data (which may well come from a pipe). Another related problem is that you have to wait for the full input data before you can start compiling the code. I don't think these situations are all that common, though, so reading in the full source code before compiling it sounds like a reasonable approach. We use the same simplification in eGenix PyRun's emulation of the Python command line interface and it has so far not caused any problems. > [1] http://bugs.python.org/issue25643 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 17 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ 2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From victor.stinner at gmail.com Tue Nov 17 05:24:07 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 17 Nov 2015 11:24:07 +0100 Subject: [Python-Dev] [Python-checkins] Daily reference leaks (97e2a6810f7f): sum=10 In-Reply-To: References: <20151116084238.37478.4533@psf.io> Message-ID: Hi, I opened the http://bugs.python.org/issue25645 It looks like a simple regression in _pickle. I wrote a patch. Victor 2015-11-16 19:49 GMT+01:00 Brett Cannon : > Just an FYI there seems to be a consistent, minor refcount leak found by > test_capi that has been there for what seems like a couple of weeks. > > On Mon, 16 Nov 2015 at 00:42 wrote: >> >> results for 97e2a6810f7f on branch "default" >> -------------------------------------------- >> >> test_asyncio leaked [0, 0, 3] memory blocks, sum=3 >> test_capi leaked [1, 1, 1] references, sum=3 >> test_functools leaked [0, 2, 2] memory blocks, sum=4 >> >> >> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', >> '3:3:/home/psf-users/antoine/refleaks/reflogBLsY2a', '--timeout', '7200'] >> _______________________________________________ >> Python-checkins mailing list >> Python-checkins at python.org >> https://mail.python.org/mailman/listinfo/python-checkins > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > From victor.stinner at gmail.com Tue Nov 17 05:33:30 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 17 Nov 2015 11:33:30 +0100 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: Message-ID: 2015-11-17 1:00 GMT+01:00 Guido van Rossum : > Hm, making Christian the BDFL-delegate would mean two out of three > authors *and* the BDFL-delegate all working for Red Hat, which clearly > has a stake (and IIUC has already committed to this approach ahead of > PEP approval). SO then it would look like this is just rubber-stamping > Red Hat's internal decision process (if it's a process -- sounds more > like an accident :-). Can we try to get a vote from maintainers of the Python2/3 packages of other Linux distributions? Debian, Ubuntu, OpenSUSE, etc.? Victor From ncoghlan at gmail.com Tue Nov 17 08:44:50 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 17 Nov 2015 23:44:50 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: Message-ID: On 17 November 2015 at 20:33, Victor Stinner wrote: > 2015-11-17 1:00 GMT+01:00 Guido van Rossum : >> Hm, making Christian the BDFL-delegate would mean two out of three >> authors *and* the BDFL-delegate all working for Red Hat, which clearly >> has a stake (and IIUC has already committed to this approach ahead of >> PEP approval). SO then it would look like this is just rubber-stamping >> Red Hat's internal decision process (if it's a process -- sounds more >> like an accident :-). > > Can we try to get a vote from maintainers of the Python2/3 packages of > other Linux distributions? Debian, Ubuntu, OpenSUSE, etc.? I know Oracle were interested based on a discussion between them and a member of Red Hat's product security team about it on oss-security, but their devs never followed up on it upstream (even after an explicit suggestion that they do so), so I'm interpreting that as willingness to go along with whatever happens in RHEL. For Debian, Ubuntu and SUSE, their original determinations for the relevant CVE were "too intrusive to backport", so folks currently need to upgrade to newer versions of those distros to get the improved default behaviour: * http://people.canonical.com/~ubuntu-security/cve/2014/CVE-2014-9365.html * https://security-tracker.debian.org/tracker/CVE-2014-9365 * https://www.suse.com/security/cve/CVE-2014-9365.html If having an opt-in backwards-compatible-by-default approach available (albeit as a PEP 466+476+493 patch set in the RHEL/CentOS system Python 2.7.5 package) prompts other distro security teams to reconsider those initial assessments, that would be a nice outcome, but it isn't my own main priority (so Guido makes a good point in favouring finding a non-Red-Hatter willing to act as BDFL-Delegate) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Tue Nov 17 10:22:37 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 17 Nov 2015 07:22:37 -0800 Subject: [Python-Dev] Reading Python source file In-Reply-To: <564AFA6A.8010505@egenix.com> References: <564AFA6A.8010505@egenix.com> Message-ID: On Tue, Nov 17, 2015 at 1:59 AM, M.-A. Lemburg wrote: > On 17.11.2015 02:53, Serhiy Storchaka wrote: >> I'm working on rewriting Python tokenizer (in particular the part that reads and decodes Python >> source file). The code is complicated. For now there are such cases: >> >> * Reading from the string in memory. >> * Interactive reading from the file. >> * Reading from the file: >> - Raw reading ignoring encoding in parser generator. >> - Raw reading UTF-8 encoded file. >> - Reading and recoding to UTF-8. >> >> The file is read by the line. It makes hard to check correctness of the first line if the encoding >> is specified in the second line. And it makes very hard problems with null bytes and with >> desynchronizing buffered C and Python files. All this problems can be easily solved if read all >> Python source file in memory and then parse it as string. This would allow to drop a large complex >> and buggy part of code. >> >> Are there disadvantages in this solution? As for memory consumption, the source text itself will >> consume only small part of the memory consumed by AST tree and other structures. As for performance, >> reading and decoding all file can be faster then by the line. > > A problem with this approach is that you can no > longer fail early and detect indentation errors et al. while > parsing the data (which may well come from a pipe). Oh, this use case I had forgotten about. I don't know how common or important it is though. But more important is the interactive REPL, which parses your input fully each time you hit ENTER. > Another related problem is that you have to wait for the full > input data before you can start compiling the code. That's always the case -- we don't start compiling before we have the full parse tree. > I don't think these situations are all that common, though, > so reading in the full source code before compiling it > sounds like a reasonable approach. > > We use the same simplification in eGenix PyRun's emulation of > the Python command line interface and it has so far not > caused any problems. Curious how you do it? I'd actually be quite disappointed if the amount of parsing done by the standard REPL went down. >> [1] http://bugs.python.org/issue25643 -- --Guido van Rossum (python.org/~guido) From storchaka at gmail.com Tue Nov 17 10:40:37 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 17 Nov 2015 17:40:37 +0200 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: Message-ID: On 17.11.15 05:00, Guido van Rossum wrote: > If you free the memory used for the source buffer before starting code > generation you should be good. Thank you. The buffer is freed just after the end of generating AST. > On Mon, Nov 16, 2015 at 5:53 PM, Serhiy Storchaka wrote: >> I'm working on rewriting Python tokenizer (in particular the part that reads >> and decodes Python source file). The code is complicated. For now there are >> such cases: >> >> * Reading from the string in memory. >> * Interactive reading from the file. >> * Reading from the file: >> - Raw reading ignoring encoding in parser generator. >> - Raw reading UTF-8 encoded file. >> - Reading and recoding to UTF-8. >> >> The file is read by the line. It makes hard to check correctness of the >> first line if the encoding is specified in the second line. And it makes >> very hard problems with null bytes and with desynchronizing buffered C and >> Python files. All this problems can be easily solved if read all Python >> source file in memory and then parse it as string. This would allow to drop >> a large complex and buggy part of code. >> >> Are there disadvantages in this solution? As for memory consumption, the >> source text itself will consume only small part of the memory consumed by >> AST tree and other structures. As for performance, reading and decoding all >> file can be faster then by the line. >> >> [1] http://bugs.python.org/issue25643 >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > From guido at python.org Tue Nov 17 11:06:01 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 17 Nov 2015 08:06:01 -0800 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: Message-ID: OK, but what are you going to do about the interactive REPL? On Tue, Nov 17, 2015 at 7:40 AM, Serhiy Storchaka wrote: > On 17.11.15 05:00, Guido van Rossum wrote: >> >> If you free the memory used for the source buffer before starting code >> generation you should be good. > > > Thank you. The buffer is freed just after the end of generating AST. > > >> On Mon, Nov 16, 2015 at 5:53 PM, Serhiy Storchaka >> wrote: >>> >>> I'm working on rewriting Python tokenizer (in particular the part that >>> reads >>> and decodes Python source file). The code is complicated. For now there >>> are >>> such cases: >>> >>> * Reading from the string in memory. >>> * Interactive reading from the file. >>> * Reading from the file: >>> - Raw reading ignoring encoding in parser generator. >>> - Raw reading UTF-8 encoded file. >>> - Reading and recoding to UTF-8. >>> >>> The file is read by the line. It makes hard to check correctness of the >>> first line if the encoding is specified in the second line. And it makes >>> very hard problems with null bytes and with desynchronizing buffered C >>> and >>> Python files. All this problems can be easily solved if read all Python >>> source file in memory and then parse it as string. This would allow to >>> drop >>> a large complex and buggy part of code. >>> >>> Are there disadvantages in this solution? As for memory consumption, the >>> source text itself will consume only small part of the memory consumed by >>> AST tree and other structures. As for performance, reading and decoding >>> all >>> file can be faster then by the line. >>> >>> [1] http://bugs.python.org/issue25643 >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> >> >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From storchaka at gmail.com Tue Nov 17 11:06:17 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 17 Nov 2015 18:06:17 +0200 Subject: [Python-Dev] Reading Python source file In-Reply-To: <564A997D.2080706@mrabarnett.plus.com> References: <564A997D.2080706@mrabarnett.plus.com> Message-ID: On 17.11.15 05:05, MRAB wrote: > As I understand it, *nix expects the shebang to be b'#!', which means > that the > first line should be ASCII-compatible (it's possible that the UTF-8 BOM > might > be present). This kind of suggests that encodings like UTF-16 would cause a > problem on such systems. > > The encoding line also needs to be ASCII-compatible. > > I believe that the recent thread "Support of UTF-16 and UTF-32 source > encodings" also concluded that UTF-16 and UTF-32 shouldn't be supported. > > This means that you could treat the first 2 lines as though they were some > kind of extended ASCII (Latin-1?), the line ending being '\n' or '\r' or > '\r\n'. > > Once you'd identify the encoding, you could decode everything (including > the > shebang line) using that encoding. Yes, that is what I were going to implement (and already halfway here). My question is whether it is worth to complicate the code further to preserve reading by the line. In any case after reading the first line that doesn't contain neither coding cookie, nor non-comment tokens, we need to wait the second line. > (What should happen if the encoding line then decoded differently, i.e. > encoding_line.decode(encoding) != encoding_line.decode('latin-1')?) The parser should got the line decoded with specified encoding. From storchaka at gmail.com Tue Nov 17 11:20:30 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 17 Nov 2015 18:20:30 +0200 Subject: [Python-Dev] Reading Python source file In-Reply-To: <564AFA6A.8010505@egenix.com> References: <564AFA6A.8010505@egenix.com> Message-ID: On 17.11.15 11:59, M.-A. Lemburg wrote: > I don't think these situations are all that common, though, > so reading in the full source code before compiling it > sounds like a reasonable approach. > > We use the same simplification in eGenix PyRun's emulation of > the Python command line interface and it has so far not > caused any problems. Current implementation of import system went the same way. As a result importing the script as a module and running it with command line can have different behaviours in corner cases. From storchaka at gmail.com Tue Nov 17 11:26:15 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 17 Nov 2015 18:26:15 +0200 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: <564AFA6A.8010505@egenix.com> Message-ID: On 17.11.15 17:22, Guido van Rossum wrote: > But more important is the interactive REPL, which parses your input > fully each time you hit ENTER. Interactive REPL runs different code. It is simpler that the code for reading from file, because it have no care about BOM or coding cookie. From storchaka at gmail.com Tue Nov 17 11:27:49 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 17 Nov 2015 18:27:49 +0200 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: Message-ID: On 17.11.15 18:06, Guido van Rossum wrote: > OK, but what are you going to do about the interactive REPL? Nothing (except some simplification). This is a separate branch of the code. From guido at python.org Tue Nov 17 11:50:53 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 17 Nov 2015 08:50:53 -0800 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: <564AFA6A.8010505@egenix.com> Message-ID: On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka wrote: > On 17.11.15 11:59, M.-A. Lemburg wrote: >> >> I don't think these situations are all that common, though, >> so reading in the full source code before compiling it >> sounds like a reasonable approach. >> >> We use the same simplification in eGenix PyRun's emulation of >> the Python command line interface and it has so far not >> caused any problems. > > Current implementation of import system went the same way. As a result > importing the script as a module and running it with command line can have > different behaviours in corner cases. I'm confused. *Of course* these two behaviors differ, since Python uses a different __name__. Not sure how this relates to the REPL. -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue Nov 17 11:51:15 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 17 Nov 2015 08:51:15 -0800 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: Message-ID: Oh, cool! Sorry for the disturbance. On Tue, Nov 17, 2015 at 8:27 AM, Serhiy Storchaka wrote: > On 17.11.15 18:06, Guido van Rossum wrote: >> >> OK, but what are you going to do about the interactive REPL? > > > Nothing (except some simplification). This is a separate branch of the code. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From rdmurray at bitdance.com Tue Nov 17 13:40:32 2015 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 17 Nov 2015 13:40:32 -0500 Subject: [Python-Dev] Benchmark results across all major Python implementations In-Reply-To: References: Message-ID: <20151117184034.1C19FB14089@webabinitio.net> On Mon, 16 Nov 2015 23:37:06 +0000, "Stewart, David C" wrote: > Last June we started publishing a daily performance report of the latest Python tip against the previous day's run and some established synch point. We mail these to the community to act as a "canary in the coal mine." I wrote about it at https://01.org/lp/blog/0-day-challenge-what-pulse-internet > > You can see our manager-style dashboard of a couple of key workloads at http://languagesperformance.intel.com/ > (I have this running constantly on a dedicated screen in my office). Just took a look at this. Pretty cool. The web page is a bit confusing, though. It doesn't give any clue as to what is being measured by the numbers presented...it isn't obvious whether those downward sloping lines represent progress or regression. Also, the intro talks about historical data, but other than the older dates[*] in the graph there's no access to it. Do you have plans to provide access to the raw data? It also doesn't show all of the test shown in the example email in your blog post or the emails to python-checkins...do you plan to make those graphs available in the future as well? Also, in the emails, what is the PGO column percentage relative to? I suppose that for this to have maximum effect someone would have to specifically be paying attention to performance and figuring out why every (real) regression happened. I don't suppose we have anyone in the community currently who is taking on that role, though we certainly do have people who are *interested* in Python performance :) --David [*] Personally I'd find it easier to read those dates in MM-DD form, but I suppose that's a US quirk, since in the US when using slashes the month comes first... From david.c.stewart at intel.com Tue Nov 17 14:22:48 2015 From: david.c.stewart at intel.com (Stewart, David C) Date: Tue, 17 Nov 2015 19:22:48 +0000 Subject: [Python-Dev] Benchmark results across all major Python implementations In-Reply-To: <20151117184034.1C19FB14089@webabinitio.net> References: <20151117184034.1C19FB14089@webabinitio.net> Message-ID: +Stefan (owner of the 0-day lab) On 11/17/15, 10:40 AM, "Python-Dev on behalf of R. David Murray" wrote: >On Mon, 16 Nov 2015 23:37:06 +0000, "Stewart, David C" wrote: >> Last June we started publishing a daily performance report of the latest Python tip against the previous day's run and some established synch point. We mail these to the community to act as a "canary in the coal mine." I wrote about it at https://01.org/lp/blog/0-day-challenge-what-pulse-internet >> >> You can see our manager-style dashboard of a couple of key workloads at http://languagesperformance.intel.com/ >> (I have this running constantly on a dedicated screen in my office). > >Just took a look at this. Pretty cool. The web page is a bit confusing, >though. It doesn't give any clue as to what is being measured by the >numbers presented...it isn't obvious whether those downward sloping >lines represent progress or regression. Also, the intro talks about >historical data, but other than the older dates[*] in the graph there's >no access to it. Do you have plans to provide access to the raw data? >It also doesn't show all of the test shown in the example email in your >blog post or the emails to python-checkins...do you plan to make those >graphs available in the future as well? The data on this website has been normalized so "up" is "good" so far as the slope of the line. The daily email has a lot more detail about the hardware and software configuration and the versions being compared. We run workloads multiple times and visually show the relative standard distribution on the graph. No plans to show the raw data. I think showing multiple workloads graphically sounds useful, we should look into that. > >Also, in the emails, what is the PGO column percentage relative to? It's the performance boost on the current rev from just using PGO. Another way to think about it is, this is the performance that you leave on the table by *not* building Cpython with PGO. For example, from last night's run, we would see an 18.54% boost in django_v2 by building Python using PGO. Note: PGO is not the default way to build Python because it is relatively slow to compile it that way. (I think it should be the default). Here are the instructions for using it (thanks to Peter Wang for the instructions): hg clone https://hg.python.org/cpython cpython cd cpython hg update 2.7 ./configure make profile-opt > >I suppose that for this to have maximum effect someone would have to >specifically be paying attention to performance and figuring out why >every (real) regression happened. I don't suppose we have anyone in the >community currently who is taking on that role, though we certainly do >have people who are *interested* in Python performance :) We're trying to fill that role as much as we can. When there is a significant (and unexplained) regression that we see, I usually ask our engineers to bisect it to identify the offending patch and root-cause it. > >--David > >[*] Personally I'd find it easier to read those dates in MM-DD form, >but I suppose that's a US quirk, since in the US when using slashes >the month comes first... You and me both. As you surmised, the site was developed by our friends in Europe. :-) >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: https://mail.python.org/mailman/options/python-dev/david.c.stewart%40intel.com From ncoghlan at gmail.com Tue Nov 17 21:31:43 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 18 Nov 2015 12:31:43 +1000 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: <564AFA6A.8010505@egenix.com> Message-ID: On 18 November 2015 at 02:50, Guido van Rossum wrote: > On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka wrote: >> On 17.11.15 11:59, M.-A. Lemburg wrote: >>> >>> I don't think these situations are all that common, though, >>> so reading in the full source code before compiling it >>> sounds like a reasonable approach. >>> >>> We use the same simplification in eGenix PyRun's emulation of >>> the Python command line interface and it has so far not >>> caused any problems. >> >> Current implementation of import system went the same way. As a result >> importing the script as a module and running it with command line can have >> different behaviours in corner cases. > > I'm confused. *Of course* these two behaviors differ, since Python > uses a different __name__. Not sure how this relates to the REPL. I think Serhiy was referring to the fact that importlib already reads the entire file before compiling it - since the import abstraction doesn't require modules to be files on disk, it uses the get_code() API on the module loader internally, which is typically implemented by calling get_source() and then passing the result to compile(). That behaviour is then inherited at the command line by both the -m switch and the support for executing directories and zip archives. When we consider that the "-c" switch also executes an in-memory string, direct script execution is currently the odd one out in *not* reading the entire source file into memory first, so Serhiy's proposed simplification of the implementation makes sense to me. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Tue Nov 17 21:46:49 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 17 Nov 2015 18:46:49 -0800 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: <564AFA6A.8010505@egenix.com> Message-ID: Aha, so the only code path that's being replaced is the code that reads the script file when invoking "python FILE" or "python wrote: > On 18 November 2015 at 02:50, Guido van Rossum wrote: >> On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka wrote: >>> On 17.11.15 11:59, M.-A. Lemburg wrote: >>>> >>>> I don't think these situations are all that common, though, >>>> so reading in the full source code before compiling it >>>> sounds like a reasonable approach. >>>> >>>> We use the same simplification in eGenix PyRun's emulation of >>>> the Python command line interface and it has so far not >>>> caused any problems. >>> >>> Current implementation of import system went the same way. As a result >>> importing the script as a module and running it with command line can have >>> different behaviours in corner cases. >> >> I'm confused. *Of course* these two behaviors differ, since Python >> uses a different __name__. Not sure how this relates to the REPL. > > I think Serhiy was referring to the fact that importlib already reads > the entire file before compiling it - since the import abstraction > doesn't require modules to be files on disk, it uses the get_code() > API on the module loader internally, which is typically implemented by > calling get_source() and then passing the result to compile(). > > That behaviour is then inherited at the command line by both the -m > switch and the support for executing directories and zip archives. > When we consider that the "-c" switch also executes an in-memory > string, direct script execution is currently the odd one out in *not* > reading the entire source file into memory first, so Serhiy's proposed > simplification of the implementation makes sense to me. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -- --Guido van Rossum (python.org/~guido) From stefan.a.popa at intel.com Wed Nov 18 01:10:30 2015 From: stefan.a.popa at intel.com (Popa, Stefan A) Date: Wed, 18 Nov 2015 06:10:30 +0000 Subject: [Python-Dev] Benchmark results across all major Python implementations In-Reply-To: References: <20151117184034.1C19FB14089@webabinitio.net>, Message-ID: <63E314D4-979E-44B8-9E3D-BE624D476E86@intel.com> Hi Python community, Thank you for your feedback! We will look into this and come up with an e-mail format proposal in the following days. Best regards, -- Stefan A. POPA Software Engineering Manager System Technologies and Optimization Division Software Services Group, Intel Romania > On 17 Nov 2015, at 21:22, Stewart, David C wrote: > > +Stefan (owner of the 0-day lab) > > > > >> On 11/17/15, 10:40 AM, "Python-Dev on behalf of R. David Murray" wrote: >> >>> On Mon, 16 Nov 2015 23:37:06 +0000, "Stewart, David C" wrote: >>> Last June we started publishing a daily performance report of the latest Python tip against the previous day's run and some established synch point. We mail these to the community to act as a "canary in the coal mine." I wrote about it at https://01.org/lp/blog/0-day-challenge-what-pulse-internet >>> >>> You can see our manager-style dashboard of a couple of key workloads at http://languagesperformance.intel.com/ >>> (I have this running constantly on a dedicated screen in my office). >> >> Just took a look at this. Pretty cool. The web page is a bit confusing, >> though. It doesn't give any clue as to what is being measured by the >> numbers presented...it isn't obvious whether those downward sloping >> lines represent progress or regression. Also, the intro talks about >> historical data, but other than the older dates[*] in the graph there's >> no access to it. Do you have plans to provide access to the raw data? >> It also doesn't show all of the test shown in the example email in your >> blog post or the emails to python-checkins...do you plan to make those >> graphs available in the future as well? > > The data on this website has been normalized so "up" is "good" so far as the slope of the line. The daily email has a lot more detail about the hardware and software configuration and the versions being compared. We run workloads multiple times and visually show the relative standard distribution on the graph. > > No plans to show the raw data. > > I think showing multiple workloads graphically sounds useful, we should look into that. > >> >> Also, in the emails, what is the PGO column percentage relative to? > > It's the performance boost on the current rev from just using PGO. Another way to think about it is, this is the performance that you leave on the table by *not* building Cpython with PGO. For example, from last night's run, we would see an 18.54% boost in django_v2 by building Python using PGO. > > Note: PGO is not the default way to build Python because it is relatively slow to compile it that way. (I think it should be the default). > > Here are the instructions for using it (thanks to Peter Wang for the instructions): > > hg clone https://hg.python.org/cpython cpython > cd cpython > hg update 2.7 > ./configure > make profile-opt > > > >> >> I suppose that for this to have maximum effect someone would have to >> specifically be paying attention to performance and figuring out why >> every (real) regression happened. I don't suppose we have anyone in the >> community currently who is taking on that role, though we certainly do >> have people who are *interested* in Python performance :) > > We're trying to fill that role as much as we can. When there is a significant (and unexplained) regression that we see, I usually ask our engineers to bisect it to identify the offending patch and root-cause it. > >> >> --David >> >> [*] Personally I'd find it easier to read those dates in MM-DD form, >> but I suppose that's a US quirk, since in the US when using slashes >> the month comes first... > > You and me both. As you surmised, the site was developed by our friends in Europe. :-) > >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/david.c.stewart%40intel.com From stephen at xemacs.org Wed Nov 18 02:11:59 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 18 Nov 2015 16:11:59 +0900 Subject: [Python-Dev] Benchmark results across all major Python implementations In-Reply-To: References: <20151117184034.1C19FB14089@webabinitio.net> Message-ID: <22092.9407.502230.151306@turnbull.sk.tsukuba.ac.jp> Stewart, David C writes: > Note: PGO is not the default way to build Python because it is > relatively slow to compile it that way. (I think it should be the > default). +1 Slow-build-fast-run should be the default if you're sure the optimization works. Only developers are likely to run a given build few enough times to save seconds, and most people are like to turn to some other task as soon as they type "make". It's a slightly different use case, but in XEmacs we have a --quick-build configure option which means that the "usual targets" don't rebuild a bunch of auxiliary targets (mostly documentation and development infrastructure such as xref caches). Never heard a complaint about that either from the developers (who learned to use --quick-build easily enough) or the beta testers (who do remark on long build times, but only once a week or so for most of them). From hrvoje.niksic at avl.com Wed Nov 18 07:15:16 2015 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Wed, 18 Nov 2015 13:15:16 +0100 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: <564AFA6A.8010505@egenix.com> Message-ID: <564C6BD4.6030205@avl.com> On 11/18/2015 03:31 AM, Nick Coghlan wrote: > That behaviour is then inherited at the command line by both the -m > switch and the support for executing directories and zip archives. > When we consider that the "-c" switch also executes an in-memory > string, direct script execution is currently the odd one out in *not* > reading the entire source file into memory first, so Serhiy's proposed > simplification of the implementation makes sense to me. Reading the whole script in memory will incur an overhead when executing scripts that contain (potentially large) data embedded after the end of script source. The technique of reading data from sys.argv[0] is probably obsolete now that Python supports executing zipped archives, but it is popular in shell scripting and might still be used for self-extracting scripts that must support older Python versions. This feature doesn't affect imports and -c which are not expected to contain non-Python data. Hrvoje From guido at python.org Wed Nov 18 10:48:41 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 18 Nov 2015 07:48:41 -0800 Subject: [Python-Dev] Reading Python source file In-Reply-To: <564C6BD4.6030205@avl.com> References: <564AFA6A.8010505@egenix.com> <564C6BD4.6030205@avl.com> Message-ID: On Wed, Nov 18, 2015 at 4:15 AM, Hrvoje Niksic wrote: > On 11/18/2015 03:31 AM, Nick Coghlan wrote: >> >> That behaviour is then inherited at the command line by both the -m >> switch and the support for executing directories and zip archives. >> When we consider that the "-c" switch also executes an in-memory >> string, direct script execution is currently the odd one out in *not* >> reading the entire source file into memory first, so Serhiy's proposed >> simplification of the implementation makes sense to me. > > > Reading the whole script in memory will incur an overhead when executing > scripts that contain (potentially large) data embedded after the end of > script source. > > The technique of reading data from sys.argv[0] is probably obsolete now that > Python supports executing zipped archives, but it is popular in shell > scripting and might still be used for self-extracting scripts that must > support older Python versions. This feature doesn't affect imports and -c > which are not expected to contain non-Python data. That trick doesn't work unless the data looks like Python comments or data (e.g. a docstring). Python has always insisted on being able to parse until EOF. The only extreme case would be a small script followed by e.g. 4 GB of comments (where the old parser would indeed be more efficient). But unless you can point me to an occurrence of this in the wild I'm going to speculate that you just made this up based on the shell analogy (which isn't perfect). -- --Guido van Rossum (python.org/~guido) From hrvoje.niksic at avl.com Wed Nov 18 10:57:31 2015 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Wed, 18 Nov 2015 16:57:31 +0100 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: <564AFA6A.8010505@egenix.com> <564C6BD4.6030205@avl.com> Message-ID: <564C9FEB.6030604@avl.com> On 11/18/2015 04:48 PM, Guido van Rossum wrote: > That trick doesn't work unless the data looks like Python comments or > data (e.g. a docstring). Python has always insisted on being able to > parse until EOF. The only extreme case would be a small script > followed by e.g. 4 GB of comments (where the old parser would indeed > be more efficient). But unless you can point me to an occurrence of > this in the wild I'm going to speculate that you just made this up > based on the shell analogy (which isn't perfect). If this never really worked in Python, feel free to drop the issue. I may be misremembering the language in which scripts I saw using this techniques years ago were written - most likely sh or Perl. Hrvoje From rdmurray at bitdance.com Wed Nov 18 11:24:03 2015 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 18 Nov 2015 11:24:03 -0500 Subject: [Python-Dev] Benchmark results across all major Python implementations In-Reply-To: <63E314D4-979E-44B8-9E3D-BE624D476E86@intel.com> References: <20151117184034.1C19FB14089@webabinitio.net>, <63E314D4-979E-44B8-9E3D-BE624D476E86@intel.com> Message-ID: <20151118162404.417A5B14089@webabinitio.net> On 17 Nov 2015, at 21:22, Stewart, David C wrote: > On 11/17/15, 10:40 AM, "Python-Dev on behalf of R. David Murray" wrote: >> >> I suppose that for this to have maximum effect someone would have to >> specifically be paying attention to performance and figuring out why >> every (real) regression happened. I don't suppose we have anyone in the >> community currently who is taking on that role, though we certainly do >> have people who are *interested* in Python performance :) > > We're trying to fill that role as much as we can. When there is a > significant (and unexplained) regression that we see, I usually ask > our engineers to bisect it to identify the offending patch and > root-cause it. That's great news. --David From rymg19 at gmail.com Wed Nov 18 11:50:48 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 18 Nov 2015 10:50:48 -0600 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: <564AFA6A.8010505@egenix.com> <564C6BD4.6030205@avl.com> Message-ID: <9A6B0389-9F66-4A1B-8D96-A3E57D42DAB4@gmail.com> Well, not quite the same thing, but https://github.com/kirbyfan64/pfbuild/blob/master/pfbuild embeds the compressed version of 16k LOC. Would it be affected negatively in any way be this? Since all the data is on one line, I'd think the old (current) parser would end up reading in the whole line anyway. On November 18, 2015 9:48:41 AM CST, Guido van Rossum wrote: >On Wed, Nov 18, 2015 at 4:15 AM, Hrvoje Niksic >wrote: >> On 11/18/2015 03:31 AM, Nick Coghlan wrote: >>> >>> That behaviour is then inherited at the command line by both the -m >>> switch and the support for executing directories and zip archives. >>> When we consider that the "-c" switch also executes an in-memory >>> string, direct script execution is currently the odd one out in >*not* >>> reading the entire source file into memory first, so Serhiy's >proposed >>> simplification of the implementation makes sense to me. >> >> >> Reading the whole script in memory will incur an overhead when >executing >> scripts that contain (potentially large) data embedded after the end >of >> script source. >> >> The technique of reading data from sys.argv[0] is probably obsolete >now that >> Python supports executing zipped archives, but it is popular in shell >> scripting and might still be used for self-extracting scripts that >must >> support older Python versions. This feature doesn't affect imports >and -c >> which are not expected to contain non-Python data. > >That trick doesn't work unless the data looks like Python comments or >data (e.g. a docstring). Python has always insisted on being able to >parse until EOF. The only extreme case would be a small script >followed by e.g. 4 GB of comments (where the old parser would indeed >be more efficient). But unless you can point me to an occurrence of >this in the wild I'm going to speculate that you just made this up >based on the shell analogy (which isn't perfect). > >-- >--Guido van Rossum (python.org/~guido) >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Nov 18 21:30:15 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 19 Nov 2015 12:30:15 +1000 Subject: [Python-Dev] Reading Python source file In-Reply-To: <9A6B0389-9F66-4A1B-8D96-A3E57D42DAB4@gmail.com> References: <564AFA6A.8010505@egenix.com> <564C6BD4.6030205@avl.com> <9A6B0389-9F66-4A1B-8D96-A3E57D42DAB4@gmail.com> Message-ID: On 19 November 2015 at 02:50, Ryan Gonzalez wrote: > Well, not quite the same thing, but > https://github.com/kirbyfan64/pfbuild/blob/master/pfbuild embeds the > compressed version of 16k LOC. Would it be affected negatively in any way be > this? > > Since all the data is on one line, I'd think the old (current) parser would > end up reading in the whole line anyway. Right. The other main "embedded binary blob" case I'm familiar with is get-pip.py, which embeds a base85 encoded copy of pip as a zip archive in a DATA variable, and there aren't any appending tricks there either - it's a normal variable, and the "if __name__ == '__main__':" block is the final two lines of the file, so Python already has to read the whole thing before it starts the process of unpacking it and executing it. Two things worth keeping in mind here is that if a script is embedding enough data for reading the whole thing into memory it to pose a problem: * compiling and executing it is likely to pose an even bigger problem than reading it in * downloading it in the first place is also likely to be annoying We couldn't make a change like this in a maintenance release, but for a new feature release, the consistency gain is definitely worth it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vadmium+py at gmail.com Wed Nov 18 21:28:40 2015 From: vadmium+py at gmail.com (Martin Panter) Date: Thu, 19 Nov 2015 02:28:40 +0000 Subject: [Python-Dev] [Python-checkins] cpython (3.4): asyncio.docs: Fix versionadded In-Reply-To: <20151118174516.21855.43706@psf.io> References: <20151118174516.21855.43706@psf.io> Message-ID: On 18 November 2015 at 17:45, yury.selivanov wrote: > https://hg.python.org/cpython/rev/b34c42e46e7b > changeset: 99204:b34c42e46e7b > branch: 3.4 > parent: 99201:89d66f912671 > user: Yury Selivanov > date: Wed Nov 18 12:44:31 2015 -0500 > summary: > asyncio.docs: Fix versionadded > > files: > Doc/library/asyncio-task.rst | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst > --- a/Doc/library/asyncio-task.rst > +++ b/Doc/library/asyncio-task.rst > @@ -699,4 +699,4 @@ > Unlike the functions above, :func:`run_coroutine_threadsafe` requires the > *loop* argument to be passed explicitely. > > - .. versionadded:: 3.4.4 > + .. versionadded:: 3.4.4, 3.5.1 This doesn?t get marked up properly; have a look at . From ben at groovie.org Wed Nov 18 22:53:14 2015 From: ben at groovie.org (Ben Bangert) Date: Wed, 18 Nov 2015 19:53:14 -0800 Subject: [Python-Dev] Python stdlib ssl.SSLContext is missing mode setting ability Message-ID: In Python 2 and 3, the ssl module's SSLContext object has a way to set SSL options, but not to set SSL modes. The set_mode command and some of the available modes: https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl library allows the setting of SSLContext modes, it seems very odd that the Python 2/3 ssl modules do not. Though I could understand that perhaps not all SSL libraries Python might build against would have this mode thing available. (BoringSSL sets this mode by default its considered such an obvious win) If there is some way to set this I happened to miss, apologies, I only went looking through the docs for it, not the code. On a side-note, in my testing, Python 3.4 had about 20kb/connection of overhead for using SSL, but Python 3.5 jumped to 30kb/connection of SSL overhead. These numbers for SSL overhead are far too high for any reasonable use of Python+SSL on highly concurrent systems. Test repo for testing SSL overhead here: https://github.com/bbangert/ssl-ram-testing/ Cheers, Ben From cory at lukasa.co.uk Thu Nov 19 03:14:17 2015 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 19 Nov 2015 08:14:17 +0000 Subject: [Python-Dev] Python stdlib ssl.SSLContext is missing mode setting ability In-Reply-To: References: Message-ID: <2E57C2E0-C03C-4057-8996-1D4BD691532D@lukasa.co.uk> > On 19 Nov 2015, at 03:53, Ben Bangert wrote: > > In Python 2 and 3, the ssl module's SSLContext object has a way to set > SSL options, but not to set SSL modes. > > The set_mode command and some of the available modes: > https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html > > The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the > SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl > library allows the setting of SSLContext modes, it seems very odd that > the Python 2/3 ssl modules do not. Though I could understand that > perhaps not all SSL libraries Python might build against would have > this mode thing available. > Ben, Do we need the ability to set arbitrary modes? Most of the modes mentioned in the OpenSSL documentation are things we actively don?t want the user to set because stuff will randomly break. With that in mind, and with the fact that SSL_MODE_RELEASE_BUFFERS is so obviously better than the standard, should we just instead have the ssl module automatically set SSL_MODE_RELEASE_BUFFERS unconditionally? If so, I?m happy to submit a bug/patch to get that to happen. Cory -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Thu Nov 19 04:28:43 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 19 Nov 2015 09:28:43 +0000 Subject: [Python-Dev] Reading Python source file In-Reply-To: <564C9FEB.6030604@avl.com> References: <564AFA6A.8010505@egenix.com> <564C6BD4.6030205@avl.com> <564C9FEB.6030604@avl.com> Message-ID: On 18 November 2015 at 15:57, Hrvoje Niksic wrote: > If this never really worked in Python, feel free to drop the issue. I may be > misremembering the language in which scripts I saw using this techniques > years ago were written - most likely sh or Perl. It was Perl. In the past I've tried to emulate this in Python and it's not been very successful, so I'd say it's not something to worry about here. Paul From mal at egenix.com Thu Nov 19 04:47:24 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 19 Nov 2015 10:47:24 +0100 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: <564AFA6A.8010505@egenix.com> Message-ID: <564D9AAC.1090409@egenix.com> On 17.11.2015 16:22, Guido van Rossum wrote: > On Tue, Nov 17, 2015 at 1:59 AM, M.-A. Lemburg wrote: >>> [moving from read source line by line to reading all in one go] >> We use the same simplification in eGenix PyRun's emulation of >> the Python command line interface and it has so far not >> caused any problems. > > Curious how you do it? I'd actually be quite disappointed if the > amount of parsing done by the standard REPL went down. Oh, that's easy: elif sys.argv[0] == '-' and not (pyrun_as_string or pyrun_as_module): # Read the script from stdin pyrun_as_string = True pyrun_script = sys.stdin.read() and then, later on: # Run the script try: pyrun_execute_script(pyrun_script, mode) except Exception as reason: if pyrun_interactive: import traceback traceback.print_exc() pyrun_prompt(banner='') else: raise else: # Enter interactive mode, in case wanted if pyrun_interactive: pyrun_prompt() The REPL is not affected by this, since we use the standard code.interact() for the prompt. This reads the entry line by line, joins the lines and tries to compile the entry every time it receives a new line until it succeeds or fails. Serhiy's proposed change should not affect this mode of operation. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 19 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ 2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From mal at egenix.com Thu Nov 19 04:58:31 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 19 Nov 2015 10:58:31 +0100 Subject: [Python-Dev] Python stdlib ssl.SSLContext is missing mode setting ability In-Reply-To: <2E57C2E0-C03C-4057-8996-1D4BD691532D@lukasa.co.uk> References: <2E57C2E0-C03C-4057-8996-1D4BD691532D@lukasa.co.uk> Message-ID: <564D9D47.1060500@egenix.com> On 19.11.2015 09:14, Cory Benfield wrote: > >> On 19 Nov 2015, at 03:53, Ben Bangert wrote: >> >> In Python 2 and 3, the ssl module's SSLContext object has a way to set >> SSL options, but not to set SSL modes. >> >> The set_mode command and some of the available modes: >> https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html >> >> The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the >> SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl >> library allows the setting of SSLContext modes, it seems very odd that >> the Python 2/3 ssl modules do not. Though I could understand that >> perhaps not all SSL libraries Python might build against would have >> this mode thing available. >> > > Ben, > > Do we need the ability to set arbitrary modes? Most of the modes mentioned in the OpenSSL documentation are things we actively don?t want the user to set because stuff will randomly break. With that in mind, and with the fact that SSL_MODE_RELEASE_BUFFERS is so obviously better than the standard, should we just instead have the ssl module automatically set SSL_MODE_RELEASE_BUFFERS unconditionally? > > If so, I?m happy to submit a bug/patch to get that to happen. The mode should only be enabled for OpenSSL versions which are not affected by this vulnerability: https://www.rapid7.com/db/vulnerabilities/http-openssl-cve-2014-0198 Other than that it seems like a good way forward. Plenty other projects have had this enabled per default for years: http://www.dovecot.org/list/dovecot/2011-October/131381.html https://svn.boost.org/trac/boost/changeset/71706 https://community.openvpn.net/openvpn/ticket/157 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 19 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ 2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From storchaka at gmail.com Thu Nov 19 06:51:39 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 19 Nov 2015 13:51:39 +0200 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: <564AFA6A.8010505@egenix.com> Message-ID: On 17.11.15 18:50, Guido van Rossum wrote: > On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka wrote: >> Current implementation of import system went the same way. As a result >> importing the script as a module and running it with command line can have >> different behaviours in corner cases. > > I'm confused. *Of course* these two behaviors differ, since Python > uses a different __name__. Not sure how this relates to the REPL. Sorry for confusing. I meant parser level. File parser has a few bugs, that can cause that the source will be differently interpreted with file and string parsers. For example attached script produces different output, "?" if executed as a script, and "??" if imported as a module. And there is a question about the null byte. Now compile(), exec(), eval() raises an exception if the script contains the null byte. Formerly they accepted it, but the null byte ended the script. The behavior of file parser is more weird. The null byte makes parser to ignore the end of script including the newline byte [1]. E.g. "#\0\nprint('a')" is interpreted as "#print('a')". This is different from PyPy (and may be other implementations) that interprets the null byte just as ordinal character. The question is wherever we should support the null byte in Python sources. [1] http://bugs.python.org/issue20115 -------------- next part -------------- A non-text attachment was scrubbed... Name: long_first_coding_line.py Type: text/x-python Size: 8223 bytes Desc: not available URL: From rosuav at gmail.com Thu Nov 19 07:10:18 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 19 Nov 2015 23:10:18 +1100 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: <564AFA6A.8010505@egenix.com> Message-ID: On Thu, Nov 19, 2015 at 10:51 PM, Serhiy Storchaka wrote: > http://bugs.python.org/issue20115 Interestingly, the file linked in the last comment on that issue [1] ties in with another part of this thread, regarding binary blobs in Python scripts. It uses open(sys.argv[0],'rb') to find itself, and has the binary data embedded as a comment. (Though the particular technique strikes me as fragile; to prevent newlines from messing things up, \n becomes #% and \r becomes #$, but I don't see any protection against those sequences occurring naturally in the blob. Given that it's a bz2 archive, I would expect any two-byte sequence to be capable of occurring.) To be quite honest, I wouldn't mind if Python objected to this kind of code. If I were writing it myself, I'd use a triple-quoted string containing some kind of textualized version - either the repr of the string, or some kind of base 64 or base 85 encoding. Binary data embedded literally will prevent non-ASCII file encodings. ChrisA [1] http://ftp.waf.io/pub/release/waf-1.7.16 From ncoghlan at gmail.com Thu Nov 19 08:35:01 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 19 Nov 2015 23:35:01 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: Message-ID: On 11 November 2015 at 10:47, Nick Coghlan wrote: > Our last discussion back in July seemed to show that folks either > didn't care about the question (because they're using unmodified > upstream versions so the PEP didn't affect them), or else thought the > approach described in the PEP was reasonable, so I'm hoping the > consequences of my mistake won't be too severe. > RHEL 7.2 is out today, together with details of what we've now committed to supporting for certification verification configuration in the RHEL 7 system Python: https://access.redhat.com/articles/2039753 That article also describes the PEP 476 status across the different Python builds Red Hat supports, which I've summarised below. Versions with PEP 476 implemented: * Python 3.4 SCL: cert verification always enabled by default * system Python in RHEL 7.2+: still off by default due to the compatibility break, but with PEP 493's file based configuration setting to enable it by default Versions without PEP 476 implemented: * Python 2.7 SCL * Python 3.3 SCL * system Python in RHEL 7.1 and earlier * system Python in all versions of RHEL 6 I assume that status of the Python 2.7 SCL will change at some point, but don't have an ETA or any technical details to share at this point. The system Python versions in RHEL 5 and earlier didn't include the ssl module at all (since that was only added to the standard library in Python 2.6), so they're not affected by the concerns raised in PEP 476 in the first place. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at groovie.org Thu Nov 19 10:26:28 2015 From: ben at groovie.org (Ben Bangert) Date: Thu, 19 Nov 2015 07:26:28 -0800 Subject: [Python-Dev] Python stdlib ssl.SSLContext is missing mode setting ability In-Reply-To: <564D9D47.1060500@egenix.com> References: <2E57C2E0-C03C-4057-8996-1D4BD691532D@lukasa.co.uk> <564D9D47.1060500@egenix.com> Message-ID: On Thu, Nov 19, 2015 at 1:58 AM, M.-A. Lemburg wrote: > On 19.11.2015 09:14, Cory Benfield wrote: >> >>> On 19 Nov 2015, at 03:53, Ben Bangert wrote: >>> >>> In Python 2 and 3, the ssl module's SSLContext object has a way to set >>> SSL options, but not to set SSL modes. >>> >>> The set_mode command and some of the available modes: >>> https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html >>> >>> The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the >>> SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl >>> library allows the setting of SSLContext modes, it seems very odd that >>> the Python 2/3 ssl modules do not. Though I could understand that >>> perhaps not all SSL libraries Python might build against would have >>> this mode thing available. >>> >> >> Ben, >> >> Do we need the ability to set arbitrary modes? Most of the modes mentioned in the OpenSSL documentation are things we actively don?t want the user to set because stuff will randomly break. With that in mind, and with the fact that SSL_MODE_RELEASE_BUFFERS is so obviously better than the standard, should we just instead have the ssl module automatically set SSL_MODE_RELEASE_BUFFERS unconditionally? >> >> If so, I?m happy to submit a bug/patch to get that to happen. > > The mode should only be enabled for OpenSSL versions which are > not affected by this vulnerability: > > https://www.rapid7.com/db/vulnerabilities/http-openssl-cve-2014-0198 > > Other than that it seems like a good way forward. Plenty other > projects have had this enabled per default for years: I can't think of any other mode to set, setting this with the condition cited for that vulnerability looks like a good idea. Cheers, Ben From guido at python.org Thu Nov 19 12:41:48 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 19 Nov 2015 09:41:48 -0800 Subject: [Python-Dev] Reading Python source file In-Reply-To: <564D9AAC.1090409@egenix.com> References: <564AFA6A.8010505@egenix.com> <564D9AAC.1090409@egenix.com> Message-ID: On Thu, Nov 19, 2015 at 1:47 AM, M.-A. Lemburg wrote: > On 17.11.2015 16:22, Guido van Rossum wrote: >> On Tue, Nov 17, 2015 at 1:59 AM, M.-A. Lemburg wrote: >>>> [moving from read source line by line to reading all in one go] >>> We use the same simplification in eGenix PyRun's emulation of >>> the Python command line interface and it has so far not >>> caused any problems. >> >> Curious how you do it? I'd actually be quite disappointed if the >> amount of parsing done by the standard REPL went down. > > Oh, that's easy: > > elif sys.argv[0] == '-' and not (pyrun_as_string or pyrun_as_module): > # Read the script from stdin > pyrun_as_string = True > pyrun_script = sys.stdin.read() > > and then, later on: > > # Run the script > try: > pyrun_execute_script(pyrun_script, mode) > except Exception as reason: > if pyrun_interactive: > import traceback > traceback.print_exc() > pyrun_prompt(banner='') > else: > raise > else: > # Enter interactive mode, in case wanted > if pyrun_interactive: > pyrun_prompt() Yes, this makes sense. > The REPL is not affected by this, since we use the standard > code.interact() for the prompt. This reads the entry line > by line, joins the lines and tries to compile the entry every > time it receives a new line until it succeeds or fails. Heh, I wrote code.interact() as a poor-man's substitute for what the "real" REPL (implemented in C) does. :-) It usually ends up doing the same thing, but I'm sure there are edge cases where the "real" REPL is better. It doesn't re-parse after each line is read, it actually keeps the parser state and adds new tokens read from the tty. There is even a special grammar root symbol ('single') for this mode. > Serhiy's proposed change should not affect this mode of > operation. I sure hope not. Though there is actually one case that IIRC doesn't work today: if sys.stdin is a stream that doesn't wrap a file descriptor. Would be nice to make that work. (Pretty esoteric use case though.) -- --Guido van Rossum (python.org/~guido) From cory at lukasa.co.uk Thu Nov 19 13:17:41 2015 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 19 Nov 2015 18:17:41 +0000 Subject: [Python-Dev] Python stdlib ssl.SSLContext is missing mode setting ability In-Reply-To: References: <2E57C2E0-C03C-4057-8996-1D4BD691532D@lukasa.co.uk> <564D9D47.1060500@egenix.com> Message-ID: <53DD9408-F0DF-42D4-809B-19F29E83D870@lukasa.co.uk> > On 19 Nov 2015, at 15:26, Ben Bangert wrote: > > I can't think of any other mode to set, setting this with the > condition cited for that vulnerability looks like a good idea. > > Cheers, > Ben Ok, we?re agreed. The work can be tracked under Issue 25672: https://bugs.python.org/issue25672 Cory -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From guido at python.org Thu Nov 19 13:27:18 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 19 Nov 2015 10:27:18 -0800 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: <564AFA6A.8010505@egenix.com> Message-ID: On Thu, Nov 19, 2015 at 3:51 AM, Serhiy Storchaka wrote: > On 17.11.15 18:50, Guido van Rossum wrote: >> >> On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka >> wrote: >>> >>> Current implementation of import system went the same way. As a result >>> importing the script as a module and running it with command line can >>> have >>> different behaviours in corner cases. >> >> >> I'm confused. *Of course* these two behaviors differ, since Python >> uses a different __name__. Not sure how this relates to the REPL. > > Sorry for confusing. I meant parser level. File parser has a few bugs, that > can cause that the source will be differently interpreted with file and > string parsers. For example attached script produces different output, "?" > if executed as a script, and "??" if imported as a module. I see. Well, I trust you in this area (it's been too long since I wrote the original code for all that :-). > And there is a question about the null byte. Now compile(), exec(), eval() > raises an exception if the script contains the null byte. Formerly they > accepted it, but the null byte ended the script. I like erroring out better. > The behavior of file parser > is more weird. The null byte makes parser to ignore the end of script > including the newline byte [1]. E.g. "#\0\nprint('a')" is interpreted as > "#print('a')". This is different from PyPy (and may be other > implementations) that interprets the null byte just as ordinal character. Yeah, this is just poorly written old code that didn't expect anyone to care about null bytes. IMO here too the null byte should be an error. > The question is wherever we should support the null byte in Python sources. > > [1] http://bugs.python.org/issue20115 Good luck, and thanks for working on this! -- --Guido van Rossum (python.org/~guido) From guido at python.org Thu Nov 19 13:28:02 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 19 Nov 2015 10:28:02 -0800 Subject: [Python-Dev] Reading Python source file In-Reply-To: References: <564AFA6A.8010505@egenix.com> Message-ID: Yeah, let's kill this undefined behavior. On Thu, Nov 19, 2015 at 4:10 AM, Chris Angelico wrote: > On Thu, Nov 19, 2015 at 10:51 PM, Serhiy Storchaka wrote: >> http://bugs.python.org/issue20115 > > Interestingly, the file linked in the last comment on that issue [1] > ties in with another part of this thread, regarding binary blobs in > Python scripts. It uses open(sys.argv[0],'rb') to find itself, and has > the binary data embedded as a comment. (Though the particular > technique strikes me as fragile; to prevent newlines from messing > things up, \n becomes #% and \r becomes #$, but I don't see any > protection against those sequences occurring naturally in the blob. > Given that it's a bz2 archive, I would expect any two-byte sequence to > be capable of occurring.) To be quite honest, I wouldn't mind if > Python objected to this kind of code. If I were writing it myself, I'd > use a triple-quoted string containing some kind of textualized version > - either the repr of the string, or some kind of base 64 or base 85 > encoding. Binary data embedded literally will prevent non-ASCII file > encodings. > > ChrisA > > [1] http://ftp.waf.io/pub/release/waf-1.7.16 > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From drekin at gmail.com Thu Nov 19 16:50:10 2015 From: drekin at gmail.com (=?UTF-8?B?QWRhbSBCYXJ0b8Wh?=) Date: Thu, 19 Nov 2015 22:50:10 +0100 Subject: [Python-Dev] Improving the reading part of REPL Message-ID: It seems that there will be some refactoring of the tokenizer code. Regarding this, I'd like to recall my proposal on readline hooks. It would be nice if char* based PyOS_Readline API was replaced by a Python str based hook customizable by Python code. I propose to add function sys.readlinehook accepting optional prompt and returning a line read interactively from a user. There would also be sys.__readlinehook__ containing the original value of sys.readlinehook (similarly to sys.(__)displayhook(__), sys.(__)excepthook(__) and sys.(__)std(in/out/err)(__)). Currently, the input is read from C stdin even if sys.stdin is changed (see http://bugs.python.org/issue17620). This complicates fixing http://bugs.python.org/issue1602 ? the standard sys.std* streams are not capable of communicating in Unicode with Windows console, and replacing the streams with custom ones is not enough ? one has also to install a custom readline hook, which is currently complicated. And even after installing a custom readine hook one finds out that Python tokenizer cannot handle UTF-16, so he has to wrap the custom stream objects just to let their encoding attribute have a different value, because readlinehook currently returns char* rather than a Python string. For more details see the documentation of my package: https://github.com/Drekin/win-unicode-console. The pyreadline package also sets up a custom readline so it would benefit if doing so would be easier. Moreover, the two consumers of PyOS_Readline API ? the input function and the tokenizer ? assume a different encoding of the bytes returned by the readlinehook. Effectively, one assumes sys.stdout.encoding and the other sys.stdin.encoding, so if these two are different, there is no way to implement a correct readline hook. If sys.readlinehook was added, the builting input function would be just a thin wrapper over sys.readlinehook removing the newline character and turning no input into EOFError. I thing that the best default value for sys.readlinehook on Windows would be stdio_readline ? just write the prompt to sys.stdout and read a line from sys.stdin. On Linux, the default implementation would call GNU readline if it is available and sys.stdin and sys.stdout are standard TTYs (the check present in the current implementation of the input function), and it would call stdio_readline otherwise. Regards, Adam Barto? -------------- next part -------------- An HTML attachment was scrubbed... URL: From drekin at gmail.com Fri Nov 20 10:23:49 2015 From: drekin at gmail.com (=?UTF-8?B?QWRhbSBCYXJ0b8Wh?=) Date: Fri, 20 Nov 2015 16:23:49 +0100 Subject: [Python-Dev] Improving the reading part of REPL In-Reply-To: References: Message-ID: Another issue with the current implementation is http://bugs.python.org/issue24829. Even if I fix my Python environment by win_unicode_console so >>> "?" really results in "?" rather than "?", the feature vanishes when I try to redirect stdout. On Thu, Nov 19, 2015 at 10:50 PM, Adam Barto? wrote: > It seems that there will be some refactoring of the tokenizer code. > Regarding this, I'd like to recall my proposal on readline hooks. It would > be nice if char* based PyOS_Readline API was replaced by a Python str based > hook customizable by Python code. I propose to add function > sys.readlinehook accepting optional prompt and returning a line read > interactively from a user. There would also be sys.__readlinehook__ > containing the original value of sys.readlinehook (similarly to > sys.(__)displayhook(__), sys.(__)excepthook(__) and > sys.(__)std(in/out/err)(__)). > > Currently, the input is read from C stdin even if sys.stdin is changed > (see http://bugs.python.org/issue17620). This complicates fixing > http://bugs.python.org/issue1602 ? the standard sys.std* streams are not > capable of communicating in Unicode with Windows console, and replacing the > streams with custom ones is not enough ? one has also to install a custom > readline hook, which is currently complicated. And even after installing a > custom readine hook one finds out that Python tokenizer cannot handle > UTF-16, so he has to wrap the custom stream objects just to let their > encoding attribute have a different value, because readlinehook currently > returns char* rather than a Python string. For more details see the > documentation of my package: https://github.com/Drekin/win-unicode-console > . > > The pyreadline package also sets up a custom readline so it would benefit > if doing so would be easier. Moreover, the two consumers of PyOS_Readline > API ? the input function and the tokenizer ? assume a different encoding of > the bytes returned by the readlinehook. Effectively, one assumes > sys.stdout.encoding and the other sys.stdin.encoding, so if these two are > different, there is no way to implement a correct readline hook. > > If sys.readlinehook was added, the builting input function would be just a > thin wrapper over sys.readlinehook removing the newline character and > turning no input into EOFError. I thing that the best default value for > sys.readlinehook on Windows would be stdio_readline ? just write the prompt > to sys.stdout and read a line from sys.stdin. On Linux, the default > implementation would call GNU readline if it is available and sys.stdin and > sys.stdout are standard TTYs (the check present in the current > implementation of the input function), and it would call stdio_readline > otherwise. > > Regards, Adam Barto? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Fri Nov 20 11:30:40 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 20 Nov 2015 11:30:40 -0500 Subject: [Python-Dev] nested try..except bug in generators Message-ID: <564F4AB0.5060409@gmail.com> Hi, For anyone interested in tickling the low-level CPython internals (in this case eval loop and generators implementations), here's an interesting bug: http://bugs.python.org/issue25612. I have a patch that fixes it, but I'm not sure that it's correct, and I desperately need someone to review it. While debugging the issue, I found another oddity: class MainError(Exception): pass class SubError(Exception): pass def main(): try: raise MainError() except MainError: yield coro = main() coro.send(None) coro.throw(SubError()) On the last line of the above snippet, SubError will propagate. However, it won't have it's __context__ set to MainError. Martin Panter suggests that it's not a bug, and after giving this some thought, I now don't think it's a bug too -- SubError was instantiated outside of MainError from a code location where there was no error. Thoughts? Here's a separate issue for the __context__ bug/feature: http://bugs.python.org/issue25683 Thanks, Yury From status at bugs.python.org Fri Nov 20 12:08:26 2015 From: status at bugs.python.org (Python tracker) Date: Fri, 20 Nov 2015 18:08:26 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20151120170826.D92B5567EC@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2015-11-13 - 2015-11-20) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5260 (+28) closed 32181 (+38) total 37441 (+66) Open issues with patches: 2311 Issues opened (51) ================== #25589: test_ascii_formatd fails on Mac when built with Intel compiler http://bugs.python.org/issue25589 reopened by christopher.hogan #25622: Enable ctypes.test.test_values.Win_ValuesTestCase for non-Wind http://bugs.python.org/issue25622 opened by martin.panter #25623: Keep the link to Python implementation of OrderedDict http://bugs.python.org/issue25623 opened by serhiy.storchaka #25624: shutil.make_archive makes invalid directory entries http://bugs.python.org/issue25624 opened by gumblex #25625: "chdir" Contex manager for pathlib http://bugs.python.org/issue25625 opened by J??chym Barv??nek #25626: Gzip fails for file over 2**32 bytes http://bugs.python.org/issue25626 opened by Ben Cipollini #25627: distutils : file "bdist_rpm.py" does not quote filenames when http://bugs.python.org/issue25627 opened by TheRegRunner #25628: Make namedtuple "verbose" and "rename" parameters into keyword http://bugs.python.org/issue25628 opened by rhettinger #25631: Segmentation fault with invalid Unicode command-line arguments http://bugs.python.org/issue25631 opened by itaibn #25632: Document BUILD_*_UNPACK opcodes http://bugs.python.org/issue25632 opened by Antony.Lee #25633: The documentation for urllib.request should mention http.clien http://bugs.python.org/issue25633 opened by jon orebro #25635: urllib2 cannot fully read FTP file http://bugs.python.org/issue25635 opened by keakon #25636: Let launcher autoselect best 32-bit Python on 64-bit system http://bugs.python.org/issue25636 opened by Christian.Ullrich #25637: Move non-collections-related ABCs out of collections.abc http://bugs.python.org/issue25637 opened by brett.cannon #25638: Verify the etree_parse and etree_iterparse benchmarks are work http://bugs.python.org/issue25638 opened by brett.cannon #25639: open 'PhysicalDriveN' on windows fails (since python 3.5) with http://bugs.python.org/issue25639 opened by rokozh #25640: EmailMessage should throw an error if policy is set to compat3 http://bugs.python.org/issue25640 opened by Matthieu Pepin #25641: urllib/request.py/getproxies_environment() throws "dictionary http://bugs.python.org/issue25641 opened by alphalfalfa #25642: Setting maxsize breaks asyncio.JoinableQueue/Queue http://bugs.python.org/issue25642 opened by Chris Seto #25643: Python tokenizer rewriting http://bugs.python.org/issue25643 opened by serhiy.storchaka #25644: Unable to open IDLE on Windows 10 with Python 2.7.10 http://bugs.python.org/issue25644 opened by jatanig #25646: Distutils and Windows SDK 7.1 http://bugs.python.org/issue25646 opened by JGoutin #25647: Return of asyncio.coroutine from asyncio.coroutine doesn't wor http://bugs.python.org/issue25647 opened by vrutsky #25649: Come up with a good way to handle module aliasing http://bugs.python.org/issue25649 opened by brett.cannon #25650: Mismatching documentation <=> behaviour for typing.Any http://bugs.python.org/issue25650 opened by ebarry #25651: Confusing output for TestCase.subTest(0) http://bugs.python.org/issue25651 opened by ezio.melotti #25652: collections.UserString.__rmod__() raises NameError http://bugs.python.org/issue25652 opened by Jonathan Goble #25653: ctypes+callbacks+fork+selinux = crash http://bugs.python.org/issue25653 opened by arigo #25654: test_multiprocessing_spawn ResourceWarning with -Werror http://bugs.python.org/issue25654 opened by martin.panter #25655: Python errors related to failures loading DLL's lack informati http://bugs.python.org/issue25655 opened by never-eat-yellow-snow #25656: multiprocessing.dummy: pool.map hangs on empty list http://bugs.python.org/issue25656 opened by tomer70 #25657: virtualenv's activate does not update LD_LIBRARY_PATH http://bugs.python.org/issue25657 opened by serge-sans-paille #25658: PyThread assumes pthread_key_t is an integer, which is against http://bugs.python.org/issue25658 opened by EdSchouten #25659: ctypes.Array.from_buffer segmentation fault when trying to cre http://bugs.python.org/issue25659 opened by spearsem at gmail.com #25660: tabs don't work correctly in python repl http://bugs.python.org/issue25660 opened by yselivanov #25661: tokenize.untokenize does not maintain the order of tabbed inde http://bugs.python.org/issue25661 opened by Blake Hilliard #25663: Make rlcompleter avoid duplicate global names http://bugs.python.org/issue25663 opened by martin.panter #25664: Unexpected UnicodeDecodeError in logging module http://bugs.python.org/issue25664 opened by zephor #25666: Python unexpectedly ignores a signal after fork http://bugs.python.org/issue25666 opened by jon orebro #25667: Supply dual-stack (IPv4/IPv6) socket bind routine http://bugs.python.org/issue25667 opened by jason.coombs #25668: Deadlock in logging caused by a possible race condition with " http://bugs.python.org/issue25668 opened by fviard #25669: unittest.assertEqual() on un-equal types inheriting from colle http://bugs.python.org/issue25669 opened by Nathan Herring #25671: Fix venv activate.fish to maintain $status http://bugs.python.org/issue25671 opened by Lucretiel #25672: set SSL_MODE_RELEASE_BUFFERS http://bugs.python.org/issue25672 opened by Lukasa #25674: test_ssl (test_algorithms) failures on bolen-ubuntu slaves: sh http://bugs.python.org/issue25674 opened by db3l #25675: doc for BaseEventLoop.run_in_executor() says its a coroutine, http://bugs.python.org/issue25675 opened by ealfie #25677: Syntax error caret confused by indentation http://bugs.python.org/issue25677 opened by Michael Layzell #25678: int() from a buffer reads past the buffer boundaries http://bugs.python.org/issue25678 opened by svenberkvens #25680: Selector.select() hangs when there is nothing to select http://bugs.python.org/issue25680 opened by Aleksey Kladov #25682: __package__ not set to None under pdb in Python 3 http://bugs.python.org/issue25682 opened by Keith Prussing #25683: __context__ for yields inside except clause http://bugs.python.org/issue25683 opened by yselivanov Most recent 15 issues with no replies (15) ========================================== #25683: __context__ for yields inside except clause http://bugs.python.org/issue25683 #25682: __package__ not set to None under pdb in Python 3 http://bugs.python.org/issue25682 #25675: doc for BaseEventLoop.run_in_executor() says its a coroutine, http://bugs.python.org/issue25675 #25671: Fix venv activate.fish to maintain $status http://bugs.python.org/issue25671 #25664: Unexpected UnicodeDecodeError in logging module http://bugs.python.org/issue25664 #25661: tokenize.untokenize does not maintain the order of tabbed inde http://bugs.python.org/issue25661 #25658: PyThread assumes pthread_key_t is an integer, which is against http://bugs.python.org/issue25658 #25657: virtualenv's activate does not update LD_LIBRARY_PATH http://bugs.python.org/issue25657 #25653: ctypes+callbacks+fork+selinux = crash http://bugs.python.org/issue25653 #25651: Confusing output for TestCase.subTest(0) http://bugs.python.org/issue25651 #25650: Mismatching documentation <=> behaviour for typing.Any http://bugs.python.org/issue25650 #25647: Return of asyncio.coroutine from asyncio.coroutine doesn't wor http://bugs.python.org/issue25647 #25644: Unable to open IDLE on Windows 10 with Python 2.7.10 http://bugs.python.org/issue25644 #25643: Python tokenizer rewriting http://bugs.python.org/issue25643 #25636: Let launcher autoselect best 32-bit Python on 64-bit system http://bugs.python.org/issue25636 Most recent 15 issues waiting for review (15) ============================================= #25678: int() from a buffer reads past the buffer boundaries http://bugs.python.org/issue25678 #25677: Syntax error caret confused by indentation http://bugs.python.org/issue25677 #25672: set SSL_MODE_RELEASE_BUFFERS http://bugs.python.org/issue25672 #25671: Fix venv activate.fish to maintain $status http://bugs.python.org/issue25671 #25663: Make rlcompleter avoid duplicate global names http://bugs.python.org/issue25663 #25660: tabs don't work correctly in python repl http://bugs.python.org/issue25660 #25654: test_multiprocessing_spawn ResourceWarning with -Werror http://bugs.python.org/issue25654 #25647: Return of asyncio.coroutine from asyncio.coroutine doesn't wor http://bugs.python.org/issue25647 #25643: Python tokenizer rewriting http://bugs.python.org/issue25643 #25628: Make namedtuple "verbose" and "rename" parameters into keyword http://bugs.python.org/issue25628 #25627: distutils : file "bdist_rpm.py" does not quote filenames when http://bugs.python.org/issue25627 #25626: Gzip fails for file over 2**32 bytes http://bugs.python.org/issue25626 #25624: shutil.make_archive makes invalid directory entries http://bugs.python.org/issue25624 #25623: Keep the link to Python implementation of OrderedDict http://bugs.python.org/issue25623 #25622: Enable ctypes.test.test_values.Win_ValuesTestCase for non-Wind http://bugs.python.org/issue25622 Top 10 most discussed issues (10) ================================= #25626: Gzip fails for file over 2**32 bytes http://bugs.python.org/issue25626 13 msgs #25612: nested try..excepts don't work correctly for generators http://bugs.python.org/issue25612 12 msgs #25660: tabs don't work correctly in python repl http://bugs.python.org/issue25660 8 msgs #25654: test_multiprocessing_spawn ResourceWarning with -Werror http://bugs.python.org/issue25654 7 msgs #25672: set SSL_MODE_RELEASE_BUFFERS http://bugs.python.org/issue25672 7 msgs #23883: __all__ lists are incomplete http://bugs.python.org/issue23883 6 msgs #25638: Verify the etree_parse and etree_iterparse benchmarks are work http://bugs.python.org/issue25638 6 msgs #25646: Distutils and Windows SDK 7.1 http://bugs.python.org/issue25646 6 msgs #25628: Make namedtuple "verbose" and "rename" parameters into keyword http://bugs.python.org/issue25628 5 msgs #25640: EmailMessage should throw an error if policy is set to compat3 http://bugs.python.org/issue25640 5 msgs Issues closed (36) ================== #6973: subprocess.Popen.send_signal doesn't check whether the process http://bugs.python.org/issue6973 closed by gregory.p.smith #8823: urllib2 does not catch httplib.BadStatusLine http://bugs.python.org/issue8823 closed by martin.panter #17131: subprocess.Popen.terminate can raise exception on Posix http://bugs.python.org/issue17131 closed by gregory.p.smith #20220: test_datetime, test_tarfile, test_strptime time zone failures http://bugs.python.org/issue20220 closed by martin.panter #20387: tokenize/untokenize roundtrip fails with tabs http://bugs.python.org/issue20387 closed by jason.coombs #21779: test_multiprocessing_spawn fails when ran with -Werror http://bugs.python.org/issue21779 closed by serhiy.storchaka #23564: Patch fixing sanity check for ordered fd sequence in _posixsub http://bugs.python.org/issue23564 closed by gregory.p.smith #24172: Errors in resource.getpagesize module documentation http://bugs.python.org/issue24172 closed by martin.panter #24623: Parser: broken line numbers for triple-quoted strings http://bugs.python.org/issue24623 closed by serhiy.storchaka #25017: htmllib deprecated: Which library to use? Missing sane default http://bugs.python.org/issue25017 closed by martin.panter #25168: test_datetime.test_strptime() random failures on "s390x SLES" http://bugs.python.org/issue25168 closed by martin.panter #25388: tokenizer crash/misbehavior -- heap use-after-free http://bugs.python.org/issue25388 closed by serhiy.storchaka #25472: Typing: Specialized subclasses of generics cannot be unpickled http://bugs.python.org/issue25472 closed by gvanrossum #25498: Python 3.4.3 regression in ctypes.frombuffer() http://bugs.python.org/issue25498 closed by martin.panter #25557: Optimize LOAD_NAME bytecode http://bugs.python.org/issue25557 closed by haypo #25578: Memory leak in SSLSocket.getpeercert() with 0-length AIA exten http://bugs.python.org/issue25578 closed by python-dev #25583: os.makedirs with exist_ok=True raises PermissionError on Windo http://bugs.python.org/issue25583 closed by martin.panter #25590: tab-completition on instances repeatedly accesses attribute/de http://bugs.python.org/issue25590 closed by martin.panter #25611: test_deque failure on Gentoo and OpenIndiana buildbots on 2.7 http://bugs.python.org/issue25611 closed by rhettinger #25615: Document unsorted behaviour of glob.glob http://bugs.python.org/issue25615 closed by martin.panter #25618: gdk attribute error http://bugs.python.org/issue25618 closed by r.david.murray #25619: Exception AttributeError: "'NoneType'.... thrown on exit http://bugs.python.org/issue25619 closed by r.david.murray #25620: Bytes Issue http://bugs.python.org/issue25620 closed by SilentGhost #25621: ast.literal_eval() fails on docstrings http://bugs.python.org/issue25621 closed by r.david.murray #25629: Move set fill/used updates out of set_insert_clean() http://bugs.python.org/issue25629 closed by rhettinger #25630: SIGSEGV in os.putenv() http://bugs.python.org/issue25630 closed by python-dev #25634: Add a dedicated subclass for attribute missing errors http://bugs.python.org/issue25634 closed by r.david.murray #25645: Reference leak in test_capi, on "import _pickle" in a subinter http://bugs.python.org/issue25645 closed by python-dev #25648: asyncio.coroutine fails if asyncio debug mode is enabled and w http://bugs.python.org/issue25648 closed by gvanrossum #25662: _tkinter.TclError: bad event type or keysym "Alt" http://bugs.python.org/issue25662 closed by ronixx #25665: typing.NamedTuple instances are not picklable. http://bugs.python.org/issue25665 closed by gvanrossum #25670: ast.NodeTransformer: remove duplicate getattr() http://bugs.python.org/issue25670 closed by haypo #25673: unittest assertLessEqual not working properly with lists http://bugs.python.org/issue25673 closed by ezio.melotti #25676: Spam http://bugs.python.org/issue25676 closed by zach.ware #25679: Fix typo in Doc/reference/executionmodel.rst http://bugs.python.org/issue25679 closed by r.david.murray #25681: Assignment of one element in nested list changes multiple elem http://bugs.python.org/issue25681 closed by haypo From benjamin at python.org Sat Nov 21 21:48:27 2015 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 21 Nov 2015 18:48:27 -0800 Subject: [Python-Dev] [RELASE] Python 2.7.11 release candidate 1 Message-ID: <1448160507.1985354.446437097.5E584CD6@webmail.messagingengine.com> Greetings, It is my duty and honor to announce the immediate availability of Python 2.7.11 release candidate 1. Python 2.7.11 will be another bug fix release for the 2.7.x series. Downloads are at: https://www.python.org/downloads/release/python-2711rc1/ Please test the candidate and report bugs to https://bugs.python.org If no serious problems are found, 2.7.11 final will be released in two weeks. Regards, Benjamin Peterson (on behalf of Python 2.7.11's contributors) From larry at hastings.org Sun Nov 22 03:12:35 2015 From: larry at hastings.org (Larry Hastings) Date: Sun, 22 Nov 2015 00:12:35 -0800 Subject: [Python-Dev] Release schedule for 3.4.4 Message-ID: <565178F3.5070701@hastings.org> Sorry for the short notice of this schedule, but... here goes: Sat Dec 05 - tag 3.4.4rc1 Sun Dec 06 - release 3.4.4rc1 Sat Dec 19 - tag 3.4.4 final Sun Dec 20 - release 3.4.4 final The reason for the short notice: we may have to change personnel for this release, and we need to work around some holiday vacation schedules too. However, note that we have two weeks for the RC release. Because of the short notice, I plan to be a little more flexible about changes between RC1 and final. It also means we can pretty easily add a second RC if we need to. Unless something amazing happens, 3.4.4 will be the final release of 3.4 with binary installers. 3.4 will also move into "security fixes only" mode at that time. Best wishes, and again my apologies for the short notice, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Mon Nov 23 01:21:43 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Mon, 23 Nov 2015 01:21:43 -0500 Subject: [Python-Dev] collections.Counter __add__ implementation quirk Message-ID: <5652B077.9000606@sdamon.com> collections.Counter.__add__ as a bit of a quirk. Counters allow for negative numbers. You can subtract from a counter into the negative no problem. However, if you have a counter with a negative value and add it to another counter, and if that value, after addition, would still be negative... that value is not included in the resulting Counter object. This is kind of weird, to the point of thinking I had a bug in other code for several hours until I went and checked how Counters are implemented. Is there any particular reason counters drop negative values when you add them together? I definitely expected them to act like ints do when you add negatives, and had to subclass it to get what I think is the obvious behavior. From larry at hastings.org Mon Nov 23 02:16:44 2015 From: larry at hastings.org (Larry Hastings) Date: Sun, 22 Nov 2015 23:16:44 -0800 Subject: [Python-Dev] [RELEASED] Python 3.5.1rc1 is now available Message-ID: <5652BD5C.6040107@hastings.org> On behalf of the Python development community and the Python 3.5 release team, I'm pleased to announce the availability of Python 3.5.1rc1. Python 3.5.1 will be the first update for Python 3.5. Python 3.5 is the newest version of the Python language, and it contains many exciting new features and optimizations. You can see what's changed in Python 3.5.1rc1 (as compared to 3.5.0) here: https://docs.python.org/3.5/whatsnew/changelog.html And you can download Python 3.5.1 here: https://www.python.org/downloads/release/python-351rc1/ Windows and Mac users: please read the important platform-specific "Notes on this release" section near the end of that page. We hope you enjoy Python 3.5.1! //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Mon Nov 23 02:47:33 2015 From: larry at hastings.org (Larry Hastings) Date: Sun, 22 Nov 2015 23:47:33 -0800 Subject: [Python-Dev] Reminder: checkins into 3.5 after rc1 won't automatically go into final Message-ID: <5652C495.2080804@hastings.org> 3.5.1rc1 is tagged, merged, and pushed back into the central repo. At this point I don't plan to merge further changes from hg.python.org into 3.5.1 final. If you have an important bug fix that didn't make it into 3.5.1rc1 and *has* to go into 3.5.1 final, add me to the issue and we'll talk about it. If it happens I'll have to manually cherry-pick the change. I've added a 3.5.2rc1 section to Misc/NEWS; please add new entries there. Happy holidays, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlastimil.brom at gmail.com Mon Nov 23 13:43:15 2015 From: vlastimil.brom at gmail.com (Vlastimil Brom) Date: Mon, 23 Nov 2015 19:43:15 +0100 Subject: [Python-Dev] collections.Counter __add__ implementation quirk In-Reply-To: <5652B077.9000606@sdamon.com> References: <5652B077.9000606@sdamon.com> Message-ID: 2015-11-23 7:21 GMT+01:00 Alexander Walters : > collections.Counter.__add__ as a bit of a quirk. > > Counters allow for negative numbers. You can subtract from a counter into > the negative no problem. However, if you have a counter with a negative > value and add it to another counter, and if that value, after addition, > would still be negative... that value is not included in the resulting > Counter object. This is kind of weird, to the point of thinking I had a bug > in other code for several hours until I went and checked how Counters are > implemented. > > Is there any particular reason counters drop negative values when you add > them together? I definitely expected them to act like ints do when you add > negatives, and had to subclass it to get what I think is the obvious > behavior. > _______________________________________________ > Python-Dev mailing list ... Hi, this is probably more appropriate for the general python list rathere then this developers' maillist, however, as I asked a similar question some time ago, I got some detailed explanations for the the current design decissions from the original developer; cf.: https://mail.python.org/pipermail/python-list/2010-March/570618.html (I didn't check possible changes in Counter since that version (3.1 at that time).) hth, vbr From christian at python.org Mon Nov 23 14:35:06 2015 From: christian at python.org (Christian Heimes) Date: Mon, 23 Nov 2015 20:35:06 +0100 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: Message-ID: <56536A6A.9030902@python.org> On 2015-11-17 01:00, Guido van Rossum wrote: > Hm, making Christian the BDFL-delegate would mean two out of three > authors *and* the BDFL-delegate all working for Red Hat, which clearly > has a stake (and IIUC has already committed to this approach ahead of > PEP approval). SO then it would look like this is just rubber-stamping > Red Hat's internal decision process (if it's a process -- sounds more > like an accident :-). > > So, Alex, do you want to approve this PEP? I haven't read this thread until now. Independently from your objection I have raised the same concern with Nick today. I'd be willing to BDFL the PEP but I'd rather have somebody outside of Red Hat. Alex is a great choice. In the same mail I sent Nick a quick review of the latest PEP version in private. 1) The example implementation of the function doesn't check the sys.flags.ignore_environment. Internally CPython has specialized getenv function that ignores env vars with PYTHON prefix when the flag is set. PYTHON* env vars aren't removed from os.environ. Modules have to check the flag. 2) The PEP is rather Linux-centric. What's the recommended path to the config file on other platforms like BDS (/usr/local/etc/ is preferred for additional dependencies on FreeBSD), OSX and Windows? 3) What's the interaction between the location of the config file and virtual envs? Would it make sense to search for the file in a venv's etc/ first and then dispatch to global /etc/? That way venvs can influence the setting, too. 4) It makes sense to make the cert-verification.cfg file future-proof and reserve it for other cert-related configuration in the future. For example it could be used to define new contexts, set protocols, ciphers or hashes for cert pinning. It should be enough to say that CPython reserves the right to add more sections and keys later. 5) I'm not particular fond of the section name [https]. For one It is ambiguous because it doesn't distinguish between server certs and client certs. It's also not correct. The default context is used for other protocols like imap, smtp etc. over TLS. Christian From christian at python.org Mon Nov 23 14:35:06 2015 From: christian at python.org (Christian Heimes) Date: Mon, 23 Nov 2015 20:35:06 +0100 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: Message-ID: <56536A6A.9030902@python.org> On 2015-11-17 01:00, Guido van Rossum wrote: > Hm, making Christian the BDFL-delegate would mean two out of three > authors *and* the BDFL-delegate all working for Red Hat, which clearly > has a stake (and IIUC has already committed to this approach ahead of > PEP approval). SO then it would look like this is just rubber-stamping > Red Hat's internal decision process (if it's a process -- sounds more > like an accident :-). > > So, Alex, do you want to approve this PEP? I haven't read this thread until now. Independently from your objection I have raised the same concern with Nick today. I'd be willing to BDFL the PEP but I'd rather have somebody outside of Red Hat. Alex is a great choice. In the same mail I sent Nick a quick review of the latest PEP version in private. 1) The example implementation of the function doesn't check the sys.flags.ignore_environment. Internally CPython has specialized getenv function that ignores env vars with PYTHON prefix when the flag is set. PYTHON* env vars aren't removed from os.environ. Modules have to check the flag. 2) The PEP is rather Linux-centric. What's the recommended path to the config file on other platforms like BDS (/usr/local/etc/ is preferred for additional dependencies on FreeBSD), OSX and Windows? 3) What's the interaction between the location of the config file and virtual envs? Would it make sense to search for the file in a venv's etc/ first and then dispatch to global /etc/? That way venvs can influence the setting, too. 4) It makes sense to make the cert-verification.cfg file future-proof and reserve it for other cert-related configuration in the future. For example it could be used to define new contexts, set protocols, ciphers or hashes for cert pinning. It should be enough to say that CPython reserves the right to add more sections and keys later. 5) I'm not particular fond of the section name [https]. For one It is ambiguous because it doesn't distinguish between server certs and client certs. It's also not correct. The default context is used for other protocols like imap, smtp etc. over TLS. Christian From alex.gaynor at gmail.com Mon Nov 23 14:38:36 2015 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 23 Nov 2015 14:38:36 -0500 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <56536A6A.9030902@python.org> References: <56536A6A.9030902@python.org> Message-ID: Hi all, While I appreciate the vote of confidence from everyone, I'm not interested in being the BDFL-delegate for this. I don't think it's a good idea, and I'm not willing to put further time into. If he's interested, Donald Stufft would make a good choice for delegate. Really do appreciate everyone's confidence. Cheers, Alex On Mon, Nov 23, 2015 at 2:35 PM, Christian Heimes wrote: > On 2015-11-17 01:00, Guido van Rossum wrote: > > Hm, making Christian the BDFL-delegate would mean two out of three > > authors *and* the BDFL-delegate all working for Red Hat, which clearly > > has a stake (and IIUC has already committed to this approach ahead of > > PEP approval). SO then it would look like this is just rubber-stamping > > Red Hat's internal decision process (if it's a process -- sounds more > > like an accident :-). > > > > So, Alex, do you want to approve this PEP? > > I haven't read this thread until now. Independently from your objection > I have raised the same concern with Nick today. I'd be willing to BDFL > the PEP but I'd rather have somebody outside of Red Hat. Alex is a great > choice. > > > In the same mail I sent Nick a quick review of the latest PEP version in > private. > > > 1) The example implementation of the function doesn't check the > sys.flags.ignore_environment. Internally CPython has specialized getenv > function that ignores env vars with PYTHON prefix when the flag is set. > PYTHON* env vars aren't removed from os.environ. Modules have to check > the flag. > > > 2) The PEP is rather Linux-centric. What's the recommended path to the > config file on other platforms like BDS (/usr/local/etc/ is preferred > for additional dependencies on FreeBSD), OSX and Windows? > > > 3) What's the interaction between the location of the config file and > virtual envs? Would it make sense to search for the file in a venv's > etc/ first and then dispatch to global /etc/? That way venvs can > influence the setting, too. > > > 4) It makes sense to make the cert-verification.cfg file future-proof > and reserve it for other cert-related configuration in the future. For > example it could be used to define new contexts, set protocols, ciphers > or hashes for cert pinning. It should be enough to say that CPython > reserves the right to add more sections and keys later. > > > 5) I'm not particular fond of the section name [https]. For one It is > ambiguous because it doesn't distinguish between server certs and client > certs. It's also not correct. The default context is used for other > protocols like imap, smtp etc. over TLS. > > Christian > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Mon Nov 23 15:47:52 2015 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 23 Nov 2015 14:47:52 -0600 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: Message-ID: 1. Does this affect easy_install? 2. If/because this affects easy_install, should the guidance / suggested package installation tool be [pip]; because pip install_requires backports.ssl_match_hostname https://pypi.python.org/pypi/backports.ssl_match_hostname On Nov 10, 2015 6:48 PM, "Nick Coghlan" wrote: > Hi folks, > > I have a confession to make - I dropped the ball on the HTTPS > verification backport proposals in PEP 493, and let the upstream and > downstream approval processes get out of sequence. > > As a result, the RHEL 7.2 beta released back in September incorporates > the HTTPS verification feature backport based on the current PEP 493 > draft, even though that hasn't formally been pronounced as an Active > recommendation by python-dev yet. > > Accordingly, I'm belatedly submitting it for pronouncement now: > https://www.python.org/dev/peps/pep-0493/ > > There's currently no BDFL-Delegate assigned, so if Guido doesn't want > to handle it, we'll need to address that question first. > > Our last discussion back in July seemed to show that folks either > didn't care about the question (because they're using unmodified > upstream versions so the PEP didn't affect them), or else thought the > approach described in the PEP was reasonable, so I'm hoping the > consequences of my mistake won't be too severe. > > Regards, > Nick. > > P.S. I'm aware that this looks like presenting a fait accompli at a > point where it's too late to realistically say "No", but the truth is > that preparation for the Python in Education miniconf at PyCon > Australia ramped up immediately after the July discussion, and then I > personally got confused as to the scope of what was being included in > 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493 > being deferred to a later release, but it's actually the whole package > of 466+476+493). That's my fault for trying to keep track of too many > things at once (and thus failing at some of them), not anyone else's. > > ================================ > > PEP: 493 > Title: HTTPS verification recommendations for Python 2.7 redistributors > Version: $Revision$ > Last-Modified: $Date$ > Author: Nick Coghlan , > Robert Kuska , > Marc-Andr? Lemburg > Status: Draft > Type: Informational > Content-Type: text/x-rst > Created: 10-May-2015 > Post-History: 06-Jul-2015 > > > Abstract > ======== > > PEP 476 updated Python's default handling of HTTPS certificates to be > appropriate for communication over the public internet. The Python 2.7 long > term maintenance series was judged to be in scope for this change, with the > new behaviour introduced in the Python 2.7.9 maintenance release. > > This PEP provides recommendations to downstream redistributors wishing to > provide a smoother migration experience when helping their users to manage > this change in Python's default behaviour. > > > Rationale > ========= > > PEP 476 changed Python's default behaviour to better match the needs and > expectations of developers operating over the public internet, a category > which appears to include most new Python developers. It is the position of > the authors of this PEP that this was a correct decision. > > However, it is also the case that this change *does* cause problems for > infrastructure administrators operating private intranets that rely on > self-signed certificates, or otherwise encounter problems with the new > default > certificate verification settings. > > The long term answer for such environments is to update their internal > certificate management to at least match the standards set by the public > internet, but in the meantime, it is desirable to offer these > administrators > a way to continue receiving maintenance updates to the Python 2.7 series, > without having to gate that on upgrades to their certificate management > infrastructure. > > PEP 476 did attempt to address this question, by covering how to revert the > new settings process wide by monkeypatching the ``ssl`` module to restore > the > old behaviour. Unfortunately, the ``sitecustomize.py`` based technique > proposed > to allow system administrators to disable the feature by default in their > Standard Operating Environment definition has been determined to be > insufficient in at least some cases. The specific case of interest to the > authors of this PEP is the one where a Linux distributor aims to provide > their users with a > `smoother migration path > `__ > than the standard one provided by consuming upstream CPython 2.7 releases > directly, but other potential challenges have also been pointed out with > updating embedded Python runtimes and other user level installations of > Python. > > Rather than allowing a plethora of mutually incompatibile migration > techniques > to bloom, this PEP proposes two alternative approaches that redistributors > may take when addressing these problems. Redistributors may choose to > implement > one, both, or neither of these approaches based on their assessment of the > needs of their particular userbase. > > These designs are being proposed as a recommendation for redistributors, > rather > than as new upstream features, as they are needed purely to support legacy > environments migrating from older versions of Python 2.7. Neither approach > is being proposed as an upstream Python 2.7 feature, nor as a feature in > any > version of Python 3 (whether published directly by the Python Software > Foundation or by a redistributor). > > > Requirements for capability detection > ===================================== > > As these recommendations are intended to cover backports to earlier Python > versions, the Python version number cannot be used as a reliable means for > detecting them. Instead, the recommendations are defined to allow the > presence > or absence of the feature to be determined using the following technique:: > > python -c "import ssl; ssl._relevant_attribute" > > This will fail with `AttributeError` (and hence a non-zero return code) if > the > relevant capability is not available. > > The marker attributes are prefixed with an underscore to indicate the > implementation dependent nature of these capabilities - not all Python > distributions will offer them, only those that are providing a multi-stage > migration process from the legacy HTTPS handling to the new default > behaviour. > > > Recommendation for an environment variable based security downgrade > =================================================================== > > Some redistributors may wish to provide a per-application option to disable > certificate verification in selected applications that run on or embed > CPython > without needing to modify the application itself. > > In these cases, a configuration mechanism is needed that provides: > > * an opt-out model that allows certificate verification to be selectively > turned off for particular applications after upgrading to a version of > Python that verifies certificates by default > * the ability for all users to configure this setting on a per-application > basis, rather than on a per-system, or per-Python-installation basis > > This approach may be used for any redistributor provided version of Python > 2.7, > including those that advertise themselves as providing Python 2.7.9 or > later. > > > Required marker attribute > ------------------------- > > The required marker attribute on the ``ssl`` module when implementing this > recommendation is:: > > _https_verify_envvar = 'PYTHONHTTPSVERIFY' > > This not only makes it straightforward to detect the presence (or absence) > of > the capability, it also makes it possible to programmatically determine the > relevant environment variable name. > > > Recommended modifications to the Python standard library > -------------------------------------------------------- > > The recommended approach to providing a per-application configuration > setting > for HTTPS certificate verification that doesn't require modifications to > the > application itself is to: > > * modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment > variable when the module is first imported into a Python process > * set the ``ssl._create_default_https_context`` function to be an alias for > ``ssl._create_unverified_context`` if this environment variable is > present > and set to ``'0'`` > * otherwise, set the ``ssl._create_default_https_context`` function to be > an > alias for ``ssl.create_default_context`` as usual > > > Example implementation > ---------------------- > > :: > > _https_verify_envvar = 'PYTHONHTTPSVERIFY' > > def _get_https_context_factory(): > config_setting = os.environ.get(_https_verify_envvar) > if config_setting == '0': > return _create_unverified_context > return create_default_context > > _create_default_https_context = _get_https_context_factory() > > > Security Considerations > ----------------------- > > Relative to an unmodified version of CPython 2.7.9 or later, this approach > does introduce a new downgrade attack against the default security settings > that potentially allows a sufficiently determined attacker to revert Python > to the vulnerable configuration used in CPython 2.7.8 and earlier releases. > However, such an attack requires the ability to modify the execution > environment of a Python process prior to the import of the ``ssl`` module, > and any attacker with such access would already be able to modify the > behaviour of the underlying OpenSSL implementation. > > > Recommendation for backporting to earlier Python versions > ========================================================= > > Some redistributors, most notably Linux distributions, may choose to > backport > the PEP 476 HTTPS verification changes to modified Python versions based on > earlier Python 2 maintenance releases. In these cases, a configuration > mechanism is needed that provides: > > * an opt-in model that allows the decision to enable HTTPS certificate > verification to be made independently of the decision to upgrade to the > Python version where the feature was first backported > * the ability for system administrators to set the default behaviour of > Python > applications and scripts run directly in the system Python installation > * the ability for the redistributor to consider changing the default > behaviour > of *new* installations at some point in the future without impacting > existing > installations that have been explicitly configured to skip verifying > HTTPS > certificates by default > > This approach should not be used for any Python installation that > advertises > itself as providing Python 2.7.9 or later, as most Python users will have > the > reasonable expectation that all such environments will validate HTTPS > certificates by default. > > > Required marker attribute > ------------------------- > > The required marker attribute on the ``ssl`` module when implementing this > recommendation is:: > > _cert_verification_config = '' > > This not only makes it straightforward to detect the presence (or absence) > of > the capability, it also makes it possible to programmatically determine the > relevant configuration file name. > > > Recommended modifications to the Python standard library > -------------------------------------------------------- > > The recommended approach to backporting the PEP 476 modifications to an > earlier > point release is to implement the following changes relative to the default > PEP 476 behaviour implemented in Python 2.7.9+: > > * modify the ``ssl`` module to read a system wide configuration file when > the > module is first imported into a Python process > * define a platform default behaviour (either verifying or not verifying > HTTPS > certificates) to be used if this configuration file is not present > * support selection between the following three modes of operation: > > * ensure HTTPS certificate verification is enabled > * ensure HTTPS certificate verification is disabled > * delegate the decision to the redistributor providing this Python > version > > * set the ``ssl._create_default_https_context`` function to be an alias for > either ``ssl.create_default_context`` or > ``ssl._create_unverified_context`` > based on the given configuration setting. > > > Recommended file location > ------------------------- > > This approach is currently only defined for \*nix system Python > installations. > > The recommended configuration file name is > ``/etc/python/cert-verification.cfg``. > > The ``.cfg`` filename extension is recommended for consistency with the > ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library. > > > Recommended file format > ----------------------- > > The configuration file should use a ConfigParser ini-style format with a > single section named ``[https]`` containing one required setting > ``verify``. > > Permitted values for ``verify`` are: > > * ``enable``: ensure HTTPS certificate verification is enabled by default > * ``disable``: ensure HTTPS certificate verification is disabled by default > * ``platform_default``: delegate the decision to the redistributor > providing > this particular Python version > > If the ``[https]`` section or the ``verify`` setting are missing, or if the > ``verify`` setting is set to an unknown value, it should be treated as if > the > configuration file is not present. > > > Example implementation > ---------------------- > > :: > > _cert_verification_config = '/etc/python/cert-verification.cfg' > > def _get_https_context_factory(): > # Check for a system-wide override of the default behaviour > context_factories = { > 'enable': create_default_context, > 'disable': _create_unverified_context, > 'platform_default': _create_unverified_context, # For now :) > } > import ConfigParser > config = ConfigParser.RawConfigParser() > config.read(_cert_verification_config) > try: > verify_mode = config.get('https', 'verify') > except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): > verify_mode = 'platform_default' > default_factory = context_factories.get('platform_default') > return context_factories.get(verify_mode, default_factory) > > _create_default_https_context = _get_https_context_factory() > > > Security Considerations > ----------------------- > > The specific recommendations for the backporting case are designed to work > for > privileged, security sensitive processes, even those being run in the > following > locked down configuration: > > * run from a locked down administrator controlled directory rather than a > normal > user directory (preventing ``sys.path[0]`` based privilege escalation > attacks) > * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable > based > privilege escalation attacks) > * run using the ``-s`` switch (preventing user site directory based > privilege > escalation attacks) > * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege > escalation attacks) > > The intent is that the *only* reason HTTPS verification should be getting > turned off system wide when using this approach is because: > > * an end user is running a redistributor provided version of CPython rather > than running upstream CPython directly > * that redistributor has decided to provide a smoother migration path to > verifying HTTPS certificates by default than that being provided by the > upstream project > * either the redistributor or the local infrastructure administrator has > determined that it is appropriate to override the default upstream > behaviour > (at least for the time being) > > Using an administrator controlled configuration file rather than an > environment > variable has the essential feature of providing a smoother migration path, > even > for applications being run with the ``-E`` switch. > > > Combining the recommendations > ============================= > > If a redistributor chooses to implement both recommendations, then the > environment variable should take precedence over the system-wide > configuration > setting. This allows the setting to be changed for a given user, virtual > environment or application, regardless of the system-wide default > behaviour. > > In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is > defined, and > set to anything *other* than ``'0'``, then HTTPS certificate verification > should be enabled. > > Example implementation > ---------------------- > > :: > > _https_verify_envvar = 'PYTHONHTTPSVERIFY' > _cert_verification_config = '/etc/python/cert-verification.cfg' > > def _get_https_context_factory(): > # Check for am environmental override of the default behaviour > config_setting = os.environ.get(_https_verify_envvar) > if config_setting is not None: > if config_setting == '0': > return _create_unverified_context > return create_default_context > > # Check for a system-wide override of the default behaviour > context_factories = { > 'enable': create_default_context, > 'disable': _create_unverified_context, > 'platform_default': _create_unverified_context, # For now :) > } > import ConfigParser > config = ConfigParser.RawConfigParser() > config.read(_cert_verification_config) > try: > verify_mode = config.get('https', 'verify') > except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): > verify_mode = 'platform_default' > default_factory = context_factories.get('platform_default') > return context_factories.get(verify_mode, default_factory) > > _create_default_https_context = _get_https_context_factory() > > > Copyright > ========= > > This document has been placed into the public domain. > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Mon Nov 23 15:51:01 2015 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 23 Nov 2015 14:51:01 -0600 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: Message-ID: ... Just had this discussion in regards to easy_install, Ubuntu 14.04 LTS, and the ReadTheDocs Docker images (as well as: ~why should I have to wget/curl get-pip.py) https://github.com/rtfd/readthedocs-docker-images/pull/3 On Nov 23, 2015 2:47 PM, "Wes Turner" wrote: > 1. Does this affect easy_install? > 2. If/because this affects easy_install, > should the guidance / suggested package installation tool be [pip]; > because pip install_requires backports.ssl_match_hostname > > https://pypi.python.org/pypi/backports.ssl_match_hostname > On Nov 10, 2015 6:48 PM, "Nick Coghlan" wrote: > >> Hi folks, >> >> I have a confession to make - I dropped the ball on the HTTPS >> verification backport proposals in PEP 493, and let the upstream and >> downstream approval processes get out of sequence. >> >> As a result, the RHEL 7.2 beta released back in September incorporates >> the HTTPS verification feature backport based on the current PEP 493 >> draft, even though that hasn't formally been pronounced as an Active >> recommendation by python-dev yet. >> >> Accordingly, I'm belatedly submitting it for pronouncement now: >> https://www.python.org/dev/peps/pep-0493/ >> >> There's currently no BDFL-Delegate assigned, so if Guido doesn't want >> to handle it, we'll need to address that question first. >> >> Our last discussion back in July seemed to show that folks either >> didn't care about the question (because they're using unmodified >> upstream versions so the PEP didn't affect them), or else thought the >> approach described in the PEP was reasonable, so I'm hoping the >> consequences of my mistake won't be too severe. >> >> Regards, >> Nick. >> >> P.S. I'm aware that this looks like presenting a fait accompli at a >> point where it's too late to realistically say "No", but the truth is >> that preparation for the Python in Education miniconf at PyCon >> Australia ramped up immediately after the July discussion, and then I >> personally got confused as to the scope of what was being included in >> 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493 >> being deferred to a later release, but it's actually the whole package >> of 466+476+493). That's my fault for trying to keep track of too many >> things at once (and thus failing at some of them), not anyone else's. >> >> ================================ >> >> PEP: 493 >> Title: HTTPS verification recommendations for Python 2.7 redistributors >> Version: $Revision$ >> Last-Modified: $Date$ >> Author: Nick Coghlan , >> Robert Kuska , >> Marc-Andr? Lemburg >> Status: Draft >> Type: Informational >> Content-Type: text/x-rst >> Created: 10-May-2015 >> Post-History: 06-Jul-2015 >> >> >> Abstract >> ======== >> >> PEP 476 updated Python's default handling of HTTPS certificates to be >> appropriate for communication over the public internet. The Python 2.7 >> long >> term maintenance series was judged to be in scope for this change, with >> the >> new behaviour introduced in the Python 2.7.9 maintenance release. >> >> This PEP provides recommendations to downstream redistributors wishing to >> provide a smoother migration experience when helping their users to manage >> this change in Python's default behaviour. >> >> >> Rationale >> ========= >> >> PEP 476 changed Python's default behaviour to better match the needs and >> expectations of developers operating over the public internet, a category >> which appears to include most new Python developers. It is the position of >> the authors of this PEP that this was a correct decision. >> >> However, it is also the case that this change *does* cause problems for >> infrastructure administrators operating private intranets that rely on >> self-signed certificates, or otherwise encounter problems with the new >> default >> certificate verification settings. >> >> The long term answer for such environments is to update their internal >> certificate management to at least match the standards set by the public >> internet, but in the meantime, it is desirable to offer these >> administrators >> a way to continue receiving maintenance updates to the Python 2.7 series, >> without having to gate that on upgrades to their certificate management >> infrastructure. >> >> PEP 476 did attempt to address this question, by covering how to revert >> the >> new settings process wide by monkeypatching the ``ssl`` module to restore >> the >> old behaviour. Unfortunately, the ``sitecustomize.py`` based technique >> proposed >> to allow system administrators to disable the feature by default in their >> Standard Operating Environment definition has been determined to be >> insufficient in at least some cases. The specific case of interest to the >> authors of this PEP is the one where a Linux distributor aims to provide >> their users with a >> `smoother migration path >> `__ >> than the standard one provided by consuming upstream CPython 2.7 releases >> directly, but other potential challenges have also been pointed out with >> updating embedded Python runtimes and other user level installations of >> Python. >> >> Rather than allowing a plethora of mutually incompatibile migration >> techniques >> to bloom, this PEP proposes two alternative approaches that redistributors >> may take when addressing these problems. Redistributors may choose to >> implement >> one, both, or neither of these approaches based on their assessment of the >> needs of their particular userbase. >> >> These designs are being proposed as a recommendation for redistributors, >> rather >> than as new upstream features, as they are needed purely to support legacy >> environments migrating from older versions of Python 2.7. Neither approach >> is being proposed as an upstream Python 2.7 feature, nor as a feature in >> any >> version of Python 3 (whether published directly by the Python Software >> Foundation or by a redistributor). >> >> >> Requirements for capability detection >> ===================================== >> >> As these recommendations are intended to cover backports to earlier Python >> versions, the Python version number cannot be used as a reliable means for >> detecting them. Instead, the recommendations are defined to allow the >> presence >> or absence of the feature to be determined using the following technique:: >> >> python -c "import ssl; ssl._relevant_attribute" >> >> This will fail with `AttributeError` (and hence a non-zero return code) >> if the >> relevant capability is not available. >> >> The marker attributes are prefixed with an underscore to indicate the >> implementation dependent nature of these capabilities - not all Python >> distributions will offer them, only those that are providing a multi-stage >> migration process from the legacy HTTPS handling to the new default >> behaviour. >> >> >> Recommendation for an environment variable based security downgrade >> =================================================================== >> >> Some redistributors may wish to provide a per-application option to >> disable >> certificate verification in selected applications that run on or embed >> CPython >> without needing to modify the application itself. >> >> In these cases, a configuration mechanism is needed that provides: >> >> * an opt-out model that allows certificate verification to be selectively >> turned off for particular applications after upgrading to a version of >> Python that verifies certificates by default >> * the ability for all users to configure this setting on a per-application >> basis, rather than on a per-system, or per-Python-installation basis >> >> This approach may be used for any redistributor provided version of >> Python 2.7, >> including those that advertise themselves as providing Python 2.7.9 or >> later. >> >> >> Required marker attribute >> ------------------------- >> >> The required marker attribute on the ``ssl`` module when implementing this >> recommendation is:: >> >> _https_verify_envvar = 'PYTHONHTTPSVERIFY' >> >> This not only makes it straightforward to detect the presence (or >> absence) of >> the capability, it also makes it possible to programmatically determine >> the >> relevant environment variable name. >> >> >> Recommended modifications to the Python standard library >> -------------------------------------------------------- >> >> The recommended approach to providing a per-application configuration >> setting >> for HTTPS certificate verification that doesn't require modifications to >> the >> application itself is to: >> >> * modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment >> variable when the module is first imported into a Python process >> * set the ``ssl._create_default_https_context`` function to be an alias >> for >> ``ssl._create_unverified_context`` if this environment variable is >> present >> and set to ``'0'`` >> * otherwise, set the ``ssl._create_default_https_context`` function to be >> an >> alias for ``ssl.create_default_context`` as usual >> >> >> Example implementation >> ---------------------- >> >> :: >> >> _https_verify_envvar = 'PYTHONHTTPSVERIFY' >> >> def _get_https_context_factory(): >> config_setting = os.environ.get(_https_verify_envvar) >> if config_setting == '0': >> return _create_unverified_context >> return create_default_context >> >> _create_default_https_context = _get_https_context_factory() >> >> >> Security Considerations >> ----------------------- >> >> Relative to an unmodified version of CPython 2.7.9 or later, this approach >> does introduce a new downgrade attack against the default security >> settings >> that potentially allows a sufficiently determined attacker to revert >> Python >> to the vulnerable configuration used in CPython 2.7.8 and earlier >> releases. >> However, such an attack requires the ability to modify the execution >> environment of a Python process prior to the import of the ``ssl`` module, >> and any attacker with such access would already be able to modify the >> behaviour of the underlying OpenSSL implementation. >> >> >> Recommendation for backporting to earlier Python versions >> ========================================================= >> >> Some redistributors, most notably Linux distributions, may choose to >> backport >> the PEP 476 HTTPS verification changes to modified Python versions based >> on >> earlier Python 2 maintenance releases. In these cases, a configuration >> mechanism is needed that provides: >> >> * an opt-in model that allows the decision to enable HTTPS certificate >> verification to be made independently of the decision to upgrade to the >> Python version where the feature was first backported >> * the ability for system administrators to set the default behaviour of >> Python >> applications and scripts run directly in the system Python installation >> * the ability for the redistributor to consider changing the default >> behaviour >> of *new* installations at some point in the future without impacting >> existing >> installations that have been explicitly configured to skip verifying >> HTTPS >> certificates by default >> >> This approach should not be used for any Python installation that >> advertises >> itself as providing Python 2.7.9 or later, as most Python users will have >> the >> reasonable expectation that all such environments will validate HTTPS >> certificates by default. >> >> >> Required marker attribute >> ------------------------- >> >> The required marker attribute on the ``ssl`` module when implementing this >> recommendation is:: >> >> _cert_verification_config = '' >> >> This not only makes it straightforward to detect the presence (or >> absence) of >> the capability, it also makes it possible to programmatically determine >> the >> relevant configuration file name. >> >> >> Recommended modifications to the Python standard library >> -------------------------------------------------------- >> >> The recommended approach to backporting the PEP 476 modifications to an >> earlier >> point release is to implement the following changes relative to the >> default >> PEP 476 behaviour implemented in Python 2.7.9+: >> >> * modify the ``ssl`` module to read a system wide configuration file when >> the >> module is first imported into a Python process >> * define a platform default behaviour (either verifying or not verifying >> HTTPS >> certificates) to be used if this configuration file is not present >> * support selection between the following three modes of operation: >> >> * ensure HTTPS certificate verification is enabled >> * ensure HTTPS certificate verification is disabled >> * delegate the decision to the redistributor providing this Python >> version >> >> * set the ``ssl._create_default_https_context`` function to be an alias >> for >> either ``ssl.create_default_context`` or >> ``ssl._create_unverified_context`` >> based on the given configuration setting. >> >> >> Recommended file location >> ------------------------- >> >> This approach is currently only defined for \*nix system Python >> installations. >> >> The recommended configuration file name is >> ``/etc/python/cert-verification.cfg``. >> >> The ``.cfg`` filename extension is recommended for consistency with the >> ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library. >> >> >> Recommended file format >> ----------------------- >> >> The configuration file should use a ConfigParser ini-style format with a >> single section named ``[https]`` containing one required setting >> ``verify``. >> >> Permitted values for ``verify`` are: >> >> * ``enable``: ensure HTTPS certificate verification is enabled by default >> * ``disable``: ensure HTTPS certificate verification is disabled by >> default >> * ``platform_default``: delegate the decision to the redistributor >> providing >> this particular Python version >> >> If the ``[https]`` section or the ``verify`` setting are missing, or if >> the >> ``verify`` setting is set to an unknown value, it should be treated as if >> the >> configuration file is not present. >> >> >> Example implementation >> ---------------------- >> >> :: >> >> _cert_verification_config = '/etc/python/cert-verification.cfg' >> >> def _get_https_context_factory(): >> # Check for a system-wide override of the default behaviour >> context_factories = { >> 'enable': create_default_context, >> 'disable': _create_unverified_context, >> 'platform_default': _create_unverified_context, # For now :) >> } >> import ConfigParser >> config = ConfigParser.RawConfigParser() >> config.read(_cert_verification_config) >> try: >> verify_mode = config.get('https', 'verify') >> except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): >> verify_mode = 'platform_default' >> default_factory = context_factories.get('platform_default') >> return context_factories.get(verify_mode, default_factory) >> >> _create_default_https_context = _get_https_context_factory() >> >> >> Security Considerations >> ----------------------- >> >> The specific recommendations for the backporting case are designed to >> work for >> privileged, security sensitive processes, even those being run in the >> following >> locked down configuration: >> >> * run from a locked down administrator controlled directory rather than a >> normal >> user directory (preventing ``sys.path[0]`` based privilege escalation >> attacks) >> * run using the ``-E`` switch (preventing ``PYTHON*`` environment >> variable based >> privilege escalation attacks) >> * run using the ``-s`` switch (preventing user site directory based >> privilege >> escalation attacks) >> * run using the ``-S`` switch (preventing ``sitecustomize`` based >> privilege >> escalation attacks) >> >> The intent is that the *only* reason HTTPS verification should be getting >> turned off system wide when using this approach is because: >> >> * an end user is running a redistributor provided version of CPython >> rather >> than running upstream CPython directly >> * that redistributor has decided to provide a smoother migration path to >> verifying HTTPS certificates by default than that being provided by the >> upstream project >> * either the redistributor or the local infrastructure administrator has >> determined that it is appropriate to override the default upstream >> behaviour >> (at least for the time being) >> >> Using an administrator controlled configuration file rather than an >> environment >> variable has the essential feature of providing a smoother migration >> path, even >> for applications being run with the ``-E`` switch. >> >> >> Combining the recommendations >> ============================= >> >> If a redistributor chooses to implement both recommendations, then the >> environment variable should take precedence over the system-wide >> configuration >> setting. This allows the setting to be changed for a given user, virtual >> environment or application, regardless of the system-wide default >> behaviour. >> >> In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is >> defined, and >> set to anything *other* than ``'0'``, then HTTPS certificate verification >> should be enabled. >> >> Example implementation >> ---------------------- >> >> :: >> >> _https_verify_envvar = 'PYTHONHTTPSVERIFY' >> _cert_verification_config = '/etc/python/cert-verification.cfg' >> >> def _get_https_context_factory(): >> # Check for am environmental override of the default behaviour >> config_setting = os.environ.get(_https_verify_envvar) >> if config_setting is not None: >> if config_setting == '0': >> return _create_unverified_context >> return create_default_context >> >> # Check for a system-wide override of the default behaviour >> context_factories = { >> 'enable': create_default_context, >> 'disable': _create_unverified_context, >> 'platform_default': _create_unverified_context, # For now :) >> } >> import ConfigParser >> config = ConfigParser.RawConfigParser() >> config.read(_cert_verification_config) >> try: >> verify_mode = config.get('https', 'verify') >> except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): >> verify_mode = 'platform_default' >> default_factory = context_factories.get('platform_default') >> return context_factories.get(verify_mode, default_factory) >> >> _create_default_https_context = _get_https_context_factory() >> >> >> Copyright >> ========= >> >> This document has been placed into the public domain. >> >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Nov 23 18:47:04 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Nov 2015 09:47:04 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <56536A6A.9030902@python.org> References: <56536A6A.9030902@python.org> Message-ID: Updated version of the PEP posted: https://hg.python.org/peps/rev/8decac213ebf On 24 November 2015 at 05:35, Christian Heimes wrote: > 1) The example implementation of the function doesn't check the > sys.flags.ignore_environment. Internally CPython has specialized getenv > function that ignores env vars with PYTHON prefix when the flag is set. > PYTHON* env vars aren't removed from os.environ. Modules have to check > the flag. I guarded the relevant sections in the examples with "if not sys.flags.ignore_environment:" > 2) The PEP is rather Linux-centric. What's the recommended path to the > config file on other platforms like BDS (/usr/local/etc/ is preferred > for additional dependencies on FreeBSD), OSX and Windows? The environment variable section should be generic, and the configuration file section is largely specific to vendors offering long term commercial support options for Linux distros. There was already a qualification in the "Recommended file location" section limiting the scope to *nix systems, so rather than trying to cover non-Linux systems, I've instead updated that qualification to limit the recommendation even further to specifically Linux systems. If *BSD folks pipe up saying they'd like to be included, then adding another filename recommendation wouldn't be difficult. > 3) What's the interaction between the location of the config file and > virtual envs? Would it make sense to search for the file in a venv's > etc/ first and then dispatch to global /etc/? That way venvs can > influence the setting, too. This is a system level configuration setting to preserve backwards compatible behaviour in the system Python, so scoping it at the interpreter level was a deliberate design decision. However, I added a new section to both recommendations describing the lack of interaction with virtual environments. > 4) It makes sense to make the cert-verification.cfg file future-proof > and reserve it for other cert-related configuration in the future. For > example it could be used to define new contexts, set protocols, ciphers > or hashes for cert pinning. It should be enough to say that CPython > reserves the right to add more sections and keys later. For future releases, I think a different filename makes sense, as we don't really want a global "turn off HTTPS certificate verification" flag. Perhaps something like "/etc/python/network-security.cfg", for example. > 5) I'm not particular fond of the section name [https]. For one It is > ambiguous because it doesn't distinguish between server certs and client > certs. I added a note to clarify that the section name comes from the "https" URL schema passed to the relevant client APIs. > It's also not correct. The default context is used for other > protocols like imap, smtp etc. over TLS. That changed in PEP 476 - there's a separate private context for HTTPS standard library clients now, which allows HTTPS to verify hostnames by default, while other clients are still permissive. This is noted in PEP 476: https://www.python.org/dev/peps/pep-0476/#other-protocols The reason for this is that browsers have provided a pretty good forcing function in getting folks to use properly signed certificates, at least on the public internet, but self-signed and improperly signed certificates are still significantly more common for other protocols. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Nov 23 18:56:16 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Nov 2015 09:56:16 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: Message-ID: On 24 November 2015 at 06:47, Wes Turner wrote: > 1. Does this affect easy_install? easy_install has validated certificates since distribute was merged back into the project as part of setuptools 0.7 [1], and aside from one issue with HTTPS tunnelling [2], the certificate verification code has been stable since setuptools 1.3 [3]. > 2. If/because this affects easy_install, > should the guidance / suggested package installation tool be [pip]; > because pip install_requires backports.ssl_match_hostname setuptools/easy_install uses backports.ssl_match_hostname if it's available, and otherwise has its own implementation. Cheers, Nick. [1] https://pythonhosted.org/setuptools/history.html#id159 [2] https://pythonhosted.org/setuptools/history.html#id80 [3] https://pythonhosted.org/setuptools/history.html#id123 -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Nov 23 19:18:50 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Nov 2015 10:18:50 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <56536A6A.9030902@python.org> References: <56536A6A.9030902@python.org> Message-ID: On 24 November 2015 at 05:35, Christian Heimes wrote: > On 2015-11-17 01:00, Guido van Rossum wrote: >> Hm, making Christian the BDFL-delegate would mean two out of three >> authors *and* the BDFL-delegate all working for Red Hat, which clearly >> has a stake (and IIUC has already committed to this approach ahead of >> PEP approval). SO then it would look like this is just rubber-stamping >> Red Hat's internal decision process (if it's a process -- sounds more >> like an accident :-). >> >> So, Alex, do you want to approve this PEP? > > I haven't read this thread until now. Independently from your objection > I have raised the same concern with Nick today. I'd be willing to BDFL > the PEP but I'd rather have somebody outside of Red Hat. Likewise, but the intersection between "wants to get PEP 476 into the hands of as many system operators as possible as soon as possible", "is a CPython core developer", and "doesn't work for Red Hat" is looking to be a rather select group :) Since we already know Red Hat are OK with the draft recommendations, and I missed the RHEL 7.2 release date anyway, perhaps Barry or Matthias might be interested in tilting at the Ubuntu 14.04 LTS stable release update windmill? I know there was previously a decision from Ubuntu Security not to backport PEPs 466 & 476 to 2.7.5 due to the stability risks [1], but the configuration file based approach recommended in PEP 493 is backwards compatible by default, with the decision to opt in to the improved settings after upgrading current systems being made by system administrators rather than the distro vendor. With around 3 1/2 years still to run on 14.04's support lifecycle, that has the potential to reach quite a few systems that otherwise wouldn't benefit from the change until well after Ubuntu 16.04 is released next year. Regards, Nick. [1] http://people.canonical.com/~ubuntu-security/cve/2014/CVE-2014-9365.html -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From wes.turner at gmail.com Mon Nov 23 20:17:41 2015 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 23 Nov 2015 19:17:41 -0600 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: Message-ID: On Mon, Nov 23, 2015 at 5:56 PM, Nick Coghlan wrote: > On 24 November 2015 at 06:47, Wes Turner wrote: > > 1. Does this affect easy_install? > > easy_install has validated certificates since distribute was merged > back into the project as part of setuptools 0.7 [1], and aside from > one issue with HTTPS tunnelling [2], the certificate verification code > has been stable since setuptools 1.3 [3]. > Got it; thanks. * https://bitbucket.org/pypa/setuptools/src/tip/setuptools/ssl_support.py > > > 2. If/because this affects easy_install, > > should the guidance / suggested package installation tool be [pip]; > > because pip install_requires backports.ssl_match_hostname > > setuptools/easy_install uses backports.ssl_match_hostname if it's > available, and otherwise has its own implementation. > * setuptools 0.7 [src] -- 2013-06-?? -- https://bitbucket.org/pypa/setuptools/src/7f2c08e9/?at=0.7 * setuptools 0.7 [distro] -- 201X-XX-XX [see: whohas, ] Setuptools latest is now at version 18.5 (2015-11-02). > > Cheers, > Nick. > > [1] https://pythonhosted.org/setuptools/history.html#id159 > [2] https://pythonhosted.org/setuptools/history.html#id80 > [3] https://pythonhosted.org/setuptools/history.html#id123 > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Nov 23 20:59:06 2015 From: barry at python.org (Barry Warsaw) Date: Mon, 23 Nov 2015 20:59:06 -0500 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <56536A6A.9030902@python.org> Message-ID: <20151123205906.02cd57ec@limelight.wooz.org> On Nov 24, 2015, at 10:18 AM, Nick Coghlan wrote: >Since we already know Red Hat are OK with the draft recommendations, >and I missed the RHEL 7.2 release date anyway, perhaps Barry or >Matthias might be interested in tilting at the Ubuntu 14.04 LTS stable >release update windmill? I know there was previously a decision from >Ubuntu Security not to backport PEPs 466 & 476 to 2.7.5 due to the >stability risks [1], but the configuration file based approach >recommended in PEP 493 is backwards compatible by default Right, but this isn't a patch we'd particularly want to carry ourselves. Maybe if it were available upstream, tried and tested, it could be considered for backporting, but it still wouldn't be zero cost. We'd have to also handle migration paths to newer Ubuntu releases, which probably means removing the config file on future upgrades. There's also the possibility of implementing different defaults on new installs of 14.04 versus upgrades to 14.04. And even if a system administrator enabled it for one particular application, it could break other applications on the same machine, so it just punts a difficult decision down the line. We're also not seeing much (any?) demand from our users, and the initial attempt at turning this on by default *did* get a strong negative reaction because of the compatibility break. I'm concerned about accepting PEP 493 making a strong recommendation to downstreams. Yes, in an ideal world we all want security by default, but I think the backward compatibility concerns of the PEP are understated, especially as they relate to a maintenance release of a stable long term support version of the OS. I don't want PEP 493 to be a cudgel that people beat us up with instead of having an honest discussion of the difficult trade-offs involved. Having said all that, I think informing people of the issue, and letting any future reconsideration be demand driven is the right approach for now. $0.02-ly y'rs, -Barry From barry at python.org Mon Nov 23 21:05:57 2015 From: barry at python.org (Barry Warsaw) Date: Mon, 23 Nov 2015 21:05:57 -0500 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: Message-ID: <20151123210557.57b7e732@limelight.wooz.org> On Nov 17, 2015, at 11:44 PM, Nick Coghlan wrote: >For Debian, Ubuntu and SUSE, their original determinations for the >relevant CVE were "too intrusive to backport", so folks currently need >to upgrade to newer versions of those distros to get the improved >default behaviour: This is an example of my problem with the tone of PEP 493 (sorry Nick, nothing personal!). "Improved default behavior"... for whom? It's not improved for the folks whose applications are broken by changing the default. Cheers, -Barry From ncoghlan at gmail.com Mon Nov 23 22:46:55 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Nov 2015 13:46:55 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <20151123210557.57b7e732@limelight.wooz.org> References: <20151123210557.57b7e732@limelight.wooz.org> Message-ID: On 24 November 2015 at 12:05, Barry Warsaw wrote: > On Nov 17, 2015, at 11:44 PM, Nick Coghlan wrote: > >>For Debian, Ubuntu and SUSE, their original determinations for the >>relevant CVE were "too intrusive to backport", so folks currently need >>to upgrade to newer versions of those distros to get the improved >>default behaviour: > > This is an example of my problem with the tone of PEP 493 (sorry Nick, nothing > personal!). "Improved default behavior"... for whom? It's not improved for > the folks whose applications are broken by changing the default. I think there are three relevant categories here: 1. Folks who assume that "https" means the same thing in Python that it means in web browsers, and are currently experiencing a silent security failure 2. Folks who already know it doesn't, and are relying on that to keep their infrastructure working 3. Folks currently in group 2 who would like to improve their infrastructure to default to verifying certificates The upstream change in PEP 476 was driven by the interests of folks in group 1. This makes a lot of sense, given that a popular way of introducing folks to programming now is by writing programs that interact with web APIs over HTTPS. Rebasing is fine for reaching desktop audiences, so this group should already have been covered by the updated python.org binaries for Windows and Mac OS X, new binary releases from the cross-platform redistributors, and updates to non-LTS Linux distros. For folks in group 2, the main goal is "Don't break their stuff as a side effect of updating a supported version", which is why PEP 493 recommends leaving certificate verification disabled by default if backporting the PEP 476 changes. The target audience for PEP 493 is then the folks in group 3: infrastructure operators that *want* to turn failures to validate certificates in Python applications into a noisy failure, but *don't* want to have to switch to a new major version of their entire base operating system to do it. With the PEP, opting in to PEP 476 for Python 2 applications and services is just a configuration file setting, readily managed with any configuration management tool. If just the configuration file approach is supported (as is the case for RHEL 7.2), then that decision needs to be made on a system-wide basis, and you need to use a chroot (or a full Linux container) in order to opt out (or in) for a single application. If the environment variable downgrade is also supported, then the environments for individual applications can be tweaked, even while the overall environment is upgraded to complain about missing security checks by default. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Nov 23 23:12:56 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Nov 2015 14:12:56 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <20151123205906.02cd57ec@limelight.wooz.org> References: <56536A6A.9030902@python.org> <20151123205906.02cd57ec@limelight.wooz.org> Message-ID: On 24 November 2015 at 11:59, Barry Warsaw wrote: > On Nov 24, 2015, at 10:18 AM, Nick Coghlan wrote: > >>Since we already know Red Hat are OK with the draft recommendations, >>and I missed the RHEL 7.2 release date anyway, perhaps Barry or >>Matthias might be interested in tilting at the Ubuntu 14.04 LTS stable >>release update windmill? I know there was previously a decision from >>Ubuntu Security not to backport PEPs 466 & 476 to 2.7.5 due to the >>stability risks [1], but the configuration file based approach >>recommended in PEP 493 is backwards compatible by default > > Right, but this isn't a patch we'd particularly want to carry ourselves. > Maybe if it were available upstream, tried and tested, it could be considered > for backporting, but it still wouldn't be zero cost. If anyone wants to look through the patches implementing the backport, they're available through the CentOS git repositories: https://git.centos.org/commit/rpms!python.git/refs!heads!c7 > We'd have to also handle > migration paths to newer Ubuntu releases, which probably means removing the > config file on future upgrades. You'll need to figure out those migration paths anyway, as the certificate verification compatibility break will exist between 14.04 and 16.04 regardless of what happens with PEP 493. > There's also the possibility of implementing > different defaults on new installs of 14.04 versus upgrades to 14.04. For those aspects, we left the defaults in RHEL 7.2 as implementing the 2.7.5 behaviour (i.e. no cert verification), but used the "platform default" mechanism in PEP 493 to ensure we had the ability to change our mind if circumstances change. For the immediately foreseeable future though, it will be up to operations teams to decide how they want the Python 2.7 standard library to behave with respect to HTTPS client certificate verification on RHEL & CentOS 7. > And > even if a system administrator enabled it for one particular application, it > could break other applications on the same machine, so it just punts a > difficult decision down the line. > > We're also not seeing much (any?) demand from our users, and the initial > attempt at turning this on by default *did* get a strong negative reaction > because of the compatibility break. My own main concern is providing an additional layer of defence against certificate verification related security bugs in network services and clients written in Python like OpenStack, Ansible, FreeIPA, Pulp, etc, which is why I'm happy with the approach of giving system operators the tools they need to make their own decision about whether or not they want that additional layer of protection on a given host. Folks building Docker images can also easily opt in or out by including a suitable configuration file. >From a change management perspective, there's also a very big difference between "We made this decision for you, and you'll need to update your systems accordingly in order to upgrade", and "We made it possible for you to make your own decision about this new configuration setting, based on what you think is best for your particular situation". There's still some risk in the latter case (since you may accidentally break things), but the upstream releases flushed out a lot of those problems for this particular update (e.g. the gevent compatibility issue that affected the upstream PEP 466 backport). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From raymond.hettinger at gmail.com Tue Nov 24 01:01:10 2015 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 23 Nov 2015 22:01:10 -0800 Subject: [Python-Dev] collections.Counter __add__ implementation quirk In-Reply-To: References: <5652B077.9000606@sdamon.com> Message-ID: > On Nov 23, 2015, at 10:43 AM, Vlastimil Brom wrote: > >> Is there any particular reason counters drop negative values when you add >> them together? I definitely expected them to act like ints do when you add >> negatives, and had to subclass it to get what I think is the obvious >> behavior. >> _______________________________________________ >> Python-Dev mailing list > ... > Hi, > this is probably more appropriate for the general python list rathere > then this developers' maillist, however, as I asked a similar question > some time ago, I got some detailed explanations for the the current > design decissions from the original developer; cf.: > https://mail.python.org/pipermail/python-list/2010-March/570618.html > > (I didn't check possible changes in Counter since that version (3.1 at > that time).) In Python3.2, Counter grew a subtract() method: >>> c = Counter(a=4, b=2, c=0, d=-2) >>> d = Counter(a=1, b=2, c=3, d=4) >>> c.subtract(d) >>> c Counter({'a': 3, 'b': 0, 'c': -3, 'd': -6}) The update() method has been around since the beginning: >>> from collections import Counter >>> c = Counter(a=4, b=2, c=0, d=-2) >>> d = Counter(a=1, b=-5, c=-2, d=6) >>> c.update(d) >>> d Counter({'d': 6, 'a': 1, 'c': -2, 'b': -5}) So, you have two ways of doing counter math: 1. Normal integer arithmetic using update() and subtract() does straight addition and subtraction, either starting with or ending-up with negative values. 2. Saturating arithmetic using the operators: + - & | excludes non-positive results. This supports bag-like behavior (c.f. smalltalk) and multiset operations (https://en.wikipedia.org/wiki/Multiset). Raymond From p.f.moore at gmail.com Tue Nov 24 05:12:10 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Nov 2015 10:12:10 +0000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <20151123210557.57b7e732@limelight.wooz.org> Message-ID: On 24 November 2015 at 03:46, Nick Coghlan wrote: > I think there are three relevant categories here: > > 1. Folks who assume that "https" means the same thing in Python that > it means in web browsers, and are currently experiencing a silent > security failure > 2. Folks who already know it doesn't, and are relying on that to keep > their infrastructure working > 3. Folks currently in group 2 who would like to improve their > infrastructure to default to verifying certificates I'm not directly interested in this PEP (any code I write that is relevant will run on Windows) but there's another group: 4. People who use https because it's the published API, but are only looking at internal systems (where the threat window is minimised) and the internal infrastructure is out of their control but insecure. Browser users and other apps have to go through pain to work (accepting self-signed certs, repeatedly responding "Yes" to security warnings). Python code written for internal use just works at the moment. Maybe it "shouldn't", in some abstract sense, but it *does* and that is important to users. This group may know that they are taking risks, and may want to improve the infrastructure, but they *can't*. And this change breaks their apps (possibly in ways they can't fix easily - not all client environments offer ways to add trust, there's a reason tools like wget have "ignore certificate checks" flags even though everyone knows they are a bad idea). The push for "secure by default" is important and needed, but needs to be handled in a more pragmatic way, recognising that Python is used in situations like the above, where people know they aren't doing the right thing, but have no better options. Shutting down their least bad option on a point of principle isn't helpful. Paul From christian at python.org Tue Nov 24 07:53:48 2015 From: christian at python.org (Christian Heimes) Date: Tue, 24 Nov 2015 13:53:48 +0100 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <56536A6A.9030902@python.org> Message-ID: <56545DDC.7000407@python.org> On 2015-11-24 01:18, Nick Coghlan wrote: > On 24 November 2015 at 05:35, Christian Heimes wrote: >> On 2015-11-17 01:00, Guido van Rossum wrote: >>> Hm, making Christian the BDFL-delegate would mean two out of three >>> authors *and* the BDFL-delegate all working for Red Hat, which clearly >>> has a stake (and IIUC has already committed to this approach ahead of >>> PEP approval). SO then it would look like this is just rubber-stamping >>> Red Hat's internal decision process (if it's a process -- sounds more >>> like an accident :-). >>> >>> So, Alex, do you want to approve this PEP? >> >> I haven't read this thread until now. Independently from your objection >> I have raised the same concern with Nick today. I'd be willing to BDFL >> the PEP but I'd rather have somebody outside of Red Hat. > > Likewise, but the intersection between "wants to get PEP 476 into the > hands of as many system operators as possible as soon as possible", > "is a CPython core developer", and "doesn't work for Red Hat" is > looking to be a rather select group :) Right, with Antoine and Alex out of scope and you, Victor and me working for Red Hat, the air is getting thin. Benjamin is familiar with the ssl module. Or we can follow Alex's advice and ask somebody from the PyCA group (Donald, Paul, lvh) or requests (Cory) to get some outside perspective. > Since we already know Red Hat are OK with the draft recommendations, > and I missed the RHEL 7.2 release date anyway, perhaps Barry or > Matthias might be interested in tilting at the Ubuntu 14.04 LTS stable > release update windmill? I know there was previously a decision from > Ubuntu Security not to backport PEPs 466 & 476 to 2.7.5 due to the > stability risks [1], but the configuration file based approach > recommended in PEP 493 is backwards compatible by default, with the > decision to opt in to the improved settings after upgrading current > systems being made by system administrators rather than the distro > vendor. With around 3 1/2 years still to run on 14.04's support > lifecycle, that has the potential to reach quite a few systems that > otherwise wouldn't benefit from the change until well after Ubuntu > 16.04 is released next year. Yes, that makes a lot of sense. Christian From donald at stufft.io Tue Nov 24 07:59:37 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 24 Nov 2015 07:59:37 -0500 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <56545DDC.7000407@python.org> References: <56536A6A.9030902@python.org> <56545DDC.7000407@python.org> Message-ID: <905691F9-40EB-4745-9DA0-BDA769FB9541@stufft.io> > On Nov 24, 2015, at 7:53 AM, Christian Heimes wrote: > > On 2015-11-24 01:18, Nick Coghlan wrote: >> On 24 November 2015 at 05:35, Christian Heimes wrote: >>> On 2015-11-17 01:00, Guido van Rossum wrote: >>>> Hm, making Christian the BDFL-delegate would mean two out of three >>>> authors *and* the BDFL-delegate all working for Red Hat, which clearly >>>> has a stake (and IIUC has already committed to this approach ahead of >>>> PEP approval). SO then it would look like this is just rubber-stamping >>>> Red Hat's internal decision process (if it's a process -- sounds more >>>> like an accident :-). >>>> >>>> So, Alex, do you want to approve this PEP? >>> >>> I haven't read this thread until now. Independently from your objection >>> I have raised the same concern with Nick today. I'd be willing to BDFL >>> the PEP but I'd rather have somebody outside of Red Hat. >> >> Likewise, but the intersection between "wants to get PEP 476 into the >> hands of as many system operators as possible as soon as possible", >> "is a CPython core developer", and "doesn't work for Red Hat" is >> looking to be a rather select group :) > > Right, with Antoine and Alex out of scope and you, Victor and me working > for Red Hat, the air is getting thin. Benjamin is familiar with the ssl > module. Or we can follow Alex's advice and ask somebody from the PyCA > group (Donald, Paul, lvh) or requests (Cory) to get some outside > perspective. > Under normal circumstances I'd probably be willing to do it even though I have doubts about the value of it. However, I'm less than two weeks away from closing on a house and will be moving into it after that, so my time is very limited right now. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Tue Nov 24 08:20:16 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Nov 2015 23:20:16 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <20151123210557.57b7e732@limelight.wooz.org> Message-ID: On 24 Nov 2015 8:12 pm, "Paul Moore" wrote: > > On 24 November 2015 at 03:46, Nick Coghlan wrote: > > I think there are three relevant categories here: > > > > 1. Folks who assume that "https" means the same thing in Python that > > it means in web browsers, and are currently experiencing a silent > > security failure > > 2. Folks who already know it doesn't, and are relying on that to keep > > their infrastructure working > > 3. Folks currently in group 2 who would like to improve their > > infrastructure to default to verifying certificates > > I'm not directly interested in this PEP (any code I write that is > relevant will run on Windows) but there's another group: > > 4. People who use https because it's the published API, but are only > looking at internal systems (where the threat window is minimised) and > the internal infrastructure is out of their control but insecure. > Browser users and other apps have to go through pain to work > (accepting self-signed certs, repeatedly responding "Yes" to security > warnings). Python code written for internal use just works at the > moment. Maybe it "shouldn't", in some abstract sense, but it *does* > and that is important to users. > > This group may know that they are taking risks, and may want to > improve the infrastructure, but they *can't*. And this change breaks > their apps (possibly in ways they can't fix easily - not all client > environments offer ways to add trust, there's a reason tools like wget > have "ignore certificate checks" flags even though everyone knows they > are a bad idea). I believe you're referring mainly to the original PEP 476 change there. In the context of PEP 493, this is another group that would potentially benefit from the suggested "security downgrade" environment variable (if any redistributors decide to implement that - RHEL doesn't as yet), since it would provide a way to restore the old behaviour without changing their client code or monkeypatching the SSL module as described in PEP 476. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Nov 24 09:05:53 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Nov 2015 14:05:53 +0000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <20151123210557.57b7e732@limelight.wooz.org> Message-ID: On 24 November 2015 at 13:20, Nick Coghlan wrote: > I believe you're referring mainly to the original PEP 476 change there. In > the context of PEP 493, this is another group that would potentially benefit > from the suggested "security downgrade" environment variable (if any > redistributors decide to implement that - RHEL doesn't as yet), since it > would provide a way to restore the old behaviour without changing their > client code or monkeypatching the SSL module as described in PEP 476. I'm actually referring to the fact that your classification didn't seem to include people who have no control over their infrastructure (except in class 1 which implies ignorance rather than powerlessness...). PEP 493 is of benefit to such people, so there's now downside in explicitly noting this. My concern is that *because* people consistently forget about the class of people who have to put up with bad infrastructure but can't do anything about it, we risk promoting a sense of "security as the enemy" - which is the direct opposite of what we're trying to do. I have no interest or opinion regarding this PEP itself, but I would like to see "people who have to put up with whatever infrastructure they are dumped with, and use Python to ease that burden" recognised as an important class of user. They are very under-represented in discussions, as it's usually big business closed source and similar environments that are in that situation. Simply adding "people who have no control over their broken infrastructure" with a note that this PEP helps them, would be sufficient here (and actually helps the case for the PEP, so why not? ;-)) Apologies, this is a bit of a hobby horse of mine, I'll pipe down now. Paul. From lac at openend.se Tue Nov 24 09:27:15 2015 From: lac at openend.se (Laura Creighton) Date: Tue, 24 Nov 2015 15:27:15 +0100 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <20151123210557.57b7e732@limelight.wooz.org> Message-ID: <201511241427.tAOERFJC028474@fido.openend.se> In a message of Tue, 24 Nov 2015 14:05:53 +0000, Paul Moore writes: >Simply adding "people who have no control over their broken >infrastructure" with a note that this PEP helps them, would be >sufficient here (and actually helps the case for the PEP, so why not? >;-)) But does it help them? Or does it increase the power of those who hand out certificates and who are intensely security conscious over those who would like to get some work done this afternoon? Laura From ncoghlan at gmail.com Tue Nov 24 09:59:30 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Nov 2015 00:59:30 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <201511241427.tAOERFJC028474@fido.openend.se> References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> Message-ID: On 25 November 2015 at 00:27, Laura Creighton wrote: > In a message of Tue, 24 Nov 2015 14:05:53 +0000, Paul Moore writes: >>Simply adding "people who have no control over their broken >>infrastructure" with a note that this PEP helps them, would be >>sufficient here (and actually helps the case for the PEP, so why not? >>;-)) > > But does it help them? Or does it increase the power of those who > hand out certificates and who are intensely security conscious over > those who would like to get some work done this afternoon? In situations where IT are still the "Department of No", rather than focusing on facilitating folks in getting their work done, I think the most likely outcome of the configuration file recommendation in PEP 493 is preservation of the status quo: admins simply won't change the config setting, even if they deploy a version of Linux that adopts the approach suggested in the PEP. If they do enable full certificate verification (or upgrade the environments they provide to a version of Python that has it enabled by default) without doing appropriate compatibility testing first, then they're going to hit the compatibility problems Paul is talking about. The aspect of the PEP that has the potential to help in the case of poor infrastructure management is providing the ability to globally turn off certificate verification on a per-process basis. It's the networking equivalent of monkeypatching - you know there are risks with doing it, but also judge the near term benefits to outweigh those longer term risks in your particular situation. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mal at egenix.com Tue Nov 24 10:04:08 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 24 Nov 2015 16:04:08 +0100 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <201511241427.tAOERFJC028474@fido.openend.se> References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> Message-ID: <56547C68.8060506@egenix.com> I think the PEP is a good step forward to compromise between the crypto purists (use whatever technologies makes us more secure even if it breaks things) and those who cannot upgrade their Python 2.7 because of the PEP 476 change, since it causes their applications to fail (e.g. because the embedded devices they want to interface to only support self signed certs). I would still find having built-in support for the recommendations in the Python stdlib a better approach, but PEP 493 is good enough in at least solving real problems people are having. PS: Would be great to have a PyPI package which implements these recommendations so that you can simply add it as dependency - and then please for Python 3 as well, since people with embedded devices will want to be able to use Python 3 as well ;-) On 24.11.2015 15:27, Laura Creighton wrote: > In a message of Tue, 24 Nov 2015 14:05:53 +0000, Paul Moore writes: >> Simply adding "people who have no control over their broken >> infrastructure" with a note that this PEP helps them, would be >> sufficient here (and actually helps the case for the PEP, so why not? >> ;-)) > > But does it help them? Or does it increase the power of those who > hand out certificates and who are intensely security conscious over > those who would like to get some work done this afternoon? > > Laura > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 24 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From a.badger at gmail.com Tue Nov 24 10:19:22 2015 From: a.badger at gmail.com (Toshio Kuratomi) Date: Tue, 24 Nov 2015 07:19:22 -0800 Subject: [Python-Dev] Fwd: Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> Message-ID: On Nov 24, 2015 6:28 AM, "Laura Creighton" wrote: > > In a message of Tue, 24 Nov 2015 14:05:53 +0000, Paul Moore writes: > >Simply adding "people who have no control over their broken > >infrastructure" with a note that this PEP helps them, would be > >sufficient here (and actually helps the case for the PEP, so why not? > >;-)) > > But does it help them? Or does it increase the power of those who > hand out certificates and who are intensely security conscious over > those who would like to get some work done this afternoon? > My reading is that it will help more people but lockdown environments can still trump their users if they wish. If a distribution wishes to give users of older python versions the option of verifying certificates then they will need to backport changes authorized by previous peps. By themselves, those changes would make it so environment owners and application authors are in complete control. If an application is coded to do cert verification and the remote end has certificates that aren't recognized as valid on the client end then the user would have to change the client application code to be able to use it in their environment (or figure out how to get the ca for the remote end into their local certificate store... in extreme cases, this might be impossible - the ca cert has been lost or belongs to another company). This pep tells distributions how they might give the client end a bit more power when they backport. The settings file allows the client to toggle verification site wide. The environment variable allows clients to toggle it per application invocation. Both of these situations are better for a client than having the backport and nothing else. Both of these can be shut down by an environment owner with sufficient authority to limit what's running on the client (not sure the scope of the environment owner's powers here so I thought I should acknowledge this factor). So basically: backporting other peps (to increase security) will subtract power from the clients. This pep specifies several facilities the backporters can implement to give some of that power back to the clients. -Toshio > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Tue Nov 24 10:30:07 2015 From: christian at python.org (Christian Heimes) Date: Tue, 24 Nov 2015 16:30:07 +0100 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <56536A6A.9030902@python.org> Message-ID: <5654827F.9010900@python.org> On 2015-11-24 00:47, Nick Coghlan wrote: > Updated version of the PEP posted: https://hg.python.org/peps/rev/8decac213ebf > > On 24 November 2015 at 05:35, Christian Heimes wrote: >> 1) The example implementation of the function doesn't check the >> sys.flags.ignore_environment. Internally CPython has specialized getenv >> function that ignores env vars with PYTHON prefix when the flag is set. >> PYTHON* env vars aren't removed from os.environ. Modules have to check >> the flag. > > I guarded the relevant sections in the examples with "if not > sys.flags.ignore_environment:" Thanks! >> 2) The PEP is rather Linux-centric. What's the recommended path to the >> config file on other platforms like BDS (/usr/local/etc/ is preferred >> for additional dependencies on FreeBSD), OSX and Windows? > > The environment variable section should be generic, and the > configuration file section is largely specific to vendors offering > long term commercial support options for Linux distros. > > There was already a qualification in the "Recommended file location" > section limiting the scope to *nix systems, so rather than trying to > cover non-Linux systems, I've instead updated that qualification to > limit the recommendation even further to specifically Linux systems. > If *BSD folks pipe up saying they'd like to be included, then adding > another filename recommendation wouldn't be difficult. >> 3) What's the interaction between the location of the config file and >> virtual envs? Would it make sense to search for the file in a venv's >> etc/ first and then dispatch to global /etc/? That way venvs can >> influence the setting, too. > > This is a system level configuration setting to preserve backwards > compatible behaviour in the system Python, so scoping it at the > interpreter level was a deliberate design decision. However, I added a > new section to both recommendations describing the lack of interaction > with virtual environments. I mentioned 2) and 3) because I suspect that some people will want to use the new setting to configure applications. You know how people think. They get a new shiny tool and then they abuse it as a hammer to drive in all their problems. :) >> 4) It makes sense to make the cert-verification.cfg file future-proof >> and reserve it for other cert-related configuration in the future. For >> example it could be used to define new contexts, set protocols, ciphers >> or hashes for cert pinning. It should be enough to say that CPython >> reserves the right to add more sections and keys later. > > For future releases, I think a different filename makes sense, as we > don't really want a global "turn off HTTPS certificate verification" > flag. Perhaps something like "/etc/python/network-security.cfg", for > example. Are you planning to remove the "disable verification flag" in the future? >> 5) I'm not particular fond of the section name [https]. For one It is >> ambiguous because it doesn't distinguish between server certs and client >> certs. > > I added a note to clarify that the section name comes from the "https" > URL schema passed to the relevant client APIs. > >> It's also not correct. The default context is used for other >> protocols like imap, smtp etc. over TLS. > > That changed in PEP 476 - there's a separate private context for HTTPS > standard library clients now, which allows HTTPS to verify hostnames > by default, while other clients are still permissive. This is noted in > PEP 476: https://www.python.org/dev/peps/pep-0476/#other-protocols > > The reason for this is that browsers have provided a pretty good > forcing function in getting folks to use properly signed certificates, > at least on the public internet, but self-signed and improperly signed > certificates are still significantly more common for other protocols. I'm sorry, I forgot about PEP 476 and the separate context function for HTTPS. Somehow I expected the setting to influence all TLS/SSL connection, not just HTTPS. I should have read the topic line. In the light of the new information the setting name makes sense. Christian From p.f.moore at gmail.com Tue Nov 24 10:34:53 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Nov 2015 15:34:53 +0000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <201511241427.tAOERFJC028474@fido.openend.se> References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> Message-ID: On 24 November 2015 at 14:27, Laura Creighton wrote: > In a message of Tue, 24 Nov 2015 14:05:53 +0000, Paul Moore writes: >>Simply adding "people who have no control over their broken >>infrastructure" with a note that this PEP helps them, would be >>sufficient here (and actually helps the case for the PEP, so why not? >>;-)) > > But does it help them? Or does it increase the power of those who > hand out certificates and who are intensely security conscious over > those who would like to get some work done this afternoon? My reading is that if fully implemented (and Nick has already confirmed that Red Hat didn't do this) it would add an environment variable that would allow the user to (in effect) say "I can't fix my security infrastructure, so just leave me alone and let me take the risk". So in theory this PEP would give back some of the ability to ignore the problem that previous PEPs took away. (And by "ignore the problem", here I mean "just try to get some work done in spite of a security and infrastructure group that don't understand how to implement security and infrastructure, dammit!") Like it or not, in many organisations, security and development are a huge "us and them" battle. For me, it's important that Python doesn't take sides in that battle, while still offering education to anyone willing to listen. (All I've learned about security is as a result of working with Python - sadly, that knowledge has not made my job one iota easier, it's just increased my stress levels :-() Paul From mike at selik.org Tue Nov 24 03:19:36 2015 From: mike at selik.org (Michael Selik) Date: Tue, 24 Nov 2015 08:19:36 +0000 Subject: [Python-Dev] collections.Counter __add__ implementation quirk In-Reply-To: References: <5652B077.9000606@sdamon.com> Message-ID: Raymond, I think you made a typographical error in your Counter.update example. >>> from collections import Counter >>> c = Counter(a=4, b=2, c=0, d=-2) >>> d = Counter(a=1, b=-5, c=-2, d=6) >>> c.update(d) >>> c Counter({'a': 5, 'd': 4, 'c': -2, 'b': -3}) Pair programming ;-) On Tue, Nov 24, 2015 at 1:02 AM Raymond Hettinger < raymond.hettinger at gmail.com> wrote: > > > On Nov 23, 2015, at 10:43 AM, Vlastimil Brom > wrote: > > > >> Is there any particular reason counters drop negative values when you > add > >> them together? I definitely expected them to act like ints do when you > add > >> negatives, and had to subclass it to get what I think is the obvious > >> behavior. > >> _______________________________________________ > >> Python-Dev mailing list > > ... > > Hi, > > this is probably more appropriate for the general python list rathere > > then this developers' maillist, however, as I asked a similar question > > some time ago, I got some detailed explanations for the the current > > design decissions from the original developer; cf.: > > https://mail.python.org/pipermail/python-list/2010-March/570618.html > > > > (I didn't check possible changes in Counter since that version (3.1 at > > that time).) > > In Python3.2, Counter grew a subtract() method: > > >>> c = Counter(a=4, b=2, c=0, d=-2) > >>> d = Counter(a=1, b=2, c=3, d=4) > >>> c.subtract(d) > >>> c > Counter({'a': 3, 'b': 0, 'c': -3, 'd': -6}) > > The update() method has been around since the beginning: > > >>> from collections import Counter > >>> c = Counter(a=4, b=2, c=0, d=-2) > >>> d = Counter(a=1, b=-5, c=-2, d=6) > >>> c.update(d) > >>> d > Counter({'d': 6, 'a': 1, 'c': -2, 'b': -5}) > > > So, you have two ways of doing counter math: > > 1. Normal integer arithmetic using update() and subtract() does straight > addition and subtraction, either starting with or ending-up with negative > values. > > 2. Saturating arithmetic using the operators: + - & | excludes > non-positive results. This supports bag-like behavior (c.f. smalltalk) and > multiset operations (https://en.wikipedia.org/wiki/Multiset). > > > Raymond > > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mike%40selik.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.badger at gmail.com Tue Nov 24 12:16:30 2015 From: a.badger at gmail.com (Toshio Kuratomi) Date: Tue, 24 Nov 2015 09:16:30 -0800 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <20151123205906.02cd57ec@limelight.wooz.org> References: <56536A6A.9030902@python.org> <20151123205906.02cd57ec@limelight.wooz.org> Message-ID: On Mon, Nov 23, 2015 at 5:59 PM, Barry Warsaw wrote: > I'm concerned about accepting PEP 493 making a strong recommendation to > downstreams. Yes, in an ideal world we all want security by default, but I > think the backward compatibility concerns of the PEP are understated, > especially as they relate to a maintenance release of a stable long term > support version of the OS. I don't want PEP 493 to be a cudgel that people > beat us up with instead of having an honest discussion of the difficult > trade-offs involved. > It sounds like the implementation sections of the PEP are acceptable but that the PEP's general tone seems to assume that distributors are champing at the bit to backport and that the recommendations here make it so that backporting is a no-brainer -- which does not seem to reflect the real-world? I think the tone could be changed to address that as it doesn't seem like forcing distros to backport is a real goal of the PEP. The main purposes of the PEP seem to be: * Enumerate several ways that distributors can backport these 2.7.9 features to older releases * Allow programmers to detect the presence of the features from their code * Give end-users the ability to choose between backwards compatibility and enhanced security Here's some ideas for changing the tone: Abstract ======== PEP 476 updated Python's default handling of HTTPS certificates to be appropriate for communication over the public internet. The Python 2.7 long term maintenance series was judged to be in scope for this change, with the new behaviour introduced in the Python 2.7.9 maintenance release. + Change to "PEP 476 updated Python's default handling of HTTPS certificates to validate that the certs belonged to the server". This way we're saying what the change is rather than making a value judgement of whether people who don't choose to backport are "appropriate" or not. Appropriate-ness is probably best left as an argument in the text of PEP 476. This PEP provides recommendations to downstream redistributors wishing to provide a smoother migration experience when helping their users to manage this change in Python's default behaviour. + Change to "downstream redistributors wishing to backport the enhancements in a way that allows users to choose between backwards compatible behaviour or more secure certificate handling." As barry noted, this PEP doesn't change the amount of work needed to migrate. It does, however, give users some choice in when they are going to perform that work. Additionally, this isn't simply about distributors who want to make the transition smoother... (there's no downstreams that want to make it "more painful" are there? ;-) It's really about making backporting of the enhancements less painful for users. Rationale ========= PEP 476 changed Python's default behaviour to better match the needs and expectations of developers operating over the public internet, a category which appears to include most new Python developers. It is the position of the authors of this PEP that this was a correct decision. However, it is also the case that this change *does* cause problems for infrastructure administrators operating private intranets that rely on self-signed certificates, or otherwise encounter problems with the new default certificate verification settings. + per barry's mesage, it would be good to either devote a paragraph to the backwards compatibility implications here or link to https://www.python.org/dev/peps/pep-0476/#backwards-compatibility The long term answer for such environments is to update their internal certificate management to at least match the standards set by the public internet, but in the meantime, it is desirable to offer these administrators a way to continue receiving maintenance updates to the Python 2.7 series, without having to gate that on upgrades to their certificate management infrastructure. + The wording here seems to reflect a different scope than merely backporting by distros. Perhaps we should change it to: "[...]set by the public internet. Distributions may wish to help these sites transition by backporting the PEP 476 changes to earlier versions of python in a way that does not require the administrators to upgrade their certificate management infrastructure immediately. This would allow sites to choose to use the distribution suppiied python in a backwards compatible fashion until their certificate management infrastructure was updated and then toggle their site to utilize the more secure features provided by PEP 476." [...] These designs are being proposed as a recommendation for redistributors, rather than as new upstream features, as they are needed purely to support legacy environments migrating from older versions of Python 2.7. Neither approach is being proposed as an upstream Python 2.7 feature, nor as a feature in any version of Python 3 (whether published directly by the Python Software Foundation or by a redistributor). + This paragraph seems a little out of place here as it is not about why this PEP is useful so much as who will implement it. It also seems to tie into Abstract closely as it reinforces and expands on where this is going to make a difference. Perhaps move it to Abstract or create a "Scope" section that can follow abstract and contain this information. wording can be changed a bit to show that it isn't expected that all redistributors are expected to follow this but only those who are already backporting: "These recommendations are being proposed to redistributors who wish to backport PEP 476 features rather than as a new upstream feature as they are needed purely to support migrating legacy environments from older versions of python 2.7" -Toshio From p.f.moore at gmail.com Tue Nov 24 13:08:58 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Nov 2015 18:08:58 +0000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <56536A6A.9030902@python.org> <20151123205906.02cd57ec@limelight.wooz.org> Message-ID: On 24 November 2015 at 17:16, Toshio Kuratomi wrote: > The long term answer for such environments is to update their internal > certificate management to at least match the standards set by the public > internet, but in the meantime, it is desirable to offer these administrators > a way to continue receiving maintenance updates to the Python 2.7 series, > without having to gate that on upgrades to their certificate management > infrastructure. > > + The wording here seems to reflect a different scope than merely > backporting by distros. Perhaps we should change it to: "[...]set by > the public internet. Distributions may wish to help these sites > transition by backporting the PEP 476 changes to earlier versions of > python in a way that does not require the administrators to upgrade > their certificate management infrastructure immediately. This would > allow sites to choose to use the distribution suppiied python in a > backwards compatible fashion until their certificate management > infrastructure was updated and then toggle their site to utilize the > more secure features provided by PEP 476." I'm not actually sure that it's the place of this PEP to even comment on what the long term answer for such environments should be (or indeed, any answer, long term or not). I've argued the position that in some organisations it feels like security don't actually understand the issues of carefully balancing secure operation against flexible development practices, but conversely it's certainly true that in many organisations, they *have* weighed the various arguments and made an informed decision on how to set up their internal network. It's entirely possible that self-signed certificates are entirely the right decision for their circumstances. Why would a Python PEP be qualified to comment on that decision? In my opinion, we should take all of the value judgements out of this paragraph, and just state the facts. How about: """ In order to provide additional flexibility to allow infrastructure administrators to provide the appropriate solution for their environment, this PEP offers a way for administrators to upgrade to later versions of the Python 2.7 series without being forced to update their existing security certificate management infrastructure as a prerequisite. """ Paul From a.badger at gmail.com Tue Nov 24 13:37:09 2015 From: a.badger at gmail.com (Toshio Kuratomi) Date: Tue, 24 Nov 2015 10:37:09 -0800 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <56536A6A.9030902@python.org> <20151123205906.02cd57ec@limelight.wooz.org> Message-ID: On Tue, Nov 24, 2015 at 10:08 AM, Paul Moore wrote: > I'm not actually sure that it's the place of this PEP to even comment > on what the long term answer for such environments should be (or > indeed, any answer, long term or not). I've argued the position that > in some organisations it feels like security don't actually understand > the issues of carefully balancing secure operation against flexible > development practices, I agree with this. > but conversely it's certainly true that in many > organisations, they *have* weighed the various arguments and made an > informed decision on how to set up their internal network. It's > entirely possible that self-signed certificates are entirely the right > decision for their circumstances. Why would a Python PEP be qualified > to comment on that decision? I don't quite agree with this but it's probably moot in the face of the previous and certain cornercases. Self-signed certs work just fine with the backports to python-2.7.9+ but you have to add the ca to the clients. An organization that has weighed the arguments and made an informed decision to use self-signed certs should either do this (to prevent MITM) or they should switch to using http instead of https (because MITM isn't a feasible attack here). The cornercases come into play because you don't always control all of the devices and services on your network. The site could evaluate and decide that MITM isn't a threat to their switch's configuration interface but that interface might be served over https using a certificate signed by their network vendor who doesn't give out their ca certificate (simply stated: your security team knows what they're doing but your vendor's does not). > In my opinion, we should take all of the value judgements out of this > paragraph, and just state the facts. How about: > > """ > In order to provide additional flexibility to allow infrastructure > administrators to provide the appropriate solution for their > environment, this PEP offers a way for administrators to upgrade to > later versions of the Python 2.7 series without being forced to update > their existing security certificate management infrastructure as a > prerequisite. > """ Two notes: * python-2.7.9+ doesn't give you flexibility in this regard so organizations do have to update their certificate management infrastructure. The cornercase described above becomes something that has to be addressed at the code level. Environments that are simply misconfigured have to be fixed. So in that regard, a value judgement does seem appropriate here. the judgement is "Listen guys, this PEP advises redistributors on how they might provide a migration path for you but it *does not bandaid the problem indefinitely*. So long term, you have to change your practices or you'll be out in the cold when your redistributor upgrades to python-2.7.9+" * Your proposed text actually removes the fix that I was adding -- this version of the paragraph implies that if your environment is compatible with the redistributors' python-2.7.8 (or less) then it will also be compatible with the redistributors' python-2.7.9+. That is not true. Whether or not we take out any value judgement as to the user's present environment this paragraph needs to be fixed to make it clear that this only affects redistributor's packages which have backported pep 476 to python-2.7.8 or older. Once the redistributor updates to a newer python sites which relied on this crutch will break. -Toshio From p.f.moore at gmail.com Tue Nov 24 13:56:47 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Nov 2015 18:56:47 +0000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <56536A6A.9030902@python.org> <20151123205906.02cd57ec@limelight.wooz.org> Message-ID: On 24 November 2015 at 18:37, Toshio Kuratomi wrote: > I don't quite agree with this but it's probably moot in the face of > the previous and certain cornercases. Self-signed certs work just > fine with the backports to python-2.7.9+ but you have to add the ca to > the clients. An organization that has weighed the arguments and made > an informed decision to use self-signed certs should either do this > (to prevent MITM) or they should switch to using http instead of https > (because MITM isn't a feasible attack here). There's an assumption (which is untrue for the big organisations I'm familiar with) that security are involved with the full stack - typically in my experience "corporate security" set rules that are filtered down through many layers, and development teams have no influence on those decisions - conversely, the people setting security policies aren't aware of the details of development day to day needs. It's easy to say that users just click on the "allow" button in their browser if they want to use self-signed certificates, but not have any viable solution for automation code that doesn't use a browser - and such automation code is "not corporately supported", so policy won't get updated to address the disconnect. > The cornercases come > into play because you don't always control all of the devices and > services on your network. The site could evaluate and decide that > MITM isn't a threat to their switch's configuration interface but that > interface might be served over https using a certificate signed by > their network vendor who doesn't give out their ca certificate (simply > stated: your security team knows what they're doing but your vendor's > does not). This sounds like a similar situation to what I described above. I'm not sure I'd see these as corner cases, though - they are pretty much day to day business in my experience :-( >> In my opinion, we should take all of the value judgements out of this >> paragraph, and just state the facts. How about: >> >> """ >> In order to provide additional flexibility to allow infrastructure >> administrators to provide the appropriate solution for their >> environment, this PEP offers a way for administrators to upgrade to >> later versions of the Python 2.7 series without being forced to update >> their existing security certificate management infrastructure as a >> prerequisite. >> """ > > Two notes: > > * python-2.7.9+ doesn't give you flexibility in this regard so > organizations do have to update their certificate management > infrastructure. The cornercase described above becomes something that > has to be addressed at the code level. Environments that are simply > misconfigured have to be fixed. So in that regard, a value judgement > does seem appropriate here. the judgement is "Listen guys, this PEP > advises redistributors on how they might provide a migration path for > you but it *does not bandaid the problem indefinitely*. So long term, > you have to change your practices or you'll be out in the cold when > your redistributor upgrades to python-2.7.9+" Hmm, maybe I misread the PEP (I only skimmed it - as I say, Linux is of limited interest to me). I thought that the environment variable gave developers a "get out" clause. Maybe it's not what we want them to do (for some value of "we") but isn't that the point of the PEP? Admittedly if distributions don't *implement* that part of the PEP (and I understand Red Hat haven't) then people are still stuck. But "this PEP offers a way" is not incompatible with "your vendor didn't implement the PEP so you're still stuck, sorry"... > * Your proposed text actually removes the fix that I was adding -- > this version of the paragraph implies that if your environment is > compatible with the redistributors' python-2.7.8 (or less) then it > will also be compatible with the redistributors' python-2.7.9+. That > is not true. Whether or not we take out any value judgement as to the > user's present environment this paragraph needs to be fixed to make it > clear that this only affects redistributor's packages which have > backported pep 476 to python-2.7.8 or older. Once the redistributor > updates to a newer python sites which relied on this crutch will > break. Sorry for that. Certainly getting the facts right is crucial, and it looks like my suggestion didn't do that. But hopefully someone can fix it up (if people think it's a good way to go). Paul From a.badger at gmail.com Tue Nov 24 14:40:50 2015 From: a.badger at gmail.com (Toshio Kuratomi) Date: Tue, 24 Nov 2015 11:40:50 -0800 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <56536A6A.9030902@python.org> <20151123205906.02cd57ec@limelight.wooz.org> Message-ID: On Tue, Nov 24, 2015 at 10:56 AM, Paul Moore wrote: > On 24 November 2015 at 18:37, Toshio Kuratomi wrote: >> The cornercases come >> into play because you don't always control all of the devices and >> services on your network. The site could evaluate and decide that >> MITM isn't a threat to their switch's configuration interface but that >> interface might be served over https using a certificate signed by >> their network vendor who doesn't give out their ca certificate (simply >> stated: your security team knows what they're doing but your vendor's >> does not). > > This sounds like a similar situation to what I described above. I'm > not sure I'd see these as corner cases, though - they are pretty much > day to day business in my experience :-( > It sounds like you're coming from a Windows background and I'm coming from a Linux background which might be a small disconnect here -- we do seem to be in agreement that what's "right to do" isn't always easy or possible for the client to accomplish so I think we should probably leave it at that. >>> In my opinion, we should take all of the value judgements out of this >>> paragraph, and just state the facts. How about: >>> >>> """ >>> In order to provide additional flexibility to allow infrastructure >>> administrators to provide the appropriate solution for their >>> environment, this PEP offers a way for administrators to upgrade to >>> later versions of the Python 2.7 series without being forced to update >>> their existing security certificate management infrastructure as a >>> prerequisite. >>> """ >> >> Two notes: >> >> * python-2.7.9+ doesn't give you flexibility in this regard so >> organizations do have to update their certificate management >> infrastructure. The cornercase described above becomes something that >> has to be addressed at the code level. Environments that are simply >> misconfigured have to be fixed. So in that regard, a value judgement >> does seem appropriate here. the judgement is "Listen guys, this PEP >> advises redistributors on how they might provide a migration path for >> you but it *does not bandaid the problem indefinitely*. So long term, >> you have to change your practices or you'll be out in the cold when >> your redistributor upgrades to python-2.7.9+" > > Hmm, maybe I misread the PEP (I only skimmed it - as I say, Linux is > of limited interest to me). I thought that the environment variable > gave developers a "get out" clause. Maybe it's not what we want them > to do (for some value of "we") but isn't that the point of the PEP? > > Admittedly if distributions don't *implement* that part of the PEP > (and I understand Red Hat haven't) then people are still stuck. But > "this PEP offers a way" is not incompatible with "your vendor didn't > implement the PEP so you're still stuck, sorry"... > Yeah, I think you are correct in your understanding of what actual changes to the python distrribution are being proposed for redistributors in the PEP. Reading through the PEP again, I'm not sure if I'm correct in thinking that this only applies to backporting... it may be that the environment section of the PEP applies to any python-2 while the config file section only applies to backporting. Nick, could you clarify? The PEP is clear that it doesn't apply to python-3 or cross-distro. So that means that sites still can't rely on this long-term (but their long term would extend to the lifetime of their vendor supporting python2 rather than when their vendor updated to 2.7.9+) and also that developers can't depend on this if they're developing portable code. >> * Your proposed text actually removes the fix that I was adding -- >> this version of the paragraph implies that if your environment is >> compatible with the redistributors' python-2.7.8 (or less) then it >> will also be compatible with the redistributors' python-2.7.9+. That >> is not true. Whether or not we take out any value judgement as to the >> user's present environment this paragraph needs to be fixed to make it >> clear that this only affects redistributor's packages which have >> backported pep 476 to python-2.7.8 or older. Once the redistributor >> updates to a newer python sites which relied on this crutch will >> break. > > Sorry for that. Certainly getting the facts right is crucial, and it > looks like my suggestion didn't do that. But hopefully someone can fix > it up (if people think it's a good way to go). Could be that I'm wrong -- will wait for Nick to clarify before I think about what could be done to make this wording better. -Toshio From brett at snarky.ca Tue Nov 24 14:31:48 2015 From: brett at snarky.ca (Brett Cannon) Date: Tue, 24 Nov 2015 19:31:48 +0000 Subject: [Python-Dev] [Python-checkins] Daily reference leaks (e5e507a357a6): sum=103 In-Reply-To: <20151124084506.81870.97482@psf.io> References: <20151124084506.81870.97482@psf.io> Message-ID: Someone just added a leak to pickle. On Tue, 24 Nov 2015 at 01:45 wrote: > results for e5e507a357a6 on branch "default" > -------------------------------------------- > > test_asyncio leaked [3, 0, 0] memory blocks, sum=3 > test_functools leaked [0, 2, 2] memory blocks, sum=4 > test_pickle leaked [20, 20, 20] references, sum=60 > test_pickle leaked [12, 12, 12] memory blocks, sum=36 > > > Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', > '3:3:/home/psf-users/antoine/refleaks/reflogUP9efi', '--timeout', '7200'] > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Nov 24 16:21:21 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 24 Nov 2015 23:21:21 +0200 Subject: [Python-Dev] Deleting with setting C API functions Message-ID: Slots like PyTypeObject.tp_setattr, PySequenceMethods.sq_ass_item, PyMappingMethods.mp_ass_subscript are used not only for setting attribute/item value, but also for deleting attribute/item (if value is NULL). This is not documented and should be. [1] Correspondingly public API functions like PyObject_SetAttr, PyObject_SetItem, PySequence_SetItem, PySequence_SetSlice, PyMapping_SetItemString can be used for deleting. But all these functions have special counterparts for deleting: PyObject_DelAttr etc. The question is wherever deleting ability of Set-functions is intentional, should we document this or deprecate and then delete? [1] http://bugs.python.org/issue25701 From storchaka at gmail.com Tue Nov 24 16:22:22 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 24 Nov 2015 23:22:22 +0200 Subject: [Python-Dev] [Python-checkins] Daily reference leaks (e5e507a357a6): sum=103 In-Reply-To: References: <20151124084506.81870.97482@psf.io> Message-ID: On 24.11.15 21:31, Brett Cannon wrote: > Someone just added a leak to pickle. It always was here. I just added tests that expose it. http://bugs.python.org/issue25725 From guido at python.org Tue Nov 24 16:33:09 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 24 Nov 2015 13:33:09 -0800 Subject: [Python-Dev] Deleting with setting C API functions In-Reply-To: References: Message-ID: Ooooh, that's probably really old code. I guess for the slots the reasoning is to save on slots. For the public functions, alas it will be hard to know if anyone is depending on it, even if it's undocumented. Perhaps add a deprecation warning to these if the value is NULL for one release cycle? On Tue, Nov 24, 2015 at 1:21 PM, Serhiy Storchaka wrote: > Slots like PyTypeObject.tp_setattr, PySequenceMethods.sq_ass_item, > PyMappingMethods.mp_ass_subscript are used not only for setting > attribute/item value, but also for deleting attribute/item (if value is > NULL). This is not documented and should be. [1] Correspondingly public API > functions like PyObject_SetAttr, PyObject_SetItem, PySequence_SetItem, > PySequence_SetSlice, PyMapping_SetItemString can be used for deleting. But > all these functions have special counterparts for deleting: PyObject_DelAttr > etc. > > The question is wherever deleting ability of Set-functions is intentional, > should we document this or deprecate and then delete? > > [1] http://bugs.python.org/issue25701 > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Tue Nov 24 20:07:02 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Nov 2015 11:07:02 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <56536A6A.9030902@python.org> <20151123205906.02cd57ec@limelight.wooz.org> Message-ID: On 25 Nov 2015 05:42, "Toshio Kuratomi" wrote: > > > > Yeah, I think you are correct in your understanding of what actual > changes to the python distrribution are being proposed for > redistributors in the PEP. Reading through the PEP again, I'm not > sure if I'm correct in thinking that this only applies to > backporting... it may be that the environment section of the PEP > applies to any python-2 while the config file section only applies to > backporting. Nick, could you clarify? Yep, the environment variable section is the part MAL persuaded me was a good idea during the original discussion of this PEP, based on the interests of eGenix's customers. It's a cross-platform suggestion for distributors wanting to provide a smoother upgrade path, applicable to both backports and rebasing on 2.7.9+. (I originally didn't like it, but became amenable once it was pointed out that anyone with the ability to set process environment variables can already alter OpenSSL's settings to use a different CA cert file or directory) The config file recommendation is specific to Linux distro backports, with the aim of saying "IF you backport PEP 476 to an older Python 2.7 point release, THEN you should do it like this for cross-distro consistency". As the current discussion shows, I'm *personally* an advocate for backporting, but I don't think the PEP itself should reflect that - the suggestions to adjust the tone are good ones, so I'll update the text accordingly. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Nov 25 01:39:27 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Nov 2015 16:39:27 +1000 Subject: [Python-Dev] Deleting with setting C API functions In-Reply-To: References: Message-ID: On 25 November 2015 at 07:33, Guido van Rossum wrote: > Ooooh, that's probably really old code. I guess for the slots the > reasoning is to save on slots. For the public functions, alas it will > be hard to know if anyone is depending on it, even if it's > undocumented. Perhaps add a deprecation warning to these if the value > is NULL for one release cycle? I did a quick scan for "PyObject_SetAttr", and it turns out PyObject_DelAttr is only a convenience macro for calling PyObject_SetAttr with NULL as the value argument. bltinmodule.c and ceval.c also both include direct calls to PyObject_SetAttr with "(PyObject *)NULL" as the value argument. Investigating some of the uses that passed a variable as the value argument, one case is the weakref proxy implementation, which uses PyObject_SetAttr on the underlying object in its implementation of the setattr slot in the proxy. So it looks to me like replicating the NULL-handling behaviour of the slots in the public Set* APIs was intentional, and it's just the documentation of that detail that was missed (since most folks presumably use the Del* convenience APIs instead). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From cory at lukasa.co.uk Wed Nov 25 04:12:29 2015 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 25 Nov 2015 09:12:29 +0000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <56545DDC.7000407@python.org> References: <56536A6A.9030902@python.org> <56545DDC.7000407@python.org> Message-ID: > On 24 Nov 2015, at 12:53, Christian Heimes wrote: > Right, with Antoine and Alex out of scope and you, Victor and me working > for Red Hat, the air is getting thin. Benjamin is familiar with the ssl > module. Or we can follow Alex's advice and ask somebody from the PyCA > group (Donald, Paul, lvh) or requests (Cory) to get some outside > perspective. I?m not a CPython core developer, so I think that excludes me. For what it?s worth, I think that this is a good PEP, and I?m strongly in favour of it being approved in some form. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From barry at python.org Wed Nov 25 14:57:14 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 25 Nov 2015 14:57:14 -0500 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <56547C68.8060506@egenix.com> References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> Message-ID: <20151125145714.2a02651d@limelight.wooz.org> There's a lot to process in this thread, but as I see it, the issue breaks down to these questions: * How should PEP 493 be implemented? * What should the default be? * How should PEP 493 be worded to express the right tone to redistributors? Let me take on the implementation details here. On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote: >I would still find having built-in support for the recommendations >in the Python stdlib a better approach As would I. Currently the PEP contains some sample code for reading a configuration file and an environment variable, and assigning the default behavior based on these values. The first improvement to the stdlib I would have made would have been to add a convenience function to ssl to enable and disable the verification. Something like this: def ssl.verify_certificates(enable) Then at the simplest level, an application that wants to enable higher security could do this early in its main entry point: import ssl if hasattr(ssl, 'verify_certificates'): ssl.verify_certificates(True) or ... ssl.verify_certificates(False) and that would work regardless of the platform default, and it would work properly on older versions of Python 2.7 that didn't have the backport. This seems so straightforward, I'm wondering why PEP 493 doesn't consider it. I would like to see the sample code in PEP 493 implemented in the stdlib. It would then be a matter of cherry picking that should a distro decide to backport it. It means one less non-trivial delta from upstream which has to be managed separately. The only difference to the combined approach I'd make is to handle the case where the _cert_verification_config file doesn't exist. (If you included the convenience method above, then the sample code could be simplified down to a default for the `enable` parameter and a call to ssl.verify_certificates().) Is it too late to include both of those suggestions in PEP 493 and Python 2.7.11? Cheers, -Barry From robertc at robertcollins.net Wed Nov 25 15:17:02 2015 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 26 Nov 2015 09:17:02 +1300 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <20151125145714.2a02651d@limelight.wooz.org> References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> Message-ID: On 26 November 2015 at 08:57, Barry Warsaw wrote: > There's a lot to process in this thread, but as I see it, the issue breaks > down to these questions: > > * How should PEP 493 be implemented? > > * What should the default be? > > * How should PEP 493 be worded to express the right tone to redistributors? > > Let me take on the implementation details here. > > On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote: > >>I would still find having built-in support for the recommendations >>in the Python stdlib a better approach > > As would I. For what its worth: a PEP telling distributors to patch the standard library is really distasteful to me. We've spent a long time trying to build close relations such that when something doesn't work distributors can share their needs with us and we can make Python out of the box be a good fit. This seems to fly in the exact opposite direction: we're explicitly making it so that Python builds on these vendor's platforms will not be the same as you get by checking out the Python source code. Ugh. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From rdmurray at bitdance.com Wed Nov 25 15:39:54 2015 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 25 Nov 2015 15:39:54 -0500 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> Message-ID: <20151125203955.D1791B1408E@webabinitio.net> On Thu, 26 Nov 2015 09:17:02 +1300, Robert Collins wrote: > On 26 November 2015 at 08:57, Barry Warsaw wrote: > > There's a lot to process in this thread, but as I see it, the issue breaks > > down to these questions: > > > > * How should PEP 493 be implemented? > > > > * What should the default be? > > > > * How should PEP 493 be worded to express the right tone to redistributors? > > > > Let me take on the implementation details here. > > > > On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote: > > > >>I would still find having built-in support for the recommendations > >>in the Python stdlib a better approach > > > > As would I. > > For what its worth: a PEP telling distributors to patch the standard > library is really distasteful to me. > > We've spent a long time trying to build close relations such that when > something doesn't work distributors can share their needs with us and > we can make Python out of the box be a good fit. This seems to fly in > the exact opposite direction: we're explicitly making it so that > Python builds on these vendor's platforms will not be the same as you > get by checking out the Python source code. I think we should include the environment variable support in CPython and be done with it (nuke the PEP otherwise). Which is what I've thought from the beginning :) --David From sad2project at gmail.com Wed Nov 25 15:31:09 2015 From: sad2project at gmail.com (Jacob Zimmerman) Date: Wed, 25 Nov 2015 14:31:09 -0600 Subject: [Python-Dev] Help with a book Message-ID: Hello Python Developers! I'm writing a book about descriptors, and I'm hoping to make it a comprehensive guide, covering just about everything there is to know about descriptors, including many tips to get past certain knowledge barriers and how to avoid certain pitfalls. I'm hoping for it to become the definitive guide for those who want to really understand descriptors. Raymond's Descriptor HowTo Guide is great, but focused. I want to expand on what that guide teaches. Anyway, I am currently reading through my initial rough draft and making corrections for a revised draft. Once I'm done with that, I'd like a few volunteers to read through it. First, to find spelling, grammatical, or consistency mistakes, though those should be few in number, but also to give input as to whether something seems too repetitive, if some more information should be given in a spot, and if I'm even blatantly wrong about anything (hopefully not). Anyone who helps will be able to receive a free ebook of the final copy as well as mention in the Special Thanks. I appreciate any help anyone can give to assist in making this book as great as it can be. Thanks, Jacob Zimmerman Twitter: @jacobz_20 Blog: http://programmingideaswithjake.wordpress.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Nov 25 15:57:13 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 25 Nov 2015 15:57:13 -0500 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <56536A6A.9030902@python.org> <20151123205906.02cd57ec@limelight.wooz.org> Message-ID: <20151125155713.42936cff@limelight.wooz.org> On Nov 24, 2015, at 09:16 AM, Toshio Kuratomi wrote: >It sounds like the implementation sections of the PEP are acceptable Well, see my follow up to MAL's email. From Rob's and RDM's responses, it seems we're not alone. :) I like all your other proposed changes, although I'll note the preference to minimize the unique-to-downstream changes. Cheers, -Barry From lac at openend.se Wed Nov 25 15:58:44 2015 From: lac at openend.se (Laura Creighton) Date: Wed, 25 Nov 2015 21:58:44 +0100 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <20151125203955.D1791B1408E@webabinitio.net> References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> <20151125203955.D1791B1408E@webabinitio.net> Message-ID: <201511252058.tAPKwiiY025031@fido.openend.se> In a message of Wed, 25 Nov 2015 15:39:54 -0500, "R. David Murray" writes: >I think we should include the environment variable support in CPython >and be done with it (nuke the PEP otherwise). Which is what I've >thought from the beginning :) > >--David I like this idea too. Laura From barry at python.org Wed Nov 25 16:04:35 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 25 Nov 2015 16:04:35 -0500 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <56536A6A.9030902@python.org> <20151123205906.02cd57ec@limelight.wooz.org> Message-ID: <20151125160435.2de0791c@limelight.wooz.org> On Nov 24, 2015, at 02:12 PM, Nick Coghlan wrote: >On 24 November 2015 at 11:59, Barry Warsaw wrote: >> We'd have to also handle >> migration paths to newer Ubuntu releases, which probably means removing the >> config file on future upgrades. > >You'll need to figure out those migration paths anyway, as the certificate >verification compatibility break will exist between 14.04 and 16.04 >regardless of what happens with PEP 493. Sure, but that's a very different situation. If someone upgraded from 14.04.3 to 14.04.4, they definitely wouldn't expect it to break working code. If it did, it would be a pretty serious regression. Upgrading to LTS maintenance releases should be a no-brainer. Upgrading from one LTS to a newer one, you *know* things are going to change, so you'll review the release notes and do some very careful analysis of your mission critical code. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Wed Nov 25 16:13:32 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 25 Nov 2015 16:13:32 -0500 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <20151125203955.D1791B1408E@webabinitio.net> References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> <20151125203955.D1791B1408E@webabinitio.net> Message-ID: <20151125161332.43b0dd5b@limelight.wooz.org> On Nov 25, 2015, at 03:39 PM, R. David Murray wrote: >I think we should include the environment variable support in CPython >and be done with it (nuke the PEP otherwise). Which is what I've >thought from the beginning :) +1. I'd like to see my proposed convenience function added too. I'm -0 on the config file though since while I think it might be useful, I'm not sure we'd actually install that on Ubuntu. Cheers, -Barry From vgr255 at live.ca Wed Nov 25 17:48:00 2015 From: vgr255 at live.ca (Emanuel Barry) Date: Wed, 25 Nov 2015 17:48:00 -0500 Subject: [Python-Dev] Help with a book In-Reply-To: References: Message-ID: As someone who plays around a *lot* with descriptors, I would be interested in reviewing it. I may not have the same knowledge as a core developer, but I can still help. Date: Wed, 25 Nov 2015 14:31:09 -0600 From: sad2project at gmail.com To: python-dev at python.org Subject: [Python-Dev] Help with a book Hello Python Developers! I'm writing a book about descriptors, and I'm hoping to make it a comprehensive guide, covering just about everything there is to know about descriptors, including many tips to get past certain knowledge barriers and how to avoid certain pitfalls. I'm hoping for it to become the definitive guide for those who want to really understand descriptors. Raymond's Descriptor HowTo Guide is great, but focused. I want to expand on what that guide teaches. Anyway, I am currently reading through my initial rough draft and making corrections for a revised draft. Once I'm done with that, I'd like a few volunteers to read through it. First, to find spelling, grammatical, or consistency mistakes, though those should be few in number, but also to give input as to whether something seems too repetitive, if some more information should be given in a spot, and if I'm even blatantly wrong about anything (hopefully not). Anyone who helps will be able to receive a free ebook of the final copy as well as mention in the Special Thanks. I appreciate any help anyone can give to assist in making this book as great as it can be. Thanks,Jacob Zimmerman Twitter: @jacobz_20Blog: http://programmingideaswithjake.wordpress.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Nov 25 18:55:05 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 26 Nov 2015 10:55:05 +1100 Subject: [Python-Dev] Help with a book In-Reply-To: References: Message-ID: On Thu, Nov 26, 2015 at 7:31 AM, Jacob Zimmerman wrote: > Anyway, I am currently reading through my initial rough draft and making > corrections for a revised draft. Once I'm done with that, I'd like a few > volunteers to read through it. First, to find spelling, grammatical, or > consistency mistakes, though those should be few in number, but also to give > input as to whether something seems too repetitive, if some more information > should be given in a spot, and if I'm even blatantly wrong about anything > (hopefully not). I'd be happy to help out. Wordsmithing is my life, and I've used descriptors some (not as much as Emanuel will have, though!). ChrisA From ncoghlan at gmail.com Wed Nov 25 23:13:14 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Nov 2015 14:13:14 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <20151125145714.2a02651d@limelight.wooz.org> References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> Message-ID: On 26 November 2015 at 05:57, Barry Warsaw wrote: > There's a lot to process in this thread, but as I see it, the issue breaks > down to these questions: > > * How should PEP 493 be implemented? > > * What should the default be? > > * How should PEP 493 be worded to express the right tone to redistributors? > > Let me take on the implementation details here. > > On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote: > >>I would still find having built-in support for the recommendations >>in the Python stdlib a better approach > > As would I. > > Currently the PEP contains some sample code for reading a configuration file > and an environment variable, and assigning the default behavior based on these > values. > > The first improvement to the stdlib I would have made would have been to add > a convenience function to ssl to enable and disable the verification. > Something like this: > > def ssl.verify_certificates(enable) > > Then at the simplest level, an application that wants to enable higher > security could do this early in its main entry point: > > import ssl > if hasattr(ssl, 'verify_certificates'): > ssl.verify_certificates(True) > > or > > ... > ssl.verify_certificates(False) > > and that would work regardless of the platform default, and it would work > properly on older versions of Python 2.7 that didn't have the backport. This > seems so straightforward, I'm wondering why PEP 493 doesn't consider it. PEP 476 rejected providing a public indefinitely maintained API for this, so PEP 493 is specifically about helping commercial redistributors offer a smoother transition plan to customers without affecting the public Python level API, and without encouraging a plethora of mutually incompatible transition schemes. > I would like to see the sample code in PEP 493 implemented in the stdlib. It > would then be a matter of cherry picking that should a distro decide to > backport it. It means one less non-trivial delta from upstream which has to > be managed separately. The only difference to the combined approach I'd make > is to handle the case where the _cert_verification_config file doesn't exist. > > (If you included the convenience method above, then the sample code could be > simplified down to a default for the `enable` parameter and a call to > ssl.verify_certificates().) > > Is it too late to include both of those suggestions in PEP 493 and Python > 2.7.11? PEP 493 isn't about attempting to rehash the PEP 476 discussions in search of a different conclusion, so this would need to be a different PEP, preferably one that targets Python 3.6 first and covers more than just HTTPS. Such an API could then be considered as a backport candidate for 2.7.12. (2.7.11rc1 has already been published, so there isn't any scope for updates there). I actually think that's a nicer approach, but also think the recommendations in PEP 493 are much simpler to implement and test, while still meeting the bar of "good enough to solve the problem at hand". If anyone does decide to write that PEP, then the fact that SELinux offers "setenforce 0" and AppArmor can be turned off with a kernel boot option can be used as examples of platform developers letting end users make their own security decisions, even if the platform provider recommends leaving the more opinionated default behaviour enabled. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Nov 26 00:06:00 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Nov 2015 15:06:00 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> Message-ID: On 26 November 2015 at 06:17, Robert Collins wrote: > On 26 November 2015 at 08:57, Barry Warsaw wrote: >> There's a lot to process in this thread, but as I see it, the issue breaks >> down to these questions: >> >> * How should PEP 493 be implemented? >> >> * What should the default be? >> >> * How should PEP 493 be worded to express the right tone to redistributors? >> >> Let me take on the implementation details here. >> >> On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote: >> >>>I would still find having built-in support for the recommendations >>>in the Python stdlib a better approach >> >> As would I. > > For what its worth: a PEP telling distributors to patch the standard > library is really distasteful to me. I'm not a big fan of it either, but it's the way sustainable commercial open source distribution works in practice: customers want selective upgrades, so customers get selective upgrades; customers want greater platform integration, so customers get greater platform integration; etc, etc. Large Python shops like Google will typically maintain their own custom Python variants with their own patches applied that are suitable and sufficient for their use cases, but may not be suitable for pushing upstream (e.g. they may be Linux specific patches, without support for other platforms). That's why I've focused on providing guidance to redistributors to keep their patches within reasonably consistent bounds for PEPs like bundling pip, backporting the network security enhancements to Python 2.7, and changing the default behaviour for HTTPS verification. Different redistributors have different customer bases, which also differ from the audience for upstream python.org releases, so saying "don't patch Python" is denying the reality of the obligations commercial vendors have to their customers, while "when you patch Python, please ensure you abide by these guidelines" is something redistributors can realistically do. > We've spent a long time trying to build close relations such that when > something doesn't work distributors can share their needs with us and > we can make Python out of the box be a good fit. This seems to fly in > the exact opposite direction: we're explicitly making it so that > Python builds on these vendor's platforms will not be the same as you > get by checking out the Python source code. This has always been true for the RHEL system Python - it gets backported changes, just like the kernel does (albeit nowhere near as many). That's one of the main things long term support customers are paying for - developers continuing to care about a code base after the volunteers working on the project upstream have (entirely reasonably) shifted their attentions to newer versions. We try to keep the patches to a minimum, and avoid having to carry large patches indefinitely by only backporting changes that have already been accepted in later versions, but "never apply patches to upstream projects as a downstream redistributor" isn't a realistic goal. In this particular case, the migration problems were already raised in the PEP 476 discussions, and the decision was made to *not* provide a public API specifically for globally turning off HTTPS certificate verification, since that would either: 1. Need to be added to Python 3 and maintained indefinitely; or 2. Be implemented in Python 2.7 only, and thus create another barrier to 2->3 migration A HTTPS specific flag wasn't considered a suitable long term API design for Python 3, while a multi-protocol configuration design that also handled setting the defaults for other protocols like secure SMTP, IMAP, POP3, SFTP, etc seemed generally desirable (but also doesn't exist anywhere, and poses signficant challenges in designing and implementing a good test matrix). The only concrete things that have changed since PEP 476 was approved is that: * we figured out the monkeypatching based migration recommendations in that PEP didn't actually work for the redistributor case * Robert Kuska came up with a way to do a backwards compatible backport of PEP 476 to RHEL 7.2 that doesn't raise any new barriers to migrating from Python 2 to Python 3 PEP 493 originally only described the latter idea - I added MAL's design for the environment variable based security downgrade based on the original PEP 493 discussions in July. Splitting the latter out to its own PEP that proposes it as a forwards compatible configuration setting akin to "curl --insecure" and "wget --no-check-certificate" for Python 2.7.12, 3.4.4, 3.5.2 and 3.6 upstream would be fine by me, since that would also significantly clarify the fact that the two ideas are entirely orthogonal to each other (except for the need to define a relative priority when you combine the two). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From barry at python.org Thu Nov 26 11:48:43 2015 From: barry at python.org (Barry Warsaw) Date: Thu, 26 Nov 2015 11:48:43 -0500 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> Message-ID: <20151126114843.3db83d5f@anarchist.wooz.org> On Nov 26, 2015, at 02:13 PM, Nick Coghlan wrote: >PEP 476 rejected providing a public indefinitely maintained API for this, so >PEP 493 is specifically about helping commercial redistributors offer a >smoother transition plan to customers without affecting the public Python >level API, and without encouraging a plethora of mutually incompatible >transition schemes. Of course, the API would only have to be support for the life of 2.7; it would never go in 3.x so the burden is minimal. >PEP 493 isn't about attempting to rehash the PEP 476 discussions in >search of a different conclusion, so this would need to be a different >PEP, preferably one that targets Python 3.6 first and covers more than >just HTTPS. That seems like overkill. PEP 493 is specifically about Python 2.7 and providing ways for downstreams to facilitate more choice for end-users and end-administrators. Although I think it could safely sneak in after rc1, that would be for the RM to decide. Even if it were deferred to 2.7.12, it would still provide a better, more consistent experience if implemented upstream. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Thu Nov 26 12:15:10 2015 From: barry at python.org (Barry Warsaw) Date: Thu, 26 Nov 2015 12:15:10 -0500 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> Message-ID: <20151126121510.32c9a500@anarchist.wooz.org> On Nov 26, 2015, at 03:06 PM, Nick Coghlan wrote: >I'm not a big fan of it either, but it's the way sustainable >commercial open source distribution works in practice: While it's inevitable that redistributors have to deviate from upstream, in Debian and Ubuntu, we really try to keep that at a minimum. Debian, of course is an all volunteer organization so our "customers" are just normal users who I think want an experience as close to installing from source as possible, consistent with the principles and policies of the Debian project. Debian at least tries to document the handful of deviations from upstream. Likewise in Ubuntu, we try to keep deviations from Debian at a minimum, and document them when we must deviate. Ubuntu is a community driven distro so while Canonical itself has customers, it's much more likely that feedback about the Python stack comes from ordinary users. Again, my personal goal is to make Python on Ubuntu a pleasant and comfortable environment, as close to installing from source as possible, consistent with the principles and policies of the project. All this to say that IMHO, solutions to problems should go as far upstream as possible. Guidelines certainly are useful (e.g. PEP 394) to maintain consistency across downstream providers where the decisions are primarily in the sphere of influence of the downstreams, but IMHO there's still more that upstream can do to promote PEP 493. >Different redistributors have different customer bases, which also >differ from the audience for upstream python.org releases, so saying >"don't patch Python" is denying the reality of the obligations >commercial vendors have to their customers, while "when you patch >Python, please ensure you abide by these guidelines" is something >redistributors can realistically do. ISTM that the same forces are in play regardless of whether the change is in code or in an informational PEP. Best to get consensus where possible, and manifest those decisions in code, but if competing goals are the outcome of a code change or informational PEP, downstream consumers will still make what they judge to be the best decision in the interest of their users, balanced against their own competing constraints. >In this particular case, the migration problems were already raised in >the PEP 476 discussions, and the decision was made to *not* provide a >public API specifically for globally turning off HTTPS certificate >verification, since that would either: > >1. Need to be added to Python 3 and maintained indefinitely; or >2. Be implemented in Python 2.7 only, and thus create another barrier >to 2->3 migration A minor one, for sure. It wouldn't significantly bother me if the API were underscore-prefixed as a big honkin' clue about the guarantees being made. >* Robert Kuska came up with a way to do a backwards compatible >backport of PEP 476 to RHEL 7.2 that doesn't raise any new barriers to >migrating from Python 2 to Python 3 I'm not sure I agree with that. If you have to set an environment variable to get your application to work in Python 2.7, it's *still* a migration issue when you move to Python 3. You're just moving the change out of the code and into the environment, where it's arguably more difficult to re-discover down the road. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From ncoghlan at gmail.com Thu Nov 26 19:52:27 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 27 Nov 2015 10:52:27 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <20151126121510.32c9a500@anarchist.wooz.org> References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> <20151126121510.32c9a500@anarchist.wooz.org> Message-ID: On 27 November 2015 at 03:15, Barry Warsaw wrote: > On Nov 26, 2015, at 03:06 PM, Nick Coghlan wrote: > >>I'm not a big fan of it either, but it's the way sustainable >>commercial open source distribution works in practice: > > While it's inevitable that redistributors have to deviate from upstream, in > Debian and Ubuntu, we really try to keep that at a minimum. Debian, of course > is an all volunteer organization so our "customers" are just normal users who > I think want an experience as close to installing from source as possible, > consistent with the principles and policies of the Debian project. Debian at > least tries to document the handful of deviations from upstream. Yeah, we need to be better about that - while https://access.redhat.com/articles/2039753 covers the details for the certificate verification changes, there's currently no easy way to get a description of the changes backported to the RHEL/CentOS system Python without trawling through old Errata announcements :( > Likewise in Ubuntu, we try to keep deviations from Debian at a minimum, and > document them when we must deviate. Ubuntu is a community driven distro so > while Canonical itself has customers, it's much more likely that feedback > about the Python stack comes from ordinary users. Again, my personal goal is > to make Python on Ubuntu a pleasant and comfortable environment, as close to > installing from source as possible, consistent with the principles and > policies of the project. I'd strongly agree with that description for Fedora and softwarecollections.org, but for the RHEL/CentOS system Python I think the situation is slightly different: there, the goal is to meet the long term support commitments involved in being a base RHEL package. As the nominal base version of the package (2.7.5 in the case of RHEL 7) doesn't change, there is naturally going to be increasing divergence from the nominal version. > All this to say that IMHO, solutions to problems should go as far upstream as > possible. Guidelines certainly are useful (e.g. PEP 394) to maintain > consistency across downstream providers where the decisions are primarily in > the sphere of influence of the downstreams, but IMHO there's still more that > upstream can do to promote PEP 493. I tried to go down the "upstream first" path with a properly supported "off switch" in PEP 476, and didn't succeed (hence the monkeypatch compromise). It sounds like several folks would like to see us revisit that decision, though. >>Different redistributors have different customer bases, which also >>differ from the audience for upstream python.org releases, so saying >>"don't patch Python" is denying the reality of the obligations >>commercial vendors have to their customers, while "when you patch >>Python, please ensure you abide by these guidelines" is something >>redistributors can realistically do. > > ISTM that the same forces are in play regardless of whether the change is in > code or in an informational PEP. Best to get consensus where possible, and > manifest those decisions in code, but if competing goals are the outcome of a > code change or informational PEP, downstream consumers will still make what > they judge to be the best decision in the interest of their users, balanced > against their own competing constraints. > >>In this particular case, the migration problems were already raised in >>the PEP 476 discussions, and the decision was made to *not* provide a >>public API specifically for globally turning off HTTPS certificate >>verification, since that would either: >> >>1. Need to be added to Python 3 and maintained indefinitely; or >>2. Be implemented in Python 2.7 only, and thus create another barrier >>to 2->3 migration > > A minor one, for sure. It wouldn't significantly bother me if the API were > underscore-prefixed as a big honkin' clue about the guarantees being made. OK, you've persuaded me to recast the PEP from an Informational one to a Standards track one. The section about backporting to versions prior to 2.7.9 won't change much (except in tone), but the ability to opt out on a process-wide basis will be pitched as a language level feature. >>* Robert Kuska came up with a way to do a backwards compatible >>backport of PEP 476 to RHEL 7.2 that doesn't raise any new barriers to >>migrating from Python 2 to Python 3 > > I'm not sure I agree with that. If you have to set an environment variable to > get your application to work in Python 2.7, it's *still* a migration issue > when you move to Python 3. You're just moving the change out of the code and > into the environment, where it's arguably more difficult to re-discover down > the road. By that I meant that setting the environment variable won't *break* anything in Python 3, whereas if we add a new Python level API and people call it without a hasattr() guard, then even if their infrastructure has been upgraded to use verifiable certificates, the client applications will break on Python 3. However, with an underscore prefix and an admonition to use a hasattr() check, I agree a 2.7.x-only Python level API isn't really any more of a barrier to migration than only offering the environment variable. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ldeller at iress.com.au Thu Nov 26 21:06:53 2015 From: ldeller at iress.com.au (Luke Deller) Date: Fri, 27 Nov 2015 02:06:53 +0000 Subject: [Python-Dev] crash bug in weakref_richcompare? Message-ID: Hi I have come across some dubious code in Objects/weakrefobject.c which looks like a bug to me, but wanted to run it past others. This was discovered from looking at crash dumps from a multithreaded python app (using Python 2.7.9, but the same weakref code exists in 3.5 and hg tip). The code that worries me is at the end of the "weakref_richcompare" function: return PyObject_RichCompare(PyWeakref_GET_OBJECT(self), PyWeakref_GET_OBJECT(other), op); At this point the code has established that the referents are still alive, and it is trying to compare the referents. However it has not acquired a strong reference to the referents, so I think it is possible for one of them to be deleted half way through this comparison. This can lead to a crash, because PyObject_RichCompare assumes that the PyObject*'s it was passed will remain usable for the duration of the call. The crashes I have seen involve data corruption consistent with one of these PyObject's being deleted and the memory reused for something else, eg: 00 python27!try_3way_compare+0x15 [objects\object.c @ 712] 01 python27!try_3way_to_rich_compare+0xb [objects\object.c @ 901] 02 python27!do_richcmp+0x2c [objects\object.c @ 935] 03 python27!PyObject_RichCompare+0x99 [objects\object.c @ 982] 04 python27!weakref_richcompare+0x6a [objects\weakrefobject.c @ 212] (In this example v->ob_type was 0x5f637865 which is ASCII "exc_", not a valid pointer at all) Other places in weakrefobject.c seem to have a similar weakness, eg in weakref_hash and weakref_repr. I have not been able to produce a small test case to exhibit this crash, but from this inspection of the code it looks like a bug - am I understanding this correctly? Thanks, Luke. ********************************************************************************************** Important Note This email (including any attachments) contains information which is confidential and may be subject to legal privilege. If you are not the intended recipient you must not use, distribute or copy this email. If you have received this email in error please notify the sender immediately and delete this email. Any views expressed in this email are not necessarily the views of IRESS Limited. It is the duty of the recipient to virus scan and otherwise test the information provided before loading onto any computer system. IRESS Limited does not warrant that the information is free of a virus or any other defect or error. ********************************************************************************************** From a.badger at gmail.com Thu Nov 26 23:02:58 2015 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 26 Nov 2015 20:02:58 -0800 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> <20151126121510.32c9a500@anarchist.wooz.org> Message-ID: On Nov 26, 2015 4:53 PM, "Nick Coghlan" wrote: > > On 27 November 2015 at 03:15, Barry Warsaw wrote: > > > Likewise in Ubuntu, we try to keep deviations from Debian at a minimum, and > > document them when we must deviate. Ubuntu is a community driven distro so > > while Canonical itself has customers, it's much more likely that feedback > > about the Python stack comes from ordinary users. Again, my personal goal is > > to make Python on Ubuntu a pleasant and comfortable environment, as close to > > installing from source as possible, consistent with the principles and > > policies of the project. > > I'd strongly agree with that description for Fedora and > softwarecollections.org, but for the RHEL/CentOS system Python I think > the situation is slightly different: there, the goal is to meet the > long term support commitments involved in being a base RHEL package. > As the nominal base version of the package (2.7.5 in the case of RHEL > 7) doesn't change, there is naturally going to be increasing > divergence from the nominal version. I think the goal in rhel/centos is similar, actually. The maintenance burden for non upstream changes has been acknowledged as a problem to be avoided by rhel maintainers before. The caveat for those distributions is that they accumulate more *backports*. However, backports are easier to maintain than non upstream changes. The test of the upstream community helps to find and fix bugs in the code; the downstream maintainer just needs to stay aware of whether fixes are going into the code they've backported. > I tried to go down the "upstream first" path with a properly supported > "off switch" in PEP 476, and didn't succeed (hence the monkeypatch > compromise). It sounds like several folks would like to see us revisit > that decision, though. > That's the rub. If there's now enough support to push this upstream I think everyone downstream will be happier. If it turns out there's still enough resistance to keep it from upstream then I suppose you cross that bridge if it becomes necessary. -Toshio -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Nov 27 01:04:30 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 27 Nov 2015 16:04:30 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> <20151126121510.32c9a500@anarchist.wooz.org> Message-ID: On 27 November 2015 at 10:52, Nick Coghlan wrote: > On 27 November 2015 at 03:15, Barry Warsaw wrote: >> On Nov 26, 2015, at 03:06 PM, Nick Coghlan wrote: >>>In this particular case, the migration problems were already raised in >>>the PEP 476 discussions, and the decision was made to *not* provide a >>>public API specifically for globally turning off HTTPS certificate >>>verification, since that would either: >>> >>>1. Need to be added to Python 3 and maintained indefinitely; or >>>2. Be implemented in Python 2.7 only, and thus create another barrier >>>to 2->3 migration >> >> A minor one, for sure. It wouldn't significantly bother me if the API were >> underscore-prefixed as a big honkin' clue about the guarantees being made. > > OK, you've persuaded me to recast the PEP from an Informational one to > a Standards track one. The section about backporting to versions prior > to 2.7.9 won't change much (except in tone), but the ability to opt > out on a process-wide basis will be pitched as a language level > feature. New draft pushed: https://hg.python.org/peps/rev/f602a47ea795 This is a significant rewrite that switches the PEP to a Standards Track PEP proposing two new features for 2.7.12+: an "ssl._verify_https_certificates()" configuration function, and a "PYTHONHTTPSVERIFY" environment variable (although writing them together like that makes me wonder if the latter should now be "PYTHONVERIFYHTTPS" instead). There are then 3 backport recommendation sections: * one for backporting the ability to enable HTTPS verification by default for the entire Python installation * one for backporting the new features described in this PEP * one for combining both backports in a single implementation I tried to trim some of the resulting duplication between the sections, but I suspect the overall text is still longer than it needs to be. I've also added links describing the origin of the configuration file based backport in RHEL development, and where to find the source code for the patches in Fedora. Cheers, Nick. =================================== PEP: 493 Title: HTTPS verification migration tools for Python 2.7 Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan , Robert Kuska , Marc-Andr? Lemburg Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-May-2015 Python-Version: 2.7.12 Post-History: 06-Jul-2015, 11-Nov-2015, 24-Nov-2015 Abstract ======== PEP 476 updated Python's default handling of HTTPS certificates in client modules to align with certificate handling in web browsers, by validating that the certificates received belonged to the server the client was attempting to contact. The Python 2.7 long term maintenance series was judged to be in scope for this change, with the new behaviour introduced in the Python 2.7.9 maintenance release. This has created a non-trivial barrier to adoption for affected Python 2.7 maintenance releases, so this PEP proposes additional Python 2.7 specific features that allow system administrators and other users to more easily decouple the decision to verify server certificates in HTTPS client modules from the decision to update to newer Python 2.7 maintenance releases. Rationale ========= PEP 476 changed Python's default behaviour to align with expectations established by web browsers in regards to the semantics of HTTPS URLs: starting with Python 2.7.9 and 3.4.3, HTTPS clients in the standard library validate server certificates by default. However, it is also the case that this change *does* cause problems for infrastructure administrators operating private intranets that rely on self-signed certificates, or otherwise encounter problems with the new default certificate verification settings. To manage these kinds of situations, web browsers provide users with "click through" warnings that allow the user to add the server's certificate to the browser's certificate store. Network client tools like ``curl`` and ``wget`` offer options to switch off certificate checking entirely (by way of ``curl --insecure`` and ``wget --no-check-certificate``, respectively). At a different layer of the technology stack, Linux security modules like `SELinux` and `AppArmor`, while enabled by default by distribution vendors, offer relatively straightforward mechanisms for turning them off. At the moment, no such convenient mechanisms exist to disable Python's default certificate checking for a whole process. PEP 476 did attempt to address this question, by covering how to revert to the old settings process wide by monkeypatching the ``ssl`` module to restore the old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed to allow system administrators to disable the feature by default in their Standard Operating Environment definition has been determined to be insufficient in at least some cases. The specific case that led to the initial creation of this PEP is the one where a Linux distributor aims to provide their users with a `smoother migration path `__ than the standard one provided by consuming upstream CPython 2.7 releases directly, but other potential challenges have also been pointed out with updating embedded Python runtimes and other user level installations of Python. Rather than allowing a plethora of mutually incompatibile migration techniques to bloom, this PEP proposes an additional feature to be added to Python 2.7.12 to make it easier to revert a process to the past behaviour of skipping certificate validation in HTTPS client modules. It also provides additional recommendations to redistributors backporting these features to versions of Python prior to Python 2.7.9. These designs are being proposed purely as tools for helping to manage the transition to the new default certificate handling behaviour in the context of Python 2.7. They are not being proposed as new features for Python 3, as it is expected that the vast majority of client applications affected by this problem without the ability to update the application itself will be Python 2 applications. It would likely be desirable for a future version of Python 3 to allow default certificate handling for secure protocols to be configurable on a per-protocol basis, but that question is beyond the scope of this PEP. Alternatives ------------ In the absence of clear upstream guidance and recommendations, commercial redistributors will still make their own design decisions in the interests of their customers. The main approaches available are: * Continuing to rebase on new Python 2.7.x releases, while providing no additional assistance beyond the mechanisms defined in PEP 476 in migrating from unchecked to checked hostnames in standard library HTTPS clients * Gating availability of the changes in default handling of HTTPS connections on upgrading from Python 2 to Python 3 * For Linux distribution vendors, gating availability of the changes in default handling of HTTPS connections on upgrading to a new operating system version * Implementing one or both of the backport suggestions described in this PEP, regardless of the formal status of the PEP Requirements for capability detection ===================================== As the proposals in this PEP aim to facilitate backports to earlier Python versions, the Python version number cannot be used as a reliable means for detecting them. Instead, they are designed to allow the presence or absence of the feature to be determined using the following technique:: python -c "import ssl; ssl._relevant_attribute" This will fail with `AttributeError` (and hence a non-zero return code) if the relevant capability is not available. The marker attributes are prefixed with an underscore to indicate the implementation dependent nature of these capabilities - not all Python distributions will offer them, only those that are providing a multi-stage migration process from the original Python 2.7 HTTPS handling to the new default behaviour. Feature: Configuration API ========================== This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x releases. It consists of a new ``ssl._verify_https_certificates()`` to specify the default handling of HTTPS certificates in standard library client libraries. It is not proposed to forward port this change to Python 3, so Python 3 applications that need to support skipping certificate verification will still need to define their own suitable security context. Feature detection ----------------- The marker attribute on the ``ssl`` module related to this feature is the ``ssl._verify_https_certificates`` function itself. Specification ------------- The ``ssl._verify_https_certificates`` function will work as follows:: def _verify_https_certificates(enable=True): """Verify server HTTPS certificates by default?""" global _create_default_https_context if enable: _create_default_https_context = create_default_context else: _create_default_https_context = _create_unverified_context If called without arguments, or with ``enable`` set to a true value, then standard library client modules will subsequently verify HTTPS certificates by default, otherwise they will skip verification. If called with ``enable`` set to a false value, then standard library client modules will subsequently skip verifying HTTPS certificates by default. Security Considerations ----------------------- The inclusion of this feature will allow security sensitive applications to include the following forward-compatible snippet in their code:: if hasattr(ssl, "_verify_https_certificates"): ssl._verify_https_certificates() Some developers may also choose to opt out of certificate checking using ``ssl._verify_https_certificates(enable=False)``. This doesn't introduce any major new security concerns, as monkeypatching the affected internal APIs was already possible. Feature: environment based configuration ======================================== This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x releases. It consists of a new ``PYTHONHTTPSVERIFY`` environment variable that allows the default verification to be disabled without modifying the application source code (which may not even be available in cases of bytecode-only application distribution) It is not proposed to forward port this change to Python 3, so Python 3 applications that need to support skipping certificate verification will still need to define their own suitable security context. Feature detection ----------------- The marker attribute on the ``ssl`` module related to this feature is: * the ``ssl._https_verify_envvar`` attribute, giving the name of environment variable affecting the default behaviour This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant environment variable name. Specification ------------- Rather than always defaulting to the use of ``ssl.create_default_context``, the ``ssl`` module will be modified to: * read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first imported into a Python process * set the ``ssl._create_default_https_context`` function to be an alias for ``ssl._create_unverified_context`` if this environment variable is present and set to ``'0'`` * otherwise, set the ``ssl._create_default_https_context`` function to be an alias for ``ssl.create_default_context`` as usual Example implementation ---------------------- :: _https_verify_envvar = 'PYTHONHTTPSVERIFY' def _get_https_context_factory(): if not sys.flags.ignore_environment: config_setting = os.environ.get(_https_verify_envvar) if config_setting == '0': return _create_unverified_context return create_default_context _create_default_https_context = _get_https_context_factory() Security Considerations ----------------------- Relative to the behaviour in Python 3.4.3+ and Python 2.7.9->2.7.11, this approach does introduce a new downgrade attack against the default security settings that potentially allows a sufficiently determined attacker to revert Python to the default behaviour used in CPython 2.7.8 and earlier releases. However, such an attack requires the ability to modify the execution environment of a Python process prior to the import of the ``ssl`` module, and any attacker with such access would already be able to modify the behaviour of the underlying OpenSSL implementation. Interaction with Python virtual environments -------------------------------------------- The default setting is read directly from the process environment, and hence works the same way regardless of whether or not the interpreter is being run inside an activated Python virtual environment. Backporting PEP 476 to earlier Python versions ============================================== Some redistributors, most notably Linux distributions, may choose to backport the PEP 476 HTTPS verification changes to modified Python versions based on earlier Python 2 maintenance releases. In these cases, a configuration mechanism is needed that provides: * an opt-in model that allows the decision to enable HTTPS certificate verification to be made independently of the decision to upgrade to the Python version where the feature was first backported * the ability for system administrators to set the default behaviour of Python applications and scripts run directly in the system Python installation * the ability for the redistributor to consider changing the default behaviour of *new* installations at some point in the future without impacting existing installations that have been explicitly configured to skip verifying HTTPS certificates by default As it only affects backports to earlier releases of Python 2.7, this change is not proposed for inclusion in upstream CPython, but rather is offered as guidance to redistributors to reduce the likelihood of multiple mutually incompatible approaches to backporting being adopted. This approach SHOULD NOT be used for any Python installation that advertises itself as providing Python 2.7.9 or later, as most Python users will have the reasonable expectation that all such environments will validate HTTPS certificates by default. Feature detection ----------------- The marker attribute on the ``ssl`` module related to this feature is:: _cert_verification_config = '' This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant configuration file name. Recommended modifications to the Python standard library -------------------------------------------------------- The recommended approach to backporting the PEP 476 modifications to an earlier point release is to implement the following changes relative to the default PEP 476 behaviour implemented in Python 2.7.9+: * modify the ``ssl`` module to read a system wide configuration file when the module is first imported into a Python process * define a platform default behaviour (either verifying or not verifying HTTPS certificates) to be used if this configuration file is not present * support selection between the following three modes of operation: * ensure HTTPS certificate verification is enabled * ensure HTTPS certificate verification is disabled * delegate the decision to the redistributor providing this Python version * set the ``ssl._create_default_https_context`` function to be an alias for either ``ssl.create_default_context`` or ``ssl._create_unverified_context`` based on the given configuration setting. Recommended file location ------------------------- As the PEP authors are not aware of any vendors providing long-term support releases targeting Windows, Mac OS X or \*BSD systems, this approach is currently only specifically defined for Linux system Python installations. The recommended configuration file name on Linux systems is ``/etc/python/cert-verification.cfg``. The ``.cfg`` filename extension is recommended for consistency with the ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library. Recommended file format ----------------------- The configuration file should use a ConfigParser ini-style format with a single section named ``[https]`` containing one required setting ``verify``. The suggested section name is taken from the "https" URL schema passed to affected client APIs. Permitted values for ``verify`` are: * ``enable``: ensure HTTPS certificate verification is enabled by default * ``disable``: ensure HTTPS certificate verification is disabled by default * ``platform_default``: delegate the decision to the redistributor providing this particular Python version If the ``[https]`` section or the ``verify`` setting are missing, or if the ``verify`` setting is set to an unknown value, it should be treated as if the configuration file is not present. Example implementation ---------------------- :: _cert_verification_config = '/etc/python/cert-verification.cfg' def _get_https_context_factory(): # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory) _create_default_https_context = _get_https_context_factory() Security Considerations ----------------------- The specific recommendations for this backporting case are designed to work for privileged, security sensitive processes, even those being run in the following locked down configuration: * run from a locked down administrator controlled directory rather than a normal user directory (preventing ``sys.path[0]`` based privilege escalation attacks) * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based privilege escalation attacks) * run using the ``-s`` switch (preventing user site directory based privilege escalation attacks) * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege escalation attacks) The intent is that the *only* reason HTTPS verification should be getting turned off system wide when using this approach is because: * an end user is running a redistributor provided version of CPython rather than running upstream CPython directly * that redistributor has decided to provide a smoother migration path to verifying HTTPS certificates by default than that being provided by the upstream project * either the redistributor or the local infrastructure administrator has determined that it is appropriate to override the default upstream behaviour (at least for the time being) Using an administrator controlled configuration file rather than an environment variable has the essential feature of providing a smoother migration path, even for applications being run with the ``-E`` switch. Interaction with Python virtual environments -------------------------------------------- This setting is scoped by the interpreter installation and affects all Python processes using that interpreter, regardless of whether or not the interpreter is being run inside an activated Python virtual environment. Origins of this recommendation ------------------------------ This recommendation is based on the backporting approach adopted for Red Hat Enterprise Linux 7.2, as published in the original July 2015 draft of this PEP and described in detail in `this KnowledgeBase article `__. Red Hat's patches implementing this backport for Python 2.7.5 can be found in the `CentOS git repository `__. Backporting this PEP to earlier Python versions =============================================== The configuration file based backport described above is designed to cover backporting the PEP 476 changes to default certificate handling without the additional configuration mechanisms defined in this PEP. If this PEP is accepted, then an additional backporting option becomes available, which is to backport the per-process configuration mechanisms defined in this PEP, without backporting the ability to change the default behaviour of the overall Python installation. Such a backport would differ from the mechanism proposed in this PEP solely in the default behaviour when ``PYTHONHTTPSVERIFY`` was not set at all: it would continue to default to skipping certificate validation. In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and set to anything *other* than ``'0'``, then HTTPS certificate verification should be enabled. Feature detection ----------------- There's no specific attribute indicating that this situation applies. Rather, it is indicated by the ``ssl._verify_https_certificates`` and ``ssl._https_verify_envvar`` attributes being present in a Python version that is nominally older than Python 2.7.9. Specification ------------- Implementing this backport involves backporting the changes in PEP 466, 476 and this PEP, with the following change to the handling of the ``PYTHONHTTPSVERIFY`` environment variable in the ``ssl`` module: * read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first imported into a Python process * set the ``ssl._create_default_https_context`` function to be an alias for ``ssl.create_default_context`` if this environment variable is present and set to any value other than ``'0'`` * otherwise, set the ``ssl._create_default_https_context`` function to be an alias for ``ssl._create_unverified_context`` Example implementation ---------------------- :: _https_verify_envvar = 'PYTHONHTTPSVERIFY' def _get_https_context_factory(): if not sys.flags.ignore_environment: config_setting = os.environ.get(_https_verify_envvar) if config_setting != '0': return create_default_context return _create_unverified_context _create_default_https_context = _get_https_context_factory() def _disable_https_default_verification(): """Skip verification of HTTPS certificates by default""" global _create_default_https_context _create_default_https_context = _create_unverified_context Security Considerations ----------------------- This change would be a strict security upgrade for any Python version that currently defaults to skipping certificate validation in standard library HTTPS clients. The technical trade-offs to be taken into account relate largely to the magnitude of the PEP 466 backport also required rather than to anything security related. Interaction with Python virtual environments -------------------------------------------- The default setting is read directly from the process environment, and hence works the same way regardless of whether or not the interpreter is being run inside an activated Python virtual environment. Recommendation for combined feature backports ============================================= If a redistributor chooses to backport the environment variable based configuration setting from this PEP to a modified Python version that also implements the configuration file based PEP 476 , then the environment variable should take precedence over the system-wide configuration setting. This allows the setting to be changed for a given user or application, regardless of the installation-wide default behaviour. Example implementation ---------------------- :: _https_verify_envvar = 'PYTHONHTTPSVERIFY' _cert_verification_config = '/etc/python/cert-verification.cfg' def _get_https_context_factory(): # Check for an environmental override of the default behaviour if not sys.flags.ignore_environment: config_setting = os.environ.get(_https_verify_envvar) if config_setting is not None: if config_setting == '0': return _create_unverified_context return create_default_context # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory) _create_default_https_context = _get_https_context_factory() -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Nov 27 02:47:33 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 27 Nov 2015 17:47:33 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <22104.2412.671436.247736@turnbull.sk.tsukuba.ac.jp> References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> <20151126121510.32c9a500@anarchist.wooz.org> <22104.2412.671436.247736@turnbull.sk.tsukuba.ac.jp> Message-ID: On 27 November 2015 at 17:42, Stephen J. Turnbull wrote: > Nick Coghlan writes: > > > This is a significant rewrite that switches the PEP to a Standards > > Track PEP proposing two new features for 2.7.12+: an > > "ssl._verify_https_certificates()" configuration function, and a > > "PYTHONHTTPSVERIFY" environment variable (although writing them > > together like that makes me wonder if the latter should now be > > "PYTHONVERIFYHTTPS" instead). > > +1 on the same order of "words" in the function name and the > evnironment variable. I tend to prefer > "ssl._https_verify_certificates" since these aren't "HTTPS > certificates". Ah, yes, that would make sense, so I've just pushed an update that switches the function name order to match the environment variable. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephen at xemacs.org Fri Nov 27 02:42:36 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 27 Nov 2015 16:42:36 +0900 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> <20151126121510.32c9a500@anarchist.wooz.org> Message-ID: <22104.2412.671436.247736@turnbull.sk.tsukuba.ac.jp> Nick Coghlan writes: > This is a significant rewrite that switches the PEP to a Standards > Track PEP proposing two new features for 2.7.12+: an > "ssl._verify_https_certificates()" configuration function, and a > "PYTHONHTTPSVERIFY" environment variable (although writing them > together like that makes me wonder if the latter should now be > "PYTHONVERIFYHTTPS" instead). +1 on the same order of "words" in the function name and the evnironment variable. I tend to prefer "ssl._https_verify_certificates" since these aren't "HTTPS certificates". From cory at lukasa.co.uk Fri Nov 27 03:47:17 2015 From: cory at lukasa.co.uk (Cory Benfield) Date: Fri, 27 Nov 2015 08:47:17 +0000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> <20151126121510.32c9a500@anarchist.wooz.org> Message-ID: <27C511D5-0393-4473-8CAA-272298A2BAE1@lukasa.co.uk> > On 27 Nov 2015, at 06:04, Nick Coghlan wrote: > > Feature: Configuration API > ========================== > > This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x > releases. It consists of a new ``ssl._verify_https_certificates()`` to specify > the default handling of HTTPS certificates in standard library client libraries. > > It is not proposed to forward port this change to Python 3, so Python 3 > applications that need to support skipping certificate verification will still > need to define their own suitable security context. > > Feature detection > ----------------- > > The marker attribute on the ``ssl`` module related to this feature is the > ``ssl._verify_https_certificates`` function itself. > > Specification > ------------- > > The ``ssl._verify_https_certificates`` function will work as follows:: > > def _verify_https_certificates(enable=True): > """Verify server HTTPS certificates by default?""" > global _create_default_https_context > if enable: > _create_default_https_context = create_default_context > else: > _create_default_https_context = _create_unverified_context > > If called without arguments, or with ``enable`` set to a true value, then > standard library client modules will subsequently verify HTTPS > certificates by default, otherwise they will skip verification. Perhaps I missed this, Nick, but what happens if multiple third party libraries apply updates to call this function in incompatible ways? For example, if you depend on libfoo which calls ssl._verify_https_certificates(False) and libbar which calls ssl._verify_https_certificates(True)? Is it?last import wins? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Fri Nov 27 06:38:11 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 27 Nov 2015 21:38:11 +1000 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: <27C511D5-0393-4473-8CAA-272298A2BAE1@lukasa.co.uk> References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> <20151126121510.32c9a500@anarchist.wooz.org> <27C511D5-0393-4473-8CAA-272298A2BAE1@lukasa.co.uk> Message-ID: On 27 November 2015 at 18:47, Cory Benfield wrote: > Perhaps I missed this, Nick, but what happens if multiple third party libraries apply updates to call this function in incompatible ways? For example, if you depend on libfoo which calls ssl._verify_https_certificates(False) and libbar which calls ssl._verify_https_certificates(True)? Is it?last import wins? Last import wins, but libaries shouldn't be mutating process global state as a side effect of import - like the sys module, the ssl module configuration should only be modified (directly or indirectly) from __main__. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From wjun77 at gmail.com Fri Nov 27 03:40:12 2015 From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=) Date: Fri, 27 Nov 2015 16:40:12 +0800 Subject: [Python-Dev] The AttributeError/__getattr__ mechanism Message-ID: Hello everyone: I'm suggesting a modification to the AttributeError/__getattr__ mechanism, see issue25634: http://bugs.python.org/issue25634 I used __getattr__ sometimes, and *descriptor* especially *property* is so widely used. I wonder whether someone had encountered the same problem with me. However, this is a complicated problem including performance issues, and backward compatibility. Thanks for your attention, Jun Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri Nov 27 11:31:24 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 27 Nov 2015 17:31:24 +0100 Subject: [Python-Dev] The AttributeError/__getattr__ mechanism In-Reply-To: References: Message-ID: Hi, As R. David Murray wrote on the issue, the python-ideas mailing list is better suited to start such discussion ;-) Victor 2015-11-27 9:40 GMT+01:00 ?? : > Hello everyone: > > I'm suggesting a modification to the AttributeError/__getattr__ mechanism, > see issue25634: > http://bugs.python.org/issue25634 > > I used __getattr__ sometimes, and descriptor especially property is so > widely used. I wonder whether someone > had encountered the same problem with me. > > However, this is a complicated problem including performance issues, and > backward compatibility. > > Thanks for your attention, > Jun Wang > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > From status at bugs.python.org Fri Nov 27 12:07:11 2015 From: status at bugs.python.org (Python tracker) Date: Fri, 27 Nov 2015 18:07:11 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20151127170711.79B8256755@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2015-11-20 - 2015-11-27) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5263 ( +3) closed 32238 (+57) total 37501 (+60) Open issues with patches: 2312 Issues opened (41) ================== #20923: Explain ConfigParser 'valid section name' and .SECTCRE http://bugs.python.org/issue20923 reopened by terry.reedy #25684: ttk.OptionMenu radiobuttons aren't unique between two instance http://bugs.python.org/issue25684 opened by Bryan.Oakley #25685: Inefficiency with SocketHandler - may send log record message http://bugs.python.org/issue25685 opened by Marvin Greenberg #25687: Error during test case and tearDown http://bugs.python.org/issue25687 opened by bennoleslie #25690: Replacement for unittest.mock.mock_open http://bugs.python.org/issue25690 opened by Niv Ben-David #25694: test.libregrtest not installed http://bugs.python.org/issue25694 opened by Arfrever #25696: "make -j9 install" fails because bininstall target requires th http://bugs.python.org/issue25696 opened by haypo #25698: test regressions introduced with the fix for #22995 http://bugs.python.org/issue25698 opened by doko #25699: Easier way to specify reduced globals for doctest http://bugs.python.org/issue25699 opened by John.Mark.Vandenberg #25701: Document that tp_setattro and tp_setattr are used for deleting http://bugs.python.org/issue25701 opened by serhiy.storchaka #25702: Link Time Optimizations support for GCC and CLANG http://bugs.python.org/issue25702 opened by alecsandru.patrascu #25704: Update the devguide to 3.5 http://bugs.python.org/issue25704 opened by matrixise #25707: Add the close method for ElementTree.iterparse() object http://bugs.python.org/issue25707 opened by serhiy.storchaka #25708: runpy hides traceback for some exceptions http://bugs.python.org/issue25708 opened by sleepycal #25709: Problem with string concatenation and utf-8 cache. http://bugs.python.org/issue25709 opened by ??rp??d K??sa #25710: zipimport is not PEP 3147 or PEP 488 compliant http://bugs.python.org/issue25710 opened by byrnes #25711: Rewrite zipimport from scratch http://bugs.python.org/issue25711 opened by brett.cannon #25713: Setuptools included with 64-bit Windows installer is outdated http://bugs.python.org/issue25713 opened by James.Paget #25714: Consider isinstance(..., numbers.Integral) instead of isinstan http://bugs.python.org/issue25714 opened by posita #25715: Python 3.5.1 installer shows wrong upgrade path http://bugs.python.org/issue25715 opened by steve.dower #25716: typeobject.c call_method & call_maybe can leak references on ' http://bugs.python.org/issue25716 opened by Myron Walker #25717: tempfile.TemporaryFile fails when dir option set to directory http://bugs.python.org/issue25717 opened by Hans Lawrenz #25718: itertools.accumulate __reduce__/__setstate__ bug http://bugs.python.org/issue25718 opened by arigo #25719: Deprecate spitfire benchmark http://bugs.python.org/issue25719 opened by florin.papa #25720: Fix curses module compilation with ncurses6 http://bugs.python.org/issue25720 opened by donmez #25723: ConfigParser should never write broken configurations http://bugs.python.org/issue25723 opened by spaceone #25724: SSLv3 test failure on Ubuntu 16.04 LTS http://bugs.python.org/issue25724 opened by mdeslaur #25726: sys.setprofile / sys.getprofile asymetry http://bugs.python.org/issue25726 opened by stefan #25728: email parser ignores inner multipart boundary when outer messa http://bugs.python.org/issue25728 opened by forest #25729: update pure python datetime.timedelta creation http://bugs.python.org/issue25729 opened by bdkearns #25730: invisible sidebar content with code snippets http://bugs.python.org/issue25730 opened by xwhhsprings #25731: Assigning and deleting __new__ attr on the class does not allo http://bugs.python.org/issue25731 opened by Stanis??aw Skonieczny (Uosiu) #25732: functools.total_ordering does not correctly implement not equa http://bugs.python.org/issue25732 opened by David Seddon #25733: Idle does not handle non-standard compile errors http://bugs.python.org/issue25733 opened by ppperry #25735: math.factorial doc should mention integer return type http://bugs.python.org/issue25735 opened by John.Yeung #25738: http.server doesn't handle RESET CONTENT status correctly http://bugs.python.org/issue25738 opened by spaceone #25739: Add PAYLOAD_TOO_LARGE / URI_TOO_LONG (new name in RFC 7231) http://bugs.python.org/issue25739 opened by spaceone #25740: multiple issues in http.client http://bugs.python.org/issue25740 opened by spaceone #25741: Usual Installation Directory http://bugs.python.org/issue25741 opened by firatozgul #25742: locale.setlocale does not work with unicode strings http://bugs.python.org/issue25742 opened by tierlieb #25743: Clarify exactly what \w matches in UNICODE mode http://bugs.python.org/issue25743 opened by zwol Most recent 15 issues with no replies (15) ========================================== #25742: locale.setlocale does not work with unicode strings http://bugs.python.org/issue25742 #25735: math.factorial doc should mention integer return type http://bugs.python.org/issue25735 #25733: Idle does not handle non-standard compile errors http://bugs.python.org/issue25733 #25726: sys.setprofile / sys.getprofile asymetry http://bugs.python.org/issue25726 #25724: SSLv3 test failure on Ubuntu 16.04 LTS http://bugs.python.org/issue25724 #25720: Fix curses module compilation with ncurses6 http://bugs.python.org/issue25720 #25713: Setuptools included with 64-bit Windows installer is outdated http://bugs.python.org/issue25713 #25708: runpy hides traceback for some exceptions http://bugs.python.org/issue25708 #25687: Error during test case and tearDown http://bugs.python.org/issue25687 #25685: Inefficiency with SocketHandler - may send log record message http://bugs.python.org/issue25685 #25682: __package__ not set to None under pdb in Python 3 http://bugs.python.org/issue25682 #25675: doc for BaseEventLoop.run_in_executor() says its a coroutine, http://bugs.python.org/issue25675 #25661: tokenize.untokenize does not maintain the order of tabbed inde http://bugs.python.org/issue25661 #25658: PyThread assumes pthread_key_t is an integer, which is against http://bugs.python.org/issue25658 #25653: ctypes+callbacks+fork+selinux = crash http://bugs.python.org/issue25653 Most recent 15 issues waiting for review (15) ============================================= #25740: multiple issues in http.client http://bugs.python.org/issue25740 #25739: Add PAYLOAD_TOO_LARGE / URI_TOO_LONG (new name in RFC 7231) http://bugs.python.org/issue25739 #25738: http.server doesn't handle RESET CONTENT status correctly http://bugs.python.org/issue25738 #25729: update pure python datetime.timedelta creation http://bugs.python.org/issue25729 #25724: SSLv3 test failure on Ubuntu 16.04 LTS http://bugs.python.org/issue25724 #25720: Fix curses module compilation with ncurses6 http://bugs.python.org/issue25720 #25719: Deprecate spitfire benchmark http://bugs.python.org/issue25719 #25718: itertools.accumulate __reduce__/__setstate__ bug http://bugs.python.org/issue25718 #25717: tempfile.TemporaryFile fails when dir option set to directory http://bugs.python.org/issue25717 #25716: typeobject.c call_method & call_maybe can leak references on ' http://bugs.python.org/issue25716 #25715: Python 3.5.1 installer shows wrong upgrade path http://bugs.python.org/issue25715 #25709: Problem with string concatenation and utf-8 cache. http://bugs.python.org/issue25709 #25704: Update the devguide to 3.5 http://bugs.python.org/issue25704 #25702: Link Time Optimizations support for GCC and CLANG http://bugs.python.org/issue25702 #25701: Document that tp_setattro and tp_setattr are used for deleting http://bugs.python.org/issue25701 Top 10 most discussed issues (10) ================================= #25709: Problem with string concatenation and utf-8 cache. http://bugs.python.org/issue25709 24 msgs #25717: tempfile.TemporaryFile fails when dir option set to directory http://bugs.python.org/issue25717 13 msgs #25723: ConfigParser should never write broken configurations http://bugs.python.org/issue25723 11 msgs #20923: Explain ConfigParser 'valid section name' and .SECTCRE http://bugs.python.org/issue20923 10 msgs #25660: tabs don't work correctly in python repl http://bugs.python.org/issue25660 10 msgs #25730: invisible sidebar content with code snippets http://bugs.python.org/issue25730 7 msgs #25677: Syntax error caret confused by indentation http://bugs.python.org/issue25677 6 msgs #25694: test.libregrtest not installed http://bugs.python.org/issue25694 6 msgs #25729: update pure python datetime.timedelta creation http://bugs.python.org/issue25729 6 msgs #15348: IDLE - shell becomes unresponsive if debugger windows is close http://bugs.python.org/issue15348 5 msgs Issues closed (51) ================== #4744: asynchat documentation needs to be more precise http://bugs.python.org/issue4744 closed by r.david.murray #7990: xml.etree.cElementTree lacks full dir() on Element http://bugs.python.org/issue7990 closed by serhiy.storchaka #10131: deepcopying an xml.dom.minidom.Document generates an invalid X http://bugs.python.org/issue10131 closed by serhiy.storchaka #10942: xml.etree.ElementTree.tostring returns type bytes, expected ty http://bugs.python.org/issue10942 closed by serhiy.storchaka #13275: Recommend xml.etree for XML processing http://bugs.python.org/issue13275 closed by serhiy.storchaka #14044: IncompleteRead error with urllib2 or urllib.request -- fine wi http://bugs.python.org/issue14044 closed by martin.panter #14988: _elementtree: Raise ImportError when importing of pyexpat fail http://bugs.python.org/issue14988 closed by serhiy.storchaka #19687: Fixes for elementtree integer overflow http://bugs.python.org/issue19687 closed by serhiy.storchaka #21228: Missing enumeration of HTTPResponse Objects methods of urllib. http://bugs.python.org/issue21228 closed by martin.panter #22989: HTTPResponse.msg not as documented http://bugs.python.org/issue22989 closed by martin.panter #23914: pickle fails with SystemError http://bugs.python.org/issue23914 closed by serhiy.storchaka #24097: Use after free in PyObject_GetState http://bugs.python.org/issue24097 closed by serhiy.storchaka #24455: IDLE debugger causes crash if not quitted properly before next http://bugs.python.org/issue24455 closed by terry.reedy #24731: Incorrect assert in str_subtype_new http://bugs.python.org/issue24731 closed by serhiy.storchaka #24858: python3 -m test -ugui -v test_tk gives 3 failures under Debian http://bugs.python.org/issue24858 closed by serhiy.storchaka #25508: LogRecord attributes are not tuple, when logging only dict http://bugs.python.org/issue25508 closed by python-dev #25576: Remove ???Content-Type: application/x-www-form-urlencoded; cha http://bugs.python.org/issue25576 closed by martin.panter #25594: enum instance attribute access possible http://bugs.python.org/issue25594 closed by python-dev #25616: Extract OrderedDict tests into separate file http://bugs.python.org/issue25616 closed by serhiy.storchaka #25622: Enable ctypes.test.test_values.Win_ValuesTestCase for non-Wind http://bugs.python.org/issue25622 closed by martin.panter #25623: Keep the link to Python implementation of OrderedDict http://bugs.python.org/issue25623 closed by serhiy.storchaka #25624: shutil.make_archive makes invalid directory entries http://bugs.python.org/issue25624 closed by serhiy.storchaka #25625: "chdir" Contex manager for pathlib http://bugs.python.org/issue25625 closed by serhiy.storchaka #25626: Gzip fails for file over 2**32 bytes http://bugs.python.org/issue25626 closed by martin.panter #25646: Distutils and Windows SDK 7.1 http://bugs.python.org/issue25646 closed by JGoutin #25656: multiprocessing.dummy: pool.map hangs on empty list http://bugs.python.org/issue25656 closed by ebarry #25657: virtualenv's activate does not update LD_LIBRARY_PATH http://bugs.python.org/issue25657 closed by serge-sans-paille #25663: Make rlcompleter avoid duplicate global names http://bugs.python.org/issue25663 closed by martin.panter #25668: Deadlock in logging caused by a possible race condition with " http://bugs.python.org/issue25668 closed by vinay.sajip #25669: unittest.assertEqual() on un-equal types inheriting from colle http://bugs.python.org/issue25669 closed by michael.foord #25678: int() from a buffer reads past the buffer boundaries http://bugs.python.org/issue25678 closed by serhiy.storchaka #25686: Don't use distutils in test_shutil http://bugs.python.org/issue25686 closed by serhiy.storchaka #25688: File leak in ElementTree.iterparse() http://bugs.python.org/issue25688 closed by serhiy.storchaka #25689: ftplib docs language fix http://bugs.python.org/issue25689 closed by martin.panter #25691: Crash on deleting Element attribute http://bugs.python.org/issue25691 closed by serhiy.storchaka #25692: input function outputs prompt to stderr http://bugs.python.org/issue25692 closed by martin.panter #25693: test_asyncio prints some messages http://bugs.python.org/issue25693 closed by martin.panter #25695: test___all__ and test_support alter execution environment http://bugs.python.org/issue25695 closed by martin.panter #25697: Fix rough alphabetical order in Misc/ACKS http://bugs.python.org/issue25697 closed by serhiy.storchaka #25700: namedtuple documentation http://bugs.python.org/issue25700 closed by Laur Joost #25703: test_sys is failed if standard stream is redirected in 2.7 http://bugs.python.org/issue25703 closed by serhiy.storchaka #25705: Fix regex_compile benchmark crash http://bugs.python.org/issue25705 closed by zach.ware #25706: problems in library/base64.rst http://bugs.python.org/issue25706 closed by serhiy.storchaka #25712: Dead link in the PEP-3147 http://bugs.python.org/issue25712 closed by haypo #25721: Fix pybench always_display error http://bugs.python.org/issue25721 closed by haypo #25722: Lib/ssl.py breaks certificate validation for wildcard domains, http://bugs.python.org/issue25722 closed by christian.heimes #25725: Memory leak on unpickling bogus data http://bugs.python.org/issue25725 closed by serhiy.storchaka #25727: os.startfile implementation for other OS'es besides Windows http://bugs.python.org/issue25727 closed by r.david.murray #25734: fnmatch regular expression can be improved http://bugs.python.org/issue25734 closed by r.david.murray #25736: smtplib can't send 8bit encoded utf-8 message http://bugs.python.org/issue25736 closed by r.david.murray #25737: array is not a Sequence http://bugs.python.org/issue25737 closed by eryksun From lac at openend.se Fri Nov 27 12:45:18 2015 From: lac at openend.se (Laura Creighton) Date: Fri, 27 Nov 2015 18:45:18 +0100 Subject: [Python-Dev] ACTIVITY SUMMARY (2015-11-20 - 2015-11-27) Message-ID: <201511271745.tARHjIl3001173@fido.openend.se> My mailer just barfed trying to read that summary. The problem is that the mail comes out with the encoding: Content-Type: text/plain; charset="us-ascii" but then wants to print 'http://bugs.python.org/issue25709 opened by ??rp??d K??sa' Can we change the encoding to utf-8 ? Laura From ethan at stoneleaf.us Sat Nov 28 10:37:31 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 28 Nov 2015 07:37:31 -0800 Subject: [Python-Dev] Help with a book In-Reply-To: References: Message-ID: <5659CA3B.20509@stoneleaf.us> On 11/25/2015 12:31 PM, Jacob Zimmerman wrote: > I appreciate any help anyone can give to assist in making this book as > great as it can be. I'm a pretty good editor, and have used descriptors quite a bit. I'd be happy to help. -- ~Ethan~ From ethan at stoneleaf.us Sat Nov 28 10:35:33 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 28 Nov 2015 07:35:33 -0800 Subject: [Python-Dev] crash bug in weakref_richcompare? In-Reply-To: References: Message-ID: <5659C9C5.3090204@stoneleaf.us> On 11/26/2015 06:06 PM, Luke Deller wrote: > I have come across some dubious code in Objects/weakrefobject.c which looks like a bug to me, but wanted to run it past others. > > This was discovered from looking at crash dumps from a multithreaded python app (using Python 2.7.9, but the same weakref code exists in 3.5 and hg tip). > > The code that worries me is at the end of the "weakref_richcompare" function: > > return PyObject_RichCompare(PyWeakref_GET_OBJECT(self), > PyWeakref_GET_OBJECT(other), op); > > At this point the code has established that the referents are still alive, and it is trying to compare the referents. However it has not acquired a strong reference to the referents, so I think it is possible for one of them to be deleted half way through this comparison. This can lead to a crash, because PyObject_RichCompare assumes that the PyObject*'s it was passed will remain usable for the duration of the call. > > The crashes I have seen involve data corruption consistent with one of these PyObject's being deleted and the memory reused for something else, eg: > > 00 python27!try_3way_compare+0x15 [objects\object.c @ 712] > 01 python27!try_3way_to_rich_compare+0xb [objects\object.c @ 901] > 02 python27!do_richcmp+0x2c [objects\object.c @ 935] > 03 python27!PyObject_RichCompare+0x99 [objects\object.c @ 982] > 04 python27!weakref_richcompare+0x6a [objects\weakrefobject.c @ 212] > > (In this example v->ob_type was 0x5f637865 which is ASCII "exc_", not a valid pointer at all) > > Other places in weakrefobject.c seem to have a similar weakness, eg in weakref_hash and weakref_repr. > > I have not been able to produce a small test case to exhibit this crash, but from this inspection of the code it looks like a bug - am I understanding this correctly? Luke, go ahead and open an issue on the bug tracker [1] for this; email threads are too easily lost. -- ~Ethan~ [1] bugs.python.org From 15834119171 at 163.com Sat Nov 28 20:34:43 2015 From: 15834119171 at 163.com (maochongzhao) Date: Sun, 29 Nov 2015 09:34:43 +0800 Subject: [Python-Dev] Can not pass configuration on mac os Message-ID: Hi, I am new here wishing to contribute to python. I tried to go through the tutorial for seting up Cpython, while I cannot pass the configuration. My system is Mac OS and I have installed the command tool of Xcode and hg. But on terminal it said that no such file or directory when I type ?./configure --with-pydebug?. What was wrong with my action? How can I fix the issue? This may a little question, but thanks for your help and tutoring! Sincerely, Chongzhao Mao -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Nov 28 20:58:16 2015 From: brett at python.org (Brett Cannon) Date: Sun, 29 Nov 2015 01:58:16 +0000 Subject: [Python-Dev] Can not pass configuration on mac os In-Reply-To: References: Message-ID: The best to get help is the tutor mailing list: https://mail.python.org/mailman/listinfo/tutor . After that you can get help contributing by reading https://docs.python.org/devguide/ . On Sat, 28 Nov 2015, 18:53 maochongzhao <15834119171 at 163.com> wrote: > Hi, > > I am new here wishing to contribute to python. > > I tried to go through the tutorial for seting up Cpython, while I cannot > pass the configuration. My system is Mac OS and I have installed the > command tool of Xcode and hg. But on terminal it said that no such file or > directory when I type ?./configure --with-pydebug?. What was wrong with > my action? How can I fix the issue? > > This may a little question, but thanks for your help and tutoring! > > > Sincerely, > > Chongzhao Mao > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Fri Nov 27 02:30:10 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 27 Nov 2015 16:30:10 +0900 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> <20151126121510.32c9a500@anarchist.wooz.org> Message-ID: Nick Coghlan writes: > This is a significant rewrite that switches the PEP to a Standards > Track PEP proposing two new features for 2.7.12+: an > "ssl._verify_https_certificates()" configuration function, and a > "PYTHONHTTPSVERIFY" environment variable (although writing them > together like that makes me wonder if the latter should now be > "PYTHONVERIFYHTTPS" instead). +1 on the same order of "words" in the function name and the evnironment variable. I tend to prefer "ssl._https_verify_certificates" since these aren't "HTTPS certificates". From fabiofz at gmail.com Mon Nov 30 06:02:12 2015 From: fabiofz at gmail.com (Fabio Zadrozny) Date: Mon, 30 Nov 2015 09:02:12 -0200 Subject: [Python-Dev] Avoiding CPython performance regressions Message-ID: Hi python-dev, I've seen that on and off CPython had attempts to measure benchmarks over time to avoid performance regressions (i.e.: https://speed.python.org), but had nothing concrete so far, so, I ended up creating a hosted service for that (https://www.speedtin.com) and I'd like to help in setting up a structure to run the benchmarks from https://hg.python.org/benchmarks/ and properly upload them to SpeedTin (if CPython devs are Ok with that) -- note that I don't really have server to run the benchmarks, only to host the data (but https://speed.python.org seems to indicate that such a server is available...). There's a sample report at: https://www.speedtin.com/reports/1_CPython27x_Performance_Over_Time/ (it has real data from running using the PyPy benchmarks as I only discovered about the benchmarks from https://hg.python.org/benchmarks/ later on -- also, it doesn't seem to support Python 3 right now, so, it's probably not that useful for the current Python dev, but it does have some nice insight on CPython 2.7.x performance over time). Later on, the idea is being able to compare across different Python implementations which use the same benchmark set... (although that needs other implementations to also post to the data to SpeedTin). Note that uploading the data to SpeedTin should be pretty straightforward (by using https://github.com/fabioz/pyspeedtin, so, the main issue would be setting up o machine to run the benchmarks). Best Regards, Fabio -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Mon Nov 30 08:52:39 2015 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 30 Nov 2015 08:52:39 -0500 Subject: [Python-Dev] Avoiding CPython performance regressions In-Reply-To: References: Message-ID: <20151130135240.3E341251104@webabinitio.net> On Mon, 30 Nov 2015 09:02:12 -0200, Fabio Zadrozny wrote: > Note that uploading the data to SpeedTin should be pretty straightforward > (by using https://github.com/fabioz/pyspeedtin, so, the main issue would be > setting up o machine to run the benchmarks). Thanks, but Zach almost has this working using codespeed (he's still waiting on a review from infrastructure, I think). The server was not in fact running; a large part of what Zach did was to get that server set up. I don't know what it would take to export the data to another consumer, but if you want to work on that I'm guessing there would be no objection. And I'm sure there would be no objection if you want to get involved in maintaining the benchmark server! There's also an Intel project posted about here recently that checks individual benchmarks for performance regressions and posts the results to python-checkins. --David From sad2project at gmail.com Mon Nov 30 12:23:51 2015 From: sad2project at gmail.com (Jacob Zimmerman) Date: Mon, 30 Nov 2015 11:23:51 -0600 Subject: [Python-Dev] Help with a book In-Reply-To: References: Message-ID: Thanks to everyone who has offered to look at and edit my book. I have enough help now, so I won't be taking anymore requests. On Wed, Nov 25, 2015 at 2:31 PM, Jacob Zimmerman wrote: > Hello Python Developers! > > I'm writing a book about descriptors, and I'm hoping to make it a > comprehensive guide, covering just about everything there is to know about > descriptors, including many tips to get past certain knowledge barriers and > how to avoid certain pitfalls. I'm hoping for it to become the definitive > guide for those who want to really understand descriptors. Raymond's > Descriptor HowTo Guide is great, but focused. I want to expand on what that > guide teaches. > > Anyway, I am currently reading through my initial rough draft and making > corrections for a revised draft. Once I'm done with that, I'd like a few > volunteers to read through it. First, to find spelling, grammatical, or > consistency mistakes, though those should be few in number, but also to > give input as to whether something seems too repetitive, if some more > information should be given in a spot, and if I'm even blatantly wrong > about anything (hopefully not). > > Anyone who helps will be able to receive a free ebook of the final copy as > well as mention in the Special Thanks. > > I appreciate any help anyone can give to assist in making this book as > great as it can be. > > Thanks, > Jacob Zimmerman > > Twitter: @jacobz_20 > Blog: http://programmingideaswithjake.wordpress.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.c.stewart at intel.com Mon Nov 30 12:33:05 2015 From: david.c.stewart at intel.com (Stewart, David C) Date: Mon, 30 Nov 2015 17:33:05 +0000 Subject: [Python-Dev] Avoiding CPython performance regressions In-Reply-To: <20151130135240.3E341251104@webabinitio.net> References: <20151130135240.3E341251104@webabinitio.net> Message-ID: <5A37912B-B69D-452D-847E-EACBF8E5F3E4@intel.com> On 11/30/15, 5:52 AM, "Python-Dev on behalf of R. David Murray" wrote: > >There's also an Intel project posted about here recently that checks >individual benchmarks for performance regressions and posts the results >to python-checkins. The description of the project is at https://01.org/lp - Python results are indeed sent daily to python-checkins. (No results for Nov 30 and Dec 1 due to Romania National Day holiday!) There is also a graphic dashboard at http://languagesperformance.intel.com/ Dave From barry at python.org Mon Nov 30 15:32:30 2015 From: barry at python.org (Barry Warsaw) Date: Mon, 30 Nov 2015 15:32:30 -0500 Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance) In-Reply-To: References: <20151123210557.57b7e732@limelight.wooz.org> <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com> <20151125145714.2a02651d@limelight.wooz.org> <20151126121510.32c9a500@anarchist.wooz.org> Message-ID: <20151130153230.2117981c@anarchist.wooz.org> On Nov 27, 2015, at 04:04 PM, Nick Coghlan wrote: >New draft pushed: https://hg.python.org/peps/rev/f602a47ea795 > >This is a significant rewrite that switches the PEP to a Standards Track PEP >proposing two new features for 2.7.12+: an "ssl._verify_https_certificates()" >configuration function, and a "PYTHONHTTPSVERIFY" environment variable >(although writing them together like that makes me wonder if the latter >should now be "PYTHONVERIFYHTTPS" instead). Thanks for this, and +1 on Stephen's suggested name change (which you've already pushed). Two comments: the PEP still describes the configuration file implementation. Is this slated for 2.7.12 also? If not, should it just be dropped from the PEP? I'd mildly prefer no default value for `enable` in _https_verify_certificates(). I'd have preferred a keyword-only argument, but of course this is Python 2. Instead, I'd like to force passing True or False (and document using `enable=True` or `enable=False`) and not rely on a default argument. But I'm only +0 on that detail. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: