From ja.py at farowl.co.uk Sat Mar 1 00:03:27 2014 From: ja.py at farowl.co.uk (Jeff Allen) Date: Fri, 28 Feb 2014 23:03:27 +0000 Subject: [Python-Dev] Python 3.3.4150 In-Reply-To: <9DB8A1E75492864D8C3016CE2126AADD28BDD1C8@CREXHTMAIL02.us.aegon.com> References: <9DB8A1E75492864D8C3016CE2126AADD28BDD1C8@CREXHTMAIL02.us.aegon.com> Message-ID: <531115BF.2070001@farowl.co.uk> Jason: I get that too, now I try it. The place to report bugs is: http://bugs.python.org/ However, please take a look at http://bugs.python.org/issue14512 before you file a new one. Jeff Allen On 28/02/2014 17:05, Burgoon, Jason wrote: > > Good day Python Dev Team -- > > One of our users has reported the following: > > I have installed the given msi on 64 bit as per the install > instructions document. > > One of the shortcuts 'Start Menu\Programs\Python 3.3\ Module Docs' is > not getting launched. When I launch this shortcut, it is not opening > any window. > > I have tried with admin user and non-admin user. > > Is this expected behavior? > > Please adviseand thanks for your help. > > Jason > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Mar 1 01:46:17 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 28 Feb 2014 19:46:17 -0500 Subject: [Python-Dev] Python 3.3.4150 In-Reply-To: <9DB8A1E75492864D8C3016CE2126AADD28BDD1C8@CREXHTMAIL02.us.aegon.com> References: <9DB8A1E75492864D8C3016CE2126AADD28BDD1C8@CREXHTMAIL02.us.aegon.com> Message-ID: On 2/28/2014 12:05 PM, Burgoon, Jason wrote: > One of the shortcuts ?Start Menu\Programs\Python 3.3\ Module Docs? is > not getting launched. When I launch this shortcut, it is not opening any > window. > I have tried with admin user and non-admin user. > Is this expected behavior? No, it is a bug that I already reported. Usage questions and problem reports like this should better be posted to python-list. Also, plain text is preferred to html. -- Terry Jan Reedy From emm.odeke at gmail.com Sat Mar 1 02:03:24 2014 From: emm.odeke at gmail.com (emm.odeke at gmail.com) Date: Fri, 28 Feb 2014 18:03:24 -0700 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: <20140228174917.48ec22da@anarchist.wooz.org> References: <53104F2B.5080200@hastings.org> <20140228101948.29966a30@anarchist.wooz.org> <20140228174917.48ec22da@anarchist.wooz.org> Message-ID: <20140301010324.6029459.39353.768@gmail.com> Sent from my BlackBerry 10 smartphone on the Koodo network. ? Original Message ? From: Barry Warsaw Sent: Friday, February 28, 2014 3:50 PM To: python-dev at python.org Subject: Re: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc On Mar 01, 2014, at 08:15 AM, Nick Coghlan wrote: >It *is* playing refcounting games - it's decrefing the existing target >while stealing a reference to the new target, just like the existing >SET_ITEM macros and somewhat like Py_CLEAR (although in that case, it's >more obvious that we will never incref NULL). Okay, but "setting the reference" isn't one of them, which is what I read when I see Py_SETREF. ;) >The whole point of this macro is to take an *existing* reference and safely >*overwrite* another existing reference, exactly as the SET_ITEM macros do. > >That actually gives me an idea that wasn't on Serhiy's original list: >Py_SET_ATTR(target, value). That does seem better. -Barry From v+python at g.nevcal.com Sat Mar 1 02:24:22 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 28 Feb 2014 17:24:22 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> <530FF075.8020303@g.nevcal.com> <53103D11.9030204@g.nevcal.com> <53105743.5080407@g.nevcal.com> Message-ID: <531136C6.1050108@g.nevcal.com> On 2/28/2014 4:51 AM, Nick Coghlan wrote: > The preferred notation in the PEP most resembles the existing lambda > use case, with "except" instead of "lambda", an exception handling > spec instead of an argument list and an additional leading expression: > > (expr except Exception: default) > > Lots of people don't like the lambda notation though, so it isn't > necessarily a particularly compelling parallel to use. Thank you for explaining why I find the above notation awkward. ": as introducing a suite" never bothered me, because, as you've now enumerated, there are other uses of :. But the lambda syntax parallel is what I don't like about it... I find the lambda syntax hard to read. > By contrast, > it's rare to hear any objections to the {key:value} dict display > syntax. Hence the proposed tweak to the syntax to define an "exception > handler expression" syntax that is analogous to a dict display rather > than a lambda expression: > > expr except (Exception: default) > > However, I have realised that there*is* a major downside to that > notation, which is that it lacks the connotations of lazy evaluation > associated with lambda expressions, whereas the default result of an > except expression won't be evaluated at all if the exception isn't > thrown. You are overlooking that the keyword except provides exactly the connotation of lazy evaluation, so if this is your only reason for preferring the lambda syntax, you just erased it :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Mar 1 02:40:47 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 1 Mar 2014 12:40:47 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <531136C6.1050108@g.nevcal.com> References: <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> <530FF075.8020303@g.nevcal.com> <53103D11.9030204@g.nevcal.com> <53105743.5080407@g.nevcal.com> <531136C6.1050108@g.nevcal.com> Message-ID: On Sat, Mar 1, 2014 at 12:24 PM, Glenn Linderman wrote: > You are overlooking that the keyword except provides exactly the connotation > of lazy evaluation, so if this is your only reason for preferring the lambda > syntax, you just erased it :) Statements are always executed sequentially. That's not "lazy evaluation", that's just the way statements are :) ChrisA From g.rodola at gmail.com Sat Mar 1 19:30:12 2014 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sat, 1 Mar 2014 19:30:12 +0100 Subject: [Python-Dev] Removing doc from pythonhosted.org Message-ID: I'm not sure whether this is the right mailing list where to post this. However, it seems the pypi UI currently provides a way to upload doc at the bottom of the page https://pypi.python.org/pypi?%3Aaction=pkg_edit&name=PROJECT_NAME but there's currently no way to remove it: https://bitbucket.org/pypa/pypi/issue/24/allow-deleting-project-documentation-from http://sourceforge.net/p/pypi/support-requests/294/ https://github.com/litl/rauth/issues/81 The only workaround as of http://stackoverflow.com/questions/6521931/how-to-delete-documentation-from-pypiappears to be uploading a new .zip file with a redirect. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at sebastiankraft.net Sat Mar 1 20:57:37 2014 From: mail at sebastiankraft.net (Sebastian Kraft) Date: Sat, 01 Mar 2014 20:57:37 +0100 Subject: [Python-Dev] Wave module support for floating point data Message-ID: <53123BB1.7030903@sebastiankraft.net> Hi everybody, more than a year ago I have submitted a patch to enhance the Wave module with read/write support for floating point data. http://bugs.python.org/issue16525 Up till now this patch has not been applied nor did I get feedback if anything needs to be enhanced or changed before it can be committed. I have never been in contact with your development process and Python core developers, so please tell me what I should improve... Regards, Sebastian From tjreedy at udel.edu Sat Mar 1 21:08:00 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 01 Mar 2014 15:08:00 -0500 Subject: [Python-Dev] Wave module support for floating point data In-Reply-To: <53123BB1.7030903@sebastiankraft.net> References: <53123BB1.7030903@sebastiankraft.net> Message-ID: On 3/1/2014 2:57 PM, Sebastian Kraft wrote: > Hi everybody, > > more than a year ago I have submitted a patch to enhance the Wave module > with read/write support for floating point data. > > http://bugs.python.org/issue16525 > > Up till now this patch has not been applied nor did I get feedback if > anything needs to be enhanced or changed before it can be committed. > I have never been in contact with your development process and Python > core developers, so please tell me what I should improve... Please subscribe to core-mentorship list and post your question there. -- Terry Jan Reedy From solipsis at pitrou.net Sat Mar 1 21:25:43 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 1 Mar 2014 21:25:43 +0100 Subject: [Python-Dev] Wave module support for floating point data References: <53123BB1.7030903@sebastiankraft.net> Message-ID: <20140301212543.6cac867c@fsol> On Sat, 01 Mar 2014 15:08:00 -0500 Terry Reedy wrote: > On 3/1/2014 2:57 PM, Sebastian Kraft wrote: > > Hi everybody, > > > > more than a year ago I have submitted a patch to enhance the Wave module > > with read/write support for floating point data. > > > > http://bugs.python.org/issue16525 > > > > Up till now this patch has not been applied nor did I get feedback if > > anything needs to be enhanced or changed before it can be committed. > > I have never been in contact with your development process and Python > > core developers, so please tell me what I should improve... > > Please subscribe to core-mentorship list and post your question there. I don't understand this response. You seem to be assuming that Sebastian is asking for guidance, but he's simply telling us about a patch that hasn't received any review yet, despite having been posted one year ago. It's not obvious he has been doing something wrong that he needs to be taught about. Regards Antoine. From tjreedy at udel.edu Sun Mar 2 00:04:09 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 01 Mar 2014 18:04:09 -0500 Subject: [Python-Dev] Wave module support for floating point data In-Reply-To: <20140301212543.6cac867c@fsol> References: <53123BB1.7030903@sebastiankraft.net> <20140301212543.6cac867c@fsol> Message-ID: On 3/1/2014 3:25 PM, Antoine Pitrou wrote: > On Sat, 01 Mar 2014 15:08:00 -0500 > Terry Reedy wrote: >> On 3/1/2014 2:57 PM, Sebastian Kraft wrote: >>> Hi everybody, >>> >>> more than a year ago I have submitted a patch to enhance the Wave module >>> with read/write support for floating point data. >>> >>> http://bugs.python.org/issue16525 >>> >>> Up till now this patch has not been applied nor did I get feedback if >>> anything needs to be enhanced or changed before it can be committed. >>> I have never been in contact with your development process and Python >>> core developers, so please tell me what I should improve... >> >> Please subscribe to core-mentorship list and post your question there. > > I don't understand this response. You seem to be assuming that > Sebastian is asking for guidance, I am reading what he posted, which ended with "please tell me what I should improve...". This sort of question-request is routinely posted, in much the same words, on core-mentorship, and routinely gets a response there. > but he's simply telling us about a patch that hasn't received > any review yet, despite having been posted one year ago. If that were all he said (and it is not) it would not be too useful. There are about 4000 open issues. About half have the 'patch' keyword set, and probably more have patches without the keyword. About 700 with the keywork have seen no activity for a year. Some fraction of those have never received a review. I would guess at least 100. In this sense, there is, unfortunately in my opinion, nothing too special about this issue or patch. If you are really interested in this subset of issues, someone should do a custom search against the database. Issues that are open, have a patch (a file name ending in .diff or .patch) posted more than a year ago, and have no subsequent responses from a core developer, would be a start. Perhaps we should add a 'reviewed' field to the table of uploads, which would be automatically marked True when is a completed Rietveld response. Perhaps there should be a way to connect review messages, as opposed to mere comment messages, to the patch they review. We should think about this in the process re-design Nick has planned. > It's not obvious he has been doing something wrong that > he needs to be taught about. The same could initially be said of all the similar posts on core-mentorship. But please change 'wrong' to 'incomplete or something. Most patches get revised and augmented without being 'wrong'. This is especialy true of patches from people 'have never been in contact with [our] development process'. What makes Sebastian's request worth extra attention is the extra information that he, the author of the neglected patch, is still around and desirous of discussing, editing, and augmenting the patch as necessary. -- Terry Jan Reedy From rosuav at gmail.com Sun Mar 2 01:11:01 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 2 Mar 2014 11:11:01 +1100 Subject: [Python-Dev] "Five reviews to get yours reviewed"? Message-ID: Way back in 2012, Martin L?wis declared a standing offer on this list to get issue patches reviewed: review five issues and he'll review one of yours. [1] Is that offer still around? Have any other devs made any similar offer? I have a couple of patches outstanding, notably issue 20249 [2], which is a small change, has a patch, and has no activity or nosying since its creation. More importantly, if there is such an offer, it'd be great to mention it somewhere, so people can know what they can do to move an issue forward. (And preferably with a link somewhere to what it means to review a patch - what it takes to make a useful and helpful review, which I'm not entirely sure of at the moment.) If there's not, is it something that could be considered? I'd love to see some downward movement on the Open Issues figure, but am not really sure what I can personally do to help. ChrisA [1] https://mail.python.org/pipermail/python-dev/2012-October/122157.html [2] http://bugs.python.org/issue20249 From rosuav at gmail.com Sun Mar 2 01:20:29 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 2 Mar 2014 11:20:29 +1100 Subject: [Python-Dev] "Five reviews to get yours reviewed"? In-Reply-To: References: Message-ID: On Sun, Mar 2, 2014 at 11:11 AM, Chris Angelico wrote: > I have a couple of patches outstanding, notably issue 20249 [2], which > is a small change, has a patch, and has no activity or nosying since > its creation. And Benjamin Peterson has just looked into this one and committed it, not three minutes after I posted. Looks like that crossed in the ether :) Thanks! I'm still willing to help out with reviewing, though! ChrisA From solipsis at pitrou.net Sun Mar 2 01:25:33 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 2 Mar 2014 01:25:33 +0100 Subject: [Python-Dev] "Five reviews to get yours reviewed"? References: Message-ID: <20140302012533.74073b22@fsol> On Sun, 2 Mar 2014 11:11:01 +1100 Chris Angelico wrote: > More importantly, if there is such an offer, it'd be great to mention > it somewhere, so people can know what they can do to move an issue > forward. (And preferably with a link somewhere to what it means to > review a patch - what it takes to make a useful and helpful review, > which I'm not entirely sure of at the moment.) If there's not, is it > something that could be considered? I'd love to see some downward > movement on the Open Issues figure, but am not really sure what I can > personally do to help. It's such an unbalanced offer that it's understandable why it never worked. "One review against another" would be reasonable. That said, it's not a mere issue of time. It's also that occasional contributors may not have (or may not feel they have) the required expertise to review other people's patches. Regards Antoine. From rosuav at gmail.com Sun Mar 2 02:00:03 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 2 Mar 2014 12:00:03 +1100 Subject: [Python-Dev] "Five reviews to get yours reviewed"? In-Reply-To: <20140302012533.74073b22@fsol> References: <20140302012533.74073b22@fsol> Message-ID: On Sun, Mar 2, 2014 at 11:25 AM, Antoine Pitrou wrote: > On Sun, 2 Mar 2014 11:11:01 +1100 > Chris Angelico wrote: >> More importantly, if there is such an offer, it'd be great to mention >> it somewhere, so people can know what they can do to move an issue >> forward. (And preferably with a link somewhere to what it means to >> review a patch - what it takes to make a useful and helpful review, >> which I'm not entirely sure of at the moment.) If there's not, is it >> something that could be considered? I'd love to see some downward >> movement on the Open Issues figure, but am not really sure what I can >> personally do to help. > > It's such an unbalanced offer that it's understandable why it never > worked. "One review against another" would be reasonable. > > That said, it's not a mere issue of time. It's also that occasional > contributors may not have (or may not feel they have) the required > expertise to review other people's patches. Since there's a skill level difference, I can understand that I'd have to do more work than I'm asking someone else to do. But it's the other part that's more important. How would someone know whether or not they're capable of making useful reviews? Are there guidelines somewhere? Obviously you have to be able to apply the patch, compile (if appropriate), and probably run the test suite, but beyond that, what does it take to review? (The buildbots have the intelligence to do that.) ChrisA From tjreedy at udel.edu Sun Mar 2 06:07:15 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 02 Mar 2014 00:07:15 -0500 Subject: [Python-Dev] "Five reviews to get yours reviewed"? In-Reply-To: References: Message-ID: On 3/1/2014 7:11 PM, Chris Angelico wrote: > Way back in 2012, Martin L?wis declared a standing offer on this list > to get issue patches reviewed: review five issues and he'll review one > of yours. As I remember, he set a pretty low bar for 'review', lowing that I think you are thinking. > I have a couple of patches outstanding, notably issue 20249 [2], which > is a small change, has a patch, and has no activity or nosying since > its creation. And the other? > More importantly, if there is such an offer, it'd be great to mention > it somewhere, so people can know what they can do to move an issue > forward. (And preferably with a link somewhere to what it means to The question has been asked on core-mentorship list. I have considered making an offer, but haven't yet. > review a patch - what it takes to make a useful and helpful review, > which I'm not entirely sure of at the moment.) If there's not, is it > something that could be considered? I'd love to see some downward > movement on the Open Issues figure, but am not really sure what I can > personally do to help. You are active on python-ideas, so build on that. There are 1551 open enhancement issues. Some have no response. Some should be rejected (for instance, if a couple of core devs have given negative responses, and none positive). Some should probably be closed, but possibly discussed on python-ideas. You could open either suggest that the OP post on python-ideas or open a discussion yourself. Many should have been discussed on python-ideas first, to garner support, but may have been posted before it existed, or at least before it was very well known. Some might be obsolete given what has otherwise been added, or by changes from py2 to py3. Any that are left open should be marked for 3.5. -- Terry Jan Reedy From rosuav at gmail.com Sun Mar 2 07:51:11 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 2 Mar 2014 17:51:11 +1100 Subject: [Python-Dev] "Five reviews to get yours reviewed"? In-Reply-To: References: Message-ID: On Sun, Mar 2, 2014 at 4:07 PM, Terry Reedy wrote: > On 3/1/2014 7:11 PM, Chris Angelico wrote: >> I have a couple of patches outstanding, notably issue 20249 [2], which >> is a small change, has a patch, and has no activity or nosying since >> its creation. > > And the other? http://bugs.python.org/issue19494 has a patch that I uploaded, but it's more accurately someone else's patch and I just made a slight tweak to it. http://bugs.python.org/issue20729 is an issue that I opened, and there's a patch at the issue, but I didn't write the patch. Technically, neither really counts, but I was checking over the "Followed by you" issues list and saw that several had patches. >> I'd love to see some downward >> movement on the Open Issues figure, but am not really sure what I can >> personally do to help. > > You are active on python-ideas, so build on that. There are 1551 open > enhancement issues. ... > You could open either suggest that the OP post on python-ideas > or open a discussion yourself. Okay! I'll poke around at some issues tonight and see what I can find. *thumb up* Thanks for the pointer. ChrisA From tjreedy at udel.edu Sun Mar 2 09:48:52 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 02 Mar 2014 03:48:52 -0500 Subject: [Python-Dev] "Five reviews to get yours reviewed"? In-Reply-To: References: Message-ID: On 3/2/2014 1:51 AM, Chris Angelico wrote: > On Sun, Mar 2, 2014 at 4:07 PM, Terry Reedy wrote: >> On 3/1/2014 7:11 PM, Chris Angelico wrote: >>> I have a couple of patches outstanding, notably issue 20249 [2], which >>> is a small change, has a patch, and has no activity or nosying since >>> its creation. >> >> And the other? > > http://bugs.python.org/issue19494 has a patch that I uploaded, but > it's more accurately someone else's patch and I just made a slight > tweak to it. The line numbers in your patch do not match the line numbers in the 3.4 file. Did you prepare against 3.3? The base issue here is a policy question about accommodating violations of the standard. The main maintainer of the urllib.requests module is Senthil. I would not decide the policy question. > http://bugs.python.org/issue20729 is an issue that I opened, and > there's a patch at the issue, but I didn't write the patch. I think Serhiy's patch is 'conservative', so I could look at it and see if I agree that it is the right minimal change. > Technically, neither really counts, Martin's offer was to review a patch that one wanted reviewed, not necessarily one that one wrote. -- Terry Jan Reedy From rosuav at gmail.com Sun Mar 2 10:39:05 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 2 Mar 2014 20:39:05 +1100 Subject: [Python-Dev] "Five reviews to get yours reviewed"? In-Reply-To: References: Message-ID: On Sun, Mar 2, 2014 at 7:48 PM, Terry Reedy wrote: > On 3/2/2014 1:51 AM, Chris Angelico wrote: >> >> On Sun, Mar 2, 2014 at 4:07 PM, Terry Reedy wrote: >>> >>> On 3/1/2014 7:11 PM, Chris Angelico wrote: >>>> >>>> I have a couple of patches outstanding, notably issue 20249 [2], which >>>> is a small change, has a patch, and has no activity or nosying since >>>> its creation. >>> >>> >>> And the other? >> >> >> http://bugs.python.org/issue19494 has a patch that I uploaded, but >> it's more accurately someone else's patch and I just made a slight >> tweak to it. > > > The line numbers in your patch do not match the line numbers in the 3.4 > file. Did you prepare against 3.3? > Hmm. I definitely made sure it worked on 3.4, but IIRC the patch was manually fudged from the 2.7 patch, so possibly I didn't get everything perfect. But it does apply without errors: rosuav at sikorsky:~/cpython$ hg impor aaa.patch applying aaa.patch (Yeah, I'm not imaginative with temporary file names. So sue me. :) ChrisA From xdegaye at gmail.com Sun Mar 2 11:25:38 2014 From: xdegaye at gmail.com (Xavier de Gaye) Date: Sun, 02 Mar 2014 11:25:38 +0100 Subject: [Python-Dev] Wave module support for floating point data In-Reply-To: References: <53123BB1.7030903@sebastiankraft.net> <20140301212543.6cac867c@fsol> Message-ID: <53130722.8040708@gmail.com> On 03/02/2014 12:04 AM, Terry Reedy wrote: > On 3/1/2014 3:25 PM, Antoine Pitrou wrote: >> On Sat, 01 Mar 2014 15:08:00 -0500 >> Terry Reedy wrote: >>> On 3/1/2014 2:57 PM, Sebastian Kraft wrote: >>>> Hi everybody, >>>> >>>> more than a year ago I have submitted a patch to enhance the Wave module >>>> with read/write support for floating point data. >>>> >>>> http://bugs.python.org/issue16525 >>>> >>>> Up till now this patch has not been applied nor did I get feedback if >>>> anything needs to be enhanced or changed before it can be committed. >>>> I have never been in contact with your development process and Python >>>> core developers, so please tell me what I should improve... >>> >>> Please subscribe to core-mentorship list and post your question there. >> >> I don't understand this response. You seem to be assuming that >> Sebastian is asking for guidance, > > I am reading what he posted, which ended with "please tell me what I should improve...". > > This sort of question-request is routinely posted, in much the same words, on core-mentorship, and routinely gets a response there. The core-mentorship current archive is only available to the list members as stated at the core-mentorship web site at http://pythonmentors.com/. Curious to know why. Xavier From mail at timgolden.me.uk Sun Mar 2 11:57:38 2014 From: mail at timgolden.me.uk (Tim Golden) Date: Sun, 02 Mar 2014 10:57:38 +0000 Subject: [Python-Dev] Wave module support for floating point data In-Reply-To: <53130722.8040708@gmail.com> References: <53123BB1.7030903@sebastiankraft.net> <20140301212543.6cac867c@fsol> <53130722.8040708@gmail.com> Message-ID: <53130EA2.90201@timgolden.me.uk> On 02/03/2014 10:25, Xavier de Gaye wrote: > The core-mentorship current archive is only available to the list > members as stated at the core-mentorship web site at > http://pythonmentors.com/. Curious to know why. The thinking behind c-m is to lower the entry bar for contribution. To that end, people can ask questions which might, to the more knowledgeable, appear naive. ("You don't know what Mercurial is?" "You've never used make?") If the archives were public then less-experienced questioners might shrink from exposing their ignorance and would ultimately not engage with Python development. Another "rule" is that you should not quote c-m posts literally outside the c-m list. You can restate the question in general terms to, eg, get input from python-dev or python-committers. Advice *from* the committers on the list can be quoted elsewhere. Note that the archives are open to all subscribers, and subscription is open to anyone. So it's not particularly secret, just a low barrier to remind people to respect the possible inexperience of posters. TJG From mail at sebastiankraft.net Sun Mar 2 10:02:20 2014 From: mail at sebastiankraft.net (Sebastian Kraft) Date: Sun, 02 Mar 2014 10:02:20 +0100 Subject: [Python-Dev] Wave module support for floating point data In-Reply-To: <20140301212543.6cac867c@fsol> References: <53123BB1.7030903@sebastiankraft.net> <20140301212543.6cac867c@fsol> Message-ID: <5312F39C.9040101@sebastiankraft.net> Am 01.03.2014 21:25, schrieb Antoine Pitrou: > On Sat, 01 Mar 2014 15:08:00 -0500 > Terry Reedy wrote: >> On 3/1/2014 2:57 PM, Sebastian Kraft wrote: >>> Hi everybody, >>> >>> more than a year ago I have submitted a patch to enhance the Wave module >>> with read/write support for floating point data. >>> >>> http://bugs.python.org/issue16525 >>> >>> Up till now this patch has not been applied nor did I get feedback if >>> anything needs to be enhanced or changed before it can be committed. >>> I have never been in contact with your development process and Python >>> core developers, so please tell me what I should improve... >> >> Please subscribe to core-mentorship list and post your question there. > > I don't understand this response. You seem to be assuming that It seems there is some confusion about how to handle this... Anyway I looked at the patch again and after the recent changes towards version 3.4 the patch cannot be applied anymore. Taking into account that there was no interest in this area for the last years, I will not take the time to rework the patch. Anyway please consider to commit the new documentation patch I have uploaded to the issue. http://bugs.python.org/issue16525 It is really necessary to clarify the suppoprted sample formats in the documentation. Sebastian From solipsis at pitrou.net Sun Mar 2 17:11:28 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 2 Mar 2014 17:11:28 +0100 Subject: [Python-Dev] Wave module support for floating point data References: <53123BB1.7030903@sebastiankraft.net> <20140301212543.6cac867c@fsol> Message-ID: <20140302171128.70231604@fsol> On Sat, 01 Mar 2014 18:04:09 -0500 Terry Reedy wrote: > >> > >> Please subscribe to core-mentorship list and post your question there. > > > > I don't understand this response. You seem to be assuming that > > Sebastian is asking for guidance, > > I am reading what he posted, which ended with "please tell me what I > should improve...". > > This sort of question-request is routinely posted, in much the same > words, on core-mentorship, and routinely gets a response there. I'm sure people who answer questions on core-mentorship also hang out on python-dev, so I don't understand why it wouldn't get a response *here*. Regardless, the main problem with stale patches like Sebastian's is that there isn't a core developer interested in reviewing them (or with the required time to do so). Asking those people to post on core-mentorship is a distraction: it won't make core developers magically more available. Regards Antoine. From brett at python.org Sun Mar 2 18:17:03 2014 From: brett at python.org (Brett Cannon) Date: Sun, 2 Mar 2014 12:17:03 -0500 Subject: [Python-Dev] Removing doc from pythonhosted.org In-Reply-To: References: Message-ID: Best place for PyPI-related things is distutils-sig at python.org. But for bug reports related to PyPI, file them at https://bitbucket.org/pypa/pypi/issues(it's actually linked off of PyPI). On Sat, Mar 1, 2014 at 1:30 PM, Giampaolo Rodola' wrote: > I'm not sure whether this is the right mailing list where to post this. > However, it seems the pypi UI currently provides a way to upload doc at > the bottom of the page > https://pypi.python.org/pypi?%3Aaction=pkg_edit&name=PROJECT_NAME but > there's currently no way to remove it: > > https://bitbucket.org/pypa/pypi/issue/24/allow-deleting-project-documentation-from > http://sourceforge.net/p/pypi/support-requests/294/ > https://github.com/litl/rauth/issues/81 > The only workaround as of > http://stackoverflow.com/questions/6521931/how-to-delete-documentation-from-pypiappears to be uploading a new .zip file with a redirect. > > -- > Giampaolo - http://grodola.blogspot.com > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Mar 2 21:01:01 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 02 Mar 2014 15:01:01 -0500 Subject: [Python-Dev] unicode_string future, str -> basestring, fix or feature Message-ID: Suppose a 2.7 standard library function is documented as taking a 'string' argument, such as these examples from the turtle module. pencolor(colorstring) Set pencolor to colorstring, which is a Tk color specification string, such as "red", "yellow", or "#33cc8c". turtle.shape(name=None) Parameters: name ? a string which is a valid shapename class turtle.Shape(type_, data) Parameters: type_ ? one of the strings ?polygon?, ?image?, ?compound? Suppose adding from __future__ import unicode_literals to a working program causes an exception, such as with turtle http://bugs.python.org/issue15618 (Note: unicode_literals is not indexed.) Is this a programmer error for passing unicode instead of string, or a library error for not accepting unicode? Is changing 'isinstance(x, str)' in the library (with whatever other changes are needed) a bugfix to be pushed or a prohibited API expansion? -- Terry Jan Reedy From solipsis at pitrou.net Sun Mar 2 21:08:51 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 2 Mar 2014 21:08:51 +0100 Subject: [Python-Dev] unicode_string future, str -> basestring, fix or feature References: Message-ID: <20140302210851.5143cbec@fsol> On Sun, 02 Mar 2014 15:01:01 -0500 Terry Reedy wrote: > Suppose a 2.7 standard library function is documented as taking a > 'string' argument, such as these examples from the turtle module. > > pencolor(colorstring) > Set pencolor to colorstring, which is a Tk color specification > string, such as "red", "yellow", or "#33cc8c". > > turtle.shape(name=None) > Parameters: name ? a string which is a valid shapename > > class turtle.Shape(type_, data) > Parameters: type_ ? one of the strings ?polygon?, ?image?, ?compound? > > Suppose adding > from __future__ import unicode_literals > to a working program causes an exception, such as with turtle > http://bugs.python.org/issue15618 > (Note: unicode_literals is not indexed.) > > Is this a programmer error for passing unicode instead of string, or a > library error for not accepting unicode? In most cases I would say it's a library error. The only exception is when the argument is clearly meant as a byte string rather than a text string, such as when writing to a binary file or a socket. Regards Antoine. From guido at python.org Sun Mar 2 21:12:30 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 2 Mar 2014 12:12:30 -0800 Subject: [Python-Dev] unicode_string future, str -> basestring, fix or feature In-Reply-To: References: Message-ID: It looks to me like a defect in the library (*), and you are making a reasonable argument that we should fix it in 2.7 to help people be more prepared for the transition to Python 3. (*) As Antoine points out, pretty much the only time where it's not a good idea to switch from str to basestring is when the data is meant to be binary -- but in this case it's clearly text (we can also tell from what the same code looks like in Python 3 :-). On Sun, Mar 2, 2014 at 12:01 PM, Terry Reedy wrote: > Suppose a 2.7 standard library function is documented as taking a 'string' > argument, such as these examples from the turtle module. > > pencolor(colorstring) > Set pencolor to colorstring, which is a Tk color specification string, > such as "red", "yellow", or "#33cc8c". > > turtle.shape(name=None) > Parameters: name - a string which is a valid shapename > > class turtle.Shape(type_, data) > Parameters: type_ - one of the strings "polygon", "image", "compound" > > Suppose adding > from __future__ import unicode_literals > to a working program causes an exception, such as with turtle > http://bugs.python.org/issue15618 > (Note: unicode_literals is not indexed.) > > Is this a programmer error for passing unicode instead of string, or a > library error for not accepting unicode? > Is changing 'isinstance(x, str)' in the library (with whatever other > changes are needed) a bugfix to be pushed or a prohibited API expansion? > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sun Mar 2 21:31:42 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 02 Mar 2014 22:31:42 +0200 Subject: [Python-Dev] Wave module support for floating point data In-Reply-To: <53123BB1.7030903@sebastiankraft.net> References: <53123BB1.7030903@sebastiankraft.net> Message-ID: 01.03.14 21:57, Sebastian Kraft ???????(??): > more than a year ago I have submitted a patch to enhance the Wave module > with read/write support for floating point data. > > http://bugs.python.org/issue16525 > > Up till now this patch has not been applied nor did I get feedback if > anything needs to be enhanced or changed before it can be committed. > I have never been in contact with your development process and Python > core developers, so please tell me what I should improve... I had assigned this issue to myself because I were going to solve it. There was too little time before features freeze, so the patch had been delayed until 3.5. As for the patch itself, there were many changes in the wave module last months, especially in the tests, so the patch can't be applied cleanly and should be rewritten. The main problem is that this issue could not be resolved separately. First of all we should add support for floating point in the audioop module, we also need to add support for floating point in aifc and sunau modules. Need to design a new interface for audio modules. From tjreedy at udel.edu Sun Mar 2 21:40:31 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 02 Mar 2014 15:40:31 -0500 Subject: [Python-Dev] unicode_string future, str -> basestring, fix or feature In-Reply-To: References: Message-ID: On 3/2/2014 3:12 PM, Guido van Rossum wrote: > It looks to me like a defect in the library (*), and you are making a > reasonable argument that we should fix it in 2.7 to help people be more > prepared for the transition to Python 3. > > (*) As Antoine points out, pretty much the only time where it's not a > good idea to switch from str to basestring is when the data is meant to > be binary -- but in this case it's clearly text (we can also tell from > what the same code looks like in Python 3 :-). Thanks to both of you. 'bugfix' noted on the issue. > > On Sun, Mar 2, 2014 at 12:01 PM, Terry Reedy > wrote: > > Suppose a 2.7 standard library function is documented as taking a > 'string' argument, such as these examples from the turtle module. > > pencolor(colorstring) > Set pencolor to colorstring, which is a Tk color specification > string, such as "red", "yellow", or "#33cc8c". > > turtle.shape(name=None) > Parameters: name ? a string which is a valid shapename > > class turtle.Shape(type_, data) > Parameters: type_ ? one of the strings ?polygon?, ?image?, > ?compound? > > Suppose adding > from __future__ import unicode_literals > to a working program causes an exception, such as with turtle > http://bugs.python.org/__issue15618 > (Note: unicode_literals is not indexed.) > > Is this a programmer error for passing unicode instead of string, or > a library error for not accepting unicode? > Is changing 'isinstance(x, str)' in the library (with whatever other > changes are needed) a bugfix to be pushed or a prohibited API expansion? -- Terry Jan Reedy From storchaka at gmail.com Sun Mar 2 22:23:55 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 02 Mar 2014 23:23:55 +0200 Subject: [Python-Dev] unicode_string future, str -> basestring, fix or feature In-Reply-To: References: Message-ID: 02.03.14 22:01, Terry Reedy ???????(??): > Is this a programmer error for passing unicode instead of string, or a > library error for not accepting unicode? > Is changing 'isinstance(x, str)' in the library (with whatever other > changes are needed) a bugfix to be pushed or a prohibited API expansion? Patches which add support for unicode strings were accepted for one issues (e.g. http://bugs.python.org/issue19099) and rejected for other issues (e.g. http://bugs.python.org/issue20014 and http://bugs.python.org/issue20015). Some issues (e.g. http://bugs.python.org/issue18695) hang in undefined state. From berker.peksag at gmail.com Sun Mar 2 23:45:03 2014 From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=) Date: Mon, 3 Mar 2014 00:45:03 +0200 Subject: [Python-Dev] unicode_string future, str -> basestring, fix or feature In-Reply-To: References: Message-ID: On Sun, Mar 2, 2014 at 11:23 PM, Serhiy Storchaka wrote: > Patches which add support for unicode strings were accepted for one issues > (e.g. http://bugs.python.org/issue19099) and rejected for other issues (e.g. > http://bugs.python.org/issue20014 and http://bugs.python.org/issue20015). See also http://bugs.python.org/issue15843. --Berker From tjreedy at udel.edu Mon Mar 3 01:02:00 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 02 Mar 2014 19:02:00 -0500 Subject: [Python-Dev] unicode_string future, str -> basestring, fix or feature In-Reply-To: References: Message-ID: On 3/2/2014 4:23 PM, Serhiy Storchaka wrote: > 02.03.14 22:01, Terry Reedy ???????(??): >> Is this a programmer error for passing unicode instead of string, or a >> library error for not accepting unicode? >> Is changing 'isinstance(x, str)' in the library (with whatever other >> changes are needed) a bugfix to be pushed or a prohibited API expansion? > > Patches which add support for unicode strings were accepted for one > issues (e.g. http://bugs.python.org/issue19099) and rejected for other > issues (e.g. http://bugs.python.org/issue20014 and > http://bugs.python.org/issue20015). Some issues (e.g. > http://bugs.python.org/issue18695) hang in undefined state. If Antoine and Guido don't reverse themselves, those could perhaps be re-opened. It strikes me as borderline, depending interpretation of 'string'. I am not surprised there have been different resolutions. -- Terry Jan Reedy From ncoghlan at gmail.com Mon Mar 3 03:09:26 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 3 Mar 2014 12:09:26 +1000 Subject: [Python-Dev] unicode_string future, str -> basestring, fix or feature In-Reply-To: References: Message-ID: On 3 March 2014 10:02, Terry Reedy wrote: > On 3/2/2014 4:23 PM, Serhiy Storchaka wrote: >> >> 02.03.14 22:01, Terry Reedy ???????(??): >>> >>> Is this a programmer error for passing unicode instead of string, or a >>> library error for not accepting unicode? >>> Is changing 'isinstance(x, str)' in the library (with whatever other >>> changes are needed) a bugfix to be pushed or a prohibited API expansion? >> >> >> Patches which add support for unicode strings were accepted for one >> issues (e.g. http://bugs.python.org/issue19099) and rejected for other >> issues (e.g. http://bugs.python.org/issue20014 and >> http://bugs.python.org/issue20015). Some issues (e.g. >> http://bugs.python.org/issue18695) hang in undefined state. > > > If Antoine and Guido don't reverse themselves, those could perhaps be > re-opened. It strikes me as borderline, depending interpretation of > 'string'. I am not surprised there have been different resolutions. It occurs to me that it would be good to have a "bug fix or feature?" section in the developer guide to provide a more permanent record of dicussions like this. That would also be the place to document tricks like defining a private API to fix a bug in a maintenance release, and then potentially making that new API public for the next feature release if it's potentially useful to end users. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephen at xemacs.org Mon Mar 3 03:26:46 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 03 Mar 2014 11:26:46 +0900 Subject: [Python-Dev] unicode_string future, str -> basestring, fix or feature In-Reply-To: References: Message-ID: <87d2i4rp95.fsf@uwakimon.sk.tsukuba.ac.jp> Terry Reedy writes: > On 3/2/2014 4:23 PM, Serhiy Storchaka wrote: > > Patches which add support for unicode strings were accepted for one > > issues (e.g. http://bugs.python.org/issue19099) and rejected for other > > issues (e.g. http://bugs.python.org/issue20014 and > > http://bugs.python.org/issue20015). Some issues (e.g. > > http://bugs.python.org/issue18695) hang in undefined state. > > If Antoine and Guido don't reverse themselves, those could perhaps be > re-opened. It strikes me as borderline, depending interpretation of > 'string'. I am not surprised there have been different resolutions. I agree with Victor in http://bugs.python.org/issue18695#msg208857: there's no "bug". It is just that in the design of 2.x 'str' is not Unicode, and the "fix" is Python 3. This may be an area where 2to3 could give more help. As Victor points out in that message, the issue-by-issue approach to this inconsistency is just whack-a-mole. I would worry not only about the whack-a-mole aspect where 'unicode' objects leak into contexts where they're not supported, but also that this could confuse tools like 2to3. I agree that usage of the word "string" is all too often ambiguous in the documentation, but that doesn't justify a wholesale overhaul of the Python 2.7 API to make everything polymorphic. From guido at python.org Mon Mar 3 03:44:20 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 2 Mar 2014 18:44:20 -0800 Subject: [Python-Dev] unicode_string future, str -> basestring, fix or feature In-Reply-To: <87d2i4rp95.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87d2i4rp95.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: AFACT, in that message Victor was only talking about allowing Unicode filenames. Making everything polymorphic is clearly pulling on the thread that will unravel the entire sweater. But... The start of this thread was about changing a few occurrences of isinstance(..., str) to use basestring, and that's a different matter. The Python 2 Unicode design calls for mixing of Unicode and 8-bit strings as long as the latter contain 7-bit ASCII -- the code in turtle violates that design by insisting on an 8-bit string. The underlying Tkinter module handles Unicode strings just fine (and not just 7-bit ASCII). As far as lib2to3 goes, using basestring instead of str actually disambiguates things -- with str it can't tell for sure whether text or binary was meant, but with basestring it's a safe bet that the intention was text. Finally, in most places Python 2.7 *does* handle Unicode filenames just fine. On Sun, Mar 2, 2014 at 6:26 PM, Stephen J. Turnbull wrote: > Terry Reedy writes: > > On 3/2/2014 4:23 PM, Serhiy Storchaka wrote: > > > > Patches which add support for unicode strings were accepted for one > > > issues (e.g. http://bugs.python.org/issue19099) and rejected for > other > > > issues (e.g. http://bugs.python.org/issue20014 and > > > http://bugs.python.org/issue20015). Some issues (e.g. > > > http://bugs.python.org/issue18695) hang in undefined state. > > > > If Antoine and Guido don't reverse themselves, those could perhaps be > > re-opened. It strikes me as borderline, depending interpretation of > > 'string'. I am not surprised there have been different resolutions. > > I agree with Victor in http://bugs.python.org/issue18695#msg208857: > there's no "bug". It is just that in the design of 2.x 'str' is not > Unicode, and the "fix" is Python 3. This may be an area where 2to3 > could give more help. > > As Victor points out in that message, the issue-by-issue approach to > this inconsistency is just whack-a-mole. > > I would worry not only about the whack-a-mole aspect where 'unicode' > objects leak into contexts where they're not supported, but also that > this could confuse tools like 2to3. > > I agree that usage of the word "string" is all too often ambiguous in > the documentation, but that doesn't justify a wholesale overhaul of > the Python 2.7 API to make everything polymorphic. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Mon Mar 3 08:41:04 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 03 Mar 2014 09:41:04 +0200 Subject: [Python-Dev] unicode_string future, str -> basestring, fix or feature In-Reply-To: References: Message-ID: 03.03.14 02:02, Terry Reedy ???????(??): > On 3/2/2014 4:23 PM, Serhiy Storchaka wrote: >> 02.03.14 22:01, Terry Reedy ???????(??): >>> Is this a programmer error for passing unicode instead of string, or a >>> library error for not accepting unicode? >>> Is changing 'isinstance(x, str)' in the library (with whatever other >>> changes are needed) a bugfix to be pushed or a prohibited API expansion? >> >> Patches which add support for unicode strings were accepted for one >> issues (e.g. http://bugs.python.org/issue19099) and rejected for other >> issues (e.g. http://bugs.python.org/issue20014 and >> http://bugs.python.org/issue20015). Some issues (e.g. >> http://bugs.python.org/issue18695) hang in undefined state. > > If Antoine and Guido don't reverse themselves, those could perhaps be > re-opened. It strikes me as borderline, depending interpretation of > 'string'. I am not surprised there have been different resolutions. I believe that in all cases when valid values are ASCII-only strings (format specifiers for array, struct, memoryview, etc), we can accept both str and unicode. Especially when they are likely literals. But when valid value can be non-ASCII (e.g. file names), it is a different case, because it requires additional and may be totally different code. From georg at python.org Mon Mar 3 10:51:38 2014 From: georg at python.org (Georg Brandl) Date: Mon, 03 Mar 2014 10:51:38 +0100 Subject: [Python-Dev] [RELEASED] Python 3.3.5 release candidate 2 Message-ID: <531450AA.3030005@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team, I'm happy to announce the release of Python 3.3.5, release candidate 2. Python 3.3.5 includes a fix for a regression in zipimport in 3.3.4 (see http://bugs.python.org/issue20621) and a few other bugs. Python 3.3 includes a range of improvements of the 3.x series, as well as easier porting between 2.x and 3.x. In total, almost 500 API items are new or improved in Python 3.3. For a more extensive list of changes in the 3.3 series, see http://docs.python.org/3.3/whatsnew/3.3.html To download Python 3.3.5 visit: http://www.python.org/download/releases/3.3.5/ This is a preview release, please report any bugs to http://bugs.python.org/ The final release is scheduled one week from now. Enjoy! - -- Georg Brandl, Release Manager georg at python.org (on behalf of the entire python-dev team and 3.3's contributors) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlMUUKoACgkQN9GcIYhpnLD5OACfTpRkcM9aXUx2XbiXoZtIgSE7 BqwAnjwpAuqc9lKJ0O3XOw5qDvDPYsNb =EGuB -----END PGP SIGNATURE----- From victor.stinner at gmail.com Mon Mar 3 12:01:40 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 3 Mar 2014 12:01:40 +0100 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? Message-ID: Hi, I would like to know if the cherry-picking rule still applies for Python 3.4 final? Can I open an issue if I want to see a changeset in the final version? I'm asking for a typo in tracemalloc documentation: http://bugs.python.org/issue20814 http://hg.python.org/cpython/rev/a9058b772807 Victor From larry at hastings.org Mon Mar 3 13:13:21 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 03 Mar 2014 04:13:21 -0800 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: References: Message-ID: <531471E1.1010509@hastings.org> On 03/03/2014 03:01 AM, Victor Stinner wrote: > Hi, > > I would like to know if the cherry-picking rule still applies for > Python 3.4 final? Can I open an issue if I want to see a changeset in > the final version? Sadly, yes. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Mar 3 14:05:57 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 3 Mar 2014 14:05:57 +0100 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: <531471E1.1010509@hastings.org> References: <531471E1.1010509@hastings.org> Message-ID: 2014-03-03 13:13 GMT+01:00 Larry Hastings : > I would like to know if the cherry-picking rule still applies for > Python 3.4 final? Can I open an issue if I want to see a changeset in > the final version? > > Sadly, yes. Ok, I created: http://bugs.python.org/issue20843 Why do you say "sadly"? It's up to you to decide if a change can wait Python 3.4.1 or not. Feel free to close my cherry-pick issue as wontfix. Victor From chris.barker at noaa.gov Mon Mar 3 17:37:15 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 3 Mar 2014 08:37:15 -0800 Subject: [Python-Dev] unicode_string future, str -> basestring, fix or feature In-Reply-To: References: <87d2i4rp95.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Sun, Mar 2, 2014 at 6:44 PM, Guido van Rossum wrote: > AFACT, in that message Victor was only talking about allowing Unicode > filenames. > ... > Finally, in most places Python 2.7 *does* handle Unicode filenames just > fine. > I'm a bit confused. In this example: http://bugs.python.org/issue18695 You are proposing that the issue should be considered a bug and a well-written patch accepted? Or is is just too late for 2.7 ? Personally I think that having some, but not all file functions accept unicode paths is pretty broken....and fixing these kinds of thing will ease 2 to 3 transition, so a good thing overall. - Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Mar 3 18:40:01 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Mar 2014 09:40:01 -0800 Subject: [Python-Dev] unicode_string future, str -> basestring, fix or feature In-Reply-To: References: <87d2i4rp95.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Mar 3, 2014 at 8:37 AM, Chris Barker wrote: > On Sun, Mar 2, 2014 at 6:44 PM, Guido van Rossum wrote: > >> AFACT, in that message Victor was only talking about allowing Unicode >> filenames. >> > ... > >> Finally, in most places Python 2.7 *does* handle Unicode filenames just >> fine. >> > > I'm a bit confused. In this example: > > http://bugs.python.org/issue18695 > > You are proposing that the issue should be considered a bug and a > well-written patch accepted? > > Or is is just too late for 2.7 ? > > Personally I think that having some, but not all file functions accept > unicode paths is pretty broken....and fixing these kinds of thing will ease > 2 to 3 transition, so a good thing overall. > Agreed. Given that the claim "Python 2 doesn't support Unicode filenames" is factually incorrect (in Python 2.7, most filesystem calls in fact do support Unicode, at least on some platforms), I think individual functions in the os module that are found lacking should be considered bugs, and if someone goes through the effort to supply an otherwise acceptable fix, we shouldn't reject it on the basis that we don't want to consider supporting Unicode filenames. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Mar 3 19:31:20 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 03 Mar 2014 13:31:20 -0500 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: <531471E1.1010509@hastings.org> References: <531471E1.1010509@hastings.org> Message-ID: On 3/3/2014 7:13 AM, Larry Hastings wrote: > On 03/03/2014 03:01 AM, Victor Stinner wrote: >> Hi, >> >> I would like to know if the cherry-picking rule still applies for >> Python 3.4 final? Can I open an issue if I want to see a changeset in >> the final version? > > Sadly, yes. Doc changes appear online within hours. I should expect that releases pull in and bundle the latest version of the doc possible, just before the release. Is this not the usual procedure? -- Terry Jan Reedy From g.brandl at gmx.net Mon Mar 3 20:04:13 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 03 Mar 2014 20:04:13 +0100 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: References: <531471E1.1010509@hastings.org> Message-ID: Am 03.03.2014 19:31, schrieb Terry Reedy: > On 3/3/2014 7:13 AM, Larry Hastings wrote: >> On 03/03/2014 03:01 AM, Victor Stinner wrote: >>> Hi, >>> >>> I would like to know if the cherry-picking rule still applies for >>> Python 3.4 final? Can I open an issue if I want to see a changeset in >>> the final version? >> >> Sadly, yes. > > Doc changes appear online within hours. I should expect that releases > pull in and bundle the latest version of the doc possible, just before > the release. Is this not the usual procedure? No, that would be mostly pointless churn, with all the usual dangers of last minute changes. Georg From larry at hastings.org Mon Mar 3 22:25:57 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 03 Mar 2014 13:25:57 -0800 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: References: <531471E1.1010509@hastings.org> Message-ID: <5314F365.80104@hastings.org> On 03/03/2014 05:05 AM, Victor Stinner wrote: > 2014-03-03 13:13 GMT+01:00 Larry Hastings : >> I would like to know if the cherry-picking rule still applies for >> Python 3.4 final? Can I open an issue if I want to see a changeset in >> the final version? >> >> Sadly, yes. > Ok, I created: > http://bugs.python.org/issue20843 > > Why do you say "sadly"? It's up to you to decide if a change can wait > Python 3.4.1 or not. Feel free to close my cherry-pick issue as > wontfix. It was intended as gentle comedy. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Mar 3 22:38:36 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 4 Mar 2014 07:38:36 +1000 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: <5314F365.80104@hastings.org> References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> Message-ID: On 4 Mar 2014 07:32, "Larry Hastings" wrote: > > On 03/03/2014 05:05 AM, Victor Stinner wrote: >> >> 2014-03-03 13:13 GMT+01:00 Larry Hastings : >>> >>> I would like to know if the cherry-picking rule still applies for >>> Python 3.4 final? Can I open an issue if I want to see a changeset in >>> the final version? >>> >>> Sadly, yes. >> >> Ok, I created: >> http://bugs.python.org/issue20843 >> >> Why do you say "sadly"? It's up to you to decide if a change can wait >> Python 3.4.1 or not. Feel free to close my cherry-pick issue as >> wontfix. > > > It was intended as gentle comedy. Related question - have you decided yet whether or not to do an rc3? I ask, as I believe it would be good to give the folks like Mike Bayer and Armin Ronacher (who picked up test coverage gaps in rc2 via the Alembic and Flask test suites respectively) a chance to rerun their tests before we declare 3.4 final. Cheers, Nick. > > > /arry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Mon Mar 3 23:36:44 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 03 Mar 2014 22:36:44 +0000 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> Message-ID: On 03/03/2014 21:38, Nick Coghlan wrote: > > On 4 Mar 2014 07:32, "Larry Hastings" > wrote: > > > > On 03/03/2014 05:05 AM, Victor Stinner wrote: > >> > >> 2014-03-03 13:13 GMT+01:00 Larry Hastings >: > >>> > >>> I would like to know if the cherry-picking rule still applies for > >>> Python 3.4 final? Can I open an issue if I want to see a changeset in > >>> the final version? > >>> > >>> Sadly, yes. > >> > >> Ok, I created: > >> http://bugs.python.org/issue20843 > >> > >> Why do you say "sadly"? It's up to you to decide if a change can wait > >> Python 3.4.1 or not. Feel free to close my cherry-pick issue as > >> wontfix. > > > > > > It was intended as gentle comedy. > > Related question - have you decided yet whether or not to do an rc3? > > I ask, as I believe it would be good to give the folks like Mike Bayer > and Armin Ronacher (who picked up test coverage gaps in rc2 via the > Alembic and Flask test suites respectively) a chance to rerun their > tests before we declare 3.4 final. > > Cheers, > Nick. > Will this impact on the decision http://bugs.python.org/issue20846 ? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com From barry at python.org Mon Mar 3 23:44:35 2014 From: barry at python.org (Barry Warsaw) Date: Mon, 3 Mar 2014 17:44:35 -0500 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> Message-ID: <20140303174435.4b35c80b@anarchist.wooz.org> On Mar 03, 2014, at 10:36 PM, Mark Lawrence wrote: >Will this impact on the decision http://bugs.python.org/issue20846 ? Issue 20808 is my own pet cherry pick for 3.4. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From victor.stinner at gmail.com Mon Mar 3 23:51:37 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 3 Mar 2014 23:51:37 +0100 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> Message-ID: > > >> Will this impact on the decision http://bugs.python.org/issue20846 ? > This issue has been closed as wontfix. It has no patch and must be reported to pip, not python. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Mar 4 00:15:50 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 4 Mar 2014 09:15:50 +1000 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> Message-ID: On 4 Mar 2014 08:40, "Mark Lawrence" wrote: > > On 03/03/2014 21:38, Nick Coghlan wrote: >> >> >> On 4 Mar 2014 07:32, "Larry Hastings" > > wrote: >> > >> > On 03/03/2014 05:05 AM, Victor Stinner wrote: >> >> >> >> 2014-03-03 13:13 GMT+01:00 Larry Hastings > >: >> >> >>> >> >>> I would like to know if the cherry-picking rule still applies for >> >>> Python 3.4 final? Can I open an issue if I want to see a changeset in >> >>> the final version? >> >>> >> >>> Sadly, yes. >> >> >> >> Ok, I created: >> >> http://bugs.python.org/issue20843 >> >> >> >> Why do you say "sadly"? It's up to you to decide if a change can wait >> >> Python 3.4.1 or not. Feel free to close my cherry-pick issue as >> >> wontfix. >> > >> > >> > It was intended as gentle comedy. >> >> Related question - have you decided yet whether or not to do an rc3? >> >> I ask, as I believe it would be good to give the folks like Mike Bayer >> and Armin Ronacher (who picked up test coverage gaps in rc2 via the >> Alembic and Flask test suites respectively) a chance to rerun their >> tests before we declare 3.4 final. >> >> Cheers, >> Nick. >> > > Will this impact on the decision http://bugs.python.org/issue20846 ? No. I never claimed pip was bug free (any more than CPython itself is), merely the best available option. File a bug against pip (assuming there isn't one already), get it fixed, and we'll bundle the fixed version with the next CPython maintenance release. Cheers, Nick. > > -- > My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. > > Mark Lawrence > > --- > This email is free from viruses and malware because avast! Antivirus protection is active. > http://www.avast.com > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Tue Mar 4 00:47:37 2014 From: nad at acm.org (Ned Deily) Date: Mon, 03 Mar 2014 15:47:37 -0800 Subject: [Python-Dev] Issue20846 [was: Cherry-pick between Python 3.4 RC2 and final?] References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> Message-ID: In article , Nick Coghlan wrote: > On 4 Mar 2014 08:40, "Mark Lawrence" wrote: > > Will this impact on the decision http://bugs.python.org/issue20846 ? > No. I never claimed pip was bug free (any more than CPython itself is), > merely the best available option. > > File a bug against pip (assuming there isn't one already), get it fixed, > and we'll bundle the fixed version with the next CPython maintenance > release. FTR, the behavior documented in Issue20846 has nothing to do with either pip, setuptools, or python3.4. See updates to the issue. One could argue about whether pip should check and, by default, not offer to download and install packages that are marked in PyPI as not being compatible with the Python version being used, e.g. trying to install a Python 2 only package with a Python 3 or vice versa. AFAIK, neither pip nor easy_install (setuptools) has ever done that. And, in any case, it would not have helped with the package in question as it does not specify in its package metadata (as shown on PyPI) with which versions of Pythons it is supported. I don't know whether a request for such a feature has ever been made for pip; I didn't see any on a quick search of the pip tracker. But this is off-topic for python-dev. -- Ned Deily, nad at acm.org From ncoghlan at gmail.com Tue Mar 4 03:30:27 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 4 Mar 2014 12:30:27 +1000 Subject: [Python-Dev] Issue20846 [was: Cherry-pick between Python 3.4 RC2 and final?] In-Reply-To: References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> Message-ID: On 4 Mar 2014 09:49, "Ned Deily" wrote: > > In article > , > Nick Coghlan wrote: > > On 4 Mar 2014 08:40, "Mark Lawrence" wrote: > > > Will this impact on the decision http://bugs.python.org/issue20846 ? > > No. I never claimed pip was bug free (any more than CPython itself is), > > merely the best available option. > > > > File a bug against pip (assuming there isn't one already), get it fixed, > > and we'll bundle the fixed version with the next CPython maintenance > > release. > > FTR, the behavior documented in Issue20846 has nothing to do with either > pip, setuptools, or python3.4. See updates to the issue. > > One could argue about whether pip should check and, by default, not > offer to download and install packages that are marked in PyPI as not > being compatible with the Python version being used, e.g. trying to > install a Python 2 only package with a Python 3 or vice versa. AFAIK, > neither pip nor easy_install (setuptools) has ever done that. And, in > any case, it would not have helped with the package in question as it > does not specify in its package metadata (as shown on PyPI) with which > versions of Pythons it is supported. I don't know whether a request for > such a feature has ever been made for pip; I didn't see any on a quick > search of the pip tracker. But this is off-topic for python-dev. As one final comment on this: allowing proper environmental constraints that installers can reliably enforce is one of the goals of the metadata 2.0 effort. The existing metadata is not considered sufficiently accurate to allow it to be used as the basis for automated enforcement. Regards, Nick. > > -- > Ned Deily, > nad at acm.org > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Tue Mar 4 04:35:29 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 03 Mar 2014 19:35:29 -0800 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> Message-ID: <53154A01.3030008@hastings.org> On 03/03/2014 01:38 PM, Nick Coghlan wrote: > Related question - have you decided yet whether or not to do an rc3? > > I ask, as I believe it would be good to give the folks like Mike Bayer > and Armin Ronacher (who picked up test coverage gaps in rc2 via the > Alembic and Flask test suites respectively) a chance to rerun their > tests before we declare 3.4 final. > I hadn't planned on it. I'll reach out to those two and see if they can deal with tarballs, rather than requiring binary installers--if tarballs works for them, I'll steer them towards the 3.4 merge status tarballs and ping them when I spin up some new ones. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Mar 4 04:58:48 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 4 Mar 2014 13:58:48 +1000 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: <53154A01.3030008@hastings.org> References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> Message-ID: On 4 March 2014 13:35, Larry Hastings wrote: > On 03/03/2014 01:38 PM, Nick Coghlan wrote: > > Related question - have you decided yet whether or not to do an rc3? > > I ask, as I believe it would be good to give the folks like Mike Bayer and > Armin Ronacher (who picked up test coverage gaps in rc2 via the Alembic and > Flask test suites respectively) a chance to rerun their tests before we > declare 3.4 final. > > > I hadn't planned on it. I'll reach out to those two and see if they can > deal with tarballs, rather than requiring binary installers--if tarballs > works for them, I'll steer them towards the 3.4 merge status tarballs and > ping them when I spin up some new ones. All of our development guides for testing against trunk are designed around running from a Mercurial checkout - it would *really* be whole lot easier for everyone else that is trying to test the release if you could just do a push from your release clone to a server side clone on hg.python.org (the link to create one is on http://hg.python.org/cpython/, and then it's just a hg push to publish a mirror of the exact state of your current clone). If you don't want to do an rc3 despite the cherry picked changes since rc2, then you need to make it easy for people to test the changes directly from the release branch. An opaque intermittently updated tarball is not acceptable when none of our infrastructure is designed to work that way. I was OK with just the tarball when I assumed you would an rc3 if non-trivial defects were found in rc2. If that's not the case, then we *need* a public mirror of your release clone. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From larry at hastings.org Tue Mar 4 05:20:14 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 03 Mar 2014 20:20:14 -0800 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> Message-ID: <5315547E.4000702@hastings.org> On 03/03/2014 07:58 PM, Nick Coghlan wrote: > All of our development guides for testing against trunk are designed > around running from a Mercurial checkout - it would *really* be whole > lot easier for everyone else that is trying to test the release if you > could just do a push from your release clone to a server side clone on > hg.python.org (the link to create one is on > http://hg.python.org/cpython/, and then it's just a hg push to publish > a mirror of the exact state of your current clone). I've pulled enough shenanigans over here (discarding and recreating the 3.4 branch, rebasing) that I'm really glad I haven't made the revisions public. And I'm not done, either, *yet another* really old revision has cropped up for cherry-picking. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Mar 4 07:23:12 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 4 Mar 2014 16:23:12 +1000 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: <5315547E.4000702@hastings.org> References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> <5315547E.4000702@hastings.org> Message-ID: On 4 March 2014 14:20, Larry Hastings wrote: > On 03/03/2014 07:58 PM, Nick Coghlan wrote: > > All of our development guides for testing against trunk are designed > around running from a Mercurial checkout - it would *really* be whole > lot easier for everyone else that is trying to test the release if you > could just do a push from your release clone to a server side clone on > hg.python.org (the link to create one is on > http://hg.python.org/cpython/, and then it's just a hg push to publish > a mirror of the exact state of your current clone). > > > I've pulled enough shenanigans over here (discarding and recreating the 3.4 > branch, rebasing) that I'm really glad I haven't made the revisions public. > And I'm not done, either, *yet another* really old revision has cropped up > for cherry-picking. The clones doesn't need to be updatable, we just need an easy way to do "hg clone " to get a release branch to test against. It doesn't matter if hg pull doesn't work later - we just need you to make it as easy as possible for us to help you test things, because if there's not going to be an rc3 then we're running out of time to make sure SQL Alchemy and Flask actually work properly on Python 3.4 final (and I've been told the pkgutil bug that affected Flask may have affected Vim as well). Let *us* deal with the Mercurial problems - you can rebase and cherry pick however the heck you want. But at the moment you're making it *hard* for people to test the release, and that is scaring me - our test coverage is a long way from 100%, so we *need* people running third party test suites on the pre-release to spot coverage gaps the way the Alembic and Flask test suites did. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From larry at hastings.org Tue Mar 4 07:50:35 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 03 Mar 2014 22:50:35 -0800 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> <5315547E.4000702@hastings.org> Message-ID: <531577BB.1060802@hastings.org> On 03/03/2014 10:23 PM, Nick Coghlan wrote: > But at the moment you're making it > *hard* for people to test the release, How? How is testing against a tarball fundamentally different from testing against an hg-cloned repository? I'm really not buying this. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Mar 4 08:33:29 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 4 Mar 2014 17:33:29 +1000 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: <531577BB.1060802@hastings.org> References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> <5315547E.4000702@hastings.org> <531577BB.1060802@hastings.org> Message-ID: On 4 March 2014 16:50, Larry Hastings wrote: > > On 03/03/2014 10:23 PM, Nick Coghlan wrote: > > But at the moment you're making it > *hard* for people to test the release, > > > How? How is testing against a tarball fundamentally different from testing > against an hg-cloned repository? > > I'm really not buying this. Because there are *zero* instructions in the devguide for tarball based testing. Can it be done? Yes. Is it properly documented such that it is acceptable to rely on it as an essential part of the release process? No. *Never* have we done a feature release where we went dark for most of the release candidate cycle - for past feature releases, the release branch was made at the time of the first rc, and everything merged during that time was subject to two committer review, and everything merged to the release branch was automatically considered as a candidate for including in the next rc/final release. After the switch to Mercurial, the contents of the release branch might not be *exactly* what ended up being released, but they were close enough for all the purposes that anyone cared about. That's not the case here - by instituting the new process where you stopped checking every commit and instead required the creation of specific tracker issues, the default branch of the main repo now has a bunch of stuff listed for 3.4.1 that is a mixture of 3.4.0 and 3.4.1 changes, so it's hard to tell whether or not a particular change is going to make it into 3.4 final. I was prepared to go along with that (since those that do the work get to make the rules), but then you said you were going to go straight from a release candidate that broke two of the most popular Python projects to a final release with the release clone *still* offline for reasons I do not understand. That is *not* OK - if we're skipping rc3 even though rc2 broke both Flask and SQL Alchemy (and possibly Vim), then we need to be able to see *exactly* what is going to be published, including the full history of which commits have been cherry-picked, not just the end result as a tarball. You've given people plenty of warning that you'll be rewriting history in your release clone. That's fine, we can deal, especially for throwaway testing clones - git users handle rewritten history as a matter of course, and it really isn't that scary, even in Mercurial. But the combination of skipping rc3 *and* keeping the release clone offline is not a responsible course of action. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From martin at v.loewis.de Tue Mar 4 11:16:41 2014 From: martin at v.loewis.de (martin at v.loewis.de) Date: Tue, 04 Mar 2014 11:16:41 +0100 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> Message-ID: <20140304111641.Horde.WTUtNWKjH-nv8-JUkM3R2w2@webmail.df.eu> Quoting Nick Coghlan : > If you don't want to do an rc3 despite the cherry picked changes since > rc2, then you need to make it easy for people to test the changes > directly from the release branch. An opaque intermittently updated > tarball is not acceptable when none of our infrastructure is designed > to work that way. I was OK with just the tarball when I assumed you > would an rc3 if non-trivial defects were found in rc2. If that's not > the case, then we *need* a public mirror of your release clone. Another rc or not - I expect to see a 3.4.1 release *really* shortly after the 3.4.0 release. So except for issues where Python does not work at all, any glitches that remain in the code can be fixed in the bug fix release. Regards, Martin From ncoghlan at gmail.com Tue Mar 4 11:37:39 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 4 Mar 2014 20:37:39 +1000 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: <20140304111641.Horde.WTUtNWKjH-nv8-JUkM3R2w2@webmail.df.eu> References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> <20140304111641.Horde.WTUtNWKjH-nv8-JUkM3R2w2@webmail.df.eu> Message-ID: On 4 March 2014 20:16, wrote: > > Quoting Nick Coghlan : > >> If you don't want to do an rc3 despite the cherry picked changes since >> rc2, then you need to make it easy for people to test the changes >> directly from the release branch. An opaque intermittently updated >> tarball is not acceptable when none of our infrastructure is designed >> to work that way. I was OK with just the tarball when I assumed you >> would an rc3 if non-trivial defects were found in rc2. If that's not >> the case, then we *need* a public mirror of your release clone. > > > Another rc or not - I expect to see a 3.4.1 release *really* shortly after > the 3.4.0 release. So except for issues where Python does not work at all, > any glitches that remain in the code can be fixed in the bug fix release. Right, but I don't see the need to rush 3.4.0 itself - these were definitely edge cases (that's why our test suite missed them), but they were a result of two of the deeper changes in 3.4 (the import system changes and the introspection changes). Mike and Armin found, investigated and reported them when testing on rc2 turned up a couple of relatively obscure and hard to diagnose failures in their own test suites, and it seems discourteous to then turn around and put pressure on them to double check the fix in our original release time frame rather than granting the extra two weeks a 3rd rc would provide not just to us, but to anyone doing third party testing. (Barry and Matthias already confirmed we can do that much while still having the final release make it directly in Ubuntu 14.04) Maybe I'm being overly conservative, and if Larry, you, and Ned are all comfortable with the idea of going straight to a final release from here, I'm not going to argue further (since you're the ones that would incur the additional work from an additional release). I just wanted to be clear that I'd be more comfortable with either a 3rd release candidate given the pip installation, pkgutil & type slot introspection changes I'm aware of, or at least the ability for people to test the in-development final release using the usual processes documented in the devguide. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Tue Mar 4 11:55:46 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 4 Mar 2014 11:55:46 +0100 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> Message-ID: Hi, 2014-03-03 22:38 GMT+01:00 Nick Coghlan : > Related question - have you decided yet whether or not to do an rc3? I take a look at current release blocker issues for Python 3.4. I saw bugfixes (ex: upgrade SQLite from 3.8.3 to 3.8.3.1) but also fixes for regressions between Python 3.4rc2 and Python 3.4rc1, especially about pip on Windows. It looks like the Windows installer *and* pip are currently broken on Windows with Python 3.4rc2. It would be sad to release Python 3.4 with pip broken (on Windows) since it was announced (*) that Python 3.4 finally fixes the "pip bootstrap" issue, and users may expect this feature. If we fix the Windows installer and pip on Windows, it would be safer to have a RC3 because I don't know how to test Windows installer and I don't want to learn how to do that. Note: There are also requested changes in OS X installer (upgrade SQLite). IMO a perfect final version is exactly the same than the last RC except that the version changed. Here I see a long list of cherry-pick issues... @Larry: What do you think of a RC3 for March 16th, and a final release one week later (March 23th)? (Or maybe a RC3 for this week-end if you have some free time :-)) (*) Read for example https://lwn.net/Articles/570471/ "Rationalizing Python packaging" Victor From stephen at xemacs.org Tue Mar 4 14:23:52 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 04 Mar 2014 22:23:52 +0900 Subject: [Python-Dev] unicode_string future, str -> basestring, fix or feature In-Reply-To: References: <87d2i4rp95.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87vbvuqeqf.fsf@uwakimon.sk.tsukuba.ac.jp> >>>>> Guido van Rossum writes: > Given that the claim "Python 2 doesn't support Unicode filenames" > is factually incorrect (in Python 2.7, most filesystem calls in > fact do support Unicode, at least on some platforms), I don't understand what "support Unicode" means. Just that with open(u"\u4e00", "w") as f: f.write("works!\n") does what is expected[1] if the user knows what he is doing (ie, has set PYTHONIOENCODING to a Unicode UTF or one of the Asian encodings)? > I think individual functions in the os module that are found > lacking should be considered bugs, and if someone goes through > the effort to supply an otherwise acceptable fix, we shouldn't > reject it on the basis that we don't want to consider supporting > Unicode filenames. As above, "acceptable fix" means take whatever the current value is for file system name encoding, and use that to encode and decode unicode objects to/from str, or raise a UnicodeError if it doesn't work? I think it's important to define this somewhat carefully, because this is an area that has a strong tendency to "mission creep". Given that builtin open "works" by the above definition, I guess it's reasonable to accept such patches. Footnotes: [1] It writes the line "works!\n" to a file whose name consists of the single Chinese character for "one". From solipsis at pitrou.net Tue Mar 4 14:33:56 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 4 Mar 2014 14:33:56 +0100 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> <20140304111641.Horde.WTUtNWKjH-nv8-JUkM3R2w2@webmail.df.eu> Message-ID: <20140304143356.2ac47d37@fsol> On Tue, 04 Mar 2014 11:16:41 +0100 martin at v.loewis.de wrote: > > Quoting Nick Coghlan : > > > If you don't want to do an rc3 despite the cherry picked changes since > > rc2, then you need to make it easy for people to test the changes > > directly from the release branch. An opaque intermittently updated > > tarball is not acceptable when none of our infrastructure is designed > > to work that way. I was OK with just the tarball when I assumed you > > would an rc3 if non-trivial defects were found in rc2. If that's not > > the case, then we *need* a public mirror of your release clone. > > Another rc or not - I expect to see a 3.4.1 release *really* shortly after > the 3.4.0 release. So except for issues where Python does not work at all, > any glitches that remain in the code can be fixed in the bug fix release. I agree with Martin. At this point, we'd better release 3.4.0 (fixing any critical regressions, but leaving non-critical ones aside), then patiently collect evidence of issues, and fix them in non-rush mode for 3.4.1. Regards Antoine. From brett at python.org Tue Mar 4 15:52:30 2014 From: brett at python.org (Brett Cannon) Date: Tue, 4 Mar 2014 09:52:30 -0500 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: <20140304143356.2ac47d37@fsol> References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> <20140304111641.Horde.WTUtNWKjH-nv8-JUkM3R2w2@webmail.df.eu> <20140304143356.2ac47d37@fsol> Message-ID: On Tue, Mar 4, 2014 at 8:33 AM, Antoine Pitrou wrote: > On Tue, 04 Mar 2014 11:16:41 +0100 > martin at v.loewis.de wrote: > > > > Quoting Nick Coghlan : > > > > > If you don't want to do an rc3 despite the cherry picked changes since > > > rc2, then you need to make it easy for people to test the changes > > > directly from the release branch. An opaque intermittently updated > > > tarball is not acceptable when none of our infrastructure is designed > > > to work that way. I was OK with just the tarball when I assumed you > > > would an rc3 if non-trivial defects were found in rc2. If that's not > > > the case, then we *need* a public mirror of your release clone. > > > > Another rc or not - I expect to see a 3.4.1 release *really* shortly > after > > the 3.4.0 release. So except for issues where Python does not work at > all, > > any glitches that remain in the code can be fixed in the bug fix release. > > I agree with Martin. At this point, we'd better release 3.4.0 (fixing > any critical regressions, but leaving non-critical ones aside), then > patiently collect evidence of issues, and fix them in non-rush > mode for 3.4.1. > How about this (if Larry is amenable): critical now/this week (e.g. pip on Windows works), non-critical in 3.4.1 with Larry aiming to release 3.4.1 by the end of April? That way the PyCon sprints can be used by projects to test against 3.4.1 and the core sprint can work on squashing any bugs that will inevitably come up from 3.4.0 being out the door. This also lets us make our stated 3.4.0 release date. Every release we feel pressure to squeeze on those last few bugs and this release is the same. Heck, you can say it's enhanced as we had a longer rc cycle. Normally the bugs from Armin and Mike -- which I appreciate -- would have just happened after 3.4.0 went out the door and thus be rolled into 3.4.1 anyway. I suspect the community in general views *.0 releases as stable but not perfect while *.1 releases as the solid one where no surprises will pop up because of new code. I have also filed http://bugs.python.org/issue20851 to make sure the devguide covers running tests from a tarball. If the way the release has been handled has still bugged you enough it can be discussed at the language summit, but it would be the first time we consider putting any restrictions on the RM (which I would loathe to do; I'm fine with nudging Larry to do his stuff in public next time in hopes he's more comfortable with the whole process, but I wouldn't want to require it). -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at bytereef.org Tue Mar 4 10:55:27 2014 From: stefan at bytereef.org (Stefan Krah) Date: Tue, 4 Mar 2014 10:55:27 +0100 Subject: [Python-Dev] Start writing inlines rather than macros? In-Reply-To: References: <1900238651415214229.843004sturla.molden-gmail.com@news.gmane.org> <20140227195156.1750ddf9@fsol> Message-ID: <20140304095527.GA4457@sleipnir.bytereef.org> Skip Montanaro wrote: > I do have one question though. Suppose you encounter a compiler that > doesn't understand the inline keyword, so you choose the static > declaration as Kristj?n suggested. The resulting Python executable > should be functionally correct, but if the optimizer doesn't happen to > inline a given static function you might be stuck with some bad > performance across-the-board (if it never inlines, or doesn't inline > where we really need it to), or only under some circumstances (as a > hypothetical example, inlining in dictobject.c, but not in ceval.c). > Is there a configurable way to tell if a compiler will inline > functions which are declared static, and possibly under what > conditions they might not? It might still be necessary to maintain > macros for those platforms. I think that all modern compilers can handle "static inline" in header files. If you have a compiler that cannot, chances are that the platform is horribly outdated and this particular performance issue will be relatively benign compared to other ones. Stefan Krah From guido at python.org Tue Mar 4 17:07:17 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 4 Mar 2014 08:07:17 -0800 Subject: [Python-Dev] unicode_string future, str -> basestring, fix or feature In-Reply-To: <87vbvuqeqf.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87d2i4rp95.fsf@uwakimon.sk.tsukuba.ac.jp> <87vbvuqeqf.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Tue, Mar 4, 2014 at 5:23 AM, Stephen J. Turnbull wrote: > >>>>> Guido van Rossum writes: > > > Given that the claim "Python 2 doesn't support Unicode filenames" > > is factually incorrect (in Python 2.7, most filesystem calls in > > fact do support Unicode, at least on some platforms), > > I don't understand what "support Unicode" means. Just that > > with open(u"\u4e00", "w") as f: f.write("works!\n") > > does what is expected[1] if the user knows what he is doing (ie, has > set PYTHONIOENCODING to a Unicode UTF or one of the Asian encodings)? > That's all I'm asking for, since that's what most functions in 2.7 already do. > > I think individual functions in the os module that are found > > lacking should be considered bugs, and if someone goes through > > the effort to supply an otherwise acceptable fix, we shouldn't > > reject it on the basis that we don't want to consider supporting > > Unicode filenames. > > As above, "acceptable fix" means take whatever the current value is > for file system name encoding, and use that to encode and decode > unicode objects to/from str, or raise a UnicodeError if it doesn't > work? > The same thing that is done for other functions that take filenames. > I think it's important to define this somewhat carefully, because this > is an area that has a strong tendency to "mission creep". Given that > builtin open "works" by the above definition, I guess it's reasonable > to accept such patches. > Right, the interpretation given to Unicode filenames by builtin open() should be propagated to other functions (I actually suspect that os.statvfs(), which apparently doesn't, is in the minority here). AFAIK that's also roughly what happens in Python 3. > Footnotes: > [1] It writes the line "works!\n" to a file whose name consists of the > single Chinese character for "one". > > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From doko at ubuntu.com Tue Mar 4 23:04:30 2014 From: doko at ubuntu.com (Matthias Klose) Date: Tue, 04 Mar 2014 23:04:30 +0100 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> <20140304111641.Horde.WTUtNWKjH-nv8-JUkM3R2w2@webmail.df.eu> <20140304143356.2ac47d37@fsol> Message-ID: <53164DEE.1000904@ubuntu.com> Am 04.03.2014 15:52, schrieb Brett Cannon: > I have also filed http://bugs.python.org/issue20851 to make sure the > devguide covers running tests from a tarball. If the way the release has > been handled has still bugged you enough it can be discussed at the > language summit, but it would be the first time we consider putting any > restrictions on the RM. I wouldn't put it this way, but instead ask how to enable the RM to do this kind of work more publicly. I really would like it more if we can extend our buildd infrastructure to automatically test during the time where the trunk and the release candidates don't match. I am aware that this was never done for any release in the past, but maybe this is something that can be enabled for 3.5. But before documenting a process which may change depending on the RM why not try to align the process? Matthias From ncoghlan at gmail.com Wed Mar 5 00:24:56 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 5 Mar 2014 09:24:56 +1000 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: <53164DEE.1000904@ubuntu.com> References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> <20140304111641.Horde.WTUtNWKjH-nv8-JUkM3R2w2@webmail.df.eu> <20140304143356.2ac47d37@fsol> <53164DEE.1000904@ubuntu.com> Message-ID: On 5 Mar 2014 08:15, "Matthias Klose" wrote: > > Am 04.03.2014 15:52, schrieb Brett Cannon: > > I have also filed http://bugs.python.org/issue20851 to make sure the > > devguide covers running tests from a tarball. If the way the release has > > been handled has still bugged you enough it can be discussed at the > > language summit, but it would be the first time we consider putting any > > restrictions on the RM. > > I wouldn't put it this way, but instead ask how to enable the RM to do this kind > of work more publicly. I really would like it more if we can extend our buildd > infrastructure to automatically test during the time where the trunk and the > release candidates don't match. I am aware that this was never done for any > release in the past, but maybe this is something that can be enabled for 3.5. > But before documenting a process which may change depending on the RM why not > try to align the process? I think it's also the fact that new feature releases are rare and changes of release manager even more so, meaning there's a fair bit of relearning involved every time (since what was appropriate a couple of years earlier may not be appropriate the next time, and we'll usually have new developers involved that weren't around for the previous release). While I'd still strongly prefer an rc3, I think we at least need to get the remaining release blockers sorted in the tracker (either fixing them or reclassifying them, or else closing them if they're a rejected cherry pick request) and a tarball or release clone available for core developer and third party testing. We won't be able to get people to test the pip integration fixes on Windows that way (that's one of the reasons I would like an rc3 and don't understand the apparent desire to avoid one), but we would at least be able to check that the Flask and Alembic test suites pass. I'm being fussy about this for two reasons. Firstly, because my view is the same as Victor's: the time between the last rc and the final release should be completely uninteresting, with no significant regressions reported relative to the previous major version (or any such cases clearly being rare enough to wait for the first maintenance release). Secondly, I care because I think we need to take into account the social benefits of ensuring that we treat third party testers as valued members of the release process by giving them a clear chance to confirm that the problems they reported have been addressed before we proceed with the release. That third party testing does help improve the stability of the final release, and wherever practical, we should be doing our best to encourage it. For 3.4rc2, the Alembic and Flask test suites both hit clear regression bugs (one due to pkgutil still calling a now deprecated function, the other due to a type slot publishing an incorrect signature), and users also found that upgrading pip on Windows would prevent subsequently uninstalling CPython. I can politely ask Mike and Armin to test against a pre-release tarball, or against the default branch and assure them the patch has been included in the release tag, but I have no reasonable answer to give them if they ask "why not just publish an rc3 to make this easy to test?". For the folks that reported the Windows installer issues, we can't ask them to double check the fixes at all if we don't do an rc3. Regards, Nick. > > Matthias > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Mar 5 00:59:58 2014 From: barry at python.org (Barry Warsaw) Date: Tue, 4 Mar 2014 18:59:58 -0500 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> <20140304111641.Horde.WTUtNWKjH-nv8-JUkM3R2w2@webmail.df.eu> <20140304143356.2ac47d37@fsol> <53164DEE.1000904@ubuntu.com> Message-ID: <20140304185958.5b5bff17@limelight.wooz.org> On Mar 05, 2014, at 09:24 AM, Nick Coghlan wrote: >I think it's also the fact that new feature releases are rare and changes >of release manager even more so, meaning there's a fair bit of relearning >involved every time (since what was appropriate a couple of years earlier >may not be appropriate the next time, and we'll usually have new developers >involved that weren't around for the previous release). This is true. Be aware though that Larry is in communication with Benjamin, Georg, and myself, and the PSU is running a closed mailing list for past, present, and future release managers[1]. But you're right that the process and tools can change fairly significantly between releases, and certainly between release manager stints. We have PEP 101 and a bunch of scripts to automate as much as possible. I'm fully supportive of RMs taking on at least two releases in a row, for a similar reason it makes sense to hold PyconNA's in the same city more than one year in a row. Larry has to figure out what works for him on the fly, and I'm positive that he takes everyone's comments and feedback seriously. RMing is just not that easy. Hopefully, once the heat of the release subsides, we and Larry can review the process and make whatever changes are necessary for next time. >While I'd still strongly prefer an rc3, I think we at least need to get the >remaining release blockers sorted in the tracker (either fixing them or >reclassifying them, or else closing them if they're a rejected cherry pick >request) and a tarball or release clone available for core developer and >third party testing. I too would like an rc3, especially to see if issue 19021 can be fixed, which I suspect will hit a lot of people. Despite Serhiy's comment (msg212475) I haven't quite gotten the Ubuntu package working entirely, and I hope to return to it once I have some other unrelated high priority issues resolved. It's fine to close as rejected any cherry picks that won't make it into 3.4.0. For the related bugs that are fixed in trunk though, those marked as release blockers should IMO be demoted to deferred blockers so they can be handled in 3.4.1. (Unless of course the disposition is to not fix them until 3.5). >I'm being fussy about this for two reasons. Firstly, because my view is the >same as Victor's: the time between the last rc and the final release should >be completely uninteresting, with no significant regressions reported >relative to the previous major version (or any such cases clearly being >rare enough to wait for the first maintenance release). I completely agree. I don't even trust seemingly innocent changes -- they have broken us before! Ideally, the only difference between the last rc and the final release is the version bump. That may or may not be possible. >Secondly, I care because I think we need to take into account the social >benefits of ensuring that we treat third party testers as valued members of >the release process by giving them a clear chance to confirm that the >problems they reported have been addressed before we proceed with the >release. That third party testing does help improve the stability of the >final release, and wherever practical, we should be doing our best to >encourage it. Given how difficult it is to get people to test pre-final release versions, I definitely agree we need to encourage and support enthusiastic third parties. We all want to avoid the brown paper bag release. ;) Cheers, -Barry [1] Both the PSU and this mailing list emphatically do not exist. From larry at hastings.org Wed Mar 5 03:46:46 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 04 Mar 2014 18:46:46 -0800 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: <20140304185958.5b5bff17@limelight.wooz.org> References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> <20140304111641.Horde.WTUtNWKjH-nv8-JUkM3R2w2@webmail.df.eu> <20140304143356.2ac47d37@fsol> <53164DEE.1000904@ubuntu.com> <20140304185958.5b5bff17@limelight.wooz.org> Message-ID: <53169016.5050802@hastings.org> On 03/04/2014 03:59 PM, Barry Warsaw wrote: > I too would like an rc3, especially to see if issue 19021 can be fixed, which > I suspect will hit a lot of people. I talked to the other guys on the 3.4 team, and we're all willing to do an rc3 this weekend. I'll add that to PEP 429. In other news, I'm thrilled to confirm something Antoine mentioned a week or two ago: it is literally impossible for garden-variety core devs to push new branches back into trunk. I tried to, early this morning (PST), with someone logged in to hg.python.org ready to clean up in case it actually worked. But happily hg hooks on the server reject new branches every time. Since this banishes my chief objection to publishing the repo where I do my cherry picking, I'll be publishing that repo this week. I think I still have a rebase ahead of me, so I'm going wait until after the latest (and hopefully last!) round of cherry-picking. I'll post to python-dev once the tree is public. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Mar 5 14:32:11 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 5 Mar 2014 23:32:11 +1000 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: <53169016.5050802@hastings.org> References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> <20140304111641.Horde.WTUtNWKjH-nv8-JUkM3R2w2@webmail.df.eu> <20140304143356.2ac47d37@fsol> <53164DEE.1000904@ubuntu.com> <20140304185958.5b5bff17@limelight.wooz.org> <53169016.5050802@hastings.org> Message-ID: On 5 Mar 2014 12:48, "Larry Hastings" wrote: > > On 03/04/2014 03:59 PM, Barry Warsaw wrote: >> >> I too would like an rc3, especially to see if issue 19021 can be fixed, which >> I suspect will hit a lot of people. > > > I talked to the other guys on the 3.4 team, and we're all willing to do an rc3 this weekend. I'll add that to PEP 429. Thanks Larry, I just saw that commit. It's genuinely appreciated! :) > In other news, I'm thrilled to confirm something Antoine mentioned a week or two ago: it is literally impossible for garden-variety core devs to push new branches back into trunk. I tried to, early this morning (PST), with someone logged in to hg.python.org ready to clean up in case it actually worked. But happily hg hooks on the server reject new branches every time. Ah, *now* I understand your concern - yes, that hook was put in place during the initial migration to Mercurial, but if you weren't aware of it, then your desire to keep the release clone offline to prevent erroneous pushes makes a lot more sense to me :) > Since this banishes my chief objection to publishing the repo where I do my cherry picking, I'll be publishing that repo this week. I think I still have a rebase ahead of me, so I'm going wait until after the latest (and hopefully last!) round of cherry-picking. I'll post to python-dev once the tree is public. Sweet, thanks! Cheers, Nick. > > > /arry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Mar 5 16:30:21 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 05 Mar 2014 16:30:21 +0100 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: <53169016.5050802@hastings.org> References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> <20140304111641.Horde.WTUtNWKjH-nv8-JUkM3R2w2@webmail.df.eu> <20140304143356.2ac47d37@fsol> <53164DEE.1000904@ubuntu.com> <20140304185958.5b5bff17@limelight.wooz.org> <53169016.5050802@hastings.org> Message-ID: Le 05/03/2014 03:46, Larry Hastings a ?crit : > On 03/04/2014 03:59 PM, Barry Warsaw wrote: >> I too would like an rc3, especially to see if issue 19021 can be fixed, which >> I suspect will hit a lot of people. > > I talked to the other guys on the 3.4 team, and we're all willing to do > an rc3 this weekend. I'll add that to PEP 429. > > > In other news, I'm thrilled to confirm something Antoine mentioned a > week or two ago: it is literally impossible for garden-variety core devs > to push new branches back into trunk. I tried to, early this morning > (PST), with someone logged in to hg.python.org ready to clean up in case > it actually worked. But happily hg hooks on the server reject new > branches every time. And I thought the hook would be overkill! Regards Antoine. From victor.stinner at gmail.com Wed Mar 5 17:37:12 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 5 Mar 2014 17:37:12 +0100 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ Message-ID: Hi, Python 3 now stores the traceback object in Exception.__traceback__ and exceptions can be chained through Exception.__context__. It's convinient but it introduced tricky reference cycles if the exception object is used out of the except block. Refrences: Exception.__traceback__ -> traceback -> frames -> local variables Exception.__traceback__ keeps strong references to local variables which will only be deleted after the exception is destroyed. It becomes worse if the exception is stored in an object which is also a local variables somewhere in the traceback: DebugObject -> Exception -> ... frame -> DebugObject For a concrete example of object storing an exception, see Future.set_exception() of the ayncio module. Python 3.4 introduced frame.clear(), but frame.clear() raises an RuntimeError if the frame is still running. And it doesn't break all reference cycles. An obvious workaround is to store the traceback as text, but this operation is "expensive" especially if the traceback is only needed in rare cases. I tried to write "views" of the traceback (and frames), but Exception.__traceback__ rejects types other than traceback and traceback instances cannot be created. It's possible to store the traceback somewhere else and set Exception.__traceback__ to None, but there is still the problem with chained exceptions. Any idea for a generic fix to such problem? Another workaround is to set Exception.__traceback__ to None in release mode, and leaves it unchanged in debug mode. But it's not a good solution because bugs also occur in production, and most tricky bugs *only* occur in production :-) For more info, see: http://bugs.python.org/issue20032 http://code.google.com/p/tulip/issues/detail?id=42 http://code.google.com/p/tulip/issues/detail?id=155 Victor From solipsis at pitrou.net Wed Mar 5 17:41:42 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 05 Mar 2014 17:41:42 +0100 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: Message-ID: Le 05/03/2014 17:37, Victor Stinner a ?crit : > > Python 3.4 introduced frame.clear(), but frame.clear() raises an > RuntimeError if the frame is still running. And it doesn't break all > reference cycles. > > An obvious workaround is to store the traceback as text, but this > operation is "expensive" especially if the traceback is only needed in > rare cases. Another "obvious" workaround is to call frame.clear() from the parent function (i.e. have a dedicated wrapping layer). Regards Antoine. From yselivanov.ml at gmail.com Wed Mar 5 17:51:42 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 05 Mar 2014 11:51:42 -0500 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: Message-ID: <5317561E.8090301@gmail.com> On 2014-03-05, 11:37 AM, Victor Stinner wrote: > [snip] > > I tried to write "views" of the traceback (and frames), but > Exception.__traceback__ rejects types other than traceback and > traceback instances cannot be created. It's possible to store the > traceback somewhere else and set Exception.__traceback__ to None, but > there is still the problem with chained exceptions. > Can we allow instantiation of types.FrameType and types.TracebackType? If it's allowed, we can write a simple code, that will recreate tracebacks with cleaned-up frames without references to locals etc. Yury From thomas at python.org Wed Mar 5 21:57:03 2014 From: thomas at python.org (Thomas Wouters) Date: Wed, 5 Mar 2014 12:57:03 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> Message-ID: On Thu, Feb 27, 2014 at 1:29 PM, Chris Angelico wrote: > +Had this facility existed early in Python's history, there would have been > +no need to create dict.get() and related methods; FWIW, after experimenting and some consideration I've come to the conclusion that this is incorrect. 'd[k] except KeyError: default' is still much broader than dict.get(k): Python 3.4.0rc1+ (default:aa2ae744e701+, Feb 24 2014, 01:22:15) [GCC 4.6.3] on linux Type "help", "copyright", "credits" or "license" for more information. >>> expensive_calculation = hash >>> class C: ... _hash_cache = {} ... def __init__(self, value): ... self.value = value ... if value not in self._hash_cache: ... self._hash_cache[value] = expensive_calculation(value) ... def __hash__(self): ... return self._hash_cache[self.value] ... def __eq__(self, other): ... return self.value == other ... >>> a, b, c, d = C(1), C(2), C(3), C(4) >>> D = {a: 1, b: 2, c: 3, d: 4} >>> >>> a.value = 5 >>> print("except expr:", (D[a] except KeyError: 'default')) except expr: default >>> print("dict.get:", D.get(a, 'default')) Traceback (most recent call last): File "", line 1, in File "", line 8, in __hash__ KeyError: 5 All in all I believe I will continue to prefer specific methods for specific use-cases; I'm -0 on the idea of an except-expression, -0 on the syntax with the mandatory parentheses around the whole thing (and so far -1 on any of the other suggested forms.) I can see the attractiveness, but frankly, all the suggested changes to the stdlib fall in two categories: easier to express (and narrower in its exception handling) with e.g. dict.get for the trivial ones, or much better written out using temporary variables for the complex ones. As soon as an except-expression has trouble fitting on two lines it becomes an unsightly mess; it no longer becomes obvious what it does from a glance. Not having except-expr may mean we keep adding methods (with different names, and slightly different semantics) to cover specific use-cases of specific types, but I can live with that. -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Mar 5 22:50:59 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 6 Mar 2014 07:50:59 +1000 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> <20140304111641.Horde.WTUtNWKjH-nv8-JUkM3R2w2@webmail.df.eu> <20140304143356.2ac47d37@fsol> <53164DEE.1000904@ubuntu.com> <20140304185958.5b5bff17@limelight.wooz.org> <53169016.5050802@hastings.org> Message-ID: On 6 Mar 2014 01:32, "Antoine Pitrou" wrote: > > Le 05/03/2014 03:46, Larry Hastings a ?crit : > >> On 03/04/2014 03:59 PM, Barry Warsaw wrote: >>> >>> I too would like an rc3, especially to see if issue 19021 can be fixed, which >>> I suspect will hit a lot of people. >> >> >> I talked to the other guys on the 3.4 team, and we're all willing to do >> an rc3 this weekend. I'll add that to PEP 429. >> >> >> In other news, I'm thrilled to confirm something Antoine mentioned a >> week or two ago: it is literally impossible for garden-variety core devs >> to push new branches back into trunk. I tried to, early this morning >> (PST), with someone logged in to hg.python.org ready to clean up in case >> it actually worked. But happily hg hooks on the server reject new >> branches every time. > > > And I thought the hook would be overkill! I found it to be quite a handy failsafe in the early days of using Mercurial. Haven't triggered it since I set up my separate sandbox repo, though :) Cheers, Nick. > > Regards > > Antoine. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Mar 5 22:54:42 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 6 Mar 2014 07:54:42 +1000 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: Message-ID: On 6 Mar 2014 02:43, "Antoine Pitrou" wrote: > > Le 05/03/2014 17:37, Victor Stinner a ?crit : > >> >> Python 3.4 introduced frame.clear(), but frame.clear() raises an >> RuntimeError if the frame is still running. And it doesn't break all >> reference cycles. >> >> An obvious workaround is to store the traceback as text, but this >> operation is "expensive" especially if the traceback is only needed in >> rare cases. > > > Another "obvious" workaround is to call frame.clear() from the parent function (i.e. have a dedicated wrapping layer). Guido had a fix in mind - splitting traceback formatting into two pieces. The first would convert a traceback to a sequence of named tuples, the second would do the actual string formatting. There's an RFE for it on the tracker, but I don't have a link handy. Cheers, Nick. > > Regards > > Antoine. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Mar 5 23:12:33 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Mar 2014 14:12:33 -0800 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: Message-ID: I think it was http://bugs.python.org/issue17911 On Wed, Mar 5, 2014 at 1:54 PM, Nick Coghlan wrote: > > On 6 Mar 2014 02:43, "Antoine Pitrou" wrote: > > > > Le 05/03/2014 17:37, Victor Stinner a ?crit : > > > >> > >> Python 3.4 introduced frame.clear(), but frame.clear() raises an > >> RuntimeError if the frame is still running. And it doesn't break all > >> reference cycles. > >> > >> An obvious workaround is to store the traceback as text, but this > >> operation is "expensive" especially if the traceback is only needed in > >> rare cases. > > > > > > Another "obvious" workaround is to call frame.clear() from the parent > function (i.e. have a dedicated wrapping layer). > > Guido had a fix in mind - splitting traceback formatting into two pieces. > The first would convert a traceback to a sequence of named tuples, the > second would do the actual string formatting. > > There's an RFE for it on the tracker, but I don't have a link handy. > > Cheers, > Nick. > > > > > Regards > > > > Antoine. > > > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Wed Mar 5 23:17:07 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 5 Mar 2014 15:17:07 -0700 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: Message-ID: On Wed, Mar 5, 2014 at 2:54 PM, Nick Coghlan wrote: > Guido had a fix in mind - splitting traceback formatting into two pieces. > The first would convert a traceback to a sequence of named tuples, the > second would do the actual string formatting. +1 This is basically how I've handled pickling tracebacks in a project at work (using 2.7). You can already using extract_tb() and format_list() from the traceback module to get the effect. As to how to use this to avoid holding those references (without taking a performance hit), that will take some more thought. :) -eric From yselivanov.ml at gmail.com Wed Mar 5 23:31:01 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 05 Mar 2014 17:31:01 -0500 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: Message-ID: <5317A5A5.5000305@gmail.com> On 2014-03-05, 4:54 PM, Nick Coghlan wrote: > On 6 Mar 2014 02:43, "Antoine Pitrou" wrote: >> Le 05/03/2014 17:37, Victor Stinner a ?crit : >> >>> Python 3.4 introduced frame.clear(), but frame.clear() raises an >>> RuntimeError if the frame is still running. And it doesn't break all >>> reference cycles. >>> >>> An obvious workaround is to store the traceback as text, but this >>> operation is "expensive" especially if the traceback is only needed in >>> rare cases. >> >> Another "obvious" workaround is to call frame.clear() from the parent > function (i.e. have a dedicated wrapping layer). > > Guido had a fix in mind - splitting traceback formatting into two pieces. > The first would convert a traceback to a sequence of named tuples, the > second would do the actual string formatting. I don't like the idea of using tuples instead of tracebacks in 'exception.__traceback__'. Lots of introspection and logging tools except to find tracebacks in exceptions, and do custom formatting or submitting data to remote servers for further analysis (sometimes with values of local variables in frames). If asyncio starts stripping exception.__traceback__ and using namedtuples, it will make it harder to write/integrate such tools. Yury From ncoghlan at gmail.com Wed Mar 5 23:53:07 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 6 Mar 2014 08:53:07 +1000 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: <5317A5A5.5000305@gmail.com> References: <5317A5A5.5000305@gmail.com> Message-ID: On 6 Mar 2014 08:32, "Yury Selivanov" wrote: > > > On 2014-03-05, 4:54 PM, Nick Coghlan wrote: >> >> On 6 Mar 2014 02:43, "Antoine Pitrou" wrote: >>> >>> Le 05/03/2014 17:37, Victor Stinner a ?crit : >>> >>>> Python 3.4 introduced frame.clear(), but frame.clear() raises an >>>> RuntimeError if the frame is still running. And it doesn't break all >>>> reference cycles. >>>> >>>> An obvious workaround is to store the traceback as text, but this >>>> operation is "expensive" especially if the traceback is only needed in >>>> rare cases. >>> >>> >>> Another "obvious" workaround is to call frame.clear() from the parent >> >> function (i.e. have a dedicated wrapping layer). >> >> Guido had a fix in mind - splitting traceback formatting into two pieces. >> The first would convert a traceback to a sequence of named tuples, the >> second would do the actual string formatting. > > > I don't like the idea of using tuples instead of tracebacks > in 'exception.__traceback__'. Lots of introspection and logging > tools except to find tracebacks in exceptions, and do custom > formatting or submitting data to remote servers for further > analysis (sometimes with values of local variables in frames). > If asyncio starts stripping exception.__traceback__ and using > namedtuples, it will make it harder to write/integrate such > tools. __traceback__ wouldn't change - this is a variant on Victor's point that it is currently easy to store the already formatted traceback as a string, but that wastes time formatting a string you may never display, and is hard to manipulate programmatically. Extracting a traceback summary from __traceback__ instead can address both limitations. Cheers, Nick. > > Yury > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Mar 6 01:22:00 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 6 Mar 2014 11:22:00 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> Message-ID: On Thu, Mar 6, 2014 at 7:57 AM, Thomas Wouters wrote: > On Thu, Feb 27, 2014 at 1:29 PM, Chris Angelico wrote: >> >> +Had this facility existed early in Python's history, there would have >> been >> +no need to create dict.get() and related methods; > > > FWIW, after experimenting and some consideration I've come to the conclusion > that this is incorrect. 'd[k] except KeyError: default' is still much > broader than dict.get(k): *Much* broader? You prove that it's broader, yes, but most types aren't defining __hash__ methods that can fail with KeyError. Can you show any real-world code that trips this? I'd say the broadened exception scope (or, putting it the other way, the narrowed exception scope of adding dict.get() after except-expressions had already existed) is insufficiently significant to justify adding an extra method to the dict. Since the method does exist, it will continue to be useful, but if except expressions did and dict.get() didn't, there'd have been very little justification for them. And certainly hasattr() wouldn't need to exist, since it exactly _does_ try to get the attribute and see if AttributeError is raised. ChrisA From rosuav at gmail.com Thu Mar 6 01:28:31 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 6 Mar 2014 11:28:31 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> Message-ID: On Thu, Mar 6, 2014 at 7:57 AM, Thomas Wouters wrote: > All in all I believe I will continue to prefer specific methods for specific > use-cases; I'm -0 on the idea of an except-expression, -0 on the syntax with > the mandatory parentheses around the whole thing (and so far -1 on any of > the other suggested forms.) I can see the attractiveness, but frankly, all > the suggested changes to the stdlib fall in two categories: easier to > express (and narrower in its exception handling) with e.g. dict.get for the > trivial ones, or much better written out using temporary variables for the > complex ones. As soon as an except-expression has trouble fitting on two > lines it becomes an unsightly mess; it no longer becomes obvious what it > does from a glance. Not having except-expr may mean we keep adding methods > (with different names, and slightly different semantics) to cover specific > use-cases of specific types, but I can live with that. Most of those concerns could also be aimed at the if/else expression. There are definitely places that do not merit its use, but that doesn't mean the feature is a bad one. The PEP has a number of examples that fit quite happily on a single line, and it's those that I'm advocating. We have comprehensions, genexps, etc, etc, all (or at least most) of which can be written out in some form of long-hand, and it's usually better to use the short-hand - it's not just laziness, it's expressiveness. Speaking of the PEP, if someone could apply the latest changes, I think it's pretty much ready for pronouncement. Thanks! Content: https://raw.github.com/Rosuav/ExceptExpr/master/pep-0463.txt Diff from current peps repo: diff -r 2cf89e9e50a3 pep-0463.txt --- a/pep-0463.txt Tue Mar 04 18:47:44 2014 -0800 +++ b/pep-0463.txt Thu Mar 06 11:12:44 2014 +1100 @@ -250,7 +250,8 @@ alternatives listed above must (by the nature of functions) evaluate their default values eagerly. The preferred form, using the colon, parallels try/except by using "except exception_list:", and parallels lambda by having -"keyword name_list: subexpression". Using the arrow introduces a token many +"keyword name_list: subexpression"; it also can be read as mapping Exception +to the default value, dict-style. Using the arrow introduces a token many programmers will not be familiar with, and which currently has no similar meaning, but is otherwise quite readable. The English word "pass" has a vaguely similar meaning (consider the common usage "pass by value/reference" @@ -271,6 +272,18 @@ Using the preferred order, subexpressions will always be evaluated from left to right, no matter how the syntax is nested. +Keeping the existing notation, but shifting the mandatory parentheses, we +have the following suggestion:: + + value = expr except (Exception: default) + value = expr except(Exception: default) + +This is reminiscent of a function call, or a dict initializer. The colon +cannot be confused with introducing a suite, but on the other hand, the new +syntax guarantees lazy evaluation, which a dict does not. The potential +to reduce confusion is considered unjustified by the corresponding potential +to increase it. + Example usage ============= @@ -854,6 +867,32 @@ expression to achieve this. +Common objections +================= + +Colons always introduce suites +------------------------------ + +While it is true that many of Python's syntactic elements use the colon to +introduce a statement suite (if, while, with, for, etcetera), this is not +by any means the sole use of the colon. Currently, Python syntax includes +four cases where a colon introduces a subexpression: + +* dict display - { ... key:value ... } +* slice notation - [start:stop:step] +* function definition - parameter : annotation +* lambda - arg list: return value + +This proposal simply adds a fifth: + +* except-expression - exception list: result + +Style guides and PEP 8 should recommend not having the colon at the end of +a wrapped line, which could potentially look like the introduction of a +suite, but instead advocate wrapping before the exception list, keeping the +colon clearly between two expressions. + + Copyright ========= ChrisA From thomas at python.org Thu Mar 6 02:03:45 2014 From: thomas at python.org (Thomas Wouters) Date: Wed, 5 Mar 2014 17:03:45 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> Message-ID: On Wed, Mar 5, 2014 at 4:28 PM, Chris Angelico wrote: > On Thu, Mar 6, 2014 at 7:57 AM, Thomas Wouters wrote: > > All in all I believe I will continue to prefer specific methods for > specific > > use-cases; I'm -0 on the idea of an except-expression, -0 on the syntax > with > > the mandatory parentheses around the whole thing (and so far -1 on any of > > the other suggested forms.) I can see the attractiveness, but frankly, > all > > the suggested changes to the stdlib fall in two categories: easier to > > express (and narrower in its exception handling) with e.g. dict.get for > the > > trivial ones, or much better written out using temporary variables for > the > > complex ones. As soon as an except-expression has trouble fitting on two > > lines it becomes an unsightly mess; it no longer becomes obvious what it > > does from a glance. Not having except-expr may mean we keep adding > methods > > (with different names, and slightly different semantics) to cover > specific > > use-cases of specific types, but I can live with that. > > Most of those concerns could also be aimed at the if/else expression. > And I did :) But the if-else expression had a single thing going for it, the thing that landed the actual feature: it prevents people from using buggy alternatives in the quest for short code, the and-or trick. You may remember that Guido initially rejected the if-else expression, until he realized people were going to use something like it whether he liked it or not. The except-expression has a different issue: people catching exceptions too broadly. I don't believe that's as big a problem, nor is it as subtly wrong. And the except-expression doesn't solve _all_ such issues, just a very small subset. It's just another thing to learn when you're new, and just another thing to consider when reviewing code. > There are definitely places that do not merit its use, but that > doesn't mean the feature is a bad one. The PEP has a number of > examples that fit quite happily on a single line, and it's those that > I'm advocating. We have comprehensions, genexps, etc, etc, all (or at > least most) of which can be written out in some form of long-hand, and > it's usually better to use the short-hand - it's not just laziness, > it's expressiveness. > It's not a question of it being expressive, it's a question of it being worth separate syntax. I don't think it's worth syntax. > > Speaking of the PEP, if someone could apply the latest changes, I > think it's pretty much ready for pronouncement. Thanks! > PEP update pushed (changeset 59653081cdf6.) > > Content: > https://raw.github.com/Rosuav/ExceptExpr/master/pep-0463.txt > > Diff from current peps repo: > > diff -r 2cf89e9e50a3 pep-0463.txt > --- a/pep-0463.txt Tue Mar 04 18:47:44 2014 -0800 > +++ b/pep-0463.txt Thu Mar 06 11:12:44 2014 +1100 > @@ -250,7 +250,8 @@ > alternatives listed above must (by the nature of functions) evaluate their > default values eagerly. The preferred form, using the colon, parallels > try/except by using "except exception_list:", and parallels lambda by > having > -"keyword name_list: subexpression". Using the arrow introduces a token > many > +"keyword name_list: subexpression"; it also can be read as mapping > Exception > +to the default value, dict-style. Using the arrow introduces a token many > programmers will not be familiar with, and which currently has no similar > meaning, but is otherwise quite readable. The English word "pass" has a > vaguely similar meaning (consider the common usage "pass by > value/reference" > @@ -271,6 +272,18 @@ > Using the preferred order, subexpressions will always be evaluated from > left to right, no matter how the syntax is nested. > > +Keeping the existing notation, but shifting the mandatory parentheses, we > +have the following suggestion:: > + > + value = expr except (Exception: default) > + value = expr except(Exception: default) > + > +This is reminiscent of a function call, or a dict initializer. The colon > +cannot be confused with introducing a suite, but on the other hand, the > new > +syntax guarantees lazy evaluation, which a dict does not. The potential > +to reduce confusion is considered unjustified by the corresponding > potential > +to increase it. > + > > Example usage > ============= > @@ -854,6 +867,32 @@ > expression to achieve this. > > > +Common objections > +================= > + > +Colons always introduce suites > +------------------------------ > + > +While it is true that many of Python's syntactic elements use the colon to > +introduce a statement suite (if, while, with, for, etcetera), this is not > +by any means the sole use of the colon. Currently, Python syntax includes > +four cases where a colon introduces a subexpression: > + > +* dict display - { ... key:value ... } > +* slice notation - [start:stop:step] > +* function definition - parameter : annotation > +* lambda - arg list: return value > + > +This proposal simply adds a fifth: > + > +* except-expression - exception list: result > + > +Style guides and PEP 8 should recommend not having the colon at the end of > +a wrapped line, which could potentially look like the introduction of a > +suite, but instead advocate wrapping before the exception list, keeping > the > +colon clearly between two expressions. > + > + > Copyright > ========= > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/thomas%40python.org > -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Mar 6 02:10:38 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 6 Mar 2014 12:10:38 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> Message-ID: On Thu, Mar 6, 2014 at 12:03 PM, Thomas Wouters wrote: > PEP update pushed (changeset 59653081cdf6.) > Thanks for that! ChrisA From steve at pearwood.info Thu Mar 6 02:15:38 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 6 Mar 2014 12:15:38 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> Message-ID: <20140306011538.GE28804@ando> On Wed, Mar 05, 2014 at 12:57:03PM -0800, Thomas Wouters wrote: > On Thu, Feb 27, 2014 at 1:29 PM, Chris Angelico wrote: > > > +Had this facility existed early in Python's history, there would have been > > +no need to create dict.get() and related methods; > > > FWIW, after experimenting and some consideration I've come to the > conclusion that this is incorrect. 'd[k] except KeyError: default' is still > much broader than dict.get(k): I don't think your example proves what you think it does. I think it demonstrates a bug in the dict.get method. The documentation for get states clearly that get will never raise KeyError: Return the value for key if key is in the dictionary, else default. If default is not given, it defaults to None, so that this method never raises a KeyError. http://docs.python.org/3/library/stdtypes.html#dict.get but your example demonstrates that in fact it can raise KeyError (albeit under some rather unusual circumstances): > Python 3.4.0rc1+ (default:aa2ae744e701+, Feb 24 2014, 01:22:15) > [GCC 4.6.3] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> expensive_calculation = hash > >>> class C: > ... _hash_cache = {} > ... def __init__(self, value): > ... self.value = value > ... if value not in self._hash_cache: > ... self._hash_cache[value] = expensive_calculation(value) > ... def __hash__(self): > ... return self._hash_cache[self.value] > ... def __eq__(self, other): > ... return self.value == other > ... > >>> a, b, c, d = C(1), C(2), C(3), C(4) > >>> D = {a: 1, b: 2, c: 3, d: 4} > >>> a.value = 5 > >>> print("except expr:", (D[a] except KeyError: 'default')) > except expr: default > >>> print("dict.get:", D.get(a, 'default')) > Traceback (most recent call last): > File "", line 1, in > File "", line 8, in __hash__ > KeyError: 5 According to the documentation, this behaviour is wrong. Now, you might argue that the documentation is wrong. I'm sympathetic to that argument, but *as documented now*, dict.get is documented as being logically equivalent to: try: return d[key] except KeyError: return default The fact that it actually isn't is an artifact of the specific implementation used. If it were a deliberate design choice, that design is not reflected in the documentation. Whether the current behaviour is wrong, or the documentation is wrong, is irrelevant to the question of whether or not the developers back in nineteen-ninety-whatever would have choosen to add dict.get had there been syntax for catching the KeyError in an expression. Perhaps they would have argued: "Sure, you can catch the KeyError yourself, but 'get' is a fundamental operation for mappings, and I think that dict should implement a 'get' method just to be complete." Or perhaps not. Some developers prefer minimalist APIs, some developers prefer more exhaustive APIs. Regardless of what might have happened back in 199x when dict.get was first discussed, I think we can agree that an except expression will lower the pressure on Python to add *more* "get-like" methods, or add default arguments, in the future. -- Steven From tjreedy at udel.edu Thu Mar 6 04:29:44 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 05 Mar 2014 22:29:44 -0500 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140306011538.GE28804@ando> References: <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> <20140306011538.GE28804@ando> Message-ID: On 3/5/2014 8:15 PM, Steven D'Aprano wrote: > On Wed, Mar 05, 2014 at 12:57:03PM -0800, Thomas Wouters wrote: >> On Thu, Feb 27, 2014 at 1:29 PM, Chris Angelico wrote: >> >>> +Had this facility existed early in Python's history, there would have been >>> +no need to create dict.get() and related methods; >> >> >> FWIW, after experimenting and some consideration I've come to the >> conclusion that this is incorrect. 'd[k] except KeyError: default' is still >> much broader than dict.get(k): > > I don't think your example proves what you think it does. I think it > demonstrates a bug in the dict.get method. The documentation for get > states clearly that get will never raise KeyError: > > Return the value for key if key is in the dictionary, else default. > If default is not given, it defaults to None, so that this method > never raises a KeyError. > > http://docs.python.org/3/library/stdtypes.html#dict.get > > > but your example demonstrates that in fact it can raise KeyError > (albeit under some rather unusual circumstances): >> Python 3.4.0rc1+ (default:aa2ae744e701+, Feb 24 2014, 01:22:15) >> [GCC 4.6.3] on linux >> Type "help", "copyright", "credits" or "license" for more information. >>>>> expensive_calculation = hash >>>>> class C: >> ... _hash_cache = {} >> ... def __init__(self, value): >> ... self.value = value >> ... if value not in self._hash_cache: >> ... self._hash_cache[value] = expensive_calculation(value) >> ... def __hash__(self): >> ... return self._hash_cache[self.value] This is a buggy special method. According to the docs for hash and __hash__ and the general convention on exceptions, a __hash__ method should return an int or raise TypeError. >> ... def __eq__(self, other): >> ... return self.value == other >> ... >>>>> a, b, c, d = C(1), C(2), C(3), C(4) >>>>> D = {a: 1, b: 2, c: 3, d: 4} >>>>> a.value = 5 This breaks the implied C invariant and makes the object 'a' incoherent and buggy >>>>> print("dict.get:", D.get(a, 'default')) >> Traceback (most recent call last): >> File "", line 1, in >> File "", line 8, in __hash__ >> KeyError: 5 > According to the documentation, this behaviour is wrong. One could argue that an error raised in a special method is not raised *by* a function that uses the special method. The docs constantly assume that special methods are coded correctly. '''bool([x]) Convert a value to a Boolean, using the standard truth testing procedure. If x is false or omitted, this returns False; otherwise it returns True.''' ... unless x.__bool__ raises or returns something other than True/False -- in which case bool itself raises. TypeError: __bool__ should return bool, returned int > Now, you might argue that the documentation is wrong. I'm sympathetic to > that argument, but *as documented now*, dict.get is documented as being > logically equivalent to: > > try: > return d[key] > except KeyError: > return default It appears to be actually equivalent to key_hash = hash(key) try: return d._hashlookup(key_has) except KeyError: return default > The fact that it actually isn't is an artifact of the specific > implementation used. If it were a deliberate design choice, Given that the choice is that bugs in special methods should not pass silently, I would presume that it is intentional. > that design is not reflected in the documentation. The docs generally describe behavior in the absence of coding errors. -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Thu Mar 6 06:47:28 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 06 Mar 2014 18:47:28 +1300 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140306011538.GE28804@ando> References: <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> <20140306011538.GE28804@ando> Message-ID: <53180BF0.4090203@canterbury.ac.nz> Steven D'Aprano wrote: > Return the value for key if key is in the dictionary, else default. > If default is not given, it defaults to None, so that this method > never raises a KeyError. I think that's supposed to mean that it won't raise KeyError as a result of the key not being in the dictionary. The actual behaviour is correct, IMO, because it avoids masking bugs, so this could probably be worded better. -- Greg From storchaka at gmail.com Thu Mar 6 09:02:52 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 06 Mar 2014 10:02:52 +0200 Subject: [Python-Dev] cpython (3.3): Make the various iterators' "setstate" sliently and consistently clip the In-Reply-To: <3ffJmG2K2Yz7Llf@mail.python.org> References: <3ffJmG2K2Yz7Llf@mail.python.org> Message-ID: 05.03.14 17:24, kristjan.jonsson ???????(??): > http://hg.python.org/cpython/rev/3b2c28061184 > changeset: 89477:3b2c28061184 > branch: 3.3 > parent: 89475:24d4e52f4f87 > user: Kristj?n Valur J?nsson > date: Wed Mar 05 13:47:57 2014 +0000 > summary: > Make the various iterators' "setstate" sliently and consistently clip the > index. This avoids the possibility of setting an iterator to an invalid > state. Why indexes are silently clipped instead of raising an exception? > files: > Lib/test/test_range.py | 12 ++++++++++ > Modules/arraymodule.c | 2 + > Objects/bytearrayobject.c | 10 ++++++-- > Objects/bytesobject.c | 10 ++++++-- > Objects/listobject.c | 2 + > Objects/rangeobject.c | 31 +++++++++++++++++++++++--- > Objects/tupleobject.c | 4 +- > Objects/unicodeobject.c | 10 ++++++-- > 8 files changed, 66 insertions(+), 15 deletions(-) And it would be better first discuss and review such large changes on the bugtracker. From solipsis at pitrou.net Thu Mar 6 14:42:44 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 06 Mar 2014 14:42:44 +0100 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: <5317A5A5.5000305@gmail.com> Message-ID: Le 05/03/2014 23:53, Nick Coghlan a ?crit : > > __traceback__ wouldn't change [...] Uh, really? If you want to suppress all reference cycles, you *have* to remove __traceback__. The problem is to make computation of the traceback summary lightweight enough that it doesn't degrade performance in the common case where you don't have to print the traceback later. Regards Antoine. From yselivanov.ml at gmail.com Thu Mar 6 16:03:09 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Thu, 06 Mar 2014 10:03:09 -0500 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: <5317A5A5.5000305@gmail.com> Message-ID: <53188E2D.6020904@gmail.com> On 2014-03-06, 8:42 AM, Antoine Pitrou wrote: > Le 05/03/2014 23:53, Nick Coghlan a ?crit : >> >> __traceback__ wouldn't change [...] > > Uh, really? If you want to suppress all reference cycles, you *have* > to remove __traceback__. > > The problem is to make computation of the traceback summary > lightweight enough that it doesn't degrade performance in the common > case where you don't have to print the traceback later. So why can't we allow instantiation of types.TracebackType & types.FrameType? It should be about the same time to reconstruct traceback and its frames without locals, as to create named tuples. Yury From victor.stinner at gmail.com Thu Mar 6 16:19:14 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 6 Mar 2014 16:19:14 +0100 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: <5317A5A5.5000305@gmail.com> Message-ID: 2014-03-06 14:42 GMT+01:00 Antoine Pitrou : > Le 05/03/2014 23:53, Nick Coghlan a ?crit : >> >> >> __traceback__ wouldn't change [...] > > > Uh, really? If you want to suppress all reference cycles, you *have* to > remove __traceback__. > > The problem is to make computation of the traceback summary lightweight > enough that it doesn't degrade performance in the common case where you > don't have to print the traceback later. By the way, here is my test script to try to create a lightweight traceback object without references to locals: https://bitbucket.org/haypo/misc/src/tip/python/suppress_locals.py It works if there is no chained exception. The problem is to build something working with the traceback module. I should maybe write my own formatting function reusing some traceback functions. Victor From solipsis at pitrou.net Thu Mar 6 16:52:56 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 06 Mar 2014 16:52:56 +0100 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: <53188E2D.6020904@gmail.com> References: <5317A5A5.5000305@gmail.com> <53188E2D.6020904@gmail.com> Message-ID: Le 06/03/2014 16:03, Yury Selivanov a ?crit : > > On 2014-03-06, 8:42 AM, Antoine Pitrou wrote: >> Le 05/03/2014 23:53, Nick Coghlan a ?crit : >>> >>> __traceback__ wouldn't change [...] >> >> Uh, really? If you want to suppress all reference cycles, you *have* >> to remove __traceback__. >> >> The problem is to make computation of the traceback summary >> lightweight enough that it doesn't degrade performance in the common >> case where you don't have to print the traceback later. > > So why can't we allow instantiation of types.TracebackType & > types.FrameType? IMO it is absolutely out of question to allow creation of arbitrary frames from Python code, because the structure and initialization of frames embody too many low-level implementation details. We might allow the creation of traceback objects, but without any custom frame objects it is unclear how useful that would be. Regards Antoine. From python-dev at masklinn.net Thu Mar 6 19:10:12 2014 From: python-dev at masklinn.net (Xavier Morel) Date: Thu, 6 Mar 2014 19:10:12 +0100 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: <5317A5A5.5000305@gmail.com> <53188E2D.6020904@gmail.com> Message-ID: On 2014-03-06, at 16:52 , Antoine Pitrou wrote: > Le 06/03/2014 16:03, Yury Selivanov a ?crit : >> >> On 2014-03-06, 8:42 AM, Antoine Pitrou wrote: >>> Le 05/03/2014 23:53, Nick Coghlan a ?crit : >>>> >>>> __traceback__ wouldn't change [...] >>> >>> Uh, really? If you want to suppress all reference cycles, you *have* >>> to remove __traceback__. >>> >>> The problem is to make computation of the traceback summary >>> lightweight enough that it doesn't degrade performance in the common >>> case where you don't have to print the traceback later. >> >> So why can't we allow instantiation of types.TracebackType & >> types.FrameType? > > IMO it is absolutely out of question to allow creation of arbitrary frames from Python code, because the structure and initialization of frames embody too many low-level implementation details. > > We might allow the creation of traceback objects, but without any custom frame objects it is unclear how useful that would be. Some bits of the standard library (and probably third-party libraries transitively relying on getinnerframes) really, really want traceback objects and can't be used with a stack or frames extracted via inspect.currentframe() or whatever. For instance cgitb.text calls inspect.getinnerframes which accesses param.tb_frame then calls getframeinfo(param). getframeinfo blows up if it does not get either an actual traceback object or an actual frame object. A frame object does not have a tb_frame attribute, and will thus fail in getinnerframes, a fake traceback object will fail in getframeinfo. Therefore it's not possible to call cgitb.text outside of an exception handling context (that is, not possible to use it as a traceback.print_stack providing extra information). If it were possible to create traceback objects, there would be no issue there at least. From guido at python.org Thu Mar 6 19:32:00 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 6 Mar 2014 10:32:00 -0800 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: <5317A5A5.5000305@gmail.com> <53188E2D.6020904@gmail.com> Message-ID: But inspect is in the stdlib. Surely changing inspect.py is less controversial than amending the semantics of frame objects. On Thu, Mar 6, 2014 at 10:10 AM, Xavier Morel wrote: > On 2014-03-06, at 16:52 , Antoine Pitrou wrote: > > Le 06/03/2014 16:03, Yury Selivanov a ?crit : > >> > >> On 2014-03-06, 8:42 AM, Antoine Pitrou wrote: > >>> Le 05/03/2014 23:53, Nick Coghlan a ?crit : > >>>> > >>>> __traceback__ wouldn't change [...] > >>> > >>> Uh, really? If you want to suppress all reference cycles, you *have* > >>> to remove __traceback__. > >>> > >>> The problem is to make computation of the traceback summary > >>> lightweight enough that it doesn't degrade performance in the common > >>> case where you don't have to print the traceback later. > >> > >> So why can't we allow instantiation of types.TracebackType & > >> types.FrameType? > > > > IMO it is absolutely out of question to allow creation of arbitrary > frames from Python code, because the structure and initialization of frames > embody too many low-level implementation details. > > > > We might allow the creation of traceback objects, but without any custom > frame objects it is unclear how useful that would be. > > Some bits of the standard library (and probably third-party libraries > transitively relying on getinnerframes) really, really want traceback > objects and can't be used with a stack or frames extracted via > inspect.currentframe() or whatever. For instance cgitb.text calls > inspect.getinnerframes which accesses param.tb_frame then calls > getframeinfo(param). getframeinfo blows up if it does not get either an > actual traceback object or an actual frame object. > > A frame object does not have a tb_frame attribute, and will thus fail > in getinnerframes, a fake traceback object will fail in getframeinfo. > > Therefore it's not possible to call cgitb.text outside of an exception > handling context (that is, not possible to use it as a > traceback.print_stack providing extra information). If it were possible > to create traceback objects, there would be no issue there at least. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Mar 6 18:44:45 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 06 Mar 2014 09:44:45 -0800 Subject: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final? In-Reply-To: <53169016.5050802@hastings.org> References: <531471E1.1010509@hastings.org> <5314F365.80104@hastings.org> <53154A01.3030008@hastings.org> <20140304111641.Horde.WTUtNWKjH-nv8-JUkM3R2w2@webmail.df.eu> <20140304143356.2ac47d37@fsol> <53164DEE.1000904@ubuntu.com> <20140304185958.5b5bff17@limelight.wooz.org> <53169016.5050802@hastings.org> Message-ID: <5318B40D.2010606@stoneleaf.us> On 03/04/2014 06:46 PM, Larry Hastings wrote: > On 03/04/2014 03:59 PM, Barry Warsaw wrote: >> I too would like an rc3, especially to see if issue 19021 can be fixed, which >> I suspect will hit a lot of people. > > I talked to the other guys on the 3.4 team, and we're all willing to do an rc3 this weekend. I'll add that to PEP 429. > > > In other news, I'm thrilled to confirm something Antoine mentioned a week or two ago: it is literally impossible for > garden-variety core devs to push new branches back into trunk. I tried to, early this morning (PST), with someone > logged in to hg.python.org ready to clean up in case it actually worked. But happily hg hooks on the server reject new > branches every time. > > Since this banishes my chief objection to publishing the repo where I do my cherry picking, I'll be publishing that repo > this week. I think I still have a rebase ahead of me, so I'm going wait until after the latest (and hopefully last!) > round of cherry-picking. I'll post to python-dev once the tree is public. Thanks, Larry! I know I have no idea how much work it takes to be an RM, but I appreciate you taking the time, learning the ropes, and getting this done for the rest of us. -- ~Ethan~ From yselivanov.ml at gmail.com Thu Mar 6 20:02:52 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Thu, 06 Mar 2014 14:02:52 -0500 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: <5317A5A5.5000305@gmail.com> <53188E2D.6020904@gmail.com> Message-ID: <5318C65C.1060508@gmail.com> But having exceptions without '__traceback__' affects not only the inspect module, but also lots of other code. It will make harder to write generic error reporting code, as you'd need to check for __traceback__ first, and then, if it's None, look for some other, asyncio specific attribute. I think that having tracebacks with fake frames is much more backwards compatible and consistent solution. I don't understand why we should break the established protocol here. [Or we can allow instantiation of immutable frame objects without __code__] Yury On 2014-03-06, 1:32 PM, Guido van Rossum wrote: > But inspect is in the stdlib. Surely changing inspect.py is less > controversial than amending the semantics of frame objects. > > > On Thu, Mar 6, 2014 at 10:10 AM, Xavier Morel wrote: > >> On 2014-03-06, at 16:52 , Antoine Pitrou wrote: >>> Le 06/03/2014 16:03, Yury Selivanov a ?crit : >>>> On 2014-03-06, 8:42 AM, Antoine Pitrou wrote: >>>>> Le 05/03/2014 23:53, Nick Coghlan a ?crit : >>>>>> __traceback__ wouldn't change [...] >>>>> Uh, really? If you want to suppress all reference cycles, you *have* >>>>> to remove __traceback__. >>>>> >>>>> The problem is to make computation of the traceback summary >>>>> lightweight enough that it doesn't degrade performance in the common >>>>> case where you don't have to print the traceback later. >>>> So why can't we allow instantiation of types.TracebackType & >>>> types.FrameType? >>> IMO it is absolutely out of question to allow creation of arbitrary >> frames from Python code, because the structure and initialization of frames >> embody too many low-level implementation details. >>> We might allow the creation of traceback objects, but without any custom >> frame objects it is unclear how useful that would be. >> >> Some bits of the standard library (and probably third-party libraries >> transitively relying on getinnerframes) really, really want traceback >> objects and can't be used with a stack or frames extracted via >> inspect.currentframe() or whatever. For instance cgitb.text calls >> inspect.getinnerframes which accesses param.tb_frame then calls >> getframeinfo(param). getframeinfo blows up if it does not get either an >> actual traceback object or an actual frame object. >> >> A frame object does not have a tb_frame attribute, and will thus fail >> in getinnerframes, a fake traceback object will fail in getframeinfo. >> >> Therefore it's not possible to call cgitb.text outside of an exception >> handling context (that is, not possible to use it as a >> traceback.print_stack providing extra information). If it were possible >> to create traceback objects, there would be no issue there at least. >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com From jimjjewett at gmail.com Thu Mar 6 21:29:44 2014 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Thu, 06 Mar 2014 12:29:44 -0800 (PST) Subject: [Python-Dev] Alternative forms [was: PEP 463: Exception-catching expressions] In-Reply-To: Message-ID: <5318dab8.e55e8c0a.6939.fffff18b@mx.google.com> The PEP currently says: > Alternative Proposals > ===================== > Discussion on python-ideas brought up the following syntax suggestions:: > value = expr except default if Exception [as e] This one was rejected because of the out-of-order evaluation. Note, however, that the (farthest left) expr is always evaluated first; the only out-of-order evaluation is "default if Exception". "default if Exception" is precisely the same evaluation order (clause after the "if" skips ahead of the clause before the "if") as in the existing if-expression, and the existing if-filters in comprehensions. The same justifications for that order violation generally apply here too. You can argue that they weren't sufficient justification in the first place, but that is water under the bridge; *re*-using out-of-order-"if" shouldn't add any additional costs. [Err... unless people take the "if" too literally, and treat the Exception clause as a boolean value, instead of as an argument to the "except" keyword.] The advantages of this form get much stronger with [as e] or multiple different except clauses, but some of them do apply to even the simplest form. Notably, the "say it like you would in English" that convinced Perl still applies: "if" *without* a "then" is normally an extra condition added after the main point: Normally ham, but fish if it's a Friday. (Admittedly, the word "then" *can* be elided (and represented by a slight pause), and python programmers are already used to seeing it represented only by ":\n") I also give a fair amount of weight to the fact that this form starts to look awkward at pretty much the same time the logic gets too complicated for an expression -- that should discourage abuse. [The analogies to if-expressions and if-filters and to spoken English, along with discouragement for abuse, make this my preferred form.] ... > value = expr except (Exception [as e]: default) (and the similar but unmentioned) value = expr except (Exception [as e] -> default) The mapping analogy for ":" is good -- and is the reason to place parentheses there, as opposed to around the whole expression. Your preferred form -- without the internal parentheses -- looks very much like a suite-introduction, and not at all like the uses where an inline colon is acceptable. I do understand your concern that the parentheses make "except (...)" look too much like a function call -- but I'm not sure that is all bad, let alone as bad as looking like a suite introduction. Both ":" and "->" are defined for signatures; the signature meaning of ":" is tolerable, and the signature meaning of "->" is good. ... > value = expr except Exception [as e] continue with default This one works for me, but only because I read "continue with" as a compound keyword. I assume the parser would too. :D But I do recognize that it is a poor choice for those who see the space as a more solid barrier. ... > value = expr except(Exception) default # Catches only the named type(s) This looks too much like the pre-"as" way of capturing an exception. > value = default if expr raise Exception (Without new keyword "raises",) I would have to work really hard not to read that as: __temp = default if expr: raise Exception value = __temp > value = expr or else default if Exception To me, this just seems like a wordier and more awkward version of expr except (default if Exception [as e]) including the implicit parentheses around "default if Exception". > value = expr except Exception [as e] -> default Without parens to group Exception and default, this looks too much like an annotation describing what the expr should return. value = expr except Exception [as e] pass default I would assume that this skipped the statement, like an if-filter in a comprehension. > All forms involving the 'as' capturing clause have been deferred from > this proposal in the interests of simplicity, but are preserved in the > table above as an accurate record of suggestions. Nick is right that you should specify whether it is deferred or rejected, because the simplest implementation may lock you into too broad a scope if it is added later. > The four forms most supported by this proposal are, in order:: > value = (expr except Exception: default) > value = (expr except Exception -> default) ... If there are not parentheses after "except", it will be very tempting (and arguably correct) to (at least mentally) insert them around the first two clauses -- which are evaluated first. But that leaks into value = (expr except Exception): default which strongly resembles the suite-starter ":", but has very little in common with the mapping ":" or the signature ":". value = (expr except Exception) -> default which looks like an annotation, rather than a part of the value-determination. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From python-dev at masklinn.net Thu Mar 6 21:38:39 2014 From: python-dev at masklinn.net (Xavier Morel) Date: Thu, 6 Mar 2014 21:38:39 +0100 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: <5317A5A5.5000305@gmail.com> <53188E2D.6020904@gmail.com> Message-ID: On 2014-03-06, at 19:32 , Guido van Rossum wrote: > But inspect is in the stdlib. Surely changing inspect.py is less controversial than amending the semantics of frame objects. I've no idea, I'm just giving a case where I could have used the ability to create traceback objects even without the ability to create stack frames (changing the stdlib may also be an option, though I'd hope the explicit type checks were put in there for a reason) From guido at python.org Thu Mar 6 22:00:12 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 6 Mar 2014 13:00:12 -0800 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: <5317A5A5.5000305@gmail.com> <53188E2D.6020904@gmail.com> Message-ID: It's all very old C code that's deeply intertwined with interpreter internals. Who knows what kind of assumptions are made by some of the C code consuming frames and tracebacks... On Thu, Mar 6, 2014 at 12:38 PM, Xavier Morel wrote: > On 2014-03-06, at 19:32 , Guido van Rossum wrote: > > But inspect is in the stdlib. Surely changing inspect.py is less > controversial than amending the semantics of frame objects. > > I've no idea, I'm just giving a case where I could have used the ability > to create traceback objects even without the ability to create stack > frames (changing the stdlib may also be an option, though I'd hope the > explicit type checks were put in there for a reason) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Mar 6 22:41:44 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 07 Mar 2014 10:41:44 +1300 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: <5317A5A5.5000305@gmail.com> <53188E2D.6020904@gmail.com> Message-ID: <5318EB98.7010700@canterbury.ac.nz> Antoine Pitrou wrote: > We might allow the creation of traceback objects, but without any custom > frame objects it is unclear how useful that would be. When I was implementing Pyrex, I would have found it very useful to be able to create Traceback objects without Frames, but only if the Traceback were able to hold a line number. As it was, I had to create an entire fake Frame object filled with mostly dummy values, just to get line numbers printed in the traceback. -- Greg From rosuav at gmail.com Thu Mar 6 23:26:47 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 7 Mar 2014 09:26:47 +1100 Subject: [Python-Dev] Alternative forms [was: PEP 463: Exception-catching expressions] In-Reply-To: <5318dab8.e55e8c0a.6939.fffff18b@mx.google.com> References: <5318dab8.e55e8c0a.6939.fffff18b@mx.google.com> Message-ID: On Fri, Mar 7, 2014 at 7:29 AM, Jim J. Jewett wrote: > > The PEP currently says: > >> Alternative Proposals >> ===================== > >> Discussion on python-ideas brought up the following syntax suggestions:: > >> value = expr except default if Exception [as e] > > This one was rejected because of the out-of-order evaluation. > > Note, however, that the (farthest left) expr is always evaluated first; > the only out-of-order evaluation is "default if Exception". > > "default if Exception" is precisely the same evaluation order > (clause after the "if" skips ahead of the clause before the "if") > as in the existing if-expression, and the existing if-filters in > comprehensions. Yes, but that's still out of order. > The same justifications for that order violation generally apply here too. > You can argue that they weren't sufficient justification in the first > place, but that is water under the bridge; *re*-using out-of-order-"if" > shouldn't add any additional costs. > > [Err... unless people take the "if" too literally, and treat the > Exception clause as a boolean value, instead of as an argument to the > "except" keyword.] The other thing to note is that it's somewhat ambiguous. Until you find that there isn't an else clause, it could be the equally valid "expr except (default if cond else other_default)", with the actual "if Exception" part still to come. (Style guides should, of course, decry this as unreadable, but both the machine parser and any humans reading the code have to assume style guides mightn't be followed.) > The advantages of this form get much stronger with [as e] or multiple > different except clauses, but some of them do apply to even the simplest > form. Multiple different except clauses would make for an even messier evaluation order: expr1 except expr3 if expr2 except expr5 if expr4 If you consider the exception type to be the condition, then this makes sense (that is, if you read it as "if isinstance(thrown_exception, Exception)"); but the most obvious reading of "if", both in its statement form and as part of "true if expr else false", is that it is followed by something that's evaluated as boolean; and all exception types and tuples will be true in that context. > Notably, the "say it like you would in English" that convinced > Perl still applies: "if" *without* a "then" is normally an extra condition > added after the main point: > > Normally ham, but fish if it's a Friday. That's not how Python words ternary if, though. "Ham, if it's not Friday; otherwise fish" is closer, or inverting that: "Fish on Fridays, but normally ham". English is pretty flexible with how you lay things out :) >> value = expr except (Exception [as e]: default) > > (and the similar but unmentioned) > > value = expr except (Exception [as e] -> default) The parenthesizing question and the choice of tokens are considered independent, so not all the cross-multiplications are listed. > The mapping analogy for ":" is good -- and is the reason to place > parentheses there, as opposed to around the whole expression. Your > preferred form -- without the internal parentheses -- looks very > much like a suite-introduction, and not at all like the uses > where an inline colon is acceptable. I have some notes on that down the bottom: http://www.python.org/dev/peps/pep-0463/#colons-always-introduce-suites >> value = expr except Exception [as e] continue with default > > This one works for me, but only because I read "continue with" as a > compound keyword. I assume the parser would too. :D But I do > recognize that it is a poor choice for those who see the space as a > more solid barrier. That depends on all parsers (computer and human) being okay with the two-keyword unit. Would have to poll human parsers to see how they feel about it. >> value = expr except(Exception) default # Catches only the named type(s) > > This looks too much like the pre-"as" way of capturing an exception. Not sure what the connection is, but I don't like the style anyway: putting an expression immediately after a close parenthesis seems odd (I can't think of anything else in Python that has that). >> value = default if expr raise Exception > > (Without new keyword "raises",) I would have to work really hard not to > read that as: > > __temp = default > if expr: > raise Exception > value = __temp Yeah; on the flip side, "raises" doesn't add a huge amount of clarity, and it's creating a new keyword that's not identical, but so all-but - oh Saphir, is it not quite too "all-but"? >> value = expr or else default if Exception > > To me, this just seems like a wordier and more awkward version of > > expr except (default if Exception [as e]) > > including the implicit parentheses around "default if Exception". And mainly, it abuses three keywords that can all already exist in an expression, and doesn't use either "try" or "except". Suppose you saw that, and wanted to know what it does. What would you search the docs for? >> value = expr except Exception [as e] -> default > > Without parens to group Exception and default, this looks too much like > an annotation describing what the expr should return. Can expressions have annotations? >> All forms involving the 'as' capturing clause have been deferred from >> this proposal in the interests of simplicity, but are preserved in the >> table above as an accurate record of suggestions. > > Nick is right that you should specify whether it is deferred or > rejected, because the simplest implementation may lock you into > too broad a scope if it is added later. Put it this way: It's deferred until there's a non-closure means of creating a sub-scope. If that never happens, then this is postponed indefinitely (like pay-day for the Dormouse's good workers); and that doesn't majorly bother me, because the "as" clause isn't very much used in the simple try/except that want to be made into expressions. Philosophically it's a nice thing to support, so it's not Rejected; but it's dependent on a feature that CPython doesn't have, and may never have. In theory, someone could implement subscopes in a different Python, and then reopen this part of the proposal as "CPython won't support it, but MyPy will, can we make this official please?"; I don't know that anyone would bother, but it'd be plausible with the syntax as given. >> The four forms most supported by this proposal are, in order:: > >> value = (expr except Exception: default) >> value = (expr except Exception -> default) > > If there are not parentheses after "except", it will be very tempting > (and arguably correct) to (at least mentally) insert them around the > first two clauses -- which are evaluated first. But that leaks into > > value = (expr except Exception): default > > which strongly resembles the suite-starter ":", but has very little > in common with the mapping ":" or the signature ":". > > value = (expr except Exception) -> default > > which looks like an annotation, rather than a part of the value-determination. value = true_expr if cond else false_expr You can put parens around true_expr or cond, no problem, but you can't put them around both: >>> value = (true_expr if cond) else false_expr SyntaxError: invalid syntax Why would you want to put them around the first two expressions here? It's exactly the same: you have a ternary operator, so you can't parenthesize two of its terms and one of its keywords. That breaks up the parse unit, visually and logically. So that simply wouldn't work. :) ChrisA From victor.stinner at gmail.com Fri Mar 7 00:38:27 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 7 Mar 2014 00:38:27 +0100 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: <5317A5A5.5000305@gmail.com> Message-ID: Hi, 2014-03-06 16:19 GMT+01:00 Victor Stinner : > By the way, here is my test script to try to create a lightweight > traceback object without references to locals: > https://bitbucket.org/haypo/misc/src/tip/python/suppress_locals.py > > It works if there is no chained exception. I updated my proof-of-concept to support Exception.__context__, Exception.__cause__ and Exception.__suppress_context__: https://bitbucket.org/haypo/misc/src/tip/python/suppress_locals.py It creates a lightweight "view" of an exception only keeping information needed to format a traceback. > The problem is to build something working with the traceback module. I > should maybe write my own formatting function reusing some traceback > functions. I reused as much code as possible of the traceback module. I only reimplemented _iter_chain() and _format_exception_iter(). View classes could be simplified, but it would need to modify more code in the traceback module. What do you think of my approach? Would it be something useful in other applications? If yes, I can maybe work on a a better implementation, and maybe on a PEP, to see how such thing could be integrated in Python stdlib. It can maybe directly be part of the traceback module. I don't know if storing an exception to maybe format it later as a traceback is a common use case. Usually, exceptions are immediatly formatted or dropped, no? Victor From ncoghlan at gmail.com Fri Mar 7 06:25:01 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 7 Mar 2014 15:25:01 +1000 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: <5317A5A5.5000305@gmail.com> Message-ID: On 6 Mar 2014 23:44, "Antoine Pitrou" wrote: > > Le 05/03/2014 23:53, Nick Coghlan a ?crit : >> >> >> __traceback__ wouldn't change [...] > > > Uh, really? If you want to suppress all reference cycles, you *have* to remove __traceback__. > > The problem is to make computation of the traceback summary lightweight enough that it doesn't degrade performance in the common case where you don't have to print the traceback later. The proposed summary extraction only keeps the exception type and its str output, not the exception itself (as you don't need that to create the formatted traceback). Cheers, Nick. > > Regards > > Antoine. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Fri Mar 7 12:00:02 2014 From: larry at hastings.org (Larry Hastings) Date: Fri, 07 Mar 2014 03:00:02 -0800 Subject: [Python-Dev] First attempt at Python 3.4.0rc3 is up Message-ID: <5319A6B2.8080809@hastings.org> It's published here: http://hg.python.org/releasing/3.4/ There's also a tarball at the old web site: http://midwinter.com/~larry/3.4.status/ However, the "merge.status.html" page is broken. I'm not going to bother to fix it, instead please just look at the hg repo above. I think part of the problem is that some revisions I had to manually patch in (instead of grafting) wound up with my name instead of the original core dev's name, so the translation between the 3.4 and default branches got broken. Anyway, it locally builds and runs tests without errors. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri Mar 7 12:14:15 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 7 Mar 2014 12:14:15 +0100 Subject: [Python-Dev] Reference cycles in Exception.__traceback__ In-Reply-To: References: <5317A5A5.5000305@gmail.com> Message-ID: 2014-03-07 6:25 GMT+01:00 Nick Coghlan : >> Uh, really? If you want to suppress all reference cycles, you *have* to >> remove __traceback__. >> >> The problem is to make computation of the traceback summary lightweight >> enough that it doesn't degrade performance in the common case where you >> don't have to print the traceback later. > > The proposed summary extraction only keeps the exception type and its str > output, not the exception itself (as you don't need that to create the > formatted traceback). My patch keeps the exception object, but it breaks links to the other exceptions (__cause__ and __context__) and to the traceback (__traceback__): https://bitbucket.org/haypo/misc/src/tip/python/suppress_locals.py Then I saw that http://bugs.python.org/issue17911 already has a patch for the traceback module, nice! And it's very close to what I wrote: http://bugs.python.org/review/17911/#ps8639 This patch stores the exception as string (traceback._format_value(exc)). I prefer to avoid any useless formatting, since the formatted exception is only needed in rare cases. Just drop the "exception summary/view" is the most common case. If you want to full original exception object, don't use the summary/view but handle the exception in the except block. I realized that "memory leaks" (reference cycles) is a common issue with traceback objects (in Python 2) and exception objects (in Python 3): Extracting tracebacks does too much work [open] http://bugs.python.org/issue17911 Local variables not freed when Exception raises in function called from cycle [wont fix] http://bugs.python.org/issue5641 asyncio.Future.set_exception() creates a reference cycle [invalid] http://bugs.python.org/issue20032 Do we need to call gc.collect() occasionally through the event loop? http://code.google.com/p/tulip/issues/detail?id=42 Traceback objects not properly garbage-collected [invalid] http://bugs.python.org/issue226254 Reference cycle in _TracebackLogger and bug in _TracebackLogger.__del__() http://code.google.com/p/tulip/issues/detail?id=155 Twisted fake Traceback object: http://twistedmatrix.com/trac/browser/trunk/twisted/python/failure.py#L89 frame.f_locals keeps references to things for too long [open since 2009], request from Twisted http://bugs.python.org/issue6116 http://twistedmatrix.com/trac/ticket/3853 assertRaises as a context manager keeps tracebacks and frames alive [open] http://bugs.python.org/issue9815 Expose called function on frame object http://bugs.python.org/issue12857 tracebacks eat up memory by holding references to locals and globals when they are not wanted [fixed by traceback.clear_frames()] http://bugs.python.org/issue1565525 Add a frame method to clear expensive details [fixed by frame.clear()] http://bugs.python.org/issue17934 Generator cleanup without tp_del [rejected] http://bugs.python.org/issue17807 Generator memory leak [duplicate] http://bugs.python.org/issue17468 asyncio: remove _TracebackLogger [fixed] http://bugs.python.org/issue19967 sys.exc_info() should not be stored on a local variable [fixed] https://code.djangoproject.com/ticket/10758 Capturing the Currently Raised Exception http://docs.python.org/3/howto/pyporting.html#capturing-the-currently-raised-exception In Python 3, the traceback is attached to the exception instance through the __traceback__ attribute. If the instance is saved in a local variable that persists outside of the except block, the traceback will create a reference cycle with the current frame and its dictionary of local variables. This will delay reclaiming dead resources until the next cyclic garbage collection pass. In Python 2, this problem only occurs if you save the traceback itself (e.g. the third element of the tuple returned by sys.exc_info()) in a variable. => http://hewgill.com/journal/entries/541-python-2-to-3-upgrade-and-exception-handling [Python-Dev] new unbounded memory leak in exception handling? https://mail.python.org/pipermail/python-dev/2009-November/094304.html PEP 3134: Exception Chaining and Embedded Tracebacks [final] http://legacy.python.org/dev/peps/pep-3134/ PEP 344: Exception Chaining and Embedded Tracebacks [superseded] http://legacy.python.org/dev/peps/pep-0344/ Victor From ncoghlan at gmail.com Fri Mar 7 12:14:57 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 7 Mar 2014 21:14:57 +1000 Subject: [Python-Dev] First attempt at Python 3.4.0rc3 is up In-Reply-To: <5319A6B2.8080809@hastings.org> References: <5319A6B2.8080809@hastings.org> Message-ID: On 7 March 2014 21:00, Larry Hastings wrote: > > > It's published here: > > http://hg.python.org/releasing/3.4/ Thanks for that Larry - commits and NEWS for the pip bundling, pkgutil and inspect changes all look good to me :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From larry at hastings.org Fri Mar 7 12:19:19 2014 From: larry at hastings.org (Larry Hastings) Date: Fri, 07 Mar 2014 03:19:19 -0800 Subject: [Python-Dev] First attempt at Python 3.4.0rc3 is up In-Reply-To: References: <5319A6B2.8080809@hastings.org> Message-ID: <5319AB37.6070703@hastings.org> On 03/07/2014 03:14 AM, Nick Coghlan wrote: > On 7 March 2014 21:00, Larry Hastings wrote: >> >> It's published here: >> >> http://hg.python.org/releasing/3.4/ > Thanks for that Larry - commits and NEWS for the pip bundling, pkgutil > and inspect changes all look good to me :) It's been an anxious process. I have some ideas for 3.5 that I think will be popular and make the process easier. But that's a discussion for another time, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Mar 7 18:07:44 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 7 Mar 2014 18:07:44 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140307170744.59D6956912@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-02-28 - 2014-03-07) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4593 (+11) closed 28044 (+45) total 32637 (+56) Open issues with patches: 2084 Issues opened (33) ================== #13936: RFE: change bool(datetime.time(0, 0, 0)) to evaluate as True http://bugs.python.org/issue13936 reopened by ncoghlan #19494: urllib2.HTTPBasicAuthHandler (or urllib.request.HTTPBasicAuthH http://bugs.python.org/issue19494 reopened by r.david.murray #20014: Makes array.array constructor accepts ascii-unicode typecode http://bugs.python.org/issue20014 reopened by serhiy.storchaka #20015: Allow 1-character ASCII unicode where 1-character str is requi http://bugs.python.org/issue20015 reopened by serhiy.storchaka #20811: str.format for fixed width float can return a string longer th http://bugs.python.org/issue20811 opened by aubmoon #20812: Explicitly cover application migration in the 2->3 guide http://bugs.python.org/issue20812 opened by ncoghlan #20813: Backport revised 2to3 guide to older branches http://bugs.python.org/issue20813 opened by ncoghlan #20815: ipaddress unit tests PEP8 http://bugs.python.org/issue20815 opened by exhuma #20816: inspect.getcallargs() attempts to iterate over None http://bugs.python.org/issue20816 opened by jlowin #20817: inspect.getcallargs() raises the wrong error if 3+ arguments a http://bugs.python.org/issue20817 opened by jlowin #20819: reinitialize_command doesn't clear install_lib on install and http://bugs.python.org/issue20819 opened by jason.coombs #20823: Clarify copyreg.pickle() documentation http://bugs.python.org/issue20823 opened by peter.otten #20825: containment test for "ip_network in ip_network" http://bugs.python.org/issue20825 opened by exhuma #20826: Faster implementation to collapse consecutive ip-networks http://bugs.python.org/issue20826 opened by exhuma #20827: IDLE : Display function argument list in ClassBrowser http://bugs.python.org/issue20827 opened by sahutd #20833: scripts\pydocgui.pyw out of date http://bugs.python.org/issue20833 opened by loewis #20836: Pickle Nonetype http://bugs.python.org/issue20836 opened by Hans.Polak #20837: Ambiguity words in base64 documentation http://bugs.python.org/issue20837 opened by naoki #20840: AttributeError: 'module' object has no attribute 'ArgumentPars http://bugs.python.org/issue20840 opened by bsduni #20842: pkgutil docs should reference glossary terms not PEP 302 http://bugs.python.org/issue20842 opened by ncoghlan #20844: coding bug remains in 3.3.5rc2 http://bugs.python.org/issue20844 opened by miwa #20847: asyncio docs should call out that network logging is a no-no http://bugs.python.org/issue20847 opened by gvanrossum #20849: add exist_ok to shutil.copytree http://bugs.python.org/issue20849 opened by thehesiod #20851: Update devguide to cover testing from a tarball http://bugs.python.org/issue20851 opened by brett.cannon #20853: pdb "args" crashes when an arg is not printable http://bugs.python.org/issue20853 opened by jneb #20854: multiprocessing.managers.Server: problem with returning proxy http://bugs.python.org/issue20854 opened by allista #20857: ipaddress "is_private" and "is_global" are insufficiently docu http://bugs.python.org/issue20857 opened by r.david.murray #20858: Enhancements/fixes to pure-python datetime module http://bugs.python.org/issue20858 opened by bdkearns #20859: Context of documentation for conditional expressions http://bugs.python.org/issue20859 opened by terry.reedy #20860: ipaddress network subnets() method should return object with _ http://bugs.python.org/issue20860 opened by Warren.Turkal #20861: datetime argument handling inconsistent; should accept logical http://bugs.python.org/issue20861 opened by josh.rosenberg #20863: IDLE not opening http://bugs.python.org/issue20863 opened by chester.burns #20864: getattr does not work well with descriptor http://bugs.python.org/issue20864 opened by Martin.Thurau Most recent 15 issues with no replies (15) ========================================== #20864: getattr does not work well with descriptor http://bugs.python.org/issue20864 #20859: Context of documentation for conditional expressions http://bugs.python.org/issue20859 #20854: multiprocessing.managers.Server: problem with returning proxy http://bugs.python.org/issue20854 #20851: Update devguide to cover testing from a tarball http://bugs.python.org/issue20851 #20842: pkgutil docs should reference glossary terms not PEP 302 http://bugs.python.org/issue20842 #20833: scripts\pydocgui.pyw out of date http://bugs.python.org/issue20833 #20827: IDLE : Display function argument list in ClassBrowser http://bugs.python.org/issue20827 #20826: Faster implementation to collapse consecutive ip-networks http://bugs.python.org/issue20826 #20823: Clarify copyreg.pickle() documentation http://bugs.python.org/issue20823 #20819: reinitialize_command doesn't clear install_lib on install and http://bugs.python.org/issue20819 #20797: zipfile.extractall should accept bytes path as parameter http://bugs.python.org/issue20797 #20787: typo in asyncio docs for subprocess_exec() http://bugs.python.org/issue20787 #20785: Missing symbols in Python27.lib (Windows 64bit) http://bugs.python.org/issue20785 #20782: base64 module docs do not use the terms 'bytes' and 'string' c http://bugs.python.org/issue20782 #20780: Shadowed (duplicate name but different body) test in test_stat http://bugs.python.org/issue20780 Most recent 15 issues waiting for review (15) ============================================= #20858: Enhancements/fixes to pure-python datetime module http://bugs.python.org/issue20858 #20849: add exist_ok to shutil.copytree http://bugs.python.org/issue20849 #20837: Ambiguity words in base64 documentation http://bugs.python.org/issue20837 #20827: IDLE : Display function argument list in ClassBrowser http://bugs.python.org/issue20827 #20826: Faster implementation to collapse consecutive ip-networks http://bugs.python.org/issue20826 #20825: containment test for "ip_network in ip_network" http://bugs.python.org/issue20825 #20823: Clarify copyreg.pickle() documentation http://bugs.python.org/issue20823 #20817: inspect.getcallargs() raises the wrong error if 3+ arguments a http://bugs.python.org/issue20817 #20816: inspect.getcallargs() attempts to iterate over None http://bugs.python.org/issue20816 #20815: ipaddress unit tests PEP8 http://bugs.python.org/issue20815 #20792: IDLE: Extend tests for PathBrowser http://bugs.python.org/issue20792 #20787: typo in asyncio docs for subprocess_exec() http://bugs.python.org/issue20787 #20780: Shadowed (duplicate name but different body) test in test_stat http://bugs.python.org/issue20780 #20779: Add pathlib.chown method http://bugs.python.org/issue20779 #20770: Inform caller of smtplib STARTTLS failures http://bugs.python.org/issue20770 Top 10 most discussed issues (10) ================================= #13936: RFE: change bool(datetime.time(0, 0, 0)) to evaluate as True http://bugs.python.org/issue13936 32 msgs #20811: str.format for fixed width float can return a string longer th http://bugs.python.org/issue20811 12 msgs #20858: Enhancements/fixes to pure-python datetime module http://bugs.python.org/issue20858 10 msgs #15014: smtplib: add support for arbitrary auth methods http://bugs.python.org/issue15014 7 msgs #20815: ipaddress unit tests PEP8 http://bugs.python.org/issue20815 7 msgs #20825: containment test for "ip_network in ip_network" http://bugs.python.org/issue20825 7 msgs #20015: Allow 1-character ASCII unicode where 1-character str is requi http://bugs.python.org/issue20015 6 msgs #20804: Sentinels identity lost when pickled (unittest.mock) http://bugs.python.org/issue20804 6 msgs #20860: ipaddress network subnets() method should return object with _ http://bugs.python.org/issue20860 6 msgs #18139: email module should have a way to prepend and insert headers http://bugs.python.org/issue18139 4 msgs Issues closed (47) ================== #11558: Raise a more helpful exception in email.message.Message.attach http://bugs.python.org/issue11558 closed by r.david.murray #14512: Pydocs module docs server not working on Windows. http://bugs.python.org/issue14512 closed by loewis #15618: turtle.pencolor() chokes on unicode http://bugs.python.org/issue15618 closed by terry.reedy #19021: AttributeError in Popen.__del__ http://bugs.python.org/issue19021 closed by larry #19085: Add tkinter basic options tests http://bugs.python.org/issue19085 closed by serhiy.storchaka #19619: Blacklist base64, hex, ... codecs from bytes.decode() and str. http://bugs.python.org/issue19619 closed by georg.brandl #20249: test_posix.test_initgroups fails when running with no suppleme http://bugs.python.org/issue20249 closed by python-dev #20283: Wrong keyword parameter name in regex pattern methods http://bugs.python.org/issue20283 closed by Arfrever #20465: Upgrade SQLite to 3.8.3 with 3.4.0 Windows and OS X installers http://bugs.python.org/issue20465 closed by loewis #20646: 3.4 cherry-pick: 180e4b678003 select and kqueue round the time http://bugs.python.org/issue20646 closed by haypo #20648: 3.4 cherry-pick: multiple changesets for asyncio http://bugs.python.org/issue20648 closed by haypo #20728: Remove unused import from base64 http://bugs.python.org/issue20728 closed by terry.reedy #20748: 3.4rc2 MSI uninstallation leaves behind ensurepip _uninstall . http://bugs.python.org/issue20748 closed by loewis #20761: os.path.join doesn't strip LF or CR http://bugs.python.org/issue20761 closed by georg.brandl #20786: inspect.getargspec() returns wrong answer with property.__dele http://bugs.python.org/issue20786 closed by yselivanov #20789: [3.4] cherrypick 5dec1604322c: old sys.path_hooks importer doe http://bugs.python.org/issue20789 closed by larry #20799: Turn on all tests in test.test_importlib.source.test_file_load http://bugs.python.org/issue20799 closed by brett.cannon #20807: 3.4 cherry pick: 82ec02db7fe6 & ec42ab5e0cb3 Windows installer http://bugs.python.org/issue20807 closed by larry #20808: 3.4 cherry pick: 6a1711c96fa6 (Popen.__del__ traceback) http://bugs.python.org/issue20808 closed by larry #20809: isabspath receiving path value of None in distutils.dist.py http://bugs.python.org/issue20809 closed by ned.deily #20810: literal re fails to match http://bugs.python.org/issue20810 closed by eric.smith #20814: tracemalloc example - Pretty Top http://bugs.python.org/issue20814 closed by python-dev #20818: 3.4 cherry pick: 5fa3f6d82d61 and a8470f88e7b4 - OS X installe http://bugs.python.org/issue20818 closed by larry #20820: HTML being escaped on some release pages http://bugs.python.org/issue20820 closed by ned.deily #20821: Should be able to break long numbers across lines http://bugs.python.org/issue20821 closed by eric.smith #20822: WINhelp file loads Table of Contents, but not contents: "Navig http://bugs.python.org/issue20822 closed by Nemo.Neminibus #20824: setup.py install doesn't parse spaces in path http://bugs.python.org/issue20824 closed by ned.deily #20828: inspect.getargspec() returns wrong answer with datetime.today. http://bugs.python.org/issue20828 closed by yselivanov #20829: 3.4 cherry pick: c9861ec8754c Fix signatures for dict.__delite http://bugs.python.org/issue20829 closed by larry #20830: 3.4 cherry-pick: 16f91d87ff39: "pip install " didn't http://bugs.python.org/issue20830 closed by larry #20831: 3.4 cherry-pick: 7b3c40510a08 Windows uninstaller doesn't unin http://bugs.python.org/issue20831 closed by larry #20832: 3.4 cherry-pick: 9459f517d854 Update Windows installer to SQLi http://bugs.python.org/issue20832 closed by larry #20834: 3.4 cherry-pick: de81e0fe4905 Pydocs module docs server not wo http://bugs.python.org/issue20834 closed by larry #20835: (9-(9%2) http://bugs.python.org/issue20835 closed by skrah #20838: Documents disagree about release state of Python 3.3.5 http://bugs.python.org/issue20838 closed by georg.brandl #20839: pkgutil.get_loader throws deprecation warning due to internal http://bugs.python.org/issue20839 closed by python-dev #20841: asyncio.StreamReader.readline() lack lines limit optional para http://bugs.python.org/issue20841 closed by gvanrossum #20843: 3.4 cherry-pick: a9058b772807 fix tracemalloc doc http://bugs.python.org/issue20843 closed by larry #20845: email.utils.formataddr encodes incorrectly http://bugs.python.org/issue20845 closed by Michael.JasonSmith #20846: pip can fail on windows but gives success message http://bugs.python.org/issue20846 closed by r.david.murray #20848: 3.4 cherry-pick: b637064cc696 Improve enum subclass behavior http://bugs.python.org/issue20848 closed by larry #20850: 3.4 cherry pick: ea827c809765 fix pkgutil.find_loader http://bugs.python.org/issue20850 closed by larry #20852: Embedding with zipped standard library is broken in 3.3.4 (Win http://bugs.python.org/issue20852 closed by ned.deily #20855: RFE: change bool(0.0) to evaluate as True http://bugs.python.org/issue20855 closed by dstufft #20856: bz2.peek always peeks all the remaining bytes ignoring n argum http://bugs.python.org/issue20856 closed by serhiy.storchaka #20862: Problems running python 3.3 http://bugs.python.org/issue20862 closed by ned.deily #20865: A run has overwrite my code save http://bugs.python.org/issue20865 closed by zach.ware From jimjjewett at gmail.com Fri Mar 7 19:58:25 2014 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Fri, 07 Mar 2014 10:58:25 -0800 (PST) Subject: [Python-Dev] Alternative forms [was: PEP 463: Exception-catching expressions] In-Reply-To: Message-ID: <531a16d1.1473e00a.10f4.ffffd7ef@mx.google.com> (Thu Mar 6 23:26:47 CET 2014) Chris Angelico responded: > On Fri, Mar 7, 2014 at 7:29 AM, Jim J. Jewett wrote: >> [ note that "x if y" already occurs in multiple contexts, and >> always evaluates y before x. ] > Yes, but that's still out of order. Yeah, but local consistency is more important than global guidelines. :D >> ... *re*-using out-of-order-"if" shouldn't add any additional costs. > The other thing to note is that it's somewhat ambiguous. Until you > find that there isn't an else clause, it could be the equally valid > "expr except (default if cond else other_default)", with the actual > "if Exception" part still to come. True -- and I'm not a parser expert. But my belief is that the current parser allows lookahead for exactly one token, and that the "else" would fit within that limit. > ... humans reading the code have to assume style guides mightn't > be followed. True ... but I hope any non-trivial use of this (including use with a non-trivial ternary if) will look bad enough to serve as its own warning. >> The advantages of this form get much stronger with [as e] or >> multiple different except clauses, but some of them do apply >> to even the simplest form. > Multiple different except clauses would make for an even > messier evaluation order: > expr1 except expr3 if expr2 except expr5 if expr4 > If you consider the exception type to be the condition, then > this makes sense (that is, if you read it as > "if isinstance(thrown_exception, Exception)"); > [but the most obvious reading is boolean; as always True] I phrased that badly. I agree that without parentheses for good spacing, the above is at least ambiguous -- that is what you get for stringing multiple clauses together without internal grouping. I do think parentheses help, (but are less important when there is only a single "if") and I strongly prefer that they be internal (which you fear looks too much like calling a function named except). In that case, it is: expr1 except (expr3 if expr2) and the extension to multiple except clauses would be: expr1 except (expr3 if expr2, expr5 if expr4) though as I discuss later, placing parentheses there also makes a colon or arrow more tolerable. It does this because the nearby parens make it look more like the existing (non-lambda) uses of inline-colon to associate the two things on either side. (Without nearby brackets, the scope of the colon or arrow is more easily taken to be the whole line.) expr1 except (expr2: expr3, expr4: expr5) expr1 except (expr2 -> expr3, expr4 -> expr5) >> Notably, the "say it like you would in English" that convinced >> Perl still applies: "if" *without* a "then" is normally an extra >> condition added after the main point: >> Normally ham, but fish if it's a Friday. > That's not how Python words ternary if, though. Agreed ... the "say it like you would in English" applies only to the "expr if expr" form (proposed here and) used by comprehensions: [1/x for x in data if x] >> value = expr except (Exception [as e]: default) > > (and the similar but unmentioned) > > value = expr except (Exception [as e] -> default) The parenthesizing question and the choice of tokens are considered independent, so not all the cross-multiplications are listed. >> The mapping analogy for ":" is good -- and is the reason to place >> parentheses there, as opposed to around the whole expression. Your >> preferred form -- without the internal parentheses -- looks very >> much like a suite-introduction, and not at all like the uses >> where an inline colon is acceptable. > I have some notes on that down the bottom: > http://www.python.org/dev/peps/pep-0463/#colons-always-introduce-suites I know that they don't always introduce suites. I can't speak to the lambda precedent, but I do know that I personally often stumble when trying to parse it, so I don't consider it a good model. The other three inline uses (dict display, slide notation, and function parameter annotation) are effectively conjunction operators, saying that expr1 and expr2 are bound more tightly than you would assume if they were separated by commas. They only occur inside a fairly close bracket (of some sort), and if the bracket isn't *very* close, then there are usually multiple associates-colons inside the same bracket. data[3:5] data[-1:-3:-1] def myfunc(a:int=5, b:str="Jim", c:float=3.14) {'b': 2, 'c': 3, 'a': 1} With parentheses after the except, the except expression will match this pattern too -- particularly if there are multiple types of exception treated differently. expr1 except (expr2: expr3) Without (preferably internal) parentheses, it will instead look like a long line with a colon near the end, and a short continuation suite that got moved up a line because it was only one statement long. def nullfunc(self, a): pass expr1 except expr3: expr2 >>> value = expr except Exception [as e] -> default >> >> Without parens to group Exception and default, this looks too much like >> an annotation describing what the expr should return. > Can expressions have annotations? Not yet. I have seen proposals. Other syntax (notably, Decorations) have expanded their domain. I would be mildly surprised if there isn't already a 3rd party template system using -> in places where it isn't currently defined. >>> All forms involving the 'as' capturing clause have been deferred ... >> Nick is right that you should specify whether it is deferred or >> rejected, because the simplest implementation may lock you into >> too broad a scope if it is added later. > Put it this way: It's deferred until there's a non-closure means of > creating a sub-scope. The problem is that once it is deployed as leaking into the parent scope, backwards compatibility may force it to always leak into the parent scope. (You could document the leakage as a bug or as implementation-defined, but ... those choices are also sub-optimal.) >>> The four forms most supported by this proposal are, in order:: >>> value = (expr except Exception: default) >>> value = (expr except Exception -> default) >> If there are not parentheses after "except", it will be very tempting >> (and arguably correct) to (at least mentally) insert them around the >> first two clauses -- which are evaluated first. But that leaks into >> value = (expr except Exception): default ... > You can put parens around true_expr or cond, no problem, but you > can't put them around both: > >>> value = (true_expr if cond) else false_expr > SyntaxError: invalid syntax The fact that short-circuiting will prevent false_expr from even being evaluated means that "(true_expr if cond)" is a useful mental approximation, even though "true_expr if cond" is already to the left. Commutivity and Associativity are pretty strong biases; testing order of execution for apparently parallel expressions (such as different arguments to the same function in the same call) is a source of trick questions. When associativity is violated, computation normally proceeds from the left, and any time you need to jump ahead to simplify something that isn't left-most, that special case needs to be signalled. For short enough jumps, order of operations can be sufficient (e=mc^2), but you'll still see plenty of programmers inserting redundant parentheses just to make things clear (or maybe just to avoid having to look up the order of operations). The replacement value and the exception that triggers it are clearly associated more closely with each other than either is to the primary expression. Therefore, they *should* be grouped more tightly. This is a proposal with no perfect syntax, but I don't really see any upside to *not* grouping the exception with its exceptional result. At the moment, I don't know how to spell that closer grouping without at least one of: Parentheses or brackets of some sort A line break/new suite: - kills the proposal An inline associates-colon: - needs brackets anyhow, or it will be mistaken for a suite-introduction-colon. A new grouping operator, such as "->" - risk that it will be misinterpreted Relying on convention and taste: (main_expr except exc_expr fallback_expr) -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From contact at bigbizmail.com Fri Mar 7 19:29:52 2014 From: contact at bigbizmail.com (Bolyai S) Date: Fri, 7 Mar 2014 18:29:52 +0000 Subject: [Python-Dev] =?iso-8859-1?q?Relax=E1l=F3_massz=E1zs_otthon=2E?= Message-ID: Tudja meg,hogyan r?szes?lhetnek az otthoni massz?zsnak,20%-kal olcs?bb ?rban mint a massz?zs szalonokban. ????????????????????????????????????????????????????????? -----> Kattintson ide <------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Mar 7 20:54:31 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 8 Mar 2014 06:54:31 +1100 Subject: [Python-Dev] Alternative forms [was: PEP 463: Exception-catching expressions] In-Reply-To: <531a16d1.1473e00a.10f4.ffffd7ef@mx.google.com> References: <531a16d1.1473e00a.10f4.ffffd7ef@mx.google.com> Message-ID: On Sat, Mar 8, 2014 at 5:58 AM, Jim J. Jewett wrote: > > (Thu Mar 6 23:26:47 CET 2014) Chris Angelico responded: > >> On Fri, Mar 7, 2014 at 7:29 AM, Jim J. Jewett wrote: > >>> [ note that "x if y" already occurs in multiple contexts, and >>> always evaluates y before x. ] > >> Yes, but that's still out of order. > > Yeah, but local consistency is more important than global guidelines. :D I don't see except expressions as fundamentally more associated with if/else than with, say, an or chain, which works left to right. Aside from both being ternary operators, of course, which isn't really much of a justification. >> The other thing to note is that it's somewhat ambiguous. Until you >> find that there isn't an else clause, it could be the equally valid >> "expr except (default if cond else other_default)", with the actual >> "if Exception" part still to come. > > True -- and I'm not a parser expert. But my belief is that the > current parser allows lookahead for exactly one token, and that > the "else" would fit within that limit. > >> ... humans reading the code have to assume style guides mightn't >> be followed. > > True ... but I hope any non-trivial use of this (including use > with a non-trivial ternary if) will look bad enough to serve as > its own warning. Not sure whether the parser would be able to handle it or not, but the human would have to if the machine can, and that's going to be a source of confusion. I'd rather avoid it if I can. Remember, everywhere else in Python, the word "if" is followed by something that's interpreted as a boolean. You wouldn't expect to see this somewhere: if None: do_stuff() do_stuff() if sys else do_other_stuff() So it'd cause a mental glitch to see some other constant and always-true expression there: ... if ZeroDivisionError ... You don't expect ZeroDivisionError ever to be false. Seeing it following an if would leave you wondering if maybe it is. IMO that syntax is abuse of the 'if' keyword, hammering it into a hole it's not designed to fit into. > I do think parentheses help, (but are less important when there > is only a single "if") Analysis of the Python standard library suggests that the single-if situation is *by far* the most common, to the extent that it'd hardly impact the stdlib at all to add multiple except clauses to the proposal. Do you have a strong use-case for the more full syntax? It tends to get long, which rather damages the readability. A number of people have said that if the except expression goes over a line break, it should probably be a statement instead. I'm not sure that extreme is fully justified, but it certainly does have merit. In a codebase of mine I wouldn't *forbid* breaking an except expression over multiple lines, but it'd certainly be a reason to consider whether it's really necessary or not. > and I strongly prefer that they [the parentheses] be internal > (which you fear looks too much like calling a function named except). > In that case, it is: > > expr1 except (expr3 if expr2) I'm still not really seeing how this is better. With the colon version, it looks very much like dict display, only with different brackets around it; in some fonts, that'll be very easily confused. With the if, it looks like an incomplete expression waiting for its else. And I'm still not enamored of constructing syntax that has the evaluation order (a) not simple left-to-right, like most expressions are, and (b) different from the except statement, which puts the exception_list ahead of the suite. > Agreed ... the "say it like you would in English" applies only > to the "expr if expr" form (proposed here and) used by comprehensions: > > [1/x for x in data > if x] Sure. That usage makes a _lot_ of sense, I really like list comprehensions. There's no room in them for an 'else' clause though, so it's very strictly one-way. Makes them a tricky comparison for this. > I can't speak to the lambda precedent, but I do know that I personally > often stumble when trying to parse it, so I don't consider it a good model. The only confusion I have with lambda is its precedence, which trips me up now and then: >>> def f(x): ... return lambda y: x,y ... >>> f(5) Traceback (most recent call last): File "", line 1, in File "", line 2, in f NameError: name 'y' is not defined That's not returning a function that returns a tuple. It's returning a tuple of a function and the global y, and the function will ignore its arg. > The other three inline uses (dict display, slide notation, and > function parameter annotation) are effectively conjunction operators, > saying that expr1 and expr2 are bound more tightly than you would > assume if they were separated by commas. They only occur inside > a fairly close bracket (of some sort), and if the bracket isn't > *very* close, then there are usually multiple associates-colons > inside the same bracket. Not sure that really helps. This isn't going to be as tight as a slice, and it's most often not going to have multiple colons inside the brackets. > Without (preferably internal) parentheses, it will instead look like > a long line with a colon near the end, and a short continuation suite > that got moved up a line because it was only one statement long. > > def nullfunc(self, a): pass > > expr1 except expr3: expr2 That should have expr2 and exp3 the other way round - the bit before the colon is evaluated before the bit after the colon. What about putting it all in parens, which is the current proposal? There's no way you can have a statement with a one-line suite on the same line, all enclosed in parens. (expr1 except expr2: expr3) >>>> All forms involving the 'as' capturing clause have been deferred ... > >>> Nick is right that you should specify whether it is deferred or >>> rejected, because the simplest implementation may lock you into >>> too broad a scope if it is added later. > >> Put it this way: It's deferred until there's a non-closure means of >> creating a sub-scope. > > The problem is that once it is deployed as leaking into the parent > scope, backwards compatibility may force it to always leak into > the parent scope. (You could document the leakage as a bug or as > implementation-defined, but ... those choices are also sub-optimal.) It'll never be deployed as leaking, for the same reason that the current 'except' statement doesn't leak: it creates a refloop. The entire notion of an 'as' clause is deferred. If there's no way to create a sub-scope that isn't a closure, then there will be no 'as' clause in an except expression. It's that simple. Then later, if such a feature is added, it can be retrofitted without breaking anything. But there's little need of it; the cases where the exception needs to be captured are usually the ones that need the full statement anyway. Of all 'except' statements in the stdlib, almost exactly one in five uses 'as'; of the ones simple enough to potentially be changed to this form, just 7 out of 236 do. (And those seven are among the more dubious ones.) So it's no great loss to the proposal, assuming the stdlib is indicative. (If you have a good-sized Py3 codebase to try it on, feel free to enhance those stats with more data. The script's part of my repo.) > The fact that short-circuiting will prevent false_expr from even > being evaluated means that "(true_expr if cond)" is a useful > mental approximation, even though "true_expr if cond" is already > to the left. Hrm, I don't really like that way of describing it. To me, the entire expression is one unit - you can't parenthesize it. It's like parenthesizing a chained comparison: if 5 < x < 10: if (5 < x) < 10: if 5 < (x < 10): You can't add parens around one part of the expression; it completely changes the meaning of it. > Commutivity and Associativity are pretty strong biases; testing > order of execution for apparently parallel expressions (such as > different arguments to the same function in the same call) is a > source of trick questions. In C, the answer is usually "Implementation-defined". In Python, the answer is usually "Left to right". > The replacement value and the exception that triggers it are > clearly associated more closely with each other than either is > to the primary expression. Therefore, they *should* be grouped > more tightly. Maybe. On the other hand, the replacement value and the original expression are two things of the same broad type, and the other expression is an exception list, so the primary and the replacement should be grouped more tightly. You're going to get this or that. I don't think this is a strong reason for parentheses. > This is a proposal with no perfect syntax, but I don't really > see any upside to *not* grouping the exception with its > exceptional result. At the moment, I don't know how to spell > that closer grouping without at least one of: > > Parentheses or brackets of some sort > > A line break/new suite: > - kills the proposal > > An inline associates-colon: > - needs brackets anyhow, or it will be mistaken for > a suite-introduction-colon. > > A new grouping operator, such as "->" > - risk that it will be misinterpreted > > Relying on convention and taste: > (main_expr except exc_expr fallback_expr) Definitely not the last one. It needs some kind of syntactic separation, not just whitespace (what if the fallback_expr is a negative number?). The entire expression should be representing a single thought, or else it probably should be written as a statement. Like the thought "Get me the current working directory, None if there isn't one". pwd = (os.getcwd() except OSError: None) Or: "Append the IP, or failing that the network address". ips.append(ip.ip except AttributeError: ip.network_address) With those, there's really no separation between "Get the current working directory", "if there isn't one", and "None". Parenthesizing the part after 'except' makes good sense if you want to have multiple except clauses, but there's no reason to do that, so we're back with this very simple case. I'd rather not mandate any parens at all, personally, but as was said in another thread, you don't need to see a Frenchman on the parapet to know where the wind's blowing. ChrisA From jurko.gospodnetic at pke.hr Fri Mar 7 21:10:54 2014 From: jurko.gospodnetic at pke.hr (=?windows-1250?Q?Jurko_Gospodneti=E6?=) Date: Fri, 07 Mar 2014 21:10:54 +0100 Subject: [Python-Dev] undocumented help() function change in Python 3.4? Message-ID: Hi. I just noticed that the way help() function displays a function signature changed between Python 3.3 & 3.4 but I can not find this documented anywhere. Here's a matching example in both Python 3.3 & Python 3.4 for comparison: ---------------------------------------- > Python 3.3.3 (v3.3.3:c3896275c0f6, Nov 18 2013, 21:19:30) [MSC v.1600 64 bit (AMD64)] on win32 > Type "help", "copyright", "credits" or "license" for more information. >>>> def f(a, b, c): > ... print(a, b, c) > ... >>>> def g(*args, **kwargs): > ... f(*args, **kwargs) > ... >>>> import inspect >>>> print(inspect.signature(f)) > (a, b, c) >>>> print(inspect.signature(g)) > (*args, **kwargs) >>>> g.__wrapped__ = f >>>> print(inspect.signature(f)) > (a, b, c) >>>> print(inspect.signature(g)) > (a, b, c) >>>> help(f) > Help on function f in module __main__: > > f(a, b, c) > >>>> help(g) > Help on function g in module __main__: > > g(*args, **kwargs) ---------------------------------------- > Python 3.4.0b3 (v3.4.0b3:a97ce3ecc96a, Jan 26 2014, 17:50:55) [MSC v.1600 64 bit (AMD64)] on win32 > Type "help", "copyright", "credits" or "license" for more information. >>>> def f(a, b, c): > ... print(a, b, c) > ... >>>> def g(*args, **kwargs): > ... f(*args, **kwargs) > ... >>>> import inspect >>>> print(inspect.signature(f)) > (a, b, c) >>>> print(inspect.signature(g)) > (*args, **kwargs) >>>> g.__wrapped__ = f >>>> print(inspect.signature(f)) > (a, b, c) >>>> print(inspect.signature(g)) > (a, b, c) >>>> help(f) > Help on function f in module __main__: > > f(a, b, c) > >>>> help(g) > Help on function g in module __main__: > > g(a, b, c) ---------------------------------------- As you can see by comparing those two outputs, setting the __wrapped__ attribute on a wrapper function affects the inspect.signature() results on that function. This behaviour is the same in both Python 3.3. & 3.4 and is (somewhat) described in the Python documentation. However, help() output is not affected by this in Python 3.3, but is affected in Python 3.4, and I can not find anything regarding this in the Python 3.4 docs. Can something related to this be added at least to the 'what's changed' docs, if not to the help() documentation as well? Best regards, Jurko Gospodenti? From nad at acm.org Fri Mar 7 22:00:50 2014 From: nad at acm.org (Ned Deily) Date: Fri, 07 Mar 2014 13:00:50 -0800 Subject: [Python-Dev] First attempt at Python 3.4.0rc3 is up References: <5319A6B2.8080809@hastings.org> Message-ID: In article <5319A6B2.8080809 at hastings.org>, Larry Hastings wrote: > It's published here: > > http://hg.python.org/releasing/3.4/ Note, to get the current state of what will be in "3.4.0.rc3", you need to use the "3.4" branch and not "default". hg pull http://hg.python.org/releasing/3.4/ hg update 3.4 -- Ned Deily, nad at acm.org From jimjjewett at gmail.com Fri Mar 7 22:20:49 2014 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Fri, 07 Mar 2014 13:20:49 -0800 (PST) Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: Message-ID: <531a3831.8589e00a.7525.01d4@mx.google.com> On Wed Mar 5 17:37:12 CET 2014, Victor Stinner wrote: > Python 3 now stores the traceback object in Exception.__traceback__ > and exceptions can be chained through Exception.__context__. It's > convenient but it introduced tricky reference cycles if the exception > object is used out of the except block. > ... see Future.set_exception() of the ayncio module. > ... frame.clear() raises an RuntimeError if the frame is still > running. And it doesn't break all reference cycles. > An obvious workaround is to store the traceback as text, but this > operation is "expensive" especially if the traceback is only needed in > rare cases. > I tried to write "views" of the traceback (and frames), but > Exception.__traceback__ rejects types other than traceback and > traceback instances cannot be created. It's possible to store the > traceback somewhere else and set Exception.__traceback__ to None, but > there is still the problem with chained exceptions. > Any idea for a generic fix to such problem? Could you clarify what the problem actually is? I can imagine any of the following: (A) Exceptions take a lot of memory, because of all the related details. + But sometimes the details are needed, so there is no good solution. (B) Exceptions take a lot of memory, because of all the related details. There is a common use case that knows it will never need certain types of details, and releasing just those details would save a lot of memory. But frame.clear() picks the wrong details to release, at least for this case. + So write another function (or even a method) that does work, and have your framework call it. (Also see (F)) + Instead of saving the original exception, could you instead create and store a new (copied?) one, which obviously won't (yet) be referenced by the traceback you assign to it? (C) Exceptions take a lot of memory, because of all the related details. There is a common use case that knows it can make do with a summary of certain types of details, and releasing just those details would save a lot of memory. But generating the summary is expensive. + It would help to have the summarize method available. + It would help to have feedback from gc saying when there is enough memory pressure to make this call worthwhile. (D) Exceptions are not released until cyclic gc, and so they eat a lot of memory for a long time prior to that. + This may be like case B + Are there references that can be replaced by weak references? + Are there references that you can replace with weak references when your framework stores the exception? (Also see (F)) (E) Exceptions are not released even during cyclic gc, because of ambiguity over which __del__ to run first. + This may be like case B or case E + This may be a concrete use case for the __close__ protocol. __close__ is similar to __del__, except that it promises not to care about order of finalization, and it is run eagerly. As soon as an instance is known to be in a garbage cycle, __close__ should be run without worrying about whether other objects also have __close__ or __del__ methods. Hopefully, this will break the cycle, or at least reduce the number of objects with __del__ methods. (Whether to require that __close__ be idempotent, or to guarantee that it is run only once/instance -- that would be part of the bikeshedding.) (F) You know what to delete (or turn into weakrefs), but can't actually do it without changing a type. (F1) Why does Exception.__traceback__ reject other objects which are neither tracebacks nor None? + Can that restriction be relaxed? + Can you create a mixin subtype of Exception, which relaxes the constraint, and gets used by your framework? + Can the restriction on creating tracebacks be relaxed? + Can traceback's restriction on frames' types be relaxed? (F2) Do you need the original Exception? (see (B)) (F3) Do you care about frame.clear() raising a runtime exception? Could you suppress it (or, better, get clear() to raise something more specific, and suppress that)? It would still have released what memory it reasonably could. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From ethan at stoneleaf.us Fri Mar 7 22:25:34 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 07 Mar 2014 13:25:34 -0800 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: <531a3831.8589e00a.7525.01d4@mx.google.com> References: <531a3831.8589e00a.7525.01d4@mx.google.com> Message-ID: <531A394E.3000809@stoneleaf.us> On 03/07/2014 01:20 PM, Jim J. Jewett wrote: > > (E) Exceptions are not released even during cyclic gc, because > of ambiguity over which __del__ to run first. > > + This may be like case B or case E Um, this is case E. ;) -- ~Ethan~ From jimjjewett at gmail.com Fri Mar 7 22:53:17 2014 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Fri, 07 Mar 2014 13:53:17 -0800 (PST) Subject: [Python-Dev] Why not make frames? [was: Reference cycles in Exception.__traceback__] In-Reply-To: Message-ID: <531a3fcd.8a49e00a.7c70.0741@mx.google.com> On Thu Mar 6 16:52:56 CET 2014, Antoine Pitrou wrote: > IMO it is absolutely out of question to allow creation of arbitrary > frames from Python code, because the structure and initialization of > frames embody too many low-level implementation details. So? Does any of that matter until the frame is used to actually evaluate something? So what is the harm in creating a (likely partially invalid) frame for inspection purposes? For that matter, what is the point in tracebacks requiring frames, as opposed to any object, with the caveat that not having the expected attributes may cause grief -- as happens with any duck typing? -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From v+python at g.nevcal.com Fri Mar 7 23:38:32 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 07 Mar 2014 14:38:32 -0800 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: <531A394E.3000809@stoneleaf.us> References: <531a3831.8589e00a.7525.01d4@mx.google.com> <531A394E.3000809@stoneleaf.us> Message-ID: <531A4A68.4000903@g.nevcal.com> On 3/7/2014 1:25 PM, Ethan Furman wrote: > On 03/07/2014 01:20 PM, Jim J. Jewett wrote: >> >> (E) Exceptions are not released even during cyclic gc, because >> of ambiguity over which __del__ to run first. >> >> + This may be like case B or case E > > Um, this is case E. ;) Yeah, quite a bit like case E :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri Mar 7 23:56:44 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 7 Mar 2014 23:56:44 +0100 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: <531a3831.8589e00a.7525.01d4@mx.google.com> References: <531a3831.8589e00a.7525.01d4@mx.google.com> Message-ID: > Could you clarify what the problem actually is? Please see: http://bugs.python.org/file33238/never_deleted.py Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Mar 8 02:29:26 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 07 Mar 2014 20:29:26 -0500 Subject: [Python-Dev] undocumented help() function change in Python 3.4? In-Reply-To: References: Message-ID: On 3/7/2014 3:10 PM, Jurko Gospodneti? wrote: > Hi. > > I just noticed that the way help() function displays a function > signature changed between Python 3.3 & 3.4 but I can not find this > documented anywhere. Here's a matching example in both Python 3.3 & > Python 3.4 for comparison: > > ---------------------------------------- > >> Python 3.3.3 (v3.3.3:c3896275c0f6, Nov 18 2013, 21:19:30) [MSC v.1600 >> 64 bit (AMD64)] on win32 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> def f(a, b, c): >> ... print(a, b, c) >> ... >>>>> def g(*args, **kwargs): >> ... f(*args, **kwargs) >> ... >>>>> import inspect >>>>> print(inspect.signature(f)) >> (a, b, c) >>>>> print(inspect.signature(g)) >> (*args, **kwargs) >>>>> g.__wrapped__ = f >>>>> print(inspect.signature(f)) >> (a, b, c) >>>>> print(inspect.signature(g)) >> (a, b, c) >>>>> help(f) >> Help on function f in module __main__: >> >> f(a, b, c) >> >>>>> help(g) >> Help on function g in module __main__: >> >> g(*args, **kwargs) > > ---------------------------------------- > >> Python 3.4.0b3 (v3.4.0b3:a97ce3ecc96a, Jan 26 2014, 17:50:55) [MSC >> v.1600 64 bit (AMD64)] on win32 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> def f(a, b, c): >> ... print(a, b, c) >> ... >>>>> def g(*args, **kwargs): >> ... f(*args, **kwargs) >> ... >>>>> import inspect >>>>> print(inspect.signature(f)) >> (a, b, c) >>>>> print(inspect.signature(g)) >> (*args, **kwargs) >>>>> g.__wrapped__ = f >>>>> print(inspect.signature(f)) >> (a, b, c) >>>>> print(inspect.signature(g)) >> (a, b, c) >>>>> help(f) >> Help on function f in module __main__: >> >> f(a, b, c) >> >>>>> help(g) >> Help on function g in module __main__: >> >> g(a, b, c) > > ---------------------------------------- > > As you can see by comparing those two outputs, setting the __wrapped__ > attribute on a wrapper function affects the inspect.signature() results > on that function. This behaviour is the same in both Python 3.3. & 3.4 > and is (somewhat) described in the Python documentation. I suspect that is it intentional that the output of help is not exactly defined, just as exception messages are not. We consider either fair game to be changed in new versions. > However, help() output is not affected by this in Python 3.3, That was a bug. > but is > affected in Python 3.4, and I can not find anything regarding this in > the Python 3.4 docs. Assuming that pydoc (and hence help()) have not (yet?) been switched to using .signature directly, I suspect this is a side-effect of http://bugs.python.org/issue17481 inspect.getfullargspec should use __signature__ > Can something related to this be added at least to the 'what's changed' > docs, if not to the help() documentation as well? Perhaps add something to Other Language Changes ---------------------- * Signatures reported by help() have been modified and improved by Argument Clinic and changes to the inspect module. -- Terry Jan Reedy From ncoghlan at gmail.com Sat Mar 8 06:48:56 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 8 Mar 2014 15:48:56 +1000 Subject: [Python-Dev] undocumented help() function change in Python 3.4? In-Reply-To: References: Message-ID: On 8 March 2014 11:29, Terry Reedy wrote: > Perhaps add something to > > Other Language Changes > ---------------------- > * Signatures reported by help() have been modified and improved by Argument > Clinic and changes to the inspect module. Yeah, the changes to help() are doubly indirect - help() uses pydoc which uses inspect, and inspect saw a lot of changes. I'll make a few updates to the What's New to help make the consequences of this clearer. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From g.brandl at gmx.net Sat Mar 8 08:01:23 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 08 Mar 2014 08:01:23 +0100 Subject: [Python-Dev] cpython (3.3): Make the various iterators' "setstate" sliently and consistently clip the In-Reply-To: References: <3ffJmG2K2Yz7Llf@mail.python.org> Message-ID: Am 06.03.2014 09:02, schrieb Serhiy Storchaka: > 05.03.14 17:24, kristjan.jonsson ???????(??): >> http://hg.python.org/cpython/rev/3b2c28061184 >> changeset: 89477:3b2c28061184 >> branch: 3.3 >> parent: 89475:24d4e52f4f87 >> user: Kristj?n Valur J?nsson >> date: Wed Mar 05 13:47:57 2014 +0000 >> summary: >> Make the various iterators' "setstate" sliently and consistently clip the >> index. This avoids the possibility of setting an iterator to an invalid >> state. > > Why indexes are silently clipped instead of raising an exception? > >> files: >> Lib/test/test_range.py | 12 ++++++++++ >> Modules/arraymodule.c | 2 + >> Objects/bytearrayobject.c | 10 ++++++-- >> Objects/bytesobject.c | 10 ++++++-- >> Objects/listobject.c | 2 + >> Objects/rangeobject.c | 31 +++++++++++++++++++++++--- >> Objects/tupleobject.c | 4 +- >> Objects/unicodeobject.c | 10 ++++++-- >> 8 files changed, 66 insertions(+), 15 deletions(-) > > And it would be better first discuss and review such large changes on > the bugtracker. Agreed. Kristjan, could you please explain a bit more about this change and use the tracker in the future? Georg From victor.stinner at gmail.com Sat Mar 8 11:06:54 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 8 Mar 2014 11:06:54 +0100 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: References: <531a3831.8589e00a.7525.01d4@mx.google.com> Message-ID: 2014-03-08 1:14 GMT+01:00 Jim Jewett : >>> Could you clarify what the problem actually is? > >> Please see: >> http://bugs.python.org/file33238/never_deleted.py > > I would not expect it to be cleared at least until go runs ... and reading > the ticket, it sounds like it is cleared then. Attached script: never_deleted2.py, it's almost the same but it explains better the problem. The script creates MyObject and Future objects which are never deleted. Calling gc.collect() does *not* break the reference cycle (between the future, the exception, traceback and frames). Stopping the event loop does not remove Future nor MyObject objects. Only exiting Python does remove the Future object. The Future destructor calls code between this code fails between Python is exiting and so most symbols are set to None. I'm testing with Python 3.4 default, so with Antoine's PEP 442 and Serhiy's changes on Python shutdown (restore builtins at exit). The Future destructor is used in asyncio to notice the developer that the application has a bug, the exception was not handled which may be a major bug. And MyObject is not destroyed which is an obvious memory leak, beause there is no more explicit reference to it. Script output: --- gc.collect stop! exit --- Logging error --- Traceback (most recent call last): --- Logging error --- Traceback (most recent call last): Exception ignored in: > Traceback (most recent call last): File "Lib/asyncio/futures.py", line 184, in __del__ File "Lib/asyncio/base_events.py", line 704, in call_exception_handler File "Lib/logging/__init__.py", line 1280, in error File "Lib/logging/__init__.py", line 1386, in _log File "Lib/logging/__init__.py", line 1396, in handle File "Lib/logging/__init__.py", line 1466, in callHandlers File "Lib/logging/__init__.py", line 837, in handle File "Lib/logging/__init__.py", line 961, in emit File "Lib/logging/__init__.py", line 890, in handleError File "Lib/traceback.py", line 169, in print_exception File "Lib/traceback.py", line 153, in _format_exception_iter File "Lib/traceback.py", line 18, in _format_list_iter File "Lib/traceback.py", line 65, in _extract_tb_or_stack_iter File "Lib/linecache.py", line 15, in getline File "Lib/linecache.py", line 41, in getlines File "Lib/linecache.py", line 126, in updatecache File "Lib/tokenize.py", line 437, in open AttributeError: 'module' object has no attribute 'open' DELETE OBJET --- Victor -------------- next part -------------- A non-text attachment was scrubbed... Name: never_deleted2.py Type: text/x-python Size: 544 bytes Desc: not available URL: From ether.joe at gmail.com Sat Mar 8 11:27:25 2014 From: ether.joe at gmail.com (Sean Felipe Wolfe) Date: Sat, 8 Mar 2014 02:27:25 -0800 Subject: [Python-Dev] for Python + Java devs - real-world importance of the security model Message-ID: Hello everybody, I'm getting back into some Java game programming using the (excellent) libgdx library. It's been a couple years since I've written Java classes from scratch and it's got me thinking. The Java code I'm going through has lots 'final' and 'static' variable declarations, along with public vs private vs protected. For those of you who are Java devs in the real world ... is this security model really important? Is it protection against other developers somehow modifying state in your program via another package somewhere in the system? I've been doing Ruby code for a couple years now, and Python for a bit longer, and while I'm not a senior level dev by any means, I have yet to see where this security model is actually useful. Why not just keep it simple? For now I am writing my classes with very little protection keywords ... with as little keywords as possible actually. But I'm just getting started and I need to read up more. Any Java devs out there care to chime in? I'm not trying to troll, just interested in a response from those who appreciate KISS as a general principle. Thanks! -- A musician must make music, an artist must paint, a poet must write, if he is to be ultimately at peace with himself. - Abraham Maslow From rosuav at gmail.com Sat Mar 8 11:39:38 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 8 Mar 2014 21:39:38 +1100 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: References: <531a3831.8589e00a.7525.01d4@mx.google.com> Message-ID: On Sat, Mar 8, 2014 at 9:06 PM, Victor Stinner wrote: > And MyObject is not destroyed which is an obvious memory leak, beause > there is no more explicit reference to it. And it doesn't seem to be getting put into gc.garbage, either, which is probably worth mentioning. You have __del__ in there, so it's entirely possible it would get dropped into the garbage rather than actually disposed of, but I added a print(gc.garbage) after gc.collect() and it showed empty. ChrisA From ncoghlan at gmail.com Sat Mar 8 12:25:51 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 8 Mar 2014 21:25:51 +1000 Subject: [Python-Dev] for Python + Java devs - real-world importance of the security model In-Reply-To: References: Message-ID: On 8 March 2014 20:27, Sean Felipe Wolfe wrote: > Hello everybody, > > I'm getting back into some Java game programming using the (excellent) > libgdx library. It's been a couple years since I've written Java > classes from scratch and it's got me thinking. Sean, did you mean to send this to core Python list? It is not relevant to CPython development (it's really off topic even for python-ideas). A more appropriate venue for this kind of general programming question is the "Programmers" subsite of Stack Exchange. For example, you can see some discussion of the "private" variable question at https://programmers.stackexchange.com/questions/143736/why-do-we-need-private-variables Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sat Mar 8 12:45:01 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 8 Mar 2014 12:45:01 +0100 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] References: <531a3831.8589e00a.7525.01d4@mx.google.com> Message-ID: <20140308124501.6d809148@fsol> On Sat, 8 Mar 2014 11:06:54 +0100 Victor Stinner wrote: > > Attached script: never_deleted2.py, it's almost the same but it > explains better the problem. The script creates MyObject and Future > objects which are never deleted. Calling gc.collect() does *not* break > the reference cycle (between the future, the exception, traceback and > frames). Stopping the event loop does not remove Future nor MyObject > objects. Only exiting Python does remove the Future object. So clearly the coroutine must be kept alive by something. Regards Antoine. From victor.stinner at gmail.com Sat Mar 8 14:01:50 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 8 Mar 2014 14:01:50 +0100 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: <20140308124501.6d809148@fsol> References: <531a3831.8589e00a.7525.01d4@mx.google.com> <20140308124501.6d809148@fsol> Message-ID: 2014-03-08 12:45 GMT+01:00 Antoine Pitrou : >> Attached script: never_deleted2.py, it's almost the same but it >> explains better the problem. The script creates MyObject and Future >> objects which are never deleted. Calling gc.collect() does *not* break >> the reference cycle (between the future, the exception, traceback and >> frames). Stopping the event loop does not remove Future nor MyObject >> objects. Only exiting Python does remove the Future object. > > So clearly the coroutine must be kept alive by something. It's a reference cycle. Something like that: Future -> Exception -> Traceback -> Frames -> Local variables -> {Future, MyObject} Victor From ncoghlan at gmail.com Sat Mar 8 14:16:07 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 8 Mar 2014 23:16:07 +1000 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: References: <531a3831.8589e00a.7525.01d4@mx.google.com> <20140308124501.6d809148@fsol> Message-ID: On 8 March 2014 23:01, Victor Stinner wrote: > 2014-03-08 12:45 GMT+01:00 Antoine Pitrou : >>> Attached script: never_deleted2.py, it's almost the same but it >>> explains better the problem. The script creates MyObject and Future >>> objects which are never deleted. Calling gc.collect() does *not* break >>> the reference cycle (between the future, the exception, traceback and >>> frames). Stopping the event loop does not remove Future nor MyObject >>> objects. Only exiting Python does remove the Future object. >> >> So clearly the coroutine must be kept alive by something. > > It's a reference cycle. Something like that: > > Future -> Exception -> Traceback -> Frames -> Local variables -> > {Future, MyObject} It seems unlikely we could have an uncollectable reference cycle that doesn't end up in gc.garbage. Are you sure there are no external references to one of the objects in the cycle, thus keeping the whole thing alive? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sat Mar 8 14:33:02 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 8 Mar 2014 14:33:02 +0100 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: References: <531a3831.8589e00a.7525.01d4@mx.google.com> <20140308124501.6d809148@fsol> Message-ID: <20140308143302.39cb9435@fsol> On Sat, 8 Mar 2014 23:16:07 +1000 Nick Coghlan wrote: > On 8 March 2014 23:01, Victor Stinner wrote: > > 2014-03-08 12:45 GMT+01:00 Antoine Pitrou : > >>> Attached script: never_deleted2.py, it's almost the same but it > >>> explains better the problem. The script creates MyObject and Future > >>> objects which are never deleted. Calling gc.collect() does *not* break > >>> the reference cycle (between the future, the exception, traceback and > >>> frames). Stopping the event loop does not remove Future nor MyObject > >>> objects. Only exiting Python does remove the Future object. > >> > >> So clearly the coroutine must be kept alive by something. > > > > It's a reference cycle. Something like that: > > > > Future -> Exception -> Traceback -> Frames -> Local variables -> > > {Future, MyObject} > > It seems unlikely we could have an uncollectable reference cycle that > doesn't end up in gc.garbage. Are you sure there are no external > references to one of the objects in the cycle, thus keeping the whole > thing alive? Ok, it's actually quite trivial. The whole chain is kept alive by the "fut" global variable. If you arrange for it to be disposed of: fut = asyncio.Future() asyncio.Task(func(fut)) del fut [etc.] then the problem disappears: as soon as gc.collect() happens, the MyObject instance is destroyed, the future is collected, and the future's traceback is printed out. Regards Antoine. From victor.stinner at gmail.com Sat Mar 8 16:14:23 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 8 Mar 2014 16:14:23 +0100 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: <20140308143302.39cb9435@fsol> References: <531a3831.8589e00a.7525.01d4@mx.google.com> <20140308124501.6d809148@fsol> <20140308143302.39cb9435@fsol> Message-ID: 2014-03-08 14:33 GMT+01:00 Antoine Pitrou : > Ok, it's actually quite trivial. The whole chain is kept alive by the > "fut" global variable. If you arrange for it to be disposed of: > > fut = asyncio.Future() > asyncio.Task(func(fut)) > del fut > [etc.] > > then the problem disappears: as soon as gc.collect() happens, the > MyObject instance is destroyed, the future is collected, and the > future's traceback is printed out. Well, the problem is more general than this specific example. I would like to implement a general solution which would not hold references to local variables, to destroy objects when Python exits the except block. It looks like a "exception summary" containing only data to format the traceback would fit asyncio needs. If you don't want it in the traceback module, I will try to implement it in asyncio. It would be nice to provide an "exception summary" in the traceback module, because it looks like reference cycles related to exception and/or traceback is a common issue (see the list of links I gave in a previous email). Victor From fijall at gmail.com Sat Mar 8 16:30:12 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 8 Mar 2014 17:30:12 +0200 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: References: <531a3831.8589e00a.7525.01d4@mx.google.com> <20140308124501.6d809148@fsol> <20140308143302.39cb9435@fsol> Message-ID: On Sat, Mar 8, 2014 at 5:14 PM, Victor Stinner wrote: > 2014-03-08 14:33 GMT+01:00 Antoine Pitrou : >> Ok, it's actually quite trivial. The whole chain is kept alive by the >> "fut" global variable. If you arrange for it to be disposed of: >> >> fut = asyncio.Future() >> asyncio.Task(func(fut)) >> del fut >> [etc.] >> >> then the problem disappears: as soon as gc.collect() happens, the >> MyObject instance is destroyed, the future is collected, and the >> future's traceback is printed out. > > Well, the problem is more general than this specific example. I would > like to implement a general solution which would not hold references > to local variables, to destroy objects when Python exits the except > block. > > It looks like a "exception summary" containing only data to format the > traceback would fit asyncio needs. If you don't want it in the > traceback module, I will try to implement it in asyncio. > > It would be nice to provide an "exception summary" in the traceback > module, because it looks like reference cycles related to exception > and/or traceback is a common issue (see the list of links I gave in a > previous email). > > Victor How about fixing cyclic gc to deal with __del__ instead? That sounds like an awful change to the semantics. From pradyunsg at gmail.com Sat Mar 8 09:37:36 2014 From: pradyunsg at gmail.com (Pradyun Gedam) Date: Sat, 8 Mar 2014 14:07:36 +0530 Subject: [Python-Dev] unittest.TestCase.assert* methods calling TestCase.fail method directly. Message-ID: Hi All, This is my first time on any mailing list... Please point out any mistakes.. I had a suggestion about the implementation of unittest.TestCase.assert* methods. They all call failureException separately. This is also what the unittest.TestCase.fail method does. Is there any specific reason it is so? If not, then is there any reason stopping them from calling the unittest.TestCase.fail method? As then we have one point where all the tests fail meaning that one can reimplement the unittest.TestCase.fail method for handling test failures in their own custom way. Regards, Pradyun From solipsis at pitrou.net Sat Mar 8 18:24:21 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 8 Mar 2014 18:24:21 +0100 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: References: <531a3831.8589e00a.7525.01d4@mx.google.com> <20140308124501.6d809148@fsol> <20140308143302.39cb9435@fsol> Message-ID: <20140308182421.09e5b100@fsol> On Sat, 8 Mar 2014 16:14:23 +0100 Victor Stinner wrote: > 2014-03-08 14:33 GMT+01:00 Antoine Pitrou : > > Ok, it's actually quite trivial. The whole chain is kept alive by the > > "fut" global variable. If you arrange for it to be disposed of: > > > > fut = asyncio.Future() > > asyncio.Task(func(fut)) > > del fut > > [etc.] > > > > then the problem disappears: as soon as gc.collect() happens, the > > MyObject instance is destroyed, the future is collected, and the > > future's traceback is printed out. > > Well, the problem is more general than this specific example. I would > like to implement a general solution which would not hold references > to local variables, to destroy objects when Python exits the except > block. How about the following patch: diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -315,6 +315,7 @@ class Future: self._schedule_callbacks() if _PY34: self._log_traceback = True + self._loop.call_soon(traceback.clear_frames, exception.__traceback__) else: self._tb_logger = _TracebackLogger(exception, self._loop) # Arrange for the logger to be activated after all callbacks That said, I agree an automated mechanism would be useful. It is overwhelmingly rare to want to analyze local variables in a traceback, yet Python always keeps a reference to those. Perhaps tracebacks could have a __debug__ attribute which, when sent to False, would prevent the locals from being kept alive (but how?). Regards Antoine. From ether.joe at gmail.com Sat Mar 8 21:30:46 2014 From: ether.joe at gmail.com (Sean Felipe Wolfe) Date: Sat, 8 Mar 2014 12:30:46 -0800 Subject: [Python-Dev] for Python + Java devs - real-world importance of the security model In-Reply-To: References: Message-ID: Oops sorry I meant to send it elsewhere. Stupid auto complete :P On Sat, Mar 8, 2014 at 3:25 AM, Nick Coghlan wrote: > On 8 March 2014 20:27, Sean Felipe Wolfe wrote: >> Hello everybody, >> >> I'm getting back into some Java game programming using the (excellent) >> libgdx library. It's been a couple years since I've written Java >> classes from scratch and it's got me thinking. > > Sean, did you mean to send this to core Python list? It is not > relevant to CPython development (it's really off topic even for > python-ideas). > > A more appropriate venue for this kind of general programming question > is the "Programmers" subsite of Stack Exchange. For example, you can > see some discussion of the "private" variable question at > https://programmers.stackexchange.com/questions/143736/why-do-we-need-private-variables > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -- A musician must make music, an artist must paint, a poet must write, if he is to be ultimately at peace with himself. - Abraham Maslow From georg at python.org Sun Mar 9 11:39:14 2014 From: georg at python.org (Georg Brandl) Date: Sun, 09 Mar 2014 11:39:14 +0100 Subject: [Python-Dev] [RELEASED] Python 3.3.5 Message-ID: <531C44D2.9070501@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team, I'm very happy to announce the release of Python 3.3.5. Python 3.3.5 includes fixes for these important issues: * a 3.3.4 regression in zipimport (see http://bugs.python.org/issue20621) * a 3.3.4 regression executing scripts with a coding declared and Windows newlines (see http://bugs.python.org/issue20731) * potential DOS using compression codecs in bytes.decode() (see http://bugs.python.org/issue19619 and http://bugs.python.org/issue20404) and also fixes quite a few other bugs. Python 3.3 includes a range of improvements of the 3.x series, as well as easier porting between 2.x and 3.x. In total, almost 500 API items are new or improved in Python 3.3. For a more extensive list of changes in the 3.3 series, see http://docs.python.org/3.3/whatsnew/3.3.html To download Python 3.3.5 visit: http://www.python.org/downloads/release/python-335/ This is a production release, please report any bugs to http://bugs.python.org/ The final release is scheduled one week from now. Enjoy! - -- Georg Brandl, Release Manager georg at python.org (on behalf of the entire python-dev team and 3.3's contributors) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlMcRNEACgkQN9GcIYhpnLAVqACeMoOOuuNX5iP6at9zbIZDnkqK idwAoKb2FxAJlirhs2BmdESEaQiqBDJd =smeC -----END PGP SIGNATURE----- From jimjjewett at gmail.com Mon Mar 10 03:16:38 2014 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Sun, 09 Mar 2014 19:16:38 -0700 (PDT) Subject: [Python-Dev] Why not make frames? [was: Alternative forms [was: PEP 463: Exception-catching expressions]] In-Reply-To: Message-ID: <531d2086.b1528c0a.0878.ffffa138@mx.google.com> TL;DR: expr except (default if exc_expr) expr (except default if exc_expr) expr except (exc_expr: default) expr (except exc_expr: default) (1) Group the exceptions with the default they imply. (2) inline-":" still needs () or [] or {}. (3) Consider the expression inside a longer line. (3a) Does the except expression need to be general, or would it work if it were limited to a subclause of variable assignments? (3b) What about comprehensions? On Fri Mar 7 20:54:31 CET 2014, Chris Angelico wrote: >On Sat, Mar 8, 2014 at 5:58 AM, Jim J. Jewett wrote: >> (Thu Mar 6 23:26:47 CET 2014) Chris Angelico responded: >>> On Fri, Mar 7, 2014 at 7:29 AM, Jim J. Jewett wrote: >>>> [ note that "x if y" already occurs in multiple contexts, and >>>> always evaluates y before x. ] ... > I don't see except expressions as fundamentally more associated with > if/else than with, say, an or chain, which works left to right. I do, because of the skipping portion. Short-circuiting operators, such as an "or" chain, never skip a clause unless they are skipping *every* subsequent clause. An "if" statement sometimes skips the (unlabeled in python) "then" clause, but still processes the even-later "else" clause. A "try" statement sometimes skips the remainder of the try suite but still executes the later subordinate "except" and "finally" clauses. Note that this only explains why I see "except" as more closely related to "if" than to "or"; it isn't sufficient to justify going back to execute the skipped clause later. That said, going back to a previous location is a lot easier to excuse after an error handler than in "regular" code. > Analysis of the Python standard library suggests that the single-if > situation is *by far* the most common, to the extent that it'd hardly > impact the stdlib at all to add multiple except clauses to the > proposal. Do you have a strong use-case for the more full syntax? I do not. I dislike the arbitrary restriction, and I worry that lifting it later (while maintaining backwards compatibility) will result in a syntax wart, but I do not have a compelling use case for that later relaxation. >> and I strongly prefer that they [the parentheses] be internal >> (which you fear looks too much like calling a function named except). >> In that case, it is: >> expr1 except (expr3 if expr2) > I'm still not really seeing how this is better. For one thing, it makes it clear that the "if" keyword may be messing with the order of evaluation. I don't claim that syntax is perfect. I do think it is less flawed than the no-parentheses (or external parentheses) versions: (expr1 except expr3 if expr2) expr1 except expr3 if expr2 because the tigher parentheses correctly indicate that expr2 and expr3 should be considered as a (what-to-do-in-case-of-error) group, which interacts (as a single unit) with the main expression. I also think it is (very slighly) better than the colon+internal-parentheses version: expr1 except (expr2: expr3) which in turn is far, far better than the colon versions with external or missing parentheses: (expr1 except expr2: expr3) expr1 except expr2: expr3 because I cannot imagine reading an embedded version of either of those without having to mentally re-parse at the colon. An example assuming a precedence level that may not be what the PEP proposes: if myfunc(5, expr1 except expr2: expr3, "label"): for i in range(3, 3*max(data) except TypeError: 9, 3): ... if myfunc(5, (expr1 except expr2: expr3), "label"): for i in range(3, (3*max(data) except TypeError: 9), 3): ... if myfunc(5, expr1 except (expr2: expr3), "label"): for i in range(3, 3*max(data) except (TypeError: 9), 3): ... if myfunc(5, expr1 except (expr2: expr3), "label"): for i in range(3, 3*max(data) (except TypeError: 9), 3): ... if myfunc(5, expr1 except (expr3 if expr3), "label"): for i in range(3, 3*max(data) (except 9 if TypeError), 3): ... if myfunc(5, expr1 except (expr3 if expr3), "label"): for i in range(3, 3*max(data) except (9 if TypeError), 3): myarg = expr1 except (expr3 if expr2) if myfunc(5, myarg, "label"): limit = 3*max(data) except (9 if TypeError) for i in range(3, limit, 3): Yes, I would prefer to create a variable naming those expressions, but these are all still simple enough that I would expect to have to read them. (I like constructions that get ugly just a bit faster than they get hard to understand.) If I have to parse any of them, the ones at the bottom are less difficult than the ones at the top. > With the colon version, it looks very much like dict display, which is good, since that is one of the acceptable uses of inline-colon. > only with different brackets around it; in some fonts, that'll be > very easily confused. I've had more trouble with comma vs period than with different types of bracket. But let's assume that there is confusion, and someone sees: expr1 except [expr2:expr3] or expr1 except {expr2:expr3} These are not yet defined any more than the tuple-with-colon version is, nor do they have an obvious-yet-incompatible meaning. In fact, I would prefer either of them to any version that does not syntactically associate the exception list with the result those exceptions imply. >> The other three inline uses (dict display, slide notation, and >> function parameter annotation) are effectively conjunction operators, >> saying that expr1 and expr2 are bound more tightly than you would >> assume if they were separated by commas. They only occur inside >> a fairly close bracket (of some sort), and if the bracket isn't >> *very* close, then there are usually multiple associates-colons >> inside the same bracket. > Not sure that really helps. This isn't going to be as tight as a > slice, and it's most often not going to have multiple colons inside > the brackets. The colon is acceptable only to the extent that this similarity does help. So if a colon is used, I want the similarity to be as strong as possible -- which, I suppose, is another argument for tightening the parentheses, and possibly an argument for using [] instead of (). -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From jimjjewett at gmail.com Mon Mar 10 03:35:51 2014 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Sun, 09 Mar 2014 19:35:51 -0700 (PDT) Subject: [Python-Dev] Scope issues [was: Alternative forms [was: PEP 463: Exception-catching expressions]] In-Reply-To: Message-ID: <531d2507.4f138c0a.234d.ffffa519@mx.google.com> On Fri Mar 7 20:54:31 CET 2014, Chris Angelico wrote: > On Sat, Mar 8, 2014 at 5:58 AM, Jim J. Jewett wrote: >> (Thu Mar 6 23:26:47 CET 2014) Chris Angelico responded: >>> ...[as-capturing is] deferred until there's a non-closure means of >>> creating a sub-scope. >> The problem is that once it is deployed as leaking into the parent >> scope, backwards compatibility may force it to always leak into >> the parent scope. (You could document the leakage as a bug or as >> implementation-defined, but ... those choices are also sub-optimal.) > It'll never be deployed as leaking, for the same reason that the > current 'except' statement doesn't leak: I don't think that is the full extent of the problem. From Nick's description, this is a nasty enough corner case that there may be glitches no one notices in time. The PEP should therefore explicitly state that implementation details may force the deferral to be permanent, and that this is considered an acceptable trade-off. -jJ -- Sorry for the botched subject line on the last previous message. If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From rosuav at gmail.com Mon Mar 10 04:26:14 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 10 Mar 2014 14:26:14 +1100 Subject: [Python-Dev] Why not make frames? [was: Alternative forms [was: PEP 463: Exception-catching expressions]] In-Reply-To: <531d2086.b1528c0a.0878.ffffa138@mx.google.com> References: <531d2086.b1528c0a.0878.ffffa138@mx.google.com> Message-ID: On Mon, Mar 10, 2014 at 1:16 PM, Jim J. Jewett wrote: > On Fri Mar 7 20:54:31 CET 2014, Chris Angelico wrote: >> I don't see except expressions as fundamentally more associated with >> if/else than with, say, an or chain, which works left to right. > > I do, because of the skipping portion. > > Short-circuiting operators, such as an "or" chain, never skip a clause > unless they are skipping *every* subsequent clause. > > An "if" statement sometimes skips the (unlabeled in python) "then" > clause, but still processes the even-later "else" clause. > > A "try" statement sometimes skips the remainder of the try suite but > still executes the later subordinate "except" and "finally" clauses. > > Note that this only explains why I see "except" as more closely related > to "if" than to "or"; it isn't sufficient to justify going back to > execute the skipped clause later. That said, going back to a previous > location is a lot easier to excuse after an error handler than in > "regular" code. This is a rather tenuous connection, I think. It also doesn't justify a fundamentally out-of-order evaluation. Compare all of these, which are - unless I've made a mistake somewhere - evaluated in the order shown (that is, no lower-numbered will be evaluated after a higher-numbered, but of course not everything will be evaluated): if expr1: stmt2 elif expr3: stmt4 else: stmt5 try: stmt1 except expr2: stmt3 except expr4: stmt5 except: stmt6 else: stmt7 finally: stmt8 value = expr1 or expr2 or expr3 or expr4 value = expr1 and expr2 and expr3 and expr4 # With the current recommendations in PEP 463 value = (expr1 except expr2: expr3) value = expr2 if expr1 else expr3 The if expression is the *only* one that's out of order, and it's justified by (a) "reading" correctly (ie being similar to English), and (b) using two existing keywords, rather than creating a 'then' keyword or using symbols. Useful reading: http://legacy.python.org/dev/peps/pep-0308/ Skip down to the "Detailed Results of Voting" section (sorry, text/plain PEPs don't do hash links) and note that these four proposals received the most support: A. x if C else y B. if C then x else y C. (if C: x else: y) D. C ? x : y with more support for C and D than for A and B. B requires a new keyword, D is all symbols, C is indistinguishable from the statement without the parens, and A puts the expressions in the wrong order. All of them are somewhat flawed but all of them have some merit, too. The BDFL went for A (which proves that Python isn't a democracy), and there's no explanation of the reasoning behind that, but in the absence of a strong statement on the subject, I would say that the gap between the above four options is sufficiently narrow that this can't be used to justify out-of-order syntax in any other proposal. >> Analysis of the Python standard library suggests that the single-if >> situation is *by far* the most common, to the extent that it'd hardly >> impact the stdlib at all to add multiple except clauses to the >> proposal. Do you have a strong use-case for the more full syntax? > > I do not. > > I dislike the arbitrary restriction, and I worry that lifting it later > (while maintaining backwards compatibility) will result in a syntax > wart, but I do not have a compelling use case for that later relaxation. That's one reason for the mandatory external parentheses. With some real-world usage data from Python 3.5, if this were implemented exactly per the PEP, a decision could be made in 3.6 to lift one of two restrictions: 3.5: (expr1 except Exception2: expr3) 3.6 choice 1: expr1 except Exception2: expr3 3.6 choice 2: (expr1 except Exception2: expr3 except Exception4: expr5) But these are incompatible. Lifting either restriction mandates the retention of the other. Note also that internal parens are less easily removed - it becomes two distinct code branches. Removing the requirement to have external parents simply means that old-style code is putting parens around an expression, no different from: value = (1 if x else 2) value = (3 + 5) and completely harmless. Allowing optional internal parens means maintaining two syntaxes forever, and both human and computer parsers would have to be compatible with both. >>> and I strongly prefer that they [the parentheses] be internal >>> (which you fear looks too much like calling a function named except). >>> In that case, it is: > >>> expr1 except (expr3 if expr2) > >> I'm still not really seeing how this is better. > > For one thing, it makes it clear that the "if" keyword may be messing > with the order of evaluation. It still could be waiting for its else. I mean, it's bad code style, but you theoretically could do this: foo[x] except (IndexError if isinstance(foo,list) else KeyError): None Really bad style and horribly unclear (and anyway, you could just catch LookupError), but grammatically legal. Anything using the word 'if' in an expression context has to consider that. (Note that the converse of using 'else' on its own wouldn't be a problem, as it wouldn't be following an if.) > I don't claim that syntax is perfect. I do think it is less flawed > than the no-parentheses (or external parentheses) versions: > > (expr1 except expr3 if expr2) > expr1 except expr3 if expr2 > > because the tigher parentheses correctly indicate that expr2 and expr3 > should be considered as a (what-to-do-in-case-of-error) group, which > interacts (as a single unit) with the main expression. But it doesn't, really. The entire set of three expressions is a single unit. You can't break out the bit inside the parens and give that a name, like you can in most places where something "acts as a single unit" to interact with something else. (Yes, there are special cases, like the syntax for constructing slice objects that works only inside square brackets. And you can't break out a function's arguments, as a unit, into a single object (the nearest is *args,**kw). I said most places, and I don't want to add more to the special-case set.) > which in turn is far, far better than the colon versions with external > or missing parentheses: > > (expr1 except expr2: expr3) > expr1 except expr2: expr3 > > because I cannot imagine reading an embedded version of either of those > without having to mentally re-parse at the colon. What do you mean by "re-parse"? The general recommendation is that the entire except expression should fit inside a human's 7 +/- 2 limit. Fundamentally, a good use of this syntax will still be saying basically the same thing at the end that it seemed to be saying at the beginning, in the same way that short-circuiting or can be used. >>> dic = {7:"Seven", 9:None, 5:"Five"} >>> dic[7] or "Default" 'Seven' >>> dic[9] or "Default" 'Default' It's still looking something up, but now it adds a default for the case where the stored value is falsy. Now compare the (slightly more verbose) version that catches an exception: >>> (dic[3] except KeyError: "Not present") 'Not present' Again, it's still looking something up, but now coping with an additional situation. > I've had more trouble with comma vs period than with different types > of bracket. But let's assume that there is confusion, and someone sees: > > expr1 except {expr2:expr3} (most fonts will make square brackets clear enough, it's only braces and round brackets that are problematic) > These are not yet defined any more than the tuple-with-colon version is, > nor do they have an obvious-yet-incompatible meaning. In fact, I would > prefer either of them to any version that does not syntactically > associate the exception list with the result those exceptions imply. Actually, braces with colons inside *is* defined. What's not defined is the word "except" followed by a dictionary. But at the point where you're reading that, it looks like a dict. The trouble is that that's *almost* valid; you can think of the exception handling as mapping an exception to its result. But that's not valid for two reasons: firstly, exception handling is inherently ordered, and secondly, the expressions are not evaluated eagerly, as they would be in a dict. Only the one that matches will be evaluated. Making the syntax look like it's laying out a dictionary would be awkwardly confusing. > The colon is acceptable only to the extent that this similarity does > help. So if a colon is used, I want the similarity to be as strong > as possible -- which, I suppose, is another argument for tightening > the parentheses, and possibly an argument for using [] instead of (). Tightening them doesn't particularly help, but is an arguable point. But using square brackets? Definitely not. Parentheses are used syntactically in many places (grouping, function calls, and they're associated with tuples), but square brackets are not; plus, see argument above about the option of relaxing the restrictions. Using square brackets would be confusing, and would often be interpreted as creating a list. Not something we want to imply. ChrisA From rosuav at gmail.com Mon Mar 10 04:29:47 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 10 Mar 2014 14:29:47 +1100 Subject: [Python-Dev] Scope issues [was: Alternative forms [was: PEP 463: Exception-catching expressions]] In-Reply-To: <531d2507.4f138c0a.234d.ffffa519@mx.google.com> References: <531d2507.4f138c0a.234d.ffffa519@mx.google.com> Message-ID: On Mon, Mar 10, 2014 at 1:35 PM, Jim J. Jewett wrote: > The PEP should therefore explicitly state that implementation details > may force the deferral to be permanent, and that this is considered an > acceptable trade-off. How about words to this effect? """Should there be, in future, a way to create a true subscope (which could simplify comprehensions, except expressions, with blocks, and possibly more), then this proposal could be revived; until then, its loss is not a great one...""" (I parked the time machine in the PSU's garage, thanks for lending me it!) (The PSU does not exist, and therefore does not have a garage.) ChrisA From larry at hastings.org Mon Mar 10 08:47:05 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 10 Mar 2014 00:47:05 -0700 Subject: [Python-Dev] [RELEASED] Python 3.4.0 release candidate 3 Message-ID: <531D6DF9.1020305@hastings.org> On behalf of the Python development team, I'm pleased to announce the third and final** release candidate of Python 3.4. This is a preview release, and its use is not recommended for production settings. Python 3.4 includes a range of improvements of the 3.x series, including hundreds of small improvements and bug fixes. Major new features and changes in the 3.4 release series include: * PEP 428, a "pathlib" module providing object-oriented filesystem paths * PEP 435, a standardized "enum" module * PEP 436, a build enhancement that will help generate introspection information for builtins * PEP 442, improved semantics for object finalization * PEP 443, adding single-dispatch generic functions to the standard library * PEP 445, a new C API for implementing custom memory allocators * PEP 446, changing file descriptors to not be inherited by default in subprocesses * PEP 450, a new "statistics" module * PEP 451, standardizing module metadata for Python's module import system * PEP 453, a bundled installer for the *pip* package manager * PEP 454, a new "tracemalloc" module for tracing Python memory allocations * PEP 456, a new hash algorithm for Python strings and binary data * PEP 3154, a new and improved protocol for pickled objects * PEP 3156, a new "asyncio" module, a new framework for asynchronous I/O Python 3.4 is now in "feature freeze", meaning that no new features will be added. The final release is projected for March 16, 2014. To download Python 3.4.0rc3 visit: http://www.python.org/download/releases/3.4.0/ Please consider trying Python 3.4.0rc3 with your code and reporting any new issues you notice to: http://bugs.python.org/ Enjoy! ** This time we really mean it! No foolin'! -- Larry Hastings, Release Manager larry at hastings.org (on behalf of the entire python-dev team and 3.4's contributors) From jurko.gospodnetic at pke.hr Mon Mar 10 10:13:54 2014 From: jurko.gospodnetic at pke.hr (=?windows-1250?Q?Jurko_Gospodneti=E6?=) Date: Mon, 10 Mar 2014 10:13:54 +0100 Subject: [Python-Dev] Python 3.4 change in importlib/__init__.py breaking cxFreeze? Message-ID: Hi all. Python 3.4 introduced a change to Lib/importlib/__init__.py that added the following code to it: > else: > # importlib._bootstrap is the built-in import, ensure we don't create > # a second copy of the module. > _bootstrap.__name__ = 'importlib._bootstrap' > _bootstrap.__package__ = 'importlib' > _bootstrap.__file__ = __file__.replace('__init__.py', '_bootstrap.py') > sys.modules['importlib._bootstrap'] = _bootstrap When attempting to use cxFreeze on a project, using Python 3.4. we ran into a problem with '__file__' identifier not being defined. Could this be a python bug? Why is this code expecting the module loaded from importlib/__init__.py to always have a __file__ identifier? What is supposed to happen when that code gets loaded from a ZIP archive? Just wanted to check here before filing an issue... but if this is an issue I hope it can be resolved before the final 3.4 release. Best regards, Jurko Gospodneti? From mark at hotpy.org Mon Mar 10 10:45:44 2014 From: mark at hotpy.org (Mark Shannon) Date: Mon, 10 Mar 2014 09:45:44 +0000 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: References: <531a3831.8589e00a.7525.01d4@mx.google.com> <20140308124501.6d809148@fsol> <20140308143302.39cb9435@fsol> Message-ID: <531D89C8.90101@hotpy.org> On 08/03/14 15:30, Maciej Fijalkowski wrote: > On Sat, Mar 8, 2014 at 5:14 PM, Victor Stinner wrote: >> 2014-03-08 14:33 GMT+01:00 Antoine Pitrou : >>> Ok, it's actually quite trivial. The whole chain is kept alive by the >>> "fut" global variable. If you arrange for it to be disposed of: >>> >>> fut = asyncio.Future() >>> asyncio.Task(func(fut)) >>> del fut >>> [etc.] >>> >>> then the problem disappears: as soon as gc.collect() happens, the >>> MyObject instance is destroyed, the future is collected, and the >>> future's traceback is printed out. >> >> Well, the problem is more general than this specific example. I would >> like to implement a general solution which would not hold references >> to local variables, to destroy objects when Python exits the except >> block. >> >> It looks like a "exception summary" containing only data to format the >> traceback would fit asyncio needs. If you don't want it in the >> traceback module, I will try to implement it in asyncio. >> >> It would be nice to provide an "exception summary" in the traceback >> module, because it looks like reference cycles related to exception >> and/or traceback is a common issue (see the list of links I gave in a >> previous email). >> >> Victor > > How about fixing cyclic gc to deal with __del__ instead? That sounds > like an awful change to the semantics. +1 Cheers, Mark. From victor.stinner at gmail.com Mon Mar 10 10:43:47 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 10 Mar 2014 10:43:47 +0100 Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.0 release candidate 3 In-Reply-To: <531D6DF9.1020305@hastings.org> References: <531D6DF9.1020305@hastings.org> Message-ID: Hi, I tested Python 3.4rc3 installer on Windows: - all menu entries work (doc, doc server, help, IDLE, command line, etc.) - pip is installed, yeah! - my tracemalloc module works on Windows too ;-) - The uninstaller fully removes C:\Python34 The major Windows installer issues are fixed in the RC3. Note: As fully, I got a popup telling me that the MSI is untrusted, and I had to click on [OK] to confirm that I trust python.org. I also tried to change pip version: I downgraded it from 1.5.4 to 1.5.3. In this case, C:\Python34 is not removed, I filled an issue for that, but it's maybe expected? http://bugs.python.org/issue20878 When I tested tracemalloc, I also found a "memory regression": the "base64 module of Python 3.4 uses 920 kB of memory". I filled an issue for this. It's not critical, it can wait Python 3.4.1. I'm happy that tracemalloc is now built in Python and so memory can be tracked more easily! http://bugs.python.org/issue20879 Victor 2014-03-10 8:47 GMT+01:00 Larry Hastings : > > > On behalf of the Python development team, I'm pleased to announce > the third and final** release candidate of Python 3.4. > > This is a preview release, and its use is not recommended for > production settings. > > Python 3.4 includes a range of improvements of the 3.x series, including > hundreds of small improvements and bug fixes. Major new features and > changes in the 3.4 release series include: > > * PEP 428, a "pathlib" module providing object-oriented filesystem paths > * PEP 435, a standardized "enum" module > * PEP 436, a build enhancement that will help generate introspection > information for builtins > * PEP 442, improved semantics for object finalization > * PEP 443, adding single-dispatch generic functions to the standard library > * PEP 445, a new C API for implementing custom memory allocators > * PEP 446, changing file descriptors to not be inherited by default > in subprocesses > * PEP 450, a new "statistics" module > * PEP 451, standardizing module metadata for Python's module import system > * PEP 453, a bundled installer for the *pip* package manager > * PEP 454, a new "tracemalloc" module for tracing Python memory allocations > * PEP 456, a new hash algorithm for Python strings and binary data > * PEP 3154, a new and improved protocol for pickled objects > * PEP 3156, a new "asyncio" module, a new framework for asynchronous I/O > > Python 3.4 is now in "feature freeze", meaning that no new features will be > added. The final release is projected for March 16, 2014. > > > To download Python 3.4.0rc3 visit: > > http://www.python.org/download/releases/3.4.0/ > > > Please consider trying Python 3.4.0rc3 with your code and reporting any > new issues you notice to: > > http://bugs.python.org/ > > > Enjoy! > > > ** This time we really mean it! No foolin'! > > -- > Larry Hastings, Release Manager > larry at hastings.org > (on behalf of the entire python-dev team and 3.4's contributors) > _______________________________________________ > python-committers mailing list > python-committers at python.org > https://mail.python.org/mailman/listinfo/python-committers From victor.stinner at gmail.com Mon Mar 10 11:10:26 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 10 Mar 2014 11:10:26 +0100 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: References: <531a3831.8589e00a.7525.01d4@mx.google.com> <20140308124501.6d809148@fsol> <20140308143302.39cb9435@fsol> Message-ID: 2014-03-08 16:30 GMT+01:00 Maciej Fijalkowski : > How about fixing cyclic gc to deal with __del__ instead? That sounds > like an awful change to the semantics. Hum? That's the purpose of the PEP 442 which is implemented in Python 3.4. As I wrote, it's not enough to fix all issues. Usually, I see an explicit call to gc.collect() as a workaround to a deeper issue. I prefer to modify my program to run smoothly without explict garbage collection. That's why I would prefer to avoid creating reference cycles from the beginning. Victor From fijall at gmail.com Mon Mar 10 13:11:40 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 10 Mar 2014 14:11:40 +0200 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: References: <531a3831.8589e00a.7525.01d4@mx.google.com> <20140308124501.6d809148@fsol> <20140308143302.39cb9435@fsol> Message-ID: On Mon, Mar 10, 2014 at 12:10 PM, Victor Stinner wrote: > 2014-03-08 16:30 GMT+01:00 Maciej Fijalkowski : >> How about fixing cyclic gc to deal with __del__ instead? That sounds >> like an awful change to the semantics. > > Hum? That's the purpose of the PEP 442 which is implemented in Python 3.4. > > As I wrote, it's not enough to fix all issues. > > Usually, I see an explicit call to gc.collect() as a workaround to a > deeper issue. I prefer to modify my program to run smoothly without > explict garbage collection. > > That's why I would prefer to avoid creating reference cycles from the beginning. > > Victor It was agreed long time ago that the immediate finalization is an implementation specific detail and it's not guaranteed. You should not rely on __del__s being called timely one way or another. Why would you require this for the program to work correctly in the particular example of __traceback__? Cheers, fijal From jurko.gospodnetic at pke.hr Mon Mar 10 14:03:24 2014 From: jurko.gospodnetic at pke.hr (=?windows-1250?Q?Jurko_Gospodneti=E6?=) Date: Mon, 10 Mar 2014 14:03:24 +0100 Subject: [Python-Dev] Windows 'for current user' installation - 32/64-bit registrations overwrite each other Message-ID: Hi. When running the Python Windows installer 'for all users', the 32-bit installation and the 64-bit installation each gets a separate registration in the Windows registry. E.g. under: HKEY_LOCAL_MACHINE\Software\Python\PythonCore\3.4 and: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Python\PythonCore\3.4 However, when running the installer 'for the current user only', the 32-bit and the 64-bit installation both use the same Windows registry key: HKEY_CURRENT_USER\Software\Python\PythonCore\3.4 Note that a 64-bit installation does not automatically uninstall a previous 32-bit installation in either case, or vice versa. Why the mismatch? As I understand it, whichever installation you use, you should be able to run a specific Python interpreter using the py.exe runner, as it: py.exe -3.4 py.exe -3.4-32 As it stands now - both of those run the same interpreter when they have been installed 'for the current user only'. Is this as issue or desired behaviour? Should I open an issue for it? Best regards, Jurko Gospodneti? From ncoghlan at gmail.com Mon Mar 10 14:04:19 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 10 Mar 2014 23:04:19 +1000 Subject: [Python-Dev] [Python-checkins] cpython: whatsnew: venv with_pip, pyvenv --without-pip (#19552) In-Reply-To: <3fhyT2594sz7Lm1@mail.python.org> References: <3fhyT2594sz7Lm1@mail.python.org> Message-ID: On 10 Mar 2014 08:51, "r.david.murray" wrote: > > http://hg.python.org/cpython/rev/a140caad76bc > changeset: 89539:a140caad76bc > user: R David Murray > date: Sun Mar 09 18:09:51 2014 -0400 > summary: > whatsnew: venv with_pip, pyvenv --without-pip (#19552) > > files: > Doc/whatsnew/3.4.rst | 11 +++++++++++ > 1 files changed, 11 insertions(+), 0 deletions(-) > > > diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst > --- a/Doc/whatsnew/3.4.rst > +++ b/Doc/whatsnew/3.4.rst > @@ -1507,6 +1507,12 @@ > :mod:`venv` now includes activation scripts for the ``csh`` and ``fish`` > shells (Contributed by Andrew Svetlov in :issue:`15417`.) > > +:class:`~venv.EnvBuilder` and the :func:`~venv.create` convenience function > +take a new keyword argument *with_pip*, which defaults to ``False``, that > +controls whether or not :class:`~venv.EnvBuilder` ensures that ``pip`` is > +installed in the virtual environment. (Contributed by Nick Coghlan in > +:issue:`19552` as part of the :pep:`453` implementation.) > + > > wave > ---- > @@ -1772,6 +1778,11 @@ > to use copies rather than symlinks even on systems where symlinks are the > default. (Contributed by Vinay Sajip in :issue:`18807`.) > > +* The :ref:`pyvenv ` command also accepts a ``--without-pip`` > + option to suppress the otherwise-automatic bootstrapping of pip into > + the virtual environment. (Contributed by Nick Coghlan in :issue:`19552` > + as part of the :pep:`453` implementation.) It just occurred to me that we should probably have a porting note about that CLI change. We decided that changing the default CLI behaviour, while preserving the default module API was the best available compromise, but anyone that is invoking the CLI programmatically and wants to skip bootstrapping pip will still need to change their command. Cheers, Nick. > + > * The encoding name is now optional in the value set for the > :envvar:`PYTHONIOENCODING` environment variable. This makes it possible to > set just the error handler, without changing the default encoding. > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Mar 10 14:09:53 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 10 Mar 2014 23:09:53 +1000 Subject: [Python-Dev] [Python-checkins] cpython: whatsnew: 'U' mode deprecation (#15204). In-Reply-To: <3fj2742w8Tz7Lm2@mail.python.org> References: <3fj2742w8Tz7Lm2@mail.python.org> Message-ID: On 10 Mar 2014 11:36, "r.david.murray" wrote: > > http://hg.python.org/cpython/rev/2d5544afb510 > changeset: 89547:2d5544afb510 > user: R David Murray > date: Sun Mar 09 20:33:35 2014 -0400 > summary: > whatsnew: 'U' mode deprecation (#15204). > > files: > Doc/whatsnew/3.4.rst | 14 +++++++++++++- > 1 files changed, 13 insertions(+), 1 deletions(-) > > > diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst > --- a/Doc/whatsnew/3.4.rst > +++ b/Doc/whatsnew/3.4.rst > @@ -1948,6 +1948,11 @@ > * The :mod:`sysconfig` key ``SO`` is deprecated, it has been replaced by > ``EXT_SUFFIX``. > > +* The ``U`` mode accepted by various ``open`` functions is deprecated. > + In Python3 it does not do anything useful, and should be replaced by > + appropriate uses of :class:`io.TextIOWrapper` (if needed) and its *newline* > + argument. > + Huh, I missed that change. Don't we still need U when writing 2/3 compatible code at this point? If so, perhaps we may want to reconsider this particular deprecation in 3.4.1. Cheers, Nick. > > Deprecations in the C API > ------------------------- > @@ -2242,7 +2247,14 @@ > > * Since the *digestmod* argument to the :func:`hmac.new` function will in the > future have no default, all calls to :func:`hmac.new` should be changed to > - explicitly specify a *digestmod*. (:issue:`17276`). > + explicitly specify a *digestmod* (:issue:`17276`). > + > +* Any calls to ``open`` functions that specify ``U`` should be modified. > + ``U`` is ineffective in Python3 and will eventually raise an error if used. > + Depending on the function, the equivalent of its old Python2 behavior can be > + achieved using either a *newline* argument, or if necessary by wrapping the > + stream in :mod:`~io.TextIOWrapper` to use its *newline* argument > + (:issue:`15204`). > > > Changes in the C API > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jurko.gospodnetic at pke.hr Mon Mar 10 14:15:28 2014 From: jurko.gospodnetic at pke.hr (=?UTF-8?B?SnVya28gR29zcG9kbmV0acSH?=) Date: Mon, 10 Mar 2014 14:15:28 +0100 Subject: [Python-Dev] undocumented help() function change in Python 3.4? In-Reply-To: References: Message-ID: <531DBAF0.7010903@pke.hr> Hi. On 8.3.2014. 6:48, Nick Coghlan wrote: > Yeah, the changes to help() are doubly indirect - help() uses pydoc > which uses inspect, and inspect saw a lot of changes. > > I'll make a few updates to the What's New to help make the > consequences of this clearer. Just looked through the latest 3.4rc3 release and nothing related to this seems to have been included yet. Best regards, Jurko Gospodneti? From ncoghlan at gmail.com Mon Mar 10 14:17:57 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 10 Mar 2014 23:17:57 +1000 Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.0 release candidate 3 In-Reply-To: References: <531D6DF9.1020305@hastings.org> Message-ID: On 10 Mar 2014 19:50, "Victor Stinner" wrote: > > Hi, > > I tested Python 3.4rc3 installer on Windows: > > - all menu entries work (doc, doc server, help, IDLE, command line, etc.) > - pip is installed, yeah! > - my tracemalloc module works on Windows too ;-) > - The uninstaller fully removes C:\Python34 > > The major Windows installer issues are fixed in the RC3. Huzzah! :) > Note: As fully, I got a popup telling me that the MSI is untrusted, > and I had to click on [OK] to confirm that I trust python.org. > > I also tried to change pip version: I downgraded it from 1.5.4 to > 1.5.3. In this case, C:\Python34 is not removed, I filled an issue for > that, but it's maybe expected? > http://bugs.python.org/issue20878 Yeah, that's expected - if pip is used to install extra packages, the uninstaller has historically left them alone. Changing the version of pip causes it to be treated like an independently installed package, so the uninstaller leaves it behind. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Mon Mar 10 14:18:03 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 10 Mar 2014 09:18:03 -0400 Subject: [Python-Dev] [Python-checkins] cpython: whatsnew: 'U' mode deprecation (#15204). In-Reply-To: References: <3fj2742w8Tz7Lm2@mail.python.org> Message-ID: <20140310131803.C68BC250CBE@webabinitio.net> On Mon, 10 Mar 2014 23:09:53 +1000, Nick Coghlan wrote: > On 10 Mar 2014 11:36, "r.david.murray" wrote: > > > > http://hg.python.org/cpython/rev/2d5544afb510 > > changeset: 89547:2d5544afb510 > > user: R David Murray > > date: Sun Mar 09 20:33:35 2014 -0400 > > summary: > > whatsnew: 'U' mode deprecation (#15204). > > > > files: > > Doc/whatsnew/3.4.rst | 14 +++++++++++++- > > 1 files changed, 13 insertions(+), 1 deletions(-) > > > > > > diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst > > --- a/Doc/whatsnew/3.4.rst > > +++ b/Doc/whatsnew/3.4.rst > > @@ -1948,6 +1948,11 @@ > > * The :mod:`sysconfig` key ``SO`` is deprecated, it has been replaced by > > ``EXT_SUFFIX``. > > > > +* The ``U`` mode accepted by various ``open`` functions is deprecated. > > + In Python3 it does not do anything useful, and should be replaced by > > + appropriate uses of :class:`io.TextIOWrapper` (if needed) and its > *newline* > > + argument. > > + > > Huh, I missed that change. Don't we still need U when writing 2/3 > compatible code at this point? If so, perhaps we may want to reconsider > this particular deprecation in 3.4.1. It's a deprecation, not a removal. I believe the idea is to not remove it before 4.0. --David From victor.stinner at gmail.com Mon Mar 10 14:23:35 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 10 Mar 2014 14:23:35 +0100 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: References: <531a3831.8589e00a.7525.01d4@mx.google.com> <20140308124501.6d809148@fsol> <20140308143302.39cb9435@fsol> Message-ID: 2014-03-10 13:11 GMT+01:00 Maciej Fijalkowski : > It was agreed long time ago that the immediate finalization is an > implementation specific detail and it's not guaranteed. You should not > rely on __del__s being called timely one way or another. Why would you > require this for the program to work correctly in the particular > example of __traceback__? For asyncio, it's very useful to see unhandled exceptions as early as possible. Otherwise, your program is blocked and you don't know why. Guido van Rossum suggests to run gc.collect() regulary: http://code.google.com/p/tulip/issues/detail?id=42 Victor From ncoghlan at gmail.com Mon Mar 10 14:25:17 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 10 Mar 2014 23:25:17 +1000 Subject: [Python-Dev] Python 3.4 change in importlib/__init__.py breaking cxFreeze? In-Reply-To: References: Message-ID: On 10 Mar 2014 19:15, "Jurko Gospodneti?" wrote: > > Hi all. > > Python 3.4 introduced a change to Lib/importlib/__init__.py that added the following code to it: > >> else: >> # importlib._bootstrap is the built-in import, ensure we don't create >> # a second copy of the module. >> _bootstrap.__name__ = 'importlib._bootstrap' >> _bootstrap.__package__ = 'importlib' >> _bootstrap.__file__ = __file__.replace('__init__.py', '_bootstrap.py') >> sys.modules['importlib._bootstrap'] = _bootstrap > > > When attempting to use cxFreeze on a project, using Python 3.4. we ran into a problem with '__file__' identifier not being defined. > > Could this be a python bug? Why is this code expecting the module loaded from importlib/__init__.py to always have a __file__ identifier? What is supposed to happen when that code gets loaded from a ZIP archive? __file__ is expected to always be set (including when loaded from a zipfile - in that case it's the zipfile name concatenated with the path within the zip file). If it isn't set, there's a buggy loader involved somewhere that isn't setting it properly. Cheers, Nick. > > Just wanted to check here before filing an issue... but if this is an issue I hope it can be resolved before the final 3.4 release. > > Best regards, > Jurko Gospodneti? > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Mar 10 14:36:19 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 10 Mar 2014 23:36:19 +1000 Subject: [Python-Dev] [Python-checkins] cpython: whatsnew: 'U' mode deprecation (#15204). In-Reply-To: <20140310142521.69c3f957@fsol> References: <3fj2742w8Tz7Lm2@mail.python.org> <20140310142521.69c3f957@fsol> Message-ID: On 10 Mar 2014 23:26, "Antoine Pitrou" wrote: > > On Mon, 10 Mar 2014 23:09:53 +1000 > Nick Coghlan wrote: > > On 10 Mar 2014 11:36, "r.david.murray" wrote: > > > > > > +* The ``U`` mode accepted by various ``open`` functions is deprecated. > > > + In Python3 it does not do anything useful, and should be replaced by > > > + appropriate uses of :class:`io.TextIOWrapper` (if needed) and its > > *newline* > > > + argument. > > > + > > > > Huh, I missed that change. Don't we still need U when writing 2/3 > > compatible code at this point? > > Define "need" :-) I rarely if ever see "U" used in 2.x code, so I don't > think the need is very pressing in general. True, normal text mode is much more common. I'm OK with waiting to see if anyone actually complains about the deprecation :) Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Mar 10 14:55:26 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 10 Mar 2014 14:55:26 +0100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility Message-ID: Hi, Last 5 years, I spend significant time to port a lot of Python 2 code on Python 3. First, using the 2to3 tool + extra manual patches. Sorry, it was not usable in practice. The conversion was very slow, it didn't fix doctests nor all other minor "details". "Fixing Python 2 code" was no always possible. I now use the six module (or a smaller subset) to port applications to Python 3. Today, they are still many dependencies blocking applications to be ported to Python 3. So can we please try to stop scheduling another major Python version breaking almost all modules and all applications just to be pendantic? No, we should not remove any old feature in Python 4. Python 4 should be just a minor release following the previous 3.x release. I don't want another six, nine or whatever module to fill the gap between Python 3 and Python 4. For example, I propose to release the next major Python version (3.5) with the version 4.0 but without removing anything. (It's just an example, it can wait another release.) I mean that any incompatible change must following the classic transition plan: tag the feature as deprecated and wait at least one major version before dropping it, or just keep it forever. We can expect that just changing the major version (3 => 4) will already break enough applications (where "python3" or "version == 3" is hardcoded")... Instead of asking application developers to try to follow each major Python release, we should try to keep the maintaince pain in Python core. What do you think? Victor From p.f.moore at gmail.com Mon Mar 10 14:54:10 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 10 Mar 2014 13:54:10 +0000 Subject: [Python-Dev] Windows 'for current user' installation - 32/64-bit registrations overwrite each other In-Reply-To: References: Message-ID: On 10 March 2014 13:03, Jurko Gospodneti? wrote: > Is this as issue or desired behaviour? Should I open an issue for it? Sounds like a bug, but a pretty long-standing one. I can't think that the registry schema Python uses would *ever* have distinguished (the WOW64 component of the all-users entries is a system-generated thing, AIUI). So yes, raise a bug, but (a) it would be useful to test whether the same bug exists in 3.3 and 2.7, and (b) it's going to be tricky to fix without a backward-incompatible change to the registry settings (which will affect things like the launcher and virtualenv, for a start). Paul From barry at python.org Mon Mar 10 15:01:49 2014 From: barry at python.org (Barry Warsaw) Date: Mon, 10 Mar 2014 10:01:49 -0400 Subject: [Python-Dev] Python 3.4 change in importlib/__init__.py breaking cxFreeze? In-Reply-To: References: Message-ID: <20140310100149.2fae22d8@limelight.wooz.org> On Mar 10, 2014, at 11:25 PM, Nick Coghlan wrote: >__file__ is expected to always be set (including when loaded from a zipfile Actually, __file__ is an optional attribute on modules since PEP 420 and Python 3.2. It's *usually* there, but unlike in previous version of Python, if the module doesn't actually come from a "file", there's no longer a requirement to craft some mythical value for it if it doesn't make sense. -Barry From barry at python.org Mon Mar 10 15:04:51 2014 From: barry at python.org (Barry Warsaw) Date: Mon, 10 Mar 2014 10:04:51 -0400 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: Message-ID: <20140310100451.2494242d@limelight.wooz.org> On Mar 10, 2014, at 02:55 PM, Victor Stinner wrote: >So can we please try to stop scheduling another major Python version >breaking almost all modules and all applications just to be pendantic? > >What do you think? Just that it's crazy to be talking about Python 4 right now. We have at least 9 years before we have to worry about that. ((~3.10 - 3.4) * 18months)). -Barry From steve at pearwood.info Mon Mar 10 16:04:52 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 11 Mar 2014 02:04:52 +1100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: Message-ID: <20140310150452.GC12595@ando> On Mon, Mar 10, 2014 at 02:55:26PM +0100, Victor Stinner wrote: [...] > So can we please try to stop scheduling another major Python version > breaking almost all modules and all applications just to be pendantic? > > No, we should not remove any old feature in Python 4. Python 4 should > be just a minor release following the previous 3.x release. I often talk about "Python 4000" as the next possible opportunity for major backwards incompatible changes, but of course that's not decided yet, and given the long term pain of the 2->3 transition, it may be quite conservative, with no radical changes. Perhaps I ought to use Python 5000 as my target for radical language changes? > For example, I propose to release the next major Python version (3.5) > with the version 4.0 but without removing anything. (It's just an > example, it can wait another release.) If Python 4 is a conservative release, I don't see any reason to bump the major version number until after Python 3.9. So, assuming no further radical changes to the language like 2->3, we'd have five more point releases before needing to deal with 4.0. Perhaps we need a long-term schedule? 3.5: August 2015 3.6: February 2017 3.7: August 2018 3.8: February 2020 3.9: August 2021 4.0: February 2023 give or take a few months. -- Steven From rdmurray at bitdance.com Mon Mar 10 16:06:26 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 10 Mar 2014 11:06:26 -0400 Subject: [Python-Dev] Python 3.4 change in importlib/__init__.py breaking cxFreeze? In-Reply-To: References: Message-ID: <20140310150627.0CB4E250CC5@webabinitio.net> On Mon, 10 Mar 2014 23:25:17 +1000, Nick Coghlan wrote: > On 10 Mar 2014 19:15, "Jurko Gospodneti??" wrote: > > > > Hi all. > > > > Python 3.4 introduced a change to Lib/importlib/__init__.py that added > the following code to it: > > > >> else: > >> # importlib._bootstrap is the built-in import, ensure we don't create > >> # a second copy of the module. > >> _bootstrap.__name__ = 'importlib._bootstrap' > >> _bootstrap.__package__ = 'importlib' > >> _bootstrap.__file__ = __file__.replace('__init__.py', > '_bootstrap.py') > >> sys.modules['importlib._bootstrap'] = _bootstrap > > > > > > When attempting to use cxFreeze on a project, using Python 3.4. we ran > into a problem with '__file__' identifier not being defined. > > > > Could this be a python bug? Why is this code expecting the module > loaded from importlib/__init__.py to always have a __file__ identifier? > What is supposed to happen when that code gets loaded from a ZIP archive? > > __file__ is expected to always be set (including when loaded from a zipfile > - in that case it's the zipfile name concatenated with the path within the > zip file). If it isn't set, there's a buggy loader involved somewhere that > isn't setting it properly. I noticed while using cx_Freeze that __file__ is not set on the main module that cx_Freeze calls (and I found that very annoying :). I'm guessing this is a cx_Freeze artifact and will need to be fixed in cx_Freeze. --David From brian at python.org Mon Mar 10 16:31:42 2014 From: brian at python.org (Brian Curtin) Date: Mon, 10 Mar 2014 10:31:42 -0500 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: Message-ID: On Mon, Mar 10, 2014 at 8:55 AM, Victor Stinner wrote: > For example, I propose to release the next major Python version (3.5) > with the version 4.0 but without removing anything. People put a lot of weight behind version numbers, often much more than they should. Jumping to 4.0 would be a PR nightmare and would ultimately hurt this project as more people decide that switching to another language will solve their problem better than jumping from 2.x to 4.0. People already think 2.7 to 3.4 is enough of a burden. Please do not do this. From Stefan.Richthofer at gmx.de Mon Mar 10 16:25:08 2014 From: Stefan.Richthofer at gmx.de (Stefan Richthofer) Date: Mon, 10 Mar 2014 16:25:08 +0100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <20140310150452.GC12595@ando> References: , <20140310150452.GC12595@ando> Message-ID: I don't see the point in this discussion. As far as I know, the major version is INTENDED to indicate backward-incompatible changes. The meaning of the versioning scheme is literally [API compatibility].[new features].[bug fixes], isn't it? So all you are asking for is never do produce a Python 4.x This is not necessarily bad, for instance Java has never increased its major version ever (it is still 1.7.x). An indeed, it never dropped features, just "deprecated". > I don't see any reason to bump > the major version number until after Python 3.9. Even then, there is no need for 4.0; you can just have 3.10, 3.11 etc. Cheers Stefan > Gesendet: Montag, 10. M?rz 2014 um 16:04 Uhr > Von: "Steven D'Aprano" > An: python-dev at python.org > Betreff: Re: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility > > On Mon, Mar 10, 2014 at 02:55:26PM +0100, Victor Stinner wrote: > [...] > > So can we please try to stop scheduling another major Python version > > breaking almost all modules and all applications just to be pendantic? > > > > No, we should not remove any old feature in Python 4. Python 4 should > > be just a minor release following the previous 3.x release. > > I often talk about "Python 4000" as the next possible opportunity for > major backwards incompatible changes, but of course that's not decided > yet, and given the long term pain of the 2->3 transition, it may be > quite conservative, with no radical changes. Perhaps I ought to use > Python 5000 as my target for radical language changes? > > > > For example, I propose to release the next major Python version (3.5) > > with the version 4.0 but without removing anything. (It's just an > > example, it can wait another release.) > > If Python 4 is a conservative release, I don't see any reason to bump > the major version number until after Python 3.9. So, assuming no further > radical changes to the language like 2->3, we'd have five more point > releases before needing to deal with 4.0. > > Perhaps we need a long-term schedule? > > 3.5: August 2015 > 3.6: February 2017 > 3.7: August 2018 > 3.8: February 2020 > 3.9: August 2021 > 4.0: February 2023 > > give or take a few months. > > > -- > Steven > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/stefan.richthofer%40gmx.de > From jurko.gospodnetic at pke.hr Mon Mar 10 16:18:58 2014 From: jurko.gospodnetic at pke.hr (=?UTF-8?B?SnVya28gR29zcG9kbmV0acSH?=) Date: Mon, 10 Mar 2014 16:18:58 +0100 Subject: [Python-Dev] Python 3.4 change in importlib/__init__.py breaking cxFreeze? In-Reply-To: References: Message-ID: <531DD7E2.5080005@pke.hr> Hi Nick. On 10.3.2014. 14:25, Nick Coghlan wrote:> What is supposed to happen when that code gets loaded from a ZIP archive? > > __file__ is expected to always be set (including when loaded from a > zipfile - in that case it's the zipfile name concatenated with the > path within the zip file). If it isn't set, there's a buggy loader > involved somewhere that isn't setting it properly. I don't recall seeing that ever explicitly stated. For that matter, Python 3.4.0rc3 documentation explicitly states: > __file__ is optional. If set, this attribute?s value must be a string. > The import system may opt to leave __file__ unset if it has no > semantic meaning (e.g. a module loaded from a database). and: > Ultimately, the loader is what makes use of __file__ and/or __cached__. Or is this some rule specific to the importlib/__init__.py stdlib module? As I recall, I first learned that not all loaded modules need to have their __file__ attribute set by researching a failure in some package when installed as a zipped-egg using setuptools. Admittedly though, that was some old setuptools version. Best regards, Jurko Gospodneti? From jurko.gospodnetic at pke.hr Mon Mar 10 16:35:46 2014 From: jurko.gospodnetic at pke.hr (=?UTF-8?B?SnVya28gR29zcG9kbmV0acSH?=) Date: Mon, 10 Mar 2014 16:35:46 +0100 Subject: [Python-Dev] Windows 'for current user' installation - 32/64-bit registrations overwrite each other In-Reply-To: References: Message-ID: <531DDBD2.7080405@pke.hr> Hi Paul. On 10.3.2014. 14:54, Paul Moore wrote: > On 10 March 2014 13:03, Jurko Gospodneti? wrote: >> Is this as issue or desired behaviour? Should I open an issue for it? > > Sounds like a bug, but a pretty long-standing one. I can't think that > the registry schema Python uses would *ever* have distinguished (the > WOW64 component of the all-users entries is a system-generated thing, > AIUI). > > So yes, raise a bug, but (a) it would be useful to test whether the > same bug exists in 3.3 and 2.7, and (b) it's going to be tricky to fix > without a backward-incompatible change to the registry settings (which > will affect things like the launcher and virtualenv, for a start). Reported as issue #20883 (http://bugs.python.org/issue20883). I'll report the extra details there then. Best regards, Jurko Gospodneti? From python at mrabarnett.plus.com Mon Mar 10 16:42:45 2014 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 10 Mar 2014 15:42:45 +0000 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <20140310150452.GC12595@ando> References: <20140310150452.GC12595@ando> Message-ID: <531DDD75.2030200@mrabarnett.plus.com> On 2014-03-10 15:04, Steven D'Aprano wrote: > On Mon, Mar 10, 2014 at 02:55:26PM +0100, Victor Stinner wrote: > [...] >> So can we please try to stop scheduling another major Python version >> breaking almost all modules and all applications just to be pendantic? >> >> No, we should not remove any old feature in Python 4. Python 4 should >> be just a minor release following the previous 3.x release. > > I often talk about "Python 4000" as the next possible opportunity for > major backwards incompatible changes, but of course that's not decided > yet, and given the long term pain of the 2->3 transition, it may be > quite conservative, with no radical changes. Perhaps I ought to use > Python 5000 as my target for radical language changes? > You mean that backwards-incompatible changes should be limited to prime- number major versions only? :-) [snip] From bcannon at gmail.com Mon Mar 10 16:44:22 2014 From: bcannon at gmail.com (Brett Cannon) Date: Mon, 10 Mar 2014 15:44:22 +0000 Subject: [Python-Dev] Python 3.4 change in importlib/__init__.py breaking cxFreeze? References: <531DD7E2.5080005@pke.hr> Message-ID: On Mon Mar 10 2014 at 11:41:27 AM, Jurko Gospodneti? < jurko.gospodnetic at pke.hr> wrote: > Hi Nick. > > On 10.3.2014. 14:25, Nick Coghlan wrote:> What is supposed to happen > when that code gets loaded from a ZIP archive? > > > > __file__ is expected to always be set (including when loaded from a > > zipfile - in that case it's the zipfile name concatenated with the > > path within the zip file). If it isn't set, there's a buggy loader > > involved somewhere that isn't setting it properly. > > I don't recall seeing that ever explicitly stated. For that matter, > Python 3.4.0rc3 documentation explicitly states: > > > __file__ is optional. If set, this attribute?s value must be a string. > > The import system may opt to leave __file__ unset if it has no > > semantic meaning (e.g. a module loaded from a database). > > and: > > > Ultimately, the loader is what makes use of __file__ and/or __cached__. > > Or is this some rule specific to the importlib/__init__.py stdlib > module? > No, Nick was mistaken and Barry's response is accurate: __file__ is optional and left off when it doesn't make any sense. Since importlib._bootstrap is a frozen module by default it doesn't have __file__ set. -Brett > > As I recall, I first learned that not all loaded modules need to have > their __file__ attribute set by researching a failure in some package > when installed as a zipped-egg using setuptools. Admittedly though, that > was some old setuptools version. > > Best regards, > Jurko Gospodneti? > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Mar 10 16:50:12 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 10 Mar 2014 16:50:12 +0100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> Message-ID: 2014-03-10 16:25 GMT+01:00 Stefan Richthofer : > I don't see the point in this discussion. > As far as I know, the major version is INTENDED to > indicate backward-incompatible changes. This is not a strict rule. I would like to follow Linux 3 which didn't break the API between Linux 2 and Linux 3. > Even then, there is no need for 4.0; you can just have 3.10, 3.11 etc. The major version is sometimes seen as the age of a project. I propose to bump Python to version 4 because people understand that Python 4 is much better than Python 3 :-) Firefox changes its major version every 4 months or something like that. I suggest to wait less than 8 years for Python 4. http://en.wikipedia.org/wiki/Software_versioning#Political_and_cultural_significance_of_version_numbers Or maybe we should jump directly to Python 5 for asian users and to follow PHP version! http://en.wikipedia.org/wiki/Tetraphobia Victor From breamoreboy at yahoo.co.uk Mon Mar 10 16:57:16 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 10 Mar 2014 15:57:16 +0000 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: Message-ID: On 10/03/2014 13:55, Victor Stinner wrote: > > So can we please try to stop scheduling another major Python version > breaking almost all modules and all applications just to be pendantic? > I've missed the announcement about this, can we have a link please? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com From barry at python.org Mon Mar 10 16:59:18 2014 From: barry at python.org (Barry Warsaw) Date: Mon, 10 Mar 2014 11:59:18 -0400 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> Message-ID: <20140310115918.7b848934@anarchist.wooz.org> On Mar 10, 2014, at 04:25 PM, Stefan Richthofer wrote: >> I don't see any reason to bump >> the major version number until after Python 3.9. > >Even then, there is no need for 4.0; you can just have 3.10, 3.11 etc. Guido famously hates two digit minor version numbers. :) -Barry From bcannon at gmail.com Mon Mar 10 17:06:22 2014 From: bcannon at gmail.com (Brett Cannon) Date: Mon, 10 Mar 2014 16:06:22 +0000 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility References: <20140310150452.GC12595@ando> Message-ID: On Mon Mar 10 2014 at 11:50:54 AM, Victor Stinner wrote: > 2014-03-10 16:25 GMT+01:00 Stefan Richthofer : > > I don't see the point in this discussion. > > As far as I know, the major version is INTENDED to > > indicate backward-incompatible changes. > > This is not a strict rule. I would like to follow Linux 3 which didn't > break the API between Linux 2 and Linux 3. > I disagree. I don't think 3->4 will be as drastic as it was for 2->3, but I view Python 4 as a chance to drop all deprecated APIs that we left in for convenience in porting from Python 2 (e.g. the imp module). We can't put a removal date as we can't really declare Python 2 dead for the whole community. But when Python 4 does come out next decade I would like to say that we have moved entirely beyond Python 2 as a team and thus don't turn into Java and support deprecated code forever. > > > Even then, there is no need for 4.0; you can just have 3.10, 3.11 etc. > > The major version is sometimes seen as the age of a project. I propose > to bump Python to version 4 because people understand that Python 4 is > much better than Python 3 :-) Sure, but you also understand 3.5 is better than 3.4. You're also making the assumption that people are going to pick up on the fact that major version shifts from Python are suddenly not a big deal, even though it is still well known the shift from 2 to 3 was a big deal. > Firefox changes its major version every > 4 months or something like that. Sure, but we are not about to do 4 month release cycles. Releasing that often does make minor version numbers silly since they become double digits so quickly. But since we are not shifting from our 18 month cycle we don't have that issue; 15 years (typically) between a .0 and .9 release doesn't really lead to a worry of exhausting reasonable minor version numbers. > I suggest to wait less than 8 years > for Python 4. > Why? What's special about 8 years? -------------- next part -------------- An HTML attachment was scrubbed... URL: From Stefan.Richthofer at gmx.de Mon Mar 10 17:04:08 2014 From: Stefan.Richthofer at gmx.de (Stefan Richthofer) Date: Mon, 10 Mar 2014 17:04:08 +0100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <20140310115918.7b848934@anarchist.wooz.org> References: <20140310150452.GC12595@ando> , <20140310115918.7b848934@anarchist.wooz.org> Message-ID: > Guido famously hates two digit minor version numbers. :) This is no problem either. Simply switch to hexadecimal numbering ;) > Gesendet: Montag, 10. M?rz 2014 um 16:59 Uhr > Von: "Barry Warsaw" > An: python-dev at python.org > Betreff: Re: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility > > On Mar 10, 2014, at 04:25 PM, Stefan Richthofer wrote: > > >> I don't see any reason to bump > >> the major version number until after Python 3.9. > > > >Even then, there is no need for 4.0; you can just have 3.10, 3.11 etc. > > Guido famously hates two digit minor version numbers. :) > > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/stefan.richthofer%40gmx.de > From victor.stinner at gmail.com Mon Mar 10 17:08:35 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 10 Mar 2014 17:08:35 +0100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> Message-ID: >> I suggest to wait less than 8 years >> for Python 4. > > Why? What's special about 8 years? It's the time between Python 2.0 and 3.0. Victor From jurko.gospodnetic at pke.hr Mon Mar 10 17:18:37 2014 From: jurko.gospodnetic at pke.hr (=?UTF-8?B?SnVya28gR29zcG9kbmV0acSH?=) Date: Mon, 10 Mar 2014 17:18:37 +0100 Subject: [Python-Dev] Python 3.4 change in importlib/__init__.py breaking cxFreeze? In-Reply-To: References: <531DD7E2.5080005@pke.hr> Message-ID: <531DE5DD.8010205@pke.hr> Hi Brett. On 10.3.2014. 16:44, Brett Cannon wrote: > __file__ is optional and left off when it doesn't make any sense. > Since importlib._bootstrap is a frozen module by default it doesn't > have __file__ set. Issue #20884 opened for this (http://bugs.python.org/issue20884). Best regards, Jurko Gospodneti? From tseaver at palladion.com Mon Mar 10 17:26:24 2014 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 10 Mar 2014 12:26:24 -0400 Subject: [Python-Dev] cpython: whatsnew: 'U' mode deprecation (#15204). In-Reply-To: References: <3fj2742w8Tz7Lm2@mail.python.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/10/2014 09:09 AM, Nick Coghlan wrote: > Don't we still need U when writing 2/3 compatible code at this point? io.TextIOWrapper was already the superior strategy for anyone straddling Python 2.6+ - Py3k. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlMd56sACgkQ+gerLs4ltQ4+kQCgm2a0KDP/aGTCZD9eUdImjBC2 ny4AoLmyLOlBZOksvu3nwKawo6BTG0Ab =H7l+ -----END PGP SIGNATURE----- From storchaka at gmail.com Mon Mar 10 17:32:13 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 10 Mar 2014 18:32:13 +0200 Subject: [Python-Dev] cpython: whatsnew: 'U' mode deprecation (#15204). In-Reply-To: References: <3fj2742w8Tz7Lm2@mail.python.org> Message-ID: 10.03.14 15:09, Nick Coghlan ???????(??): > Huh, I missed that change. Don't we still need U when writing 2/3 > compatible code at this point? If so, perhaps we may want to reconsider > this particular deprecation in 3.4.1. In most cases the 'U' mode either no-op or forbidden in Python 3. It may be needed only when you use built-in open() in Python 2 and open without the encoding and newlines arguments in Python 3 (but it will be better to use io.open()). http://bugs.python.org/issue15204 From bcannon at gmail.com Mon Mar 10 17:37:45 2014 From: bcannon at gmail.com (Brett Cannon) Date: Mon, 10 Mar 2014 16:37:45 +0000 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility References: <20140310150452.GC12595@ando> Message-ID: On Mon Mar 10 2014 at 12:08:55 PM, Victor Stinner wrote: > >> I suggest to wait less than 8 years > >> for Python 4. > > > > Why? What's special about 8 years? > > It's the time between Python 2.0 and 3.0. > But I'm willing to bet that's going to be an anomaly. Python 3 came into existence when it did so we didn't wait too long for Python 2 to get even more of a foothold; the expectation is that long-term more non-Python 2 code will be written than not. There's no rush on Python 4 since there are no plans to try and tweak something as drastic as str/bytes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Mar 10 16:38:41 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 10 Mar 2014 08:38:41 -0700 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <20140310150452.GC12595@ando> References: <20140310150452.GC12595@ando> Message-ID: On Mon, Mar 10, 2014 at 8:04 AM, Steven D'Aprano wrote: > If Python 4 is a conservative release, I don't see any reason to bump > the major version number until after Python 3.9. and why even then? > Perhaps we need a long-term schedule? > why not: 3.5: August 2015 > 3.6: February 2017 > 3.7: August 2018 > 3.8: February 2020 > 3.9: August 2021 3.10: February 2023 > 3.11 August 2023 3.12 February 2024 .... version numbering is not decimal -- a bump to 4.0 should mean _something_ (Though from the above, we've got a few years before we need to worry about that!) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Mon Mar 10 17:49:44 2014 From: bcannon at gmail.com (Brett Cannon) Date: Mon, 10 Mar 2014 16:49:44 +0000 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility References: <20140310150452.GC12595@ando> Message-ID: On Mon Mar 10 2014 at 12:47:21 PM, Chris Barker wrote: > On Mon, Mar 10, 2014 at 8:04 AM, Steven D'Aprano wrote: > > >> If Python 4 is a conservative release, I don't see any reason to bump >> the major version number until after Python 3.9. > > > and why even then? > > >> Perhaps we need a long-term schedule? >> > > why not: > > 3.5: August 2015 >> 3.6: February 2017 >> 3.7: August 2018 >> 3.8: February 2020 >> 3.9: August 2021 > > 3.10: February 2023 >> > 3.11 August 2023 > 3.12 February 2024 > .... > > version numbering is not decimal -- a bump to 4.0 should mean _something_ > > (Though from the above, we've got a few years before we need to worry > about that!) > I think it got lost in email threading, but Barry pointed out that Guido famously hates double digit version numbers (as do I, probably partially because he does after all these years =). -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Mar 10 17:20:24 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 10 Mar 2014 09:20:24 -0700 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: Message-ID: <531DE648.4070503@stoneleaf.us> On 03/10/2014 06:55 AM, Victor Stinner wrote: > > What do you think? I think Python 4.0 will follow Python 3.9. No need to rush things [1]. -- ~Ethan~ [1] The Python 2 line ended early because we had a major paradigm shift with moving to Unicode by default. Unless we experience another major paradigm shift (which would absolutely require backwards incompatible changes), there is no need to end the 3.x line early. From solipsis at pitrou.net Mon Mar 10 17:54:37 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 10 Mar 2014 17:54:37 +0100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility References: <20140310150452.GC12595@ando> Message-ID: <20140310175437.5644dd45@fsol> On Mon, 10 Mar 2014 16:50:12 +0100 Victor Stinner wrote: > 2014-03-10 16:25 GMT+01:00 Stefan Richthofer : > > I don't see the point in this discussion. > > As far as I know, the major version is INTENDED to > > indicate backward-incompatible changes. > > This is not a strict rule. I would like to follow Linux 3 which didn't > break the API between Linux 2 and Linux 3. > > > Even then, there is no need for 4.0; you can just have 3.10, 3.11 etc. > > The major version is sometimes seen as the age of a project. I propose > to bump Python to version 4 because people understand that Python 4 is > much better than Python 3 :-) Firefox changes its major version every > 4 months or something like that. I suggest to wait less than 8 years > for Python 4. But then, three is the number, not four, and five is right out: "First shalt thou take out the Holy Pin, then shalt thou count to three, no more, no less. Three shall be the number thou shalt count, and the number of the counting shall be three. Four shalt thou not count, neither count thou two, excepting that thou then proceed to three. Five is right out. Once the number three, being the third number, be reached, then lobbest thou thy Holy Hand Grenade of Antioch towards thy foe, who being naughty in My sight, shall snuff it." Regards Antoine. From solipsis at pitrou.net Mon Mar 10 17:56:28 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 10 Mar 2014 17:56:28 +0100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility References: <20140310150452.GC12595@ando> <20140310115918.7b848934@anarchist.wooz.org> Message-ID: <20140310175628.71f80415@fsol> On Mon, 10 Mar 2014 17:04:08 +0100 "Stefan Richthofer" wrote: > > Guido famously hates two digit minor version numbers. :) > > This is no problem either. Simply switch to hexadecimal numbering ;) Or wrap around to negative numbers (a minus sign isn't technically a digit, is it?). Regards Antoine. From tseaver at palladion.com Mon Mar 10 18:01:05 2014 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 10 Mar 2014 13:01:05 -0400 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/10/2014 12:49 PM, Brett Cannon wrote: > I think it got lost in email threading, but Barry pointed out that > Guido famously hates double digit version numbers (as do I, probably > partially because he does after all these years =). "Guido hates them" isn't an argument: its a ukase. Version numbers are tuples, not decimal fractions. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlMd78wACgkQ+gerLs4ltQ4s/ACdEkOvaYjP2d9IZ4g8bVJC/OZl h8gAoIu0IY1qYAhvZQzEo9Up9swJnZ60 =7tsf -----END PGP SIGNATURE----- From rosuav at gmail.com Mon Mar 10 18:06:24 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 11 Mar 2014 04:06:24 +1100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <20140310175628.71f80415@fsol> References: <20140310150452.GC12595@ando> <20140310115918.7b848934@anarchist.wooz.org> <20140310175628.71f80415@fsol> Message-ID: On Tue, Mar 11, 2014 at 3:56 AM, Antoine Pitrou wrote: > On Mon, 10 Mar 2014 17:04:08 +0100 > "Stefan Richthofer" wrote: > >> > Guido famously hates two digit minor version numbers. :) >> >> This is no problem either. Simply switch to hexadecimal numbering ;) > > Or wrap around to negative numbers (a minus sign isn't technically a > digit, is it?). Terrible idea. Would wreak havoc with comparisons. No. Python 3 is all about Unicode, so the right way to proceed is 3.8, 3.9, 3.:, 3.;, 3.<, 3.=, 3.>, 3.?, 3.@, 3.A. ChrisA From rdmurray at bitdance.com Mon Mar 10 18:08:56 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 10 Mar 2014 13:08:56 -0400 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> Message-ID: <20140310170857.31EBF250C36@webabinitio.net> On Mon, 10 Mar 2014 16:06:22 -0000, Brett Cannon wrote: > On Mon Mar 10 2014 at 11:50:54 AM, Victor Stinner > wrote: > > > 2014-03-10 16:25 GMT+01:00 Stefan Richthofer : > > > I don't see the point in this discussion. > > > As far as I know, the major version is INTENDED to > > > indicate backward-incompatible changes. > > > > This is not a strict rule. I would like to follow Linux 3 which didn't > > break the API between Linux 2 and Linux 3. > > > > I disagree. I don't think 3->4 will be as drastic as it was for 2->3, but I > view Python 4 as a chance to drop all deprecated APIs that we left in for > convenience in porting from Python 2 (e.g. the imp module). We can't put a > removal date as we can't really declare Python 2 dead for the whole > community. But when Python 4 does come out next decade I would like to say > that we have moved entirely beyond Python 2 as a team and thus don't turn > into Java and support deprecated code forever. We had this discussion a bit ago, and my sense was that we tentatively decided that we were just going to deprecate and remove things as appropriate, irregardless of version number. I used "4.0" in my message about 'U' as a shorthand for "some time after python2 is no longer an issue". Sorry for the confusion. (That said, I do see some merit to doing some extra cleaning at the 4.0 boundary, just for mental convenience.) --David From rosuav at gmail.com Mon Mar 10 18:13:37 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 11 Mar 2014 04:13:37 +1100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <20140310170857.31EBF250C36@webabinitio.net> References: <20140310150452.GC12595@ando> <20140310170857.31EBF250C36@webabinitio.net> Message-ID: On Tue, Mar 11, 2014 at 4:08 AM, R. David Murray wrote: > (That said, I > do see some merit to doing some extra cleaning at the 4.0 > boundary, just for mental convenience.) A transition from 3.9 to 4.0 that removes a whole lot of deprecated aliases and such wouldn't be a bad thing. It's technically breaking backward compat (and thus justifying the major version bump), but any code broken by it would have been unidiomatic for the past X versions anyway. ChrisA From fijall at gmail.com Mon Mar 10 18:30:20 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 10 Mar 2014 19:30:20 +0200 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: References: <531a3831.8589e00a.7525.01d4@mx.google.com> <20140308124501.6d809148@fsol> <20140308143302.39cb9435@fsol> Message-ID: On Mon, Mar 10, 2014 at 3:23 PM, Victor Stinner wrote: > 2014-03-10 13:11 GMT+01:00 Maciej Fijalkowski : >> It was agreed long time ago that the immediate finalization is an >> implementation specific detail and it's not guaranteed. You should not >> rely on __del__s being called timely one way or another. Why would you >> require this for the program to work correctly in the particular >> example of __traceback__? > > For asyncio, it's very useful to see unhandled exceptions as early as > possible. Otherwise, your program is blocked and you don't know why. > > Guido van Rossum suggests to run gc.collect() regulary: > http://code.google.com/p/tulip/issues/detail?id=42 > > Victor twisted goes around it by attaching errback by hand. Would that work for tulip? From guido at python.org Mon Mar 10 18:35:36 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 10 Mar 2014 10:35:36 -0700 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: References: <531a3831.8589e00a.7525.01d4@mx.google.com> <20140308124501.6d809148@fsol> <20140308143302.39cb9435@fsol> Message-ID: On Mon, Mar 10, 2014 at 10:30 AM, Maciej Fijalkowski wrote: > On Mon, Mar 10, 2014 at 3:23 PM, Victor Stinner > wrote: > > 2014-03-10 13:11 GMT+01:00 Maciej Fijalkowski : > >> It was agreed long time ago that the immediate finalization is an > >> implementation specific detail and it's not guaranteed. You should not > >> rely on __del__s being called timely one way or another. Why would you > >> require this for the program to work correctly in the particular > >> example of __traceback__? > > > > For asyncio, it's very useful to see unhandled exceptions as early as > > possible. Otherwise, your program is blocked and you don't know why. > > > > Guido van Rossum suggests to run gc.collect() regulary: > > http://code.google.com/p/tulip/issues/detail?id=42 > > > > Victor > > twisted goes around it by attaching errback by hand. Would that work for > tulip? > Can you describe that idea in more detail? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Mon Mar 10 18:39:42 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 10 Mar 2014 19:39:42 +0200 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: References: <531a3831.8589e00a.7525.01d4@mx.google.com> <20140308124501.6d809148@fsol> <20140308143302.39cb9435@fsol> Message-ID: On Mon, Mar 10, 2014 at 7:35 PM, Guido van Rossum wrote: > On Mon, Mar 10, 2014 at 10:30 AM, Maciej Fijalkowski > wrote: >> >> On Mon, Mar 10, 2014 at 3:23 PM, Victor Stinner >> wrote: >> > 2014-03-10 13:11 GMT+01:00 Maciej Fijalkowski : >> >> It was agreed long time ago that the immediate finalization is an >> >> implementation specific detail and it's not guaranteed. You should not >> >> rely on __del__s being called timely one way or another. Why would you >> >> require this for the program to work correctly in the particular >> >> example of __traceback__? >> > >> > For asyncio, it's very useful to see unhandled exceptions as early as >> > possible. Otherwise, your program is blocked and you don't know why. >> > >> > Guido van Rossum suggests to run gc.collect() regulary: >> > http://code.google.com/p/tulip/issues/detail?id=42 >> > >> > Victor >> >> twisted goes around it by attaching errback by hand. Would that work for >> tulip? > > > Can you describe that idea in more detail? Essentially, instead of relying on deferred to be garbage collected, you attach an errback like this: deferred.addErrback(callback_that_writes_to_log) so in case of a failure, you get a traceback directly in the callback immediately, without relying on garbage collection. I'm sorry if I'm using twisted nomenclature here (it's also awfully off-topic for python-dev), but making programs rely on refcounting sounds like a bad idea for us (PyPy). Cheers, fijal From phd at phdru.name Mon Mar 10 18:44:56 2014 From: phd at phdru.name (Oleg Broytman) Date: Mon, 10 Mar 2014 18:44:56 +0100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> Message-ID: <20140310174456.GA1360@phdru.name> Hi! On Mon, Mar 10, 2014 at 04:49:44PM +0000, Brett Cannon wrote: > I think it got lost in email threading, but Barry pointed out that Guido > famously hates double digit version numbers (as do I, probably partially > because he does after all these years =). There is one minor annoyance with double digits: $ ls -l total 16 drwx------ 2 phd phd 4096 Mar 10 21:42 3.1 drwx------ 2 phd phd 4096 Mar 10 21:42 3.10 drwx------ 2 phd phd 4096 Mar 10 21:42 3.2 ... ... drwx------ 2 phd phd 4096 Mar 10 21:42 4.0 Other than that I don't see any problem, and don't see any need to jump from 3.9 to 4.0. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From guido at python.org Mon Mar 10 18:49:40 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 10 Mar 2014 10:49:40 -0700 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: References: <531a3831.8589e00a.7525.01d4@mx.google.com> <20140308124501.6d809148@fsol> <20140308143302.39cb9435@fsol> Message-ID: On Mon, Mar 10, 2014 at 10:39 AM, Maciej Fijalkowski wrote: > On Mon, Mar 10, 2014 at 7:35 PM, Guido van Rossum > wrote: > > On Mon, Mar 10, 2014 at 10:30 AM, Maciej Fijalkowski > > wrote: > >> > >> On Mon, Mar 10, 2014 at 3:23 PM, Victor Stinner > >> wrote: > >> > 2014-03-10 13:11 GMT+01:00 Maciej Fijalkowski : > >> >> It was agreed long time ago that the immediate finalization is an > >> >> implementation specific detail and it's not guaranteed. You should > not > >> >> rely on __del__s being called timely one way or another. Why would > you > >> >> require this for the program to work correctly in the particular > >> >> example of __traceback__? > >> > > >> > For asyncio, it's very useful to see unhandled exceptions as early as > >> > possible. Otherwise, your program is blocked and you don't know why. > >> > > >> > Guido van Rossum suggests to run gc.collect() regulary: > >> > http://code.google.com/p/tulip/issues/detail?id=42 > >> > > >> > Victor > >> > >> twisted goes around it by attaching errback by hand. Would that work for > >> tulip? > > > > > > Can you describe that idea in more detail? > > Essentially, instead of relying on deferred to be garbage collected, > you attach an errback like this: > > deferred.addErrback(callback_that_writes_to_log) > > so in case of a failure, you get a traceback directly in the callback > immediately, without relying on garbage collection. > > I'm sorry if I'm using twisted nomenclature here (it's also awfully > off-topic for python-dev), but making programs rely on refcounting > sounds like a bad idea for us (PyPy). > IIUC the problem that Victor is trying to solve is what to do if nobody thought to attach an errback. Tulip makes a best effort to still log a traceback. We've found this very helpful (just as it is helpful that Python prints a traceback when synchronous code raises an exception and no except clause caught it). The best effort relies on GC. I am guessing that refcounting makes the traceback appear sooner, but there would be other ways to force it, like occasionally calling gc.collect() during idle times (presumably during busy times it will be called soon enough. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Mon Mar 10 18:50:44 2014 From: phd at phdru.name (Oleg Broytman) Date: Mon, 10 Mar 2014 18:50:44 +0100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> Message-ID: <20140310175044.GB1360@phdru.name> On Mon, Mar 10, 2014 at 04:37:45PM +0000, Brett Cannon wrote: > On Mon Mar 10 2014 at 12:08:55 PM, Victor Stinner > wrote: > > > >> I suggest to wait less than 8 years > > >> for Python 4. > > > > > > Why? What's special about 8 years? > > > > It's the time between Python 2.0 and 3.0. > > But I'm willing to bet that's going to be an anomaly. Python 3 came into > existence when it did so we didn't wait too long for Python 2 to get even > more of a foothold; the expectation is that long-term more non-Python 2 > code will be written than not. There's no rush on Python 4 since there are > no plans to try and tweak something as drastic as str/bytes. Well, the entire discussion is about "please don't do major changes -- we've got enough already". Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From martin at v.loewis.de Mon Mar 10 18:52:39 2014 From: martin at v.loewis.de (=?windows-1250?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 10 Mar 2014 18:52:39 +0100 Subject: [Python-Dev] Windows 'for current user' installation - 32/64-bit registrations overwrite each other In-Reply-To: References: Message-ID: <531DFBE7.3070809@v.loewis.de> Am 10.03.14 14:03, schrieb Jurko Gospodneti?: > Is this as issue or desired behaviour? See my response in the tracker. It's desired by Microsoft. Regards, Martin From fijall at gmail.com Mon Mar 10 18:56:17 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 10 Mar 2014 19:56:17 +0200 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: References: <531a3831.8589e00a.7525.01d4@mx.google.com> <20140308124501.6d809148@fsol> <20140308143302.39cb9435@fsol> Message-ID: On Mon, Mar 10, 2014 at 7:49 PM, Guido van Rossum wrote: > On Mon, Mar 10, 2014 at 10:39 AM, Maciej Fijalkowski > wrote: >> >> On Mon, Mar 10, 2014 at 7:35 PM, Guido van Rossum >> wrote: >> > On Mon, Mar 10, 2014 at 10:30 AM, Maciej Fijalkowski >> > wrote: >> >> >> >> On Mon, Mar 10, 2014 at 3:23 PM, Victor Stinner >> >> wrote: >> >> > 2014-03-10 13:11 GMT+01:00 Maciej Fijalkowski : >> >> >> It was agreed long time ago that the immediate finalization is an >> >> >> implementation specific detail and it's not guaranteed. You should >> >> >> not >> >> >> rely on __del__s being called timely one way or another. Why would >> >> >> you >> >> >> require this for the program to work correctly in the particular >> >> >> example of __traceback__? >> >> > >> >> > For asyncio, it's very useful to see unhandled exceptions as early as >> >> > possible. Otherwise, your program is blocked and you don't know why. >> >> > >> >> > Guido van Rossum suggests to run gc.collect() regulary: >> >> > http://code.google.com/p/tulip/issues/detail?id=42 >> >> > >> >> > Victor >> >> >> >> twisted goes around it by attaching errback by hand. Would that work >> >> for >> >> tulip? >> > >> > >> > Can you describe that idea in more detail? >> >> Essentially, instead of relying on deferred to be garbage collected, >> you attach an errback like this: >> >> deferred.addErrback(callback_that_writes_to_log) >> >> so in case of a failure, you get a traceback directly in the callback >> immediately, without relying on garbage collection. >> >> I'm sorry if I'm using twisted nomenclature here (it's also awfully >> off-topic for python-dev), but making programs rely on refcounting >> sounds like a bad idea for us (PyPy). > > > IIUC the problem that Victor is trying to solve is what to do if nobody > thought to attach an errback. Tulip makes a best effort to still log a > traceback. We've found this very helpful (just as it is helpful that Python > prints a traceback when synchronous code raises an exception and no except > clause caught it). > > The best effort relies on GC. I am guessing that refcounting makes the > traceback appear sooner, but there would be other ways to force it, like > occasionally calling gc.collect() during idle times (presumably during busy > times it will be called soon enough. :-) > > -- > --Guido van Rossum (python.org/~guido) I agree this sounds like a solution. However I'm very skeptical about changing details of __traceback__ and frames, just in order to make refcounting work (since it would create something that would not work on pypy for example). Cheers, fijal From larry at hastings.org Mon Mar 10 18:58:30 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 10 Mar 2014 10:58:30 -0700 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <20140310115918.7b848934@anarchist.wooz.org> References: <20140310150452.GC12595@ando> <20140310115918.7b848934@anarchist.wooz.org> Message-ID: <531DFD46.8040706@hastings.org> On 03/10/2014 08:59 AM, Barry Warsaw wrote: > On Mar 10, 2014, at 04:25 PM, Stefan Richthofer wrote: > >>> I don't see any reason to bump >>> the major version number until after Python 3.9. >> Even then, there is no need for 4.0; you can just have 3.10, 3.11 etc. > Guido famously hates two digit minor version numbers. :) Yeah, but he'll be retired by then. So shh! No wonder he doesn't use Ubuntu, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Mar 10 18:58:34 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 10 Mar 2014 10:58:34 -0700 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: References: <531a3831.8589e00a.7525.01d4@mx.google.com> <20140308124501.6d809148@fsol> <20140308143302.39cb9435@fsol> Message-ID: On Mon, Mar 10, 2014 at 10:56 AM, Maciej Fijalkowski wrote: > On Mon, Mar 10, 2014 at 7:49 PM, Guido van Rossum > wrote: > > On Mon, Mar 10, 2014 at 10:39 AM, Maciej Fijalkowski > > wrote: > >> > >> On Mon, Mar 10, 2014 at 7:35 PM, Guido van Rossum > >> wrote: > >> > On Mon, Mar 10, 2014 at 10:30 AM, Maciej Fijalkowski < > fijall at gmail.com> > >> > wrote: > >> >> > >> >> On Mon, Mar 10, 2014 at 3:23 PM, Victor Stinner > >> >> wrote: > >> >> > 2014-03-10 13:11 GMT+01:00 Maciej Fijalkowski : > >> >> >> It was agreed long time ago that the immediate finalization is an > >> >> >> implementation specific detail and it's not guaranteed. You should > >> >> >> not > >> >> >> rely on __del__s being called timely one way or another. Why would > >> >> >> you > >> >> >> require this for the program to work correctly in the particular > >> >> >> example of __traceback__? > >> >> > > >> >> > For asyncio, it's very useful to see unhandled exceptions as early > as > >> >> > possible. Otherwise, your program is blocked and you don't know > why. > >> >> > > >> >> > Guido van Rossum suggests to run gc.collect() regulary: > >> >> > http://code.google.com/p/tulip/issues/detail?id=42 > >> >> > > >> >> > Victor > >> >> > >> >> twisted goes around it by attaching errback by hand. Would that work > >> >> for > >> >> tulip? > >> > > >> > > >> > Can you describe that idea in more detail? > >> > >> Essentially, instead of relying on deferred to be garbage collected, > >> you attach an errback like this: > >> > >> deferred.addErrback(callback_that_writes_to_log) > >> > >> so in case of a failure, you get a traceback directly in the callback > >> immediately, without relying on garbage collection. > >> > >> I'm sorry if I'm using twisted nomenclature here (it's also awfully > >> off-topic for python-dev), but making programs rely on refcounting > >> sounds like a bad idea for us (PyPy). > > > > > > IIUC the problem that Victor is trying to solve is what to do if nobody > > thought to attach an errback. Tulip makes a best effort to still log a > > traceback. We've found this very helpful (just as it is helpful that > Python > > prints a traceback when synchronous code raises an exception and no > except > > clause caught it). > > > > The best effort relies on GC. I am guessing that refcounting makes the > > traceback appear sooner, but there would be other ways to force it, like > > occasionally calling gc.collect() during idle times (presumably during > busy > > times it will be called soon enough. :-) > > > > -- > > --Guido van Rossum (python.org/~guido) > > I agree this sounds like a solution. However I'm very skeptical about > changing details of __traceback__ and frames, just in order to make > refcounting work (since it would create something that would not work > on pypy for example). > Agreed, I think we knew what we were in for when we added __traceback__ to Python 3, and I don't think now is the time to reconsider. Now is the time to learn to live with it. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Mon Mar 10 19:05:28 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 10 Mar 2014 19:05:28 +0100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> Message-ID: <531DFEE8.70109@v.loewis.de> Am 10.03.14 18:01, schrieb Tres Seaver: > On 03/10/2014 12:49 PM, Brett Cannon wrote: > >> I think it got lost in email threading, but Barry pointed out that >> Guido famously hates double digit version numbers (as do I, probably >> partially because he does after all these years =). > > "Guido hates them" isn't an argument: its a ukase. Indeed, it is - so obey it then :-) We live in a dictatorship, after all. Regards, Martin From python at mrabarnett.plus.com Mon Mar 10 19:19:02 2014 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 10 Mar 2014 18:19:02 +0000 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> <20140310115918.7b848934@anarchist.wooz.org> <20140310175628.71f80415@fsol> Message-ID: <531E0216.105@mrabarnett.plus.com> On 2014-03-10 17:06, Chris Angelico wrote: > On Tue, Mar 11, 2014 at 3:56 AM, Antoine Pitrou wrote: >> On Mon, 10 Mar 2014 17:04:08 +0100 >> "Stefan Richthofer" wrote: >> >>> > Guido famously hates two digit minor version numbers. :) >>> >>> This is no problem either. Simply switch to hexadecimal numbering ;) >> >> Or wrap around to negative numbers (a minus sign isn't technically a >> digit, is it?). > > Terrible idea. Would wreak havoc with comparisons. No. Python 3 is all > about Unicode, so the right way to proceed is 3.8, 3.9, 3.:, 3.;, 3.<, > 3.=, 3.>, 3.?, 3.@, 3.A. > Let's stick to digits. We could move on to Devanagari or Gurmukhi digits. From python at mrabarnett.plus.com Mon Mar 10 19:21:51 2014 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 10 Mar 2014 18:21:51 +0000 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <20140310170857.31EBF250C36@webabinitio.net> References: <20140310150452.GC12595@ando> <20140310170857.31EBF250C36@webabinitio.net> Message-ID: <531E02BF.6020908@mrabarnett.plus.com> On 2014-03-10 17:08, R. David Murray wrote: > On Mon, 10 Mar 2014 16:06:22 -0000, Brett Cannon wrote: >> On Mon Mar 10 2014 at 11:50:54 AM, Victor Stinner >> wrote: >> >> > 2014-03-10 16:25 GMT+01:00 Stefan Richthofer : >> > > I don't see the point in this discussion. >> > > As far as I know, the major version is INTENDED to >> > > indicate backward-incompatible changes. >> > >> > This is not a strict rule. I would like to follow Linux 3 which didn't >> > break the API between Linux 2 and Linux 3. >> > >> >> I disagree. I don't think 3->4 will be as drastic as it was for 2->3, but I >> view Python 4 as a chance to drop all deprecated APIs that we left in for >> convenience in porting from Python 2 (e.g. the imp module). We can't put a >> removal date as we can't really declare Python 2 dead for the whole >> community. But when Python 4 does come out next decade I would like to say >> that we have moved entirely beyond Python 2 as a team and thus don't turn >> into Java and support deprecated code forever. > > We had this discussion a bit ago, and my sense was that we tentatively > decided that we were just going to deprecate and remove things as > appropriate, irregardless of version number. I used "4.0" in my > message about 'U' as a shorthand for "some time after python2 > is no longer an issue". Sorry for the confusion. (That said, I > do see some merit to doing some extra cleaning at the 4.0 > boundary, just for mental convenience.) > What does "irregardless" mean? From donald at stufft.io Mon Mar 10 19:24:16 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 10 Mar 2014 14:24:16 -0400 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <531E02BF.6020908@mrabarnett.plus.com> References: <20140310150452.GC12595@ando> <20140310170857.31EBF250C36@webabinitio.net> <531E02BF.6020908@mrabarnett.plus.com> Message-ID: <18947EDB-D3C6-46DF-872B-43ED1597274F@stufft.io> On Mar 10, 2014, at 2:21 PM, MRAB wrote: > On 2014-03-10 17:08, R. David Murray wrote: >> On Mon, 10 Mar 2014 16:06:22 -0000, Brett Cannon wrote: >>> On Mon Mar 10 2014 at 11:50:54 AM, Victor Stinner >>> wrote: >>> >>> > 2014-03-10 16:25 GMT+01:00 Stefan Richthofer : >>> > > I don't see the point in this discussion. >>> > > As far as I know, the major version is INTENDED to >>> > > indicate backward-incompatible changes. >>> > >>> > This is not a strict rule. I would like to follow Linux 3 which didn't >>> > break the API between Linux 2 and Linux 3. >>> > >>> >>> I disagree. I don't think 3->4 will be as drastic as it was for 2->3, but I >>> view Python 4 as a chance to drop all deprecated APIs that we left in for >>> convenience in porting from Python 2 (e.g. the imp module). We can't put a >>> removal date as we can't really declare Python 2 dead for the whole >>> community. But when Python 4 does come out next decade I would like to say >>> that we have moved entirely beyond Python 2 as a team and thus don't turn >>> into Java and support deprecated code forever. >> >> We had this discussion a bit ago, and my sense was that we tentatively >> decided that we were just going to deprecate and remove things as >> appropriate, irregardless of version number. I used "4.0" in my >> message about 'U' as a shorthand for "some time after python2 >> is no longer an issue". Sorry for the confusion. (That said, I >> do see some merit to doing some extra cleaning at the 4.0 >> boundary, just for mental convenience.) >> > What does "irregardless" mean? http://www.merriam-webster.com/dictionary/irregardless > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rymg19 at gmail.com Mon Mar 10 19:17:30 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 10 Mar 2014 13:17:30 -0500 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> <20140310115918.7b848934@anarchist.wooz.org> <20140310175628.71f80415@fsol> Message-ID: You forgot 3., and 3.$. On Mon, Mar 10, 2014 at 12:06 PM, Chris Angelico wrote: > On Tue, Mar 11, 2014 at 3:56 AM, Antoine Pitrou > wrote: > > On Mon, 10 Mar 2014 17:04:08 +0100 > > "Stefan Richthofer" wrote: > > > >> > Guido famously hates two digit minor version numbers. :) > >> > >> This is no problem either. Simply switch to hexadecimal numbering ;) > > > > Or wrap around to negative numbers (a minus sign isn't technically a > > digit, is it?). > > Terrible idea. Would wreak havoc with comparisons. No. Python 3 is all > about Unicode, so the right way to proceed is 3.8, 3.9, 3.:, 3.;, 3.<, > 3.=, 3.>, 3.?, 3.@, 3.A. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon Mar 10 19:18:35 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 10 Mar 2014 13:18:35 -0500 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> <20140310170857.31EBF250C36@webabinitio.net> Message-ID: This is my standpoint. The major releases would remove the code that's been marked as "deprecated". You probably would've know for the past 3 versions anyway... On Mon, Mar 10, 2014 at 12:13 PM, Chris Angelico wrote: > On Tue, Mar 11, 2014 at 4:08 AM, R. David Murray > wrote: > > (That said, I > > do see some merit to doing some extra cleaning at the 4.0 > > boundary, just for mental convenience.) > > A transition from 3.9 to 4.0 that removes a whole lot of deprecated > aliases and such wouldn't be a bad thing. It's technically breaking > backward compat (and thus justifying the major version bump), but any > code broken by it would have been unidiomatic for the past X versions > anyway. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Mon Mar 10 19:25:29 2014 From: eric at trueblade.com (Eric V. Smith) Date: Mon, 10 Mar 2014 14:25:29 -0400 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <531E02BF.6020908@mrabarnett.plus.com> References: <20140310150452.GC12595@ando> <20140310170857.31EBF250C36@webabinitio.net> <531E02BF.6020908@mrabarnett.plus.com> Message-ID: <531E0399.1020209@trueblade.com> On 03/10/2014 02:21 PM, MRAB wrote: > On 2014-03-10 17:08, R. David Murray wrote: >> On Mon, 10 Mar 2014 16:06:22 -0000, Brett Cannon >> wrote: >>> On Mon Mar 10 2014 at 11:50:54 AM, Victor Stinner >>> >>> wrote: >>> >>> > 2014-03-10 16:25 GMT+01:00 Stefan Richthofer >>> : >>> > > I don't see the point in this discussion. >>> > > As far as I know, the major version is INTENDED to >>> > > indicate backward-incompatible changes. >>> > >>> > This is not a strict rule. I would like to follow Linux 3 which didn't >>> > break the API between Linux 2 and Linux 3. >>> > >>> >>> I disagree. I don't think 3->4 will be as drastic as it was for 2->3, >>> but I >>> view Python 4 as a chance to drop all deprecated APIs that we left in >>> for >>> convenience in porting from Python 2 (e.g. the imp module). We can't >>> put a >>> removal date as we can't really declare Python 2 dead for the whole >>> community. But when Python 4 does come out next decade I would like >>> to say >>> that we have moved entirely beyond Python 2 as a team and thus don't >>> turn >>> into Java and support deprecated code forever. >> >> We had this discussion a bit ago, and my sense was that we tentatively >> decided that we were just going to deprecate and remove things as >> appropriate, irregardless of version number. I used "4.0" in my >> message about 'U' as a shorthand for "some time after python2 >> is no longer an issue". Sorry for the confusion. (That said, I >> do see some merit to doing some extra cleaning at the 4.0 >> boundary, just for mental convenience.) >> > What does "irregardless" mean? Read it as "without regard to". From p.f.moore at gmail.com Mon Mar 10 19:50:35 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 10 Mar 2014 18:50:35 +0000 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <20140310170857.31EBF250C36@webabinitio.net> References: <20140310150452.GC12595@ando> <20140310170857.31EBF250C36@webabinitio.net> Message-ID: On 10 March 2014 17:08, R. David Murray wrote: > We had this discussion a bit ago, and my sense was that we tentatively > decided that we were just going to deprecate and remove things as > appropriate, irregardless of version number. I used "4.0" in my > message about 'U' as a shorthand for "some time after python2 > is no longer an issue". Sorry for the confusion. (That said, I > do see some merit to doing some extra cleaning at the 4.0 > boundary, just for mental convenience.) I have seen a number of postings recently pointing to things as "not until Python 4000" or "not until Python 4.0" (yours was not one that I noticed, actually, this is a more general point). I do think it's a bad idea to start talking in terms of "the next big compatibility break", even if *we* know there's no immediate plan. People are very quick to pick up messages like "Now that Python 3 is out of the way, the Python devs are talking about Python 4" which is not a message we want to see getting traction. I'm neither averse to, nor in favour of, a Python 4 compatibility break in due course, but maybe we should avoid letting the idea take hold this soon. Users are still concerned about the Python 3 change, it wouldn't do any harm if they got the clear message "at least it's a one-off exercise". Paul From rosuav at gmail.com Mon Mar 10 20:29:02 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 11 Mar 2014 06:29:02 +1100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> <20140310115918.7b848934@anarchist.wooz.org> <20140310175628.71f80415@fsol> Message-ID: On Tue, Mar 11, 2014 at 5:17 AM, Ryan Gonzalez wrote: > You forgot 3., and 3.$. > They're both earlier than digits. Comma is 2C and dollar is 24, I remember those from the earliest days of playing around in assembly language on an Epson PC-compatible running MS-DOS 5 :) But that's beside the point. I don't think there'd be huge problems with a 4.0 release that's just like 3.10 except that it's a little more free with removal of deprecateds. Maybe that could be the point at which 2.x compatibility is dropped, like the u"string" notation. Or maybe not :) ChrisA From storchaka at gmail.com Mon Mar 10 20:42:54 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 10 Mar 2014 21:42:54 +0200 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> <20140310170857.31EBF250C36@webabinitio.net> Message-ID: 10.03.14 20:50, Paul Moore ???????(??): > I have seen a number of postings recently pointing to things as "not > until Python 4000" or "not until Python 4.0" (yours was not one that I > noticed, actually, this is a more general point). This is just an euphemism for "not in observable future". From emile at fenx.com Mon Mar 10 20:43:03 2014 From: emile at fenx.com (Emile van Sebille) Date: Mon, 10 Mar 2014 12:43:03 -0700 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> <20140310115918.7b848934@anarchist.wooz.org> <20140310175628.71f80415@fsol> Message-ID: On 3/10/2014 12:29 PM, Chris Angelico wrote: > I don't think there'd be huge problems > with a 4.0 release that's just like 3.10 except that it's a little > more free with removal of deprecateds. Maybe that could be the point > at which 2.x compatibility is dropped, ... and the point at which those of us on 2.x jump over 3.x to 4.x. :) Emile From storchaka at gmail.com Mon Mar 10 20:47:32 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 10 Mar 2014 21:47:32 +0200 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <20140310174456.GA1360@phdru.name> References: <20140310150452.GC12595@ando> <20140310174456.GA1360@phdru.name> Message-ID: 10.03.14 19:44, Oleg Broytman ???????(??): > There is one minor annoyance with double digits: > > $ ls -l > total 16 > drwx------ 2 phd phd 4096 Mar 10 21:42 3.1 > drwx------ 2 phd phd 4096 Mar 10 21:42 3.10 > drwx------ 2 phd phd 4096 Mar 10 21:42 3.2 > ... ... > drwx------ 2 phd phd 4096 Mar 10 21:42 4.0 ls -lv From ethan at stoneleaf.us Mon Mar 10 20:28:25 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 10 Mar 2014 12:28:25 -0700 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <531E02BF.6020908@mrabarnett.plus.com> References: <20140310150452.GC12595@ando> <20140310170857.31EBF250C36@webabinitio.net> <531E02BF.6020908@mrabarnett.plus.com> Message-ID: <531E1259.6030609@stoneleaf.us> On 03/10/2014 11:21 AM, MRAB wrote: > > What does "irregardless" mean? The same thing as "regardless", with an extra syllable just for fun. -- ~Ethan~ From phd at phdru.name Mon Mar 10 20:57:10 2014 From: phd at phdru.name (Oleg Broytman) Date: Mon, 10 Mar 2014 20:57:10 +0100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> <20140310174456.GA1360@phdru.name> Message-ID: <20140310195710.GA8320@phdru.name> On Mon, Mar 10, 2014 at 09:47:32PM +0200, Serhiy Storchaka wrote: > 10.03.14 19:44, Oleg Broytman ???????(??): > > There is one minor annoyance with double digits: > > > >$ ls -l > >total 16 > >drwx------ 2 phd phd 4096 Mar 10 21:42 3.1 > >drwx------ 2 phd phd 4096 Mar 10 21:42 3.10 > >drwx------ 2 phd phd 4096 Mar 10 21:42 3.2 > > ... ... > >drwx------ 2 phd phd 4096 Mar 10 21:42 4.0 > > ls -lv Not every filemanager can do that. Not even every ls: $ uname -a FreeBSD 9.2-RELEASE $ man ls | grep -F -- -v Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From tseaver at palladion.com Mon Mar 10 21:14:19 2014 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 10 Mar 2014 16:14:19 -0400 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> <20140310170857.31EBF250C36@webabinitio.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/10/2014 02:50 PM, Paul Moore wrote: > I have seen a number of postings recently pointing to things as "not > until Python 4000" or "not until Python 4.0" (yours was not one that > I noticed, actually, this is a more general point). > > I do think it's a bad idea to start talking in terms of "the next big > compatibility break", even if *we* know there's no immediate plan. > People are very quick to pick up messages like "Now that Python 3 is > out of the way, the Python devs are talking about Python 4" which is > not a message we want to see getting traction. Exactly. If we can't do things sensibly / incrementally without learning from the painful lessons of the past eight years, we ought to give up on the prospect of doing language design altogether. > I'm neither averse to, nor in favour of, a Python 4 compatibility > break in due course I am. I don't think the community can stand another such break. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlMeHRsACgkQ+gerLs4ltQ4p0QCeLWgr2/qOSHmRBLLD+Wz0/+k/ EcwAn2p4lXARRCEYhyqsDpwhq/SrVEak =ZpwN -----END PGP SIGNATURE----- From ncoghlan at gmail.com Mon Mar 10 22:15:08 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 11 Mar 2014 07:15:08 +1000 Subject: [Python-Dev] Python 3.4 change in importlib/__init__.py breaking cxFreeze? In-Reply-To: References: <531DD7E2.5080005@pke.hr> Message-ID: On 11 Mar 2014 01:44, "Brett Cannon" wrote: > > > > On Mon Mar 10 2014 at 11:41:27 AM, Jurko Gospodneti? < jurko.gospodnetic at pke.hr> wrote: >> >> Hi Nick. >> >> On 10.3.2014. 14:25, Nick Coghlan wrote:> What is supposed to happen >> when that code gets loaded from a ZIP archive? >> > >> > __file__ is expected to always be set (including when loaded from a >> > zipfile - in that case it's the zipfile name concatenated with the >> > path within the zip file). If it isn't set, there's a buggy loader >> > involved somewhere that isn't setting it properly. >> >> I don't recall seeing that ever explicitly stated. For that matter, >> Python 3.4.0rc3 documentation explicitly states: >> >> > __file__ is optional. If set, this attribute?s value must be a string. >> > The import system may opt to leave __file__ unset if it has no >> > semantic meaning (e.g. a module loaded from a database). >> >> and: >> >> > Ultimately, the loader is what makes use of __file__ and/or __cached__. >> >> Or is this some rule specific to the importlib/__init__.py stdlib module? > > > No, Nick was mistaken and Barry's response is accurate: __file__ is optional and left off when it doesn't make any sense. Since importlib._bootstrap is a frozen module by default it doesn't have __file__ set. Yeah, I was thinking of the 3.4 changes to ensure that various *other* module attributes are always set, even in __main__ (and frozen modules). My apologies for the misinformation - I should have waited until I had a chance to look it up properly. Cheers, Nick. > > -Brett > >> >> >> As I recall, I first learned that not all loaded modules need to have >> their __file__ attribute set by researching a failure in some package >> when installed as a zipped-egg using setuptools. Admittedly though, that >> was some old setuptools version. >> >> Best regards, >> Jurko Gospodneti? >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Mar 10 22:39:20 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 10 Mar 2014 14:39:20 -0700 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> <20140310170857.31EBF250C36@webabinitio.net> Message-ID: On Mon, Mar 10, 2014 at 12:42 PM, Serhiy Storchaka wrote: > This is just an euphemism for "not in observable future". > > is ANY of the future observable? Oh right, The Time Machine! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Mar 10 22:54:12 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 10 Mar 2014 21:54:12 +0000 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> <20140310170857.31EBF250C36@webabinitio.net> Message-ID: On 10 March 2014 19:42, Serhiy Storchaka wrote: > 10.03.14 20:50, Paul Moore ???????(??): > >> I have seen a number of postings recently pointing to things as "not >> until Python 4000" or "not until Python 4.0" (yours was not one that I >> noticed, actually, this is a more general point). > > > This is just an euphemism for "not in observable future". I understand that - my concern is that people reading such comments out of context might not realise this ("after all, that was what Python 3000 meant, then you went and implemented it"). Paul From breamoreboy at yahoo.co.uk Mon Mar 10 23:20:04 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 10 Mar 2014 22:20:04 +0000 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <531E1259.6030609@stoneleaf.us> References: <20140310150452.GC12595@ando> <20140310170857.31EBF250C36@webabinitio.net> <531E02BF.6020908@mrabarnett.plus.com> <531E1259.6030609@stoneleaf.us> Message-ID: On 10/03/2014 19:28, Ethan Furman wrote: > On 03/10/2014 11:21 AM, MRAB wrote: >> >> What does "irregardless" mean? > > The same thing as "regardless", with an extra syllable just for fun. > > -- > ~Ethan~ Is this the UK, US, Australian or some other "regardless"? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com From greg.ewing at canterbury.ac.nz Mon Mar 10 23:28:33 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 11 Mar 2014 11:28:33 +1300 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> <20140310115918.7b848934@anarchist.wooz.org> <20140310175628.71f80415@fsol> Message-ID: <531E3C91.7020206@canterbury.ac.nz> Chris Angelico wrote: > Terrible idea. Would wreak havoc with comparisons. No. Python 3 is all > about Unicode, so the right way to proceed is 3.8, 3.9, 3.:, 3.;, 3.<, > 3.=, 3.>, 3.?, 3.@, 3.A. And we have all of UCS-4 to play with, so for all practical purposes the 3.x line can live forever! The downside is that we'll get endless complaints from jmfauth about the Flexible Version Number Representation. :-( -- Greg From greg.ewing at canterbury.ac.nz Mon Mar 10 23:29:25 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 11 Mar 2014 11:29:25 +1300 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <531E02BF.6020908@mrabarnett.plus.com> References: <20140310150452.GC12595@ando> <20140310170857.31EBF250C36@webabinitio.net> <531E02BF.6020908@mrabarnett.plus.com> Message-ID: <531E3CC5.20304@canterbury.ac.nz> MRAB wrote: > What does "irregardless" mean? It's what people say when they misunderestimate the importance of correct prefix usage in English. -- Greg From breamoreboy at yahoo.co.uk Mon Mar 10 23:35:29 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 10 Mar 2014 22:35:29 +0000 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <531E3C91.7020206@canterbury.ac.nz> References: <20140310150452.GC12595@ando> <20140310115918.7b848934@anarchist.wooz.org> <20140310175628.71f80415@fsol> <531E3C91.7020206@canterbury.ac.nz> Message-ID: On 10/03/2014 22:28, Greg Ewing wrote: > Chris Angelico wrote: >> Terrible idea. Would wreak havoc with comparisons. No. Python 3 is all >> about Unicode, so the right way to proceed is 3.8, 3.9, 3.:, 3.;, 3.<, >> 3.=, 3.>, 3.?, 3.@, 3.A. > > And we have all of UCS-4 to play with, so for all > practical purposes the 3.x line can live forever! > > The downside is that we'll get endless complaints > from jmfauth about the Flexible Version Number > Representation. :-( > Drat, drat and double drat, you beat me to it :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com From cyberdupo56 at gmail.com Mon Mar 10 23:58:59 2014 From: cyberdupo56 at gmail.com (Allen Li) Date: Mon, 10 Mar 2014 18:58:59 -0400 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> <20140310115918.7b848934@anarchist.wooz.org> <20140310175628.71f80415@fsol> <531E3C91.7020206@canterbury.ac.nz> Message-ID: <20140310225859.GA24069@avalon.amherst.edu> I'm not a dev, so my comment doesn't have that much weight, but it is possible to stop flooding the mailing list with idle chitchat about something mostly irrelevant and non-productive? There's nothing wrong with the current Python versioning scheme. Python 4 is not planned for the near future. I don't see anything else worthy of discussion on this topic. Allen Li On Mon, Mar 10, 2014 at 10:35:29PM +0000, Mark Lawrence wrote: > On 10/03/2014 22:28, Greg Ewing wrote: > >Chris Angelico wrote: > >>Terrible idea. Would wreak havoc with comparisons. No. Python 3 is all > >>about Unicode, so the right way to proceed is 3.8, 3.9, 3.:, 3.;, 3.<, > >>3.=, 3.>, 3.?, 3.@, 3.A. > > > >And we have all of UCS-4 to play with, so for all > >practical purposes the 3.x line can live forever! > > > >The downside is that we'll get endless complaints > >from jmfauth about the Flexible Version Number > >Representation. :-( > > > > Drat, drat and double drat, you beat me to it :) > > -- > My fellow Pythonistas, ask not what our language can do for you, ask what > you can do for our language. > > Mark Lawrence > > --- > This email is free from viruses and malware because avast! Antivirus protection is active. > http://www.avast.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/cyberdupo56%40gmail.com From jimjjewett at gmail.com Tue Mar 11 00:08:52 2014 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Mon, 10 Mar 2014 16:08:52 -0700 (PDT) Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] In-Reply-To: Message-ID: <531e4604.085ee00a.36c2.2cfd@mx.google.com> On Mon Mar 10 18:56:17 CET 2014 (and earlier quotes), Maciej Fijalkowski wrote: Maciej:>> You should not rely on __del__s being called timely one way or Maciej:>> another. Why would you require this for the program to work Maciej:>> correctly in the particular example of __traceback__? To the extent that I understand, he isn't requiring it for correctness; he is instead saying that without timely __del__, the Quality of Implementation suffers. I suspect there are aspects of tulip (or event processing in general) that make it more common for the frame graph to be painfully cyclic, so that live frames keep dead ones from being collected. It may also be more common to have multiple __del__ methods in the same cycle, if cycles are created by a framework. So the problems aren't new, but they may have become considerably more painful. Victor:> For asyncio, it's very useful to see unhandled exceptions as early as Victor:> possible. Otherwise, your program is blocked and you don't know why. ... Maciej:>> twisted goes around it by attaching errback by hand. Would that work Maciej:>> for tulip? Maciej:>> deferred.addErrback(callback_that_writes_to_log) What do you mean "by hand"? Does the framework automatically add a "log the exception" errback to every task, or every task that doesn't have its own errback of some sort? Or do you mean that users should do so by hand, but that it is a well-known recipe? Maciej:>> I'm very skeptical about changing details of __traceback__ and Maciej:>> frames, just in order to make refcounting work (since it would Maciej:>> create something that would not work on pypy for example). How about just loosening some constraints on exceptions, in order to permit more efficient operation, but in a way that may be particularly useful to a refcounting scheme? Can I assume that you don't object to frame.clear()? How about a hypothethetical traceback.pack() that made it easier to reclaim memory held by frame/traceback cycles? If standard traceback printing were the only likely future use, each frame/traceback pair could be replaced by 4 pointers, and allocating space/copying those 4 would be the only work that wasn't already needed for the eventual deallocation. Today, the setters for __cause__, __context__, and __traceback do typechecks to ensure that those properties are (None or) the expected type; __traceback__ doesn't even allow subclasses. The constructors for frame and traceback are similarly strict. What would be the harm in allowing arbitrary objects, let alone a few specific alternative implementations? -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From solipsis at pitrou.net Tue Mar 11 00:16:46 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 11 Mar 2014 00:16:46 +0100 Subject: [Python-Dev] What is the precise problem? [was: Reference cycles in Exception.__traceback__] References: <531e4604.085ee00a.36c2.2cfd@mx.google.com> Message-ID: <20140311001646.384a2a63@fsol> On Mon, 10 Mar 2014 16:08:52 -0700 (PDT) "Jim J. Jewett" wrote: > It may also be more common to have multiple __del__ methods in the > same cycle, if cycles are created by a framework. So the problems > aren't new, but they may have become considerably more painful. Multiple __del__s shouldn't change anything compared to a single __del__ or even none (see PEP 442). > Maciej:>> I'm very skeptical about changing details of __traceback__ and > Maciej:>> frames, just in order to make refcounting work (since it would > Maciej:>> create something that would not work on pypy for example). > > How about just loosening some constraints on exceptions, in order to > permit more efficient operation, but in a way that may be particularly > useful to a refcounting scheme? Can I assume that you don't object to > frame.clear()? I posted a prototype change (in this thread) which automatically calls frame.clear(), I didn't get any feedback about it. Regards Antoine. From rdmurray at bitdance.com Tue Mar 11 02:29:29 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 10 Mar 2014 21:29:29 -0400 Subject: [Python-Dev] undocumented help() function change in Python 3.4? In-Reply-To: <531DBAF0.7010903@pke.hr> References: <531DBAF0.7010903@pke.hr> Message-ID: <20140311012929.F00FC250C31@webabinitio.net> On Mon, 10 Mar 2014 14:15:28 +0100, =?UTF-8?B?SnVya28gR29zcG9kbmV0acSH?= wrote: > Hi. > > On 8.3.2014. 6:48, Nick Coghlan wrote: > > Yeah, the changes to help() are doubly indirect - help() uses pydoc > > which uses inspect, and inspect saw a lot of changes. > > > > I'll make a few updates to the What's New to help make the > > consequences of this clearer. > > Just looked through the latest 3.4rc3 release and nothing related to > this seems to have been included yet. The whatsnew updates (including the one for help) weren't copied into rc3. They will be in final though, unless Larry forgets. You can see them here: http://docs.python.org/dev/whatsnew/3.4.html --David From rdmurray at bitdance.com Tue Mar 11 03:05:52 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 10 Mar 2014 22:05:52 -0400 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... Message-ID: <20140311020553.16E23250C31@webabinitio.net> Well, I think What's New for 3.4 is done. I've been through all of the NEWS items from the start of 3.4 through the beta1 release. I've gone over the list of changes Serhiy found via the versionadded/versionchanged in the docs. (Since he marked some that didn't turn out to be 3.4 changes, I assume he was over-inclusive rather than under-inclusive and am not going to re-run that particular check myself...thanks Serhiy for doing it!). In addition to the items in Serhiy's list that didn't have news entries, there were a couple of features that were added after Beta1. So there might be some other features with missing versionadded/changed tags in the NEWS sections I didn't go through, but I hope not. Slightly more worrisome is the possibility that I'm missing porting notes from the beta/rc phases. But, I'm pretty much out of time for this project since Final is almost upon us. I'll be making at least one more copy-edit pass over the document, and may reformat some stuff, but the content is pretty much set at this point. If anyone knows of anything that is missing, please let me know about it. --David I track my time as a habit, so for the curious I can tell you with a fair degree of accuracy how long this little project took: about 73 hours total, starting on 12/20 last year. Let me tell you, it felt even longer than that :) From cs at zip.com.au Tue Mar 11 03:19:27 2014 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 11 Mar 2014 13:19:27 +1100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: Message-ID: <20140311021927.GA15098@cskk.homeip.net> On 10Mar2014 14:55, Victor Stinner wrote: > Last 5 years, I spend significant time to port a lot of Python 2 code > on Python 3. [... troubles ...] > So can we please try to stop scheduling another major Python version > breaking almost all modules and all applications just to be pendantic? > No, we should not remove any old feature in Python 4. Python 4 should > be just a minor release following the previous 3.x release. Maybe that will be the case, when it rolls around in the ordinary sequence of things. Assuming nothing groundbreaks shows up. > I don't want another six, nine or whatever module to fill the gap > between Python 3 and Python 4. But this I do not understand. If 4.0 is in your vision to be a regular release, why should you care that it may be years off? > For example, I propose to release the next major Python version (3.5) > with the version 4.0 but without removing anything. (It's just an > example, it can wait another release.) Just in case this is not a joke or hyperbole: I am -1 on this. Just stick with the expected numbering scheme. If there are no incompatible but desired changes queued up by the time 4.0, perhaps that will be a normal release also. If not, perhaps it will be a good time to bite the bullet. > I mean that any incompatible > change must following the classic transition plan: tag the feature as > deprecated and wait at least one major version before dropping it, or > just keep it forever. We can expect that just changing the major > version (3 => 4) will already break enough applications (where > "python3" or "version == 3" is hardcoded")... I tend to spell this >= 3, etc. Maybe I'm being simplistic. > Instead of asking application developers to try to follow each major > Python release, we should try to keep the maintaince pain in Python > core. > > What do you think? I agree there shouldn't be a major porting pain release just for the sake of a number change; anything like that should need justification. But conversely, I'm dead against bringing forward version 4.0 just to break the expectation of breakage. Cheers, -- Cameron Simpson The nice thing about standards is that you have so many to choose from; furthermore, if you do not like any of them, you can just wait for next year's model. - Andrew S. Tanenbaum From eric at trueblade.com Tue Mar 11 03:13:44 2014 From: eric at trueblade.com (Eric V. Smith) Date: Mon, 10 Mar 2014 22:13:44 -0400 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: <20140311020553.16E23250C31@webabinitio.net> References: <20140311020553.16E23250C31@webabinitio.net> Message-ID: Thanks for all of the great work! -- Eric. > On Mar 10, 2014, at 10:05 PM, "R. David Murray" wrote: > > Well, I think What's New for 3.4 is done. I've been through all of the > NEWS items from the start of 3.4 through the beta1 release. I've gone > over the list of changes Serhiy found via the versionadded/versionchanged > in the docs. (Since he marked some that didn't turn out to be 3.4 > changes, I assume he was over-inclusive rather than under-inclusive > and am not going to re-run that particular check myself...thanks > Serhiy for doing it!). > > In addition to the items in Serhiy's list that didn't have news entries, > there were a couple of features that were added after Beta1. So there > might be some other features with missing versionadded/changed tags in > the NEWS sections I didn't go through, but I hope not. Slightly more > worrisome is the possibility that I'm missing porting notes from the > beta/rc phases. But, I'm pretty much out of time for this project since > Final is almost upon us. I'll be making at least one more copy-edit > pass over the document, and may reformat some stuff, but the content is > pretty much set at this point. > > If anyone knows of anything that is missing, please let me know about > it. > > --David > > I track my time as a habit, so for the curious I can tell you with > a fair degree of accuracy how long this little project took: about 73 > hours total, starting on 12/20 last year. Let me tell you, it felt even > longer than that :) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com > From ncoghlan at gmail.com Tue Mar 11 04:55:36 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 11 Mar 2014 13:55:36 +1000 Subject: [Python-Dev] undocumented help() function change in Python 3.4? In-Reply-To: <20140311012929.F00FC250C31@webabinitio.net> References: <531DBAF0.7010903@pke.hr> <20140311012929.F00FC250C31@webabinitio.net> Message-ID: On 11 March 2014 11:29, R. David Murray wrote: > The whatsnew updates (including the one for help) weren't copied into > rc3. They will be in final though, unless Larry forgets. Oh, cool - yes, it will be good to have an up to date What's New shipped, especially as part of the compiled Windows docs. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From nad at acm.org Tue Mar 11 06:31:52 2014 From: nad at acm.org (Ned Deily) Date: Mon, 10 Mar 2014 22:31:52 -0700 Subject: [Python-Dev] undocumented help() function change in Python 3.4? References: <531DBAF0.7010903@pke.hr> <20140311012929.F00FC250C31@webabinitio.net> Message-ID: In article , Nick Coghlan wrote: > On 11 March 2014 11:29, R. David Murray wrote: > > The whatsnew updates (including the one for help) weren't copied into > > rc3. They will be in final though, unless Larry forgets. > > Oh, cool - yes, it will be good to have an up to date What's New > shipped, especially as part of the compiled Windows docs. I was going to bring that point up today. How are all the new whatsnew updates going to get into the 3.4.0 release? They've, correctly, been being pushed to the default branch. But, AFAIK, for them to show up in the 3.4.0 released docs we ship, all of the doc changes are going to need to be cherry picked (or a big mass diff from the default branch) into the 3.4 release branch, otherwise they will not be part of the release. (They certainly won't be part of the docs included with the installers I build unless they are in the 3.4 releasing branch.) Thank you for all the work you've been putting into the whatsnew doc! -- Ned Deily, nad at acm.org From ericsnowcurrently at gmail.com Tue Mar 11 06:58:39 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 10 Mar 2014 23:58:39 -0600 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: <20140311020553.16E23250C31@webabinitio.net> References: <20140311020553.16E23250C31@webabinitio.net> Message-ID: On Mon, Mar 10, 2014 at 8:05 PM, R. David Murray wrote: > Well, I think What's New for 3.4 is done. I've been through all of the > NEWS items from the start of 3.4 through the beta1 release. > > I'm pretty much out of time for this project since > Final is almost upon us. I'll be making at least one more copy-edit > pass over the document, and may reformat some stuff, but the content is > pretty much set at this point. > > > I track my time as a habit, so for the curious I can tell you with > a fair degree of accuracy how long this little project took: about 73 > hours total, starting on 12/20 last year. Let me tell you, it felt even > longer than that :) Thanks for taking the time on something that I image isn't all that fun! -eric From stephen at xemacs.org Tue Mar 11 07:14:20 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 11 Mar 2014 15:14:20 +0900 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: References: <20140310150452.GC12595@ando> <20140310170857.31EBF250C36@webabinitio.net> Message-ID: <87iorlntxf.fsf@uwakimon.sk.tsukuba.ac.jp> Paul Moore writes: > I understand that - my concern is that people reading such comments > out of context might not realise this ("after all, that was what > Python 3000 meant, then you went and implemented it"). Sure, but why worry about it? The important part of "willful ignorance" is the "willful". We can't "fix" people who are willing to accept that crap -- they'll find some other crap to believe. Python 3 already has undeniable cruft (some Python 2 compatibility stuff) and arguably some things that could have been done better but can't really be changed in Python 3 (see the backtrace thread). Python makes strong promises that those things will *not* become compatibility breaks within 3.x, even if we're pretty confident that "nobody" is using them any more, and especially not when we now just think we could have done a better job on some feature. The flip side of that is that Python 4000 is going to be mentioned as the appropriate time for changing them. If that confuses people, well, we can just unconfuse them (if that's OK with them...). From g.brandl at gmx.net Tue Mar 11 07:47:15 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 11 Mar 2014 07:47:15 +0100 Subject: [Python-Dev] Python 4: don't remove anything, don't break backward compatibility In-Reply-To: <20140310225859.GA24069@avalon.amherst.edu> References: <20140310150452.GC12595@ando> <20140310115918.7b848934@anarchist.wooz.org> <20140310175628.71f80415@fsol> <531E3C91.7020206@canterbury.ac.nz> <20140310225859.GA24069@avalon.amherst.edu> Message-ID: Am 10.03.2014 23:58, schrieb Allen Li: > I'm not a dev, so my comment doesn't have that much weight, but it is > possible to stop flooding the mailing list with idle chitchat about > something mostly irrelevant and non-productive? > > There's nothing wrong with the current Python versioning scheme. Python > 4 is not planned for the near future. I don't see anything else worthy > of discussion on this topic. Then you should just have a moderator lock the thread, and ... oh wait. Georg From g.brandl at gmx.net Tue Mar 11 07:48:34 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 11 Mar 2014 07:48:34 +0100 Subject: [Python-Dev] undocumented help() function change in Python 3.4? In-Reply-To: References: <531DBAF0.7010903@pke.hr> <20140311012929.F00FC250C31@webabinitio.net> Message-ID: Am 11.03.2014 06:31, schrieb Ned Deily: > In article > , > Nick Coghlan wrote: >> On 11 March 2014 11:29, R. David Murray wrote: >> > The whatsnew updates (including the one for help) weren't copied into >> > rc3. They will be in final though, unless Larry forgets. >> >> Oh, cool - yes, it will be good to have an up to date What's New >> shipped, especially as part of the compiled Windows docs. > > I was going to bring that point up today. How are all the new whatsnew > updates going to get into the 3.4.0 release? They've, correctly, been > being pushed to the default branch. But, AFAIK, for them to show up in > the 3.4.0 released docs we ship, all of the doc changes are going to > need to be cherry picked (or a big mass diff from the default branch) > into the 3.4 release branch, otherwise they will not be part of the > release. (They certainly won't be part of the docs included with the > installers I build unless they are in the 3.4 releasing branch.) Copying the file from default and doing just one commit in the releasing branch should be the easiest way. Georg From nad at acm.org Tue Mar 11 08:00:28 2014 From: nad at acm.org (Ned Deily) Date: Tue, 11 Mar 2014 00:00:28 -0700 Subject: [Python-Dev] undocumented help() function change in Python 3.4? References: <531DBAF0.7010903@pke.hr> <20140311012929.F00FC250C31@webabinitio.net> Message-ID: In article , Georg Brandl wrote: > Am 11.03.2014 06:31, schrieb Ned Deily: > > In article > > , > > Nick Coghlan wrote: > >> On 11 March 2014 11:29, R. David Murray wrote: > >> > The whatsnew updates (including the one for help) weren't copied into > >> > rc3. They will be in final though, unless Larry forgets. > >> > >> Oh, cool - yes, it will be good to have an up to date What's New > >> shipped, especially as part of the compiled Windows docs. > > > > I was going to bring that point up today. How are all the new whatsnew > > updates going to get into the 3.4.0 release? They've, correctly, been > > being pushed to the default branch. But, AFAIK, for them to show up in > > the 3.4.0 released docs we ship, all of the doc changes are going to > > need to be cherry picked (or a big mass diff from the default branch) > > into the 3.4 release branch, otherwise they will not be part of the > > release. (They certainly won't be part of the docs included with the > > installers I build unless they are in the 3.4 releasing branch.) > > Copying the file from default and doing just one commit in the releasing > branch should be the easiest way. That covers the whatsnew file but the changes RDM have been making affect other parts of the documentation and Misc/NEWS, too. -- Ned Deily, nad at acm.org From g.brandl at gmx.net Tue Mar 11 10:15:21 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 11 Mar 2014 10:15:21 +0100 Subject: [Python-Dev] undocumented help() function change in Python 3.4? In-Reply-To: References: <531DBAF0.7010903@pke.hr> <20140311012929.F00FC250C31@webabinitio.net> Message-ID: Am 11.03.2014 08:00, schrieb Ned Deily: > In article , > Georg Brandl wrote: > >> Am 11.03.2014 06:31, schrieb Ned Deily: >> > In article >> > , >> > Nick Coghlan wrote: >> >> On 11 March 2014 11:29, R. David Murray wrote: >> >> > The whatsnew updates (including the one for help) weren't copied into >> >> > rc3. They will be in final though, unless Larry forgets. >> >> >> >> Oh, cool - yes, it will be good to have an up to date What's New >> >> shipped, especially as part of the compiled Windows docs. >> > >> > I was going to bring that point up today. How are all the new whatsnew >> > updates going to get into the 3.4.0 release? They've, correctly, been >> > being pushed to the default branch. But, AFAIK, for them to show up in >> > the 3.4.0 released docs we ship, all of the doc changes are going to >> > need to be cherry picked (or a big mass diff from the default branch) >> > into the 3.4 release branch, otherwise they will not be part of the >> > release. (They certainly won't be part of the docs included with the >> > installers I build unless they are in the 3.4 releasing branch.) >> >> Copying the file from default and doing just one commit in the releasing >> branch should be the easiest way. > > That covers the whatsnew file but the changes RDM have been making > affect other parts of the documentation and Misc/NEWS, too. NEWS can also be copied of course. As long as the others aren't critical updates, just skip it for the final. Most people read the online docs anyway. Georg From victor.stinner at gmail.com Tue Mar 11 13:18:53 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 11 Mar 2014 13:18:53 +0100 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: <20140311020553.16E23250C31@webabinitio.net> References: <20140311020553.16E23250C31@webabinitio.net> Message-ID: Hi, Thanks David! I added a summary of security improvements: http://docs.python.org/dev/whatsnew/3.4.html#summary-release-highlights Can someone please review it? Don't hesitate to modify the text directly. Check also if the summary is complete. Victor 2014-03-11 3:05 GMT+01:00 R. David Murray : > Well, I think What's New for 3.4 is done. I've been through all of the > NEWS items from the start of 3.4 through the beta1 release. I've gone > over the list of changes Serhiy found via the versionadded/versionchanged > in the docs. (Since he marked some that didn't turn out to be 3.4 > changes, I assume he was over-inclusive rather than under-inclusive > and am not going to re-run that particular check myself...thanks > Serhiy for doing it!). > > In addition to the items in Serhiy's list that didn't have news entries, > there were a couple of features that were added after Beta1. So there > might be some other features with missing versionadded/changed tags in > the NEWS sections I didn't go through, but I hope not. Slightly more > worrisome is the possibility that I'm missing porting notes from the > beta/rc phases. But, I'm pretty much out of time for this project since > Final is almost upon us. I'll be making at least one more copy-edit > pass over the document, and may reformat some stuff, but the content is > pretty much set at this point. > > If anyone knows of anything that is missing, please let me know about > it. > > --David > > I track my time as a habit, so for the curious I can tell you with > a fair degree of accuracy how long this little project took: about 73 > hours total, starting on 12/20 last year. Let me tell you, it felt even > longer than that :) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From ncoghlan at gmail.com Tue Mar 11 13:28:25 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 11 Mar 2014 22:28:25 +1000 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: <20140311020553.16E23250C31@webabinitio.net> References: <20140311020553.16E23250C31@webabinitio.net> Message-ID: On 11 March 2014 12:05, R. David Murray wrote: > In addition to the items in Serhiy's list that didn't have news entries, > there were a couple of features that were added after Beta1. So there > might be some other features with missing versionadded/changed tags in > the NEWS sections I didn't go through, but I hope not. Slightly more > worrisome is the possibility that I'm missing porting notes from the > beta/rc phases. But, I'm pretty much out of time for this project since > Final is almost upon us. I'll be making at least one more copy-edit > pass over the document, and may reformat some stuff, but the content is > pretty much set at this point. Thank you for that! I was thinking of adding a new "Migrating from Python 2" section at the end of the porting guide, noting the changed recommendations in the migration guide (i.e. people that read it a while ago should read it again), as well as the restoration of the binary and text transform codec aliases. Sound reasonable? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Tue Mar 11 14:05:21 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 11 Mar 2014 14:05:21 +0100 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: References: <20140311020553.16E23250C31@webabinitio.net> Message-ID: 2014-03-11 13:28 GMT+01:00 Nick Coghlan : > I was thinking of adding a new "Migrating from Python 2" section at > the end of the porting guide, noting the changed recommendations in > the migration guide (i.e. people that read it a while ago should read > it again), as well as the restoration of the binary and text transform > codec aliases. Sound reasonable? Such info is useful, but I don't think that the What's New in Python 3.4 document is the right place. Or maybe add a link to another document. Victor From rdmurray at bitdance.com Tue Mar 11 14:52:28 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 11 Mar 2014 09:52:28 -0400 Subject: [Python-Dev] undocumented help() function change in Python 3.4? In-Reply-To: References: <531DBAF0.7010903@pke.hr> <20140311012929.F00FC250C31@webabinitio.net> Message-ID: <20140311135229.73D38250C31@webabinitio.net> On Tue, 11 Mar 2014 10:15:21 +0100, Georg Brandl wrote: > Am 11.03.2014 08:00, schrieb Ned Deily: > > In article , > > Georg Brandl wrote: > > > >> Am 11.03.2014 06:31, schrieb Ned Deily: > >> > In article > >> > , > >> > Nick Coghlan wrote: > >> >> On 11 March 2014 11:29, R. David Murray wrote: > >> >> > The whatsnew updates (including the one for help) weren't copied into > >> >> > rc3. They will be in final though, unless Larry forgets. > >> >> > >> >> Oh, cool - yes, it will be good to have an up to date What's New > >> >> shipped, especially as part of the compiled Windows docs. > >> > > >> > I was going to bring that point up today. How are all the new whatsnew > >> > updates going to get into the 3.4.0 release? They've, correctly, been > >> > being pushed to the default branch. But, AFAIK, for them to show up in > >> > the 3.4.0 released docs we ship, all of the doc changes are going to > >> > need to be cherry picked (or a big mass diff from the default branch) > >> > into the 3.4 release branch, otherwise they will not be part of the > >> > release. (They certainly won't be part of the docs included with the > >> > installers I build unless they are in the 3.4 releasing branch.) > >> > >> Copying the file from default and doing just one commit in the releasing > >> branch should be the easiest way. > > > > That covers the whatsnew file but the changes RDM have been making > > affect other parts of the documentation and Misc/NEWS, too. > > NEWS can also be copied of course. As long as the others aren't critical > updates, just skip it for the final. > > Most people read the online docs anyway. I don't think the NEWS changes are that important to copy either. None of the changes I made outside of whatsnew are substantive, they were either adding or fixing version tags, fixing formatting, or fixing typos or other copy-edit type things. At least, I can't remember anything that was substantive :) Oh, there was one thing that might be worth cherry picking: the change to the email docs. The new EmailMessage class was documented as being in the wrong module. But even that isn't a big deal, since it's a provisional class, and as Georg said most people read the online docs anyway, since they get the ongoing updates between releases. --David From guido at python.org Tue Mar 11 15:54:14 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 11 Mar 2014 07:54:14 -0700 Subject: [Python-Dev] undocumented help() function change in Python 3.4? In-Reply-To: <20140311135229.73D38250C31@webabinitio.net> References: <531DBAF0.7010903@pke.hr> <20140311012929.F00FC250C31@webabinitio.net> <20140311135229.73D38250C31@webabinitio.net> Message-ID: I'm not sure I agree completely with this lax attitude about the contents of the docs, and especially the What's New parts of it (both Misc/NEWS and Doc/whatsnew/3.4.rst). I find it very useful to have these pinpoint *exactly* what made it into the tarball or zipfile or whatever other form I happen to find the release -- you don't always have the full hg logs lying around, nor do you always know from which exact revision or tag a tree was built. Of course it's fine to improve the docs in an ongoing fashion, and if it's just a wording change to NEWS or whatsnew I don't mind missing it. But for specific entries I'd like to strive for completeness in each branch/tag/rc. Also, tonds of thanks to RDM for his work on the new whatsnew! On Tue, Mar 11, 2014 at 6:52 AM, R. David Murray wrote: > On Tue, 11 Mar 2014 10:15:21 +0100, Georg Brandl wrote: > > Am 11.03.2014 08:00, schrieb Ned Deily: > > > In article , > > > Georg Brandl wrote: > > > > > >> Am 11.03.2014 06:31, schrieb Ned Deily: > > >> > In article > > >> > >, > > >> > Nick Coghlan wrote: > > >> >> On 11 March 2014 11:29, R. David Murray > wrote: > > >> >> > The whatsnew updates (including the one for help) weren't copied > into > > >> >> > rc3. They will be in final though, unless Larry forgets. > > >> >> > > >> >> Oh, cool - yes, it will be good to have an up to date What's New > > >> >> shipped, especially as part of the compiled Windows docs. > > >> > > > >> > I was going to bring that point up today. How are all the new > whatsnew > > >> > updates going to get into the 3.4.0 release? They've, correctly, > been > > >> > being pushed to the default branch. But, AFAIK, for them to show > up in > > >> > the 3.4.0 released docs we ship, all of the doc changes are going to > > >> > need to be cherry picked (or a big mass diff from the default > branch) > > >> > into the 3.4 release branch, otherwise they will not be part of the > > >> > release. (They certainly won't be part of the docs included with > the > > >> > installers I build unless they are in the 3.4 releasing branch.) > > >> > > >> Copying the file from default and doing just one commit in the > releasing > > >> branch should be the easiest way. > > > > > > That covers the whatsnew file but the changes RDM have been making > > > affect other parts of the documentation and Misc/NEWS, too. > > > > NEWS can also be copied of course. As long as the others aren't critical > > updates, just skip it for the final. > > > > Most people read the online docs anyway. > > I don't think the NEWS changes are that important to copy either. None > of the changes I made outside of whatsnew are substantive, they were > either adding or fixing version tags, fixing formatting, or fixing > typos or other copy-edit type things. At least, I can't remember > anything that was substantive :) > > Oh, there was one thing that might be worth cherry picking: the change > to the email docs. The new EmailMessage class was documented as being > in the wrong module. But even that isn't a big deal, since it's > a provisional class, and as Georg said most people read the online > docs anyway, since they get the ongoing updates between releases. > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Tue Mar 11 17:17:56 2014 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 11 Mar 2014 12:17:56 -0400 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: References: <20140311020553.16E23250C31@webabinitio.net> Message-ID: <531F3734.2070102@trueblade.com> On 3/11/2014 9:05 AM, Victor Stinner wrote: > 2014-03-11 13:28 GMT+01:00 Nick Coghlan : >> I was thinking of adding a new "Migrating from Python 2" section at >> the end of the porting guide, noting the changed recommendations in >> the migration guide (i.e. people that read it a while ago should read >> it again), as well as the restoration of the binary and text transform >> codec aliases. Sound reasonable? > > Such info is useful, but I don't think that the What's New in Python > 3.4 document is the right place. Or maybe add a link to another > document. I think if the guidance has changed over time, then mentioning it in a What's New document, with a pointer to other documentation, is reasonable. Eric. From Steve.Dower at microsoft.com Tue Mar 11 19:48:56 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Tue, 11 Mar 2014 18:48:56 +0000 Subject: [Python-Dev] Issue 20891 - PyGILState_Ensure on non-Python thread causes fatal error Message-ID: <605a1fa2e4064293a0396bd567748c3b@BLUPR03MB389.namprd03.prod.outlook.com> Hi python-dev Just wanted to draw some attention to http://bugs.python.org/issue20891, which I just created. (I hope I got the right people on the nosy list, but going broad just to be safe.) Details and the discussion can go on there, but the basic gist is that C threads can't safely call PyGILState_Ensure() any more unless PyEval_InitThreads() has been called from an existing Python thread - you get a fatal error because there is no current thread state. Since PyGILState_Ensure is supposed to create this thread state, I believe this is a serious regression and would really like to see it fixed before 3.4 is released. Cheers, Steve From g.brandl at gmx.net Tue Mar 11 19:57:36 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 11 Mar 2014 19:57:36 +0100 Subject: [Python-Dev] undocumented help() function change in Python 3.4? In-Reply-To: References: <531DBAF0.7010903@pke.hr> <20140311012929.F00FC250C31@webabinitio.net> <20140311135229.73D38250C31@webabinitio.net> Message-ID: Am 11.03.2014 15:54, schrieb Guido van Rossum: > I'm not sure I agree completely with this lax attitude about the contents of the > docs, and especially the What's New parts of it (both Misc/NEWS and > Doc/whatsnew/3.4.rst). I don't think anyone here suggested not to update the whatsnew document. > I find it very useful to have these pinpoint *exactly* > what made it into the tarball or zipfile or whatever other form I happen to find > the release -- you don't always have the full hg logs lying around, nor do you > always know from which exact revision or tag a tree was built. In any case, I like to think it's not a lax attitude, but a consistent one: we basically don't want ANY non-critical changes in the RCs, so insufficient docs should be treated like any other bugfix that's not a release blocker: unfortunate, but not world-ending. That exceptions can be and are made (e.g. for the whatsnew document) is because for the docs the potential breakage is lower. > Of course it's fine to improve the docs in an ongoing fashion, and if it's just > a wording change to NEWS or whatsnew I don't mind missing it. But for specific > entries I'd like to strive for completeness in each branch/tag/rc. > > Also, tonds of thanks to RDM for his work on the new whatsnew! Definitely. Georg From ncoghlan at gmail.com Tue Mar 11 22:34:09 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 12 Mar 2014 07:34:09 +1000 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: <531F3734.2070102@trueblade.com> References: <20140311020553.16E23250C31@webabinitio.net> <531F3734.2070102@trueblade.com> Message-ID: On 12 Mar 2014 02:21, "Eric V. Smith" wrote: > > On 3/11/2014 9:05 AM, Victor Stinner wrote: > > 2014-03-11 13:28 GMT+01:00 Nick Coghlan : > >> I was thinking of adding a new "Migrating from Python 2" section at > >> the end of the porting guide, noting the changed recommendations in > >> the migration guide (i.e. people that read it a while ago should read > >> it again), as well as the restoration of the binary and text transform > >> codec aliases. Sound reasonable? > > > > Such info is useful, but I don't think that the What's New in Python > > 3.4 document is the right place. Or maybe add a link to another > > document. > > I think if the guidance has changed over time, then mentioning it in a > What's New document, with a pointer to other documentation, is reasonable. Yeah, that's what I meant - Brett already updated the guide, this would just be a pointer to that. I'll commit something tonight. Cheers, Nick. > > Eric. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Mar 11 22:59:20 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 11 Mar 2014 17:59:20 -0400 Subject: [Python-Dev] undocumented help() function change in Python 3.4? In-Reply-To: References: <531DBAF0.7010903@pke.hr> <20140311012929.F00FC250C31@webabinitio.net> <20140311135229.73D38250C31@webabinitio.net> Message-ID: <20140311215921.4311D250CF1@webabinitio.net> On Tue, 11 Mar 2014 19:57:36 +0100, Georg Brandl wrote: > Am 11.03.2014 15:54, schrieb Guido van Rossum: > > I'm not sure I agree completely with this lax attitude about the contents of the > > docs, and especially the What's New parts of it (both Misc/NEWS and > > Doc/whatsnew/3.4.rst). > > I don't think anyone here suggested not to update the whatsnew document. > > > I find it very useful to have these pinpoint *exactly* > > what made it into the tarball or zipfile or whatever other form I happen to find > > the release -- you don't always have the full hg logs lying around, nor do you > > always know from which exact revision or tag a tree was built. As far as NEWS goes, I fixed some issue callouts so they'll be links in the online version, I deleted a couple redundant entries, I fixed wording here and there, and I added...three?...entries that were missing but I found the commits from the versionadded or versionchanged doc updates they contained that Serhiy found. I'm sure there are other missing NEWS entries that weren't caught, so wanting to know *exactly* what is in the release is a lost cause unless you look at the commit log :) Especially since the revision that contains the commit related to those added news entries does *not* contain the news entry I added, so if you had a tarball built from that revision you wouldn't have that news entry... --David PS: Also, sad to say I picked up the What's New task late in the 3.3 release cycle, and even with doing additional work *after* final I did not finish. So that document may be missing stuff...and in any case, What's New never promises to include *everything*. I have been more completest in 3.4 than Raymond (intentionally) was for 3.2, but it still doesn't have *every* enhancement. From guido at python.org Tue Mar 11 23:07:54 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 11 Mar 2014 15:07:54 -0700 Subject: [Python-Dev] undocumented help() function change in Python 3.4? In-Reply-To: <20140311215921.4311D250CF1@webabinitio.net> References: <531DBAF0.7010903@pke.hr> <20140311012929.F00FC250C31@webabinitio.net> <20140311135229.73D38250C31@webabinitio.net> <20140311215921.4311D250CF1@webabinitio.net> Message-ID: On Tue, Mar 11, 2014 at 2:59 PM, R. David Murray wrote: > On Tue, 11 Mar 2014 19:57:36 +0100, Georg Brandl wrote: > > Am 11.03.2014 15:54, schrieb Guido van Rossum: > > > I'm not sure I agree completely with this lax attitude about the > contents of the > > > docs, and especially the What's New parts of it (both Misc/NEWS and > > > Doc/whatsnew/3.4.rst). > > > > I don't think anyone here suggested not to update the whatsnew document. > > > > > I find it very useful to have these pinpoint *exactly* > > > what made it into the tarball or zipfile or whatever other form I > happen to find > > > the release -- you don't always have the full hg logs lying around, > nor do you > > > always know from which exact revision or tag a tree was built. > > As far as NEWS goes, I fixed some issue callouts so they'll be links in > the online version, I deleted a couple redundant entries, I fixed wording > here and there, and I added...three?...entries that were missing but I > found the commits from the versionadded or versionchanged doc updates > they contained that Serhiy found. I'm sure there are other missing NEWS > entries that weren't caught, so wanting to know *exactly* what is in the > release is a lost cause unless you look at the commit log :) Especially > since the revision that contains the commit related to those added > news entries does *not* contain the news entry I added, so if you had > a tarball built from that revision you wouldn't have that news entry... > OK, understood. The thing is that I like the granularity of Misc/NEWS -- it doesn't cover everything, but it covers pretty much everything you might care about. The changelog is way too verbose. --David > > PS: Also, sad to say I picked up the What's New task late in the 3.3 > release cycle, and even with doing additional work *after* final I did > not finish. So that document may be missing stuff...and in any case, > What's New never promises to include *everything*. I have been more > completest in 3.4 than Raymond (intentionally) was for 3.2, but it still > doesn't have *every* enhancement. > > I think that's a feature. What's New has always been an opinionated document -- it goes from most to least important, and has a structure that makes "skimming" particularly efficient. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Mar 12 01:58:23 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 11 Mar 2014 17:58:23 -0700 Subject: [Python-Dev] Why not make frames? [was: Alternative forms [was: PEP 463: Exception-catching expressions]] In-Reply-To: <531d2086.b1528c0a.0878.ffffa138@mx.google.com> References: <531d2086.b1528c0a.0878.ffffa138@mx.google.com> Message-ID: <531FB12F.3060108@stoneleaf.us> On 03/09/2014 07:16 PM, Jim J. Jewett wrote: > > because I cannot imagine reading an embedded version of either of those > without having to mentally re-parse at the colon. An example assuming > a precedence level that may not be what the PEP proposes: > > if myfunc(5, expr1 except expr2: expr3, "label"): > for i in range(3, 3*max(data) except TypeError: 9, 3): > ... > > if myfunc(5, (expr1 except expr2: expr3), "label"): > for i in range(3, (3*max(data) except TypeError: 9), 3): > ... > > if myfunc(5, expr1 except (expr2: expr3), "label"): > for i in range(3, 3*max(data) except (TypeError: 9), 3): > ... > > if myfunc(5, expr1 except (expr2: expr3), "label"): > for i in range(3, 3*max(data) (except TypeError: 9), 3): > ... > > if myfunc(5, expr1 except (expr3 if expr3), "label"): > for i in range(3, 3*max(data) (except 9 if TypeError), 3): > ... > > if myfunc(5, expr1 except (expr3 if expr3), "label"): > for i in range(3, 3*max(data) except (9 if TypeError), 3): > > myarg = expr1 except (expr3 if expr2) > if myfunc(5, myarg, "label"): > limit = 3*max(data) except (9 if TypeError) > for i in range(3, limit, 3): > > Yes, I would prefer to create a variable naming those expressions, > but these are all still simple enough that I would expect to have > to read them. (I like constructions that get ugly just a bit faster > than they get hard to understand.) If I have to parse any of them, > the ones at the bottom are less difficult than the ones at the top. I totally disagree. I found the second one the easiest to read, and outside a function call (or other complexity, the parens wouldn't be needed. -- ~Ethan~ From ethan at stoneleaf.us Wed Mar 12 04:20:55 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 11 Mar 2014 20:20:55 -0700 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: <531FD297.4040408@stoneleaf.us> I sure hope this is accepted. I could have used it at least a half-dozen times in the last week -- which is more often than I would have used the ternary-if! :) -- ~Ethan~ From rosuav at gmail.com Wed Mar 12 04:58:56 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 12 Mar 2014 14:58:56 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <531FD297.4040408@stoneleaf.us> References: <531FD297.4040408@stoneleaf.us> Message-ID: On Wed, Mar 12, 2014 at 2:20 PM, Ethan Furman wrote: > I sure hope this is accepted. I could have used it at least a half-dozen > times in the last week -- which is more often than I would have used the > ternary-if! :) Where do we go from here? I've not made any edits for some time - no material edits for a good while - how do I request pronouncement? ChrisA From rosuav at gmail.com Wed Mar 12 05:10:30 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 12 Mar 2014 15:10:30 +1100 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions Message-ID: PEP 463, Exception-catching expressions, is stable and I believe ready for pronouncement. Drumroll please... http://www.python.org/dev/peps/pep-0463/ PEP: 463 Title: Exception-catching expressions Version: $Revision$ Last-Modified: $Date$ Author: Chris Angelico Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-Feb-2014 Python-Version: 3.5 Post-History: 20-Feb-2014, 16-Feb-2014 Abstract ======== Just as PEP 308 introduced a means of value-based conditions in an expression, this system allows exception-based conditions to be used as part of an expression. Motivation ========== A number of functions and methods have parameters which will cause them to return a specified value instead of raising an exception. The current system is ad-hoc and inconsistent, and requires that each function be individually written to have this functionality; not all support this. * dict.get(key, default) - second positional argument in place of KeyError * next(iter, default) - second positional argument in place of StopIteration * list.pop() - no way to return a default * seq[index] - no way to handle a bounds error * min(sequence, default=default) - keyword argument in place of ValueError * statistics.mean(data) - no way to handle an empty iterator Had this facility existed early in Python's history, there would have been no need to create dict.get() and related methods; the one obvious way to handle an absent key would be to respond to the exception. One method is written which signals the absence in one way, and one consistent technique is used to respond to the absence. Instead, we have dict.get(), and as of Python 3.4, we also have min(... default=default), and myriad others. We have a LBYL syntax for testing inside an expression, but there is currently no EAFP notation; compare the following:: # LBYL: if key in dic: process(dic[key]) else: process(None) # As an expression: process(dic[key] if key in dic else None) # EAFP: try: process(dic[key]) except KeyError: process(None) # As an expression: process(dic[key] except KeyError: None) Python generally recommends the EAFP policy, but must then proliferate utility functions like dic.get(key,None) to enable this. Rationale ========= The current system requires that a function author predict the need for a default, and implement support for it. If this is not done, a full try/except block is needed. Since try/except is a statement, it is impossible to catch exceptions in the middle of an expression. Just as if/else does for conditionals and lambda does for function definitions, so does this allow exception catching in an expression context. This provides a clean and consistent way for a function to provide a default: it simply raises an appropriate exception, and the caller catches it. With some situations, an LBYL technique can be used (checking if some sequence has enough length before indexing into it, for instance). This is not safe in all cases, but as it is often convenient, programmers will be tempted to sacrifice the safety of EAFP in favour of the notational brevity of LBYL. Additionally, some LBYL techniques (eg involving getattr with three arguments) warp the code into looking like literal strings rather than attribute lookup, which can impact readability. A convenient EAFP notation solves all of this. There's no convenient way to write a helper function to do this; the nearest is something ugly using either lambda:: def except_(expression, exception_list, default): try: return expression() except exception_list: return default() value = except_(lambda: 1/x, ZeroDivisionError, lambda: float("nan")) which is clunky, and unable to handle multiple exception clauses; or eval:: def except_(expression, exception_list, default): try: return eval(expression, globals_of_caller(), locals_of_caller()) except exception_list as exc: l = locals_of_caller().copy() l['exc'] = exc return eval(default, globals_of_caller(), l) def globals_of_caller(): return sys._getframe(2).f_globals def locals_of_caller(): return sys._getframe(2).f_locals value = except_("""1/x""",ZeroDivisionError,""" "Can't divide by zero" """) which is even clunkier, and relies on implementation-dependent hacks. (Writing globals_of_caller() and locals_of_caller() for interpreters other than CPython is left as an exercise for the reader.) Raymond Hettinger `expresses`__ a desire for such a consistent API. Something similar has been `requested`__ `multiple`__ `times`__ in the past. __ https://mail.python.org/pipermail/python-ideas/2014-February/025443.html __ https://mail.python.org/pipermail/python-ideas/2013-March/019760.html __ https://mail.python.org/pipermail/python-ideas/2009-August/005441.html __ https://mail.python.org/pipermail/python-ideas/2008-August/001801.html Proposal ======== Just as the 'or' operator and the three part 'if-else' expression give short circuiting methods of catching a falsy value and replacing it, this syntax gives a short-circuiting method of catching an exception and replacing it. This currently works:: lst = [1, 2, None, 3] value = lst[2] or "No value" The proposal adds this:: lst = [1, 2] value = (lst[2] except IndexError: "No value") Specifically, the syntax proposed is:: (expr except exception_list: default) where expr, exception_list, and default are all expressions. First, expr is evaluated. If no exception is raised, its value is the value of the overall expression. If any exception is raised, exception_list is evaluated, and should result in either a type or a tuple, just as with the statement form of try/except. Any matching exception will result in the corresponding default expression being evaluated and becoming the value of the expression. As with the statement form of try/except, non-matching exceptions will propagate upward. Parentheses are required around the entire expression, unless they would be completely redundant, according to the same rules as generator expressions follow. This guarantees correct interpretation of nested except-expressions, and allows for future expansion of the syntax - see below on multiple except clauses. Note that the current proposal does not allow the exception object to be captured. Where this is needed, the statement form must be used. (See below for discussion and elaboration on this.) This ternary operator would be between lambda and if/else in precedence. Consider this example of a two-level cache:: for key in sequence: x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: f(key))) # do something with x This cannot be rewritten as:: x = lvl1.get(key, lvl2.get(key, f(key))) which, despite being shorter, defeats the purpose of the cache, as it must calculate a default value to pass to get(). The .get() version calculates backwards; the exception-testing version calculates forwards, as would be expected. The nearest useful equivalent would be:: x = lvl1.get(key) or lvl2.get(key) or f(key) which depends on the values being nonzero, as well as depending on the cache object supporting this functionality. Alternative Proposals ===================== Discussion on python-ideas brought up the following syntax suggestions:: value = expr except default if Exception [as e] value = expr except default for Exception [as e] value = expr except default from Exception [as e] value = expr except Exception [as e] return default value = expr except (Exception [as e]: default) value = expr except Exception [as e] try default value = expr except Exception [as e] continue with default value = default except Exception [as e] else expr value = try expr except Exception [as e]: default value = expr except default # Catches anything value = expr except(Exception) default # Catches only the named type(s) value = default if expr raise Exception value = expr or else default if Exception value = expr except Exception [as e] -> default value = expr except Exception [as e] pass default It has also been suggested that a new keyword be created, rather than reusing an existing one. Such proposals fall into the same structure as the last form, but with a different keyword in place of 'pass'. Suggestions include 'then', 'when', and 'use'. Also, in the context of the "default if expr raise Exception" proposal, it was suggested that a new keyword "raises" be used. All forms involving the 'as' capturing clause have been deferred from this proposal in the interests of simplicity, but are preserved in the table above as an accurate record of suggestions. The four forms most supported by this proposal are, in order:: value = (expr except Exception: default) value = (expr except Exception -> default) value = (expr except Exception pass default) value = (expr except Exception then default) All four maintain left-to-right evaluation order: first the base expression, then the exception list, and lastly the default. This is important, as the expressions are evaluated lazily. By comparison, several of the ad-hoc alternatives listed above must (by the nature of functions) evaluate their default values eagerly. The preferred form, using the colon, parallels try/except by using "except exception_list:", and parallels lambda by having "keyword name_list: subexpression"; it also can be read as mapping Exception to the default value, dict-style. Using the arrow introduces a token many programmers will not be familiar with, and which currently has no similar meaning, but is otherwise quite readable. The English word "pass" has a vaguely similar meaning (consider the common usage "pass by value/reference" for function arguments), and "pass" is already a keyword, but as its meaning is distinctly unrelated, this may cause confusion. Using "then" makes sense in English, but this introduces a new keyword to the language - albeit one not in common use, but a new keyword all the same. Left to right evaluation order is extremely important to readability, as it parallels the order most expressions are evaluated. Alternatives such as:: value = (expr except default if Exception) break this, by first evaluating the two ends, and then coming to the middle; while this may not seem terrible (as the exception list will usually be a constant), it does add to the confusion when multiple clauses meet, either with multiple except/if or with the existing if/else, or a combination. Using the preferred order, subexpressions will always be evaluated from left to right, no matter how the syntax is nested. Keeping the existing notation, but shifting the mandatory parentheses, we have the following suggestion:: value = expr except (Exception: default) value = expr except(Exception: default) This is reminiscent of a function call, or a dict initializer. The colon cannot be confused with introducing a suite, but on the other hand, the new syntax guarantees lazy evaluation, which a dict does not. The potential to reduce confusion is considered unjustified by the corresponding potential to increase it. Example usage ============= For each example, an approximately-equivalent statement form is given, to show how the expression will be parsed. These are not always strictly equivalent, but will accomplish the same purpose. It is NOT safe for the interpreter to translate one into the other. A number of these examples are taken directly from the Python standard library, with file names and line numbers correct as of early Feb 2014. Many of these patterns are extremely common. Retrieve an argument, defaulting to None:: cond = (args[1] except IndexError: None) # Lib/pdb.py:803: try: cond = args[1] except IndexError: cond = None Fetch information from the system if available:: pwd = (os.getcwd() except OSError: None) # Lib/tkinter/filedialog.py:210: try: pwd = os.getcwd() except OSError: pwd = None Attempt a translation, falling back on the original:: e.widget = (self._nametowidget(W) except KeyError: W) # Lib/tkinter/__init__.py:1222: try: e.widget = self._nametowidget(W) except KeyError: e.widget = W Read from an iterator, continuing with blank lines once it's exhausted:: line = (readline() except StopIteration: '') # Lib/lib2to3/pgen2/tokenize.py:370: try: line = readline() except StopIteration: line = '' Retrieve platform-specific information (note the DRY improvement); this particular example could be taken further, turning a series of separate assignments into a single large dict initialization:: # sys.abiflags may not be defined on all platforms. _CONFIG_VARS['abiflags'] = (sys.abiflags except AttributeError: '') # Lib/sysconfig.py:529: try: _CONFIG_VARS['abiflags'] = sys.abiflags except AttributeError: # sys.abiflags may not be defined on all platforms. _CONFIG_VARS['abiflags'] = '' Retrieve an indexed item, defaulting to None (similar to dict.get):: def getNamedItem(self, name): return (self._attrs[name] except KeyError: None) # Lib/xml/dom/minidom.py:573: def getNamedItem(self, name): try: return self._attrs[name] except KeyError: return None Translate numbers to names, falling back on the numbers:: g = (grp.getgrnam(tarinfo.gname)[2] except KeyError: tarinfo.gid) u = (pwd.getpwnam(tarinfo.uname)[2] except KeyError: tarinfo.uid) # Lib/tarfile.py:2198: try: g = grp.getgrnam(tarinfo.gname)[2] except KeyError: g = tarinfo.gid try: u = pwd.getpwnam(tarinfo.uname)[2] except KeyError: u = tarinfo.uid Look up an attribute, falling back on a default:: mode = (f.mode except AttributeError: 'rb') # Lib/aifc.py:882: if hasattr(f, 'mode'): mode = f.mode else: mode = 'rb' return (sys._getframe(1) except AttributeError: None) # Lib/inspect.py:1350: return sys._getframe(1) if hasattr(sys, "_getframe") else None Perform some lengthy calculations in EAFP mode, handling division by zero as a sort of sticky NaN:: value = (calculate(x) except ZeroDivisionError: float("nan")) try: value = calculate(x) except ZeroDivisionError: value = float("nan") Calculate the mean of a series of numbers, falling back on zero:: value = (statistics.mean(lst) except statistics.StatisticsError: 0) try: value = statistics.mean(lst) except statistics.StatisticsError: value = 0 Looking up objects in a sparse list of overrides:: (overrides[x] or default except IndexError: default).ping() try: (overrides[x] or default).ping() except IndexError: default.ping() Narrowing of exception-catching scope ------------------------------------- The following examples, taken directly from Python's standard library, demonstrate how the scope of the try/except can be conveniently narrowed. To do this with the statement form of try/except would require a temporary variable, but it's far cleaner as an expression. Lib/ipaddress.py:343:: try: ips.append(ip.ip) except AttributeError: ips.append(ip.network_address) Becomes:: ips.append(ip.ip except AttributeError: ip.network_address) The expression form is nearly equivalent to this:: try: _ = ip.ip except AttributeError: _ = ip.network_address ips.append(_) Lib/tempfile.py:130:: try: dirlist.append(_os.getcwd()) except (AttributeError, OSError): dirlist.append(_os.curdir) Becomes:: dirlist.append(_os.getcwd() except (AttributeError, OSError): _os.curdir) Lib/asyncore.py:264:: try: status.append('%s:%d' % self.addr) except TypeError: status.append(repr(self.addr)) Becomes:: status.append('%s:%d' % self.addr except TypeError: repr(self.addr)) In each case, the narrowed scope of the try/except ensures that an unexpected exception (for instance, AttributeError if "append" were misspelled) does not get caught by the same handler. This is sufficiently unlikely to be reason to break the call out into a separate line (as per the five line example above), but it is a small benefit gained as a side-effect of the conversion. Comparisons with other languages ================================ (With thanks to Andrew Barnert for compiling this section. Note that the examples given here do not reflect the current version of the proposal, and need to be edited.) `Ruby's`__ "begin?rescue?rescue?else?ensure?end" is an expression (potentially with statements inside it). It has the equivalent of an "as" clause, and the equivalent of bare except. And it uses no punctuation or keyword between the bare except/exception class/exception class with as clause and the value. (And yes, it's ambiguous unless you understand Ruby's statement/expression rules.) __ http://www.skorks.com/2009/09/ruby-exceptions-and-exception-handling/ :: x = begin computation() rescue MyException => e default(e) end; x = begin computation() rescue MyException default() end; x = begin computation() rescue default() end; x = begin computation() rescue MyException default() rescue OtherException other() end; In terms of this PEP:: x = computation() except MyException as e default(e) x = computation() except MyException default(e) x = computation() except default(e) x = computation() except MyException default() except OtherException other() `Erlang`__ has a try expression that looks like this __ http://erlang.org/doc/reference_manual/expressions.html#id79284 :: x = try computation() catch MyException:e -> default(e) end; x = try computation() catch MyException:e -> default(e); OtherException:e -> other(e) end; The class and "as" name are mandatory, but you can use "_" for either. There's also an optional "when" guard on each, and a "throw" clause that you can catch, which I won't get into. To handle multiple exceptions, you just separate the clauses with semicolons, which I guess would map to commas in Python. So:: x = try computation() except MyException as e -> default(e) x = try computation() except MyException as e -> default(e), OtherException as e->other_default(e) Erlang also has a "catch" expression, which, despite using the same keyword, is completely different, and you don't want to know about it. The ML family has two different ways of dealing with this, "handle" and "try"; the difference between the two is that "try" pattern-matches the exception, which gives you the effect of multiple except clauses and as clauses. In either form, the handler clause is punctuated by "=>" in some dialects, "->" in others. To avoid confusion, I'll write the function calls in Python style. Here's `SML's`__ "handle" __ http://www.cs.cmu.edu/~rwh/introsml/core/exceptions.htm :: let x = computation() handle MyException => default();; Here's `OCaml's`__ "try" __ http://www2.lib.uchicago.edu/keith/ocaml-class/exceptions.html :: let x = try computation() with MyException explanation -> default(explanation);; let x = try computation() with MyException(e) -> default(e) | MyOtherException() -> other_default() | (e) -> fallback(e);; In terms of this PEP, these would be something like:: x = computation() except MyException => default() x = try computation() except MyException e -> default() x = (try computation() except MyException as e -> default(e) except MyOtherException -> other_default() except BaseException as e -> fallback(e)) Many ML-inspired but not-directly-related languages from academia mix things up, usually using more keywords and fewer symbols. So, the `Oz`__ would map to Python as __ http://mozart.github.io/mozart-v1/doc-1.4.0/tutorial/node5.html :: x = try computation() catch MyException as e then default(e) Many Lisp-derived languages, like `Clojure,`__ implement try/catch as special forms (if you don't know what that means, think function-like macros), so you write, effectively __ http://clojure.org/special_forms#Special%20Forms--(try%20expr*%20catch-clause*%20finally-clause?) :: try(computation(), catch(MyException, explanation, default(explanation))) try(computation(), catch(MyException, explanation, default(explanation)), catch(MyOtherException, explanation, other_default(explanation))) In Common Lisp, this is done with a slightly clunkier `"handler-case" macro,`__ but the basic idea is the same. __ http://clhs.lisp.se/Body/m_hand_1.htm The Lisp style is, surprisingly, used by some languages that don't have macros, like Lua, where `xpcall`__ takes functions. Writing lambdas Python-style instead of Lua-style __ http://www.gammon.com.au/scripts/doc.php?lua=xpcall :: x = xpcall(lambda: expression(), lambda e: default(e)) This actually returns (true, expression()) or (false, default(e)), but I think we can ignore that part. Haskell is actually similar to Lua here (except that it's all done with monads, of course):: x = do catch(lambda: expression(), lambda e: default(e)) You can write a pattern matching expression within the function to decide what to do with it; catching and re-raising exceptions you don't want is cheap enough to be idiomatic. But Haskell infixing makes this nicer:: x = do expression() `catch` lambda: default() x = do expression() `catch` lambda e: default(e) And that makes the parallel between the lambda colon and the except colon in the proposal much more obvious:: x = expression() except Exception: default() x = expression() except Exception as e: default(e) `Tcl`__ has the other half of Lua's xpcall; catch is a function which returns true if an exception was caught, false otherwise, and you get the value out in other ways. And it's all built around the the implicit quote-and-exec that everything in Tcl is based on, making it even harder to describe in Python terms than Lisp macros, but something like __ http://wiki.tcl.tk/902 :: if {[ catch("computation()") "explanation"]} { default(explanation) } `Smalltalk`__ is also somewhat hard to map to Python. The basic version would be __ http://smalltalk.gnu.org/wiki/exceptions :: x := computation() on:MyException do:default() ... but that's basically Smalltalk's passing-arguments-with-colons syntax, not its exception-handling syntax. Deferred sub-proposals ====================== Multiple except clauses ----------------------- An examination of use-cases shows that this is not needed as often as it would be with the statement form, and as its syntax is a point on which consensus has not been reached, the entire feature is deferred. Multiple 'except' keywords could be used, and they will all catch exceptions raised in the original expression (only):: # Will catch any of the listed exceptions thrown by expr; # any exception thrown by a default expression will propagate. value = (expr except Exception1: default1 except Exception2: default2 # ... except ExceptionN: defaultN ) Currently, one of the following forms must be used:: # Will catch an Exception2 thrown by either expr or default1 value = ( (expr except Exception1: default1) except Exception2: default2 ) # Will catch an Exception2 thrown by default1 only value = (expr except Exception1: (default1 except Exception2: default2) ) Listing multiple exception clauses without parentheses is a syntax error (see above), and so a future version of Python is free to add this feature without breaking any existing code. Capturing the exception object ------------------------------ In a try/except block, the use of 'as' to capture the exception object creates a local name binding, and implicitly deletes that binding (to avoid creating a reference loop) in a finally clause. In an expression context, this makes little sense, and a proper sub-scope would be required to safely capture the exception object - something akin to the way a list comprehension is handled. However, CPython currently implements a comprehension's subscope with a nested function call, which has consequences in some contexts such as class definitions, and is therefore unsuitable for this proposal. Should there be, in future, a way to create a true subscope (which could simplify comprehensions, except expressions, with blocks, and possibly more), then this proposal could be revived; until then, its loss is not a great one, as the simple exception handling that is well suited to the expression notation used here is generally concerned only with the type of the exception, and not its value - further analysis below. This syntax would, admittedly, allow a convenient way to capture exceptions in interactive Python; returned values are captured by "_", but exceptions currently are not. This could be spelled:: >>> (expr except Exception as e: e) An examination of the Python standard library shows that, while the use of 'as' is fairly common (occurring in roughly one except clause in five), it is extremely *uncommon* in the cases which could logically be converted into the expression form. Its few uses can simply be left unchanged. Consequently, in the interests of simplicity, the 'as' clause is not included in this proposal. A subsequent Python version can add this without breaking any existing code, as 'as' is already a keyword. One example where this could possibly be useful is Lib/imaplib.py:568:: try: typ, dat = self._simple_command('LOGOUT') except: typ, dat = 'NO', ['%s: %s' % sys.exc_info()[:2]] This could become:: typ, dat = (self._simple_command('LOGOUT') except BaseException as e: ('NO', '%s: %s' % (type(e), e))) Or perhaps some other variation. This is hardly the most compelling use-case, but an intelligent look at this code could tidy it up significantly. In the absence of further examples showing any need of the exception object, I have opted to defer indefinitely the recommendation. Rejected sub-proposals ====================== finally clause -------------- The statement form try... finally or try... except... finally has no logical corresponding expression form. Therefore the finally keyword is not a part of this proposal, in any way. Bare except having different meaning ------------------------------------ With several of the proposed syntaxes, omitting the exception type name would be easy and concise, and would be tempting. For convenience's sake, it might be advantageous to have a bare 'except' clause mean something more useful than "except BaseException". Proposals included having it catch Exception, or some specific set of "common exceptions" (subclasses of a new type called ExpressionError), or have it look for a tuple named ExpressionError in the current scope, with a built-in default such as (ValueError, UnicodeError, AttributeError, EOFError, IOError, OSError, LookupError, NameError, ZeroDivisionError). All of these were rejected, for several reasons. * First and foremost, consistency with the statement form of try/except would be broken. Just as a list comprehension or ternary if expression can be explained by "breaking it out" into its vertical statement form, an expression-except should be able to be explained by a relatively mechanical translation into a near-equivalent statement. Any form of syntax common to both should therefore have the same semantics in each, and above all should not have the subtle difference of catching more in one than the other, as it will tend to attract unnoticed bugs. * Secondly, the set of appropriate exceptions to catch would itself be a huge point of contention. It would be impossible to predict exactly which exceptions would "make sense" to be caught; why bless some of them with convenient syntax and not others? * And finally (this partly because the recommendation was that a bare except should be actively encouraged, once it was reduced to a "reasonable" set of exceptions), any situation where you catch an exception you don't expect to catch is an unnecessary bug magnet. Consequently, the use of a bare 'except' is down to two possibilities: either it is syntactically forbidden in the expression form, or it is permitted with the exact same semantics as in the statement form (namely, that it catch BaseException and be unable to capture it with 'as'). Bare except clauses ------------------- PEP 8 rightly advises against the use of a bare 'except'. While it is syntactically legal in a statement, and for backward compatibility must remain so, there is little value in encouraging its use. In an expression except clause, "except:" is a SyntaxError; use the equivalent long-hand form "except BaseException:" instead. A future version of Python MAY choose to reinstate this, which can be done without breaking compatibility. Parentheses around the except clauses ------------------------------------- Should it be legal to parenthesize the except clauses, separately from the expression that could raise? Example:: value = expr ( except Exception1 [as e]: default1 except Exception2 [as e]: default2 # ... except ExceptionN [as e]: defaultN ) This is more compelling when one or both of the deferred sub-proposals of multiple except clauses and/or exception capturing is included. In their absence, the parentheses would be thus:: value = expr except ExceptionType: default value = expr (except ExceptionType: default) The advantage is minimal, and the potential to confuse a reader into thinking the except clause is separate from the expression, or into thinking this is a function call, makes this non-compelling. The expression can, of course, be parenthesized if desired, as can the default:: value = (expr) except ExceptionType: (default) As the entire expression is now required to be in parentheses (which had not been decided at the time when this was debated), there is less need to delineate this section, and in many cases it would be redundant. Short-hand for "except: pass" ----------------------------- The following was been suggested as a similar short-hand, though not technically an expression:: statement except Exception: pass try: statement except Exception: pass For instance, a common use-case is attempting the removal of a file:: os.unlink(some_file) except OSError: pass There is an equivalent already in Python 3.4, however, in contextlib:: from contextlib import suppress with suppress(OSError): os.unlink(some_file) As this is already a single line (or two with a break after the colon), there is little need of new syntax and a confusion of statement vs expression to achieve this. Common objections ================= Colons always introduce suites ------------------------------ While it is true that many of Python's syntactic elements use the colon to introduce a statement suite (if, while, with, for, etcetera), this is not by any means the sole use of the colon. Currently, Python syntax includes four cases where a colon introduces a subexpression: * dict display - { ... key:value ... } * slice notation - [start:stop:step] * function definition - parameter : annotation * lambda - arg list: return value This proposal simply adds a fifth: * except-expression - exception list: result Style guides and PEP 8 should recommend not having the colon at the end of a wrapped line, which could potentially look like the introduction of a suite, but instead advocate wrapping before the exception list, keeping the colon clearly between two expressions. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From bcannon at gmail.com Wed Mar 12 15:44:01 2014 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 12 Mar 2014 14:44:01 +0000 Subject: [Python-Dev] PEP 463: Exception-catching expressions References: <531FD297.4040408@stoneleaf.us> Message-ID: On Tue Mar 11 2014 at 11:59:23 PM, Chris Angelico wrote: > On Wed, Mar 12, 2014 at 2:20 PM, Ethan Furman wrote: > > I sure hope this is accepted. I could have used it at least a half-dozen > > times in the last week -- which is more often than I would have used the > > ternary-if! :) > > Where do we go from here? I've not made any edits for some time - no > material edits for a good while - how do I request pronouncement? > You start a new email thread asking for pronouncement. Don't be surprised, though, if it is postponed until PyCon since the language summit is 4 weeks away. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Mar 12 16:14:24 2014 From: brett at python.org (Brett Cannon) Date: Wed, 12 Mar 2014 11:14:24 -0400 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Wed, Mar 12, 2014 at 12:10 AM, Chris Angelico wrote: > PEP 463, Exception-catching expressions, is stable and I believe ready > for pronouncement. Drumroll please... > > http://www.python.org/dev/peps/pep-0463/ > > PEP: 463 > Title: Exception-catching expressions > Version: $Revision$ > Last-Modified: $Date$ > Author: Chris Angelico > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 15-Feb-2014 > Python-Version: 3.5 > Post-History: 20-Feb-2014, 16-Feb-2014 > While I'm +0 on the idea, I'm -1 on the syntax; I just don't like having a colon in an expression. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Mar 12 16:16:53 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 13 Mar 2014 02:16:53 +1100 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Thu, Mar 13, 2014 at 2:14 AM, Brett Cannon wrote: > While I'm +0 on the idea, I'm -1 on the syntax; I just don't like having a > colon in an expression. Which is why there are alternatives listed, and the best four of them (including the proposed one) ranked. ChrisA From brett at python.org Wed Mar 12 16:21:22 2014 From: brett at python.org (Brett Cannon) Date: Wed, 12 Mar 2014 11:21:22 -0400 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Wed, Mar 12, 2014 at 11:16 AM, Chris Angelico wrote: > On Thu, Mar 13, 2014 at 2:14 AM, Brett Cannon wrote: > > While I'm +0 on the idea, I'm -1 on the syntax; I just don't like having > a > > colon in an expression. > > Which is why there are alternatives listed, and the best four of them > (including the proposed one) ranked. > `value = (expr except Exception then default)` would be fine with me. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Mar 12 16:44:59 2014 From: barry at python.org (Barry Warsaw) Date: Wed, 12 Mar 2014 11:44:59 -0400 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: <20140312114459.63e5bf6c@anarchist.wooz.org> On Mar 12, 2014, at 11:14 AM, Brett Cannon wrote: >While I'm +0 on the idea, I'm -1 on the syntax; I just don't like having a >colon in an expression. I'm -0 on the idea, mostly be cause it's never occurred to me to even need something like this, and because I don't personally think the existing spellings are deficient. It doesn't bother me that we have dict.get() for example. I'd also find the proposed syntax difficult to read if it was used more commonly than traditional try/except. That being said, the colon really bothers me, despite what is written in "Common objections". True, colons are used in places other than suite introduction, but with exception handling, colons *do* introduce a new suite, so its use here is jarring. There may also be tooling issues (e.g. editors, syntax colorizers, parsers, etc.). Anyway, put it on the language summit track and we can have a fun debate. :) -Barryu From rosuav at gmail.com Wed Mar 12 17:05:21 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 13 Mar 2014 03:05:21 +1100 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: <20140312114459.63e5bf6c@anarchist.wooz.org> References: <20140312114459.63e5bf6c@anarchist.wooz.org> Message-ID: On Thu, Mar 13, 2014 at 2:44 AM, Barry Warsaw wrote: > That being said, the colon really bothers me, despite what is written in > "Common objections". True, colons are used in places other than suite > introduction, but with exception handling, colons *do* introduce a new suite, > so its use here is jarring. There may also be tooling issues (e.g. editors, > syntax colorizers, parsers, etc.). Tooling issues should already have been solved for lambda, but if you don't like the colon, go with one of the other options - Brett expressed support for 'then', which makes very good sense (it does require creating a new keyword, but it's a fairly safe one). ChrisA From p.f.moore at gmail.com Wed Mar 12 17:21:35 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 12 Mar 2014 16:21:35 +0000 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: <20140312114459.63e5bf6c@anarchist.wooz.org> Message-ID: On 12 March 2014 16:05, Chris Angelico wrote: > Tooling issues should already have been solved for lambda, but if you > don't like the colon, go with one of the other options - Brett > expressed support for 'then', which makes very good sense (it does > require creating a new keyword, but it's a fairly safe one). As a data point, Vim's syntax highlighting works mostly fine out of the box for the "top 4" proposed syntaxes. The keyword versions have minor issues - "then" is not highlighted at all and "pass" is highlighted as "pythonStatement" which is slightly jarring. The punctuation-based variants just don't highlight the ":" or "->" which is fine. Making "then" a keyword in Vim's syntax wouldn't be too hard, you'd probably put it in class "pythonException" along with except, try, finally and raise (which seems a bit odd, but that's actually the only place then is used as a keyword, so it's right). Making "pass" highlight as a statement when used standalone or as an expression keyword in an except would be tricky (i.e., I wouldn't know how to do it :-)) The highlighting issue is jarring enough to put me off the "pass" option. Paul From jeremy.kloth at gmail.com Wed Mar 12 17:29:20 2014 From: jeremy.kloth at gmail.com (Jeremy Kloth) Date: Wed, 12 Mar 2014 10:29:20 -0600 Subject: [Python-Dev] Fwd: Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: <20140312114459.63e5bf6c@anarchist.wooz.org> Message-ID: [resending to list] ---------- Forwarded message ---------- From: Jeremy Kloth Date: Wed, Mar 12, 2014 at 10:27 AM Subject: Re: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions To: Chris Angelico On Wed, Mar 12, 2014 at 10:05 AM, Chris Angelico wrote: > Tooling issues should already have been solved for lambda, but if you > don't like the colon, go with one of the other options - Brett > expressed support for 'then', which makes very good sense (it does > require creating a new keyword, but it's a fairly safe one). Just to note, if a token is desired, it could be treated as a pseudo-keyword much like 'as' was prior to 2.6 (or 2.5 with future 'with_statement'). That is, it is only treated as a keyword when parsing the expression defining its use. With that, the list of available "keywords" becomes much broader. -- Jeremy Kloth From gjcarneiro at gmail.com Wed Mar 12 17:41:14 2014 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Wed, 12 Mar 2014 16:41:14 +0000 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On 12 March 2014 15:21, Brett Cannon wrote: > > > > On Wed, Mar 12, 2014 at 11:16 AM, Chris Angelico wrote: > >> On Thu, Mar 13, 2014 at 2:14 AM, Brett Cannon wrote: >> > While I'm +0 on the idea, I'm -1 on the syntax; >> > > I just don't like having a >> > colon in an expression. >> > lambda: is an expression and has a colon in it. It would be weird to use except: without a colon IMHO, because we are already used to write except: with colon everywhere else... Which is why there are alternatives listed, and the best four of them >> (including the proposed one) ranked. >> > > `value = (expr except Exception then default)` would be fine with me. > Do we really need to introduce a new keyword 'then' ? I'm +1 on the original proposal (expr except Exception: default) -- Gustavo J. A. M. Carneiro Gambit Research "The universe is always one step beyond logic." -- Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Mar 12 18:09:24 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 12 Mar 2014 10:09:24 -0700 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Tue, Mar 11, 2014 at 9:10 PM, Chris Angelico wrote: > PEP 463, Exception-catching expressions, is stable and I believe ready > for pronouncement. Drumroll please... > > http://www.python.org/dev/peps/pep-0463/ > > PEP: 463 > Title: Exception-catching expressions > Version: $Revision$ > Last-Modified: $Date$ > Author: Chris Angelico > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 15-Feb-2014 > Python-Version: 3.5 > Post-History: 20-Feb-2014, 16-Feb-2014 > > > Abstract > ======== > > Just as PEP 308 introduced a means of value-based conditions in an > expression, this system allows exception-based conditions to be used > as part of an expression. > I want to reject this PEP. I think the proposed syntax is acceptable given the desired semantics, although it's still a bit jarring. It's probably no worse than the colon used with lambda (which echoes the colon used in a def just like the colon here echoes the one in a try/except) and definitely better than the alternatives listed. But the thing I can't get behind are the motivation and rationale. I don't think that e.g. dict.get() would be unnecessary once we have except expressions, and I disagree with the position that EAFP is better than LBYL, or "generally recommended" by Python. (Where do you get that? From the same sources that are so obsessed with DRY they'd rather introduce a higher-order-function than repeat one line of code? :-) This is probably the most you can get out of me as far as a pronouncement. Given that the language summit is coming up I'd be happy to dive deeper in my reasons for rejecting it there (if there's demand). I do think that (apart from never explaining those dreadful acronyms :-) this was a well-written and well-researched PEP, and I think you've done a great job moderating the discussion, collecting objections, reviewing alternatives, and everything else that is required to turn a heated debate into a PEP. Well done Chris (and everyone who helped), and good luck with your next PEP! -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Wed Mar 12 18:27:09 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 12 Mar 2014 13:27:09 -0400 Subject: [Python-Dev] Why not make frames? [was: Alternative forms [was: PEP 463: Exception-catching expressions]] In-Reply-To: References: <531d2086.b1528c0a.0878.ffffa138@mx.google.com> Message-ID: <20140312172709.84409250CD7@webabinitio.net> On Mon, 10 Mar 2014 14:26:14 +1100, Chris Angelico wrote: > On Mon, Mar 10, 2014 at 1:16 PM, Jim J. Jewett wrote: > > I don't claim that syntax is perfect. I do think it is less flawed > > than the no-parentheses (or external parentheses) versions: > > > > (expr1 except expr3 if expr2) > > expr1 except expr3 if expr2 > > > > because the tigher parentheses correctly indicate that expr2 and expr3 > > should be considered as a (what-to-do-in-case-of-error) group, which > > interacts (as a single unit) with the main expression. > > But it doesn't, really. The entire set of three expressions is a > single unit. You can't break out the bit inside the parens and give > that a name, like you can in most places where something "acts as a > single unit" to interact with something else. (Yes, there are special > cases, like the syntax for constructing slice objects that works only > inside square brackets. And you can't break out a function's > arguments, as a unit, into a single object (the nearest is > *args,**kw). I said most places, and I don't want to add more to the > special-case set.) Actually, function arguments almost aren't a special case any more: >>> import inspect >>> def a(a, b=2): ... print(a, b) ... >>> def b(c, d=3): ... print(c, d) ... >>> sa = inspect.signature(a) >>> print(sa) (a, b=2) >>> ba = sa.bind(1, 2) >>> b(*ba.args, **ba.kwargs) 1 2 Note: I said *almost* :) But the point is that we found that the fact that we couldn't give this thing in parens a name bothersome enough to partially fix it. --David From ethan at stoneleaf.us Wed Mar 12 18:40:59 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 12 Mar 2014 10:40:59 -0700 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: <53209C2B.3010906@stoneleaf.us> On 03/12/2014 10:09 AM, Guido van Rossum wrote: > > I want to reject this PEP. [...] > > But the thing I can't get behind are the motivation and rationale. I don't think that e.g. dict.get() would be > unnecessary once we have except expressions, and I disagree with the position that EAFP is better than LBYL [...] Does this mean a better motivation and rationale may cause you to change your mind? My motivation is for simpler, easier to read code: instead of a full-blown try/except block or a double-lookup into an indexable object I would much rather do: self.date = text_to_date(date, 'mdy') except ValueError: batch_date instead of: try: self.date = text_to_date(date, 'mdy') except ValueError: self.date = batch_date > This is probably the most you can get out of me as far as a pronouncement. Given that the language summit is coming up > I'd be happy to dive deeper in my reasons for rejecting it there (if there's demand). I would like to know, but I won't be at the language summit. I can wait until after, though -- I know time is in especially short supply until then. -- ~Ethan~ From barry at python.org Wed Mar 12 21:14:45 2014 From: barry at python.org (Barry Warsaw) Date: Wed, 12 Mar 2014 16:14:45 -0400 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: <53209C2B.3010906@stoneleaf.us> References: <53209C2B.3010906@stoneleaf.us> Message-ID: <20140312161445.31b7b2db@anarchist.wooz.org> On Mar 12, 2014, at 10:40 AM, Ethan Furman wrote: >Does this mean a better motivation and rationale may cause you to change your >mind? > >My motivation is for simpler, easier to read code: instead of a full-blown >try/except block or a double-lookup into an indexable object I would much >rather do: > > self.date = text_to_date(date, 'mdy') except ValueError: batch_date > >instead of: > > try: > self.date = text_to_date(date, 'mdy') > except ValueError: > self.date = batch_date I have to challenge a couple of assertions here. First, that the exception expression is easier to read. I'm not sure it is. I suspect they will tend to be expressed on very long lines, much like ternary expressions are today. PEP 8 then would have you split them across multiple lines, which introduces parentheses or backslashes to accomplish. For ternary expressions, this can get ugly and for me sometimes draws me back to traditional if/else statements. Other times, ternaries can be worth it, but either way, it does (again for me) reduce the readability argument. Interestingly enough, where ternaries are most useful are in the same situations where I think exception expressions would be most useful, in the setting of a variable or attribute to one of two different values. In both branches of the conditional, the left hand side is the same target, and what differs is the value being assigned to the LHS. Looking over exception examples from a large code base, I find it rare (but not nonexistent) that exceptions are used in the same situation. But that could be because I'm so comfortable with .get() and other helpers. Ternary expressions seem more compelling. But there's definitely some overlap, which is perhaps why the colon seems so jarring to me. So the second assertion is that these will be common enough to warrant changing the language. Again, I'm not personally convinced. I echo Guido though: it is a well written PEP, and I commend Chris for navigating the python-dev/python-ideas gauntlet! That's an accomplishment even if the PEP is rejected. Cheers, -Barry From martin at v.loewis.de Wed Mar 12 21:26:02 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 12 Mar 2014 21:26:02 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <531FD297.4040408@stoneleaf.us> Message-ID: <5320C2DA.5070303@v.loewis.de> Am 12.03.14 04:58, schrieb Chris Angelico: > On Wed, Mar 12, 2014 at 2:20 PM, Ethan Furman wrote: >> I sure hope this is accepted. I could have used it at least a half-dozen >> times in the last week -- which is more often than I would have used the >> ternary-if! :) > > Where do we go from here? I've not made any edits for some time - no > material edits for a good while - how do I request pronouncement? I think you (or someone) first needs to find a BDFL delegate. Traditionally, with syntax changes, there is a good chance that Guido doesn't want to delegate at all, so ask him whether he wants to delegate or not. You should also add a section on the implementation status; it's unclear (to me) whether the proposed implementation actually matches the specification. Regards, Martin From rosuav at gmail.com Wed Mar 12 21:44:45 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 13 Mar 2014 07:44:45 +1100 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Thu, Mar 13, 2014 at 4:09 AM, Guido van Rossum wrote: > But the thing I can't get behind are the motivation and rationale. I don't > think that e.g. dict.get() would be unnecessary once we have except > expressions, and I disagree with the position that EAFP is better than LBYL, > or "generally recommended" by Python. (Where do you get that? From the same > sources that are so obsessed with DRY they'd rather introduce a > higher-order-function than repeat one line of code? :-) Hmm. Not actually sure where I got it from, except [1] that I've heard it periodically advocated as a way to eliminate race conditions in multitasking situations and such (in the same way that stat-followed-by-open is vulnerable, so instead use try-open-except). > This is probably the most you can get out of me as far as a pronouncement. > Given that the language summit is coming up I'd be happy to dive deeper in > my reasons for rejecting it there (if there's demand). Geography prevents me from being there, but if anyone else asks and the answer can be written down, I'm probably not the only one who would read it with interest. > I do think that (apart from never explaining those dreadful acronyms :-) Hmm, I thought for some reason they were axiomatic Python, like "BDFL" and "IDLE". :) > this was a well-written and well-researched PEP, and I think you've done a > great job moderating the discussion, collecting objections, reviewing > alternatives, and everything else that is required to turn a heated debate > into a PEP. Well done Chris (and everyone who helped), and good luck with > your next PEP! Thanks! And yes, thank you to everyone who offered suggestions, kept the intelligent discussion going (it strayed a bit but never turned into a flame war), and especially to everyone who helped with wording and such. I'm only one person, and what seems patently obvious to me isn't always obvious to someone else. I have to say this PEP has been highly successful even without being accepted. The next time anyone asks, they can be pointed to the PEP, with its list of flawed syntax options. (The most common response, by far, was "Nice idea but I don't like the syntax" - for any syntax proposed.) Plus, it taught me how to use the ast module in research. I don't know when that'll come in handy, but Sherlock Holmes was right - all knowledge comes useful! But I'm not going to devote myself to another PEP *quite* yet. Decimal literals is in need of a champion, but I need to do other things, unless someone's going to employ me to champion PEPs full time :) (Hmm. That would be an interesting job title on the resume.) ChrisA [1] Pun intended. Sorry! From rosuav at gmail.com Wed Mar 12 21:49:47 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 13 Mar 2014 07:49:47 +1100 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: <20140312161445.31b7b2db@anarchist.wooz.org> References: <53209C2B.3010906@stoneleaf.us> <20140312161445.31b7b2db@anarchist.wooz.org> Message-ID: On Thu, Mar 13, 2014 at 7:14 AM, Barry Warsaw wrote: > Interestingly enough, where ternaries are most useful are in the same > situations where I think exception expressions would be most useful, in the > setting of a variable or attribute to one of two different values. In both > branches of the conditional, the left hand side is the same target, and what > differs is the value being assigned to the LHS. > > Looking over exception examples from a large code base, I find it rare (but > not nonexistent) that exceptions are used in the same situation. But that > could be because I'm so comfortable with .get() and other helpers. Ternary > expressions seem more compelling. But there's definitely some overlap, which > is perhaps why the colon seems so jarring to me. When you say "looking over", do you mean with the Mk 1 Human Eyeball, or with the script from my research? https://github.com/Rosuav/ExceptExpr/blob/master/find_except_expr.py If you've used the script and have some stats to offer, I'd love to see them. So far, all I have is the stdlib, which isn't necessarily representative. Bear in mind, there isn't a .get() for every situation. You can use hasattr() in place of AttributeError, which isn't quite the same thing, but most of the OSError-raising APIs don't (as far as I know) offer a way to return a default instead of raising. So even if you do generally use dict.get(), there'll likely be some that the script would catch. ChrisA From rosuav at gmail.com Wed Mar 12 21:54:18 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 13 Mar 2014 07:54:18 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5320C2DA.5070303@v.loewis.de> References: <531FD297.4040408@stoneleaf.us> <5320C2DA.5070303@v.loewis.de> Message-ID: On Thu, Mar 13, 2014 at 7:26 AM, "Martin v. L?wis" wrote: > I think you (or someone) first needs to find a BDFL delegate. > Traditionally, with syntax changes, there is a good chance that Guido > doesn't want to delegate at all, so ask him whether he wants to delegate > or not. He beat you to it; the PEP has now been rejected. :) > You should also add a section on the implementation status; it's unclear > (to me) whether the proposed implementation actually matches the > specification. Hmm. I left that sort of comment out, partly because both the proposal and the implementation were fluid. I also had a bit of trouble making everything work, and had to submit some more patches. So I'm not sure what would be counted as the reference implementation. ChrisA From emile at fenx.com Wed Mar 12 22:09:57 2014 From: emile at fenx.com (Emile van Sebille) Date: Wed, 12 Mar 2014 14:09:57 -0700 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On 3/12/2014 1:44 PM, Chris Angelico wrote: > unless someone's going to employ me to champion PEPs full time :) > (Hmm. That would be an interesting job title on the resume.) It's available now -- www.pepboys.com :) Emile From tseaver at palladion.com Wed Mar 12 22:37:49 2014 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 12 Mar 2014 17:37:49 -0400 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: <53209C2B.3010906@stoneleaf.us> <20140312161445.31b7b2db@anarchist.wooz.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/12/2014 04:49 PM, Chris Angelico wrote: > You can use hasattr() in place of AttributeError I use: getattr(subject, attrname, default)? *all the time*. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlMg060ACgkQ+gerLs4ltQ70IgCgi2HFt1DRWHaeIlwgjyf1UJiR 1uEAn0+pf2fS+USmesmXtV9O63jA93hq =fLq7 -----END PGP SIGNATURE----- From rosuav at gmail.com Wed Mar 12 22:49:07 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 13 Mar 2014 08:49:07 +1100 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: <53209C2B.3010906@stoneleaf.us> <20140312161445.31b7b2db@anarchist.wooz.org> Message-ID: On Thu, Mar 13, 2014 at 8:37 AM, Tres Seaver wrote: > On 03/12/2014 04:49 PM, Chris Angelico wrote: >> You can use hasattr() in place of AttributeError > > I use: > > getattr(subject, attrname, default)? > > *all the time*. Umm, yeah, that one. Why did I think hasattr was the only way around that? Anyway, that just means I picked a bad example. There are others where you have to go for the full try/except. ChrisA From ncoghlan at gmail.com Thu Mar 13 07:23:15 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 13 Mar 2014 16:23:15 +1000 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: <53209C2B.3010906@stoneleaf.us> <20140312161445.31b7b2db@anarchist.wooz.org> Message-ID: On 13 March 2014 07:49, Chris Angelico wrote: > > Umm, yeah, that one. Why did I think hasattr was the only way around > that? Anyway, that just means I picked a bad example. There are others > where you have to go for the full try/except. You may have been thinking of item lookup on sequences. Attribute lookup can switch to getattr (at the cost of becoming more opaque to the compiler and static analysers), and mappings can fall back to get(), but there's no such alternative for sequences. Anyway - exemplary work on the PEP, even though Guido remained unconvinced. Not only does it become a useful resource if anyone else comes forward with a similar proposal, it becomes something we can point to and perhaps revisit next time someone asks for implicit exception handling to be added to yet another API :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Thu Mar 13 10:15:22 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 13 Mar 2014 10:15:22 +0100 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: Hi, 2014-03-12 18:09 GMT+01:00 Guido van Rossum : > I want to reject this PEP. (...) > this was a well-written and well-researched PEP, and I think you've done a > great job moderating the discussion, collecting objections, reviewing > alternatives, and everything else that is required to turn a heated debate > into a PEP. Well done Chris (and everyone who helped), and good luck with > your next PEP! I spend many weeks to work (write the PEP *and* implement them) on the following PEPs and Guido rejected them: - PEP 416: Add a frozendict builtin type - PEP 410: Use decimal.Decimal type for timestamps It's hard to accept that a wonderful idea at a first look is not a good idea. Some months later, I now agree and see issues of my PEPs. The PEP process ensures that the Python "language" (+ stdlib) keeps consistent and well designed. Even if a PEP is rejected, it becomes the best reference if someone requests the same or a similar feature some months or years later. Rejected PEPs explain almost how the Python language was designed. For thanks Chris, and I hope that it's not too hard for you to accept the fact that your "inline try/except" idea is maybe not as good as you expected :-) Victor From rosuav at gmail.com Thu Mar 13 11:10:42 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 13 Mar 2014 21:10:42 +1100 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Thu, Mar 13, 2014 at 8:15 PM, Victor Stinner wrote: > Even if a PEP is rejected, it becomes the best reference if someone > requests the same or a similar feature some months or years later. > Rejected PEPs explain almost how the Python language was designed. > > For thanks Chris, and I hope that it's not too hard for you to accept > the fact that your "inline try/except" idea is maybe not as good as > you expected :-) One of the original goals of the PEP was to get everything written down. I actually didn't have any sort of strong opinion on the matter until part way in; effectively, I had to first convince myself, then try to convince others :) So it's not hard to accept that the idea isn't such a good one. ChrisA From steve at pearwood.info Thu Mar 13 11:26:11 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 13 Mar 2014 21:26:11 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <531FD297.4040408@stoneleaf.us> <5320C2DA.5070303@v.loewis.de> Message-ID: <20140313102610.GM12595@ando> On Thu, Mar 13, 2014 at 07:54:18AM +1100, Chris Angelico wrote: > On Thu, Mar 13, 2014 at 7:26 AM, "Martin v. L?wis" wrote: > > I think you (or someone) first needs to find a BDFL delegate. > > Traditionally, with syntax changes, there is a good chance that Guido > > doesn't want to delegate at all, so ask him whether he wants to delegate > > or not. > > He beat you to it; the PEP has now been rejected. :) Guido said he *wanted* to reject it, and that there would be no more pronouncement until after the language summit. I think you should wait until after the summit before declaring it categorically rejected, perhaps somebody will persuade Guido to change his mind :-) But either way, thanks for tackling this, the threads about syntax were monsterous and I ran out of steam trying to keep up with it all long ago. -- Steven From christian at python.org Thu Mar 13 11:49:03 2014 From: christian at python.org (Christian Heimes) Date: Thu, 13 Mar 2014 11:49:03 +0100 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: References: <20140311020553.16E23250C31@webabinitio.net> Message-ID: <53218D1F.3030809@python.org> On 11.03.2014 13:18, Victor Stinner wrote: > Hi, > > Thanks David! I added a summary of security improvements: > http://docs.python.org/dev/whatsnew/3.4.html#summary-release-highlights > > Can someone please review it? Don't hesitate to modify the text > directly. Check also if the summary is complete. Thanks a lot David and Victor! The list of security improvements is missing one, maybe two points that are IMHO relevant: * All stdlib modules now support server cert verification including hostname matching and CRL. * http://bugs.python.org/issue16499 isolated mode is a security improvement, too. Should the section or Whats New in general mention that Python builds without compiler warnings on most platforms and that we aim for zero warnings on all supported platforms and compilers? And there is the point with Coverity Scan. We have reached zero defects about half a year ago and fixed all new defects in a matter of days. I'll try to keep the defect rate down to zero in the future, too. The tool has helped me to identify a bunch of security-relevant issues like buffer overflows, invalid casts and more. It's something worth mentioning. But I don't want it to sound like an advert... Suggestions? Christian From christian at python.org Thu Mar 13 11:49:03 2014 From: christian at python.org (Christian Heimes) Date: Thu, 13 Mar 2014 11:49:03 +0100 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: References: <20140311020553.16E23250C31@webabinitio.net> Message-ID: <53218D1F.3030809@python.org> On 11.03.2014 13:18, Victor Stinner wrote: > Hi, > > Thanks David! I added a summary of security improvements: > http://docs.python.org/dev/whatsnew/3.4.html#summary-release-highlights > > Can someone please review it? Don't hesitate to modify the text > directly. Check also if the summary is complete. Thanks a lot David and Victor! The list of security improvements is missing one, maybe two points that are IMHO relevant: * All stdlib modules now support server cert verification including hostname matching and CRL. * http://bugs.python.org/issue16499 isolated mode is a security improvement, too. Should the section or Whats New in general mention that Python builds without compiler warnings on most platforms and that we aim for zero warnings on all supported platforms and compilers? And there is the point with Coverity Scan. We have reached zero defects about half a year ago and fixed all new defects in a matter of days. I'll try to keep the defect rate down to zero in the future, too. The tool has helped me to identify a bunch of security-relevant issues like buffer overflows, invalid casts and more. It's something worth mentioning. But I don't want it to sound like an advert... Suggestions? Christian From stephen at xemacs.org Thu Mar 13 12:34:44 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 13 Mar 2014 20:34:44 +0900 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: <53218D1F.3030809@python.org> References: <20140311020553.16E23250C31@webabinitio.net> <53218D1F.3030809@python.org> Message-ID: <877g7ymiwb.fsf@uwakimon.sk.tsukuba.ac.jp> Christian Heimes writes: > But I don't want it to sound like an advert... Suggestions? Not to worry. It *can't* be an advert -- it's all true, and there are no irrelevant half-naked glistening bodies. (Former newts in the pond don't count.) Seriously, while "expect a clean build" is not "news" for Python, it's useful to state that we're at zero warnings nearly across the board, including Coverity. Accompanied by an open invitation for reports to the contrary, that's hardly like a commercial. Of course, following that up with "wink wink, nudge nudge, say no more say no more, eh?" would ruin the effect! From solipsis at pitrou.net Thu Mar 13 14:08:42 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 13 Mar 2014 14:08:42 +0100 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: <53218D1F.3030809@python.org> References: <20140311020553.16E23250C31@webabinitio.net> <53218D1F.3030809@python.org> Message-ID: Le 13/03/2014 11:49, Christian Heimes a ?crit : > Thanks a lot David and Victor! The list of security improvements is > missing one, maybe two points that are IMHO relevant: > > * All stdlib modules now support server cert verification including > hostname matching and CRL. CRL? really? I don't remember us doing automatic CRL downloads. > And there is the point with Coverity Scan. We have reached zero defects > about half a year ago and fixed all new defects in a matter of days. > I'll try to keep the defect rate down to zero in the future, too. The > tool has helped me to identify a bunch of security-relevant issues like > buffer overflows, invalid casts and more. It's something worth > mentioning. But I don't want it to sound like an advert... Suggestions? I don't think it should be mentioned at all. General code quality improvements are a given in any release, the fact that the issues were detected by Coverity rather than human scrutiny is a non-information (except as advertising for Coverity). Regards Antoine. From barry at python.org Thu Mar 13 14:28:03 2014 From: barry at python.org (Barry Warsaw) Date: Thu, 13 Mar 2014 09:28:03 -0400 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: <20140313092803.4204435c@anarchist.wooz.org> On Mar 13, 2014, at 10:15 AM, Victor Stinner wrote: >It's hard to accept that a wonderful idea at a first look is not a >good idea. Some months later, I now agree and see issues of my PEPs. >The PEP process ensures that the Python "language" (+ stdlib) keeps >consistent and well designed. > >Even if a PEP is rejected, it becomes the best reference if someone >requests the same or a similar feature some months or years later. >Rejected PEPs explain almost how the Python language was designed. > >For thanks Chris, and I hope that it's not too hard for you to accept >the fact that your "inline try/except" idea is maybe not as good as >you expected :-) Don't worry, there's always Python 4 . http://legacy.python.org/dev/peps/pep-0274/ -Barry From victor.stinner at gmail.com Thu Mar 13 14:57:41 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 13 Mar 2014 14:57:41 +0100 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: <53218D1F.3030809@python.org> References: <20140311020553.16E23250C31@webabinitio.net> <53218D1F.3030809@python.org> Message-ID: 2014-03-13 11:49 GMT+01:00 Christian Heimes : > * All stdlib modules now support server cert verification including > hostname matching and CRL. > > * http://bugs.python.org/issue16499 isolated mode is a security > improvement, too. Ok, I added these two items. Antoine wrote: > CRL? really? I don't remember us doing automatic CRL downloads. It's just the "support", nothing is automatic. I understood that you *can* load CRL and ask for CRL validation, but it must be done explicitly. There is a function to retrieve system CRLs on Windows. Victor From p.f.moore at gmail.com Thu Mar 13 15:06:39 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 13 Mar 2014 14:06:39 +0000 Subject: [Python-Dev] PEP URLs Message-ID: On 13 March 2014 13:28, Barry Warsaw wrote: > > http://legacy.python.org/dev/peps/pep-0274/ Are these "legacy.python.org" URLs the canonical locations for PEPs these days? The main page redirects there, but I'd have hoped there would be a www.python.org location... (Just checking before I update my PEP lookup bookmarklet :-)) Paul From donald at stufft.io Thu Mar 13 15:10:36 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 13 Mar 2014 10:10:36 -0400 Subject: [Python-Dev] PEP URLs In-Reply-To: References: Message-ID: On Mar 13, 2014, at 10:06 AM, Paul Moore wrote: > On 13 March 2014 13:28, Barry Warsaw wrote: >> >> http://legacy.python.org/dev/peps/pep-0274/ > > Are these "legacy.python.org" URLs the canonical locations for PEPs > these days? The main page redirects there, but I'd have hoped there > would be a www.python.org location... > > (Just checking before I update my PEP lookup bookmarklet :-)) > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io AFAIK the www.python.org PEP stuff just isn?t done yet, and the legacy redirect is a temporary stopgap. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From martin at v.loewis.de Thu Mar 13 15:44:39 2014 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 13 Mar 2014 15:44:39 +0100 Subject: [Python-Dev] PEP URLs In-Reply-To: References: Message-ID: <20140313154439.Horde.POXr_YfHbjj3SUEACHcYMw1@webmail.df.eu> Quoting Paul Moore : > On 13 March 2014 13:28, Barry Warsaw wrote: >> >> http://legacy.python.org/dev/peps/pep-0274/ > > Are these "legacy.python.org" URLs the canonical locations for PEPs > these days? The main page redirects there, but I'd have hoped there > would be a www.python.org location... http://www.python.org/dev/peps/pep-0274/ continues to work (as I thought you said yourself), so there *is* a www.python.org location. Regards, Martin From larry at hastings.org Thu Mar 13 16:16:35 2014 From: larry at hastings.org (Larry Hastings) Date: Thu, 13 Mar 2014 08:16:35 -0700 Subject: [Python-Dev] Any non-3.4.0 docs in "default"? Message-ID: <5321CBD3.2020004@hastings.org> I'm strongly considering literally copying over all the content in Doc/ from the default branch to the 3.4 branch. Not cherry-picking doc changes, simply copying everything en masse. I have two questions for the core devs. 1) Sound good? 2) Do you know of anything in Doc/ that shouldn't ship in 3.4.0? I know about the change to the build--3.4.1 won't auto-download Sphinx anymore. Is there anything else like that? Thanks, //arry/ p.s. I expect to tag 3.4.0 final somewhere between 48 and 72 hours from now. Hopefully the light at the end of the tunnel isn't an oncoming train! -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Mar 13 16:35:29 2014 From: brett at python.org (Brett Cannon) Date: Thu, 13 Mar 2014 11:35:29 -0400 Subject: [Python-Dev] Any non-3.4.0 docs in "default"? In-Reply-To: <5321CBD3.2020004@hastings.org> References: <5321CBD3.2020004@hastings.org> Message-ID: On Thu, Mar 13, 2014 at 11:16 AM, Larry Hastings wrote: > > > I'm strongly considering literally copying over all the content in Doc/ > from the default branch to the 3.4 branch. Not cherry-picking doc changes, > simply copying everything en masse. I have two questions for the core devs. > > 1) Sound good? > As long as no one answers "I know of something to not ship" to question #2, then it SGTM. -Brett > > 2) Do you know of anything in Doc/ that shouldn't ship in 3.4.0? I know > about the change to the build--3.4.1 won't auto-download Sphinx anymore. > Is there anything else like that? > > Thanks, > > > */arry* > > p.s. I expect to tag 3.4.0 final somewhere between 48 and 72 hours from > now. Hopefully the light at the end of the tunnel isn't an oncoming train! > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Thu Mar 13 16:31:00 2014 From: Nikolaus at rath.org (Nikolaus Rath) Date: Thu, 13 Mar 2014 08:31:00 -0700 Subject: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions In-Reply-To: (Tres Seaver's message of "Wed, 12 Mar 2014 17:37:49 -0400") References: <53209C2B.3010906@stoneleaf.us> <20140312161445.31b7b2db@anarchist.wooz.org> Message-ID: <878use86a3.fsf@rath.org> Tres Seaver writes: > On 03/12/2014 04:49 PM, Chris Angelico wrote: >> You can use hasattr() in place of AttributeError > > I use: > > getattr(subject, attrname, default)? > > *all the time*. In my opinion that's almost as ugly, because it still forces you to specify the attribute name as a string rather than an identifier. Best, Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ?Time flies like an arrow, fruit flies like a Banana.? From storchaka at gmail.com Thu Mar 13 17:18:24 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 13 Mar 2014 18:18:24 +0200 Subject: [Python-Dev] Any non-3.4.0 docs in "default"? In-Reply-To: <5321CBD3.2020004@hastings.org> References: <5321CBD3.2020004@hastings.org> Message-ID: 13.03.14 17:16, Larry Hastings ???????(??): > I'm strongly considering literally copying over all the content in Doc/ > from the default branch to the 3.4 branch. Not cherry-picking doc > changes, simply copying everything en masse. I have two questions for > the core devs. > > 1) Sound good? > > 2) Do you know of anything in Doc/ that shouldn't ship in 3.4.0? I know > about the change to the build--3.4.1 won't auto-download Sphinx > anymore. Is there anything else like that? David fixed not only Doc/*, but some entities in Misc/NEWS too. And may be he fixed some docstrings. From solipsis at pitrou.net Thu Mar 13 17:50:04 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 13 Mar 2014 17:50:04 +0100 Subject: [Python-Dev] Any non-3.4.0 docs in "default"? In-Reply-To: References: <5321CBD3.2020004@hastings.org> Message-ID: Le 13/03/2014 17:18, Serhiy Storchaka a ?crit : > 13.03.14 17:16, Larry Hastings ???????(??): >> I'm strongly considering literally copying over all the content in Doc/ >> from the default branch to the 3.4 branch. Not cherry-picking doc >> changes, simply copying everything en masse. I have two questions for >> the core devs. >> >> 1) Sound good? >> >> 2) Do you know of anything in Doc/ that shouldn't ship in 3.4.0? I know >> about the change to the build--3.4.1 won't auto-download Sphinx >> anymore. Is there anything else like that? > > David fixed not only Doc/*, but some entities in Misc/NEWS too. And may > be he fixed some docstrings. None of this sounds release-critical at all. Regards Antoine From g.brandl at gmx.net Thu Mar 13 18:33:51 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 13 Mar 2014 18:33:51 +0100 Subject: [Python-Dev] Any non-3.4.0 docs in "default"? In-Reply-To: References: <5321CBD3.2020004@hastings.org> Message-ID: Am 13.03.2014 17:50, schrieb Antoine Pitrou: > Le 13/03/2014 17:18, Serhiy Storchaka a ?crit : >> 13.03.14 17:16, Larry Hastings ???????(??): >>> I'm strongly considering literally copying over all the content in Doc/ >>> from the default branch to the 3.4 branch. Not cherry-picking doc >>> changes, simply copying everything en masse. I have two questions for >>> the core devs. >>> >>> 1) Sound good? >>> >>> 2) Do you know of anything in Doc/ that shouldn't ship in 3.4.0? I know >>> about the change to the build--3.4.1 won't auto-download Sphinx >>> anymore. Is there anything else like that? There's a few follow-up changes to that. But as long as you restrict yourself to .rst files you will omit those. >> David fixed not only Doc/*, but some entities in Misc/NEWS too. And may >> be he fixed some docstrings. > > None of this sounds release-critical at all. Yep. Georg From g.brandl at gmx.net Thu Mar 13 18:36:03 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 13 Mar 2014 18:36:03 +0100 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: <877g7ymiwb.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20140311020553.16E23250C31@webabinitio.net> <53218D1F.3030809@python.org> <877g7ymiwb.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Am 13.03.2014 12:34, schrieb Stephen J. Turnbull: > Christian Heimes writes: > > > But I don't want it to sound like an advert... Suggestions? > > Not to worry. It *can't* be an advert -- it's all true, and there are > no irrelevant half-naked glistening bodies. (Former newts in the pond > don't count.) > > Seriously, while "expect a clean build" is not "news" for Python, it's > useful to state that we're at zero warnings nearly across the board, > including Coverity. Accompanied by an open invitation for reports to > the contrary, that's hardly like a commercial. I think Chris meant he'd sound like an ad for Coverity. Georg From storchaka at gmail.com Thu Mar 13 18:54:13 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 13 Mar 2014 19:54:13 +0200 Subject: [Python-Dev] Any non-3.4.0 docs in "default"? In-Reply-To: <5321CBD3.2020004@hastings.org> References: <5321CBD3.2020004@hastings.org> Message-ID: 13.03.14 17:16, Larry Hastings ???????(??): > I'm strongly considering literally copying over all the content in Doc/ > from the default branch to the 3.4 branch. Not cherry-picking doc > changes, simply copying everything en masse. I have two questions for > the core devs. > > 1) Sound good? > > 2) Do you know of anything in Doc/ that shouldn't ship in 3.4.0? I know > about the change to the build--3.4.1 won't auto-download Sphinx > anymore. Is there anything else like that? d843a1caba78 adds aliases for cp273 codec and documents it. ea827c809765 updates pkgutil.find_loader() and documents this. b637064cc696 improves the pickling of enums and updates the documentation. dbf13a7d3987 adds debugging feasibility to asyncio and documents this. a63327162063 mangles __parameters in __annotations__ dict properly and mention this in whatsnews. d50a95fab002 adds a format() method to tracemalloc.Traceback and documents it. eef7899ea7ab makes Sphinx specific changes. 95de14f53cf5 makes asyncio.BaseEventLoop.sock_connect() to raise an error if the address is not resolved and document this. If this changesets are not in rc3, corresponding Doc/* changes should be reverted. From solipsis at pitrou.net Thu Mar 13 19:46:12 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 13 Mar 2014 19:46:12 +0100 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... References: <20140311020553.16E23250C31@webabinitio.net> <53218D1F.3030809@python.org> Message-ID: <20140313194612.6ec32e6f@fsol> On Thu, 13 Mar 2014 14:57:41 +0100 Victor Stinner wrote: > 2014-03-13 11:49 GMT+01:00 Christian Heimes : > > * All stdlib modules now support server cert verification including > > hostname matching and CRL. > > > > * http://bugs.python.org/issue16499 isolated mode is a security > > improvement, too. > > Ok, I added these two items. > > Antoine wrote: > > CRL? really? I don't remember us doing automatic CRL downloads. > > It's just the "support", nothing is automatic. I understood that you > *can* load CRL and ask for CRL validation, but it must be done > explicitly. There is a function to retrieve system CRLs on Windows. Then you should perhaps make your phrasing more explicit, because people may wrongly assume that CRL checking will be done automatically (IMHO). (especially since hostname checking, AFAIK, *is* automatic now) Regards Antoine. From AHettinger at Prominic.NET Thu Mar 13 21:23:14 2014 From: AHettinger at Prominic.NET (Andrew M. Hettinger) Date: Thu, 13 Mar 2014 15:23:14 -0500 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: <20140313194612.6ec32e6f@fsol> References: <20140311020553.16E23250C31@webabinitio.net> <53218D1F.3030809@python.org> <20140313194612.6ec32e6f@fsol> Message-ID: Antoine Pitrou wrote on 03/13/2014 01:46:12 PM: > On Thu, 13 Mar 2014 14:57:41 +0100 > Victor Stinner wrote: > > 2014-03-13 11:49 GMT+01:00 Christian Heimes : > > > * All stdlib modules now support server cert verification including > > > hostname matching and CRL. > > > > > > * http://bugs.python.org/issue16499 isolated mode is a security > > > improvement, too. > > > > Ok, I added these two items. > > > > Antoine wrote: > > > CRL? really? I don't remember us doing automatic CRL downloads. > > > > It's just the "support", nothing is automatic. I understood that you > > *can* load CRL and ask for CRL validation, but it must be done > > explicitly. There is a function to retrieve system CRLs on Windows. > > Then you should perhaps make your phrasing more explicit, because > people may wrongly assume that CRL checking will be done automatically > (IMHO). > > (especially since hostname checking, AFAIK, *is* automatic now) Sorry if I'm out of line on my first post to this list, but I've been using the ssl module in 3.4 some lately (indeed, I have an open RFE on it for 3.5). While hostname checking can be done automatically, it's not the default (and if it will even work at all depends on the version of openssl installed). I suppose I could see it changed to read: * All stdlib modules now support server cert verification including hostname matching and CRL verification (but not automatic download). Of course, the reality is, using the ssl module requires a vary careful attention to detail, and probably always will. If a programmer is just going by the "What's New" section for security related code, I'm not sure there's much you can to to save them. ;p > > Regards > > Antoine. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Thu Mar 13 23:24:12 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 13 Mar 2014 18:24:12 -0400 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: References: <20140311020553.16E23250C31@webabinitio.net> <53218D1F.3030809@python.org> <20140313194612.6ec32e6f@fsol> Message-ID: <20140313222412.F31E5250D05@webabinitio.net> On Thu, 13 Mar 2014 15:23:14 -0500, "Andrew M. Hettinger" wrote: > Antoine Pitrou wrote on 03/13/2014 01:46:12 PM: > > On Thu, 13 Mar 2014 14:57:41 +0100 > > Victor Stinner wrote: > > > 2014-03-13 11:49 GMT+01:00 Christian Heimes : > > > > * All stdlib modules now support server cert verification including > > > > hostname matching and CRL. > > > > > > > > * http://bugs.python.org/issue16499 isolated mode is a security > > > > improvement, too. > > > > > > Ok, I added these two items. > > > > > > Antoine wrote: > > > > CRL? really? I don't remember us doing automatic CRL downloads. > > > > > > It's just the "support", nothing is automatic. I understood that you > > > *can* load CRL and ask for CRL validation, but it must be done > > > explicitly. There is a function to retrieve system CRLs on Windows. > > > > Then you should perhaps make your phrasing more explicit, because > > people may wrongly assume that CRL checking will be done automatically > > (IMHO). > > > > (especially since hostname checking, AFAIK, *is* automatic now) > Sorry if I'm out of line on my first post to this list, but I've been using > the ssl module in 3.4 some lately (indeed, I have an open RFE on it for > 3.5). > > While hostname checking can be done automatically, it's not the default > (and if it will even work at all depends on the version of openssl > installed). > I suppose I could see it changed to read: > > * All stdlib modules now support server cert verification including > hostname matching and CRL verification (but not automatic download). > > Of course, the reality is, using the ssl module requires a vary careful > attention to detail, and probably always will. If a programmer is just > going by the "What's New" section for security related code, I'm not sure > there's much you can to to save them. ;p I opened issue 20913 to request that some sort of "best practices" documentation be added either to the SSL docs or as a separate chapter in the library reference. I do not feel competent to adjust the content of the security entries in whatsnew, so I have not. If someone wants to propose a patch or make an edit before Larry copies the files, please feel free. --David From victor.stinner at gmail.com Fri Mar 14 00:54:38 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 14 Mar 2014 00:54:38 +0100 Subject: [Python-Dev] tracemallocqt: GUI to analyze tracemalloc snapshots Message-ID: Hi, To prepare my conference on tracemalloc for Pycon Montr?al next month, I wrote a GUI to analyze tracemalloc snapshots: "tracemallocqt". https://bitbucket.org/haypo/tracemallocqt It looks like that: http://www.haypocalc.com/tmp/tracemallocqt_python34.png I'm looking for testers and feedback on the GUI. You may be curious to see which parts of Python is allocating the most memory, or maybe test your favorite application to check if it leaks memory or how its memory footprint can be reduced. To run tracemallocqt, you have to install PySide and get tracemalloc snapshots. It looks like most Linux distributions provide PySide packages. To take snapshots, see tracemalloc documentation: http://docs.python.org/dev/library/tracemalloc.html In short, use Python 3.4 with "-X tracemalloc=25" command line option and then call: --- import pickle, tracemalloc snap = tracemalloc.take_snapshot() with open("dump.pickle", "wb") as fp: pickle.dump(snap, fp, 2) snap = None --- It's more fun with at least 2 snapshots to compare them ;-) I force pickle version 2 because tracemallocqt currently only works on Python 2. (Mostly because I don't see how to get PySide for Python 3 on Fedora 20.) Not all applications work on Python 3.4 right now, you can use the pytracemalloc project which is tracemalloc for Python 2.5-3.3. Sorry, you have to patch Python and recompile it. Here are instructions to install a patched Python 2.7 with tracemalloc: http://pytracemalloc.readthedocs.org/#manual-installation If you don't want to take snapshot, you can compare these two files: http://www.haypocalc.com/tmp/python34_start.pickle http://www.haypocalc.com/tmp/python34_after_import.pickle The source browser will not find Python source code, because the snapshots contain absolute paths. Victor From tjreedy at udel.edu Fri Mar 14 02:29:45 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 13 Mar 2014 21:29:45 -0400 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: <877g7ymiwb.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20140311020553.16E23250C31@webabinitio.net> <53218D1F.3030809@python.org> <877g7ymiwb.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 3/13/2014 7:34 AM, Stephen J. Turnbull wrote: > Christian Heimes writes: > > > But I don't want it to sound like an advert... Suggestions? > > Not to worry. It *can't* be an advert -- it's all true, and there are > no irrelevant half-naked glistening bodies. (Former newts in the pond > don't count.) > > Seriously, while "expect a clean build" is not "news" for Python, It is for a Windows repository build. I just rebuilt: 3.3 gives lots of warning from multiple files; 3.4 none. > Accompanied by an open invitation for reports to > the contrary, that's hardly like a commercial. Now that no warnings is a serious goal for 3.4+, I will report them should they recur. -- Terry Jan Reedy From stephen at xemacs.org Fri Mar 14 03:13:05 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 14 Mar 2014 11:13:05 +0900 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: References: <20140311020553.16E23250C31@webabinitio.net> <53218D1F.3030809@python.org> <877g7ymiwb.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <874n31mssu.fsf@uwakimon.sk.tsukuba.ac.jp> Georg Brandl writes: > I think Chris meant he'd sound like an ad for Coverity. Oh, that. IIRC, we pay them nothing for the service. I know that they do it in part for commercial reasons and that it undoubtedly doesn't cost them a lot, but nonetheless it is a benefit that wouldn't hurt for acknowledgment. As compared to other volunteer contributors, OK, maybe What's New is a bit prominent. But the real reason is that a clean Coverity sweep for software is like a Consumer Reports 5-star safety rating for automobiles. It's a a brand that signals quality, and an assessment independent of the developing organization. And like the 5-star rating, it cost effort (I gather much of it from Chris himself) to keep that clean bill of health from release to release. It may not be "new," exactly, but like the 5-star rating, I think our users want to know that it's renewed with each release. From brian at python.org Fri Mar 14 06:59:59 2014 From: brian at python.org (Brian Curtin) Date: Fri, 14 Mar 2014 00:59:59 -0500 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: References: <20140311020553.16E23250C31@webabinitio.net> <53218D1F.3030809@python.org> <877g7ymiwb.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Thu, Mar 13, 2014 at 8:29 PM, Terry Reedy wrote: > On 3/13/2014 7:34 AM, Stephen J. Turnbull wrote: >> >> Christian Heimes writes: >> >> > But I don't want it to sound like an advert... Suggestions? >> >> Not to worry. It *can't* be an advert -- it's all true, and there are >> no irrelevant half-naked glistening bodies. (Former newts in the pond >> don't count.) >> >> Seriously, while "expect a clean build" is not "news" for Python, > > > It is for a Windows repository build. I just rebuilt: 3.3 gives lots of > warning from multiple files; 3.4 none. > > >> Accompanied by an open invitation for reports to >> the contrary, that's hardly like a commercial. > > > Now that no warnings is a serious goal for 3.4+, I will report them should > they recur. If we're at no warnings, and no warnings is a serious goal, warnings should be errors. From ncoghlan at gmail.com Fri Mar 14 07:48:29 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Mar 2014 16:48:29 +1000 Subject: [Python-Dev] PEP URLs In-Reply-To: References: Message-ID: On 14 March 2014 00:10, Donald Stufft wrote: > AFAIK the www.python.org PEP stuff just isn't done yet, and the legacy redirect is > a temporary stopgap. As Donald says, the official location hasn't changed, but the PEP repo based build process hasn't been integrated into the new site yet, so they're hitting the redirect to the legacy site. I opened https://github.com/python/pythondotorg/issues/297 to ask for an ETA on when we can expect them to be fully integrated. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Fri Mar 14 08:31:47 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 14 Mar 2014 09:31:47 +0200 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: References: <20140311020553.16E23250C31@webabinitio.net> <53218D1F.3030809@python.org> <877g7ymiwb.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: 14.03.14 07:59, Brian Curtin ???????(??): > On Thu, Mar 13, 2014 at 8:29 PM, Terry Reedy wrote: >> Now that no warnings is a serious goal for 3.4+, I will report them should >> they recur. > > If we're at no warnings, and no warnings is a serious goal, warnings > should be errors. Sources still are not C89-clean and gcc -std=c89 emits warnings/errors. From ncoghlan at gmail.com Fri Mar 14 08:39:48 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Mar 2014 17:39:48 +1000 Subject: [Python-Dev] tracemallocqt: GUI to analyze tracemalloc snapshots In-Reply-To: References: Message-ID: On 14 March 2014 09:54, Victor Stinner wrote: > I force pickle version 2 because tracemallocqt currently only works on > Python 2. (Mostly because I don't see how to get PySide for Python 3 > on Fedora 20.) The Fedora PySide package needs some spec file wrangling to actually build the Python 3 version of the packages: https://bugzilla.redhat.com/show_bug.cgi?id=1045780 (Some guidelines on handling that problem are at https://fedoraproject.org/wiki/Packaging:Python if anyone familiar with RPM and keen to see that fixed feels like taking a shot at the problem) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From zachary.ware+pydev at gmail.com Fri Mar 14 14:35:29 2014 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Fri, 14 Mar 2014 08:35:29 -0500 Subject: [Python-Dev] Whats New in 3.4 is pretty much done... In-Reply-To: References: <20140311020553.16E23250C31@webabinitio.net> <53218D1F.3030809@python.org> <877g7ymiwb.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Fri, Mar 14, 2014 at 12:59 AM, Brian Curtin wrote: > On Thu, Mar 13, 2014 at 8:29 PM, Terry Reedy wrote: >> Now that no warnings is a serious goal for 3.4+, I will report them should >> they recur. > > If we're at no warnings, and no warnings is a serious goal, warnings > should be errors. I'm planning to get that done for 3.5, at least on 32bit Windows. I haven't gotten an issue opened for it yet, though. -- Zach From barry at python.org Fri Mar 14 15:04:54 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 14 Mar 2014 10:04:54 -0400 Subject: [Python-Dev] PEP URLs In-Reply-To: References: Message-ID: <20140314100454.14ca0305@limelight.wooz.org> On Mar 14, 2014, at 04:48 PM, Nick Coghlan wrote: >I opened https://github.com/python/pythondotorg/issues/297 to ask for >an ETA on when we can expect them to be fully integrated. Thanks Nick. On the bug I suggest creating peps.python.org. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From martin at v.loewis.de Fri Mar 14 15:54:58 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 14 Mar 2014 15:54:58 +0100 Subject: [Python-Dev] PEP URLs In-Reply-To: <20140314100454.14ca0305@limelight.wooz.org> References: <20140314100454.14ca0305@limelight.wooz.org> Message-ID: <53231842.1080803@v.loewis.de> Am 14.03.14 15:04, schrieb Barry Warsaw: > On Mar 14, 2014, at 04:48 PM, Nick Coghlan wrote: > >> I opened https://github.com/python/pythondotorg/issues/297 to ask for >> an ETA on when we can expect them to be fully integrated. > > Thanks Nick. On the bug I suggest creating peps.python.org. And I suggested setting up a reverse proxy. Regards, Martin From benjamin at python.org Fri Mar 14 16:10:24 2014 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 14 Mar 2014 08:10:24 -0700 Subject: [Python-Dev] [Python-checkins] cpython: Added Doc/tools/ subdirs to .hgignore. In-Reply-To: <3flpwB0Njyz7LjX@mail.python.org> References: <3flpwB0Njyz7LjX@mail.python.org> Message-ID: <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> On Fri, Mar 14, 2014, at 07:20 AM, vinay.sajip wrote: > http://hg.python.org/cpython/rev/edb12dad7bf6 > changeset: 89654:edb12dad7bf6 > user: Vinay Sajip > date: Fri Mar 14 14:20:09 2014 +0000 > summary: > Added Doc/tools/ subdirs to .hgignore. > > files: > .hgignore | 5 +++++ > 1 files changed, 5 insertions(+), 0 deletions(-) > > > diff --git a/.hgignore b/.hgignore > --- a/.hgignore > +++ b/.hgignore > @@ -24,6 +24,11 @@ > reflog.txt$ > tags$ > Lib/plat-mac/errors.rsrc.df.rsrc > +Doc/tools/sphinx/ > +Doc/tools/docutils/ > +Doc/tools/jinja/ > +Doc/tools/jinja2/ > +Doc/tools/pygments/ Why are you readding these when they were apparently purposely removed in eef7899ea7ab? From zachary.ware+pydev at gmail.com Fri Mar 14 16:52:28 2014 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Fri, 14 Mar 2014 10:52:28 -0500 Subject: [Python-Dev] [Python-checkins] cpython: Added Doc/tools/ subdirs to .hgignore. In-Reply-To: <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> References: <3flpwB0Njyz7LjX@mail.python.org> <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> Message-ID: On Fri, Mar 14, 2014 at 10:10 AM, Benjamin Peterson wrote: > On Fri, Mar 14, 2014, at 07:20 AM, vinay.sajip wrote: >> diff --git a/.hgignore b/.hgignore >> --- a/.hgignore >> +++ b/.hgignore >> @@ -24,6 +24,11 @@ >> reflog.txt$ >> tags$ >> Lib/plat-mac/errors.rsrc.df.rsrc >> +Doc/tools/sphinx/ >> +Doc/tools/docutils/ >> +Doc/tools/jinja/ >> +Doc/tools/jinja2/ >> +Doc/tools/pygments/ > > Why are you readding these when they were apparently purposely removed > in eef7899ea7ab? It is rather ugly to build the docs in a 3.3 checkout, then update to default and run 'hg status' without those entries in .hgignore. I can understand not wanting to be bombarded by all of the untracked files, but it's also pretty easy to enable the purge extension and do "hg purge Doc/tools". -- Zach From benjamin at python.org Fri Mar 14 16:55:30 2014 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 14 Mar 2014 08:55:30 -0700 Subject: [Python-Dev] [Python-checkins] cpython: Added Doc/tools/ subdirs to .hgignore. In-Reply-To: References: <3flpwB0Njyz7LjX@mail.python.org> <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> Message-ID: <1394812530.8956.94532457.00F2EB6E@webmail.messagingengine.com> On Fri, Mar 14, 2014, at 08:52 AM, Zachary Ware wrote: > On Fri, Mar 14, 2014 at 10:10 AM, Benjamin Peterson > wrote: > > On Fri, Mar 14, 2014, at 07:20 AM, vinay.sajip wrote: > >> diff --git a/.hgignore b/.hgignore > >> --- a/.hgignore > >> +++ b/.hgignore > >> @@ -24,6 +24,11 @@ > >> reflog.txt$ > >> tags$ > >> Lib/plat-mac/errors.rsrc.df.rsrc > >> +Doc/tools/sphinx/ > >> +Doc/tools/docutils/ > >> +Doc/tools/jinja/ > >> +Doc/tools/jinja2/ > >> +Doc/tools/pygments/ > > > > Why are you readding these when they were apparently purposely removed > > in eef7899ea7ab? > > It is rather ugly to build the docs in a 3.3 checkout, then update to > default and run 'hg status' without those entries in .hgignore. I can > understand not wanting to be bombarded by all of the untracked files, > but it's also pretty easy to enable the purge extension and do "hg > purge Doc/tools". This is why everyone should use the share extension and have separate working copies for every branch. :) From g.brandl at gmx.net Fri Mar 14 17:46:10 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 14 Mar 2014 17:46:10 +0100 Subject: [Python-Dev] cpython: Added Doc/tools/ subdirs to .hgignore. In-Reply-To: <1394812530.8956.94532457.00F2EB6E@webmail.messagingengine.com> References: <3flpwB0Njyz7LjX@mail.python.org> <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> <1394812530.8956.94532457.00F2EB6E@webmail.messagingengine.com> Message-ID: <53233252.3000002@gmx.net> Am 14.03.2014 16:55, schrieb Benjamin Peterson: > > > On Fri, Mar 14, 2014, at 08:52 AM, Zachary Ware wrote: >> On Fri, Mar 14, 2014 at 10:10 AM, Benjamin Peterson >> wrote: >> > On Fri, Mar 14, 2014, at 07:20 AM, vinay.sajip wrote: >> >> diff --git a/.hgignore b/.hgignore >> >> --- a/.hgignore >> >> +++ b/.hgignore >> >> @@ -24,6 +24,11 @@ >> >> reflog.txt$ >> >> tags$ >> >> Lib/plat-mac/errors.rsrc.df.rsrc >> >> +Doc/tools/sphinx/ >> >> +Doc/tools/docutils/ >> >> +Doc/tools/jinja/ >> >> +Doc/tools/jinja2/ >> >> +Doc/tools/pygments/ >> > >> > Why are you readding these when they were apparently purposely removed >> > in eef7899ea7ab? >> >> It is rather ugly to build the docs in a 3.3 checkout, then update to >> default and run 'hg status' without those entries in .hgignore. I can >> understand not wanting to be bombarded by all of the untracked files, >> but it's also pretty easy to enable the purge extension and do "hg >> purge Doc/tools". > > This is why everyone should use the share extension and have separate > working copies for every branch. :) I agree. Georg From status at bugs.python.org Fri Mar 14 18:07:54 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 14 Mar 2014 18:07:54 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140314170754.0A29756A1B@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-03-07 - 2014-03-14) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4486 (-107) closed 28213 (+169) total 32699 (+62) Open issues with patches: 2063 Issues opened (40) ================== #5926: bdist_msi: add support for minimum Python version for pure Pyt http://bugs.python.org/issue5926 reopened by eric.araujo #18678: Wrong struct members name for spwd module http://bugs.python.org/issue18678 reopened by r.david.murray #20866: segfailt with os.popen and SIGPIPE http://bugs.python.org/issue20866 opened by hanno #20867: fix_import in 2to3 adds spurious relative import (windows) http://bugs.python.org/issue20867 opened by ccanepa #20868: Lib/test/test_socket.py: skip testGetServBy if /etc/services i http://bugs.python.org/issue20868 opened by Steap #20872: dbm/gdbm/ndbm close methods are not document http://bugs.python.org/issue20872 opened by r.david.murray #20874: Tutorial section on starting python is out of date http://bugs.python.org/issue20874 opened by r.david.murray #20876: python -m test test_pathlib fails http://bugs.python.org/issue20876 opened by Jey.Narasimhan #20879: base64 module of Python 3.4 uses 920 kB of memory http://bugs.python.org/issue20879 opened by haypo #20880: Windows installation problem with 3.3.5 http://bugs.python.org/issue20880 opened by BreamoreBoy #20884: importlib/__init__.py can not be loaded without __file__ - bre http://bugs.python.org/issue20884 opened by Jurko.Gospodneti?? #20885: Little Endian PowerPC64 Linux http://bugs.python.org/issue20885 opened by David.Edelsohn #20886: Disabling logging to ~/.python_history is not simple enough http://bugs.python.org/issue20886 opened by Sworddragon #20887: stdlib compatibility with pypy, test_zipfile.py http://bugs.python.org/issue20887 opened by mattip #20890: Miscellaneous PEP 101 additions http://bugs.python.org/issue20890 opened by ncoghlan #20891: PyGILState_Ensure on non-Python thread causes fatal error http://bugs.python.org/issue20891 opened by steve.dower #20895: Add bytes.empty_buffer and deprecate bytes(17) for the same pu http://bugs.python.org/issue20895 opened by ethan.furman #20897: @abstractmethod does not enforce method signatures http://bugs.python.org/issue20897 opened by the.mulhern #20898: Missing 507 response description http://bugs.python.org/issue20898 opened by Filip.Malczak #20899: Nested namespace imports do not work inside zip archives http://bugs.python.org/issue20899 opened by Andrey.Antsut #20900: distutils register command should print text, not bytes repr http://bugs.python.org/issue20900 opened by eric.araujo #20902: Which operand is preferred by set operations? Missing informat http://bugs.python.org/issue20902 opened by Giacomo.Alzetta #20903: smtplib.SMTP raises socket.timeout http://bugs.python.org/issue20903 opened by Chris.Clark #20904: HAVE_PY_SET_53BIT_PRECISION for m68k http://bugs.python.org/issue20904 opened by schwab #20906: Issues in Unicode HOWTO http://bugs.python.org/issue20906 opened by gwideman #20907: behavioral differences between shutil.unpack_archive and ZipFi http://bugs.python.org/issue20907 opened by peter at psantoro.net #20909: 3.4 cherry pick: d22ef969cb82 & f5be4ea5b43e & 25dc02a2acae n http://bugs.python.org/issue20909 opened by ncoghlan #20910: Make sleep configurable in tests http://bugs.python.org/issue20910 opened by haypo #20911: urllib 'headers' is not a well defined data type http://bugs.python.org/issue20911 opened by r.david.murray #20912: [zipfile.py]: Make zip directory attributes more friendly for http://bugs.python.org/issue20912 opened by vmurashev #20913: Standard Library documentation needs SSL security best practic http://bugs.python.org/issue20913 opened by r.david.murray #20915: Add "pip" section to experts list in devguide http://bugs.python.org/issue20915 opened by eric.araujo #20916: ssl.enum_certificates() will not return all certificates trust http://bugs.python.org/issue20916 opened by Adam.Goodman #20917: Idle: Enhance font change notification system http://bugs.python.org/issue20917 opened by terry.reedy #20920: Turtle module transparency. http://bugs.python.org/issue20920 opened by Eli Innis #20921: DeprecationWarning: The Windows bytes API has been deprecated, http://bugs.python.org/issue20921 opened by pramod.jadhav #20923: ConfigParser should be greedy when parsing section name http://bugs.python.org/issue20923 opened by miloskomarcevic #20924: openssl init 100% CPU utilization http://bugs.python.org/issue20924 opened by bob.bob #20926: Devguide: Enhance Quick Start portion of instructions for Wind http://bugs.python.org/issue20926 opened by kathweaver #20927: Different behaviour on Posix and Windows when using subprocess http://bugs.python.org/issue20927 opened by Jovik Most recent 15 issues with no replies (15) ========================================== #20927: Different behaviour on Posix and Windows when using subprocess http://bugs.python.org/issue20927 #20924: openssl init 100% CPU utilization http://bugs.python.org/issue20924 #20923: ConfigParser should be greedy when parsing section name http://bugs.python.org/issue20923 #20912: [zipfile.py]: Make zip directory attributes more friendly for http://bugs.python.org/issue20912 #20911: urllib 'headers' is not a well defined data type http://bugs.python.org/issue20911 #20906: Issues in Unicode HOWTO http://bugs.python.org/issue20906 #20899: Nested namespace imports do not work inside zip archives http://bugs.python.org/issue20899 #20897: @abstractmethod does not enforce method signatures http://bugs.python.org/issue20897 #20885: Little Endian PowerPC64 Linux http://bugs.python.org/issue20885 #20874: Tutorial section on starting python is out of date http://bugs.python.org/issue20874 #20872: dbm/gdbm/ndbm close methods are not document http://bugs.python.org/issue20872 #20867: fix_import in 2to3 adds spurious relative import (windows) http://bugs.python.org/issue20867 #20866: segfailt with os.popen and SIGPIPE http://bugs.python.org/issue20866 #20859: Context of documentation for conditional expressions http://bugs.python.org/issue20859 #20851: Update devguide to cover testing from a tarball http://bugs.python.org/issue20851 Most recent 15 issues waiting for review (15) ============================================= #20926: Devguide: Enhance Quick Start portion of instructions for Wind http://bugs.python.org/issue20926 #20913: Standard Library documentation needs SSL security best practic http://bugs.python.org/issue20913 #20912: [zipfile.py]: Make zip directory attributes more friendly for http://bugs.python.org/issue20912 #20910: Make sleep configurable in tests http://bugs.python.org/issue20910 #20907: behavioral differences between shutil.unpack_archive and ZipFi http://bugs.python.org/issue20907 #20904: HAVE_PY_SET_53BIT_PRECISION for m68k http://bugs.python.org/issue20904 #20884: importlib/__init__.py can not be loaded without __file__ - bre http://bugs.python.org/issue20884 #20879: base64 module of Python 3.4 uses 920 kB of memory http://bugs.python.org/issue20879 #20868: Lib/test/test_socket.py: skip testGetServBy if /etc/services i http://bugs.python.org/issue20868 #20867: fix_import in 2to3 adds spurious relative import (windows) http://bugs.python.org/issue20867 #20858: Enhancements/fixes to pure-python datetime module http://bugs.python.org/issue20858 #20853: pdb "args" crashes when an arg is not printable http://bugs.python.org/issue20853 #20849: add exist_ok to shutil.copytree http://bugs.python.org/issue20849 #20837: Ambiguity words in base64 documentation http://bugs.python.org/issue20837 #20827: IDLE : Display function argument list in ClassBrowser http://bugs.python.org/issue20827 Top 10 most discussed issues (10) ================================= #20884: importlib/__init__.py can not be loaded without __file__ - bre http://bugs.python.org/issue20884 26 msgs #20910: Make sleep configurable in tests http://bugs.python.org/issue20910 19 msgs #19861: Update What's New for Python 3.4 http://bugs.python.org/issue19861 15 msgs #16104: Use multiprocessing in compileall script http://bugs.python.org/issue16104 13 msgs #19257: Sub-optimal error message when importing a non-package http://bugs.python.org/issue19257 12 msgs #20879: base64 module of Python 3.4 uses 920 kB of memory http://bugs.python.org/issue20879 12 msgs #20265: Bring Windows docs up to date http://bugs.python.org/issue20265 10 msgs #17006: Add advice on best practices for hashing secrets http://bugs.python.org/issue17006 9 msgs #20784: 'collections.abc' is no longer defined when collections is imp http://bugs.python.org/issue20784 8 msgs #20145: unittest.assert*Regex functions should verify that expected_re http://bugs.python.org/issue20145 7 msgs Issues closed (158) =================== #3621: it would be nice if installers made by bdist_wininst stored an http://bugs.python.org/issue3621 closed by eric.araujo #4931: distutils does not show any error msg when can't build C modul http://bugs.python.org/issue4931 closed by eric.araujo #5342: packaging: add tests for old versions cleanup on update http://bugs.python.org/issue5342 closed by eric.araujo #5747: knowing the parent command http://bugs.python.org/issue5747 closed by eric.araujo #6142: distutils2 clean command: Remove extension modules built in-pl http://bugs.python.org/issue6142 closed by eric.araujo #7225: fwrite() compiler warnings http://bugs.python.org/issue7225 closed by benjamin.peterson #7894: too aggressive dependency tracking in distutils http://bugs.python.org/issue7894 closed by eric.araujo #8254: write a configure command http://bugs.python.org/issue8254 closed by eric.araujo #8255: Packaging step-by-step tutorial http://bugs.python.org/issue8255 closed by eric.araujo #8369: Add a lint command to distutils http://bugs.python.org/issue8369 closed by eric.araujo #8371: Add options to distutils2.install and pysetup install to only http://bugs.python.org/issue8371 closed by eric.araujo #8591: update mkpkg to latest coding standards http://bugs.python.org/issue8591 closed by eric.araujo #8668: Packaging: add a 'develop' command http://bugs.python.org/issue8668 closed by eric.araujo #8680: Add a sandbox in Distutils2 http://bugs.python.org/issue8680 closed by eric.araujo #9309: Add environment variable complementing command line option -no http://bugs.python.org/issue9309 closed by eric.araujo #10530: distutils2 should allow the installing of python files with in http://bugs.python.org/issue10530 closed by eric.araujo #11599: Useless error message when distutils fails compiling http://bugs.python.org/issue11599 closed by eric.araujo #11921: distutils2 should be able to compile an Extension based on the http://bugs.python.org/issue11921 closed by eric.araujo #11993: Use sub-second resolution to determine if a file is newer http://bugs.python.org/issue11993 closed by eric.araujo #12242: distutils2 environment marker for current compiler http://bugs.python.org/issue12242 closed by eric.araujo #12253: Document packaging.manifest and packaging.errors http://bugs.python.org/issue12253 closed by eric.araujo #12257: Rework/replace use of DISTUTILS_USE_SDK in packaging http://bugs.python.org/issue12257 closed by eric.araujo #12258: Clean up bytes I/O in get_compiler_versions http://bugs.python.org/issue12258 closed by eric.araujo #12302: packaging test command needs access to .dist-info http://bugs.python.org/issue12302 closed by eric.araujo #12335: pysetup create: ask before moving an existing setup.cfg http://bugs.python.org/issue12335 closed by eric.araujo #12344: Add **kwargs to reinitialize_command http://bugs.python.org/issue12344 closed by eric.araujo #12347: add an "extras" in packaging.pypi.simple.Crawler http://bugs.python.org/issue12347 closed by eric.araujo #12355: Crawler doesn't follow redirection http://bugs.python.org/issue12355 closed by eric.araujo #12366: packaging.pypi.dist should abstract download errors. http://bugs.python.org/issue12366 closed by eric.araujo #12368: packaging.pypi.simple.Crawler assumes external download links http://bugs.python.org/issue12368 closed by eric.araujo #12375: Add packages_root to sys.path for hooks http://bugs.python.org/issue12375 closed by eric.araujo #12377: Clean up use of packages_root/package_dir http://bugs.python.org/issue12377 closed by eric.araujo #12388: cannot specify recursive extra_files in packaging setup.cfg http://bugs.python.org/issue12388 closed by eric.araujo #12393: Packaging should provide support for extensible categories http://bugs.python.org/issue12393 closed by eric.araujo #12394: packaging: generate scripts from callable (dotted paths) http://bugs.python.org/issue12394 closed by eric.araujo #12405: packaging does not record/remove directories it creates http://bugs.python.org/issue12405 closed by eric.araujo #12416: packaging needs {pre,post}-{install,remove} hooks http://bugs.python.org/issue12416 closed by eric.araujo #12424: distutils2: extension section uses bad environment marker sepa http://bugs.python.org/issue12424 closed by eric.araujo #12449: Add accelerator "F" to button "Finish" in all MSI installers m http://bugs.python.org/issue12449 closed by eric.araujo #12526: packaging.pypi.Crawler and resulting objects have a circular A http://bugs.python.org/issue12526 closed by eric.araujo #12532: PyPI server index names with '/' in them http://bugs.python.org/issue12532 closed by eric.araujo #12659: Add tests for packaging.tests.support http://bugs.python.org/issue12659 closed by eric.araujo #12703: Improve error reporting for packaging.util.resolve_name http://bugs.python.org/issue12703 closed by eric.araujo #12830: --install-data doesn't effect resources destination http://bugs.python.org/issue12830 closed by eric.araujo #12891: Clean up traces of manifest template in packaging http://bugs.python.org/issue12891 closed by eric.araujo #12895: In MSI/EXE installer, allow installing Python modules in free http://bugs.python.org/issue12895 closed by eric.araujo #12900: Use universal newlines mode for setup.cfg http://bugs.python.org/issue12900 closed by eric.araujo #12925: python setup.py upload_docs doesn't ask for login and password http://bugs.python.org/issue12925 closed by eric.araujo #12937: distutils2: install action should support same options as inst http://bugs.python.org/issue12937 closed by eric.araujo #13049: distutils2 should not allow a distribution to install under an http://bugs.python.org/issue13049 closed by eric.araujo #13116: setup.cfg in [sb]dists should be static http://bugs.python.org/issue13116 closed by eric.araujo #13151: pysetup3 run bdist_wininst fails http://bugs.python.org/issue13151 closed by eric.araujo #13160: Rename install_dist to install http://bugs.python.org/issue13160 closed by eric.araujo #13166: Implement packaging.database.*Distribution.__str__ http://bugs.python.org/issue13166 closed by eric.araujo #13172: Improve detection of availability of bdist_msi command http://bugs.python.org/issue13172 closed by eric.araujo #13175: packaging uses wrong line endings in RECORD files on Windows http://bugs.python.org/issue13175 closed by eric.araujo #13178: Need tests for Unicode handling in install_distinfo and instal http://bugs.python.org/issue13178 closed by eric.araujo #13180: pysetup silently ignores invalid entries in setup.cfg http://bugs.python.org/issue13180 closed by eric.araujo #13189: New bdist_simple binary distribution format for packaging http://bugs.python.org/issue13189 closed by eric.araujo #13225: Failing packaging hooks should not stop operation http://bugs.python.org/issue13225 closed by eric.araujo #13331: Packaging cannot install resource directory trees specified in http://bugs.python.org/issue13331 closed by eric.araujo #13336: packaging.command.Command.copy_file doesn't implement preserve http://bugs.python.org/issue13336 closed by eric.araujo #13399: Don't print traceback for unrecognized actions, commands and o http://bugs.python.org/issue13399 closed by eric.araujo #13400: packaging: build command should have options to control byte-c http://bugs.python.org/issue13400 closed by eric.araujo #13408: Rename the packaging ???resources??? concept back to ???data f http://bugs.python.org/issue13408 closed by eric.araujo #13462: Improve code and tests for Mixin2to3 http://bugs.python.org/issue13462 closed by eric.araujo #13477: tarfile module should have a command line http://bugs.python.org/issue13477 closed by pitrou #13953: Get rid of doctests in packaging.tests.test_version http://bugs.python.org/issue13953 closed by eric.araujo #13967: also test for an empty pathname http://bugs.python.org/issue13967 closed by eric.araujo #13969: path name must always be string (or None) http://bugs.python.org/issue13969 closed by eric.araujo #14002: AttributeError in distutils2.pypi.wrapper http://bugs.python.org/issue14002 closed by eric.araujo #14011: packaging should use shutil archiving functions transparently http://bugs.python.org/issue14011 closed by eric.araujo #14140: packaging tests: add helpers to create and inspect a tree of f http://bugs.python.org/issue14140 closed by eric.araujo #14183: Test coverage for packaging.install and packaging.pypi.wrapper http://bugs.python.org/issue14183 closed by eric.araujo #14270: Can't install a project in a specific directory http://bugs.python.org/issue14270 closed by eric.araujo #14273: distutils2: logging handler not properly initialized http://bugs.python.org/issue14273 closed by eric.araujo #14274: pysetup does not look at requires.txt http://bugs.python.org/issue14274 closed by eric.araujo #14275: pysetup create doesn't handle install requirements http://bugs.python.org/issue14275 closed by eric.araujo #14276: installing latest version of a project http://bugs.python.org/issue14276 closed by eric.araujo #14279: packaging.pypi should support flat directories of distribution http://bugs.python.org/issue14279 closed by eric.araujo #14280: packaging.pypi should not require checksums http://bugs.python.org/issue14280 closed by eric.araujo #14294: Requirements are not properly copied into metatdata of dist-in http://bugs.python.org/issue14294 closed by eric.araujo #14319: cleanup index switching mechanism on packaging.pypi http://bugs.python.org/issue14319 closed by eric.araujo #14352: Distutils2: add logging message to report successful installat http://bugs.python.org/issue14352 closed by eric.araujo #14356: Distutils2 ignores site-local configuration http://bugs.python.org/issue14356 closed by eric.araujo #14517: distutils always recompiles all C source files http://bugs.python.org/issue14517 closed by eric.araujo #14549: Recursive inclusion of packages http://bugs.python.org/issue14549 closed by eric.araujo #14651: pysetup run cmd can't handle option values in the setup.cfg http://bugs.python.org/issue14651 closed by eric.araujo #14733: Custom commands don't work http://bugs.python.org/issue14733 closed by eric.araujo #14747: Classifiers are missing from distutils2-generated metadata http://bugs.python.org/issue14747 closed by eric.araujo #14755: Distutils2 doesn't have a Python 3 version on PyPI http://bugs.python.org/issue14755 closed by eric.araujo #14778: logging messages about bad version numbers should include the http://bugs.python.org/issue14778 closed by eric.araujo #14830: pysetup fails on non-ascii filenames http://bugs.python.org/issue14830 closed by eric.araujo #14843: support define_macros / undef_macros in setup.cfg http://bugs.python.org/issue14843 closed by eric.araujo #14848: Use shutil.move instead of os.rename in packaging http://bugs.python.org/issue14848 closed by eric.araujo #14858: 'pysetup create' off-by-one when choosing classification matur http://bugs.python.org/issue14858 closed by eric.araujo #14871: Rewrite the command line parsers and actions system used in di http://bugs.python.org/issue14871 closed by eric.araujo #14915: pysetup3.3 install is case insensitive, remove is case sensiti http://bugs.python.org/issue14915 closed by eric.araujo #14946: packaging???s pysetup script should be named differently than http://bugs.python.org/issue14946 closed by eric.araujo #14949: Documentation should state clearly the differences between "py http://bugs.python.org/issue14949 closed by eric.araujo #14950: Make packaging.install API less confusing and more extensible http://bugs.python.org/issue14950 closed by eric.araujo #14964: distutils2.utils.resolve_name cleanup http://bugs.python.org/issue14964 closed by eric.araujo #15605: Explain sphinx documentation building in devguide http://bugs.python.org/issue15605 closed by python-dev #16755: Distutils2 incorrectly works with unicode package names http://bugs.python.org/issue16755 closed by eric.araujo #16805: when building docs on Debian 7 --> ERROR: Error in "note" dire http://bugs.python.org/issue16805 closed by eric.araujo #16989: allow distutils debug mode to be enabled more easily http://bugs.python.org/issue16989 closed by eric.araujo #17002: html.entities ImportError in distutils2/pypi/simple.py on Pyth http://bugs.python.org/issue17002 closed by eric.araujo #17281: Broken links at pypi http://bugs.python.org/issue17281 closed by eric.araujo #17553: Note that distutils??? bdist_rpm command is not used to build http://bugs.python.org/issue17553 closed by eric.araujo #17574: pysetup failing with "OSError: [Errno 18] Invalid cross-device http://bugs.python.org/issue17574 closed by eric.araujo #17640: from distutils.util import byte_compile hangs http://bugs.python.org/issue17640 closed by eric.araujo #17855: Implement introspection of logger hierarchy http://bugs.python.org/issue17855 closed by vinay.sajip #18045: get_python_version is not import in bdist_rpm.py http://bugs.python.org/issue18045 closed by eric.araujo #19060: docs: note that subprocess doesn't replace os.exec* http://bugs.python.org/issue19060 closed by python-dev #19157: ipaddress.IPv6Network.hosts function omits network and broadca http://bugs.python.org/issue19157 closed by pmoody #19333: distutils.util.grok_environment_error loses the error message http://bugs.python.org/issue19333 closed by eric.araujo #19407: PEP 453: update the "Installing Python Modules" documentation http://bugs.python.org/issue19407 closed by python-dev #19614: support.temp_cwd should use support.rmtree http://bugs.python.org/issue19614 closed by python-dev #19627: python open built-in function - "updating" is not defined http://bugs.python.org/issue19627 closed by eric.araujo #19953: __iadd__() doc not strictly correct http://bugs.python.org/issue19953 closed by r.david.murray #20030: unittest.TestLoader.discover return value incorrectly document http://bugs.python.org/issue20030 closed by r.david.murray #20090: slight ambiguity in README.txt instructions for building docs http://bugs.python.org/issue20090 closed by python-dev #20104: expose posix_spawn(p) http://bugs.python.org/issue20104 closed by benjamin.peterson #20266: Bring Doc/faq/windows up to date http://bugs.python.org/issue20266 closed by eric.araujo #20720: test_create_server() of test_asyncio failure on "x86 Windows S http://bugs.python.org/issue20720 closed by haypo #20787: typo in asyncio docs for subprocess_exec() http://bugs.python.org/issue20787 closed by python-dev #20812: Explicitly cover application migration in the 2->3 guide http://bugs.python.org/issue20812 closed by brett.cannon #20813: Backport revised 2to3 guide to older branches http://bugs.python.org/issue20813 closed by brett.cannon #20863: IDLE not opening http://bugs.python.org/issue20863 closed by ned.deily #20864: getattr does not work well with descriptor http://bugs.python.org/issue20864 closed by ncoghlan #20869: IDLE lib error in IOBinding.py http://bugs.python.org/issue20869 closed by r.david.murray #20870: how to import json file in mongodb using pymongo http://bugs.python.org/issue20870 closed by r.david.murray #20871: Testing: Push email/policy.py line coverage to 100%. http://bugs.python.org/issue20871 closed by r.david.murray #20873: IDLE won't open http://bugs.python.org/issue20873 closed by ned.deily #20875: Typo in gzip._PaddedFile http://bugs.python.org/issue20875 closed by ned.deily #20877: test_grammar: assertEquals is deprecated http://bugs.python.org/issue20877 closed by python-dev #20878: Windows: uninstall Python 3.4rc3 after downgrade pip leaves fi http://bugs.python.org/issue20878 closed by loewis #20881: Can't install pip-1.5.4, ez_setup.py http://bugs.python.org/issue20881 closed by r.david.murray #20882: Link to OpenHatch ???getting started??? page from devguide http://bugs.python.org/issue20882 closed by eric.araujo #20883: Windows 'for current user' installation - 32/64-bit registrati http://bugs.python.org/issue20883 closed by loewis #20888: tracemalloc: add a get_line() method to Trace and Frame classe http://bugs.python.org/issue20888 closed by haypo #20889: asyncio.docs : asyncio.Condition acquire/release/locked method http://bugs.python.org/issue20889 closed by python-dev #20892: Typo in howto/pyporting.rst http://bugs.python.org/issue20892 closed by python-dev #20893: ctypes crash during PyFinalize when librt used http://bugs.python.org/issue20893 closed by haypo #20894: COMSPEC with multiple paths breaks subprocess.call http://bugs.python.org/issue20894 closed by r.david.murray #20896: test_ssl.test_get_server_certificate() should use PROTOCOL_SSL http://bugs.python.org/issue20896 closed by benjamin.peterson #20901: test_sqlite fails with SQLite 3.8.4 http://bugs.python.org/issue20901 closed by python-dev #20905: Adapt heapq push/pop/replace to allow passing a comparator. http://bugs.python.org/issue20905 closed by rhettinger #20908: Memory leak in Reg2Py() http://bugs.python.org/issue20908 closed by python-dev #20914: pyopenssl-0.14-1 error http://bugs.python.org/issue20914 closed by pitrou #20918: Logging of logging exceptions can fail in 3.4 if args are unpr http://bugs.python.org/issue20918 closed by vinay.sajip #20919: Image.show() does not open the image on windows xp http://bugs.python.org/issue20919 closed by ned.deily #20922: Global variables and Local Variables with same name http://bugs.python.org/issue20922 closed by ezio.melotti #20925: Update TLS version connection table http://bugs.python.org/issue20925 closed by christian.heimes #11060: distutils2 sdist does not complain about version that is not P http://bugs.python.org/issue11060 closed by eric.araujo #1682403: Transform reST to styled text in bdist_wininst-produced instal http://bugs.python.org/issue1682403 closed by eric.araujo #1294032: Distutils writes keywords comma-separated http://bugs.python.org/issue1294032 closed by eric.araujo #1692592: Stripping debugging symbols from compiled C extensions http://bugs.python.org/issue1692592 closed by eric.araujo From zachary.ware+pydev at gmail.com Fri Mar 14 18:08:22 2014 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Fri, 14 Mar 2014 12:08:22 -0500 Subject: [Python-Dev] [Python-checkins] cpython: Added Doc/tools/ subdirs to .hgignore. In-Reply-To: <1394812530.8956.94532457.00F2EB6E@webmail.messagingengine.com> References: <3flpwB0Njyz7LjX@mail.python.org> <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> <1394812530.8956.94532457.00F2EB6E@webmail.messagingengine.com> Message-ID: On Fri, Mar 14, 2014 at 10:55 AM, Benjamin Peterson wrote: > On Fri, Mar 14, 2014, at 08:52 AM, Zachary Ware wrote: >> On Fri, Mar 14, 2014 at 10:10 AM, Benjamin Peterson >> wrote: >> > Why are you readding these when they were apparently purposely removed >> > in eef7899ea7ab? >> >> It is rather ugly to build the docs in a 3.3 checkout, then update to >> default and run 'hg status' without those entries in .hgignore. I can >> understand not wanting to be bombarded by all of the untracked files, >> but it's also pretty easy to enable the purge extension and do "hg >> purge Doc/tools". > > This is why everyone should use the share extension and have separate > working copies for every branch. :) I've been using that setup for a few months now (and cursing myself for not switching sooner), but I've still found myself annoyed by this issue a time or two since the doc build chain change. I personally don't mind whether those entries are in .hgignore or not, but I don't see the harm in them sticking around (at least until 3.3 is out of maintenance; updating from 2.7 to default is a great source of problems even aside from this issue). -- Zach From solipsis at pitrou.net Fri Mar 14 18:12:30 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 14 Mar 2014 18:12:30 +0100 Subject: [Python-Dev] cpython: Added Doc/tools/ subdirs to .hgignore. References: <3flpwB0Njyz7LjX@mail.python.org> <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> <1394812530.8956.94532457.00F2EB6E@webmail.messagingengine.com> Message-ID: <20140314181230.063e4f6a@fsol> On Fri, 14 Mar 2014 12:08:22 -0500 Zachary Ware wrote: > On Fri, Mar 14, 2014 at 10:55 AM, Benjamin Peterson wrote: > > On Fri, Mar 14, 2014, at 08:52 AM, Zachary Ware wrote: > >> On Fri, Mar 14, 2014 at 10:10 AM, Benjamin Peterson > >> wrote: > >> > Why are you readding these when they were apparently purposely removed > >> > in eef7899ea7ab? > >> > >> It is rather ugly to build the docs in a 3.3 checkout, then update to > >> default and run 'hg status' without those entries in .hgignore. I can > >> understand not wanting to be bombarded by all of the untracked files, > >> but it's also pretty easy to enable the purge extension and do "hg > >> purge Doc/tools". > > > > This is why everyone should use the share extension and have separate > > working copies for every branch. :) > > I've been using that setup for a few months now (and cursing myself > for not switching sooner), but I've still found myself annoyed by this > issue a time or two since the doc build chain change. I personally > don't mind whether those entries are in .hgignore or not, but I don't > see the harm in them sticking around (at least until 3.3 is out of > maintenance; updating from 2.7 to default is a great source of > problems even aside from this issue). Indeed, I think it's ok to keep them. Regards Antoine. From rdmurray at bitdance.com Fri Mar 14 21:25:56 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 14 Mar 2014 16:25:56 -0400 Subject: [Python-Dev] Confirming status of new modules in 3.4 Message-ID: <20140314202557.0007A25047B@webabinitio.net> I just want to summarize the status of the modules that have been added to the stdlib in 3.4 to make sure they are all labeled correctly: Provisional: asyncio pathlib Not Provisional: ensurepip enum selectors statistics tracemalloc I'm especially curious about tracemalloc, since I know Victor found something he wanted to change (add?) to the API just recently. --David From solipsis at pitrou.net Fri Mar 14 23:24:28 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 14 Mar 2014 23:24:28 +0100 Subject: [Python-Dev] Confirming status of new modules in 3.4 References: <20140314202557.0007A25047B@webabinitio.net> Message-ID: <20140314232428.0e8a95ff@fsol> On Fri, 14 Mar 2014 16:25:56 -0400 "R. David Murray" wrote: > I just want to summarize the status of the modules that > have been added to the stdlib in 3.4 to make sure they > are all labeled correctly: > > Provisional: > > asyncio > pathlib These are right. Regards Antoine. From ethan at stoneleaf.us Fri Mar 14 23:27:02 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 14 Mar 2014 15:27:02 -0700 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: <20140314202557.0007A25047B@webabinitio.net> References: <20140314202557.0007A25047B@webabinitio.net> Message-ID: <53238236.4060703@stoneleaf.us> On 03/14/2014 01:25 PM, R. David Murray wrote: > > Not Provisional: > > enum This is correct. -- ~Ethan~ From vinay_sajip at yahoo.co.uk Sat Mar 15 00:06:50 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 14 Mar 2014 23:06:50 +0000 (GMT) Subject: [Python-Dev] [Python-checkins] cpython: Added Doc/tools/ subdirs to .hgignore. In-Reply-To: <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> References: <3flpwB0Njyz7LjX@mail.python.org> <1394809824.29257.94514229.6A393CDD@webmail.messagingengine.com> Message-ID: <1394838410.68259.YahooMailNeo@web172402.mail.ir2.yahoo.com> Sorry, my mistake - I should have checked more carefully. Thank you for pointing me to the commit where they were removed - I'll install a venv with Sphinx, revert the change and remove those files from my repo. Regards, Vinay Sajip ________________________________ From: Benjamin Peterson To: python-dev at python.org; python-checkins at python.org; vinay_sajip at yahoo.co.uk Sent: Friday, 14 March 2014, 15:10 Subject: Re: [Python-checkins] cpython: Added Doc/tools/ subdirs to .hgignore. On Fri, Mar 14, 2014, at 07:20 AM, vinay.sajip wrote: > http://hg.python.org/cpython/rev/edb12dad7bf6 > changeset:? 89654:edb12dad7bf6 > user:? ? ? ? Vinay Sajip > date:? ? ? ? Fri Mar 14 14:20:09 2014 +0000 > summary: >? Added Doc/tools/ subdirs to .hgignore. > > files: >? .hgignore |? 5 +++++ >? 1 files changed, 5 insertions(+), 0 deletions(-) > > > diff --git a/.hgignore b/.hgignore > --- a/.hgignore > +++ b/.hgignore > @@ -24,6 +24,11 @@ >? reflog.txt$ >? tags$ >? Lib/plat-mac/errors.rsrc.df.rsrc > +Doc/tools/sphinx/ > +Doc/tools/docutils/ > +Doc/tools/jinja/ > +Doc/tools/jinja2/ > +Doc/tools/pygments/ Why are you readding these when they were apparently purposely removed in eef7899ea7ab? -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Sat Mar 15 00:56:58 2014 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sat, 15 Mar 2014 00:56:58 +0100 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: <20140314202557.0007A25047B@webabinitio.net> References: <20140314202557.0007A25047B@webabinitio.net> Message-ID: On Fri, Mar 14, 2014 at 9:25 PM, R. David Murray wrote: > Not Provisional: > > selectors > Wouldn't it be wiser to consider this one provisional as well? -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Mar 15 00:59:08 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 14 Mar 2014 19:59:08 -0400 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: <20140314202557.0007A25047B@webabinitio.net> References: <20140314202557.0007A25047B@webabinitio.net> Message-ID: <1798D516-2B5C-4052-B64A-FA2F8FB53156@stufft.io> On Mar 14, 2014, at 4:25 PM, R. David Murray wrote: > I just want to summarize the status of the modules that > have been added to the stdlib in 3.4 to make sure they > are all labeled correctly: > > Provisional: > > asyncio > pathlib > > Not Provisional: > > ensurepip ensurepip is correct. > enum > selectors > statistics > tracemalloc > > > I'm especially curious about tracemalloc, since I know Victor found > something he wanted to change (add?) to the API just recently. > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From eliben at gmail.com Sat Mar 15 02:49:38 2014 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 14 Mar 2014 18:49:38 -0700 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: References: <20140314202557.0007A25047B@webabinitio.net> Message-ID: On Fri, Mar 14, 2014 at 4:56 PM, Giampaolo Rodola' wrote: > > On Fri, Mar 14, 2014 at 9:25 PM, R. David Murray wrote: > >> Not Provisional: >> >> selectors >> > > Wouldn't it be wiser to consider this one provisional as well? > +1 because of the intimate ties to asyncio Eli > > > > -- > Giampaolo - http://grodola.blogspot.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/eliben%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Mar 15 03:17:02 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 14 Mar 2014 19:17:02 -0700 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: References: <20140314202557.0007A25047B@webabinitio.net> Message-ID: I don't think so. asyncio depends on selectors but not vice versa. The selectors module was not part of PEP 3156. It is solid and I don't see any reason why it should get a reprieve from the usual strict backwards compatibility standards. --Guido On Fri, Mar 14, 2014 at 6:49 PM, Eli Bendersky wrote: > > > > On Fri, Mar 14, 2014 at 4:56 PM, Giampaolo Rodola' wrote: > >> >> On Fri, Mar 14, 2014 at 9:25 PM, R. David Murray wrote: >> >>> Not Provisional: >>> >>> selectors >>> >> >> Wouldn't it be wiser to consider this one provisional as well? >> > > +1 because of the intimate ties to asyncio > > Eli > > > >> >> >> > >> -- >> Giampaolo - http://grodola.blogspot.com >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/eliben%40gmail.com >> >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Mar 15 08:30:46 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 15 Mar 2014 08:30:46 +0100 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: <20140314202557.0007A25047B@webabinitio.net> References: <20140314202557.0007A25047B@webabinitio.net> Message-ID: > > > I'm especially curious about tracemalloc, since I know Victor found > something he wanted to change (add?) to the API just recently. > > I hope that the PEP process found all major design issues. I will try to avoid as much as posssible to break the backward compatibility. As you wrote, I may add new functions, methods or features. You can add the "provisional API" if you want. It would allow to fix the API if a design issue is found. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Sat Mar 15 12:02:47 2014 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sat, 15 Mar 2014 12:02:47 +0100 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: References: <20140314202557.0007A25047B@webabinitio.net> Message-ID: On Sat, Mar 15, 2014 at 3:17 AM, Guido van Rossum wrote: > I don't think so. asyncio depends on selectors but not vice versa. The > selectors module was not part of PEP 3156. It is solid and I don't see any > reason why it should get a reprieve from the usual strict backwards > compatibility standards. One part which can be improved is that right now the selectors module doesn't take advance of e/poll()'s modify() method: instead it just unregister() and register() the fd every time, which is of course considerably slower (there's also a TODO in the code about this). I guess that can be fixed later in a safely manner. Another concern I have is that we should probably rename self._epoll, self._select, self._kqueue to just "self._poller": that would make subclassing easier (see patch in issue http://bugs.python.org/issue18931) and would provide a unified interface for those users who want to reference the underlying poller object for some reason. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From cf.natali at gmail.com Sat Mar 15 12:51:17 2014 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Sat, 15 Mar 2014 11:51:17 +0000 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: References: <20140314202557.0007A25047B@webabinitio.net> Message-ID: 2014-03-15 11:02 GMT+00:00 Giampaolo Rodola' : > > One part which can be improved is that right now the selectors module doesn't take advance of e/poll()'s modify() method: instead it just unregister() and register() the fd every time, which is of course considerably slower (there's also a TODO in the code about this). > I guess that can be fixed later in a safely manner. Sure, it can be "fixed" easily, but I'd like to see the gain of this on a non-trivial benchmark (I mean a realistic workload, not just calling modify() in a tight loop). -------------- next part -------------- An HTML attachment was scrubbed... URL: From aahz at pythoncraft.com Sat Mar 15 15:38:51 2014 From: aahz at pythoncraft.com (Aahz) Date: Sat, 15 Mar 2014 07:38:51 -0700 Subject: [Python-Dev] WikiSort Message-ID: <20140315143851.GA25425@panix.com> [I'm nomail -- Cc me if you care whether I see followups] https://github.com/BonzaiThePenguin/WikiSort/tree/master WikiSort is a stable bottom-up in-place merge sort based on the work described in "Ratio based stable in-place merging", by Pok-Son Kim and Arne Kutzner [PDF]. Kim's and Kutzner's algorithm is a stable merge algorithm with great performance characteristics and proven correctness, but no attempt at adapting their work to a stable merge sort apparently existed. This is one such attempt! Probably no interest in switching over, but there might be a trick or two to add to TimSort. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "usenet imitates usenet" --Darkhawk From ncoghlan at gmail.com Sat Mar 15 16:13:44 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 16 Mar 2014 01:13:44 +1000 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: References: <20140314202557.0007A25047B@webabinitio.net> Message-ID: On 15 March 2014 21:02, Giampaolo Rodola' wrote: > > On Sat, Mar 15, 2014 at 3:17 AM, Guido van Rossum wrote: >> >> I don't think so. asyncio depends on selectors but not vice versa. The >> selectors module was not part of PEP 3156. It is solid and I don't see any >> reason why it should get a reprieve from the usual strict backwards >> compatibility standards. > > > One part which can be improved is that right now the selectors module > doesn't take advance of e/poll()'s modify() method: instead it just > unregister() and register() the fd every time, which is of course > considerably slower (there's also a TODO in the code about this). > I guess that can be fixed later in a safely manner. > > Another concern I have is that we should probably rename self._epoll, > self._select, self._kqueue to just "self._poller": that would make > subclassing easier (see patch in issue http://bugs.python.org/issue18931) > and would provide a unified interface for those users who want to reference > the underlying poller object for some reason. Those are currently private APIs regardless, so provisional or stable doesn't make a difference to our ability to change them. Provisional status is designed to cover cases like asyncio or the 3.3 ipaddress module, where we think they're largely ready for inclusion in the standard library, but aren't yet prepared to offer full backwards compatibility guarantees due to the potential discovery of issues in some of the more obscure edge cases. It's still preferred to skip that step if the contributors/maintainers of a new module are confident in the level of vetting the API has already received (the smaller and more targeted the API, the easier it is to skip an initial provisional release, especially when incorporating an existing PyPI project that doesn't require any significant API changes to be suitable for the stdlib). The experience with ipaddress (where we flipped it to stable for 3.4 with only changes that could be counted as bug fixes anyway) is still the one we expect to be typical for provisional modules. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Sat Mar 15 16:40:47 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 15 Mar 2014 08:40:47 -0700 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: References: <20140314202557.0007A25047B@webabinitio.net> Message-ID: On Sat, Mar 15, 2014 at 4:02 AM, Giampaolo Rodola' wrote: > > On Sat, Mar 15, 2014 at 3:17 AM, Guido van Rossum wrote: > >> I don't think so. asyncio depends on selectors but not vice versa. The >> selectors module was not part of PEP 3156. It is solid and I don't see any >> reason why it should get a reprieve from the usual strict backwards >> compatibility standards. > > > One part which can be improved is that right now the selectors module > doesn't take advance of e/poll()'s modify() method: instead it just > unregister() and register() the fd every time, which is of course > considerably slower (there's also a TODO in the code about this). > I guess that can be fixed later in a safely manner. > Yes, it doesn't change the API so doesn't require provisional status. > Another concern I have is that we should probably rename self._epoll, > self._select, self._kqueue to just "self._poller": that would make > subclassing easier (see patch in issue http://bugs.python.org/issue18931) > and would provide a unified interface for those users who want to reference > the underlying poller object for some reason. > Oh, but I don't want to make subclassing easier! And I don't want to open up backdoors for "advanced" users. Plus, the object they'd get is different enough that they should probably know what kind of object it is. If there's a missing API on the selector, we should add that, not tell people to help themselves. There lies madness (like the monkey-patching of asyncore in the past). The only people subclassing the Selector clases should be people proposing changes to the stdlib. Otherwise, if someone wants to write a Selector-like class for a new platform as a 3rd party class, they should write it from scratch (even using copy/paste!) rather than making a subclass of one of the existing stdlib classes, because otherwise they would (perhaps unintentionally) constrained future evolution of the latter (like the optimization you proposed above). This downside of using subclassing as an API should be well known by now and widely warned against. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Mar 15 16:55:18 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 16 Mar 2014 01:55:18 +1000 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: References: <20140314202557.0007A25047B@webabinitio.net> Message-ID: On 16 March 2014 01:40, Guido van Rossum wrote: > On Sat, Mar 15, 2014 at 4:02 AM, Giampaolo Rodola' > wrote: > > This downside of using subclassing as an API should be well known by now and > widely warned against. I've actually pondered the idea of suggesting we explicitly recommend the "procedural facade around an object oriented implementation" API design model in PEP 8, but it's not really a *coding* style guide issue, and I also haven't been able to come up with a good way of summarising it. That said, should we perhaps start codifying some of these principles as a "standard library API design guide"? We have a few additional issues to take into account that most software can ignore, like "assume that users that already know Python may be using the module and its documentation to learn a new domain, rather than the other way around, where a domain expert is just using the module to get things done". (That was the main driver for the differences between ipaddr and the accepted ipaddress API). The question about whether or not to add new boolean flags vs adding new APIs also comes up fairly often. At the moment, there's no real way for newcomers to pick up those principles other than hanging around long enough to see them come up again. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Sat Mar 15 17:19:37 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 15 Mar 2014 09:19:37 -0700 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: References: <20140314202557.0007A25047B@webabinitio.net> Message-ID: Maybe this would be a good subject for a series of blog posts? There is certainly plenty we have to say based on 20+ years of experience adding stuff to the stdlib (and not adding it). On Sat, Mar 15, 2014 at 8:55 AM, Nick Coghlan wrote: > On 16 March 2014 01:40, Guido van Rossum wrote: > > On Sat, Mar 15, 2014 at 4:02 AM, Giampaolo Rodola' > > wrote: > > > > This downside of using subclassing as an API should be well known by now > and > > widely warned against. > > I've actually pondered the idea of suggesting we explicitly recommend > the "procedural facade around an object oriented implementation" API > design model in PEP 8, but it's not really a *coding* style guide > issue, and I also haven't been able to come up with a good way of > summarising it. > > That said, should we perhaps start codifying some of these principles > as a "standard library API design guide"? We have a few additional > issues to take into account that most software can ignore, like > "assume that users that already know Python may be using the module > and its documentation to learn a new domain, rather than the other way > around, where a domain expert is just using the module to get things > done". (That was the main driver for the differences between ipaddr > and the accepted ipaddress API). The question about whether or not to > add new boolean flags vs adding new APIs also comes up fairly often. > > At the moment, there's no real way for newcomers to pick up those > principles other than hanging around long enough to see them come up > again. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Sat Mar 15 22:44:56 2014 From: Nikolaus at rath.org (Nikolaus Rath) Date: Sat, 15 Mar 2014 14:44:56 -0700 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: (Guido van Rossum's message of "Sat, 15 Mar 2014 08:40:47 -0700") References: <20140314202557.0007A25047B@webabinitio.net> Message-ID: <87zjkrm90n.fsf@vostro.rath.org> Guido van Rossum writes: > This downside of using subclassing as an API should be well known by now > and widely warned against. It wasn't known to me until now. Are these downsides described in some more detail somewhere? So far I have always thought that, as long as I avoid using private attributes, subclassing was supported like any other use of the API. Best, Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ?Time flies like an arrow, fruit flies like a Banana.? From tim.peters at gmail.com Sat Mar 15 22:51:48 2014 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 15 Mar 2014 16:51:48 -0500 Subject: [Python-Dev] WikiSort In-Reply-To: <20140315143851.GA25425@panix.com> References: <20140315143851.GA25425@panix.com> Message-ID: [Aahz ] > [I'm nomail -- Cc me if you care whether I see followups] > > https://github.com/BonzaiThePenguin/WikiSort/tree/master > > WikiSort is a stable bottom-up in-place merge sort based on the work > described in "Ratio based stable in-place merging", by Pok-Son Kim and > Arne Kutzner [PDF]. Kim's and Kutzner's algorithm is a stable merge > algorithm with great performance characteristics and proven > correctness, but no attempt at adapting their work to a stable merge > sort apparently existed. This is one such attempt! > > Probably no interest in switching over, but there might be a trick or > two to add to TimSort. It's the "in-place" here that's novel - a stable merge sort that doesn't require extra memory. C++ library authors may wish to look at this for that reason; the WikiSort text says: > * 3-15x faster than inplace_stable_sort(), which is libc++'s equivalent O(1) sort function. Last I looked, the C++ template library's inplace_stable_sort() used a merge sort variation that bought constant memory use at the cost of making runtime O(N * log(N)**2) (although the asymptotics weren't documented, they were easy enough to deduce). That's why "3-15x" isn't surprising. A similar trick _could_ be grafted into the guts of Python's sort to ensure constant memory use, but with a similar slowdown. But the tricks in WikiSort can't be grafted into Python's sort. It's a fundamentally different approach. I expect Python's sort still does significantly fewer comparisons, which is the most important thing to count for Python's sort (moving pointers around is "almost free" compared even to comparing integer objects in CPython). A very cool thing about the Kim & Kutzner paper is that the method there gracefully handles merging two vectors with very different sizes. For an extreme example, consider merging a vector of size 1 with a vector of size 1000000. "The standard" merge algorithm may require a million comparisons to do that, but it's obvious it can be done with only about 20 comparisons by using a binary search to find where the single element in the first vector belongs K&K adapt to that. So does Python's sort, but with a higher (although still logarithmic) upper bound on the number of comparisons needed. But WikiSort is a "bottom-up" mergesort, and is set up to guarantee that, at any point, the two areas it's currently merging never differ in size by more than 1 element. So the _potential_ for exploiting K&K's nice adaption to wildly different sizes can't be tapped. It does have other adaptive properties, although I bet Pyhon's current sort can exploit a lot more kinds of partial regularity. Finally, if you thought Python's sort was hard to understand ... you're partially right ;-) But it's really just a large number of tricks each of which is easy enough to understand on its own. Efficient in-place merge algorithms are inherently tricky. Indeed, one remarkable aspect of the K&K paper is that the authors actually implemented their algorithm; most prior attempts were so complex that the authors stopped after proving correctness - actually writing code to implement the mess was too depressing ;-) From greg.ewing at canterbury.ac.nz Sun Mar 16 00:00:30 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 16 Mar 2014 12:00:30 +1300 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: References: <20140314202557.0007A25047B@webabinitio.net> Message-ID: <5324DB8E.6070701@canterbury.ac.nz> Nick Coghlan wrote: > On 16 March 2014 01:40, Guido van Rossum wrote: > >>This downside of using subclassing as an API should be well known by now and >>widely warned against. > > I've actually pondered the idea of suggesting we explicitly recommend > the "procedural facade around an object oriented implementation" API > design model in PEP 8, I don't think I would call this a "procedural" API. To my mind, any API that exposes objects with methods is an object-oriented API, whether it encourages subclassing those objects or not. -- Greg From guido at python.org Sun Mar 16 00:27:59 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 15 Mar 2014 16:27:59 -0700 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: <87zjkrm90n.fsf@vostro.rath.org> References: <20140314202557.0007A25047B@webabinitio.net> <87zjkrm90n.fsf@vostro.rath.org> Message-ID: On Sat, Mar 15, 2014 at 2:44 PM, Nikolaus Rath wrote: > Guido van Rossum writes: > > This downside of using subclassing as an API should be well known by now > > and widely warned against. > > It wasn't known to me until now. Are these downsides described in some > more detail somewhere? > > So far I have always thought that, as long as I avoid using private > attributes, subclassing was supported like any other use of the API. > Googling for things like "containment vs. inheritance" finds a wealth of articles explaining the issues. Apparently even the classic GoF patterns book prefers containment. The problem with subclassing is that it tends to create tighter coupling between provider and consumer of an API. I believe this is due to the addition of "protected" APIs that are supposedly okay to use for subclasses but not for other users. If you can completely avoid those I suppose you'd be okay, but note that my remark was sparked by a proposal to create just such a protected API (_poller). Additionally, a subclass and its base class share the same namespace, so a method or attribute defined by a subclass may get in the way of something with the same name added in a new version of the base class. The main area where I think subclassing is fine is when base class and subclasses are all part of the same package/module, so their evolution is controlled by the same team. I suppose there are also some good uses for abstract base classes that avoid the above issues. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Mar 16 03:45:52 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 16 Mar 2014 12:45:52 +1000 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: <5324DB8E.6070701@canterbury.ac.nz> References: <20140314202557.0007A25047B@webabinitio.net> <5324DB8E.6070701@canterbury.ac.nz> Message-ID: On 16 March 2014 09:00, Greg Ewing wrote: > Nick Coghlan wrote: >> >> On 16 March 2014 01:40, Guido van Rossum wrote: >> >>> This downside of using subclassing as an API should be well known by now >>> and >>> widely warned against. >> >> >> I've actually pondered the idea of suggesting we explicitly recommend >> the "procedural facade around an object oriented implementation" API >> design model in PEP 8, > > > I don't think I would call this a "procedural" API. To my > mind, any API that exposes objects with methods is an > object-oriented API, whether it encourages subclassing those > objects or not. There are actually two variants of the approach. One is obviously procedural: you put basic data types (strings, numbers, containers, etc) in, you get basic data types out, and the fact an object managed the calculation in the middle is completely opaque to you as a user of the API. (from the outside, it still looks like a pure function, even though the internal calculation may have been highly stateful) The second variant is better exemplified by an API like contextlib.contextmanager(). That decorator returns a custom object type (now called contextlib._GeneratorContextManager). That type was originally undocumented with the name contextlib.GeneratorContextManager, and when the discrepancy was pointed out, I resolved it by adding the missing underscore (in 3.3 IIRC, could have been 3.2), rather than by documenting something I considered to be an implementation detail. The key problem this addresses is that telling people "this is a class" often overspecifies your API, because classes have a very broad interface. They support inheritance, typechecks, etc, etc. It's a really big commitment to your API users, and not one you should make lightly. Subclassing an object is also an *extremely* high coupling technique - it's genuinely difficult to refactor classes in any meaningful way with risking breakage of subclasses. As an example of unanticipated coupling issues that inhibit refactoring, consider a class where "method B" is originally written to call "method A". You later notice that this would be better structured by having both methods A and B call a new helper method C, rather than having B call A. If "method A" was a public API you now have a problem, since after the change, a subclass that previously only overrode A will now see different behaviour in method B (because that is now bypassing the override in the subclass). It can all get very, very messy, so experience has taught me that making your classes part of your public API before you're 100% certain you're happy with not only the public methods, but also the relationships between them, is a fine recipe for locking in bad design decisions in a hard to fix way. Sometimes the trade-off is worth it due to the power and flexibility that API users gain, but it needs to be recognised as the very high coupling technique that it is. By contrast, a callable API like contextlib.contextmanager just says "given a certain set of inputs, you will get some kind of object back with these characteristics, no more, no less". It is, in essence, ducktyping as applied to return types :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From cf.natali at gmail.com Sun Mar 16 15:08:11 2014 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Sun, 16 Mar 2014 14:08:11 +0000 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: <87zjkrm90n.fsf@vostro.rath.org> References: <20140314202557.0007A25047B@webabinitio.net> <87zjkrm90n.fsf@vostro.rath.org> Message-ID: 2014-03-15 21:44 GMT+00:00 Nikolaus Rath : > Guido van Rossum writes: > > This downside of using subclassing as an API should be well known by now > > and widely warned against. > > It wasn't known to me until now. Are these downsides described in some > more detail somewhere? > The short version is: "inheritance breaks encapsulation". As a trivial and stupid example, let's say you need a list object which counts the number of items inserted/removed (it's completely stupid, but that's not the point :-): So you might do something like: """ class CountingList(list): [...] def append(self, e): self.inserted += 1 return super().append(e) def extend(self, l): self.inserted += len(l) return super().extend(l) """ Looks fine, it would probably work. Now, it's actually very fragile: imagine what would happen if list.extend() was internally implemented by calling list.append() for each element: you'd end up counting each element twice (since the subclass append() method would be called). And that's the problem: by deriving from a class, you become dependent of its implementation, even though you're using its public API. Which means that it could work with e.g. CPython but not Pypy, or break with a new version of Python. Another related problem is, as Guido explained, that if you add a new method in the subclass, and the parent class gains a method with the same name in a new version, you're in trouble. That's why advising inheritance as a silver bullet for code reuses is IMO one of the biggest mistakes in OOP, simply because although attractive, inheritance breaks encapsulation. As a rule of thumb, you should only use inheritance within a module/package, or in other words only if you're in control of the implementation. The alternative is to use "composition" For more details, I highly encourage anyone interested in looking at the book "Effective Java" by Joshua Bloch (the example above is inspired by his book). Although Java-centric, it's packed with many advises, patterns and anti-patterns that are relevant to OOP and just programming in general (it's in my top-5 books). cf -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Sun Mar 16 19:24:34 2014 From: Nikolaus at rath.org (Nikolaus Rath) Date: Sun, 16 Mar 2014 11:24:34 -0700 Subject: [Python-Dev] Confirming status of new modules in 3.4 In-Reply-To: (=?utf-8?Q?=22Charles-Fran=C3=A7ois?= Natali"'s message of "Sun, 16 Mar 2014 14:08:11 +0000") References: <20140314202557.0007A25047B@webabinitio.net> <87zjkrm90n.fsf@vostro.rath.org> Message-ID: <8761neypb1.fsf@vostro.rath.org> Charles-Fran?ois Natali writes: > 2014-03-15 21:44 GMT+00:00 Nikolaus Rath : > >> Guido van Rossum writes: >> > This downside of using subclassing as an API should be well known by now >> > and widely warned against. >> >> It wasn't known to me until now. Are these downsides described in some >> more detail somewhere? >> > > The short version is: "inheritance breaks encapsulation". > > As a trivial and stupid example, let's say you need a list object which > counts the number of items inserted/removed (it's completely stupid, but > that's not the point :-): > > So you might do something like: [...] Very illuminating example, thanks a lot! Best, -Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ?Time flies like an arrow, fruit flies like a Banana.? From larry at hastings.org Mon Mar 17 07:29:55 2014 From: larry at hastings.org (Larry Hastings) Date: Sun, 16 Mar 2014 23:29:55 -0700 Subject: [Python-Dev] [RELEASED] Python 3.4.0 Message-ID: <53269663.9050408@hastings.org> On behalf of the Python development team, I'm thrilled to announce the official release of Python 3.4. Python 3.4 includes a range of improvements of the 3.x series, including hundreds of small improvements and bug fixes. Major new features and changes in the 3.4 release series include: * PEP 428, a "pathlib" module providing object-oriented filesystem paths * PEP 435, a standardized "enum" module * PEP 436, a build enhancement that will help generate introspection information for builtins * PEP 442, improved semantics for object finalization * PEP 443, adding single-dispatch generic functions to the standard library * PEP 445, a new C API for implementing custom memory allocators * PEP 446, changing file descriptors to not be inherited by default in subprocesses * PEP 450, a new "statistics" module * PEP 451, standardizing module metadata for Python's module import system * PEP 453, a bundled installer for the *pip* package manager * PEP 454, a new "tracemalloc" module for tracing Python memory allocations * PEP 456, a new hash algorithm for Python strings and binary data * PEP 3154, a new and improved protocol for pickled objects * PEP 3156, a new "asyncio" module, a new framework for asynchronous I/O To download Python 3.4.0 visit: http://www.python.org/download/releases/3.4.0/ This is a production release. Please report any issues you notice to: http://bugs.python.org/ Enjoy! -- Larry Hastings, Release Manager larry at hastings.org (on behalf of the entire python-dev team and 3.4's contributors) From larry at hastings.org Mon Mar 17 07:38:05 2014 From: larry at hastings.org (Larry Hastings) Date: Sun, 16 Mar 2014 23:38:05 -0700 Subject: [Python-Dev] default hg.python.org/cpython is now 3.5 Message-ID: <5326984D.1060508@hastings.org> The "3.4" branch is now checked in. It contains all the 3.4 releases since 3.4.0rc1. Its current state is effectively 3.4.1. The "default" branch is now 3.5. Cry havoc, and let slip the dogs of war, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Fri Mar 14 20:01:48 2014 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 14 Mar 2014 19:01:48 +0000 Subject: [Python-Dev] getattr vs hashattr In-Reply-To: References: <53209C2B.3010906@stoneleaf.us> <20140312161445.31b7b2db@anarchist.wooz.org> Message-ID: <5323521C.6080207@simplistix.co.uk> > On 03/12/2014 04:49 PM, Chris Angelico wrote: >> You can use hasattr() in place of AttributeError Is that true now? It used to be that hasattr swallowed all exceptions rather than just AttributeError making is a very dangerous weapon for anything (such as an orm or odb) that might do something interesting when you try and get an attribute from it. On 12/03/2014 21:37, Tres Seaver wrote: > I use: > > getattr(subject, attrname, default)? > > *all the time*. On this one, did anything ever come of making getattr have a default default of None? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From jurko.gospodnetic at pke.hr Mon Mar 17 08:30:29 2014 From: jurko.gospodnetic at pke.hr (=?UTF-8?B?SnVya28gR29zcG9kbmV0acSH?=) Date: Mon, 17 Mar 2014 08:30:29 +0100 Subject: [Python-Dev] [RELEASED] Python 3.4.0 - missing information on the web In-Reply-To: <53269663.9050408@hastings.org> References: <53269663.9050408@hastings.org> Message-ID: Hi. On 17.3.2014. 7:29, Larry Hastings wrote: > On behalf of the Python development team, I'm thrilled to announce > the official release of Python 3.4. > [...lots of great enhancements snipped...] Just noted that 'https://www.python.org/download' still lists Python 3.4.0rc2 as the latest 'testing release' (and does not mention the final Python 3.4.0 release at all). Best regards, Jurko Gospodneti? From ncoghlan at gmail.com Mon Mar 17 08:47:36 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 17 Mar 2014 17:47:36 +1000 Subject: [Python-Dev] [RELEASED] Python 3.4.0 In-Reply-To: <53269663.9050408@hastings.org> References: <53269663.9050408@hastings.org> Message-ID: On 17 March 2014 16:29, Larry Hastings wrote: > > On behalf of the Python development team, I'm thrilled to announce > the official release of Python 3.4. Huzzah! Thanks for all the work on the release wrangling Larry (and the rest of the release team), as well as to everyone that contributed to the many fine changes in this release :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ben+python at benfinney.id.au Mon Mar 17 09:43:30 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 17 Mar 2014 19:43:30 +1100 Subject: [Python-Dev] [RELEASED] Python 3.4.0 References: <53269663.9050408@hastings.org> Message-ID: <85lhw9p64t.fsf@benfinney.id.au> Larry Hastings writes: > On behalf of the Python development team, I'm thrilled to announce > the official release of Python 3.4. Hooray! This is great news, the new release looks like a major step forward. Thank you to everyone involved with making the Python 3.4 release happen. -- \ ?Why should I care about posterity? What's posterity ever done | `\ for me?? ?Groucho Marx | _o__) | Ben Finney From victor.stinner at gmail.com Mon Mar 17 09:51:01 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 17 Mar 2014 09:51:01 +0100 Subject: [Python-Dev] [python-committers] default hg.python.org/cpython is now 3.5 In-Reply-To: <5326984D.1060508@hastings.org> References: <5326984D.1060508@hastings.org> Message-ID: Until when should we fix bugs in the branch 3.3? Branches 3.1 and 3.2 only accept security fixes, right? Victor Le 17 mars 2014 07:48, "Larry Hastings" a ?crit : > > > The "3.4" branch is now checked in. It contains all the 3.4 releases > since 3.4.0rc1. Its current state is effectively 3.4.1. > > The "default" branch is now 3.5. > > Cry havoc, and let slip the dogs of war, > > > */arry* > > _______________________________________________ > python-committers mailing list > python-committers at python.org > https://mail.python.org/mailman/listinfo/python-committers > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Mon Mar 17 11:35:57 2014 From: nad at acm.org (Ned Deily) Date: Mon, 17 Mar 2014 03:35:57 -0700 Subject: [Python-Dev] [RELEASED] Python 3.4.0 - missing information on the web References: <53269663.9050408@hastings.org> Message-ID: In article , Jurko Gospodneti? wrote: > Just noted that 'https://www.python.org/download' still lists Python > 3.4.0rc2 as the latest 'testing release' (and does not mention the > final Python 3.4.0 release at all). Thanks for the report. That and other download pages should now be up-to-date. -- Ned Deily, nad at acm.org From jurko.gospodnetic at pke.hr Mon Mar 17 12:25:52 2014 From: jurko.gospodnetic at pke.hr (=?UTF-8?B?SnVya28gR29zcG9kbmV0acSH?=) Date: Mon, 17 Mar 2014 12:25:52 +0100 Subject: [Python-Dev] [RELEASED] Python 3.4.0 - PEP 429 needs updating? In-Reply-To: References: <53269663.9050408@hastings.org> Message-ID: Hi. On 17.3.2014. 11:35, Ned Deily wrote: > Thanks for the report. That and other download pages should now be > up-to-date. Something similar - does PEP 429 (Python 3.4 Release Schedule) need to be updated? It still lists 3.4.0rc3 as a 'future release'. Best regards, Jurko Gospodneti? From brett at python.org Mon Mar 17 15:05:50 2014 From: brett at python.org (Brett Cannon) Date: Mon, 17 Mar 2014 10:05:50 -0400 Subject: [Python-Dev] getattr vs hashattr In-Reply-To: <5323521C.6080207@simplistix.co.uk> References: <53209C2B.3010906@stoneleaf.us> <20140312161445.31b7b2db@anarchist.wooz.org> <5323521C.6080207@simplistix.co.uk> Message-ID: On Fri, Mar 14, 2014 at 3:01 PM, Chris Withers wrote: > On 03/12/2014 04:49 PM, Chris Angelico wrote: >> >>> You can use hasattr() in place of AttributeError >>> >> > Is that true now? It used to be that hasattr swallowed all exceptions > rather than just AttributeError making is a very dangerous weapon for > anything (such as an orm or odb) that might do something interesting when > you try and get an attribute from it. > Changed in Python 3.2: http://docs.python.org/3.2/whatsnew/3.2.html > > On 12/03/2014 21:37, Tres Seaver wrote: > >> I use: >> >> getattr(subject, attrname, default)? >> >> *all the time*. >> > > On this one, did anything ever come of making getattr have a default > default of None? No. If you want None returned as the default you still need to specify that to allow for those APIs where the attribute can have a value of None but you still want it to raise an AttributeError if the attribute isn't there. -Brett > > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Mar 17 15:10:18 2014 From: brett at python.org (Brett Cannon) Date: Mon, 17 Mar 2014 10:10:18 -0400 Subject: [Python-Dev] [python-committers] default hg.python.org/cpython is now 3.5 In-Reply-To: References: <5326984D.1060508@hastings.org> Message-ID: On Mon, Mar 17, 2014 at 4:51 AM, Victor Stinner wrote: > Until when should we fix bugs in the branch 3.3? Branches 3.1 and 3.2 only > accept security fixes, right? > Typically we do one last release before shutting the last bugfix branch down, but that's Georg's call since 3.3.5 was released so recently. -Brett > Victor > Le 17 mars 2014 07:48, "Larry Hastings" a ?crit : > >> >> >> The "3.4" branch is now checked in. It contains all the 3.4 releases >> since 3.4.0rc1. Its current state is effectively 3.4.1. >> >> The "default" branch is now 3.5. >> >> Cry havoc, and let slip the dogs of war, >> >> >> */arry* >> >> _______________________________________________ >> python-committers mailing list >> python-committers at python.org >> https://mail.python.org/mailman/listinfo/python-committers >> >> > _______________________________________________ > python-committers mailing list > python-committers at python.org > https://mail.python.org/mailman/listinfo/python-committers > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Mon Mar 17 16:18:56 2014 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 17 Mar 2014 08:18:56 -0700 Subject: [Python-Dev] [python-committers] default hg.python.org/cpython is now 3.5 In-Reply-To: References: <5326984D.1060508@hastings.org> Message-ID: <1395069536.415.95475857.4A1672A8@webmail.messagingengine.com> On Mon, Mar 17, 2014, at 7:10, Brett Cannon wrote: > On Mon, Mar 17, 2014 at 4:51 AM, Victor Stinner > wrote: > > > Until when should we fix bugs in the branch 3.3? Branches 3.1 and 3.2 only > > accept security fixes, right? > > > > Typically we do one last release before shutting the last bugfix branch > down, but that's Georg's call since 3.3.5 was released so recently. Given that, I propose 3.3 goes into security fix mode immediately. From tseaver at palladion.com Mon Mar 17 16:44:28 2014 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 17 Mar 2014 11:44:28 -0400 Subject: [Python-Dev] default hg.python.org/cpython is now 3.5 In-Reply-To: <1395069536.415.95475857.4A1672A8@webmail.messagingengine.com> References: <5326984D.1060508@hastings.org> <1395069536.415.95475857.4A1672A8@webmail.messagingengine.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/17/2014 11:18 AM, Benjamin Peterson wrote: > > > On Mon, Mar 17, 2014, at 7:10, Brett Cannon wrote: >> On Mon, Mar 17, 2014 at 4:51 AM, Victor Stinner >> wrote: >> >>> Until when should we fix bugs in the branch 3.3? Branches 3.1 and >>> 3.2 only accept security fixes, right? >>> >> >> Typically we do one last release before shutting the last bugfix >> branch down, but that's Georg's call since 3.3.5 was released so >> recently. > > Given that, I propose 3.3 goes into security fix mode immediately. Shouldn't we at least do a review of the open issues against 3.3 first, particularly those with patches? E.g. "critcal" / "patch review": http://bugs.python.org/issue?%40search_text=&ignore=file%3Acontent&title=&%40columns=title&id=&%40columns=id&stage=4&creation=&creator=&activity=&%40columns=activity&%40sort=activity&actor=&nosy=&type=&components=&versions=17&dependencies=&assignee=&keywords=&priority=2&%40group=priority&status=1&%40columns=status&resolution=&nosy_count=&message_count=&%40pagesize=50&%40startwith=0&%40action=search or "high" / "patch review": http://bugs.python.org/issue?%40search_text=&ignore=file%3Acontent&title=&%40columns=title&id=&%40columns=id&stage=4&creation=&creator=&activity=&%40columns=activity&%40sort=activity&actor=&nosy=&type=&components=&versions=17&dependencies=&assignee=&keywords=&priority=3&%40group=priority&status=1&%40columns=status&resolution=&nosy_count=&message_count=&%40pagesize=50&%40startwith=0&%40action=search (OT: man does that cry out for a URL shortener built in). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlMnGFYACgkQ+gerLs4ltQ5o5QCg2YUAmIm5POUXv40VB3DEtJPV cDMAnR9kjBJV0FaGTYrK9beA0YSDKLhQ =4yP2 -----END PGP SIGNATURE----- From martin at v.loewis.de Mon Mar 17 17:37:32 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 17 Mar 2014 17:37:32 +0100 Subject: [Python-Dev] [python-committers] default hg.python.org/cpython is now 3.5 In-Reply-To: <1395069536.415.95475857.4A1672A8@webmail.messagingengine.com> References: <5326984D.1060508@hastings.org> <1395069536.415.95475857.4A1672A8@webmail.messagingengine.com> Message-ID: <532724CC.9000502@v.loewis.de> Am 17.03.14 16:18, schrieb Benjamin Peterson: > > > On Mon, Mar 17, 2014, at 7:10, Brett Cannon wrote: >> On Mon, Mar 17, 2014 at 4:51 AM, Victor Stinner >> wrote: >> >>> Until when should we fix bugs in the branch 3.3? Branches 3.1 and 3.2 only >>> accept security fixes, right? >>> >> >> Typically we do one last release before shutting the last bugfix branch >> down, but that's Georg's call since 3.3.5 was released so recently. > > Given that, I propose 3.3 goes into security fix mode immediately. +1 Martin From yselivanov.ml at gmail.com Mon Mar 17 17:39:46 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 17 Mar 2014 12:39:46 -0400 Subject: [Python-Dev] [RELEASED] Python 3.4.0 In-Reply-To: <53269663.9050408@hastings.org> References: <53269663.9050408@hastings.org> Message-ID: <53272552.6050809@gmail.com> Congrats Larry and everybody who worked on 3.4! Yury On 2014-03-17, 2:29 AM, Larry Hastings wrote: > > On behalf of the Python development team, I'm thrilled to announce > the official release of Python 3.4. > > > Python 3.4 includes a range of improvements of the 3.x series, including > hundreds of small improvements and bug fixes. Major new features and > changes in the 3.4 release series include: > > * PEP 428, a "pathlib" module providing object-oriented filesystem paths > * PEP 435, a standardized "enum" module > * PEP 436, a build enhancement that will help generate introspection > information for builtins > * PEP 442, improved semantics for object finalization > * PEP 443, adding single-dispatch generic functions to the standard > library > * PEP 445, a new C API for implementing custom memory allocators > * PEP 446, changing file descriptors to not be inherited by default > in subprocesses > * PEP 450, a new "statistics" module > * PEP 451, standardizing module metadata for Python's module import > system > * PEP 453, a bundled installer for the *pip* package manager > * PEP 454, a new "tracemalloc" module for tracing Python memory > allocations > * PEP 456, a new hash algorithm for Python strings and binary data > * PEP 3154, a new and improved protocol for pickled objects > * PEP 3156, a new "asyncio" module, a new framework for asynchronous I/O > > > To download Python 3.4.0 visit: > > http://www.python.org/download/releases/3.4.0/ > > > This is a production release. Please report any issues you notice to: > > http://bugs.python.org/ > > > Enjoy! > > > -- > Larry Hastings, Release Manager > larry at hastings.org > (on behalf of the entire python-dev team and 3.4's contributors) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com From rymg19 at gmail.com Mon Mar 17 17:57:08 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 17 Mar 2014 11:57:08 -0500 Subject: [Python-Dev] [RELEASED] Python 3.4.0 In-Reply-To: <53269663.9050408@hastings.org> References: <53269663.9050408@hastings.org> Message-ID: YES!!! +1 to the authors of the statistics and pathlib modules. On Mon, Mar 17, 2014 at 1:29 AM, Larry Hastings wrote: > > On behalf of the Python development team, I'm thrilled to announce > the official release of Python 3.4. > > > Python 3.4 includes a range of improvements of the 3.x series, including > hundreds of small improvements and bug fixes. Major new features and > changes in the 3.4 release series include: > > * PEP 428, a "pathlib" module providing object-oriented filesystem paths > * PEP 435, a standardized "enum" module > * PEP 436, a build enhancement that will help generate introspection > information for builtins > * PEP 442, improved semantics for object finalization > * PEP 443, adding single-dispatch generic functions to the standard library > * PEP 445, a new C API for implementing custom memory allocators > * PEP 446, changing file descriptors to not be inherited by default > in subprocesses > * PEP 450, a new "statistics" module > * PEP 451, standardizing module metadata for Python's module import system > * PEP 453, a bundled installer for the *pip* package manager > * PEP 454, a new "tracemalloc" module for tracing Python memory allocations > * PEP 456, a new hash algorithm for Python strings and binary data > * PEP 3154, a new and improved protocol for pickled objects > * PEP 3156, a new "asyncio" module, a new framework for asynchronous I/O > > > To download Python 3.4.0 visit: > > http://www.python.org/download/releases/3.4.0/ > > > This is a production release. Please report any issues you notice to: > > http://bugs.python.org/ > > > Enjoy! > > > -- > Larry Hastings, Release Manager > larry at hastings.org > (on behalf of the entire python-dev team and 3.4's contributors) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > rymg19%40gmail.com > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Mar 17 18:09:18 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 17 Mar 2014 18:09:18 +0100 Subject: [Python-Dev] default hg.python.org/cpython is now 3.5 References: <5326984D.1060508@hastings.org> <1395069536.415.95475857.4A1672A8@webmail.messagingengine.com> Message-ID: <20140317180918.3e5d9c71@fsol> On Mon, 17 Mar 2014 11:44:28 -0400 Tres Seaver wrote: > > Shouldn't we at least do a review of the open issues against 3.3 first, > particularly those with patches? E.g. "critcal" / "patch review": > > http://bugs.python.org/issue?%40search_text=&ignore=file%3Acontent&title=&%40columns=title&id=&%40columns=id&stage=4&creation=&creator=&activity=&%40columns=activity&%40sort=activity&actor=&nosy=&type=&components=&versions=17&dependencies=&assignee=&keywords=&priority=2&%40group=priority&status=1&%40columns=status&resolution=&nosy_count=&message_count=&%40pagesize=50&%40startwith=0&%40action=search Looking at those, they are nice to fix, but not particularly critical (and indeed they have been open for quite some time). > or "high" / "patch review": > > http://bugs.python.org/issue?%40search_text=&ignore=file%3Acontent&title=&%40columns=title&id=&%40columns=id&stage=4&creation=&creator=&activity=&%40columns=activity&%40sort=activity&actor=&nosy=&type=&components=&versions=17&dependencies=&assignee=&keywords=&priority=3&%40group=priority&status=1&%40columns=status&resolution=&nosy_count=&message_count=&%40pagesize=50&%40startwith=0&%40action=search Ditto here. Regards Antoine. From g.rodola at gmail.com Mon Mar 17 18:11:39 2014 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Mon, 17 Mar 2014 18:11:39 +0100 Subject: [Python-Dev] [RELEASED] Python 3.4.0 In-Reply-To: References: <53269663.9050408@hastings.org> Message-ID: The what's new looks truly amazing, with pathlib and asyncio being my favourite additions. Thanks for all the hard work. On Mon, Mar 17, 2014 at 5:57 PM, Ryan Gonzalez wrote: > YES!!! +1 to the authors of the statistics and pathlib modules. > > > On Mon, Mar 17, 2014 at 1:29 AM, Larry Hastings wrote: > >> >> On behalf of the Python development team, I'm thrilled to announce >> the official release of Python 3.4. >> >> >> Python 3.4 includes a range of improvements of the 3.x series, including >> hundreds of small improvements and bug fixes. Major new features and >> changes in the 3.4 release series include: >> >> * PEP 428, a "pathlib" module providing object-oriented filesystem paths >> * PEP 435, a standardized "enum" module >> * PEP 436, a build enhancement that will help generate introspection >> information for builtins >> * PEP 442, improved semantics for object finalization >> * PEP 443, adding single-dispatch generic functions to the standard >> library >> * PEP 445, a new C API for implementing custom memory allocators >> * PEP 446, changing file descriptors to not be inherited by default >> in subprocesses >> * PEP 450, a new "statistics" module >> * PEP 451, standardizing module metadata for Python's module import system >> * PEP 453, a bundled installer for the *pip* package manager >> * PEP 454, a new "tracemalloc" module for tracing Python memory >> allocations >> * PEP 456, a new hash algorithm for Python strings and binary data >> * PEP 3154, a new and improved protocol for pickled objects >> * PEP 3156, a new "asyncio" module, a new framework for asynchronous I/O >> >> >> To download Python 3.4.0 visit: >> >> http://www.python.org/download/releases/3.4.0/ >> >> >> This is a production release. Please report any issues you notice to: >> >> http://bugs.python.org/ >> >> >> Enjoy! >> >> >> -- >> Larry Hastings, Release Manager >> larry at hastings.org >> (on behalf of the entire python-dev team and 3.4's contributors) >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ >> rymg19%40gmail.com >> > > > > -- > Ryan > If anybody ever asks me why I prefer C++ to C, my answer will be simple: > "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was > nul-terminated." > > > -- > https://mail.python.org/mailman/listinfo/python-list > > -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dia.aliounes at gmail.com Mon Mar 17 18:18:25 2014 From: dia.aliounes at gmail.com (Alioune Dia) Date: Mon, 17 Mar 2014 18:18:25 +0100 Subject: [Python-Dev] [RELEASED] Python 3.4.0 In-Reply-To: References: <53269663.9050408@hastings.org> Message-ID: yeah , asyncio is a great module, congrat for all jobs you are doing --Ad | Dakar 2014-03-17 18:11 GMT+01:00 Giampaolo Rodola' : > The what's new looks truly amazing, with pathlib and asyncio being my > favourite additions. > Thanks for all the hard work. > > > On Mon, Mar 17, 2014 at 5:57 PM, Ryan Gonzalez wrote: > >> YES!!! +1 to the authors of the statistics and pathlib modules. >> >> >> On Mon, Mar 17, 2014 at 1:29 AM, Larry Hastings wrote: >> >>> >>> On behalf of the Python development team, I'm thrilled to announce >>> the official release of Python 3.4. >>> >>> >>> Python 3.4 includes a range of improvements of the 3.x series, including >>> hundreds of small improvements and bug fixes. Major new features and >>> changes in the 3.4 release series include: >>> >>> * PEP 428, a "pathlib" module providing object-oriented filesystem paths >>> * PEP 435, a standardized "enum" module >>> * PEP 436, a build enhancement that will help generate introspection >>> information for builtins >>> * PEP 442, improved semantics for object finalization >>> * PEP 443, adding single-dispatch generic functions to the standard >>> library >>> * PEP 445, a new C API for implementing custom memory allocators >>> * PEP 446, changing file descriptors to not be inherited by default >>> in subprocesses >>> * PEP 450, a new "statistics" module >>> * PEP 451, standardizing module metadata for Python's module import >>> system >>> * PEP 453, a bundled installer for the *pip* package manager >>> * PEP 454, a new "tracemalloc" module for tracing Python memory >>> allocations >>> * PEP 456, a new hash algorithm for Python strings and binary data >>> * PEP 3154, a new and improved protocol for pickled objects >>> * PEP 3156, a new "asyncio" module, a new framework for asynchronous I/O >>> >>> >>> To download Python 3.4.0 visit: >>> >>> http://www.python.org/download/releases/3.4.0/ >>> >>> >>> This is a production release. Please report any issues you notice to: >>> >>> http://bugs.python.org/ >>> >>> >>> Enjoy! >>> >>> >>> -- >>> Larry Hastings, Release Manager >>> larry at hastings.org >>> (on behalf of the entire python-dev team and 3.4's contributors) >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ >>> rymg19%40gmail.com >>> >> >> >> >> -- >> Ryan >> If anybody ever asks me why I prefer C++ to C, my answer will be simple: >> "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was >> nul-terminated." >> >> >> -- >> https://mail.python.org/mailman/listinfo/python-list >> >> > > > -- > Giampaolo - http://grodola.blogspot.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dia.aliounes%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Mar 17 18:36:09 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 17 Mar 2014 18:36:09 +0100 Subject: [Python-Dev] peps: The final update to PEP 429, the Python 3.4 release schedule. (I think!) References: <3fnl0Y5myJz7LjM@mail.python.org> Message-ID: <20140317183609.4d0847db@fsol> On Mon, 17 Mar 2014 18:30:53 +0100 (CET) larry.hastings wrote: > +(Beta 1 was also "feature freeze"--no new features beyond this point.) Famous last words :-) Regards Antoine. From g.brandl at gmx.net Mon Mar 17 19:23:56 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 17 Mar 2014 19:23:56 +0100 Subject: [Python-Dev] 3.3 branch is now in security fix mode Message-ID: Hi all, since 3.3.5 and 3.4.0 practically coincided, it is a good point to end the bugfix maintenance of the 3.3 branch. Please only commit security-related fixes to 3.3 from now -- like for 3.2 -- and as always, please set tracker issues that relate to security fixes to "release blocker" priority. Georg From ether.joe at gmail.com Mon Mar 17 20:57:48 2014 From: ether.joe at gmail.com (Sean Felipe Wolfe) Date: Mon, 17 Mar 2014 12:57:48 -0700 Subject: [Python-Dev] trouble building 3.3 Message-ID: I'm working on some IDLE oriented bugs and I'm having some trouble building the 3.3 branch: -- make: *** No rule to make target `Modules/_operator.c', needed by `Modules/_operator.o'. Stop. -- I did a pull/update, then make clean, configure, and make. 2.7 and 3.4 are building fine. This is on OSX 10.9.0. I'll attach my full shell session. Any help is appreciated ... thanks! -- A musician must make music, an artist must paint, a poet must write, if he is to be ultimately at peace with himself. - Abraham Maslow -------------- next part -------------- seanwolfe-macbook:33 sean$ hg branch 3.3 seanwolfe-macbook:33 sean$ hg pull && hg update pulling from http://hg.python.org/cpython searching for changes no changes found 0 files updated, 0 files merged, 0 files removed, 0 files unresolved seanwolfe-macbook:33 sean$ ./configure checking for hg... found checking build system type... x86_64-apple-darwin13.0.2 checking host system type... x86_64-apple-darwin13.0.2 checking for --enable-universalsdk... no checking for --with-universal-archs... 32-bit checking MACHDEP... darwin checking for --without-gcc... no Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1 checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking for --with-cxx-main=... no checking for g++... no configure: WARNING: By default, distutils will build C++ extension modules with "g++". If this is not intended, then set CXX on the configure command line. checking for -Wl,--no-as-needed... no checking how to run the C preprocessor... gcc -E checking for grep that handles long lines and -e... /usr/bin/grep checking for egrep... /usr/bin/grep -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking minix/config.h usability... no checking minix/config.h presence... no checking for minix/config.h... no checking whether it is safe to define __EXTENSIONS__... yes checking for --with-suffix... checking for case-insensitive build directory... yes checking LIBRARY... libpython$(VERSION)$(ABIFLAGS).a checking LINKCC... $(PURIFY) $(MAINCC) checking for GNU ld... no checking for inline... inline checking for --enable-shared... no checking for --enable-profiling... no checking LDLIBRARY... libpython$(VERSION)$(ABIFLAGS).a checking for ranlib... ranlib checking for ar... ar checking for readelf... no checking for python3.3... python3.3 checking for a BSD-compatible install... /usr/bin/install -c checking for a thread-safe mkdir -p... ./install-sh -c -d checking for --with-pydebug... no checking whether gcc accepts and needs -fno-strict-aliasing... no checking if we can turn off gcc unused result warning... yes checking which compiler should be used... gcc checking whether gcc supports ParseTuple __format__... no checking whether pthreads are available without options... yes checking whether g++ also accepts flags for thread support... no checking for ANSI C header files... (cached) yes checking asm/types.h usability... no checking asm/types.h presence... no checking for asm/types.h... no checking conio.h usability... no checking conio.h presence... no checking for conio.h... no checking curses.h usability... yes checking curses.h presence... yes checking for curses.h... yes checking direct.h usability... no checking direct.h presence... no checking for direct.h... no checking dlfcn.h usability... yes checking dlfcn.h presence... yes checking for dlfcn.h... yes checking errno.h usability... yes checking errno.h presence... yes checking for errno.h... yes checking fcntl.h usability... yes checking fcntl.h presence... yes checking for fcntl.h... yes checking grp.h usability... yes checking grp.h presence... yes checking for grp.h... yes checking ieeefp.h usability... no checking ieeefp.h presence... no checking for ieeefp.h... no checking io.h usability... no checking io.h presence... no checking for io.h... no checking langinfo.h usability... yes checking langinfo.h presence... yes checking for langinfo.h... yes checking libintl.h usability... no checking libintl.h presence... no checking for libintl.h... no checking ncurses.h usability... yes checking ncurses.h presence... yes checking for ncurses.h... yes checking process.h usability... no checking process.h presence... no checking for process.h... no checking pthread.h usability... yes checking pthread.h presence... yes checking for pthread.h... yes checking sched.h usability... yes checking sched.h presence... yes checking for sched.h... yes checking shadow.h usability... no checking shadow.h presence... no checking for shadow.h... no checking signal.h usability... yes checking signal.h presence... yes checking for signal.h... yes checking for stdint.h... (cached) yes checking stropts.h usability... no checking stropts.h presence... no checking for stropts.h... no checking termios.h usability... yes checking termios.h presence... yes checking for termios.h... yes checking for unistd.h... (cached) yes checking utime.h usability... yes checking utime.h presence... yes checking for utime.h... yes checking poll.h usability... yes checking poll.h presence... yes checking for poll.h... yes checking sys/devpoll.h usability... no checking sys/devpoll.h presence... no checking for sys/devpoll.h... no checking sys/epoll.h usability... no checking sys/epoll.h presence... no checking for sys/epoll.h... no checking sys/poll.h usability... yes checking sys/poll.h presence... yes checking for sys/poll.h... yes checking sys/audioio.h usability... no checking sys/audioio.h presence... no checking for sys/audioio.h... no checking sys/xattr.h usability... yes checking sys/xattr.h presence... yes checking for sys/xattr.h... yes checking sys/bsdtty.h usability... no checking sys/bsdtty.h presence... no checking for sys/bsdtty.h... no checking sys/event.h usability... yes checking sys/event.h presence... yes checking for sys/event.h... yes checking sys/file.h usability... yes checking sys/file.h presence... yes checking for sys/file.h... yes checking sys/ioctl.h usability... yes checking sys/ioctl.h presence... yes checking for sys/ioctl.h... yes checking sys/kern_control.h usability... yes checking sys/kern_control.h presence... yes checking for sys/kern_control.h... yes checking sys/loadavg.h usability... no checking sys/loadavg.h presence... no checking for sys/loadavg.h... no checking sys/lock.h usability... yes checking sys/lock.h presence... yes checking for sys/lock.h... yes checking sys/mkdev.h usability... no checking sys/mkdev.h presence... no checking for sys/mkdev.h... no checking sys/modem.h usability... no checking sys/modem.h presence... no checking for sys/modem.h... no checking sys/param.h usability... yes checking sys/param.h presence... yes checking for sys/param.h... yes checking sys/select.h usability... yes checking sys/select.h presence... yes checking for sys/select.h... yes checking sys/sendfile.h usability... no checking sys/sendfile.h presence... no checking for sys/sendfile.h... no checking sys/socket.h usability... yes checking sys/socket.h presence... yes checking for sys/socket.h... yes checking sys/statvfs.h usability... yes checking sys/statvfs.h presence... yes checking for sys/statvfs.h... yes checking for sys/stat.h... (cached) yes checking sys/syscall.h usability... yes checking sys/syscall.h presence... yes checking for sys/syscall.h... yes checking sys/sys_domain.h usability... yes checking sys/sys_domain.h presence... yes checking for sys/sys_domain.h... yes checking sys/termio.h usability... no checking sys/termio.h presence... no checking for sys/termio.h... no checking sys/time.h usability... yes checking sys/time.h presence... yes checking for sys/time.h... yes checking sys/times.h usability... yes checking sys/times.h presence... yes checking for sys/times.h... yes checking for sys/types.h... (cached) yes checking sys/uio.h usability... yes checking sys/uio.h presence... yes checking for sys/uio.h... yes checking sys/un.h usability... yes checking sys/un.h presence... yes checking for sys/un.h... yes checking sys/utsname.h usability... yes checking sys/utsname.h presence... yes checking for sys/utsname.h... yes checking sys/wait.h usability... yes checking sys/wait.h presence... yes checking for sys/wait.h... yes checking pty.h usability... no checking pty.h presence... no checking for pty.h... no checking libutil.h usability... no checking libutil.h presence... no checking for libutil.h... no checking sys/resource.h usability... yes checking sys/resource.h presence... yes checking for sys/resource.h... yes checking netpacket/packet.h usability... no checking netpacket/packet.h presence... no checking for netpacket/packet.h... no checking sysexits.h usability... yes checking sysexits.h presence... yes checking for sysexits.h... yes checking bluetooth.h usability... no checking bluetooth.h presence... no checking for bluetooth.h... no checking bluetooth/bluetooth.h usability... no checking bluetooth/bluetooth.h presence... no checking for bluetooth/bluetooth.h... no checking linux/tipc.h usability... no checking linux/tipc.h presence... no checking for linux/tipc.h... no checking spawn.h usability... yes checking spawn.h presence... yes checking for spawn.h... yes checking util.h usability... yes checking util.h presence... yes checking for util.h... yes checking alloca.h usability... yes checking alloca.h presence... yes checking for alloca.h... yes checking for dirent.h that defines DIR... yes checking for library containing opendir... none required checking whether sys/types.h defines makedev... yes checking for net/if.h... yes checking for term.h... yes checking for linux/netlink.h... no checking for linux/can.h... no checking for linux/can/raw.h... no checking for clock_t in time.h... yes checking for makedev... yes checking Solaris LFS bug... no checking for mode_t... yes checking for off_t... yes checking for pid_t... yes checking for size_t... yes checking for uid_t in sys/types.h... yes checking for uint32_t... yes checking for uint32_t... yes checking for uint64_t... yes checking for uint64_t... yes checking for int32_t... yes checking for int32_t... yes checking for int64_t... yes checking for int64_t... yes checking for ssize_t... yes checking for __uint128_t... yes checking size of int... 4 checking size of long... 8 checking size of void *... 8 checking size of short... 2 checking size of float... 4 checking size of double... 8 checking size of fpos_t... 8 checking size of size_t... 8 checking size of pid_t... 4 checking for long long support... yes checking size of long long... 8 checking for long double support... yes checking size of long double... 16 checking for _Bool support... yes checking size of _Bool... 1 checking for uintptr_t... yes checking size of uintptr_t... 8 checking size of off_t... 8 checking whether to enable large file support... no checking size of time_t... 8 checking for pthread_t... yes checking size of pthread_t... 8 checking for --enable-framework... no checking for dyld... always on for Darwin checking the extension of shared libraries... .so checking LDSHARED... $(CC) -bundle -undefined dynamic_lookup checking CCSHARED... checking LINKFORSHARED... -framework CoreFoundation checking CFLAGSFORSHARED... checking SHLIBS... $(LIBS) checking for sendfile in -lsendfile... no checking for dlopen in -ldl... yes checking for shl_load in -ldld... no checking for library containing sem_init... none required checking for textdomain in -lintl... no checking for t_open in -lnsl... no checking for socket in -lsocket... no checking for --with-libs... no checking for pkg-config... /opt/local/bin/pkg-config checking for --with-system-expat... no checking for --with-system-ffi... no checking for --with-system-libmpdec... no checking for --enable-loadable-sqlite-extensions... no checking for --with-tcltk-includes... default checking for --with-tcltk-libs... default checking for --with-dbmliborder... checking for --with-signal-module... yes checking for --with-threads... yes checking if PTHREAD_SCOPE_SYSTEM is supported... yes checking for pthread_sigmask... yes checking for pthread_atfork... yes checking if --enable-ipv6 is specified... yes checking if RFC2553 API is available... yes checking ipv6 stack type... kame using libc checking for OSX 10.5 SDK or later... yes checking for --with-doc-strings... yes checking for --with-tsc... no checking for --with-pymalloc... yes checking for --with-valgrind... no checking for dlopen... yes checking DYNLOADFILE... dynload_shlib.o checking MACHDEP_OBJS... none checking for alarm... yes checking for accept4... no checking for setitimer... yes checking for getitimer... yes checking for bind_textdomain_codeset... no checking for chown... yes checking for clock... yes checking for confstr... yes checking for ctermid... yes checking for execv... yes checking for faccessat... no checking for fchmod... yes checking for fchmodat... no checking for fchown... yes checking for fchownat... no checking for fexecve... no checking for fdopendir... no checking for fork... yes checking for fpathconf... yes checking for fstatat... no checking for ftime... yes checking for ftruncate... yes checking for futimesat... no checking for futimens... no checking for futimes... yes checking for gai_strerror... yes checking for getgrouplist... yes checking for getgroups... yes checking for getlogin... yes checking for getloadavg... yes checking for getpeername... yes checking for getpgid... yes checking for getpid... yes checking for getpriority... yes checking for getresuid... no checking for getresgid... no checking for getpwent... yes checking for getspnam... no checking for getspent... no checking for getsid... yes checking for getwd... yes checking for if_nameindex... yes checking for initgroups... yes checking for kill... yes checking for killpg... yes checking for lchmod... yes checking for lchown... yes checking for lockf... yes checking for linkat... no checking for lstat... yes checking for lutimes... yes checking for mmap... yes checking for memrchr... no checking for mbrtowc... yes checking for mkdirat... no checking for mkfifo... yes checking for mkfifoat... no checking for mknod... yes checking for mknodat... no checking for mktime... yes checking for mremap... no checking for nice... yes checking for openat... no checking for pathconf... yes checking for pause... yes checking for pipe2... no checking for plock... no checking for poll... yes checking for posix_fallocate... no checking for posix_fadvise... no checking for pread... yes checking for pthread_init... no checking for pthread_kill... yes checking for putenv... yes checking for pwrite... yes checking for readlink... yes checking for readlinkat... no checking for readv... yes checking for realpath... yes checking for renameat... no checking for select... yes checking for sem_open... yes checking for sem_timedwait... no checking for sem_getvalue... yes checking for sem_unlink... yes checking for sendfile... yes checking for setegid... yes checking for seteuid... yes checking for setgid... yes checking for sethostname... yes checking for setlocale... yes checking for setregid... yes checking for setreuid... yes checking for setresuid... no checking for setresgid... no checking for setsid... yes checking for setpgid... yes checking for setpgrp... yes checking for setpriority... yes checking for setuid... yes checking for setvbuf... yes checking for sched_get_priority_max... yes checking for sched_setaffinity... no checking for sched_setscheduler... no checking for sched_setparam... no checking for sched_rr_get_interval... no checking for sigaction... yes checking for sigaltstack... yes checking for siginterrupt... yes checking for sigpending... yes checking for sigrelse... yes checking for sigtimedwait... no checking for sigwait... yes checking for sigwaitinfo... no checking for snprintf... yes checking for strftime... yes checking for strlcpy... yes checking for symlinkat... no checking for sync... yes checking for sysconf... yes checking for tcgetpgrp... yes checking for tcsetpgrp... yes checking for tempnam... yes checking for timegm... yes checking for times... yes checking for tmpfile... yes checking for tmpnam... yes checking for tmpnam_r... no checking for truncate... yes checking for uname... yes checking for unlinkat... no checking for unsetenv... yes checking for utimensat... no checking for utimes... yes checking for waitid... yes checking for waitpid... yes checking for wait3... yes checking for wait4... yes checking for wcscoll... yes checking for wcsftime... yes checking for wcsxfrm... yes checking for writev... yes checking for _getpty... no checking whether dirfd is declared... yes checking for chroot... yes checking for link... yes checking for symlink... yes checking for fchdir... yes checking for fsync... yes checking for fdatasync... no checking for epoll... no checking for epoll_create1... no checking for kqueue... yes checking for ctermid_r... yes checking for flock declaration... yes checking for flock... yes checking for getpagesize... yes checking for broken unsetenv... no checking for true... true checking for inet_aton in -lc... yes checking for chflags... yes checking for lchflags... yes checking for inflateCopy in -lz... yes checking for hstrerror... yes checking for inet_aton... yes checking for inet_pton... yes checking for setgroups... yes checking for openpty... yes checking for forkpty... yes checking for memmove... yes checking for fseek64... no checking for fseeko... yes checking for fstatvfs... yes checking for ftell64... no checking for ftello... yes checking for statvfs... yes checking for dup2... yes checking for getcwd... yes checking for strdup... yes checking for getpgrp... yes checking for setpgrp... (cached) yes checking for gettimeofday... yes checking for clock_gettime... no checking for clock_gettime in -lrt... no checking for clock_getres... no checking for clock_getres in -lrt... no checking for major... yes checking for getaddrinfo... yes checking getaddrinfo bug... no checking for getnameinfo... yes checking whether time.h and sys/time.h may both be included... yes checking whether struct tm is in sys/time.h or time.h... time.h checking for struct tm.tm_zone... yes checking for struct stat.st_rdev... yes checking for struct stat.st_blksize... yes checking for struct stat.st_flags... yes checking for struct stat.st_gen... yes checking for struct stat.st_birthtime... yes checking for struct stat.st_blocks... yes checking for time.h that defines altzone... no checking whether sys/select.h and sys/time.h may both be included... yes checking for addrinfo... yes checking for sockaddr_storage... yes checking whether char is unsigned... no checking for an ANSI C-conforming const... yes checking for working volatile... yes checking for working signed char... yes checking for prototypes... yes checking for variable length prototypes and stdarg.h... yes checking for socketpair... yes checking if sockaddr has sa_len member... yes checking whether va_list is an array... yes checking for gethostbyname_r... no checking for gethostbyname... yes checking for __fpu_control... no checking for __fpu_control in -lieee... no checking for --with-fpectl... no checking for --with-libm=STRING... default LIBM="" checking for --with-libc=STRING... default LIBC="" checking for x64 gcc inline assembler... yes checking whether C doubles are little-endian IEEE 754 binary64... yes checking whether C doubles are big-endian IEEE 754 binary64... no checking whether C doubles are ARM mixed-endian IEEE 754 binary64... no checking whether we can use gcc inline assembler to get and set x87 control word... yes checking for x87-style double rounding... no checking for acosh... yes checking for asinh... yes checking for atanh... yes checking for copysign... yes checking for erf... yes checking for erfc... yes checking for expm1... yes checking for finite... yes checking for gamma... yes checking for hypot... yes checking for lgamma... yes checking for log1p... yes checking for log2... yes checking for round... yes checking for tgamma... yes checking whether isinf is declared... yes checking whether isnan is declared... yes checking whether isfinite is declared... yes checking whether tanh preserves the sign of zero... yes checking whether log1p drops the sign of negative zero... no checking whether POSIX semaphores are enabled... yes checking for broken sem_getvalue... yes checking digit size for Python's longs... no value specified checking wchar.h usability... yes checking wchar.h presence... yes checking for wchar.h... yes checking size of wchar_t... 4 checking for UCS-4 tcl... no checking whether wchar_t is signed... yes no usable wchar_t found checking whether byte ordering is bigendian... no checking ABIFLAGS... m checking SOABI... cpython-33m checking LDVERSION... $(VERSION)$(ABIFLAGS) checking whether right shift extends the sign bit... yes checking for getc_unlocked() and friends... yes checking how to link readline libs... -lreadline checking for rl_callback_handler_install in -lreadline... yes checking for rl_pre_input_hook in -lreadline... yes checking for rl_completion_display_matches_hook in -lreadline... yes checking for rl_completion_matches in -lreadline... yes checking for broken nice()... no checking for broken poll()... no checking for struct tm.tm_zone... (cached) yes checking for working tzset()... yes checking for tv_nsec in struct stat... no checking for tv_nsec2 in struct stat... yes checking whether mvwdelch is an expression... yes checking whether WINDOW has _flags... no checking for is_term_resized... yes checking for resize_term... yes checking for resizeterm... yes configure: checking for device files checking for /dev/ptmx... yes checking for /dev/ptc... no checking for %lld and %llu printf() format support... yes checking for %zd printf() format support... yes checking for socklen_t... yes checking for broken mbstowcs... no checking for --with-computed-gotos... no value specified checking whether gcc supports computed gotos... yes checking for build directories... done checking for -O2... yes checking for glibc _FORTIFY_SOURCE/memmove bug... no checking for gcc ipa-pure-const bug... no configure: creating ./config.status config.status: creating Makefile.pre config.status: creating Modules/Setup.config config.status: creating Misc/python.pc config.status: creating Modules/ld_so_aix config.status: creating pyconfig.h config.status: pyconfig.h is unchanged creating Modules/Setup creating Modules/Setup.local creating Makefile seanwolfe-macbook:33 sean$ make ----------------------------------------------- Modules/Setup.dist is newer than Modules/Setup; check to make sure you have all the updates you need in your Modules/Setup file. Usually, copying Modules/Setup.dist to Modules/Setup will work. ----------------------------------------------- gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Modules/python.o ./Modules/python.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/acceler.o Parser/acceler.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/grammar1.o Parser/grammar1.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/listnode.o Parser/listnode.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/node.o Parser/node.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/parser.o Parser/parser.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/bitset.o Parser/bitset.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/metagrammar.o Parser/metagrammar.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/firstsets.o Parser/firstsets.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/grammar.o Parser/grammar.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/pgen.o Parser/pgen.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/myreadline.o Parser/myreadline.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/parsetok.o Parser/parsetok.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Parser/tokenizer.o Parser/tokenizer.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/abstract.o Objects/abstract.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/accu.o Objects/accu.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/boolobject.o Objects/boolobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/bytes_methods.o Objects/bytes_methods.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/bytearrayobject.o Objects/bytearrayobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/bytesobject.o Objects/bytesobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/cellobject.o Objects/cellobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/classobject.o Objects/classobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/codeobject.o Objects/codeobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/complexobject.o Objects/complexobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/descrobject.o Objects/descrobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/enumobject.o Objects/enumobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/exceptions.o Objects/exceptions.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/genobject.o Objects/genobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/fileobject.o Objects/fileobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/floatobject.o Objects/floatobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/frameobject.o Objects/frameobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/funcobject.o Objects/funcobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/iterobject.o Objects/iterobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/listobject.o Objects/listobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/longobject.o Objects/longobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/dictobject.o Objects/dictobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/memoryobject.o Objects/memoryobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/methodobject.o Objects/methodobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/moduleobject.o Objects/moduleobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/namespaceobject.o Objects/namespaceobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/object.o Objects/object.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/obmalloc.o Objects/obmalloc.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/capsule.o Objects/capsule.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/rangeobject.o Objects/rangeobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/setobject.o Objects/setobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/sliceobject.o Objects/sliceobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/structseq.o Objects/structseq.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/tupleobject.o Objects/tupleobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/typeobject.o Objects/typeobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/unicodeobject.o Objects/unicodeobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/unicodectype.o Objects/unicodectype.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Objects/weakrefobject.o Objects/weakrefobject.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/_warnings.o Python/_warnings.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/Python-ast.o Python/Python-ast.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/asdl.o Python/asdl.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/ast.o Python/ast.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/bltinmodule.o Python/bltinmodule.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/ceval.o Python/ceval.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/compile.o Python/compile.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/codecs.o Python/codecs.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/dynamic_annotations.o Python/dynamic_annotations.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/errors.o Python/errors.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/frozenmain.o Python/frozenmain.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/future.o Python/future.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/getargs.o Python/getargs.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/getcompiler.o Python/getcompiler.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/getcopyright.o Python/getcopyright.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -DPLATFORM='"darwin"' -o Python/getplatform.o ./Python/getplatform.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/getversion.o Python/getversion.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/graminit.o Python/graminit.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/import.o Python/import.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -I. -o Python/importdl.o ./Python/importdl.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/marshal.o Python/marshal.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/modsupport.o Python/modsupport.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/mystrtoul.o Python/mystrtoul.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/mysnprintf.o Python/mysnprintf.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/peephole.o Python/peephole.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/pyarena.o Python/pyarena.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/pyctype.o Python/pyctype.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/pyfpe.o Python/pyfpe.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/pymath.o Python/pymath.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/pystate.o Python/pystate.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/pythonrun.o Python/pythonrun.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/pytime.o Python/pytime.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/random.o Python/random.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/structmember.o Python/structmember.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/symtable.o Python/symtable.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE \ -DABIFLAGS='"m"' \ -o Python/sysmodule.o ./Python/sysmodule.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/traceback.o Python/traceback.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/getopt.o Python/getopt.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/pystrcmp.o Python/pystrcmp.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/pystrtod.o Python/pystrtod.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/dtoa.o Python/dtoa.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/formatter_unicode.o Python/formatter_unicode.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/fileutils.o Python/fileutils.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE \ -DSOABI='"cpython-33m"' \ -o Python/dynload_shlib.o ./Python/dynload_shlib.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/thread.o Python/thread.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Modules/config.o Modules/config.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -DPYTHONPATH='":plat-darwin"' \ -DPREFIX='"/usr/local"' \ -DEXEC_PREFIX='"/usr/local"' \ -DVERSION='"3.3"' \ -DVPATH='""' \ -o Modules/getpath.o ./Modules/getpath.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Modules/main.o Modules/main.c gcc -c -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Modules/gcmodule.o Modules/gcmodule.c gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -c ./Modules/_threadmodule.c -o Modules/_threadmodule.o gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -c ./Modules/signalmodule.c -o Modules/signalmodule.o gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -c ./Modules/posixmodule.c -o Modules/posixmodule.o gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -c ./Modules/errnomodule.c -o Modules/errnomodule.o gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -c ./Modules/pwdmodule.c -o Modules/pwdmodule.o gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -c ./Modules/_sre.c -o Modules/_sre.o gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -c ./Modules/_codecsmodule.c -o Modules/_codecsmodule.o gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -c ./Modules/_weakref.c -o Modules/_weakref.o gcc -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -c ./Modules/_functoolsmodule.c -o Modules/_functoolsmodule.o make: *** No rule to make target `Modules/_operator.c', needed by `Modules/_operator.o'. Stop. seanwolfe-macbook:33 sean$ From ether.joe at gmail.com Mon Mar 17 21:02:23 2014 From: ether.joe at gmail.com (Sean Felipe Wolfe) Date: Mon, 17 Mar 2014 13:02:23 -0700 Subject: [Python-Dev] hg branching + log question Message-ID: I'm getting my feet wet with the cpython sources and Mercurial. I'm a bit confused -- when I checkout a branch, eg. 3.3, and I do an 'hg log', why do I see log messages for other branches? I'm expecting different branches to be kept discreetly. If I switch to 3.3, then I expect to see only 3.3 commits and changes. Also, what does the branch 'default' mean in this context? Since cpython is concerned with 3 major concurrent release tracks, why would there even be a default ? Does it just point to 3.4? Thanks. -- A musician must make music, an artist must paint, a poet must write, if he is to be ultimately at peace with himself. - Abraham Maslow From benjamin at python.org Mon Mar 17 21:04:11 2014 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 17 Mar 2014 13:04:11 -0700 Subject: [Python-Dev] hg branching + log question In-Reply-To: References: Message-ID: <1395086651.1838.95599569.2185ADC4@webmail.messagingengine.com> On Mon, Mar 17, 2014, at 13:02, Sean Felipe Wolfe wrote: > I'm getting my feet wet with the cpython sources and Mercurial. I'm a > bit confused -- when I checkout a branch, eg. 3.3, and I do an 'hg > log', why do I see log messages for other branches? That's just the way Mercurial works. If you only want to see messages on the current branch, do "hg log -b .". > > I'm expecting different branches to be kept discreetly. If I switch to > 3.3, then I expect to see only 3.3 commits and changes. > > Also, what does the branch 'default' mean in this context? Since > cpython is concerned with 3 major concurrent release tracks, why would > there even be a default ? Does it just point to 3.4? It's what will become 3.5 as of today. From solipsis at pitrou.net Mon Mar 17 21:06:25 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 17 Mar 2014 21:06:25 +0100 Subject: [Python-Dev] trouble building 3.3 References: Message-ID: <20140317210625.33067a13@fsol> On Mon, 17 Mar 2014 12:57:48 -0700 Sean Felipe Wolfe wrote: > I'm working on some IDLE oriented bugs and I'm having some trouble > building the 3.3 branch: > > -- > make: *** No rule to make target `Modules/_operator.c', needed by > `Modules/_operator.o'. Stop. > -- > > I did a pull/update, then make clean, configure, and make. Try "make distclean" and then re-run configure. Regards Antoine. From solipsis at pitrou.net Mon Mar 17 21:07:30 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 17 Mar 2014 21:07:30 +0100 Subject: [Python-Dev] hg branching + log question References: Message-ID: <20140317210730.38e2d118@fsol> On Mon, 17 Mar 2014 13:02:23 -0700 Sean Felipe Wolfe wrote: > I'm getting my feet wet with the cpython sources and Mercurial. I'm a > bit confused -- when I checkout a branch, eg. 3.3, and I do an 'hg > log', why do I see log messages for other branches? This is a classic hg question, you would get the answer by asking Mercurial for help: hg log --help :) Basically, to restrict the log to a given branch, just use the -b option: hg log -b 3.3. Regards Antoine. From jurko.gospodnetic at pke.hr Mon Mar 17 22:10:28 2014 From: jurko.gospodnetic at pke.hr (=?windows-1250?Q?Jurko_Gospodneti=E6?=) Date: Mon, 17 Mar 2014 22:10:28 +0100 Subject: [Python-Dev] 'Add/Remove Programs' entry missing for 'current user only' 32-bit installations on 64-bit Windows Message-ID: Hi all. This seems like it should be an 'old' issue, but I have not been able to find anything related to it on the web. When you install 32-bit CPython 'for the current user only' on 64-bit Windows, the installation does not show up in the Windows 'Add/Remove Programs' dialog (a.k.a. 'Programs and Features' dialog on Windows 7). It does not show up there even for the current user. As a consequence it is possible for the user to end up in a situation where it is impossible to install a 32-bit Python installation without manually cleaning up a previous 32-bit Python installation's Windows Installer related registry entries (more on this a bit later). I tested this CPython installation behaviour with the following CPython versions: * 2.3.4 - works correctly * 2.5.4 - works correctly * 2.6.0 - works correctly * 2.6.2 - works correctly * 2.6.3 - does not work correctly * 2.6.4 - does not work correctly * 2.6.6 - does not work correctly * 2.7.6 - does not work correctly * 3.3.3 - does not work correctly * 3.3.5 - does not work correctly * 3.4.0 - does not work correctly Several related observations, indicating that this could be some sort of a Windows Installer misconfiguration issue possibly solvable in the CPython installer: * The same does not occur when you install a 64-bit CPython version - their entry is correctly displayed in the 'Add/Remove Programs' dialog. * 32-bit CPython installer still correctly detects that a specific 32-bit version has already been installed, even if that previous installation is not listed in the 'Add/Remove Programs' dialog. * When you ask Windows using its WMI interface to list all the products installed on it (e.g. by running 'wmic product list' on the command-line), all the installed CPython versions are correctly displayed, even those 32-bit versions not displayed in the 'Add/Remove Programs' dialog. I started looking into this because I had a 32-bit CPython 3.4.0rc3 installation on my system that I wanted to remove, and since I did not see it registered in the 'Add/Remove Programs' dialog, I directly deleted its installation folder. That left my system in a state where a replacement 32-bit CPython 3.4.0 installation would always fail because it would detect that there was already a valid CPython 3.4 installation on the system, and then it would fail attempting to uninstall it. Fixing this required manually cleaning up leftover CPython 3.4.0rc3 windows installer registry entries. Note that the issue could not be fixed by using the CPython 3.4.0rc3 installer as it failed due to the same problem. This situation would suggest that something should be done in the Windows installer so it does not 'fail miserably' if a previous CPython installation can not removed. Possibly allow it to detect that the previous CPython installation has already been removed and simply clean up its Windows Installer registry entries. While working on this, I did discover a 'workaround' for the 'Add/Remove Programs' dialog issue, but it will take someone more knowledgeable about the Windows Installer infrastructure to say if the workaround can be applied directly as a clean solution or if it could have hidden consequences: Each 32-bit installation on 64-bit Windows has its own 'Uninstall' registry key under: > HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall named after its Windows Installer assigned GUID, e.g.: > HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\{A37F2D73-72D1-364D-BA5D-CEA430BCC040} In that key there is a value named 'WindowsInstaller' which for CPython installations has the value '1' (of type REG_DWORD). If you change that value to '0' (again, of type REG_DWORD), the installation's 'Add/Remove Programs' dialog entry will get displayed correctly, and user will be able to run the installation (reinstall/change/uninstall) from there. I compared MSI packaging related CPython source code (Tools/msi folder) in 2.6.2 & 2.6.3 releases but I can not see anything suspicious there. It could be that the observed beaviour change between those two versions is a result of the final release packager changing his used Windows Installer version, but I have not rebuilt CPython, and its MSI installer to test this theory out. Anyone have any more information on this? Should I open a 'Add/Remove Programs' dialog related issue in the CPython issue tracker? And possibly a separate one for making CPython installations not fail without possible recovery if a previous CPython installation has already been removed? Many thanks. Best regards, Jurko Gospodneti? P.S. All this has been tested using Windows 7 SP1 (x64). From victor.stinner at gmail.com Mon Mar 17 22:36:46 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 17 Mar 2014 22:36:46 +0100 Subject: [Python-Dev] [python-committers] default hg.python.org/cpython is now 3.5 In-Reply-To: <5326984D.1060508@hastings.org> References: <5326984D.1060508@hastings.org> Message-ID: Hi, I modified the Misc/NEWS file: * I moved 3.3 sections to Misc/HISTORY: items were already present, but the format in Misc/NEWS was improved (changeset 6ba468d4fa96) * I removed 3.4.1 section: changes of 3.4 after 3.4.0 must already be present in the 3.4 branch (changeset cb161cd94e6e) Is that correct? Victor 2014-03-17 7:38 GMT+01:00 Larry Hastings : > > > The "3.4" branch is now checked in. It contains all the 3.4 releases since > 3.4.0rc1. Its current state is effectively 3.4.1. > > The "default" branch is now 3.5. > > Cry havoc, and let slip the dogs of war, > > > /arry > > _______________________________________________ > python-committers mailing list > python-committers at python.org > https://mail.python.org/mailman/listinfo/python-committers > From tjreedy at udel.edu Mon Mar 17 22:52:42 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 17 Mar 2014 17:52:42 -0400 Subject: [Python-Dev] trouble building 3.3 In-Reply-To: References: Message-ID: On 3/17/2014 3:57 PM, Sean Felipe Wolfe wrote: > I'm working on some IDLE oriented bugs and I'm having some trouble > building the 3.3 branch: Starting today, Idle for 3.3 is no more patched. 2.7, 3.4, and 3.5 will be patched for now. -- Terry Jan Reedy From ncoghlan at gmail.com Tue Mar 18 00:23:02 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 18 Mar 2014 09:23:02 +1000 Subject: [Python-Dev] [python-committers] default hg.python.org/cpython is now 3.5 In-Reply-To: References: <5326984D.1060508@hastings.org> Message-ID: On 18 Mar 2014 07:37, "Victor Stinner" wrote: > > Hi, > > I modified the Misc/NEWS file: > > * I moved 3.3 sections to Misc/HISTORY: items were already present, > but the format in Misc/NEWS was improved (changeset 6ba468d4fa96) > * I removed 3.4.1 section: changes of 3.4 after 3.4.0 must already be > present in the 3.4 branch (changeset cb161cd94e6e) > > Is that correct? Not everything was cherry picked, so we'll need to review that to move the 3.4.1 fixes back to the appropriate section. Cheers, Nick. > > Victor > > 2014-03-17 7:38 GMT+01:00 Larry Hastings : > > > > > > The "3.4" branch is now checked in. It contains all the 3.4 releases since > > 3.4.0rc1. Its current state is effectively 3.4.1. > > > > The "default" branch is now 3.5. > > > > Cry havoc, and let slip the dogs of war, > > > > > > /arry > > > > _______________________________________________ > > python-committers mailing list > > python-committers at python.org > > https://mail.python.org/mailman/listinfo/python-committers > > > _______________________________________________ > python-committers mailing list > python-committers at python.org > https://mail.python.org/mailman/listinfo/python-committers -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Tue Mar 18 00:35:17 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 17 Mar 2014 16:35:17 -0700 Subject: [Python-Dev] [python-committers] default hg.python.org/cpython is now 3.5 In-Reply-To: References: <5326984D.1060508@hastings.org> Message-ID: <532786B5.6080109@hastings.org> On 03/17/2014 04:23 PM, Nick Coghlan wrote: > > > On 18 Mar 2014 07:37, "Victor Stinner" > wrote: > > > > Hi, > > > > I modified the Misc/NEWS file: > > > > * I moved 3.3 sections to Misc/HISTORY: items were already present, > > but the format in Misc/NEWS was improved (changeset 6ba468d4fa96) > > * I removed 3.4.1 section: changes of 3.4 after 3.4.0 must already be > > present in the 3.4 branch (changeset cb161cd94e6e) > > > > Is that correct? > > Not everything was cherry picked, so we'll need to review that to move > the 3.4.1 fixes back to the appropriate section. > I merged "default" into "3.4" (3a3a83195159), so every change that was in "default" last night will automatically go into 3.4.1. Stuff that got cherry-picked back for 3.4.0 should already be in their correct sections. I worked pretty hard to get that right, so while I'd be interested to hear if I got it wrong, my assumption (my hope) for now is that Misc/NEWS is basically correct in both "3.4" and "default". //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ether.joe at gmail.com Tue Mar 18 00:51:27 2014 From: ether.joe at gmail.com (Sean Felipe Wolfe) Date: Mon, 17 Mar 2014 16:51:27 -0700 Subject: [Python-Dev] trouble building 3.3 In-Reply-To: References: Message-ID: Ah ok, got it. Antoine - I'll try your solution out though just out of curiosity, thanks :) On Mon, Mar 17, 2014 at 2:52 PM, Terry Reedy wrote: > On 3/17/2014 3:57 PM, Sean Felipe Wolfe wrote: >> >> I'm working on some IDLE oriented bugs and I'm having some trouble >> building the 3.3 branch: > > > Starting today, Idle for 3.3 is no more patched. > 2.7, 3.4, and 3.5 will be patched for now. > > > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ether.joe%40gmail.com -- A musician must make music, an artist must paint, a poet must write, if he is to be ultimately at peace with himself. - Abraham Maslow From ether.joe at gmail.com Tue Mar 18 00:53:42 2014 From: ether.joe at gmail.com (Sean Felipe Wolfe) Date: Mon, 17 Mar 2014 16:53:42 -0700 Subject: [Python-Dev] hg branching + log question In-Reply-To: <20140317210730.38e2d118@fsol> References: <20140317210730.38e2d118@fsol> Message-ID: Ah well, ok. That seems pretty counterintuitive to me though. I suppose Hg has its quirks just like ... that other DCVS system ... :P On Mon, Mar 17, 2014 at 1:07 PM, Antoine Pitrou wrote: > On Mon, 17 Mar 2014 13:02:23 -0700 > Sean Felipe Wolfe wrote: >> I'm getting my feet wet with the cpython sources and Mercurial. I'm a >> bit confused -- when I checkout a branch, eg. 3.3, and I do an 'hg >> log', why do I see log messages for other branches? > > This is a classic hg question, you would get the answer by asking > Mercurial for help: hg log --help :) > > Basically, to restrict the log to a given branch, just use the -b > option: hg log -b 3.3. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ether.joe%40gmail.com -- A musician must make music, an artist must paint, a poet must write, if he is to be ultimately at peace with himself. - Abraham Maslow From victor.stinner at gmail.com Tue Mar 18 02:54:43 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 18 Mar 2014 02:54:43 +0100 Subject: [Python-Dev] Python 3.5 now uses surrogateescape for the POSIX locale Message-ID: Hi, I modified Python 3.5 to use the "surrogateescape" error handler (PEP 383) for stdin and stdout when the LC_CTYPE locale is POSIX ("C" locale): http://bugs.python.org/issue19977 New behaviour: --- $ mkdir z $ touch z/abc? $ LC_CTYPE=C ./python -c 'import os; print(os.listdir("z")[0])' abc? --- Old behaviour, before the change (test with Python 3.3): --- $ LC_CTYPE=C python3 -c 'import os; print(os.listdir("z")[0])' Traceback (most recent call last): File "", line 1, in UnicodeEncodeError: 'ascii' codec can't encode characters in position 3-4: ordinal not in range(128) --- The POSIX locale is common because it is used by default when no other locale is set. It's common that programs started by a crontab on UNIX and daemons are using this locale. Victor From g.brandl at gmx.net Tue Mar 18 07:22:02 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 18 Mar 2014 07:22:02 +0100 Subject: [Python-Dev] cpython: Add a stub "whatsnew in 3.5" document. In-Reply-To: <3fnwDj498Xz7LjW@mail.python.org> References: <3fnwDj498Xz7LjW@mail.python.org> Message-ID: Am 18.03.2014 01:27, schrieb victor.stinner: > http://hg.python.org/cpython/rev/daa6bf71170f > changeset: 89835:daa6bf71170f > user: Victor Stinner > date: Tue Mar 18 00:53:32 2014 +0100 > summary: > Add a stub "whatsnew in 3.5" document. > > files: > Doc/whatsnew/3.5.rst | 186 +++++++++++++++++++++++++++++++ > 1 files changed, 186 insertions(+), 0 deletions(-) Nice, but please also add it to the toctree in Doc/whatsnew/index.rst. Georg From fijall at gmail.com Tue Mar 18 08:52:05 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 18 Mar 2014 09:52:05 +0200 Subject: [Python-Dev] Intricacies of calling __eq__ Message-ID: Hi I have a question about calling __eq__ in some cases. We're thinking about doing an optimization where say: if x in d: return d[x] where d is a dict would result in only one dict lookup (the second one being constant folded away). The question is whether it's ok to do it, despite the fact that it changes the semantics on how many times __eq__ is called on x. Cheers, fijal From victor.stinner at gmail.com Tue Mar 18 09:00:44 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 18 Mar 2014 09:00:44 +0100 Subject: [Python-Dev] [python-committers] default hg.python.org/cpython is now 3.5 In-Reply-To: References: <5326984D.1060508@hastings.org> Message-ID: Hi, 2014-03-18 7:19 GMT+01:00 Georg Brandl : > Am 17.03.2014 22:36, schrieb Victor Stinner: >> I modified the Misc/NEWS file: (...) Is that correct? > > The changes not merged in 3.4.0 will all be in 3.5.0; please reinstate the > NEWS entries under the 3.5 heading. Oh ok, I didn't notice that 3.4.1rc1 items were all missing from 3.5. Does it look better with the following changeset? http://hg.python.org/cpython/rev/b7f2b1bd49f3 I compared Misc/NEWS with Misc/NEWS of 3.4 branch: the only change is the 2 new changes specific to 3.5. Victor From victor.stinner at gmail.com Tue Mar 18 09:02:06 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 18 Mar 2014 09:02:06 +0100 Subject: [Python-Dev] cpython: Add a stub "whatsnew in 3.5" document. In-Reply-To: References: <3fnwDj498Xz7LjW@mail.python.org> Message-ID: 2014-03-18 7:22 GMT+01:00 Georg Brandl : > Am 18.03.2014 01:27, schrieb victor.stinner: >> http://hg.python.org/cpython/rev/daa6bf71170f >> changeset: 89835:daa6bf71170f >> user: Victor Stinner >> date: Tue Mar 18 00:53:32 2014 +0100 >> summary: >> Add a stub "whatsnew in 3.5" document. >> >> files: >> Doc/whatsnew/3.5.rst | 186 +++++++++++++++++++++++++++++++ >> 1 files changed, 186 insertions(+), 0 deletions(-) > > Nice, but please also add it to the toctree in Doc/whatsnew/index.rst. Ok, done. Is there a document explaining the steps after a release to prepare the next release? Victor From ja.py at farowl.co.uk Tue Mar 18 09:05:10 2014 From: ja.py at farowl.co.uk (Jeff Allen) Date: Tue, 18 Mar 2014 08:05:10 +0000 Subject: [Python-Dev] hg branching + log question In-Reply-To: References: <20140317210730.38e2d118@fsol> Message-ID: <5327FE36.4080803@farowl.co.uk> I have also found hg difficult to get to grips with from cold (but I like it). The hg command and its help are good, as Antoine says, but if I'm doing something complex, the visualisation of the change sets that TortoiseHG provides is invaluable (and of other invisible structures, such as the MQ patch stack). The context menus are also a clue to what you might want to do next when you can't guess what word comes after hg help ... . I found it helpful to practice extensively on something that doesn't matter. The gap for me is still examples of what I want "done well". Clearly the Python repos represent complex work, but even accepting it is all done well, are without much commentary. This is very good: http://hgbook.red-bean.com/read/ , but there are hints it has not kept up. This also: http://legacy.python.org/dev/peps/pep-0385/ Jeff Allen On 17/03/2014 23:53, Sean Felipe Wolfe wrote: > Ah well, ok. That seems pretty counterintuitive to me though. I > suppose Hg has its quirks just like ... that other DCVS system ... :P > > On Mon, Mar 17, 2014 at 1:07 PM, Antoine Pitrou wrote: >> On Mon, 17 Mar 2014 13:02:23 -0700 >> Sean Felipe Wolfe wrote: >>> I'm getting my feet wet with the cpython sources and Mercurial. I'm a >>> bit confused -- when I checkout a branch, eg. 3.3, and I do an 'hg >>> log', why do I see log messages for other branches? >> This is a classic hg question, you would get the answer by asking >> Mercurial for help: hg log --help :) >> >> Basically, to restrict the log to a given branch, just use the -b >> option: hg log -b 3.3. >> >> Regards >> >> Antoine. >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ether.joe%40gmail.com > > From ncoghlan at gmail.com Tue Mar 18 09:08:08 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 18 Mar 2014 18:08:08 +1000 Subject: [Python-Dev] Python 3.5 now uses surrogateescape for the POSIX locale In-Reply-To: References: Message-ID: On 18 Mar 2014 11:56, "Victor Stinner" wrote: > > Hi, > > I modified Python 3.5 to use the "surrogateescape" error handler (PEP > 383) for stdin and stdout when the LC_CTYPE locale is POSIX ("C" > locale): > http://bugs.python.org/issue19977 Yay, thanks Victor. I'll let the Fedora folks know this has been merged, as we may seriously consider applying this as a vendor patch to our build of Python 3.4 (while I agree this isn't a bug fix, the current behaviour also poses a problem for Fedora as more core utilities start migrating to Python 3). Cheers, Nick. > > New behaviour: > --- > $ mkdir z > $ touch z/abc? > $ LC_CTYPE=C ./python -c 'import os; print(os.listdir("z")[0])' > abc? > --- > > Old behaviour, before the change (test with Python 3.3): > --- > $ LC_CTYPE=C python3 -c 'import os; print(os.listdir("z")[0])' > Traceback (most recent call last): > File "", line 1, in > UnicodeEncodeError: 'ascii' codec can't encode characters in position > 3-4: ordinal not in range(128) > --- > > The POSIX locale is common because it is used by default when no other > locale is set. It's common that programs started by a crontab on UNIX > and daemons are using this locale. > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Tue Mar 18 09:43:01 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 18 Mar 2014 09:43:01 +0100 Subject: [Python-Dev] cpython: Add a stub "whatsnew in 3.5" document. In-Reply-To: References: <3fnwDj498Xz7LjW@mail.python.org> Message-ID: Am 18.03.2014 09:02, schrieb Victor Stinner: > 2014-03-18 7:22 GMT+01:00 Georg Brandl : >> Am 18.03.2014 01:27, schrieb victor.stinner: >>> http://hg.python.org/cpython/rev/daa6bf71170f >>> changeset: 89835:daa6bf71170f >>> user: Victor Stinner >>> date: Tue Mar 18 00:53:32 2014 +0100 >>> summary: >>> Add a stub "whatsnew in 3.5" document. >>> >>> files: >>> Doc/whatsnew/3.5.rst | 186 +++++++++++++++++++++++++++++++ >>> 1 files changed, 186 insertions(+), 0 deletions(-) >> >> Nice, but please also add it to the toctree in Doc/whatsnew/index.rst. > > Ok, done. Is there a document explaining the steps after a release to > prepare the next release? Yes, in PEP 101, but it will never be 100% complete. The step of adding a new whatsnew document could be added, for example. Georg From tjreedy at udel.edu Tue Mar 18 10:05:56 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 18 Mar 2014 05:05:56 -0400 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: Message-ID: On 3/18/2014 3:52 AM, Maciej Fijalkowski wrote: > Hi > > I have a question about calling __eq__ in some cases. > > We're thinking about doing an optimization where say: > > if x in d: > return d[x] if d.__contains__(x): return d.__getitem__(x) I do not see any requirement to call x.__eq__ any particular number of times. The implementation of d might always call somekey.__eq__(x). The concept of sets (and dicts) requires coherent equality comparisons. > where d is a dict would result in only one dict lookup (the second one > being constant folded away). The question is whether it's ok to do it, > despite the fact that it changes the semantics on how many times > __eq__ is called on x. A __eq__ that has side-effects violates the intended and expected semanitics of __eq__. -- Terry Jan Reedy From victor.stinner at gmail.com Tue Mar 18 10:13:58 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 18 Mar 2014 10:13:58 +0100 Subject: [Python-Dev] Python 3.5 now uses surrogateescape for the POSIX locale In-Reply-To: References: Message-ID: 2014-03-18 9:08 GMT+01:00 Nick Coghlan : > On 18 Mar 2014 11:56, "Victor Stinner" wrote: >> >> Hi, >> >> I modified Python 3.5 to use the "surrogateescape" error handler (PEP >> 383) for stdin and stdout when the LC_CTYPE locale is POSIX ("C" >> locale): >> http://bugs.python.org/issue19977 > > Yay, thanks Victor. I'll let the Fedora folks know this has been merged, as > we may seriously consider applying this as a vendor patch to our build of > Python 3.4 (while I agree this isn't a bug fix, the current behaviour also > poses a problem for Fedora as more core utilities start migrating to Python > 3). Please don't cherry-pick this change in Fedora if it is not done in Python 3.4. It changes the behaviour of Python and I would prefer to have the same behaviour on the same Python version on all platforms. I'm not against backporting the change in Python 3.4.1. It can be seen as a bugfix. I don't think that anyone wants a Unicode error when reading or printing non-ASCII data from stdin/to stdout. But I would like the opinion of other developers before doing that. Victor From ncoghlan at gmail.com Tue Mar 18 10:35:15 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 18 Mar 2014 19:35:15 +1000 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: Message-ID: On 18 March 2014 17:52, Maciej Fijalkowski wrote: > Hi > > I have a question about calling __eq__ in some cases. > > We're thinking about doing an optimization where say: > > if x in d: > return d[x] > > where d is a dict would result in only one dict lookup (the second one > being constant folded away). The question is whether it's ok to do it, > despite the fact that it changes the semantics on how many times > __eq__ is called on x. I'll assume the following hold: - we're only talking about true builtin dicts (the similarity between __contains__ and __getitem__ can't be assumed otherwise) - guards will trigger if d is mutated (e.g. by another thread) between the containment check and the item retrieval Semantically, what you propose is roughly equivalent to reinterpreting the look-before-you-leap version to the exception handling based fallback: try: return d[x] except KeyError: pass For a builtin dict and any *reasonable* x, those two operations will behave the same way. Differences arise only if x.__hash__ or x.__eq__ is defined in a way that most people would consider unreasonable. For an optimisation that actually changes the language semantics like that, though, I would expect it to be buying a significant payoff in speed, especially given that most cases where the key lookup is known to be a bottleneck can already be optimised by hand. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Mar 18 10:48:38 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 18 Mar 2014 19:48:38 +1000 Subject: [Python-Dev] Python 3.5 now uses surrogateescape for the POSIX locale In-Reply-To: References: Message-ID: On 18 March 2014 19:13, Victor Stinner wrote: > 2014-03-18 9:08 GMT+01:00 Nick Coghlan : >> On 18 Mar 2014 11:56, "Victor Stinner" wrote: >>> >>> Hi, >>> >>> I modified Python 3.5 to use the "surrogateescape" error handler (PEP >>> 383) for stdin and stdout when the LC_CTYPE locale is POSIX ("C" >>> locale): >>> http://bugs.python.org/issue19977 >> >> Yay, thanks Victor. I'll let the Fedora folks know this has been merged, as >> we may seriously consider applying this as a vendor patch to our build of >> Python 3.4 (while I agree this isn't a bug fix, the current behaviour also >> poses a problem for Fedora as more core utilities start migrating to Python >> 3). > > Please don't cherry-pick this change in Fedora if it is not done in > Python 3.4. It changes the behaviour of Python and I would prefer to > have the same behaviour on the same Python version on all platforms. > > I'm not against backporting the change in Python 3.4.1. It can be seen > as a bugfix. I don't think that anyone wants a Unicode error when > reading or printing non-ASCII data from stdin/to stdout. But I would > like the opinion of other developers before doing that. Well, the concern has always been the risk of silently generating bad data if there is a mismatch between the OS encoding and the stream encodings. That's why it took so long to make this change at all - we had to figure out that the underlying problem was really the ease with which even a properly configured Linux systems could end up running Python 3 code in the POSIX locale, and thus end up with improperly configured standard streams. Enabling "surrogateescape" by default only when the standard stream encoding is "ascii" helps to mitigate that risk, while still dealing with the main problem. I meant to try to get this into 3.4 (since a couple of the Fedora folks convinced me it was a problem), but there are only so many hours in the day, and it took me quite a while to fully grasp the actual problem. If folks are open to backporting this change to 3.4.1, then yes, I'd definitely prefer an upstream solution. Otherwise, it will be up to the Fedora Python maintainers to decide what they want to do. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Tue Mar 18 11:13:29 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 18 Mar 2014 11:13:29 +0100 Subject: [Python-Dev] Python 3.5 now uses surrogateescape for the POSIX locale In-Reply-To: References: Message-ID: 2014-03-18 10:48 GMT+01:00 Nick Coghlan : > Well, the concern has always been the risk of silently generating bad > data if there is a mismatch between the OS encoding and the stream > encodings. Data can be loaded from OS functions, from files and from stdin. These 3 sources may use various different and incompatible encodings. surrogateescape is used by OS functions, and now also by stdin when the POSIX locale is used. When the POSIX locale is used, OS functions and stdin can use different encodings if the PYTHONIOENCODING environment variable is used. Since we are consentent adults, I guess that you understand what you are doing when you set PYTHONIOENCODING. On Windows, the encoding of standard streams is the OEM code page, or the ANSI code page if a stream is redirected, it's unrelated to the LC_CTYPE locale. So surrogateecape can be used when if the encoding of standard streams is not ASCII. We may handle Windows differently to use strict even if the LC_CTYPE locale is "C". Note: On FreeBSD, Solaris and OpenIndiana, nl_langinfo(CODESET) announces an alias of the ASCII encoding when the LC_CTYPE locale is POSIX, whereas mbstowcs() and wcstombs() functions use the ISO-8859-1 encoding. Python 3 now uses the ASCII encoding for its "filesystem" (OS) encoding. Victor From victor.stinner at gmail.com Tue Mar 18 11:15:12 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 18 Mar 2014 11:15:12 +0100 Subject: [Python-Dev] Python 3.5 now uses surrogateescape for the POSIX locale In-Reply-To: References: Message-ID: 2014-03-18 11:02 GMT+01:00 Atsuo Ishimoto : > FYI: Guido was opposed to change error handler of stdin and stdout years ago. > > http://bugs.python.org/issue2630#msg65493 This issue proposes to use "backslashreplace" error handler for stdout. This error handler is very different to "surrogateescape" which is related to PEP 383 and used by all OS functions. Victor From fijall at gmail.com Tue Mar 18 12:11:23 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 18 Mar 2014 13:11:23 +0200 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: Message-ID: On Tue, Mar 18, 2014 at 11:35 AM, Nick Coghlan wrote: > On 18 March 2014 17:52, Maciej Fijalkowski wrote: >> Hi >> >> I have a question about calling __eq__ in some cases. >> >> We're thinking about doing an optimization where say: >> >> if x in d: >> return d[x] >> >> where d is a dict would result in only one dict lookup (the second one >> being constant folded away). The question is whether it's ok to do it, >> despite the fact that it changes the semantics on how many times >> __eq__ is called on x. > > I'll assume the following hold: > > - we're only talking about true builtin dicts (the similarity between > __contains__ and __getitem__ can't be assumed otherwise) yes > - guards will trigger if d is mutated (e.g. by another thread) between > the containment check and the item retrieval yes > > Semantically, what you propose is roughly equivalent to reinterpreting > the look-before-you-leap version to the exception handling based > fallback: > > try: > return d[x] > except KeyError: > pass > > For a builtin dict and any *reasonable* x, those two operations will > behave the same way. Differences arise only if x.__hash__ or x.__eq__ > is defined in a way that most people would consider unreasonable. > > For an optimisation that actually changes the language semantics like > that, though, I would expect it to be buying a significant payoff in > speed, especially given that most cases where the key lookup is known > to be a bottleneck can already be optimised by hand. > > Cheers, > Nick. the payoff is significant. Note that __eq__ might not be called at all (since dicts check identity first). It turns out not all people write reasonable code and we can't expect them to micro-optimize by hand. It also covers cases that are hard to optimize, like: if d[x] > 3: d[x] += 1 etc. From steve at pearwood.info Tue Mar 18 12:18:42 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 18 Mar 2014 22:18:42 +1100 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: Message-ID: <20140318111841.GK16526@ando> On Tue, Mar 18, 2014 at 05:05:56AM -0400, Terry Reedy wrote: > On 3/18/2014 3:52 AM, Maciej Fijalkowski wrote: > >Hi > > > >I have a question about calling __eq__ in some cases. > > > >We're thinking about doing an optimization where say: > > > >if x in d: > > return d[x] > > if d.__contains__(x): return d.__getitem__(x) [Aside: to be pedantic, Python only calls dunder methods on the class, not the instance, in response to operators and other special calls. That is, type(d).__contains__ rather than d.__contains__, etc. And to be even more pedantic, that's only true for new-style classes.] > I do not see any requirement to call x.__eq__ any particular number of > times. The implementation of d might always call somekey.__eq__(x). The > concept of sets (and dicts) requires coherent equality comparisons. To what extent does Python the language specify that user-defined classes must be coherent? How much latitude to shoot oneself in the foot should the language allow? What counts as coherent can depend on the types involved. For instance, I consider IEEE-754 Not-A-Numbers to be coherent, albeit weird. Python goes only so far to accomodate NANs: while it allows a NAN to test unequal even to itself (`NAN == NAN` returns False), containers are allowed to assume that instances are equal to themselves (`NAN in {NAN}` returns True). This was discussed in great detail a few years ago, and if I recall correctly, the conclusion was that containers can assume that their elements are reflexive (they equal themselves), but equality == cannot make the same assumption and bypass calling __eq__. > >where d is a dict would result in only one dict lookup (the second one > >being constant folded away). The question is whether it's ok to do it, > >despite the fact that it changes the semantics on how many times > >__eq__ is called on x. > > A __eq__ that has side-effects violates the intended and expected > semanitics of __eq__. Nevertheless, an __eq__ with side-effects is legal Python and may in fact be useful. It's a tricky one... I don't know how I feel about short-cutting normal Python semantics for speed. On the one hand, faster is good. But on the other hand, it makes it harder to reason about code when things go wrong. "Why is my __eq__ method not being called?" -- Steven From fijall at gmail.com Tue Mar 18 12:21:05 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 18 Mar 2014 13:21:05 +0200 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: <20140318111841.GK16526@ando> References: <20140318111841.GK16526@ando> Message-ID: On Tue, Mar 18, 2014 at 1:18 PM, Steven D'Aprano wrote: > On Tue, Mar 18, 2014 at 05:05:56AM -0400, Terry Reedy wrote: >> On 3/18/2014 3:52 AM, Maciej Fijalkowski wrote: >> >Hi >> > >> >I have a question about calling __eq__ in some cases. >> > >> >We're thinking about doing an optimization where say: >> > >> >if x in d: >> > return d[x] >> >> if d.__contains__(x): return d.__getitem__(x) > > [Aside: to be pedantic, Python only calls dunder methods on the class, > not the instance, in response to operators and other special calls. That > is, type(d).__contains__ rather than d.__contains__, etc. And to be even > more pedantic, that's only true for new-style classes.] > > >> I do not see any requirement to call x.__eq__ any particular number of >> times. The implementation of d might always call somekey.__eq__(x). The >> concept of sets (and dicts) requires coherent equality comparisons. > > To what extent does Python the language specify that user-defined > classes must be coherent? How much latitude to shoot oneself in the foot > should the language allow? > > What counts as coherent can depend on the types involved. For instance, > I consider IEEE-754 Not-A-Numbers to be coherent, albeit weird. Python > goes only so far to accomodate NANs: while it allows a NAN to test > unequal even to itself (`NAN == NAN` returns False), containers are > allowed to assume that instances are equal to themselves (`NAN in {NAN}` > returns True). This was discussed in great detail a few years ago, and > if I recall correctly, the conclusion was that containers can assume > that their elements are reflexive (they equal themselves), but equality > == cannot make the same assumption and bypass calling __eq__. > > >> >where d is a dict would result in only one dict lookup (the second one >> >being constant folded away). The question is whether it's ok to do it, >> >despite the fact that it changes the semantics on how many times >> >__eq__ is called on x. >> >> A __eq__ that has side-effects violates the intended and expected >> semanitics of __eq__. > > Nevertheless, an __eq__ with side-effects is legal Python and may in > fact be useful. > > It's a tricky one... I don't know how I feel about short-cutting normal > Python semantics for speed. On the one hand, faster is good. But on the > other hand, it makes it harder to reason about code when things go > wrong. "Why is my __eq__ method not being called?" > > > -- > Steven > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com note that this is specifically about dicts, where __eq__ will be called undecided number of times anyway (depending on collisions in hash/buckets which is implementation specific to start with) From steve at pearwood.info Tue Mar 18 15:21:16 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 19 Mar 2014 01:21:16 +1100 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: <20140318111841.GK16526@ando> Message-ID: <20140318142114.GM16526@ando> On Tue, Mar 18, 2014 at 01:21:05PM +0200, Maciej Fijalkowski wrote: > note that this is specifically about dicts, where __eq__ will be > called undecided number of times anyway (depending on collisions in > hash/buckets which is implementation specific to start with) Exactly. Using a __eq__ method with side-effects is a good way to find out how many collisions your dict has :-) But specifically with your example, if x in d: return d[x] my sense of this is that it falls into the same conceptual area as the identity optimization for checking list or set containment: slightly unclean, but justified. Provided d is an actual built-in dict, and it hasn't been modified between one call and the next, I think it would be okay to optimize the second lookup d[x]. A question: how far away will this optimization apply? if x in d: do_this() do_that() do_something_else() spam = d[x] Assuming no modifications to d, will the second lookup still be optimized? -- Steven From ishimoto at gembook.org Tue Mar 18 11:02:14 2014 From: ishimoto at gembook.org (Atsuo Ishimoto) Date: Tue, 18 Mar 2014 19:02:14 +0900 Subject: [Python-Dev] Python 3.5 now uses surrogateescape for the POSIX locale In-Reply-To: References: Message-ID: Hello, 2014-03-18 18:13 GMT+09:00 Victor Stinner : > I'm not against backporting the change in Python 3.4.1. It can be seen > as a bugfix. I don't think that anyone wants a Unicode error when > reading or printing non-ASCII data from stdin/to stdout. But I would > like the opinion of other developers before doing that. FYI: Guido was opposed to change error handler of stdin and stdout years ago. http://bugs.python.org/issue2630#msg65493 > Amaury: I think it would be okay to use backslashreplace as the default > error handler for sys.stderr. Probably not for sys.stdout or other > files, since I'm sure many users prefer the errors when their data > cannot be printed rather than silently writing \u escapes that might > cause other code reading their output to choke. For sys.stderr though I > think not having exceptions raised when attempting to print errors is > very valuable. -- Atsuo Ishimoto Mail: ishimoto at gembook.org Twitter: atsuoishimoto From xsalort at aldebaran-robotics.com Mon Mar 17 23:05:26 2014 From: xsalort at aldebaran-robotics.com (Xavier Salort) Date: Mon, 17 Mar 2014 18:05:26 -0400 Subject: [Python-Dev] Call for Python Developers for our humanoid Robot NAO Message-ID: Hi, We are the manufacturer of the humanoid robot NAO : http://www.youtube.com/watch?v=nNbj2G3GmAo We are now offering great opportunities for developers to use NAO and would like to get in touch with you and your members : http://www.youtube.com/watch?v=_AxErdP0YI8 Let me know if you may be interested. Thank you! Xavier -- Xavier Salort Area Sales Manager M + 1 857 247 7777 -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Mar 18 17:44:28 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 18 Mar 2014 17:44:28 +0100 Subject: [Python-Dev] Call for Python Developers for our humanoid Robot NAO In-Reply-To: References: Message-ID: Hello Xavier, It is not obvious your message is appropriate for python-dev. It looks like mere advertising; if it is not, please explain. To clarify what this mailing-list is about: "On this list the key Python developers discuss the future of the language and its implementation. Topics include Python design issues, release mechanics, and maintenance of existing releases." (from https://mail.python.org/mailman/listinfo/python-dev) Regards Antoine. Le 17/03/2014 23:05, Xavier Salort a ?crit : > Hi, > > We are the manufacturer of the humanoid robot NAO : > http://www.youtube.com/watch?v=nNbj2G3GmAo > We are now offering great opportunities for developers to use NAO and > would like to get in touch with you and your members : > http://www.youtube.com/watch?v=_AxErdP0YI8 > > Let me know if you may be interested. > Thank you! > > Xavier > -- > Xavier Salort > Area Sales Manager > M + 1 857 247 7777 > > > From fuzzyman at voidspace.org.uk Tue Mar 18 17:46:45 2014 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 18 Mar 2014 16:46:45 +0000 Subject: [Python-Dev] Call for Python Developers for our humanoid Robot NAO In-Reply-To: References: Message-ID: <53287875.9090802@voidspace.org.uk> On 18/03/14 16:44, Antoine Pitrou wrote: > > Hello Xavier, > > It is not obvious your message is appropriate for python-dev. It looks > like mere advertising; if it is not, please explain. > > To clarify what this mailing-list is about: "On this list the key > Python developers discuss the future of the language and its > implementation. Topics include Python design issues, release > mechanics, and maintenance of existing releases." > > (from https://mail.python.org/mailman/listinfo/python-dev) > Unless you're offering all the core-devs free robots. In which case it's fine. Michael > Regards > > Antoine. > > > Le 17/03/2014 23:05, Xavier Salort a ?crit : >> Hi, >> >> We are the manufacturer of the humanoid robot NAO : >> http://www.youtube.com/watch?v=nNbj2G3GmAo >> We are now offering great opportunities for developers to use NAO and >> would like to get in touch with you and your members : >> http://www.youtube.com/watch?v=_AxErdP0YI8 >> >> Let me know if you may be interested. >> Thank you! >> >> Xavier >> -- >> Xavier Salort >> Area Sales Manager >> M + 1 857 247 7777 >> >> >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk From merwok at netwok.org Tue Mar 18 18:20:09 2014 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 18 Mar 2014 13:20:09 -0400 Subject: [Python-Dev] hg branching + log question In-Reply-To: <5327FE36.4080803@farowl.co.uk> References: <20140317210730.38e2d118@fsol> <5327FE36.4080803@farowl.co.uk> Message-ID: <53288049.9090004@netwok.org> Le 18/03/2014 04:05, Jeff Allen a ?crit : > I found it helpful to practice extensively on something that doesn't > matter. The gap for me is still examples of what I want "done well". > Clearly the Python repos represent complex work, but even accepting it > is all done well, are without much commentary. This is very good: > http://hgbook.red-bean.com/read/ , but there are hints it has not kept up. > > This also: http://legacy.python.org/dev/peps/pep-0385/ Does the developer?s guide help? http://docs.python.org/devguide/patch.html http://docs.python.org/devguide/faq.html#version-control Regards From christian at python.org Tue Mar 18 19:21:52 2014 From: christian at python.org (Christian Heimes) Date: Tue, 18 Mar 2014 19:21:52 +0100 Subject: [Python-Dev] Call for Python Developers for our humanoid Robot NAO In-Reply-To: <53287875.9090802@voidspace.org.uk> References: <53287875.9090802@voidspace.org.uk> Message-ID: <53288EC0.90608@python.org> On 18.03.2014 17:46, Michael Foord wrote: > > On 18/03/14 16:44, Antoine Pitrou wrote: >> >> Hello Xavier, >> >> It is not obvious your message is appropriate for python-dev. It looks >> like mere advertising; if it is not, please explain. >> >> To clarify what this mailing-list is about: "On this list the key >> Python developers discuss the future of the language and its >> implementation. Topics include Python design issues, release >> mechanics, and maintenance of existing releases." >> >> (from https://mail.python.org/mailman/listinfo/python-dev) >> > > Unless you're offering all the core-devs free robots. In which case it's > fine. Sounds like a good deal to me. :) Can the NAO bot do The Silly Walk (tm), too? I'm even willing to film and upload movies of NAO vs. my cats... :) Christian From storchaka at gmail.com Tue Mar 18 19:29:35 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 18 Mar 2014 20:29:35 +0200 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: References: Message-ID: 26.02.14 11:40, Serhiy Storchaka ???????(??): > Let's choose the least confusing names. > > See discussions at: > > http://bugs.python.org/issue3081 > http://bugs.python.org/issue16447 > http://bugs.python.org/issue20440 > http://comments.gmane.org/gmane.comp.python.devel/145346 > Poll results: Py_(X)SETREF: +3 (Antoine, Kristj?n, Nick) Py_(X)DECREC_REPLACE: +3 (Ryan, Georg, Larry) -2 (Antoine, Kristj?n) Py_(X)ASSIGN, Py_REF_ASSIGN, Py_(X)REPLACE, Py_(X)STORE, Py_SET_AND_(X)DECREF, Py_(X)DECREF_AND_ASSIGN, Py_ASSIGN_AND_(X)DECREF: -1 (Antoine or Kristj?n) Py_CLEAR_AND_SET: -2 (Antoine, Kristj?n) From barry at python.org Tue Mar 18 19:37:06 2014 From: barry at python.org (Barry Warsaw) Date: Tue, 18 Mar 2014 14:37:06 -0400 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: References: Message-ID: <20140318143706.6cc3782f@anarchist.wooz.org> On Mar 18, 2014, at 08:29 PM, Serhiy Storchaka wrote: >Poll results: > >Py_(X)SETREF: +3 (Antoine, Kristj?n, Nick) +1 -Barry From solipsis at pitrou.net Tue Mar 18 19:47:40 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 18 Mar 2014 19:47:40 +0100 Subject: [Python-Dev] Call for Python Developers for our humanoid Robot NAO In-Reply-To: <53288EC0.90608@python.org> References: <53287875.9090802@voidspace.org.uk> <53288EC0.90608@python.org> Message-ID: <20140318194740.74e69599@fsol> On Tue, 18 Mar 2014 19:21:52 +0100 Christian Heimes wrote: > > Sounds like a good deal to me. :) Can the NAO bot do The Silly Walk > (tm), too? I'm even willing to film and upload movies of NAO vs. my > cats... :) I've heard it can mend a buildbot with its own hands. Regards Antoine. From g.brandl at gmx.net Tue Mar 18 20:05:47 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 18 Mar 2014 20:05:47 +0100 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: References: Message-ID: Am 18.03.2014 19:29, schrieb Serhiy Storchaka: > 26.02.14 11:40, Serhiy Storchaka ???????(??): >> Let's choose the least confusing names. >> >> See discussions at: >> >> http://bugs.python.org/issue3081 >> http://bugs.python.org/issue16447 >> http://bugs.python.org/issue20440 >> http://comments.gmane.org/gmane.comp.python.devel/145346 >> > > Poll results: > > Py_(X)SETREF: +3 (Antoine, Kristj?n, Nick) Well, now that I know that -1 are counted, -1 to this. > Py_(X)DECREC_REPLACE: +3 (Ryan, Georg, Larry) -2 (Antoine, Kristj?n) > > Py_(X)ASSIGN, Py_REF_ASSIGN, Py_(X)REPLACE, Py_(X)STORE, > Py_SET_AND_(X)DECREF, Py_(X)DECREF_AND_ASSIGN, Py_ASSIGN_AND_(X)DECREF: > -1 (Antoine or Kristj?n) > > Py_CLEAR_AND_SET: -2 (Antoine, Kristj?n) Georg From larry at hastings.org Tue Mar 18 20:08:13 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 18 Mar 2014 12:08:13 -0700 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: References: Message-ID: <5328999D.10701@hastings.org> On 03/18/2014 12:05 PM, Georg Brandl wrote: > Am 18.03.2014 19:29, schrieb Serhiy Storchaka: >> 26.02.14 11:40, Serhiy Storchaka ???????(??): >>> Let's choose the least confusing names. >>> >>> See discussions at: >>> >>> http://bugs.python.org/issue3081 >>> http://bugs.python.org/issue16447 >>> http://bugs.python.org/issue20440 >>> http://comments.gmane.org/gmane.comp.python.devel/145346 >>> >> Poll results: >> >> Py_(X)SETREF: +3 (Antoine, Kristj?n, Nick) > Well, now that I know that -1 are counted, -1 to this. Me too, -1 to Py_SETREF / Py_XSETREF. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Tue Mar 18 20:42:44 2014 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 18 Mar 2014 15:42:44 -0400 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: <20140318111841.GK16526@ando> References: <20140318111841.GK16526@ando> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/18/2014 07:18 AM, Steven D'Aprano wrote: > Nevertheless, an __eq__ with side-effects is legal Python and may in > fact be useful. E.g., for an LRU usecase. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlMoobQACgkQ+gerLs4ltQ62agCaA6Z++yLV6VkJ5VUu8Q38AP2N TfcAnjWyYE6/taksZQ0J3igJfv9+KhiK =/D19 -----END PGP SIGNATURE----- From fijall at gmail.com Tue Mar 18 20:46:08 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 18 Mar 2014 21:46:08 +0200 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: <20140318142114.GM16526@ando> References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> Message-ID: On Tue, Mar 18, 2014 at 4:21 PM, Steven D'Aprano wrote: > On Tue, Mar 18, 2014 at 01:21:05PM +0200, Maciej Fijalkowski wrote: > >> note that this is specifically about dicts, where __eq__ will be >> called undecided number of times anyway (depending on collisions in >> hash/buckets which is implementation specific to start with) > > Exactly. Using a __eq__ method with side-effects is a good way to find > out how many collisions your dict has :-) > > But specifically with your example, > > if x in d: > return d[x] > > my sense of this is that it falls into the same conceptual area as the > identity optimization for checking list or set containment: slightly > unclean, but justified. Provided d is an actual built-in dict, and it > hasn't been modified between one call and the next, I think it would be > okay to optimize the second lookup d[x]. > > A question: how far away will this optimization apply? > > if x in d: > do_this() > do_that() > do_something_else() > spam = d[x] it depends what those functions do. JIT will inline them and if they're small, it should work (although a modification of a different dict is illegal, since aliasing is not proven), but at some point it'll give up (Note that it'll also give up with a call to C releasing GIL since some other thread can modify it). From tjreedy at udel.edu Tue Mar 18 21:47:29 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 18 Mar 2014 16:47:29 -0400 Subject: [Python-Dev] Call for Python Developers for our humanoid Robot NAO In-Reply-To: References: Message-ID: On 3/18/2014 12:44 PM, Antoine Pitrou wrote: > Hello Xavier, > > It is not obvious your message is appropriate for python-dev. It looks > like mere advertising; The wording is typical for slightly indirect sales pitches in English. The same non-specific (no language mentioned) message was sent to python-list and, I presume, to tens or even hundreds of other lists (with language X substituted for 'Python' in the subject). For python-list, it was caught for moderation and I discarded it. > Le 17/03/2014 23:05, Xavier Salort a ?crit : >> Hi, >> >> We are the manufacturer of the humanoid robot NAO : >> http://www.youtube.com/watch?v=nNbj2G3GmAo >> We are now offering great opportunities for developers to use NAO and >> would like to get in touch with you and your members : >> http://www.youtube.com/watch?v=_AxErdP0YI8 >> >> Let me know if you may be interested. >> Thank you! >> >> Xavier >> -- >> Xavier Salort >> Area Sales Manager >> M + 1 857 247 7777 -- Terry Jan Reedy From ncoghlan at gmail.com Tue Mar 18 22:06:41 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 Mar 2014 07:06:41 +1000 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: <5328999D.10701@hastings.org> References: <5328999D.10701@hastings.org> Message-ID: On 19 Mar 2014 05:10, "Larry Hastings" wrote: > > On 03/18/2014 12:05 PM, Georg Brandl wrote: >> >> Am 18.03.2014 19:29, schrieb Serhiy Storchaka: >>> >>> 26.02.14 11:40, Serhiy Storchaka ???????(??): >>>> >>>> Let's choose the least confusing names. >>>> >>>> See discussions at: >>>> >>>> http://bugs.python.org/issue3081 >>>> http://bugs.python.org/issue16447 >>>> http://bugs.python.org/issue20440 >>>> http://comments.gmane.org/gmane.comp.python.devel/145346 >>>> >>> >>> Poll results: >>> >>> Py_(X)SETREF: +3 (Antoine, Kristj?n, Nick) >> >> Well, now that I know that -1 are counted, -1 to this. > > > Me too, -1 to Py_SETREF / Py_XSETREF. I think we're going to have to resort to BDFL appeal on this one - consensus seems unlikely at this point. Cheers, Nick. > > > /arry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Mar 18 22:19:06 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 18 Mar 2014 21:19:06 +0000 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> Message-ID: On 18 March 2014 19:46, Maciej Fijalkowski wrote: >> A question: how far away will this optimization apply? >> >> if x in d: >> do_this() >> do_that() >> do_something_else() >> spam = d[x] > > it depends what those functions do. JIT will inline them and if > they're small, it should work (although a modification of a different > dict is illegal, since aliasing is not proven), but at some point > it'll give up (Note that it'll also give up with a call to C releasing > GIL since some other thread can modify it). Surely in the presence of threads the optimisation is invalid anyway as other threads can run in between each opcode (I don't know how you'd phrase that in a way that wasn't language dependent other than "everywhere" :-)) so if x in d: # HERE spam = d[x] d can be modified at HERE. (If d is a local variable, obviously the chance that another thread has access to d is a lot lower, but do you really do that level of alias tracking?) Paul From greg.ewing at canterbury.ac.nz Tue Mar 18 22:25:15 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 19 Mar 2014 10:25:15 +1300 Subject: [Python-Dev] Call for Python Developers for our humanoid Robot NAO In-Reply-To: <53288EC0.90608@python.org> References: <53287875.9090802@voidspace.org.uk> <53288EC0.90608@python.org> Message-ID: <5328B9BB.6050702@canterbury.ac.nz> Christian Heimes wrote: > Can the NAO bot do The Silly Walk (tm), too? From what I gather, making robots do *non*-silly walks is the hard part. -- Greg From storchaka at gmail.com Tue Mar 18 22:31:31 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 18 Mar 2014 23:31:31 +0200 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: References: Message-ID: 26.02.14 11:40, Serhiy Storchaka ???????(??): > Let's choose the least confusing names. > > See discussions at: > > http://bugs.python.org/issue3081 > http://bugs.python.org/issue16447 > http://bugs.python.org/issue20440 > http://comments.gmane.org/gmane.comp.python.devel/145346 > Updated poll results. There are two leaders: Py_(X)SETREF (originally proposed by Antoine in issue3081): +4 (Antoine, Kristj?n, Nick, Barry) -2 (Georg, Larry) Py_(X)DECREC_REPLACE (originally proposed by Victor in issue16447): +3 (Ryan, Georg, Larry) -2 (Antoine, Kristj?n) From stackoverflow at mrabarnett.plus.com Tue Mar 18 22:19:26 2014 From: stackoverflow at mrabarnett.plus.com (MRAB) Date: Tue, 18 Mar 2014 21:19:26 +0000 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: References: <5328999D.10701@hastings.org> Message-ID: <5328B85E.9090305@mrabarnett.plus.com> On 2014-03-18 21:06, Nick Coghlan wrote: > > On 19 Mar 2014 05:10, "Larry Hastings" > wrote: > > > > On 03/18/2014 12:05 PM, Georg Brandl wrote: > >> > >> Am 18.03.2014 19:29, schrieb Serhiy Storchaka: > >>> > >>> 26.02.14 11:40, Serhiy Storchaka ???????(??): > >>>> > >>>> Let's choose the least confusing names. > >>>> > >>>> See discussions at: > >>>> > >>>> http://bugs.python.org/issue3081 > >>>> http://bugs.python.org/issue16447 > >>>> http://bugs.python.org/issue20440 > >>>> http://comments.gmane.org/gmane.comp.python.devel/145346 > >>>> > >>> > >>> Poll results: > >>> > >>> Py_(X)SETREF: +3 (Antoine, Kristj?n, Nick) > >> > >> Well, now that I know that -1 are counted, -1 to this. > > > > > > Me too, -1 to Py_SETREF / Py_XSETREF. > > I think we're going to have to resort to BDFL appeal on this one - > consensus seems unlikely at this point. > FWIW, I haven't been following the discussion, but, after a (very) quick look, to me: Py_XDECREF(ptr); ptr = NULL; would be clearer as: Py_CLEAR_REF(ptr); and: Py_XDECREF(ptr); ptr = new_value; would be clearer as: Py_REPLACE_REF(ptr, new_value); From ncoghlan at gmail.com Tue Mar 18 23:26:14 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 Mar 2014 08:26:14 +1000 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: <5328B85E.9090305@mrabarnett.plus.com> References: <5328999D.10701@hastings.org> <5328B85E.9090305@mrabarnett.plus.com> Message-ID: On 19 Mar 2014 07:34, "MRAB" wrote: > FWIW, I haven't been following the discussion, Note that this about correctness, not just clarity - using DECREF on member attributes is not safe, as you may end up exposing a partially destroyed object to other code. > but, after a (very) > quick look, to me: > > Py_XDECREF(ptr); > ptr = NULL; > > would be clearer as: > > Py_CLEAR_REF(ptr); Already exists as Py_CLEAR (with the correct temp variable usage). > and: > > Py_XDECREF(ptr); > ptr = new_value; > > would be clearer as: > > Py_REPLACE_REF(ptr, new_value); That is indeed the one we're aiming to find a suitable name for. Cheers, Nick. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Mar 19 00:23:04 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 18 Mar 2014 16:23:04 -0700 Subject: [Python-Dev] Call for Python Developers for our humanoid Robot NAO In-Reply-To: References: Message-ID: On Tue, Mar 18, 2014 at 1:47 PM, Terry Reedy wrote: > The wording is typical for slightly indirect sales pitches in English. > yeah, it's spam. > The same non-specific (no language mentioned) message was sent to > python-list and, I presume, to tens or even hundreds of other lists (with > language X substituted for 'Python' in the subject). For what it's worth, in the promotional video, they have folks writing python code -- so I think this is in fact Python-relevant. That doesn't make it appropriate for this list, but I think a programmable-in-python robot is pretty cool, in any case. Unless they have different versions of that video with different programming langues edited in -- that would be quite impressive, though! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From kmod at dropbox.com Wed Mar 19 00:42:29 2014 From: kmod at dropbox.com (Kevin Modzelewski) Date: Tue, 18 Mar 2014 16:42:29 -0700 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: <20140318142114.GM16526@ando> References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> Message-ID: My 2 cents: it feels like a slippery slope to start guaranteeing the number and ordering of calls to comparison functions -- for instance, doing that for the sort() function would lock in the sort implementation. It feels like the number/ordering of the calls should be "implementation-defined" in the same way that dict iteration order is, or comparisons between incomparable types; I don't think all of CPython's behavior should be guaranteed as part of the language semantics, and that this kind of change wouldn't have to necessarily represent "changing the semantics" if the semantics weren't considered guaranteed in the first place. On the other hand, even if it's theoretically not guaranteed that the sort() function calls __lt__specifically, in practice it'd be a bad idea for an alternative implementation to call __gt__ instead, so I guess there's some aspect of judgment as to what parts should be allowed to be CPython-defined (though my personal take is that this doesn't apply to this particular case). On Tue, Mar 18, 2014 at 7:21 AM, Steven D'Aprano wrote: > On Tue, Mar 18, 2014 at 01:21:05PM +0200, Maciej Fijalkowski wrote: > > > note that this is specifically about dicts, where __eq__ will be > > called undecided number of times anyway (depending on collisions in > > hash/buckets which is implementation specific to start with) > > Exactly. Using a __eq__ method with side-effects is a good way to find > out how many collisions your dict has :-) > > But specifically with your example, > > if x in d: > return d[x] > > my sense of this is that it falls into the same conceptual area as the > identity optimization for checking list or set containment: slightly > unclean, but justified. Provided d is an actual built-in dict, and it > hasn't been modified between one call and the next, I think it would be > okay to optimize the second lookup d[x]. > > A question: how far away will this optimization apply? > > if x in d: > do_this() > do_that() > do_something_else() > spam = d[x] > > Assuming no modifications to d, will the second lookup still be > optimized? > > > > -- > Steven > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/kmod%40dropbox.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Mar 19 02:09:49 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 19 Mar 2014 12:09:49 +1100 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> Message-ID: <20140319010948.GN16526@ando> On Tue, Mar 18, 2014 at 04:42:29PM -0700, Kevin Modzelewski wrote: > My 2 cents: it feels like a slippery slope to start guaranteeing the number > and ordering of calls to comparison functions -- for instance, doing that > for the sort() function would lock in the sort implementation. Although I agree with your conclusion, I'm not so sure I agree with the way you reach that conclusion. (Actually, I'm not even sure I agree with my own reasoning!) The problem here isn't that Maciej wants to change the implementation of some method or function, like sort. The problem is that (as I understand it) Maciej wants a blessing to change the semantics of multiple Python statements. Currently, the code: if key in dict: return dict[key] performs two dictionary lookups. If you read the code, you can see the two lookups: "key in dict" performs a lookup, and "dict[key]" performs a lookup. Sorry to belabour the obvious, but this gets to the core of the matter. Often this won't matter, but each lookup involves a call to __hash__ and some variable number of calls to __eq__. (Again, as I understand it) Maciej wants to optimize away the second lookup, so that even if you write code like the above, what actually gets executed (modulo guards for modifications to dicts, multiple threads running, etc.) is very different, closer to this in semantics: try: _temp = dict[key] except KeyError: pass else: return _temp I'm not suggesting that PyPy actually will translate the code exactly like this, only that this will be the semantics. The critical point here is that in the Python code you write, there are two *separate* lookups, but in the code that is actually executed, there is only one. Maciej, is my analysis of what you are doing correct? Although I have tentatively said I think this is okay, it is a change in actual semantics of Python code: what you write is no longer what gets run. That makes this *very* different from changing the implementation of sort -- by analogy, its more like changing the semantics of a = f(x) + f(x) to only call f(x) once. I don't think you would call that an implementation detail, would you? Even if justified -- f(x) is a pure, deterministic function with no side-effects -- it would still be a change to the high-level behaviour of the code. Since this proposal is limited only to built-in dicts in scenarios where they cannot be modified between the two lookups, I think that it will be okay. There's no language guarantee as to the number of times that __eq__ will be called (although I would be surprised if __hash__ isn't called twice). But I worry that I have missed something. -- Steven From tjreedy at udel.edu Wed Mar 19 02:29:05 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 18 Mar 2014 21:29:05 -0400 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: <20140319010948.GN16526@ando> References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> <20140319010948.GN16526@ando> Message-ID: On 3/18/2014 9:09 PM, Steven D'Aprano wrote: > Currently, the code: > > if key in dict: > return dict[key] > > > performs two dictionary lookups. If you read the code, you can see the > two lookups: "key in dict" performs a lookup, and "dict[key]" performs a > lookup. The doc says that @deco def f(): pass is equivalent to def f(): pass f = deco(f) That clearly implies two assignments to the containing namespace. But CPython only does one. It actually does something like def (): pass f = deco() This can be detected by defining deco to have the side-effect of checking the local namespace for a binding to 'f' (actually, .__name__) and reporting via print() before returning. When someone did detect the difference and filed a bug report, Guido declared that the difference did not matter and that the current behavior is close enough to meeting the as-if rule. Optimizing away a dict lookup strikes me as similar to optimizing away a namespace binding, which is typically a dict setting, which would involve a dict lookup. In fact, I imagine that your test class with a __eq__ method that reports calls would be an alternate way to detect the @deco optimization. -- Terry Jan Reedy From wtz_wh at foxmail.com Wed Mar 19 02:50:50 2014 From: wtz_wh at foxmail.com (=?gb18030?B?sbGx+dHz?=) Date: Wed, 19 Mar 2014 09:50:50 +0800 Subject: [Python-Dev] Issues about relative& absolute import way for Porting from python2.4 to python2.7 Message-ID: Dear, I just wrote a sample like this: testPy/ __init__.py client.py SoamFactory.c SoamFactory.so soamapi.py sample/testP.py export PYTHONPATH=$(TEST_LOCATION):$(TEST_LOCATION)/testPy Here's the source codes: __init__.py: import client client.py import soamapi class Client(soamapi.SessionCallback): def __init__(self): print "----class Client----" super(Client, self).__init__() soamapi.initialize() def create_session(self): sec_cb = soamapi.DefaultSecurityCallback() self.connection = soamapi.connect(sec_cb) soamapi.py import SoamFactory class ConnectionSecurityCallback(object): def __init__(self): print "----class ConnectionSecurityCallback----" def __del__(self): pass def test_P(self): print "test_P in class ConnectionSecurityCallback" class DefaultSecurityCallback(ConnectionSecurityCallback): def __init__(self): super(DefaultSecurityCallback, self).__init__() print "----class DefaultSecurityCallback----" def __del__(self): super(DefaultSecurityCallback, self).__del__() def test(self): print "test in class DefaultSecurityCallback" class SessionCallback(object): def __init__(self): pass def __del__(self): pass def connect(security_callback): return SoamFactory.SoamFactory_connect(security_callback) def initialize(): SoamFactory.SoamFactory_initialize() print "call soamapi" SoamFactory.c #include "Python.h" #include "PythonLoop.c" PyObject* SOAM_API_MODULE = NULL; PyObject* pyModule = NULL; static PyObject* SoamFactory_initialize(PyObject* self, PyObject* args){ PyEval_InitThreads(); init(); Py_RETURN_NONE; } static PyObject* SoamFactory_connect(PyObject* self, PyObject* args){ PyObject* pySecCallback = NULL; int ok = PyArg_ParseTuple(args, "O", &pySecCallback); if (!ok){ printf("parse tuple error!\n"); Py_RETURN_NONE; } if (! PyObject_IsInstance(pySecCallback, connectSecCallback)){ printf("pySecCallback is not an instance of ConnectionSecurityCallback!\n"); Py_RETURN_NONE; } printf("Successful!\n"); Py_RETURN_NONE; } static PyMethodDef SoamFactory[] = { {"SoamFactory_connect", SoamFactory_connect, METH_VARARGS, "connection function"}, {"SoamFactory_initialize", SoamFactory_initialize, METH_VARARGS, "SoamFactory_initialize"}, {NULL, NULL} }; void initSoamFactory(){ PyEval_InitThreads(); Py_Initialize(); pyModule = Py_InitModule("SoamFactory", SoamFactory); SOAM_API_MODULE = PyImport_ImportModule("soamapi"); } sample/testP.py import testPy print "========================================" submitter = testPy.client.Client() submitter.create_session() print "========================================" When I ran it on python2.4, it worked well, and the result was call soamapi after import soamapi------client.py ======================================== ----class Client---- ----class ConnectionSecurityCallback---- ----class DefaultSecurityCallback---- Successful! ======================================== But when I ran it on python2.7, it worked beyond my expectation, the result was call soamapi call soamapi ======================================== ----class Client---- ----class ConnectionSecurityCallback---- ----class DefaultSecurityCallback---- pySecCallback is not an instance of ConnectionSecurityCallback! ======================================== I found that soamapi was imported twice, and I investigated this is related to absolute&relative import way. PyImport_ImportModule in python2.7 uses absolute import way, it will look up sys.path to get soamapi module, and when testP.py file import testPy module, it will find local module soamapi under testPy package, and binds module's name to package, as testPy.soamapi. There are two ways to correct it for python2.7, 1) Don't use import testPy, use import client directly to avoid using relative; 2) Use from __future__ import absolute_import to enable absolute import feature. But there are two Pre-conditions: 1) Should not modify testP.py; 2) Should be ran on both python2.4 and 2.7. I don't know how to fix it. Is there any official way about how to porting this scenario or better idea? Thanks, Vatel -------------- next part -------------- An HTML attachment was scrubbed... URL: From kmod at dropbox.com Wed Mar 19 03:24:06 2014 From: kmod at dropbox.com (Kevin Modzelewski) Date: Tue, 18 Mar 2014 19:24:06 -0700 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: <20140319010948.GN16526@ando> References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> <20140319010948.GN16526@ando> Message-ID: I think in this case, though, if we say for the sake of argument that the guaranteed semantics of a dictionary lookup are zero or more calls to __hash__ plus zero or more calls to __eq__, then two back-to-back dictionary lookups wouldn't have any observable differences from doing only one, unless you start to make assumptions about the behavior of the implementation. To me there seems to be a bit of a gap between seeing a dictionary lookup and knowing the exact sequence of user-functions that get called, far more than for example something like "a < b". I would feel differently if the question was if it's ok to fold something like x = a < b y = a < b into a single comparison, since I'd agree with the way you described it that you look at this code and would expect __lt__ to be called twice. I guess maybe I just disagree about whether dictionaries are contractually bound to call __hash__ and __eq__? For instance, maybe the dict could have a small cache of recently-looked-up elements to skip hashing / equality tests if they get accessed again; I have no idea if that would be profitable or not, but it seems like that would be a roughly-equivalent change that's still "doing two dictionary lookups" except the second one simply wouldn't call back into the user's python code. Or maybe the dictionary could be implemented as a simple list for small sizes and skip calling __hash__ until it decides to switch to a hash-table strategy; again I'd still say it's "doing the lookups" but it just calls a different set of python functions. > Although I have tentatively said I think this is okay, it is a change in > actual semantics of Python code: what you write is no longer what gets > run. That makes this *very* different from changing the implementation > of sort -- by analogy, its more like changing the semantics of > > a = f(x) + f(x) > > to only call f(x) once. I don't think you would call that an > implementation detail, would you? Even if justified -- f(x) is a pure, > deterministic function with no side-effects -- it would still be a > change to the high-level behaviour of the code. > To me, the optimization stage of a compiler's job is to transform a program to an equivalent one that runs faster, where equivalence is defined w.r.t. a certain set of rules defining the behavior of the language. If f() can really be proven to be a function that is deterministic, has no side-effects, does no runtime introspection, and returns a type that supports the identity "x + x == 2 * x" (quite a bit of work for a dynamic language jit, but definitely possible!), then I'd say that I have a fairly different understanding of the "high-level behavior" the runtime is contracted to follow. As a simpler example, I think the runtime should be very free to condense "a = 1 + 1" into "a = 2" without doing the addition. Anyway, as I alluded to about the __lt__ / __gt__ usage in sort(), just because I might want alternative implementations to have flexibility to do something doesn't mean it's reasonable to say it's so. I'm biased since the implementation I'm working on uses std::unordered_map to implement Python dictionaries, and I have no idea how that's actually implemented and I'd rather not have to :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From wtz_wh at foxmail.com Wed Mar 19 03:41:33 2014 From: wtz_wh at foxmail.com (=?gb18030?B?sbGx+dHz?=) Date: Wed, 19 Mar 2014 10:41:33 +0800 Subject: [Python-Dev] Issues about relative& absolute import way for Porting from python2.4 to python2.7 Message-ID: Dear, I just wrote a sample like this: testPy/ __init__.py client.py SoamFactory.c SoamFactory.so soamapi.py sample/testP.py export PYTHONPATH=$(TEST_LOCATION):$(TEST_LOCATION)/testPy Here's the source codes: __init__.py: import client client.py import soamapi class Client(soamapi.SessionCallback): def __init__(self): print "----class Client----" super(Client, self).__init__() soamapi.initialize() def create_session(self): sec_cb = soamapi.DefaultSecurityCallback() self.connection = soamapi.connect(sec_cb) soamapi.py import SoamFactory class ConnectionSecurityCallback(object): def __init__(self): print "----class ConnectionSecurityCallback----" def __del__(self): pass def test_P(self): print "test_P in class ConnectionSecurityCallback" class DefaultSecurityCallback(ConnectionSecurityCallback): def __init__(self): super(DefaultSecurityCallback, self).__init__() print "----class DefaultSecurityCallback----" def __del__(self): super(DefaultSecurityCallback, self).__del__() def test(self): print "test in class DefaultSecurityCallback" class SessionCallback(object): def __init__(self): pass def __del__(self): pass def connect(security_callback): return SoamFactory.SoamFactory_connect(security_callback) def initialize(): SoamFactory.SoamFactory_initialize() print "call soamapi" SoamFactory.c #include "Python.h" #include "PythonLoop.c" PyObject* SOAM_API_MODULE = NULL; PyObject* pyModule = NULL; static PyObject* SoamFactory_initialize(PyObject* self, PyObject* args){ PyEval_InitThreads(); init(); Py_RETURN_NONE; } static PyObject* SoamFactory_connect(PyObject* self, PyObject* args){ PyObject* pySecCallback = NULL; int ok = PyArg_ParseTuple(args, "O", &pySecCallback); if (!ok){ printf("parse tuple error!\n"); Py_RETURN_NONE; } if (! PyObject_IsInstance(pySecCallback, connectSecCallback)){ printf("pySecCallback is not an instance of ConnectionSecurityCallback!\n"); Py_RETURN_NONE; } printf("Successful!\n"); Py_RETURN_NONE; } static PyMethodDef SoamFactory[] = { {"SoamFactory_connect", SoamFactory_connect, METH_VARARGS, "connection function"}, {"SoamFactory_initialize", SoamFactory_initialize, METH_VARARGS, "SoamFactory_initialize"}, {NULL, NULL} }; void initSoamFactory(){ PyEval_InitThreads(); Py_Initialize(); pyModule = Py_InitModule("SoamFactory", SoamFactory); SOAM_API_MODULE = PyImport_ImportModule("soamapi"); } sample/testP.py import testPy print "========================================" submitter = testPy.client.Client() submitter.create_session() print "========================================" When I ran it on python2.4, it worked well, and the result was call soamapi after import soamapi------client.py ======================================== ----class Client---- ----class ConnectionSecurityCallback---- ----class DefaultSecurityCallback---- Successful! ======================================== But when I ran it on python2.7, it worked beyond my expectation, the result was call soamapi call soamapi ======================================== ----class Client---- ----class ConnectionSecurityCallback---- ----class DefaultSecurityCallback---- pySecCallback is not an instance of ConnectionSecurityCallback! ======================================== I found that soamapi was imported twice, and I investigated this is related to absolute&relative import way. PyImport_ImportModule in python2.7 uses absolute import way, it will look up sys.path to get soamapi module, and when testP.py file import testPy module, it will find local module soamapi under testPy package, and binds module's name to package, as testPy.soamapi. There are two ways to correct it for python2.7, 1) Don't use import testPy, use import client directly to avoid using relative; 2) Use from __future__ import absolute_import to enable absolute import feature. But there are two Pre-conditions: 1) Should not modify testP.py; 2) Should be ran on both python2.4 and 2.7. I don't know how to fix it. Is there any official way about how to porting this scenario or better idea? Thanks, Vatel -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Mar 19 05:27:16 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 Mar 2014 14:27:16 +1000 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: <20140319010948.GN16526@ando> References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> <20140319010948.GN16526@ando> Message-ID: On 19 March 2014 11:09, Steven D'Aprano wrote: > Although I have tentatively said I think this is okay, it is a change in > actual semantics of Python code: what you write is no longer what gets > run. That makes this *very* different from changing the implementation > of sort -- by analogy, its more like changing the semantics of > > a = f(x) + f(x) > > to only call f(x) once. I don't think you would call that an > implementation detail, would you? Even if justified -- f(x) is a pure, > deterministic function with no side-effects -- it would still be a > change to the high-level behaviour of the code. Ah, I think this is a good alternative example. Given the stated assumptions (true builtin dict, not modified between operations), would we be OK with PyPI optimising the following to only do a single dict lookup: a = d[x] + d[x] It's essentially the same optimisation as the one being discussed - in the code *as written*, there are two lookups visible, but for any sensible __hash__ and __eq__ implementation, they should always give the same answer for a true builtin dict that isn't modified between the two lookups. (and yes, PyPy has the infrastructure to enforce those constraints safely and fall back to normal execution if they aren't met - that ability to take opportunistic advantage of known behaviours of particular types is one of the key things that makes it such a powerful tool for implicit optimisations, as compared to things like Cython and Numba, which change semantics more markedly, but also have to be explicitly applied to specific sections of your code rather than being applied automatically). I think it's certainly borderline (it's the kind of surprising behavioural change that irritates people about C/C++ optimisers), but I also think it's a defensible optimisation if the gain is significant enough. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephen at xemacs.org Wed Mar 19 05:35:25 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 19 Mar 2014 13:35:25 +0900 Subject: [Python-Dev] Python 3.5 now uses surrogateescape for the POSIX locale In-Reply-To: References: Message-ID: <87r45ykdpu.fsf@uwakimon.sk.tsukuba.ac.jp> Victor Stinner writes: > 2014-03-18 11:02 GMT+01:00 Atsuo Ishimoto : > > FYI: Guido was opposed to change error handler of stdin and > > stdout years ago. > > > > http://bugs.python.org/issue2630#msg65493 > > This issue proposes to use "backslashreplace" error handler for > stdout. This error handler is very different to "surrogateescape" > which is related to PEP 383 and used by all OS functions. I would say, it's not different in the relevant aspect, which is spewing presumably unreadable bytes that may cause other code reading the output to choke. I would think backslashreplace would generally be preferred for these use cases. From stephen at xemacs.org Wed Mar 19 05:54:09 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 19 Mar 2014 13:54:09 +0900 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> <20140319010948.GN16526@ando> Message-ID: <87pplikcum.fsf@uwakimon.sk.tsukuba.ac.jp> Kevin Modzelewski writes: > I think in this case, though, if we say for the sake of argument > that the guaranteed semantics of a dictionary lookup are zero or I don't understand the point of that argument. It's simply false that semantics are guaranteed, and all of the dunders might be user functions. > more calls to __hash__ plus zero or more calls to __eq__, then two > back-to-back dictionary lookups wouldn't have any observable > differences from doing only one, unless you start to make > assumptions about the behavior of the implementation. That's false. The inverse is true: you should allow the possibility of observable differences, unless you make assumptions about the behavior (implying there are none). > To me there seems to be a bit of a gap between seeing a dictionary > lookup and knowing the exact sequence of user-functions that get > called, far more than for example something like "a < b". The point here is that we *know* that there may be a user function (the dunder that implements []) being called, and it is very hard to determine that that function is pure. Your example of a caching hash is exactly the kind of impure function that one would expect, but who knows what might be called -- there could be a reference to a database on Mars involved (do we have a vehicle on Mars at the moment? anyway...), which calls a pile of Twisted code, and has latencies of many seconds. So Steven is precisely right -- in order to allow this optimization, it would have to be explicitly allowed. Like Steven, I have no strong feeling against it, but then, I don't have a program talking to a deep space vehicle in my near future. Darn it! :-( From mark at hotpy.org Wed Mar 19 11:10:42 2014 From: mark at hotpy.org (Mark Shannon) Date: Wed, 19 Mar 2014 10:10:42 +0000 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: Message-ID: <53296D22.6010802@hotpy.org> On 18/03/14 07:52, Maciej Fijalkowski wrote: > Hi > > I have a question about calling __eq__ in some cases. > > We're thinking about doing an optimization where say: > > if x in d: > return d[x] > > where d is a dict would result in only one dict lookup (the second one > being constant folded away). The question is whether it's ok to do it, > despite the fact that it changes the semantics on how many times > __eq__ is called on x. Yes it is OK. The number of equality checks is not part of the specification of the dictionary. In fact, it differs between a 32 and 64 bit build of the same code. Consider two objects that hash to 2**33+1 and 2**34+1 respectively. On a 32 bit machine their truncated hashes are both 1, so they must be distinguished by an equality test. On a 64 bit machine their hashes are distinct and no equality check is required. Cheers, Mark. From fijall at gmail.com Wed Mar 19 12:08:56 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 19 Mar 2014 13:08:56 +0200 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> Message-ID: On Tue, Mar 18, 2014 at 11:19 PM, Paul Moore wrote: > On 18 March 2014 19:46, Maciej Fijalkowski wrote: >>> A question: how far away will this optimization apply? >>> >>> if x in d: >>> do_this() >>> do_that() >>> do_something_else() >>> spam = d[x] >> >> it depends what those functions do. JIT will inline them and if >> they're small, it should work (although a modification of a different >> dict is illegal, since aliasing is not proven), but at some point >> it'll give up (Note that it'll also give up with a call to C releasing >> GIL since some other thread can modify it). > > Surely in the presence of threads the optimisation is invalid anyway > as other threads can run in between each opcode (I don't know how > you'd phrase that in a way that wasn't language dependent other than > "everywhere" :-)) so > > if x in d: > # HERE > spam = d[x] > > d can be modified at HERE. (If d is a local variable, obviously the > chance that another thread has access to d is a lot lower, but do you > really do that level of alias tracking?) > > Paul not in the case of JIT that *knows* where the GIL can be released. We precisely make it not possible every few bytecodes to avoid such situations. From solipsis at pitrou.net Wed Mar 19 13:42:28 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Mar 2014 13:42:28 +0100 Subject: [Python-Dev] Intricacies of calling __eq__ References: Message-ID: <20140319134228.097abd1a@fsol> On Tue, 18 Mar 2014 09:52:05 +0200 Maciej Fijalkowski wrote: > > We're thinking about doing an optimization where say: > > if x in d: > return d[x] > > where d is a dict would result in only one dict lookup (the second one > being constant folded away). The question is whether it's ok to do it, > despite the fact that it changes the semantics on how many times > __eq__ is called on x. I don't think it's ok. If the author of the code had wanted only one lookup, they would have written: try: return d[x] except KeyError: pass I agree that an __eq__ method with side effects is rather bad, of course. What you could do is instruct people that the latter idiom (EAFP) performs better on PyPy. Regards Antoine. From fijall at gmail.com Wed Mar 19 14:09:04 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 19 Mar 2014 15:09:04 +0200 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: <20140319134228.097abd1a@fsol> References: <20140319134228.097abd1a@fsol> Message-ID: On Wed, Mar 19, 2014 at 2:42 PM, Antoine Pitrou wrote: > On Tue, 18 Mar 2014 09:52:05 +0200 > Maciej Fijalkowski wrote: >> >> We're thinking about doing an optimization where say: >> >> if x in d: >> return d[x] >> >> where d is a dict would result in only one dict lookup (the second one >> being constant folded away). The question is whether it's ok to do it, >> despite the fact that it changes the semantics on how many times >> __eq__ is called on x. > > I don't think it's ok. If the author of the code had wanted only one > lookup, they would have written: > > try: > return d[x] > except KeyError: > pass > > I agree that an __eq__ method with side effects is rather bad, of > course. > What you could do is instruct people that the latter idiom (EAFP) > performs better on PyPy. > > Regards > > Antoine. I would like to point out that instructing people does not really work. Besides, other examples like this: if d[x] >= 3: d[x] += 1 don't really work. From hrvoje.niksic at avl.com Wed Mar 19 14:08:13 2014 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Wed, 19 Mar 2014 14:08:13 +0100 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> Message-ID: <532996BD.1080200@avl.com> On 03/18/2014 10:19 PM, Paul Moore wrote: > Surely in the presence of threads the optimisation is invalid anyway Why? As written, the code uses no synchronization primitives to ensure that the modifications to the dict are propagated at a particular point. As a consequence, it cannot rely on the modification done at a time that coincides with execution at HERE to be immediately propagated to all threads. The optimization is as valid as a C compiler rearranging variable assignments, which also "breaks" unsychronized threaded code. From solipsis at pitrou.net Wed Mar 19 14:17:19 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Mar 2014 14:17:19 +0100 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: <20140319134228.097abd1a@fsol> Message-ID: <20140319141719.19c3e2bf@fsol> On Wed, 19 Mar 2014 15:09:04 +0200 Maciej Fijalkowski wrote: > > I would like to point out that instructing people does not really > work. Besides, other examples like this: > > if d[x] >= 3: > d[x] += 1 don't really work. That's a good point. But then, perhaps PyPy should analyze the __eq__ method and decide whether it's likely to have side effects or not (the answer can be hard-coded for built-in types such as str). Regards Antoine. From fijall at gmail.com Wed Mar 19 14:21:16 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 19 Mar 2014 15:21:16 +0200 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: <20140319141719.19c3e2bf@fsol> References: <20140319134228.097abd1a@fsol> <20140319141719.19c3e2bf@fsol> Message-ID: On Wed, Mar 19, 2014 at 3:17 PM, Antoine Pitrou wrote: > On Wed, 19 Mar 2014 15:09:04 +0200 > Maciej Fijalkowski wrote: >> >> I would like to point out that instructing people does not really >> work. Besides, other examples like this: >> >> if d[x] >= 3: >> d[x] += 1 don't really work. > > That's a good point. But then, perhaps PyPy should analyze the __eq__ > method and decide whether it's likely to have side effects or not (the > answer can be hard-coded for built-in types such as str). > > Regards > > Antoine. Ok. But then how is it valid to have "is" fast-path? From solipsis at pitrou.net Wed Mar 19 14:26:26 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Mar 2014 14:26:26 +0100 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: <20140319134228.097abd1a@fsol> <20140319141719.19c3e2bf@fsol> Message-ID: <20140319142626.2e7cdc61@fsol> On Wed, 19 Mar 2014 15:21:16 +0200 Maciej Fijalkowski wrote: > On Wed, Mar 19, 2014 at 3:17 PM, Antoine Pitrou wrote: > > On Wed, 19 Mar 2014 15:09:04 +0200 > > Maciej Fijalkowski wrote: > >> > >> I would like to point out that instructing people does not really > >> work. Besides, other examples like this: > >> > >> if d[x] >= 3: > >> d[x] += 1 don't really work. > > > > That's a good point. But then, perhaps PyPy should analyze the __eq__ > > method and decide whether it's likely to have side effects or not (the > > answer can be hard-coded for built-in types such as str). > > > > Regards > > > > Antoine. > > Ok. But then how is it valid to have "is" fast-path? What do you mean? Regards Antoine. From thomas at python.org Wed Mar 19 15:09:23 2014 From: thomas at python.org (Thomas Wouters) Date: Wed, 19 Mar 2014 07:09:23 -0700 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: <20140319142626.2e7cdc61@fsol> References: <20140319134228.097abd1a@fsol> <20140319141719.19c3e2bf@fsol> <20140319142626.2e7cdc61@fsol> Message-ID: On Wed, Mar 19, 2014 at 6:26 AM, Antoine Pitrou wrote: > On Wed, 19 Mar 2014 15:21:16 +0200 > Maciej Fijalkowski wrote: > > > On Wed, Mar 19, 2014 at 3:17 PM, Antoine Pitrou > wrote: > > > On Wed, 19 Mar 2014 15:09:04 +0200 > > > Maciej Fijalkowski wrote: > > >> > > >> I would like to point out that instructing people does not really > > >> work. Besides, other examples like this: > > >> > > >> if d[x] >= 3: > > >> d[x] += 1 don't really work. > > > > > > That's a good point. But then, perhaps PyPy should analyze the __eq__ > > > method and decide whether it's likely to have side effects or not (the > > > answer can be hard-coded for built-in types such as str). > > > > > > Regards > > > > > > Antoine. > > > > Ok. But then how is it valid to have "is" fast-path? > > What do you mean? > He means you're being unrealistically pedantic :) The number of calls to __eq__ is _already_ unpredictable, since (as Mark Shannon said) it depends among other things on the hashing algorithm and the size of the dict. If we were serious about not changing these kinds of aspects between Python implementations, we certainly shouldn't be changing them between versions of the same Python implementation -- and so we would be banned from ever changing the details of the dict type. Absurd. -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Mar 19 15:35:45 2014 From: brett at python.org (Brett Cannon) Date: Wed, 19 Mar 2014 10:35:45 -0400 Subject: [Python-Dev] Issues about relative& absolute import way for Porting from python2.4 to python2.7 In-Reply-To: References: Message-ID: This mailing list is for the development *of* Python, the the *use* of Python. Your best option for getting an answer is to ask on python-list or python-help. On Tue, Mar 18, 2014 at 9:50 PM, ??? wrote: > Dear, > > I just wrote a sample like this: > testPy/ > __init__.py > client.py > SoamFactory.c > SoamFactory.so > soamapi.py > sample/testP.py > export PYTHONPATH=$(TEST_LOCATION):$(TEST_LOCATION)/testPy > Here's the source codes: > *__init__.py:* > import client > *client.py* > import soamapi > class Client(soamapi.SessionCallback): > > def __init__(self): > print "----class Client----" > super(Client, self).__init__() > soamapi.initialize() > > def create_session(self): > sec_cb = soamapi.DefaultSecurityCallback() > self.connection = soamapi.connect(sec_cb) > *soamapi.py* > import SoamFactory > > class ConnectionSecurityCallback(object): > def __init__(self): > print "----class ConnectionSecurityCallback----" > > def __del__(self): > pass > > def test_P(self): > print "test_P in class ConnectionSecurityCallback" > > class DefaultSecurityCallback(ConnectionSecurityCallback): > def __init__(self): > super(DefaultSecurityCallback, self).__init__() > print "----class DefaultSecurityCallback----" > > def __del__(self): > super(DefaultSecurityCallback, self).__del__() > > def test(self): > print "test in class DefaultSecurityCallback" > > class SessionCallback(object): > def __init__(self): > pass > > def __del__(self): > pass > > def connect(security_callback): > return SoamFactory.SoamFactory_connect(security_callback) > > def initialize(): > SoamFactory.SoamFactory_initialize() > > print "call soamapi" > *SoamFactory.c* > #include "Python.h" > #include "PythonLoop.c" > > PyObject* SOAM_API_MODULE = NULL; > PyObject* pyModule = NULL; > > static PyObject* SoamFactory_initialize(PyObject* self, PyObject* args){ > PyEval_InitThreads(); > init(); > Py_RETURN_NONE; > } > > static PyObject* SoamFactory_connect(PyObject* self, PyObject* args){ > PyObject* pySecCallback = NULL; > int ok = PyArg_ParseTuple(args, "O", &pySecCallback); > if (!ok){ > printf("parse tuple error!\n"); > Py_RETURN_NONE; > } > if (! PyObject_IsInstance(pySecCallback, connectSecCallback)){ > printf("pySecCallback is not an instance of > ConnectionSecurityCallback!\n"); > Py_RETURN_NONE; > } > printf("Successful!\n"); > Py_RETURN_NONE; > } > > static PyMethodDef SoamFactory[] = { > {"SoamFactory_connect", SoamFactory_connect, METH_VARARGS, "connection > function"}, > {"SoamFactory_initialize", SoamFactory_initialize, METH_VARARGS, > "SoamFactory_initialize"}, > {NULL, NULL} > }; > > void initSoamFactory(){ > PyEval_InitThreads(); > Py_Initialize(); > pyModule = Py_InitModule("SoamFactory", SoamFactory); > SOAM_API_MODULE = PyImport_ImportModule("soamapi"); > } > *sample/testP.py* > import testPy > > print "========================================" > submitter = testPy.client.Client() > submitter.create_session() > print "========================================" > > When I ran it on python2.4, it worked well, and the result was > *call soamapi* > *after import soamapi------client.py* > *========================================* > *----class Client----* > *----class ConnectionSecurityCallback----* > *----class DefaultSecurityCallback----* > *Successful!* > *========================================* > > But when I ran it on python2.7, it worked beyond my expectation, the > result was > call soamapi > call soamapi > ======================================== > ----class Client---- > ----class ConnectionSecurityCallback---- > ----class DefaultSecurityCallback---- > pySecCallback is not an instance of ConnectionSecurityCallback! > ======================================== > > I found that soamapi was imported twice, and I investigated this is > related to absolute&relative import way. PyImport_ImportModule in > python2.7 uses absolute import way, it will look up sys.path to get soamapi > module, and when testP.py file import testPy module, it will find local > module soamapi under testPy package, and binds module's name to package, as > testPy.soamapi. > There are two ways to correct it for python2.7, 1) Don't use import > testPy, use import client directly to avoid using relative; 2) Use from > __future__ import absolute_import to enable absolute import feature. > > But there are two Pre-conditions: > 1) Should not modify testP.py; > 2) Should be ran on both python2.4 and 2.7. > > I don't know how to fix it. Is there any official way about how to porting > this scenario or better idea? > > Thanks, > Vatel > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Mar 19 16:07:16 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Mar 2014 16:07:16 +0100 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: <20140319134228.097abd1a@fsol> <20140319141719.19c3e2bf@fsol> <20140319142626.2e7cdc61@fsol> Message-ID: <20140319160716.7e19de2e@fsol> On Wed, 19 Mar 2014 07:09:23 -0700 Thomas Wouters wrote: > > He means you're being unrealistically pedantic :) The number of calls to > __eq__ is _already_ unpredictable, since (as Mark Shannon said) it depends > among other things on the hashing algorithm and the size of the dict. Well, I was not arguing that the optimization would be an important compatibility break. Obviously, an __eq__ which would break this optimization would be particularly dubious (and probably wrong for other reasons) :-) Regards Antoine. From antony.lee at berkeley.edu Wed Mar 19 06:06:12 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Tue, 18 Mar 2014 22:06:12 -0700 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> <20140319010948.GN16526@ando> Message-ID: The docs don't seem to make any guarantee about calling __eq__ or __hash__: d[key] Return the item of d with key key. Raises a KeyError if key is not in the map. which seems to indicate that this kind of optimization should be fine. In fact I would very much like messing with the semantics of __eq__ be discouraged in the docs. Currently, the docs merely say "The truth of x==y does not imply that x!=y is false." Of course, once __eq__ and __ne__ are separately overridable nothing can be guaranteed but I've seen code where x == y and x != y do completely different things and that was not pretty. 2014-03-18 21:27 GMT-07:00 Nick Coghlan : > On 19 March 2014 11:09, Steven D'Aprano wrote: > > Although I have tentatively said I think this is okay, it is a change in > > actual semantics of Python code: what you write is no longer what gets > > run. That makes this *very* different from changing the implementation > > of sort -- by analogy, its more like changing the semantics of > > > > a = f(x) + f(x) > > > > to only call f(x) once. I don't think you would call that an > > implementation detail, would you? Even if justified -- f(x) is a pure, > > deterministic function with no side-effects -- it would still be a > > change to the high-level behaviour of the code. > > Ah, I think this is a good alternative example. Given the stated > assumptions (true builtin dict, not modified between operations), > would we be OK with PyPI optimising the following to only do a single > dict lookup: > > a = d[x] + d[x] > > It's essentially the same optimisation as the one being discussed - in > the code *as written*, there are two lookups visible, but for any > sensible __hash__ and __eq__ implementation, they should always give > the same answer for a true builtin dict that isn't modified between > the two lookups. (and yes, PyPy has the infrastructure to enforce > those constraints safely and fall back to normal execution if they > aren't met - that ability to take opportunistic advantage of known > behaviours of particular types is one of the key things that makes it > such a powerful tool for implicit optimisations, as compared to things > like Cython and Numba, which change semantics more markedly, but also > have to be explicitly applied to specific sections of your code rather > than being applied automatically). > > I think it's certainly borderline (it's the kind of surprising > behavioural change that irritates people about C/C++ optimisers), but > I also think it's a defensible optimisation if the gain is significant > enough. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/antony.lee%40berkeley.edu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kmod at dropbox.com Wed Mar 19 07:38:53 2014 From: kmod at dropbox.com (Kevin Modzelewski) Date: Tue, 18 Mar 2014 23:38:53 -0700 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: <87pplikcum.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> <20140319010948.GN16526@ando> <87pplikcum.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Sorry, I definitely didn't mean to imply that this kind of optimization is valid on arbitrary subscript expressions; I thought we had restricted ourselves to talking about builtin dicts. If we do, I think this becomes a discussion about what subset of the semantics of CPython's builtins are language-specified vs implementation-dependent; my argument is that just because something results in an observable behavioral difference doesn't necessarily mean that it's a change in language semantics, if it's just a change in the implementation-dependent behavior. On Tue, Mar 18, 2014 at 9:54 PM, Stephen J. Turnbull wrote: > Kevin Modzelewski writes: > > > I think in this case, though, if we say for the sake of argument > > that the guaranteed semantics of a dictionary lookup are zero or > > I don't understand the point of that argument. It's simply false that > semantics are guaranteed, and all of the dunders might be user > functions. > > > more calls to __hash__ plus zero or more calls to __eq__, then two > > back-to-back dictionary lookups wouldn't have any observable > > differences from doing only one, unless you start to make > > assumptions about the behavior of the implementation. > > That's false. The inverse is true: you should allow the possibility of > observable differences, unless you make assumptions about the behavior > (implying there are none). > > > To me there seems to be a bit of a gap between seeing a dictionary > > lookup and knowing the exact sequence of user-functions that get > > called, far more than for example something like "a < b". > > The point here is that we *know* that there may be a user function > (the dunder that implements []) being called, and it is very hard to > determine that that function is pure. > > Your example of a caching hash is exactly the kind of impure function > that one would expect, but who knows what might be called -- there > could be a reference to a database on Mars involved (do we have a > vehicle on Mars at the moment? anyway...), which calls a pile of > Twisted code, and has latencies of many seconds. > > So Steven is precisely right -- in order to allow this optimization, > it would have to be explicitly allowed. > > Like Steven, I have no strong feeling against it, but then, I don't > have a program talking to a deep space vehicle in my near future. > Darn it! :-( > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Wed Mar 19 16:38:45 2014 From: martin at v.loewis.de (=?windows-1250?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 19 Mar 2014 16:38:45 +0100 Subject: [Python-Dev] 'Add/Remove Programs' entry missing for 'current user only' 32-bit installations on 64-bit Windows In-Reply-To: References: Message-ID: <5329BA05.5030306@v.loewis.de> Am 17.03.14 22:10, schrieb Jurko Gospodneti?: > Fixing > this required manually cleaning up leftover CPython 3.4.0rc3 windows > installer registry entries. Note that the issue could not be fixed by > using the CPython 3.4.0rc3 installer as it failed due to the same problem. Did you try the 3.4.0rc3 "repair" installation? That should have undeleted (repaired) the files that you had manually deleted. > I compared MSI packaging related CPython source code (Tools/msi > folder) in 2.6.2 & 2.6.3 releases but I can not see anything suspicious > there. It could be that the observed beaviour change between those two > versions is a result of the final release packager changing his used > Windows Installer version, but I have not rebuilt CPython, and its MSI > installer to test this theory out. I believe I switched the entire Windows installation at least once in this time (from XP to Windows 7), so it's unlikely related to my personal installation. > Should I open a 'Add/Remove Programs' dialog related issue in the > CPython issue tracker? Please do. It would also good if somebody volunteered to reproduce the issue. > And possibly a separate one for making CPython > installations not fail without possible recovery if a previous CPython > installation has already been removed? Please don't. Or, at least, make it more specific. If you have manually corrupted your Python installation (by deleting essential files), you have to accept that the system installation procedures will fail. It might be possible to recover from the loss of a specific such file; for that, we would have to identify what the file is. However, the standard procedure should be to repair the installation before attempting an upgrade. Regards, Martin From jurko.gospodnetic at pke.hr Wed Mar 19 16:49:23 2014 From: jurko.gospodnetic at pke.hr (=?windows-1250?Q?Jurko_Gospodneti=E6?=) Date: Wed, 19 Mar 2014 16:49:23 +0100 Subject: [Python-Dev] 'Add/Remove Programs' entry missing for 'current user only' 32-bit installations on 64-bit Windows In-Reply-To: <5329BA05.5030306@v.loewis.de> References: <5329BA05.5030306@v.loewis.de> Message-ID: <5329BC83.4020808@pke.hr> Hi. On 19.3.2014. 16:38, "Martin v. L?wis" wrote: > Am 17.03.14 22:10, schrieb Jurko Gospodneti?: >> Fixing >> this required manually cleaning up leftover CPython 3.4.0rc3 windows >> installer registry entries. Note that the issue could not be fixed by >> using the CPython 3.4.0rc3 installer as it failed due to the same problem. > > Did you try the 3.4.0rc3 "repair" installation? That should have > undeleted (repaired) the files that you had manually deleted. I tried different things back then, and I'm 90% sure I tried that one as well. Unfortunately, my brain is already trying to block out that painful afternoon so I can't tell you for certain. :-) But, all kidding aside, I'll try to reproduce the issue again and let you know the exact details. >> Should I open a 'Add/Remove Programs' dialog related issue in the >> CPython issue tracker? > > Please do. It would also good if somebody volunteered to reproduce > the issue. Will do. FYI, I did reproduce the 'Add/Remove Programs' dialog issue on another PC in the mean time. Also Windows 7 x64 though. Unfortunately, I don't have other 64-bit OSs readily available at the moment. >> And possibly a separate one for making CPython >> installations not fail without possible recovery if a previous CPython >> installation has already been removed? > > Please don't. Or, at least, make it more specific. If you have manually > corrupted your Python installation (by deleting essential files), you > have to accept that the system installation procedures will fail. > > It might be possible to recover from the loss of a specific such file; > for that, we would have to identify what the file is. However, the > standard procedure should be to repair the installation before > attempting an upgrade. Ok, as I stated above, I'll get back to you on this once I reproduce the 'corruption' issue again. Oh, and many thanks for replying! :-) Best regards, Jurko Gospodneti? From jurko.gospodnetic at pke.hr Wed Mar 19 17:20:03 2014 From: jurko.gospodnetic at pke.hr (=?windows-1250?Q?Jurko_Gospodneti=E6?=) Date: Wed, 19 Mar 2014 17:20:03 +0100 Subject: [Python-Dev] 'Add/Remove Programs' entry missing for 'current user only' 32-bit installations on 64-bit Windows In-Reply-To: <5329BA05.5030306@v.loewis.de> References: <5329BA05.5030306@v.loewis.de> Message-ID: <5329C3B3.2090801@pke.hr> Hi. On 19.3.2014. 16:38, "Martin v. L?wis" wrote: > Am 17.03.14 22:10, schrieb Jurko Gospodneti?: >> Fixing >> this required manually cleaning up leftover CPython 3.4.0rc3 windows >> installer registry entries. Note that the issue could not be fixed >> by using the CPython 3.4.0rc3 installer as it failed due to the >> same problem. > > Did you try the 3.4.0rc3 "repair" installation? That should have > undeleted (repaired) the files that you had manually deleted. Just tested this out and repairing the same matching installation does revert the removed files. It does not reinstall pip and setuptools packages but that can be done easily by hand and so can be filed as a minor enhancement issue for the future. That at least provides a way to remove the the original installation in some way without having to manually remove its related Windows Installer registry entries. The problem with the 'Add/Remove Programs' dialog still makes this issue a bit complicated because a user not well versed in how Windows Installer tracks its installations internally would not see the installation in the 'standard place' where he is supposed to look for it, and would just be told by the new installation that 'something failed'. To use a 'repair' installation he would need to 'guess' which installation he used before and run its installer directly. When the 'Add/Remove Programs' dialog is fixed, user will be able to see & repair the installation from there, and even remove it directly if the uninstallation fails (I recall Windows asking if you want this if it can not uninstall something). I'll try to open an issue for the 'Add/Remove Programs' dialog issue soon, and I'll add a comment about the 'repair' situation there as well. Best regards, Jurko Gospodneti? From fijall at gmail.com Wed Mar 19 17:27:00 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 19 Mar 2014 18:27:00 +0200 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: <20140319142626.2e7cdc61@fsol> References: <20140319134228.097abd1a@fsol> <20140319141719.19c3e2bf@fsol> <20140319142626.2e7cdc61@fsol> Message-ID: On Wed, Mar 19, 2014 at 3:26 PM, Antoine Pitrou wrote: > On Wed, 19 Mar 2014 15:21:16 +0200 > Maciej Fijalkowski wrote: > >> On Wed, Mar 19, 2014 at 3:17 PM, Antoine Pitrou wrote: >> > On Wed, 19 Mar 2014 15:09:04 +0200 >> > Maciej Fijalkowski wrote: >> >> >> >> I would like to point out that instructing people does not really >> >> work. Besides, other examples like this: >> >> >> >> if d[x] >= 3: >> >> d[x] += 1 don't really work. >> > >> > That's a good point. But then, perhaps PyPy should analyze the __eq__ >> > method and decide whether it's likely to have side effects or not (the >> > answer can be hard-coded for built-in types such as str). >> > >> > Regards >> > >> > Antoine. >> >> Ok. But then how is it valid to have "is" fast-path? > > What do you mean? > I mean that dict starts with "is" before calling __eq__, so the number of calls to __eq__ can as well be zero. From fijall at gmail.com Wed Mar 19 17:27:29 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 19 Mar 2014 18:27:29 +0200 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> <20140319010948.GN16526@ando> <87pplikcum.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Wed, Mar 19, 2014 at 8:38 AM, Kevin Modzelewski wrote: > Sorry, I definitely didn't mean to imply that this kind of optimization is > valid on arbitrary subscript expressions; I thought we had restricted > ourselves to talking about builtin dicts. If we do, I think this becomes a > discussion about what subset of the semantics of CPython's builtins are > language-specified vs implementation-dependent; my argument is that just > because something results in an observable behavioral difference doesn't > necessarily mean that it's a change in language semantics, if it's just a > change in the implementation-dependent behavior. > > > On Tue, Mar 18, 2014 at 9:54 PM, Stephen J. Turnbull > wrote: >> >> Kevin Modzelewski writes: >> >> > I think in this case, though, if we say for the sake of argument >> > that the guaranteed semantics of a dictionary lookup are zero or >> >> I don't understand the point of that argument. It's simply false that >> semantics are guaranteed, and all of the dunders might be user >> functions. >> >> > more calls to __hash__ plus zero or more calls to __eq__, then two >> > back-to-back dictionary lookups wouldn't have any observable >> > differences from doing only one, unless you start to make >> > assumptions about the behavior of the implementation. >> >> That's false. The inverse is true: you should allow the possibility of >> observable differences, unless you make assumptions about the behavior >> (implying there are none). >> >> > To me there seems to be a bit of a gap between seeing a dictionary >> > lookup and knowing the exact sequence of user-functions that get >> > called, far more than for example something like "a < b". >> >> The point here is that we *know* that there may be a user function >> (the dunder that implements []) being called, and it is very hard to >> determine that that function is pure. >> >> Your example of a caching hash is exactly the kind of impure function >> that one would expect, but who knows what might be called -- there >> could be a reference to a database on Mars involved (do we have a >> vehicle on Mars at the moment? anyway...), which calls a pile of >> Twisted code, and has latencies of many seconds. >> >> So Steven is precisely right -- in order to allow this optimization, >> it would have to be explicitly allowed. >> >> Like Steven, I have no strong feeling against it, but then, I don't >> have a program talking to a deep space vehicle in my near future. >> Darn it! :-( >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > we're discussing builtin dicts From stephen at xemacs.org Wed Mar 19 17:35:53 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 20 Mar 2014 01:35:53 +0900 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> <20140319010948.GN16526@ando> <87pplikcum.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87wqfqi1sm.fsf@uwakimon.sk.tsukuba.ac.jp> Kevin Modzelewski writes: > Sorry, I definitely didn't mean to imply that this kind of > optimization is valid on arbitrary subscript expressions; I thought > we had restricted ourselves to talking about builtin dicts. Ah, maybe so -- Maciej made that clear later for PyPy. My bad. (With the caveat that IIUC Python does not include the technology for detecting for sure that you've got a builtin dict -- a given instance might even be monkey-patched.) >?If we do, I think this becomes a discussion about what subset of > the semantics of CPython's builtins are language-specified vs > implementation-dependent; my argument is that just because > something results in an observable behavioral difference doesn't > necessarily mean that it's a change in language semantics, if it's > just a change in the implementation-dependent behavior. I think you're wrong there. Python makes very strong guarantees of backward compatibility; there really isn't that much left to be implementation-dependent once a feature has been introduced and released. Things are a lot more flexible at the introduction of a feature (eg, the decorator example where the specification of the feature clearly involves evaluating an expression twice, but the implementation doesn't). Although I doubt anybody considers that a big deal (and if they did, they'd fix the documentation). But I don't think a *change* in that would be consided "not a change in language semantics." That doesn't mean such a change wouldn't be allowed in a 3.x.0 release. Just that it would be called a change. From jurko.gospodnetic at pke.hr Wed Mar 19 18:01:58 2014 From: jurko.gospodnetic at pke.hr (=?windows-1250?Q?Jurko_Gospodneti=E6?=) Date: Wed, 19 Mar 2014 18:01:58 +0100 Subject: [Python-Dev] 'Add/Remove Programs' entry missing for 'current user only' 32-bit installations on 64-bit Windows In-Reply-To: <5329C3B3.2090801@pke.hr> References: <5329BA05.5030306@v.loewis.de> <5329C3B3.2090801@pke.hr> Message-ID: Hi. > > Did you try the 3.4.0rc3 "repair" installation? That should have > > undeleted (repaired) the files that you had manually deleted. > > Just tested this out and repairing the same matching installation > does revert the removed files. It does not reinstall pip and setuptools > packages but that can be done easily by hand and so can be filed as a > minor enhancement issue for the future. This minor problem reported as issue #20983. http://bugs.python.org/issue20983 Best regards, Jurko Gospodneti? From imran.geriskovan at gmail.com Wed Mar 19 19:07:08 2014 From: imran.geriskovan at gmail.com (Imran Geriskovan) Date: Wed, 19 Mar 2014 20:07:08 +0200 Subject: [Python-Dev] asyncio.wait(FIRST_COMPLETED) returns more than one completions - 3.4rc2 Message-ID: Code below has been written with the intension of acquiring ONLY one lock. There are two issues: 1- Sometimes it returns more than one lock in done. 2- Sometimes, even if wait exits with zero or one locks, it seems there are other locks are acquired too. Though, I couldn't isolate the exact case for this. It sounds like some background shield() is at works. I kindly request your comments. locks = [some asyncio.Locks...] sel = [Pack(l.acquire(), l) for l in locks] done, pend = asyncio.wait(sel, timeout=10, return_when=FIRST_COMPLETED) ... @coroutine def Pack(co, obj): yield from co return obj Regards, Imran From guido at python.org Wed Mar 19 19:31:03 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Mar 2014 11:31:03 -0700 Subject: [Python-Dev] asyncio.wait(FIRST_COMPLETED) returns more than one completions - 3.4rc2 In-Reply-To: References: Message-ID: Hi Imran, The python-dev list is not the place to ask questions about the usage of Python modules or features. However, since you are asking an asyncio-related question, you should be welcome at the python-tulip list: https://groups.google.com/forum/?fromgroups#!forum/python-tulip --Guido On Wed, Mar 19, 2014 at 11:07 AM, Imran Geriskovan < imran.geriskovan at gmail.com> wrote: > Code below has been written with the intension of acquiring ONLY one lock. > There are two issues: > > 1- Sometimes it returns more than one lock in done. > 2- Sometimes, even if wait exits with zero or one locks, it seems > there are other locks are acquired too. Though, I couldn't isolate > the exact case for this. > > It sounds like some background shield() is at works. > > I kindly request your comments. > > > locks = [some asyncio.Locks...] > sel = [Pack(l.acquire(), l) for l in locks] > done, pend = asyncio.wait(sel, timeout=10, return_when=FIRST_COMPLETED) > ... > @coroutine > def Pack(co, obj): > yield from co > return obj > > > Regards, > Imran > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jurko.gospodnetic at pke.hr Wed Mar 19 19:41:19 2014 From: jurko.gospodnetic at pke.hr (=?windows-1250?Q?Jurko_Gospodneti=E6?=) Date: Wed, 19 Mar 2014 19:41:19 +0100 Subject: [Python-Dev] 'Add/Remove Programs' entry missing for 'current user only' 32-bit installations on 64-bit Windows In-Reply-To: <5329BC83.4020808@pke.hr> References: <5329BA05.5030306@v.loewis.de> <5329BC83.4020808@pke.hr> Message-ID: Hi. >> Should I open a 'Add/Remove Programs' dialog related issue in the >> CPython issue tracker? > > Please do. It would also good if somebody volunteered to reproduce > the issue. Opened issue #20984. http://bugs.python.org/issue20984 Anyone have Windows 8 x64 available to try this out by any chance? Best regards, Jurko Gospodneti? From solipsis at pitrou.net Wed Mar 19 19:46:07 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Mar 2014 19:46:07 +0100 Subject: [Python-Dev] Making proxy types easier to write and maintain Message-ID: <20140319194607.331336db@fsol> Hello, It is known to be cumbersome to write a proxy type that will correctly proxy all special methods (this has to be done manually on the type, since special methods are not looked up on the instance: a __getattr__ method would not work). Recently we've had reports of two stdlib types that forgot to implement some special methods: - weakref.proxy doesn't implement __reversed__: http://bugs.python.org/issue19359 - mock.MagicMock doesn't implement __truediv__: http://bugs.python.org/issue20968 In http://bugs.python.org/issue19359#msg213530 I proposed to introduce a "proxy protocol" (__proxy__ / tp_proxy) that would be used as a fallback by _PyObject_LookupSpecial to fetch the lookup target, i.e.: def _PyObject_LookupSpecial(obj, name): tp = type(obj) try: return getattr(tp, name) except AttributeError: return getattr(tp.tp_proxy(), name) What do you think? Regards Antoine. From ethan at stoneleaf.us Wed Mar 19 18:53:31 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 19 Mar 2014 10:53:31 -0700 Subject: [Python-Dev] unit tests for error messages Message-ID: <5329D99B.50407@stoneleaf.us> I just made a change to some error messages [1] (really, just one): old behavior: '%o' % 3.14 'float' object cannot be interpreted as an integer new behavior: '%o' % 3.14 %o format: an integer is required, not float Would we normally add a test for that? -- ~Ethan~ [1] Issue 19995: http://bugs.python.org/issue19995 From solipsis at pitrou.net Wed Mar 19 19:55:32 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Mar 2014 19:55:32 +0100 Subject: [Python-Dev] unit tests for error messages References: <5329D99B.50407@stoneleaf.us> Message-ID: <20140319195532.5ef7a721@fsol> On Wed, 19 Mar 2014 10:53:31 -0700 Ethan Furman wrote: > I just made a change to some error messages [1] (really, just one): > > old behavior: > > '%o' % 3.14 > 'float' object cannot be interpreted as an integer > > new behavior: > > '%o' % 3.14 > %o format: an integer is required, not float > > Would we normally add a test for that? Depends if you're fearing a regression. Regards Antoine. From AHettinger at Prominic.NET Wed Mar 19 19:46:48 2014 From: AHettinger at Prominic.NET (Andrew M. Hettinger) Date: Wed, 19 Mar 2014 13:46:48 -0500 Subject: [Python-Dev] 'Add/Remove Programs' entry missing for 'current user only' 32-bit installations on 64-bit Windows In-Reply-To: References: <5329BA05.5030306@v.loewis.de> <5329BC83.4020808@pke.hr> Message-ID: Sure, I'll check it this evening. Jurko Gospodneti? wrote on 03/19/2014 01:41:19 PM: > > Hi. > > >> Should I open a 'Add/Remove Programs' dialog related issue in the > >> CPython issue tracker? > > > > Please do. It would also good if somebody volunteered to reproduce > > the issue. > > Opened issue #20984. > > http://bugs.python.org/issue20984 > > Anyone have Windows 8 x64 available to try this out by any chance? > > Best regards, > Jurko Gospodneti? > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > ahettinger%40prominic.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Wed Mar 19 20:32:38 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 19 Mar 2014 20:32:38 +0100 Subject: [Python-Dev] unit tests for error messages In-Reply-To: <20140319195532.5ef7a721@fsol> References: <5329D99B.50407@stoneleaf.us> <20140319195532.5ef7a721@fsol> Message-ID: Am 19.03.2014 19:55, schrieb Antoine Pitrou: > On Wed, 19 Mar 2014 10:53:31 -0700 > Ethan Furman wrote: > >> I just made a change to some error messages [1] (really, just one): >> >> old behavior: >> >> '%o' % 3.14 >> 'float' object cannot be interpreted as an integer >> >> new behavior: >> >> '%o' % 3.14 >> %o format: an integer is required, not float >> >> Would we normally add a test for that? > > Depends if you're fearing a regression. If he fears regressions in error messages, he shouldn't be changing them in the first place :) Georg From bcannon at gmail.com Wed Mar 19 20:32:50 2014 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 19 Mar 2014 19:32:50 +0000 Subject: [Python-Dev] Making proxy types easier to write and maintain References: <20140319194607.331336db@fsol> Message-ID: On Wed Mar 19 2014 at 2:46:48 PM, Antoine Pitrou wrote: > > Hello, > > It is known to be cumbersome to write a proxy type that will correctly > proxy all special methods (this has to be done manually on the type, > since special methods are not looked up on the instance: a __getattr__ > method would not work). > > Recently we've had reports of two stdlib types that forgot to > implement some special methods: > - weakref.proxy doesn't implement __reversed__: > http://bugs.python.org/issue19359 > - mock.MagicMock doesn't implement __truediv__: > http://bugs.python.org/issue20968 > > In http://bugs.python.org/issue19359#msg213530 I proposed to introduce a > "proxy > protocol" (__proxy__ / tp_proxy) that would be used as a fallback by > _PyObject_LookupSpecial to fetch the lookup target, i.e.: > > def _PyObject_LookupSpecial(obj, name): > tp = type(obj) > try: > return getattr(tp, name) > except AttributeError: > return getattr(tp.tp_proxy(), name) > > What do you think? > Without having the code in front of me, would this only be for magic methods and attributes, or all attributes? IOW would this mean that if I assign an object to __proxy__ it would take care of the uses of __getattr__ for proxying? -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Mar 19 21:15:15 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 19 Mar 2014 20:15:15 +0000 Subject: [Python-Dev] Making proxy types easier to write and maintain In-Reply-To: <20140319194607.331336db@fsol> References: <20140319194607.331336db@fsol> Message-ID: On 19 March 2014 18:46, Antoine Pitrou wrote: > In http://bugs.python.org/issue19359#msg213530 I proposed to introduce a "proxy > protocol" (__proxy__ / tp_proxy) that would be used as a fallback by > _PyObject_LookupSpecial to fetch the lookup target, i.e.: > > def _PyObject_LookupSpecial(obj, name): > tp = type(obj) > try: > return getattr(tp, name) > except AttributeError: > return getattr(tp.tp_proxy(), name) > > What do you think? Would that increase the size of type objects? Would that matter? (There's a similar question that came up in the thread about adding the @ operator over on python-ideas, which is what made me think of it...) Paul From solipsis at pitrou.net Wed Mar 19 21:22:42 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Mar 2014 21:22:42 +0100 Subject: [Python-Dev] Making proxy types easier to write and maintain In-Reply-To: References: <20140319194607.331336db@fsol> Message-ID: <20140319212242.5d25904c@fsol> On Wed, 19 Mar 2014 19:32:50 +0000 Brett Cannon wrote: > > > > In http://bugs.python.org/issue19359#msg213530 I proposed to introduce a > > "proxy > > protocol" (__proxy__ / tp_proxy) that would be used as a fallback by > > _PyObject_LookupSpecial to fetch the lookup target, i.e.: > > > > def _PyObject_LookupSpecial(obj, name): > > tp = type(obj) > > try: > > return getattr(tp, name) > > except AttributeError: > > return getattr(tp.tp_proxy(), name) > > > > What do you think? > > > > Without having the code in front of me, would this only be for magic > methods and attributes, or all attributes? IOW would this mean that if I > assign an object to __proxy__ it would take care of the uses of __getattr__ > for proxying? In a first approach it would be only for magic methods and attributes. It if proves successful, perhaps it can be later expanded to also be used for regular lookups. By the way, Benjamin pointed me to a prior discussion: http://bugs.python.org/issue643841 Regards Antoine. From solipsis at pitrou.net Wed Mar 19 21:23:13 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Mar 2014 21:23:13 +0100 Subject: [Python-Dev] Making proxy types easier to write and maintain In-Reply-To: References: <20140319194607.331336db@fsol> Message-ID: <20140319212313.61db363a@fsol> On Wed, 19 Mar 2014 20:15:15 +0000 Paul Moore wrote: > On 19 March 2014 18:46, Antoine Pitrou wrote: > > In http://bugs.python.org/issue19359#msg213530 I proposed to introduce a "proxy > > protocol" (__proxy__ / tp_proxy) that would be used as a fallback by > > _PyObject_LookupSpecial to fetch the lookup target, i.e.: > > > > def _PyObject_LookupSpecial(obj, name): > > tp = type(obj) > > try: > > return getattr(tp, name) > > except AttributeError: > > return getattr(tp.tp_proxy(), name) > > > > What do you think? > > Would that increase the size of type objects? Would that matter? > (There's a similar question that came up in the thread about adding > the @ operator over on python-ideas, which is what made me think of > it...) One additional slot is one additional pointer field, which is mostly trivial in a type object. Regards Antoine. From ncoghlan at gmail.com Wed Mar 19 22:38:21 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Mar 2014 07:38:21 +1000 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: <87wqfqi1sm.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> <20140319010948.GN16526@ando> <87pplikcum.fsf@uwakimon.sk.tsukuba.ac.jp> <87wqfqi1sm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 20 Mar 2014 02:37, "Stephen J. Turnbull" wrote: > > Kevin Modzelewski writes: > > > Sorry, I definitely didn't mean to imply that this kind of > > optimization is valid on arbitrary subscript expressions; I thought > > we had restricted ourselves to talking about builtin dicts. > > Ah, maybe so -- Maciej made that clear later for PyPy. My bad. (With > the caveat that IIUC Python does not include the technology for > detecting for sure that you've got a builtin dict -- a given instance > might even be monkey-patched.) CPython doesn't, PyPy does. There are many reasons Armin Rigo stopped working on psyco in CPython and created PyPy instead - one of them is that making optimisations like this maintainable for a dynamic language like Python essentially required inventing a whole new approach to creating dynamic language interpreters. > > If we do, I think this becomes a discussion about what subset of > > the semantics of CPython's builtins are language-specified vs > > implementation-dependent; my argument is that just because > > something results in an observable behavioral difference doesn't > > necessarily mean that it's a change in language semantics, if it's > > just a change in the implementation-dependent behavior. > > I think you're wrong there. Python makes very strong guarantees of > backward compatibility; there really isn't that much left to be > implementation-dependent once a feature has been introduced and > released. Correct, but I think this discussion has established that "how many times dict lookup calls __eq__ on the key" is one such thing. In CPython, it already varies based on: - dict contents (due to the identity check and the distribution of entries across hash buckets) - pointer size (due to the hash bucket distribution differing between 32 bit and 64 bit builds) - dict tuning parameters (there are some settings in the dict implementation that affect when dicts resize up and down, etc, which can mean the hash bucket distribution may already change without much notice in feature releases) So that part of this PyPy optimisation shouldn't be controversial, leaving the matter of only calling __hash__ on the lookup key once rather than twice. Since "hash(x)" changing with time is just a straight up bug in the implementation of "x", that part also sounds fine. So yeah, it's certainly a subtle point, but I agree with Maciej that the PyPy team have found a legitimate optimisation opportunity here. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Mar 19 22:43:01 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Mar 2014 07:43:01 +1000 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> <20140319010948.GN16526@ando> <87pplikcum.fsf@uwakimon.sk.tsukuba.ac.jp> <87wqfqi1sm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 20 Mar 2014 07:38, "Nick Coghlan" wrote: > > Correct, but I think this discussion has established that "how many times dict lookup calls __eq__ on the key" is one such thing. In CPython, it already varies based on: > > - dict contents (due to the identity check and the distribution of entries across hash buckets) > - pointer size (due to the hash bucket distribution differing between 32 bit and 64 bit builds) > - dict tuning parameters (there are some settings in the dict implementation that affect when dicts resize up and down, etc, which can mean the hash bucket distribution may already change without much notice in feature releases) I just realised that hash randomisation also comes into play here - the distribution of entries across hash buckets is inherently variable between runs for any key types that rely directly or indirectly on a randomised hash. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Mar 19 22:54:39 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Mar 2014 07:54:39 +1000 Subject: [Python-Dev] Making proxy types easier to write and maintain In-Reply-To: <20140319212242.5d25904c@fsol> References: <20140319194607.331336db@fsol> <20140319212242.5d25904c@fsol> Message-ID: On 20 Mar 2014 06:24, "Antoine Pitrou" wrote: > > On Wed, 19 Mar 2014 19:32:50 +0000 > Brett Cannon wrote: > > > > > > In http://bugs.python.org/issue19359#msg213530 I proposed to introduce a > > > "proxy > > > protocol" (__proxy__ / tp_proxy) that would be used as a fallback by > > > _PyObject_LookupSpecial to fetch the lookup target, i.e.: > > > > > > def _PyObject_LookupSpecial(obj, name): > > > tp = type(obj) > > > try: > > > return getattr(tp, name) > > > except AttributeError: > > > return getattr(tp.tp_proxy(), name) > > > > > > What do you think? > > > > > > > Without having the code in front of me, would this only be for magic > > methods and attributes, or all attributes? IOW would this mean that if I > > assign an object to __proxy__ it would take care of the uses of __getattr__ > > for proxying? > > In a first approach it would be only for magic methods and attributes. > It if proves successful, perhaps it can be later expanded to also be > used for regular lookups. > > By the way, Benjamin pointed me to a prior discussion: > http://bugs.python.org/issue643841 Graeme Dumpleton has also subsequently written a library to handle easier creation of correct proxy types: https://pypi.python.org/pypi/wrapt It isn't as simple as changing one lookup function though - abstract.c reads the type slots directly, and those are already hairy performance critical code paths (especially the binary operator type dispatch). Cheers, Nick. > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Mar 19 23:01:39 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Mar 2014 23:01:39 +0100 Subject: [Python-Dev] Making proxy types easier to write and maintain In-Reply-To: References: <20140319194607.331336db@fsol> <20140319212242.5d25904c@fsol> Message-ID: <20140319230139.043aee2f@fsol> On Thu, 20 Mar 2014 07:54:39 +1000 Nick Coghlan wrote: > > Graeme Dumpleton has also subsequently written a library to handle easier > creation of correct proxy types: https://pypi.python.org/pypi/wrapt (is "Graeme" another spelling for "Graham"?) > It isn't as simple as changing one lookup function though - abstract.c > reads the type slots directly, and those are already hairy performance > critical code paths (especially the binary operator type dispatch). Oh indeed, you're right. I think my proposal is taking water. Regards Antoine. From ethan at stoneleaf.us Wed Mar 19 22:37:42 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 19 Mar 2014 14:37:42 -0700 Subject: [Python-Dev] unittest assertRaisesRegex bug? Message-ID: <532A0E26.5060106@stoneleaf.us> Here's the code in question: class PsuedoFloat: def __init__(self, value): self.value = float(value) def __int__(self): return int(self.value) pi = PsuedoFloat(3.1415) self.assertRaisesRegex(TypeError, '%x format: an integer is required, not PsuedoFloat', '%x'.__mod__, pi), Here's the exception: ====================================================================== ERROR: test_formatting (test.test_unicode.UnicodeTest) ---------------------------------------------------------------------- TypeError: 'PsuedoFloat' object is not callable During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ethan/source/python/issue19995/Lib/test/test_unicode.py", line 1156, in test_formatting self.assertRaisesRegex(TypeError, '%c'.__mod__, pi), File "/home/ethan/source/python/issue19995/Lib/unittest/case.py", line 1235, in assertRaisesRegex return context.handle('assertRaisesRegex', callable_obj, args, kwargs) File "/home/ethan/source/python/issue19995/Lib/unittest/case.py", line 161, in handle callable_obj(*args, **kwargs) File "/home/ethan/source/python/issue19995/Lib/unittest/case.py", line 190, in __exit__ if not expected_regex.search(str(exc_value)): AttributeError: 'method-wrapper' object has no attribute 'search' ---------------------------------------------------------------------- At worst, I was expecting a difference in the TypeError exception message; I have no idea why `pi` is being called. From the docs: ---- http://docs.python.org/3/library/unittest.html#unittest.TestCase.assertRaisesRegex ---- assertRaisesRegex(exception, regex, callable, *args, **kwds) Like assertRaises() but also tests that regex matches on the string representation of the raised exception. regex may be a regular expression object or a string containing a regular expression suitable for use by re.search(). Examples: self.assertRaisesRegex(ValueError, "invalid literal for.*XYZ'$", int, 'XYZ') -------------------------------------------------------------------------------------------- Am I correct in thinking this is a bug? -- ~Ethan~ From solipsis at pitrou.net Wed Mar 19 23:13:54 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Mar 2014 23:13:54 +0100 Subject: [Python-Dev] unittest assertRaisesRegex bug? References: <532A0E26.5060106@stoneleaf.us> Message-ID: <20140319231354.4a70b15c@fsol> On Wed, 19 Mar 2014 14:37:42 -0700 Ethan Furman wrote: > Here's the code in question: > > class PsuedoFloat: > def __init__(self, value): > self.value = float(value) > def __int__(self): > return int(self.value) > > pi = PsuedoFloat(3.1415) > > self.assertRaisesRegex(TypeError, '%x format: an integer is required, not PsuedoFloat', '%x'.__mod__, pi), > > Here's the exception: > > ====================================================================== > ERROR: test_formatting (test.test_unicode.UnicodeTest) > ---------------------------------------------------------------------- > TypeError: 'PsuedoFloat' object is not callable > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/home/ethan/source/python/issue19995/Lib/test/test_unicode.py", line 1156, in test_formatting > self.assertRaisesRegex(TypeError, '%c'.__mod__, pi), This is certainly not the code you are showing above. Regards Antoine. From ethan at stoneleaf.us Wed Mar 19 23:17:53 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 19 Mar 2014 15:17:53 -0700 Subject: [Python-Dev] unittest assertRaisesRegex bug? In-Reply-To: <20140319231354.4a70b15c@fsol> References: <532A0E26.5060106@stoneleaf.us> <20140319231354.4a70b15c@fsol> Message-ID: <532A1791.5030505@stoneleaf.us> On 03/19/2014 03:13 PM, Antoine Pitrou wrote: > On Wed, 19 Mar 2014 14:37:42 -0700 > Ethan Furman wrote: >> Here's the code in question: >> >> class PsuedoFloat: >> def __init__(self, value): >> self.value = float(value) >> def __int__(self): >> return int(self.value) >> >> pi = PsuedoFloat(3.1415) >> >> self.assertRaisesRegex(TypeError, '%x format: an integer is required, not PsuedoFloat', '%x'.__mod__, pi), >> >> Here's the exception: >> >> ====================================================================== >> ERROR: test_formatting (test.test_unicode.UnicodeTest) >> ---------------------------------------------------------------------- >> TypeError: 'PsuedoFloat' object is not callable >> >> During handling of the above exception, another exception occurred: >> >> Traceback (most recent call last): >> File "/home/ethan/source/python/issue19995/Lib/test/test_unicode.py", line 1156, in test_formatting >> self.assertRaisesRegex(TypeError, '%c'.__mod__, pi), > > This is certainly not the code you are showing above. More words, please! :) Do you mean you agree it's a bug, or do you mean you think I misstepped in reporting what's going on? -- ~Ethan~ From solipsis at pitrou.net Wed Mar 19 23:57:51 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Mar 2014 23:57:51 +0100 Subject: [Python-Dev] unittest assertRaisesRegex bug? References: <532A0E26.5060106@stoneleaf.us> <20140319231354.4a70b15c@fsol> <532A1791.5030505@stoneleaf.us> Message-ID: <20140319235751.2ecae7b1@fsol> On Wed, 19 Mar 2014 15:17:53 -0700 Ethan Furman wrote: > On 03/19/2014 03:13 PM, Antoine Pitrou wrote: > > On Wed, 19 Mar 2014 14:37:42 -0700 > > Ethan Furman wrote: > >> Here's the code in question: > >> > >> class PsuedoFloat: > >> def __init__(self, value): > >> self.value = float(value) > >> def __int__(self): > >> return int(self.value) > >> > >> pi = PsuedoFloat(3.1415) > >> > >> self.assertRaisesRegex(TypeError, '%x format: an integer is required, not PsuedoFloat', '%x'.__mod__, pi), > >> > >> Here's the exception: > >> > >> ====================================================================== > >> ERROR: test_formatting (test.test_unicode.UnicodeTest) > >> ---------------------------------------------------------------------- > >> TypeError: 'PsuedoFloat' object is not callable > >> > >> During handling of the above exception, another exception occurred: > >> > >> Traceback (most recent call last): > >> File "/home/ethan/source/python/issue19995/Lib/test/test_unicode.py", line 1156, in test_formatting > >> self.assertRaisesRegex(TypeError, '%c'.__mod__, pi), > > > > This is certainly not the code you are showing above. > > More words, please! :) I mean the line shown just above in the traceback does not match the code you presented at the top, and that line clearly has a missing regex pattern. Regards Antoine. From ethan at stoneleaf.us Thu Mar 20 00:13:01 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 19 Mar 2014 16:13:01 -0700 Subject: [Python-Dev] unittest assertRaisesRegex bug? In-Reply-To: <20140319235751.2ecae7b1@fsol> References: <532A0E26.5060106@stoneleaf.us> <20140319231354.4a70b15c@fsol> <532A1791.5030505@stoneleaf.us> <20140319235751.2ecae7b1@fsol> Message-ID: <532A247D.9020500@stoneleaf.us> On 03/19/2014 03:57 PM, Antoine Pitrou wrote: > On Wed, 19 Mar 2014 15:17:53 -0700 > Ethan Furman wrote: >> On 03/19/2014 03:13 PM, Antoine Pitrou wrote: >>> On Wed, 19 Mar 2014 14:37:42 -0700 >>> Ethan Furman wrote: >>>> Here's the code in question: >>>> >>>> class PsuedoFloat: >>>> def __init__(self, value): >>>> self.value = float(value) >>>> def __int__(self): >>>> return int(self.value) >>>> >>>> pi = PsuedoFloat(3.1415) >>>> >>>> self.assertRaisesRegex(TypeError, '%x format: an integer is required, not PsuedoFloat', '%x'.__mod__, pi), >>>> >>>> Here's the exception: >>>> >>>> ====================================================================== >>>> ERROR: test_formatting (test.test_unicode.UnicodeTest) >>>> ---------------------------------------------------------------------- >>>> TypeError: 'PsuedoFloat' object is not callable >>>> >>>> During handling of the above exception, another exception occurred: >>>> >>>> Traceback (most recent call last): >>>> File "/home/ethan/source/python/issue19995/Lib/test/test_unicode.py", line 1156, in test_formatting >>>> self.assertRaisesRegex(TypeError, '%c'.__mod__, pi), >>> >>> This is certainly not the code you are showing above. >> >> More words, please! :) > > I mean the line shown just above in the traceback does not match the > code you presented at the top, and that line clearly has a missing > regex pattern. A regex pattern can be a literal, yes? In which case exception -> TypeError regex -> '%x format: an integer is required, not PsuedoFloat' callable -> '%x'.__mod__ *args -> pi **kwargs -> None So, unless you can point to where I've gone wrong with the above (which is why I posted in the first place), I think we have a bug in unittest. Also: self.assertRaisesRegex(TypeError, '%x format: an integer is required, not float','%x'.__mod__, 3.14), self.assertRaisesRegex(TypeError, '%X format: an integer is required, not float','%X'.__mod__, 2.11), self.assertRaisesRegex(TypeError, '%o format: an integer is required, not float','%o'.__mod__, 1.79), these lines all work just fine. -- ~Ethan~ From thomas at python.org Thu Mar 20 00:41:10 2014 From: thomas at python.org (Thomas Wouters) Date: Wed, 19 Mar 2014 16:41:10 -0700 Subject: [Python-Dev] unittest assertRaisesRegex bug? In-Reply-To: <532A247D.9020500@stoneleaf.us> References: <532A0E26.5060106@stoneleaf.us> <20140319231354.4a70b15c@fsol> <532A1791.5030505@stoneleaf.us> <20140319235751.2ecae7b1@fsol> <532A247D.9020500@stoneleaf.us> Message-ID: On Wed, Mar 19, 2014 at 4:13 PM, Ethan Furman wrote: > On 03/19/2014 03:57 PM, Antoine Pitrou wrote: > >> On Wed, 19 Mar 2014 15:17:53 -0700 >> Ethan Furman wrote: >> >>> On 03/19/2014 03:13 PM, Antoine Pitrou wrote: >>> >>>> On Wed, 19 Mar 2014 14:37:42 -0700 >>>> Ethan Furman wrote: >>>> >>>>> Here's the code in question: >>>>> >>>>> class PsuedoFloat: >>>>> def __init__(self, value): >>>>> self.value = float(value) >>>>> def __int__(self): >>>>> return int(self.value) >>>>> >>>>> pi = PsuedoFloat(3.1415) >>>>> >>>>> self.assertRaisesRegex(TypeError, '%x format: an integer >>>>> is required, not PsuedoFloat', '%x'.__mod__, pi), >>>>> >>>>> Here's the exception: >>>>> >>>>> ====================================================================== >>>>> ERROR: test_formatting (test.test_unicode.UnicodeTest) >>>>> ---------------------------------------------------------------------- >>>>> TypeError: 'PsuedoFloat' object is not callable >>>>> >>>>> During handling of the above exception, another exception occurred: >>>>> >>>>> Traceback (most recent call last): >>>>> File "/home/ethan/source/python/issue19995/Lib/test/test_unicode.py", >>>>> line 1156, in test_formatting >>>>> self.assertRaisesRegex(TypeError, '%c'.__mod__, pi), >>>>> >>>> >>>> This is certainly not the code you are showing above. >>>> >>> >>> More words, please! :) >>> >> >> I mean the line shown just above in the traceback does not match the >> code you presented at the top, and that line clearly has a missing >> regex pattern. >> > > A regex pattern can be a literal, yes? In which case > > exception -> TypeError > regex -> '%x format: an integer is required, not PsuedoFloat' > callable -> '%x'.__mod__ > *args -> pi > **kwargs -> None > > So, unless you can point to where I've gone wrong with the above (which is > why I posted in the first place), I think we have a bug in unittest. > > Also: > > self.assertRaisesRegex(TypeError, '%x format: an integer is > required, not float','%x'.__mod__, 3.14), > self.assertRaisesRegex(TypeError, '%X format: an integer is > required, not float','%X'.__mod__, 2.11), > self.assertRaisesRegex(TypeError, '%o format: an integer is > required, not float','%o'.__mod__, 1.79), > > these lines all work just fine. > What Antoine is trying to tell you is that the traceback you pasted shows this: File "/home/ethan/source/python/issue19995/Lib/test/test_unicode.py", line 1156, in test_formatting self.assertRaisesRegex(TypeError, '%c'.__mod__, pi), ... which is passing '%c'.__mod__ as the 'regex' argument. '%c'.__mod__ is a method of a builtin type, a 'method-wrapper' object, which is why you get the error you're getting: AttributeError: 'method-wrapper' object has no attribute 'search'. -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Thu Mar 20 01:24:54 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 19 Mar 2014 20:24:54 -0400 Subject: [Python-Dev] unit tests for error messages In-Reply-To: References: <5329D99B.50407@stoneleaf.us> <20140319195532.5ef7a721@fsol> Message-ID: <20140320002454.99676250CC2@webabinitio.net> On Wed, 19 Mar 2014 20:32:38 +0100, Georg Brandl wrote: > Am 19.03.2014 19:55, schrieb Antoine Pitrou: > > On Wed, 19 Mar 2014 10:53:31 -0700 > > Ethan Furman wrote: > > > >> I just made a change to some error messages [1] (really, just one): > >> > >> old behavior: > >> > >> '%o' % 3.14 > >> 'float' object cannot be interpreted as an integer > >> > >> new behavior: > >> > >> '%o' % 3.14 > >> %o format: an integer is required, not float > >> > >> Would we normally add a test for that? > > > > Depends if you're fearing a regression. > > If he fears regressions in error messages, he shouldn't be changing them > in the first place :) In this type of situation I will often write a unit test that makes sure that the piece of information I just added to the message is really in the message ('%o', in this case), using assertRaisesRegex. I don't think it is required, but I like to do it, because it would indeed be a regression if that information stopped appearing in the message. --David From steve at pearwood.info Thu Mar 20 01:32:48 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 20 Mar 2014 11:32:48 +1100 Subject: [Python-Dev] Making proxy types easier to write and maintain In-Reply-To: <20140319230139.043aee2f@fsol> References: <20140319194607.331336db@fsol> <20140319212242.5d25904c@fsol> <20140319230139.043aee2f@fsol> Message-ID: <20140320003248.GP16526@ando> On Wed, Mar 19, 2014 at 11:01:39PM +0100, Antoine Pitrou wrote: > On Thu, 20 Mar 2014 07:54:39 +1000 > Nick Coghlan wrote: > > > > Graeme Dumpleton has also subsequently written a library to handle easier > > creation of correct proxy types: https://pypi.python.org/pypi/wrapt > > (is "Graeme" another spelling for "Graham"?) Yes. > > It isn't as simple as changing one lookup function though - abstract.c > > reads the type slots directly, and those are already hairy performance > > critical code paths (especially the binary operator type dispatch). > > Oh indeed, you're right. I think my proposal is taking water. Nevertheless, I would like to see a standard way to proxy dunder methods. -- Steven From rdmurray at bitdance.com Thu Mar 20 01:45:19 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 19 Mar 2014 20:45:19 -0400 Subject: [Python-Dev] unittest assertRaisesRegex bug? In-Reply-To: References: <532A0E26.5060106@stoneleaf.us> <20140319231354.4a70b15c@fsol> <532A1791.5030505@stoneleaf.us> <20140319235751.2ecae7b1@fsol> <532A247D.9020500@stoneleaf.us> Message-ID: <20140320004520.5E566250CC2@webabinitio.net> On Wed, 19 Mar 2014 16:41:10 -0700, Thomas Wouters wrote: > On Wed, Mar 19, 2014 at 4:13 PM, Ethan Furman wrote: > > > On 03/19/2014 03:57 PM, Antoine Pitrou wrote: > > > >> On Wed, 19 Mar 2014 15:17:53 -0700 > >> Ethan Furman wrote: > >> > >>> On 03/19/2014 03:13 PM, Antoine Pitrou wrote: > >>> > >>>> On Wed, 19 Mar 2014 14:37:42 -0700 > >>>> Ethan Furman wrote: > >>>>> During handling of the above exception, another exception occurred: > >>>>> > >>>>> Traceback (most recent call last): > >>>>> File "/home/ethan/source/python/issue19995/Lib/test/test_unicode.py", > >>>>> line 1156, in test_formatting > >>>>> self.assertRaisesRegex(TypeError, '%c'.__mod__, pi), > >>>>> > >>>> > >>>> This is certainly not the code you are showing above. > >>>> > >>> > >>> More words, please! :) > >>> > >> > >> I mean the line shown just above in the traceback does not match the > >> code you presented at the top, and that line clearly has a missing > >> regex pattern. > >> > > > > A regex pattern can be a literal, yes? In which case > > > > exception -> TypeError > > regex -> '%x format: an integer is required, not PsuedoFloat' > > callable -> '%x'.__mod__ > > *args -> pi > > **kwargs -> None > > > > So, unless you can point to where I've gone wrong with the above (which is > > why I posted in the first place), I think we have a bug in unittest. > > > > Also: > > > > self.assertRaisesRegex(TypeError, '%x format: an integer is > > required, not float','%x'.__mod__, 3.14), > > self.assertRaisesRegex(TypeError, '%X format: an integer is > > required, not float','%X'.__mod__, 2.11), > > self.assertRaisesRegex(TypeError, '%o format: an integer is > > required, not float','%o'.__mod__, 1.79), > > > > these lines all work just fine. > > > > What Antoine is trying to tell you is that the traceback you pasted shows > this: > > File "/home/ethan/source/python/issue19995/Lib/test/test_unicode.py", > line 1156, in test_formatting > self.assertRaisesRegex(TypeError, '%c'.__mod__, pi), > > ... which is passing '%c'.__mod__ as the 'regex' argument. '%c'.__mod__ is > a method of a builtin type, a 'method-wrapper' object, which is why you get > the error you're getting: AttributeError: 'method-wrapper' object has no > attribute 'search'. I think http://bugs.python.org/issue20145 is relevant here. --David From ethan at stoneleaf.us Thu Mar 20 00:59:22 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 19 Mar 2014 16:59:22 -0700 Subject: [Python-Dev] unittest assertRaisesRegex bug? In-Reply-To: References: <532A0E26.5060106@stoneleaf.us> <20140319231354.4a70b15c@fsol> <532A1791.5030505@stoneleaf.us> <20140319235751.2ecae7b1@fsol> <532A247D.9020500@stoneleaf.us> Message-ID: <532A2F5A.9000302@stoneleaf.us> On 03/19/2014 04:41 PM, Thomas Wouters wrote: > > What Antoine is trying to tell you is that the traceback you pasted shows this: > > File "/home/ethan/source/python/__issue19995/Lib/test/test___unicode.py", line 1156, in test_formatting > self.assertRaisesRegex(__TypeError, '%c'.__mod__, pi), > > ... which is passing '%c'.__mod__ as the 'regex' argument. Argh. Thank you, Thomas. Thank you, and my apologies for being so dense, Antoine. -- ~Ethan~ From ncoghlan at gmail.com Thu Mar 20 04:49:34 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Mar 2014 13:49:34 +1000 Subject: [Python-Dev] Making proxy types easier to write and maintain In-Reply-To: <20140320003248.GP16526@ando> References: <20140319194607.331336db@fsol> <20140319212242.5d25904c@fsol> <20140319230139.043aee2f@fsol> <20140320003248.GP16526@ando> Message-ID: On 20 Mar 2014 10:34, "Steven D'Aprano" wrote: > > On Wed, Mar 19, 2014 at 11:01:39PM +0100, Antoine Pitrou wrote: > > On Thu, 20 Mar 2014 07:54:39 +1000 > > Nick Coghlan wrote: > > > > > > Graeme Dumpleton has also subsequently written a library to handle easier > > > creation of correct proxy types: https://pypi.python.org/pypi/wrapt > > > > (is "Graeme" another spelling for "Graham"?) > > Yes. > > > > > It isn't as simple as changing one lookup function though - abstract.c > > > reads the type slots directly, and those are already hairy performance > > > critical code paths (especially the binary operator type dispatch). > > > > Oh indeed, you're right. I think my proposal is taking water. > > Nevertheless, I would like to see a standard way to proxy dunder > methods. I had vague plans to bug Graham (yay, spelling!) about proposing wrapt for inclusion in 3.5 (possibly split between functools and types rather than as a completely new module, though). functools.wraps was always intended as a near term workaround that, like most sufficiently effective workarounds, ended lasting years before its limitations irritated anyone enough for them to go back and do it right :) Cheers, Nick. > > > -- > Steven > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Thu Mar 20 08:17:05 2014 From: larry at hastings.org (Larry Hastings) Date: Thu, 20 Mar 2014 00:17:05 -0700 Subject: [Python-Dev] Making proxy types easier to write and maintain In-Reply-To: References: <20140319194607.331336db@fsol> <20140319212242.5d25904c@fsol> <20140319230139.043aee2f@fsol> <20140320003248.GP16526@ando> Message-ID: <532A95F1.6060505@hastings.org> On 03/19/2014 08:49 PM, Nick Coghlan wrote: > > I had vague plans to bug Graham (yay, spelling!) about proposing wrapt > for inclusion in 3.5 (possibly split between functools and types > rather than as a completely new module, though). > I pinged him about it for 3.4. He didn't have the time for it. I hope we can talk him into it for 3.5, wrapt is pretty sweet! //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Mar 20 08:49:20 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Mar 2014 17:49:20 +1000 Subject: [Python-Dev] Making proxy types easier to write and maintain In-Reply-To: <532A95F1.6060505@hastings.org> References: <20140319194607.331336db@fsol> <20140319212242.5d25904c@fsol> <20140319230139.043aee2f@fsol> <20140320003248.GP16526@ando> <532A95F1.6060505@hastings.org> Message-ID: On 20 March 2014 17:17, Larry Hastings wrote: > On 03/19/2014 08:49 PM, Nick Coghlan wrote: > > I had vague plans to bug Graham (yay, spelling!) about proposing wrapt for > inclusion in 3.5 (possibly split between functools and types rather than as > a completely new module, though). > > > I pinged him about it for 3.4. He didn't have the time for it. I hope we > can talk him into it for 3.5, wrapt is pretty sweet! So long as Graham's willing to go along with it, he doesn't have to to be the one to write the PEP. Just needs someone that understands the problem it's designed to solve and is willing to write it up as a PEP, and look at the question of whether it makes more sense to drop it in wholesale as a new module, or to break it up amongst existing modules. (and python-ideas discussion can help with that). One particularly thorny question to investigate: would it be possible to *implicitly* switch functools.wraps() over to using it? I suspect the backwards compatibility implications wouldn't be acceptable, but the idea at least needs to be looked into to identify the specific problems with the concept. Cheers, Nick. > > > /arry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From larry at hastings.org Thu Mar 20 18:56:17 2014 From: larry at hastings.org (Larry Hastings) Date: Thu, 20 Mar 2014 10:56:17 -0700 Subject: [Python-Dev] Making proxy types easier to write and maintain In-Reply-To: References: <20140319194607.331336db@fsol> <20140319212242.5d25904c@fsol> <20140319230139.043aee2f@fsol> <20140320003248.GP16526@ando> <532A95F1.6060505@hastings.org> Message-ID: <532B2BC1.6040603@hastings.org> On 03/20/2014 12:49 AM, Nick Coghlan wrote: > So long as Graham's willing to go along with it, he doesn't have to to > be the one to write the PEP. PEP? Why does it need a PEP? I didn't think it'd even be a new top-level library module, it'd be an addition / additions to functools. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Mar 20 21:16:18 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 20 Mar 2014 16:16:18 -0400 Subject: [Python-Dev] Making proxy types easier to write and maintain In-Reply-To: <532B2BC1.6040603@hastings.org> References: <20140319194607.331336db@fsol> <20140319212242.5d25904c@fsol> <20140319230139.043aee2f@fsol> <20140320003248.GP16526@ando> <532A95F1.6060505@hastings.org> <532B2BC1.6040603@hastings.org> Message-ID: pypy's transparent proxy feature: http://pypy.readthedocs.org/en/latest/objspace-proxies.html#transparent-proxies On Thu, Mar 20, 2014 at 1:56 PM, Larry Hastings wrote: > On 03/20/2014 12:49 AM, Nick Coghlan wrote: > > So long as Graham's willing to go along with it, he doesn't have to to > be the one to write the PEP. > > > PEP? Why does it need a PEP? I didn't think it'd even be a new top-level > library module, it'd be an addition / additions to functools. > > > /arry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > From AHettinger at Prominic.NET Fri Mar 21 02:42:17 2014 From: AHettinger at Prominic.NET (Andrew M. Hettinger) Date: Thu, 20 Mar 2014 20:42:17 -0500 Subject: [Python-Dev] 'Add/Remove Programs' entry missing for 'current user only' 32-bit installations on 64-bit Windows In-Reply-To: References: <5329BA05.5030306@v.loewis.de> <5329BC83.4020808@pke.hr> Message-ID: Jurko Gospodneti? wrote on 03/19/2014 01:41:19 PM: > > Hi. > > >> Should I open a 'Add/Remove Programs' dialog related issue in the > >> CPython issue tracker? > > > > Please do. It would also good if somebody volunteered to reproduce > > the issue. > > Opened issue #20984. > > http://bugs.python.org/issue20984 > > Anyone have Windows 8 x64 available to try this out by any chance? > > Best regards, > Jurko Gospodneti? > I confirm this behavior on Windows 8.1 x64. (I added a note to the bug as well) -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Fri Mar 21 03:35:14 2014 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 21 Mar 2014 05:35:14 +0300 Subject: [Python-Dev] Dead code in json/encoder? Message-ID: It looks like _one_shot parameter is always called with True argument and unused. What is the purpose of it? https://github.com/python/cpython/blob/de1b33f6e6071816a1fc52cd5f0c6cd47d704251/Lib/json/encoder.py#L239-L249 -- anatoly t. From status at bugs.python.org Fri Mar 21 18:07:51 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 21 Mar 2014 18:07:51 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140321170751.9B8B2568C6@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-03-14 - 2014-03-21) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4511 (+25) closed 28273 (+60) total 32784 (+85) Open issues with patches: 2085 Issues opened (53) ================== #1350: IDLE - CallTips enhancement - show full doc-string in new wind http://bugs.python.org/issue1350 reopened by terry.reedy #19690: test_logging test_race failed with PermissionError http://bugs.python.org/issue19690 reopened by ned.deily #20574: Implement incremental decoder for cp65001 http://bugs.python.org/issue20574 reopened by haypo #20928: xml.etree.ElementInclude does not include nested xincludes http://bugs.python.org/issue20928 opened by James.Bailey #20932: Undefined behavior flagged by Clang 3.4 (Python 3.5 from hg) http://bugs.python.org/issue20932 opened by Jeffrey.Walton #20934: test_multiprocessing is broken by design http://bugs.python.org/issue20934 opened by schwab #20935: Support building Python with Clang sanitizer rules http://bugs.python.org/issue20935 opened by Jeffrey.Walton #20936: test_strftime: enormous allocation, fails under Clang sanitize http://bugs.python.org/issue20936 opened by Jeffrey.Walton #20937: test_socket: buffer overflow in sock_recvmsg_guts http://bugs.python.org/issue20937 opened by Jeffrey.Walton #20939: test_geturl of test_urllibnet fails with 'https://www.python.o http://bugs.python.org/issue20939 opened by ned.deily #20941: pytime.c:184 and pytime.c:218: runtime error, outside the rang http://bugs.python.org/issue20941 opened by Jeffrey.Walton #20942: _frozen_importlib should not have a __file__ attribute http://bugs.python.org/issue20942 opened by ncoghlan #20947: -Wstrict-overflow findings http://bugs.python.org/issue20947 opened by Jeffrey.Walton #20948: -Wformat=2 -Wformat-security findings http://bugs.python.org/issue20948 opened by Jeffrey.Walton #20949: Missing platform security integrations http://bugs.python.org/issue20949 opened by Jeffrey.Walton #20951: SSLSocket.send() returns 0 for non-blocking socket http://bugs.python.org/issue20951 opened by nikratio #20952: OpenSSL and RDRAND http://bugs.python.org/issue20952 opened by Jeffrey.Walton #20953: heap-buffer-overflow in obmalloc.c:987 http://bugs.python.org/issue20953 opened by Jeffrey.Walton #20954: Bug in subprocess._args_from_interpreter_flags causes MemoryEr http://bugs.python.org/issue20954 opened by schlamar #20956: tokenize module claims tokenize.tokenize returns namedtuple, b http://bugs.python.org/issue20956 opened by FunkyBob #20957: test_smptnet Fail instead of Skip if SSL-port is unavailable http://bugs.python.org/issue20957 opened by ebfe #20961: Fix usages of the note directive in the documentation http://bugs.python.org/issue20961 opened by berker.peksag #20962: Rather modest chunk size in gzip.GzipFile http://bugs.python.org/issue20962 opened by skip.montanaro #20964: Add support.check_time_delta() http://bugs.python.org/issue20964 opened by haypo #20968: mock.MagicMock does not mock __truediv__ http://bugs.python.org/issue20968 opened by Johannes.Baiter #20969: Author of EPUB version of Python docs is set to Unknown instea http://bugs.python.org/issue20969 opened by Christian.Clauss #20970: contradictory documentation for prog option of argparse http://bugs.python.org/issue20970 opened by Aaron.Meurer #20971: HTML output of difflib http://bugs.python.org/issue20971 opened by Jasvir.Singh #20973: Implement proper comparison operations for in _TotalOrderingMi http://bugs.python.org/issue20973 opened by OrangeTux #20974: email module docs say not compatible with current python versi http://bugs.python.org/issue20974 opened by Jim.Jewett #20975: Python 3.4 build info wrong in code snippet http://bugs.python.org/issue20975 opened by Zvezdoslovec #20977: pyflakes: undefined "ctype" in 2 except blocks in the email mo http://bugs.python.org/issue20977 opened by haypo #20979: Calling getdents()/readdir64() repeatedly while closing descri http://bugs.python.org/issue20979 opened by mmarkk #20980: In multiprocessing.pool, ExceptionWithTraceback should derive http://bugs.python.org/issue20980 opened by myint #20981: ssl doesn't build anymore with OpenSSL 0.9.7 or older: X509_ch http://bugs.python.org/issue20981 opened by haypo #20983: Python 3.4 'repair' Windows installation does not install pip http://bugs.python.org/issue20983 opened by Jurko.Gospodneti?? #20984: 'Add/Remove Programs' dialog missing entries for 32-bit CPytho http://bugs.python.org/issue20984 opened by Jurko.Gospodneti?? #20986: test_startup_imports fails in test_site when executed inside v http://bugs.python.org/issue20986 opened by finitemachine #20989: XML File I/O Misbehavior with open() when the flag is 'r+' http://bugs.python.org/issue20989 opened by rexperience7 #20990: pyflakes: undefined names, get_context() and main(), in multip http://bugs.python.org/issue20990 opened by haypo #20992: reading individual bytes of multiple binary files using the Py http://bugs.python.org/issue20992 opened by Tommy.Carstensen #20994: Disable TLS Compression http://bugs.python.org/issue20994 opened by dstufft #20995: Use Better Default Ciphers for the SSL Module http://bugs.python.org/issue20995 opened by dstufft #20996: Backport TLS 1.1 and 1.2 support for ssl_version http://bugs.python.org/issue20996 opened by dstufft #20997: Wrong URL fragment identifier in search result http://bugs.python.org/issue20997 opened by bmispelon #20998: fullmatch isn't matching correctly under re.IGNORECASE http://bugs.python.org/issue20998 opened by Lucretiel #20999: setlocale, getlocale succession --> ValueError or (None, None) http://bugs.python.org/issue20999 opened by albertjan #21000: json.tool ought to have a help flag http://bugs.python.org/issue21000 opened by benjamin.peterson #21002: _sre.SRE_Scanner object should have a fullmatch() method http://bugs.python.org/issue21002 opened by Gareth.Gouldstone #21006: asyncio.docs : create_subprocess_exec example does not work on http://bugs.python.org/issue21006 opened by ajaborsk #21009: Potential deadlock in concurrent futures when garbage collecti http://bugs.python.org/issue21009 opened by simon.jagoe #21011: PyArg_ParseTupleAndKeywords doesn't take const char *keywords[ http://bugs.python.org/issue21011 opened by h.venev #21012: Figure out how to best leverage pip in devinabox http://bugs.python.org/issue21012 opened by brett.cannon Most recent 15 issues with no replies (15) ========================================== #21012: Figure out how to best leverage pip in devinabox http://bugs.python.org/issue21012 #21011: PyArg_ParseTupleAndKeywords doesn't take const char *keywords[ http://bugs.python.org/issue21011 #21009: Potential deadlock in concurrent futures when garbage collecti http://bugs.python.org/issue21009 #20997: Wrong URL fragment identifier in search result http://bugs.python.org/issue20997 #20992: reading individual bytes of multiple binary files using the Py http://bugs.python.org/issue20992 #20977: pyflakes: undefined "ctype" in 2 except blocks in the email mo http://bugs.python.org/issue20977 #20969: Author of EPUB version of Python docs is set to Unknown instea http://bugs.python.org/issue20969 #20962: Rather modest chunk size in gzip.GzipFile http://bugs.python.org/issue20962 #20961: Fix usages of the note directive in the documentation http://bugs.python.org/issue20961 #20953: heap-buffer-overflow in obmalloc.c:987 http://bugs.python.org/issue20953 #20947: -Wstrict-overflow findings http://bugs.python.org/issue20947 #20942: _frozen_importlib should not have a __file__ attribute http://bugs.python.org/issue20942 #20936: test_strftime: enormous allocation, fails under Clang sanitize http://bugs.python.org/issue20936 #20912: [zipfile.py]: Make zip directory attributes more friendly for http://bugs.python.org/issue20912 #20911: urllib 'headers' is not a well defined data type http://bugs.python.org/issue20911 Most recent 15 issues waiting for review (15) ============================================= #21006: asyncio.docs : create_subprocess_exec example does not work on http://bugs.python.org/issue21006 #21000: json.tool ought to have a help flag http://bugs.python.org/issue21000 #20998: fullmatch isn't matching correctly under re.IGNORECASE http://bugs.python.org/issue20998 #20995: Use Better Default Ciphers for the SSL Module http://bugs.python.org/issue20995 #20994: Disable TLS Compression http://bugs.python.org/issue20994 #20980: In multiprocessing.pool, ExceptionWithTraceback should derive http://bugs.python.org/issue20980 #20974: email module docs say not compatible with current python versi http://bugs.python.org/issue20974 #20970: contradictory documentation for prog option of argparse http://bugs.python.org/issue20970 #20968: mock.MagicMock does not mock __truediv__ http://bugs.python.org/issue20968 #20964: Add support.check_time_delta() http://bugs.python.org/issue20964 #20961: Fix usages of the note directive in the documentation http://bugs.python.org/issue20961 #20957: test_smptnet Fail instead of Skip if SSL-port is unavailable http://bugs.python.org/issue20957 #20954: Bug in subprocess._args_from_interpreter_flags causes MemoryEr http://bugs.python.org/issue20954 #20951: SSLSocket.send() returns 0 for non-blocking socket http://bugs.python.org/issue20951 #20939: test_geturl of test_urllibnet fails with 'https://www.python.o http://bugs.python.org/issue20939 Top 10 most discussed issues (10) ================================= #20995: Use Better Default Ciphers for the SSL Module http://bugs.python.org/issue20995 42 msgs #20927: Different behaviour on Posix and Windows when using subprocess http://bugs.python.org/issue20927 15 msgs #20951: SSLSocket.send() returns 0 for non-blocking socket http://bugs.python.org/issue20951 12 msgs #17846: Building Python on Windows - Supplementary info http://bugs.python.org/issue17846 11 msgs #20906: Issues in Unicode HOWTO http://bugs.python.org/issue20906 10 msgs #20970: contradictory documentation for prog option of argparse http://bugs.python.org/issue20970 10 msgs #10141: SocketCan support http://bugs.python.org/issue10141 7 msgs #20062: Remove emacs page from devguide http://bugs.python.org/issue20062 7 msgs #20265: Bring Windows docs up to date http://bugs.python.org/issue20265 7 msgs #20935: Support building Python with Clang sanitizer rules http://bugs.python.org/issue20935 7 msgs Issues closed (60) ================== #11362: image/webp missing from mimetypes.py http://bugs.python.org/issue11362 closed by serhiy.storchaka #11448: docs for HTTPConnection.set_tunnel are ambiguous http://bugs.python.org/issue11448 closed by python-dev #13437: Provide links to the source code for every module in the docum http://bugs.python.org/issue13437 closed by akuchling #14332: Better explain "junk" concept in difflib doc http://bugs.python.org/issue14332 closed by akuchling #16245: Update html.entities.html5 dictionary and parseentities.py http://bugs.python.org/issue16245 closed by eric.araujo #16665: doc for builtin hex() is poor http://bugs.python.org/issue16665 closed by python-dev #18931: new selectors module should support devpoll on Solaris http://bugs.python.org/issue18931 closed by giampaolo.rodola #19009: Enhance HTTPResponse.readline() performance http://bugs.python.org/issue19009 closed by kristjan.jonsson #19165: Change formatter warning to DeprecationWarning in 3.5 http://bugs.python.org/issue19165 closed by brett.cannon #19255: Don't "wipe" builtins at shutdown http://bugs.python.org/issue19255 closed by serhiy.storchaka #19861: Update What's New for Python 3.4 http://bugs.python.org/issue19861 closed by r.david.murray #20379: help(instance_of_builtin_class.method) does not display self http://bugs.python.org/issue20379 closed by larry #20444: Reduce logging.config.Converting duplication of code http://bugs.python.org/issue20444 closed by python-dev #20558: ECONNRESET value in logging.config is valid with Linux [distro http://bugs.python.org/issue20558 closed by python-dev #20627: Add context manager support to xmlrpc.client.ServerProxy http://bugs.python.org/issue20627 closed by brett.cannon #20671: test_create_at_shutdown_with_encoding() of test_io hangs on "S http://bugs.python.org/issue20671 closed by haypo #20744: shutil should not use distutils http://bugs.python.org/issue20744 closed by akuchling #20879: base64 module of Python 3.4 uses 920 kB of memory http://bugs.python.org/issue20879 closed by haypo #20897: @abstractmethod does not enforce method signatures http://bugs.python.org/issue20897 closed by r.david.murray #20902: Which operand is preferred by set operations? Missing informat http://bugs.python.org/issue20902 closed by rhettinger #20909: 3.4 cherry pick: d22ef969cb82 & f5be4ea5b43e & 25dc02a2acae n http://bugs.python.org/issue20909 closed by larry #20920: Turtle module transparency. http://bugs.python.org/issue20920 closed by terry.reedy #20921: DeprecationWarning: The Windows bytes API has been deprecated, http://bugs.python.org/issue20921 closed by r.david.murray #20923: ConfigParser should nested [] in section names. http://bugs.python.org/issue20923 closed by terry.reedy #20926: Devguide: Enhance Quick Start portion of instructions for Wind http://bugs.python.org/issue20926 closed by pitrou #20929: Undefined behavior flagged by Clang 3.4 (Python 3.4-RC3) http://bugs.python.org/issue20929 closed by benjamin.peterson #20930: Debian 7.3: This platform's pyconfig.h needs to define PY_FORM http://bugs.python.org/issue20930 closed by benjamin.peterson #20931: Confusing section title "New Expected Features for Python Impl http://bugs.python.org/issue20931 closed by python-dev #20933: bad test case in test_osx_proxy_bypass (test.test_urllib2.Hand http://bugs.python.org/issue20933 closed by r.david.murray #20938: Broken link to editors page in General FAQ http://bugs.python.org/issue20938 closed by benjamin.peterson #20940: Test 239: buffer overflow in sock_recvmsg_guts http://bugs.python.org/issue20940 closed by neologix #20943: configparser - add 'getDict' function http://bugs.python.org/issue20943 closed by lukasz.langa #20944: Engineering Process Improvements http://bugs.python.org/issue20944 closed by r.david.murray #20945: why save the item to be replaced as olditem in PyTuple_SetItem http://bugs.python.org/issue20945 closed by benjamin.peterson #20946: ctypes test fixes http://bugs.python.org/issue20946 closed by python-dev #20950: asyncio.docs : asyncio.subprocess.Process.wait() method typo http://bugs.python.org/issue20950 closed by haypo #20955: Unexpected behavior in sqlite3.Connection.set_progress_handler http://bugs.python.org/issue20955 closed by berker.peksag #20958: Undefined behavior flagged by Clang 3.4 (Python 3.4.0) http://bugs.python.org/issue20958 closed by zach.ware #20959: print gives wrong error when printing *generator http://bugs.python.org/issue20959 closed by ned.deily #20960: Fix usage of the versionchanged directive in the sys.hash_info http://bugs.python.org/issue20960 closed by python-dev #20963: side_effects swapped in Mock example http://bugs.python.org/issue20963 closed by eric.araujo #20965: Clang devguide update http://bugs.python.org/issue20965 closed by python-dev #20966: Documentation Link for "Python Tkinter Resources" is currently http://bugs.python.org/issue20966 closed by python-dev #20967: hashlib memory leak http://bugs.python.org/issue20967 closed by brett.cannon #20972: python34.dll not in System32 or SysWOW64 folders after install http://bugs.python.org/issue20972 closed by mifik #20976: pyflakes: remove unused imports http://bugs.python.org/issue20976 closed by haypo #20978: pyflakes: undefined names http://bugs.python.org/issue20978 closed by haypo #20982: http://www.python.org used in unit test is directed to HTTPS u http://bugs.python.org/issue20982 closed by ned.deily #20985: Add new style formatting to logging.config.fileConfig http://bugs.python.org/issue20985 closed by vinay.sajip #20987: Python for AS/400 (OS/400) is actually 2.7 http://bugs.python.org/issue20987 closed by ned.deily #20988: Recommend ssl.create_default_context() in "Security considerat http://bugs.python.org/issue20988 closed by pitrou #20991: Issues about relative& absolute import way for Portingfrom pyt http://bugs.python.org/issue20991 closed by brett.cannon #20993: Quicklink to PEPs is wrong. http://bugs.python.org/issue20993 closed by zach.ware #21001: Python 3.4 MSI installer doesn't work http://bugs.python.org/issue21001 closed by zach.ware #21003: how to do batch processing using python http://bugs.python.org/issue21003 closed by eric.smith #21004: how to store json data into postgresql using python script http://bugs.python.org/issue21004 closed by eric.smith #21005: asyncio.docs : asyncio.subprocess.DEVNULL doc inadequate http://bugs.python.org/issue21005 closed by python-dev #21007: List of development releases in PEPs like 429 should be links http://bugs.python.org/issue21007 closed by benjamin.peterson #21008: Update devinabox for Python 3.4 http://bugs.python.org/issue21008 closed by brett.cannon #21010: asyncio doc typo http://bugs.python.org/issue21010 closed by python-dev From fijall at gmail.com Fri Mar 21 19:48:00 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 21 Mar 2014 20:48:00 +0200 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> <20140319010948.GN16526@ando> <87pplikcum.fsf@uwakimon.sk.tsukuba.ac.jp> <87wqfqi1sm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Wed, Mar 19, 2014 at 11:43 PM, Nick Coghlan wrote: > > On 20 Mar 2014 07:38, "Nick Coghlan" wrote: >> >> Correct, but I think this discussion has established that "how many times >> dict lookup calls __eq__ on the key" is one such thing. In CPython, it >> already varies based on: >> >> - dict contents (due to the identity check and the distribution of entries >> across hash buckets) >> - pointer size (due to the hash bucket distribution differing between 32 >> bit and 64 bit builds) >> - dict tuning parameters (there are some settings in the dict >> implementation that affect when dicts resize up and down, etc, which can >> mean the hash bucket distribution may already change without much notice in >> feature releases) > > I just realised that hash randomisation also comes into play here - the > distribution of entries across hash buckets is inherently variable between > runs for any key types that rely directly or indirectly on a randomised > hash. > > Cheers, > Nick. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > at the end of the day we settled for dicts with str int or identity keys, so we're perfectly safe From theller at ctypes.org Fri Mar 21 20:46:51 2014 From: theller at ctypes.org (Thomas Heller) Date: Fri, 21 Mar 2014 20:46:51 +0100 Subject: [Python-Dev] python 3.4 and pywin32 Message-ID: With python 3.4 and pywin32 version 218 it is only possible to import win32com or win32api when pywintypes has been imported before. I have no idea if this is a bug in pywin32 or in Python 3.4. Does anyone know more? Thanks, Thomas From ncoghlan at gmail.com Sat Mar 22 00:24:04 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Mar 2014 09:24:04 +1000 Subject: [Python-Dev] Intricacies of calling __eq__ In-Reply-To: References: <20140318111841.GK16526@ando> <20140318142114.GM16526@ando> <20140319010948.GN16526@ando> <87pplikcum.fsf@uwakimon.sk.tsukuba.ac.jp> <87wqfqi1sm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 22 March 2014 04:48, Maciej Fijalkowski wrote: > at the end of the day we settled for dicts with str int or identity > keys, so we're perfectly safe Ah, Armin's original investment in PyPy's type tracking infrastructure pays off yet again :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Mar 22 00:25:44 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Mar 2014 09:25:44 +1000 Subject: [Python-Dev] python 3.4 and pywin32 In-Reply-To: References: Message-ID: On 22 March 2014 05:46, Thomas Heller wrote: > With python 3.4 and pywin32 version 218 it is only possible > to import win32com or win32api when pywintypes has been imported before. > > I have no idea if this is a bug in pywin32 or in Python 3.4. > Does anyone know more? Do you have a traceback for the failed imports? There were certainly non-trivial changes to the import system in 3.4, but without knowing more details about the failures, it's hard to say what the culprits might be. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sat Mar 22 19:10:57 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 22 Mar 2014 19:10:57 +0100 Subject: [Python-Dev] 3.4 buildbots available Message-ID: <20140322191057.1bc34e4d@fsol> Hello, I've created the 3.4 category in the buildbots setup: http://buildbot.python.org/all/waterfall?category=3.4.stable I've also retired 3.2 and 3.3 buildbots. Someone will have to update the text and URLs at https://www.python.org/dev/buildbot/. Regards Antoine. From bp at benjamin-peterson.org Sat Mar 22 22:03:04 2014 From: bp at benjamin-peterson.org (Benjamin Peterson) Date: Sat, 22 Mar 2014 14:03:04 -0700 Subject: [Python-Dev] 3.4 buildbots available In-Reply-To: <20140322191057.1bc34e4d@fsol> References: <20140322191057.1bc34e4d@fsol> Message-ID: <1395522184.31368.97664617.2CD6986F@webmail.messagingengine.com> On Sat, Mar 22, 2014, at 11:10, Antoine Pitrou wrote: > > Hello, > > I've created the 3.4 category in the buildbots setup: > http://buildbot.python.org/all/waterfall?category=3.4.stable > > I've also retired 3.2 and 3.3 buildbots. Someone will have to update > the text and URLs at https://www.python.org/dev/buildbot/. We probably want a http://buildbot.python.org/3.4.stable/ redirect, too, then. -- Regards, Benjamin From ncoghlan at gmail.com Sat Mar 22 22:11:07 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Mar 2014 07:11:07 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements Message-ID: Folks, I have just posted a proposal to change the way we treat enhancements that relate to Python's support for network security enhancements. I now see these as categorically different from most other enhancements, as they have implications for the evolution of internet security as a whole, rather than being limited to affecting the security of individual applications written in Python. It is similar to the previous IDLE policy exception PEP, where we decided that cross version consistency of IDLE superseded the general policy against backporting enhancements to maintenance branches. This new PEP proposes a similar exception for network security related changes, such that the *capabilities* of all still maintained versions should be kept equivalent, with only the *default settings* varying across up to date feature releases. The policy is also designed to make it easier for us to ratchet up the default security settings in future feature releases. However, it's more complex than the IDLE PEP (since it's more invasive), and there are still a few open questions listed. This topic involves a complex balance between encouraging and supporting good security practices and limiting the risk of failures for users upgrading to new maintenance releases, so I'd ask that folks take time to read and consider the implications of the full PEP in the broader context of today's internet before posting comments on specific details, or indicating a preference one way or the other in terms of the overall proposal. Full PEP included inline below, and available in more readable form at http://www.python.org/dev/peps/pep-0466/ Regards, Nick. PEP: 466 Title: Network Security Enhancement Exception for All Branches Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan , Status: Draft Type: Informational Content-Type: text/x-rst Created: 23-Mar-2014 Post-History: 23-Mar-2014 Abstract ======== Most CPython tracker issues are classified as errors in behaviour or proposed enhancements. Most patches to fix behavioural errors are applied to all active maintenance branches. Enhancement patches are restricted to the default branch that becomes the next Python version. This PEP relaxes the latter restriction allowing enhancements to be applied to maintenance branches for standard library components that have implications for the overall security of the internet. In particular, the exception will apply to: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the ``sha`` module (Python 2 only) * the components of other networking modules that make use of these modules * the components of the ``random`` and ``os`` modules that are relevant to cryptographic applications * the version of OpenSSL bundled with the binary installers Changes to these modules will still need to undergo normal backwards compatibility assessments, but new features will be permitted where appropriate, making it easier to implement secure networked software in Python, even for software that needs to remain compatible with older feature releases of Python. Exemption Policy ================ Under this policy, the following network security related modules are granted a blanket exemption to the restriction against adding new features in maintenance releases: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the ``sha`` module (Python 2 only) This exemption applies to *all* proposals to backport backwards compatible features in these modules to active maintenance branches. This choice is made deliberately to ensure that the "feature or fix?" argument isn't simply replaced by a "security related or not?" argument. These particular modules are inherently security related, and all enhancements to them improve Python's capabilities as a platform for development of secure networked software. As part of this policy, permission is also granted to upgrade to newer feature releases of OpenSSL when preparing the binary installers for new maintenance releases of CPython. In addition to the above blanket exemption, a conditional exemption is granted for these modules that may include some network security related features: * the ``os`` module (primarily ``os.urandom``) * the ``random`` module * networking related modules that depend on one or more of the network security related modules listed above This more limited exemption for these modules requires that the *specific* enhancement being proposed for backporting needs to be justified as being network security related. If the enhancement under discussion is designed to take advantage of a new feature in one of the network security related modules, then that will be taken as implying that the enhancement is security related. Backwards Compatibility Considerations ====================================== This PEP does *not* grant any general exemptions to the usual backwards compatibility policy for maintenance releases. Instead, it is designed to make it easier to provide more "secure by default" behaviour in future feature releases, while still limiting the risk of breaking currently working software when upgrading to a new maintenance release. In *all* cases where this policy is applied to backport enhancements to maintenance releases, it MUST be possible to write cross-version compatible code that operates by "feature detection" (for example, checking for particular attributes in the module), without needing to explicitly check the Python version. It is then up to library and framework code to provide an appropriate error message or fallback behaviour if a desired feature is found to be missing. Affected APIs SHOULD be designed to allow library and application code to perform the following actions after detecting the presence of a relevant network security related feature: * explicitly opt in to more secure settings (to allow the use of enhanced security features in older versions of Python) * explicitly opt in to less secure settings (to allow the use of newer Python versions in lower security environments) * determine the default setting for the feature (this MAY require explicit Python version checks to determine the Python feature release, but MUST NOT depend on the specific maintenance release) Documentation Requirements ========================== All modules that take advantage of this policy to backport network security related enhancements to earlier Python versions MUST include a "Security Considerations" section in their documentation. In addition to any other module specific contents, this section MUST enumerate key security enhancements and fixes (with CVE identifiers where applicable), and the Evolution of this Policy ======================== The key requirement for a module to be considered for inclusion in this policy (whether under a blanket or conditional exemption) is that it must have security implications *beyond* the specific application that is written in Python and the system that application is running on. Thus the focus on network security protocols and related cryptographic infrastructure - Python is a popular choice for the development of web services and clients, and thus the capabilities of widely used Python versions have implications for the security design of other services that may be using newer versions of Python or other development languages. The intent behind this requirement is to minimise any impact that the introduction of this policy may have on the stability and compatibility of maintenance releases. It would be thoroughly counterproductive if end users became as cautious about updating to new Python maintenance releases as they are about updating to new feature releases. Motivation and Rationale ======================== This PEP can be seen as a more targeted version of the "faster standard library release cycle" proposals discussed in PEP 407 and PEP 413, focusing specifically on those areas which have implications beyond the Python community. The creation of this PEP was prompted primarily by the aging SSL support in the Python 2 series. As of March 2014, the Python 2.7 SSL module is approaching four years of age, and the SSL support in the still popular Python 2.6 release had its feature set locked six years ago. These are simply too old to provide a foundation that can be recommended in good conscience for secure networking software that operates over the public internet, especially in an era where it is becoming quite clearly evident that advanced persistent security threats are even more widespread and more indiscriminate in their targeting than had previously been understood. While the use of the system OpenSSL installation addresses many of these concerns on Linux platforms, it doesn't address all of them, and in the case of the binary installers for Windows and Mac OS X that are published on python.org, the version of OpenSSL used is entirely within the control of the Python core development team. With increased popularity comes increased responsibility, and this policy is about recognising the fact that Python's popularity and adoption has now reached a level where some of our design and policy decisions have significant implications beyond the Python development community. As one example, the Python 2 ``ssl`` module does not support the Server Name Identification standard. While it is possible to obtain SNI support by using the third party ``requests`` client library, actually doing so currently requires using not only ``requests`` and its embedded dependencies, but also half a dozen or more additional libraries. The lack of support in the Python 2 series thus serves as an impediment to making effective use of SNI on servers, as Python 2 clients will frequently fail to handle it correctly. Another more critical example is the lack of SSL hostname matching in the Python 2 standard library - it is currently necessary to rely on a third party library, such as ``requests`` or ``backports.ssl_match_hostname`` to obtain that functionality in Python 2. The Python 2 series also remains more vulnerable to remote timing attacks on security sensitive comparisons than the Python 3 series, as it lacks a standard library equivalent to the timing attack resistant ``hmac.compare_digest()`` function. While appropriate secure comparison functions can be implemented in third party extensions, may users don't even consider the problem and use ordinary equality comparisons instead - while a standard library solution doesn't automatically fix that problem, it *does* make the barrier to resolution much lower once the problem is pointed out. My position on the ongoing transition from Python 2 to Python 3 has long been that Python 2 remains a supported platform for the core development team, and that commercial support will remain available well after upstream maintenance ends. However, in the absence of this network security enhancement policy, that position is difficult to justify when it comes to software that operates over the public internet. Just as many developers consider it too difficult to develop truly secure modern networked software in C/C++ (largely due to the challenges associated with manual memory management), I consider it too difficult to develop truly secure modern networked software using the Python 2 series without introducing this network security enhancement policy, as doing so would mean reimplementing substantial portions of the standard library as third party modules to gain access to the newer underlying network security protocols and mechanisms, and then injecting those into the module namespace to override their standard library counterparts. Requiring that latent defects in an application's Unicode correctness be addressed in order to migrate to Python 3 is not a reasonable alternative recommendation, especially given the likely existence of legacy code that lacks the kind of automated regression test suite needed to help support a migration from Python 2 to Python 3. The key point of this PEP is that those situations affect more people than just the developers and users of the affected application: their existence becomes something that developers of secure networked services need to take into account as part of their security design. By making it more feasible to enhance the security of the Python 2 series, we can help contribute to the evolution of a more secure internet for all concerned. For the Python 2 series, this proposal will most obviously impact the remaining maintenance releases of Python 2.7. However, commercial redistributors that continue to offer full support for Python 2.6, and will also do so for Python 2.7 after upstream maintenance ends, may choose to take this policy into account when deciding what changes to backport to their own maintenance updates. Providing commercial redistributors such as ActiveState, Attachmate, Canonical, Continuum Analytics, Enthought, and Red Hat that option is one of the benefits offered by requiring feature detection based forward compatibility for the modules covered by this policy. Open Questions ============== * What are the risks associated with allowing OpenSSL to be updated to new feature versions in the Windows and Mac OS X binary installers for maintenance releases? * Are there any other security relevant modules that should be covered by either a blanket or conditional exemption? * Should we enumerate a specific list of "other networking modules" rather than leaving it implicit? Acknowledgement =============== Thanks to Christian Heimes for his recent efforts on greatly improving Python's SSL support in the Python 3 series, and a variety of members of the Python community for helping me to better understand the implications of the default settings we provide in our SSL modules, and the impact that tolerating the use of SSL infrastructure that was defined in 2010 (Python 2.7) or even 2008 (Python 2.6) potentially has for the security of the web as a whole. Christian and Donald Stufft also provided valuable feedback on a preliminary draft of this proposal. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From alex.gaynor at gmail.com Sat Mar 22 22:17:58 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sat, 22 Mar 2014 21:17:58 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?PEP_466=3A_Proposed_policy_change_for_hand?= =?utf-8?q?ling_network=09security_enhancements?= References: Message-ID: Thanks for putting this together Nick. I suspect it goes without saying that I'm wildly +1 on this as a whole. I'm in favor of leaving it somewhat implicit as to exactly which networking modules concern "the health of the internet as a whole". Alex From benjamin at python.org Sat Mar 22 22:32:56 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 22 Mar 2014 14:32:56 -0700 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: Message-ID: <1395523976.4237.97669609.00EDB3FA@webmail.messagingengine.com> On Sat, Mar 22, 2014, at 14:11, Nick Coghlan wrote: > Folks, > > I have just posted a proposal to change the way we treat enhancements > that relate to Python's support for network security enhancements. I think the PEP should also address "security-mode" releases. Do the same exceptions apply? Does anyone really want to backport features to Python 3.1? From cory at lukasa.co.uk Sat Mar 22 22:17:17 2014 From: cory at lukasa.co.uk (Cory Benfield) Date: Sat, 22 Mar 2014 21:17:17 +0000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: Message-ID: On 22 March 2014 at 21:11:24, Nick Coghlan (ncoghlan at gmail.com(mailto:ncoghlan at gmail.com)) wrote: > Folks, > > I have just posted a proposal to change the way we treat enhancements > that relate to Python's support for network security enhancements. I > now see these as categorically different from most other enhancements, > as they have implications for the evolution of internet security as a > whole, rather than being limited to affecting the security of > individual applications written in Python. I am 100%, overwhelmingly in favour of this. Without this PEP, Python 2.7 is a security liability, any it becomes nothing short of irresponsible to use Python 2.7 for any form of networking code that hits the open internet. On top of that, the current state of the ssl module means that Python 2.7 and earlier cannot ever support a standard-compliant implementation of things like HTTP/2. That?s a fairly tragic state of affairs for 2.7, especially if we?re supposed to claim with a straight face that it?s acceptable to still use Python 2.7. Treat this as my strong +1. Additionally, I?m happy to volunteer my time and limited expertise to help make this happen. I?ll help work on back porting things, review code, write docs: whatever it takes to get this to happen. From tseaver at palladion.com Sat Mar 22 22:54:27 2014 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 22 Mar 2014 17:54:27 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: Message-ID: <532E0693.7000405@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/22/2014 05:11 PM, Nick Coghlan wrote: > In addition to any other module specific contents, this section MUST > enumerate key security enhancements and fixes (with CVE identifiers > where applicable), and the Incomplete sentence. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlMuBosACgkQ+gerLs4ltQ74ngCfS7MLZHtVfS7f6x+9mnOIsp+c +CUAoIc5rFIuNJvMephnlAuPhkPXZspy =N8Xk -----END PGP SIGNATURE----- From donald at stufft.io Sat Mar 22 23:05:50 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 22 Mar 2014 18:05:50 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <1395523976.4237.97669609.00EDB3FA@webmail.messagingengine.com> References: <1395523976.4237.97669609.00EDB3FA@webmail.messagingengine.com> Message-ID: <44E29DB4-2DE1-4493-9314-EB3944D3D99C@stufft.io> I think the pep doesn't mandate that someone does. It still requires someone to care enough to actually write the patch. It just allows such a patch to be merged. > On Mar 22, 2014, at 5:32 PM, Benjamin Peterson wrote: > > Does anyone really want to backport features to Python 3.1? From martin at v.loewis.de Sat Mar 22 23:16:08 2014 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 22 Mar 2014 23:16:08 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: Message-ID: <532E0BA8.5070709@v.loewis.de> Am 22.03.14 22:17, schrieb Cory Benfield: > I am 100%, overwhelmingly in favour of this. Without this PEP, Python 2.7 > is a security liability, any it becomes nothing short of irresponsible to > use Python 2.7 for any form of networking code that hits the open > internet. Agreed (although this might be a slight overstatement if taken literally). However, the right consequence should be to use Python 3.4 instead. Regards, Martin From donald at stufft.io Sat Mar 22 23:21:48 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 22 Mar 2014 18:21:48 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <532E0BA8.5070709@v.loewis.de> References: <532E0BA8.5070709@v.loewis.de> Message-ID: <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> "Just use Python 3.4" ignores the reality of production software. I wish it were that simple because I love 3.4 > On Mar 22, 2014, at 6:16 PM, "Martin v. L?wis" wrote: > > Am 22.03.14 22:17, schrieb Cory Benfield: >> I am 100%, overwhelmingly in favour of this. Without this PEP, Python 2.7 >> is a security liability, any it becomes nothing short of irresponsible to >> use Python 2.7 for any form of networking code that hits the open >> internet. > > Agreed (although this might be a slight overstatement if taken > literally). However, the right consequence should be to use Python 3.4 > instead. > > Regards, > Martin > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io From martin at v.loewis.de Sat Mar 22 23:14:19 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 22 Mar 2014 23:14:19 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: Message-ID: <532E0B3B.7070005@v.loewis.de> Am 22.03.14 22:11, schrieb Nick Coghlan: > Title: Network Security Enhancement Exception for All Branches I'm -0 on the general idea, and -1 on the inclusion of the 2.7 branch into the policy. The PEP trades security concerns against stability and maintainability. I see a maintenance threat coming out of it: existing installations will fail to operate after updating Python and some support libraries. Bug fix releases ought to focus on maintenance much more than moving along with general security threats. Software that ages needs to be retired, instead of trying to fix it endlessly (which is a futile exerciese, anyway). Everybody knows it, and has accepted it, so why should we fight what seems to be a fundamental law of software evolution? In the specific case of 2.7, I am hoping that we retire the branch soon. If so, it would be counter-productive to backport the current features to 2.7. They are still in progress, and users wouldn't be helped with getting a new feature in just the final bug fix release of 2.7, with no chance of then getting updates of the same feature, as there won't be any further bugfix releases. Finally, doing this in the 2.7 branch likely involves more effort than I'm personally willing to provide. Regards, Martin From martin at v.loewis.de Sat Mar 22 23:23:31 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 22 Mar 2014 23:23:31 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <44E29DB4-2DE1-4493-9314-EB3944D3D99C@stufft.io> References: <1395523976.4237.97669609.00EDB3FA@webmail.messagingengine.com> <44E29DB4-2DE1-4493-9314-EB3944D3D99C@stufft.io> Message-ID: <532E0D63.9050801@v.loewis.de> Am 22.03.14 23:05, schrieb Donald Stufft: > I think the pep doesn't mandate that someone does. It still requires someone to care enough to actually write the patch. It just allows such a patch to be merged. Does it actually? Unfortunately, I never got around to writing the PEP on security-only branches, so the terminology of PEP 466 is unclear. It says "maintenance brances", which could mean three things: - all branches that have ever been made off default in hg.python.org/cpython (i.e. back to 2.0). This would be the traditional definition of "maintenance branch" in a version control system, but is probably not intended by the PEP - all branches from which still releases are made (whether security-only or bug fix) - all bug fix branches Clarification of this is just a minor nit, though - although the maintainers of the old maintenance branches should (IMO) have a say whether they accept a certain change on a certain branch. Regards, Martin From ncoghlan at gmail.com Sat Mar 22 23:25:03 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Mar 2014 08:25:03 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <532E0B3B.7070005@v.loewis.de> References: <532E0B3B.7070005@v.loewis.de> Message-ID: On 23 March 2014 08:14, "Martin v. L?wis" wrote: > Am 22.03.14 22:11, schrieb Nick Coghlan: > > Finally, doing this in the 2.7 branch likely involves more effort > than I'm personally willing to provide. Believe me, I'll be escalating these concerns at work this week. We have to shoulder a lot of the responsibility for creating this problem in the first place, so I think it's fair for people to expect us to help fix it. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From martin at v.loewis.de Sat Mar 22 23:26:42 2014 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 22 Mar 2014 23:26:42 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> Message-ID: <532E0E22.6080304@v.loewis.de> Am 22.03.14 23:21, schrieb Donald Stufft: > "Just use Python 3.4" ignores the reality of production software. I wish it were that simple because I love 3.4 But I think so does the PEP (i.e. ignore the reality of production software). The risk of breaking existing installations (which might not be affected by the purported security threat being fixed) is just too high. Users that feel threatened by the lack of SSL features in Python should have long started planning on how to get to Python 3. It's not that they haven't been warned. Regards, Martin From ncoghlan at gmail.com Sat Mar 22 23:34:34 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Mar 2014 08:34:34 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <532E0D63.9050801@v.loewis.de> References: <1395523976.4237.97669609.00EDB3FA@webmail.messagingengine.com> <44E29DB4-2DE1-4493-9314-EB3944D3D99C@stufft.io> <532E0D63.9050801@v.loewis.de> Message-ID: On 23 March 2014 08:23, "Martin v. L?wis" wrote: > Am 22.03.14 23:05, schrieb Donald Stufft: >> I think the pep doesn't mandate that someone does. It still requires someone to care enough to actually write the patch. It just allows such a patch to be merged. > > Does it actually? Unfortunately, I never got around to writing the PEP > on security-only branches, so the terminology of PEP 466 is unclear. > It says "maintenance brances", which could mean three things: > - all branches that have ever been made off default in > hg.python.org/cpython (i.e. back to 2.0). This would be the > traditional definition of "maintenance branch" in a version control > system, but is probably not intended by the PEP > - all branches from which still releases are made (whether security-only > or bug fix) > - all bug fix branches > > Clarification of this is just a minor nit, though - although the > maintainers of the old maintenance branches should (IMO) have a say > whether they accept a certain change on a certain branch. Both good points, I'll clarify them in the PEP. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From nad at acm.org Sat Mar 22 23:39:28 2014 From: nad at acm.org (Ned Deily) Date: Sat, 22 Mar 2014 15:39:28 -0700 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements References: Message-ID: In article , Nick Coghlan wrote: > I have just posted a proposal to change the way we treat enhancements > that relate to Python's support for network security enhancements. +1 [...] > Open Questions > ============== > > * What are the risks associated with allowing OpenSSL to be updated to > new feature versions in the Windows and Mac OS X binary installers for > maintenance releases? Regarding the python.org binary installers, I think past practice has been for us to update third-party libraries as necessary in maintenance releases when there is good cause and with the concurrence of the release manager, so I don't see this as a big issue. For the OS X binary installer, the issue for OpenSSL has been that we dynamically link to the system-supplied OpenSSL libraries and that, for various reasons, Apple has deprecated (and frozen at non-current OpenSSL releases) the use of those libraries in favor of their own security frameworks. So, for multiple reasons, including the risk that OpenSSL may be dropped from an upcoming major release of OS X, we need to start supplying our own version with all OS X binary installers. That's the plan regardless of the outcome of this PEP. -- Ned Deily, nad at acm.org From ncoghlan at gmail.com Sat Mar 22 23:33:35 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Mar 2014 08:33:35 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <532E0BA8.5070709@v.loewis.de> References: <532E0BA8.5070709@v.loewis.de> Message-ID: On 23 March 2014 08:16, "Martin v. L?wis" wrote: > Am 22.03.14 22:17, schrieb Cory Benfield: >> I am 100%, overwhelmingly in favour of this. Without this PEP, Python 2.7 >> is a security liability, any it becomes nothing short of irresponsible to >> use Python 2.7 for any form of networking code that hits the open >> internet. > > Agreed (although this might be a slight overstatement if taken > literally). However, the right consequence should be to use Python 3.4 > instead. The PEP does try to address this point in the Motivation & Rationale sections: ================= Requiring that latent defects in an application's Unicode correctness be addressed in order to migrate to Python 3 is not a reasonable alternative recommendation, especially given the likely existence of legacy code that lacks the kind of automated regression test suite needed to help support a migration from Python 2 to Python 3. The key point of this PEP is that those situations affect more people than just the developers and users of the affected application: their existence becomes something that developers of secure networked services need to take into account as part of their security design. By making it more feasible to enhance the security of the Python 2 series, we can help contribute to the evolution of a more secure internet for all concerned. ================= Hard to maintain legacy software is a fact of life, and way too much of it is exposed to the public internet. This PEP is about doing what we can to mitigate the damage caused both by other people's mistakes, and also the inherent challenges of migrating from the error prone POSIX text model to something more reasonable. I *don't* think its reasonable to expect us to do this without support from the corporate users that caused the problem in the first place (by continuing to deploy older versions of Python without investing adequately in their upkeep), so I'd encourage everyone employed by a commercial user of Python to remind their management chains of the risks of failing to invest development time in any upstream dependencies that they expect to keep pace with the dynamic nature of the internet. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From martin at v.loewis.de Sat Mar 22 23:40:53 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 22 Mar 2014 23:40:53 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> Message-ID: <532E1175.5090902@v.loewis.de> Am 22.03.14 23:33, schrieb Nick Coghlan: > Hard to maintain legacy software is a fact of life, and way too much > of it is exposed to the public internet. This PEP is about doing what > we can to mitigate the damage caused both by other people's mistakes, > and also the inherent challenges of migrating from the error prone > POSIX text model to something more reasonable. > > I *don't* think its reasonable to expect us to do this without support > from the corporate users that caused the problem in the first place > (by continuing to deploy older versions of Python without investing > adequately in their upkeep), so I'd encourage everyone employed by a > commercial user of Python to remind their management chains of the > risks of failing to invest development time in any upstream > dependencies that they expect to keep pace with the dynamic nature of > the internet. I hope indeed you are successful in activating resources. However, putting them on this backporting project seems like a waste. They should rather go into porting stuff to 3.x where people need it. As responsible maintainers, we should just advise our users that Python 2.7 is a dead horse, and that they should stop riding it. More professionally, we should set an official end-of-life date for 2.7 (alas, we should have done that two years ago). I hope that the language summit can agree to stopping bug fix releases for 2.7 in 2014. Regards, Martin From mal at egenix.com Sat Mar 22 23:52:12 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 22 Mar 2014 23:52:12 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: Message-ID: <532E141C.3030800@egenix.com> On 22.03.2014 22:11, Nick Coghlan wrote: > PEP: 466 > Title: Network Security Enhancement Exception for All Branches +1 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 22 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-03-29: PythonCamp 2014, Cologne, Germany ... 7 days to go 2014-04-09: PyCon 2014, Montreal, Canada ... 18 days to go 2014-04-29: Python Meeting Duesseldorf ... 38 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Sat Mar 22 23:53:38 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Mar 2014 08:53:38 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <532E1175.5090902@v.loewis.de> References: <532E0BA8.5070709@v.loewis.de> <532E1175.5090902@v.loewis.de> Message-ID: On 23 March 2014 08:40, "Martin v. L?wis" wrote: > Am 22.03.14 23:33, schrieb Nick Coghlan: >> Hard to maintain legacy software is a fact of life, and way too much >> of it is exposed to the public internet. This PEP is about doing what >> we can to mitigate the damage caused both by other people's mistakes, >> and also the inherent challenges of migrating from the error prone >> POSIX text model to something more reasonable. >> >> I *don't* think its reasonable to expect us to do this without support >> from the corporate users that caused the problem in the first place >> (by continuing to deploy older versions of Python without investing >> adequately in their upkeep), so I'd encourage everyone employed by a >> commercial user of Python to remind their management chains of the >> risks of failing to invest development time in any upstream >> dependencies that they expect to keep pace with the dynamic nature of >> the internet. > > I hope indeed you are successful in activating resources. However, > putting them on this backporting project seems like a waste. They > should rather go into porting stuff to 3.x where people need it. Red Hat will be supporting Python 2.6 until 2020'ish, Python 2.7 until at least 2024 (since RHEL7 isn't actually out yet). Python 2 is part of one of our core products (RHEL), and a dependency of another one (OpenStack). While the wheels are in motion upstream to migrate both to Python 3, that's a long term project in both cases, and really hard to make a business case for. While we do believe the open source way delivers better software in the long run, we still have to make our case when we want the company to spend time and resources on something. > As responsible maintainers, we should just advise our users that > Python 2.7 is a dead horse, and that they should stop riding it. > More professionally, we should set an official end-of-life date > for 2.7 (alas, we should have done that two years ago). This completely ignores the practical realities of commercial software development, and the commitments that vendors have made to their customers. I agree it would be completely unreasonable to ask volunteers to contribute their own time to making this proposal work. However, it's likely to be much easier for those of us with influence in the commercial world to shake some resources loose if we have a pre-arranged agreement on the specific help we need to better support their interests. > I hope that the language summit can agree to stopping bug fix > releases for 2.7 in 2014. I'd rather come up with a plan to seek dedicated resources from key corporate users to help with the boring parts of long term maintenance that are really hard to get volunteers excited about, but make a great avenue for corporate sponsored contributions without infringing on the upstream community's autonomy. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ben at bendarnell.com Sat Mar 22 23:53:29 2014 From: ben at bendarnell.com (Ben Darnell) Date: Sat, 22 Mar 2014 18:53:29 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: Message-ID: I agree wholeheartedly with the sentiment behind this PEP, but I have concerns about the implementation. If we introduce new APIs into the ssl module then we will see packages and applications that depend on Python 2.7.7+, just like with the introduction of bool in 2.2.1. This will be a mess unless adoption of new Python minor versions is swift and consistent. The PEP says that new features should be testable with feature detection, but how far does that go? The Python 3 NPN interface is a method of SSLContext, which didn't exist in 2.7. Do we backport the entire SSLContext interface in a minor release, or introduce a new NPN interface that is not based on SSLContext just for older Pythons? In my experience most deployments of Python apps have better systems in place to deal with packages installed from PyPI than new versions of Python itself. Therefore it might be better to follow the lead of the unittest2 module and introduce a new ssl2 module that tracks the most recent version of Python's ssl module. My counter-proposal is: * Introduce a new module 'ssl2' which is essentially a backport of the current release's ssl module for all old versions of Python that anyone cares to support. * Third-party applications and libraries targeting anything but the latest version of Python are advised to use ssl2 instead of ssl (this might also be a good opportunity to bless certifi or an equivalent). * Minor releases of Python are allowed to make any security-related change that does not change any public interfaces (e.g. linking against a new version of openssl, or disabling SSLv2) * New interfaces in security-related modules are allowed only if those new interfaces are used by other stdlib modules (since stdlib modules are the only place where ssl2 is off-limits). For example, a backport of match_hostname could go in if httplib or urllib used it. NPN/ALPN would not be allowed under this rule since there is no stdlib code that uses them; third-party SPDY or HTTP/2 code would have to use ssl2 to get this interface. In this world, libraries like requests or tornado that want to use advanced ssl features can simply declare a dependency on ssl2, instead of saying "we require Python 2.7.7+, 3.2.6+, etc". In spite of my reservations about this PEP, I'd still love to see SSL improvements in 2.7 in any form. I'm currently planning a port for Tornado's ssl usage from the stdlib ssl module to PyOpenSSL (in order to get SNI and NPN/ALPN features), and I'd really rather not have to convert to a completely incompatible interface. -Ben On Sat, Mar 22, 2014 at 5:11 PM, Nick Coghlan wrote: > Folks, > > I have just posted a proposal to change the way we treat enhancements > that relate to Python's support for network security enhancements. I > now see these as categorically different from most other enhancements, > as they have implications for the evolution of internet security as a > whole, rather than being limited to affecting the security of > individual applications written in Python. > > It is similar to the previous IDLE policy exception PEP, where we > decided that cross version consistency of IDLE superseded the general > policy against backporting enhancements to maintenance branches. > > This new PEP proposes a similar exception for network security related > changes, such that the *capabilities* of all still maintained versions > should be kept equivalent, with only the *default settings* varying > across up to date feature releases. The policy is also designed to > make it easier for us to ratchet up the default security settings in > future feature releases. > > However, it's more complex than the IDLE PEP (since it's more > invasive), and there are still a few open questions listed. This topic > involves a complex balance between encouraging and supporting good > security practices and limiting the risk of failures for users > upgrading to new maintenance releases, so I'd ask that folks take time > to read and consider the implications of the full PEP in the broader > context of today's internet before posting comments on specific > details, or indicating a preference one way or the other in terms of > the overall proposal. > > Full PEP included inline below, and available in more readable form at > http://www.python.org/dev/peps/pep-0466/ > > Regards, > Nick. > > PEP: 466 > Title: Network Security Enhancement Exception for All Branches > Version: $Revision$ > Last-Modified: $Date$ > Author: Nick Coghlan , > Status: Draft > Type: Informational > Content-Type: text/x-rst > Created: 23-Mar-2014 > Post-History: 23-Mar-2014 > > > Abstract > ======== > > Most CPython tracker issues are classified as errors in behaviour or > proposed enhancements. Most patches to fix behavioural errors are > applied to all active maintenance branches. Enhancement patches are > restricted to the default branch that becomes the next Python version. > > This PEP relaxes the latter restriction allowing enhancements to be applied > to maintenance branches for standard library components that have > implications for the overall security of the internet. In particular, the > exception will apply to: > > * the ``ssl`` module > * the ``hashlib`` module > * the ``hmac`` module > * the ``sha`` module (Python 2 only) > * the components of other networking modules that make use of these modules > * the components of the ``random`` and ``os`` modules that are relevant to > cryptographic applications > * the version of OpenSSL bundled with the binary installers > > Changes to these modules will still need to undergo normal backwards > compatibility assessments, but new features will be permitted where > appropriate, making it easier to implement secure networked software in > Python, even for software that needs to remain compatible with older > feature > releases of Python. > > > Exemption Policy > ================ > > Under this policy, the following network security related modules are > granted a blanket exemption to the restriction against adding new features > in maintenance releases: > > * the ``ssl`` module > * the ``hashlib`` module > * the ``hmac`` module > * the ``sha`` module (Python 2 only) > > This exemption applies to *all* proposals to backport backwards compatible > features in these modules to active maintenance branches. This choice is > made deliberately to ensure that the "feature or fix?" argument isn't > simply > replaced by a "security related or not?" argument. These particular modules > are inherently security related, and all enhancements to them improve > Python's capabilities as a platform for development of secure networked > software. > > As part of this policy, permission is also granted to upgrade to newer > feature releases of OpenSSL when preparing the binary installers > for new maintenance releases of CPython. > > In addition to the above blanket exemption, a conditional exemption is > granted for these modules that may include some network security related > features: > > * the ``os`` module (primarily ``os.urandom``) > * the ``random`` module > * networking related modules that depend on one or more of the network > security related modules listed above > > This more limited exemption for these modules requires that the *specific* > enhancement being proposed for backporting needs to be justified as being > network security related. If the enhancement under discussion is designed > to take advantage of a new feature in one of the network security related > modules, then that will be taken as implying that the enhancement is > security related. > > > Backwards Compatibility Considerations > ====================================== > > This PEP does *not* grant any general exemptions to the usual backwards > compatibility policy for maintenance releases. Instead, it is designed > to make it easier to provide more "secure by default" behaviour in future > feature releases, while still limiting the risk of breaking currently > working software when upgrading to a new maintenance release. > > In *all* cases where this policy is applied to backport enhancements to > maintenance releases, it MUST be possible to write cross-version compatible > code that operates by "feature detection" (for example, checking for > particular attributes in the module), without needing to explicitly check > the Python version. > > It is then up to library and framework code to provide an appropriate error > message or fallback behaviour if a desired feature is found to be missing. > > Affected APIs SHOULD be designed to allow library and application code to > perform the following actions after detecting the presence of a relevant > network security related feature: > > * explicitly opt in to more secure settings (to allow the use of enhanced > security features in older versions of Python) > * explicitly opt in to less secure settings (to allow the use of newer > Python > versions in lower security environments) > * determine the default setting for the feature (this MAY require explicit > Python version checks to determine the Python feature release, but MUST > NOT depend on the specific maintenance release) > > > Documentation Requirements > ========================== > > All modules that take advantage of this policy to backport network > security related enhancements to earlier Python versions MUST include > a "Security Considerations" section in their documentation. > > In addition to any other module specific contents, this section MUST > enumerate key security enhancements and fixes (with CVE identifiers where > applicable), and the > > > Evolution of this Policy > ======================== > > The key requirement for a module to be considered for inclusion in this > policy (whether under a blanket or conditional exemption) is that it must > have security implications *beyond* the specific application that is > written > in Python and the system that application is running on. Thus the focus on > network security protocols and related cryptographic infrastructure - > Python > is a popular choice for the development of web services and clients, and > thus the capabilities of widely used Python versions have implications for > the security design of other services that may be using newer versions of > Python or other development languages. > > The intent behind this requirement is to minimise any impact that the > introduction of this policy may have on the stability and compatibility of > maintenance releases. It would be thoroughly counterproductive if end > users became as cautious about updating to new Python maintenance releases > as they are about updating to new feature releases. > > > Motivation and Rationale > ======================== > > This PEP can be seen as a more targeted version of the "faster standard > library release cycle" proposals discussed in PEP 407 and PEP 413, > focusing specifically on those areas which have implications beyond the > Python community. > > The creation of this PEP was prompted primarily by the aging SSL support in > the Python 2 series. As of March 2014, the Python 2.7 SSL module is > approaching four years of age, and the SSL support in the still popular > Python 2.6 release had its feature set locked six years ago. > > These are simply too old to provide a foundation that can be recommended > in good conscience for secure networking software that operates over the > public internet, especially in an era where it is becoming quite clearly > evident that advanced persistent security threats are even more widespread > and more indiscriminate in their targeting than had previously been > understood. > > While the use of the system OpenSSL installation addresses many of these > concerns on Linux platforms, it doesn't address all of them, and in the > case of the binary installers for Windows and Mac OS X that are published > on python.org, the version of OpenSSL used is entirely within the control > of the Python core development team. > > With increased popularity comes increased responsibility, and this policy > is about recognising the fact that Python's popularity and adoption has now > reached a level where some of our design and policy decisions have > significant implications beyond the Python development community. > > As one example, the Python 2 ``ssl`` module does not support the Server > Name Identification standard. While it is possible to obtain SNI support > by using the third party ``requests`` client library, actually doing so > currently requires using not only ``requests`` and its embedded > dependencies, > but also half a dozen or more additional libraries. The lack of support > in the Python 2 series thus serves as an impediment to making effective > use of SNI on servers, as Python 2 clients will frequently fail to handle > it correctly. > > Another more critical example is the lack of SSL hostname matching in the > Python 2 standard library - it is currently necessary to rely on a third > party library, such as ``requests`` or ``backports.ssl_match_hostname`` to > obtain that functionality in Python 2. > > The Python 2 series also remains more vulnerable to remote timing attacks > on security sensitive comparisons than the Python 3 series, as it lacks a > standard library equivalent to the timing attack resistant > ``hmac.compare_digest()`` function. While appropriate secure comparison > functions can be implemented in third party extensions, may users don't > even consider the problem and use ordinary equality comparisons instead > - while a standard library solution doesn't automatically fix that problem, > it *does* make the barrier to resolution much lower once the problem is > pointed out. > > My position on the ongoing transition from Python 2 to Python 3 has long > been that Python 2 remains a supported platform for the core development > team, and that commercial support will remain available well after upstream > maintenance ends. However, in the absence of this network security > enhancement policy, that position is difficult to justify when it comes to > software that operates over the public internet. Just as many developers > consider it too difficult to develop truly secure modern networked software > in C/C++ (largely due to the challenges associated with manual > memory management), I consider it too difficult to develop truly secure > modern networked software using the Python 2 series without introducing > this > network security enhancement policy, as doing so would mean reimplementing > substantial portions of the standard library as third party modules to gain > access to the newer underlying network security protocols and mechanisms, > and then injecting those into the module namespace to override their > standard library counterparts. > > Requiring that latent defects in an application's Unicode correctness be > addressed in order to migrate to Python 3 is not a reasonable alternative > recommendation, especially given the likely existence of legacy code that > lacks the kind of automated regression test suite needed to help support > a migration from Python 2 to Python 3. The key point of this PEP is that > those situations affect more people than just the developers and users of > the affected application: their existence becomes something that developers > of secure networked services need to take into account as part of their > security design. By making it more feasible to enhance the security of the > Python 2 series, we can help contribute to the evolution of a more secure > internet for all concerned. > > For the Python 2 series, this proposal will most obviously impact the > remaining maintenance releases of Python 2.7. However, commercial > redistributors that continue to offer full support for Python 2.6, and will > also do so for Python 2.7 after upstream maintenance ends, may choose > to take this policy into account when deciding what changes to backport to > their own maintenance updates. Providing commercial redistributors such > as ActiveState, Attachmate, Canonical, Continuum Analytics, Enthought, and > Red Hat that option is one of the benefits offered by requiring feature > detection based forward compatibility for the modules covered by this > policy. > > > Open Questions > ============== > > * What are the risks associated with allowing OpenSSL to be updated to > new feature versions in the Windows and Mac OS X binary installers for > maintenance releases? > > * Are there any other security relevant modules that should be covered > by either a blanket or conditional exemption? > > * Should we enumerate a specific list of "other networking modules" rather > than leaving it implicit? > > > Acknowledgement > =============== > > Thanks to Christian Heimes for his recent efforts on greatly improving > Python's SSL support in the Python 3 series, and a variety of members of > the Python community for helping me to better understand the implications > of the default settings we provide in our SSL modules, and the impact that > tolerating the use of SSL infrastructure that was defined in 2010 > (Python 2.7) or even 2008 (Python 2.6) potentially has for the security > of the web as a whole. > > Christian and Donald Stufft also provided valuable feedback on a > preliminary > draft of this proposal. > > > Copyright > ========= > > This document has been placed in the public domain. > > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ben%40bendarnell.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at python.org Sat Mar 22 23:54:37 2014 From: thomas at python.org (Thomas Wouters) Date: Sat, 22 Mar 2014 23:54:37 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <532E0E22.6080304@v.loewis.de> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> Message-ID: On Sat, Mar 22, 2014 at 11:26 PM, "Martin v. L?wis" wrote: > Am 22.03.14 23:21, schrieb Donald Stufft: > > "Just use Python 3.4" ignores the reality of production software. I wish > it were that simple because I love 3.4 > > But I think so does the PEP (i.e. ignore the reality of production > software). The risk of breaking existing installations (which might > not be affected by the purported security threat being fixed) is just > too high. > > Users that feel threatened by the lack of SSL features in Python should > have long started planning on how to get to Python 3. It's not that they > haven't been warned. Or not rely on the standard library for their security. Much as I realize it is necessary for rudimentary SSL support (for example) to exist in the standard library, any security-aware user is much better served by third-party packages they can update more easily than Python. I really wish we had no SSL *server* support in the standard library at all (but then I wish we didn't have so many middling-quality servers in the stdlib to start with :) Even when we're not considering virtualenvs, which are a great tool for preventing upgrades from breaking things, separately-installed packages (be they third-party or "official" python.org packages) are much easier to upgrade than whole Python installations. And a third-party package like PyOpenSSL has much more freedom in dealing with backward-incompatible changes (because users updating PyOpenSSL are much more likely to realize something regarding SSL could break, even if they aren't more likely to read the release documentation.) I think I'm +0 on the PEP: I understand Martin's concerns, and share them (heck, I'm preparing for an update from Python 2.7.3 to 2.7.6 at work and I'm already having nightmares) but I also understand the problem the PEP tries to solve. I wish we would separate out e.g. the SSL-related modules so they aren't part of the 'core' Python but not quite third-party packages either, but that's not something that's easily done. I do believe that besides accepting this PEP we should provide good guidance on how to write portable, secure software using the tools provided, say in a new chapter in the library docs -- and it shouldn't be afraid to suggest third-party modules if they are a much better solution :) -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sun Mar 23 00:02:36 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 22 Mar 2014 16:02:36 -0700 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <532E1175.5090902@v.loewis.de> References: <532E0BA8.5070709@v.loewis.de> <532E1175.5090902@v.loewis.de> Message-ID: <1395529356.22828.97686905.3239BCC3@webmail.messagingengine.com> On Sat, Mar 22, 2014, at 15:40, Martin v. L?wis wrote: > Am 22.03.14 23:33, schrieb Nick Coghlan: > > Hard to maintain legacy software is a fact of life, and way too much > > of it is exposed to the public internet. This PEP is about doing what > > we can to mitigate the damage caused both by other people's mistakes, > > and also the inherent challenges of migrating from the error prone > > POSIX text model to something more reasonable. > > > > I *don't* think its reasonable to expect us to do this without support > > from the corporate users that caused the problem in the first place > > (by continuing to deploy older versions of Python without investing > > adequately in their upkeep), so I'd encourage everyone employed by a > > commercial user of Python to remind their management chains of the > > risks of failing to invest development time in any upstream > > dependencies that they expect to keep pace with the dynamic nature of > > the internet. > > I hope indeed you are successful in activating resources. However, > putting them on this backporting project seems like a waste. They > should rather go into porting stuff to 3.x where people need it. > > As responsible maintainers, we should just advise our users that > Python 2.7 is a dead horse, and that they should stop riding it. > More professionally, we should set an official end-of-life date > for 2.7 (alas, we should have done that two years ago). > > I hope that the language summit can agree to stopping bug fix > releases for 2.7 in 2014. As (I believe) previously discussed and documented in PEP 373, this date currently will be May 2015. From donald at stufft.io Sun Mar 23 00:07:51 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 22 Mar 2014 19:07:51 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: Message-ID: <2C00F7E7-D1AD-47AD-8FCE-E3A72F25F7CF@stufft.io> As someone who is deeply biased towards improving the packaging tool chain and getting people to use it I think that most people will simply use the Stdlib even if a more secure alternative exists. Infact one does exist and I still see almost everyone using the stdlib ssl instead of pyopenssl. At best they have an optional dependency on it which many people who aren't security conscious won't even realize why they should install it. > On Mar 22, 2014, at 6:53 PM, Ben Darnell wrote: > > I agree wholeheartedly with the sentiment behind this PEP, but I have concerns about the implementation. If we introduce new APIs into the ssl module then we will see packages and applications that depend on Python 2.7.7+, just like with the introduction of bool in 2.2.1. This will be a mess unless adoption of new Python minor versions is swift and consistent. The PEP says that new features should be testable with feature detection, but how far does that go? The Python 3 NPN interface is a method of SSLContext, which didn't exist in 2.7. Do we backport the entire SSLContext interface in a minor release, or introduce a new NPN interface that is not based on SSLContext just for older Pythons? > > In my experience most deployments of Python apps have better systems in place to deal with packages installed from PyPI than new versions of Python itself. Therefore it might be better to follow the lead of the unittest2 module and introduce a new ssl2 module that tracks the most recent version of Python's ssl module. > > My counter-proposal is: > * Introduce a new module 'ssl2' which is essentially a backport of the current release's ssl module for all old versions of Python that anyone cares to support. > * Third-party applications and libraries targeting anything but the latest version of Python are advised to use ssl2 instead of ssl (this might also be a good opportunity to bless certifi or an equivalent). > * Minor releases of Python are allowed to make any security-related change that does not change any public interfaces (e.g. linking against a new version of openssl, or disabling SSLv2) > * New interfaces in security-related modules are allowed only if those new interfaces are used by other stdlib modules (since stdlib modules are the only place where ssl2 is off-limits). For example, a backport of match_hostname could go in if httplib or urllib used it. NPN/ALPN would not be allowed under this rule since there is no stdlib code that uses them; third-party SPDY or HTTP/2 code would have to use ssl2 to get this interface. > > In this world, libraries like requests or tornado that want to use advanced ssl features can simply declare a dependency on ssl2, instead of saying "we require Python 2.7.7+, 3.2.6+, etc". > > In spite of my reservations about this PEP, I'd still love to see SSL improvements in 2.7 in any form. I'm currently planning a port for Tornado's ssl usage from the stdlib ssl module to PyOpenSSL (in order to get SNI and NPN/ALPN features), and I'd really rather not have to convert to a completely incompatible interface. > > -Ben > > > > >> On Sat, Mar 22, 2014 at 5:11 PM, Nick Coghlan wrote: >> Folks, >> >> I have just posted a proposal to change the way we treat enhancements >> that relate to Python's support for network security enhancements. I >> now see these as categorically different from most other enhancements, >> as they have implications for the evolution of internet security as a >> whole, rather than being limited to affecting the security of >> individual applications written in Python. >> >> It is similar to the previous IDLE policy exception PEP, where we >> decided that cross version consistency of IDLE superseded the general >> policy against backporting enhancements to maintenance branches. >> >> This new PEP proposes a similar exception for network security related >> changes, such that the *capabilities* of all still maintained versions >> should be kept equivalent, with only the *default settings* varying >> across up to date feature releases. The policy is also designed to >> make it easier for us to ratchet up the default security settings in >> future feature releases. >> >> However, it's more complex than the IDLE PEP (since it's more >> invasive), and there are still a few open questions listed. This topic >> involves a complex balance between encouraging and supporting good >> security practices and limiting the risk of failures for users >> upgrading to new maintenance releases, so I'd ask that folks take time >> to read and consider the implications of the full PEP in the broader >> context of today's internet before posting comments on specific >> details, or indicating a preference one way or the other in terms of >> the overall proposal. >> >> Full PEP included inline below, and available in more readable form at >> http://www.python.org/dev/peps/pep-0466/ >> >> Regards, >> Nick. >> >> PEP: 466 >> Title: Network Security Enhancement Exception for All Branches >> Version: $Revision$ >> Last-Modified: $Date$ >> Author: Nick Coghlan , >> Status: Draft >> Type: Informational >> Content-Type: text/x-rst >> Created: 23-Mar-2014 >> Post-History: 23-Mar-2014 >> >> >> Abstract >> ======== >> >> Most CPython tracker issues are classified as errors in behaviour or >> proposed enhancements. Most patches to fix behavioural errors are >> applied to all active maintenance branches. Enhancement patches are >> restricted to the default branch that becomes the next Python version. >> >> This PEP relaxes the latter restriction allowing enhancements to be applied >> to maintenance branches for standard library components that have >> implications for the overall security of the internet. In particular, the >> exception will apply to: >> >> * the ``ssl`` module >> * the ``hashlib`` module >> * the ``hmac`` module >> * the ``sha`` module (Python 2 only) >> * the components of other networking modules that make use of these modules >> * the components of the ``random`` and ``os`` modules that are relevant to >> cryptographic applications >> * the version of OpenSSL bundled with the binary installers >> >> Changes to these modules will still need to undergo normal backwards >> compatibility assessments, but new features will be permitted where >> appropriate, making it easier to implement secure networked software in >> Python, even for software that needs to remain compatible with older feature >> releases of Python. >> >> >> Exemption Policy >> ================ >> >> Under this policy, the following network security related modules are >> granted a blanket exemption to the restriction against adding new features >> in maintenance releases: >> >> * the ``ssl`` module >> * the ``hashlib`` module >> * the ``hmac`` module >> * the ``sha`` module (Python 2 only) >> >> This exemption applies to *all* proposals to backport backwards compatible >> features in these modules to active maintenance branches. This choice is >> made deliberately to ensure that the "feature or fix?" argument isn't simply >> replaced by a "security related or not?" argument. These particular modules >> are inherently security related, and all enhancements to them improve >> Python's capabilities as a platform for development of secure networked >> software. >> >> As part of this policy, permission is also granted to upgrade to newer >> feature releases of OpenSSL when preparing the binary installers >> for new maintenance releases of CPython. >> >> In addition to the above blanket exemption, a conditional exemption is >> granted for these modules that may include some network security related >> features: >> >> * the ``os`` module (primarily ``os.urandom``) >> * the ``random`` module >> * networking related modules that depend on one or more of the network >> security related modules listed above >> >> This more limited exemption for these modules requires that the *specific* >> enhancement being proposed for backporting needs to be justified as being >> network security related. If the enhancement under discussion is designed >> to take advantage of a new feature in one of the network security related >> modules, then that will be taken as implying that the enhancement is >> security related. >> >> >> Backwards Compatibility Considerations >> ====================================== >> >> This PEP does *not* grant any general exemptions to the usual backwards >> compatibility policy for maintenance releases. Instead, it is designed >> to make it easier to provide more "secure by default" behaviour in future >> feature releases, while still limiting the risk of breaking currently >> working software when upgrading to a new maintenance release. >> >> In *all* cases where this policy is applied to backport enhancements to >> maintenance releases, it MUST be possible to write cross-version compatible >> code that operates by "feature detection" (for example, checking for >> particular attributes in the module), without needing to explicitly check >> the Python version. >> >> It is then up to library and framework code to provide an appropriate error >> message or fallback behaviour if a desired feature is found to be missing. >> >> Affected APIs SHOULD be designed to allow library and application code to >> perform the following actions after detecting the presence of a relevant >> network security related feature: >> >> * explicitly opt in to more secure settings (to allow the use of enhanced >> security features in older versions of Python) >> * explicitly opt in to less secure settings (to allow the use of newer Python >> versions in lower security environments) >> * determine the default setting for the feature (this MAY require explicit >> Python version checks to determine the Python feature release, but MUST >> NOT depend on the specific maintenance release) >> >> >> Documentation Requirements >> ========================== >> >> All modules that take advantage of this policy to backport network >> security related enhancements to earlier Python versions MUST include >> a "Security Considerations" section in their documentation. >> >> In addition to any other module specific contents, this section MUST >> enumerate key security enhancements and fixes (with CVE identifiers where >> applicable), and the >> >> >> Evolution of this Policy >> ======================== >> >> The key requirement for a module to be considered for inclusion in this >> policy (whether under a blanket or conditional exemption) is that it must >> have security implications *beyond* the specific application that is written >> in Python and the system that application is running on. Thus the focus on >> network security protocols and related cryptographic infrastructure - Python >> is a popular choice for the development of web services and clients, and >> thus the capabilities of widely used Python versions have implications for >> the security design of other services that may be using newer versions of >> Python or other development languages. >> >> The intent behind this requirement is to minimise any impact that the >> introduction of this policy may have on the stability and compatibility of >> maintenance releases. It would be thoroughly counterproductive if end >> users became as cautious about updating to new Python maintenance releases >> as they are about updating to new feature releases. >> >> >> Motivation and Rationale >> ======================== >> >> This PEP can be seen as a more targeted version of the "faster standard >> library release cycle" proposals discussed in PEP 407 and PEP 413, >> focusing specifically on those areas which have implications beyond the >> Python community. >> >> The creation of this PEP was prompted primarily by the aging SSL support in >> the Python 2 series. As of March 2014, the Python 2.7 SSL module is >> approaching four years of age, and the SSL support in the still popular >> Python 2.6 release had its feature set locked six years ago. >> >> These are simply too old to provide a foundation that can be recommended >> in good conscience for secure networking software that operates over the >> public internet, especially in an era where it is becoming quite clearly >> evident that advanced persistent security threats are even more widespread >> and more indiscriminate in their targeting than had previously been >> understood. >> >> While the use of the system OpenSSL installation addresses many of these >> concerns on Linux platforms, it doesn't address all of them, and in the >> case of the binary installers for Windows and Mac OS X that are published >> on python.org, the version of OpenSSL used is entirely within the control >> of the Python core development team. >> >> With increased popularity comes increased responsibility, and this policy >> is about recognising the fact that Python's popularity and adoption has now >> reached a level where some of our design and policy decisions have >> significant implications beyond the Python development community. >> >> As one example, the Python 2 ``ssl`` module does not support the Server >> Name Identification standard. While it is possible to obtain SNI support >> by using the third party ``requests`` client library, actually doing so >> currently requires using not only ``requests`` and its embedded dependencies, >> but also half a dozen or more additional libraries. The lack of support >> in the Python 2 series thus serves as an impediment to making effective >> use of SNI on servers, as Python 2 clients will frequently fail to handle >> it correctly. >> >> Another more critical example is the lack of SSL hostname matching in the >> Python 2 standard library - it is currently necessary to rely on a third >> party library, such as ``requests`` or ``backports.ssl_match_hostname`` to >> obtain that functionality in Python 2. >> >> The Python 2 series also remains more vulnerable to remote timing attacks >> on security sensitive comparisons than the Python 3 series, as it lacks a >> standard library equivalent to the timing attack resistant >> ``hmac.compare_digest()`` function. While appropriate secure comparison >> functions can be implemented in third party extensions, may users don't >> even consider the problem and use ordinary equality comparisons instead >> - while a standard library solution doesn't automatically fix that problem, >> it *does* make the barrier to resolution much lower once the problem is >> pointed out. >> >> My position on the ongoing transition from Python 2 to Python 3 has long >> been that Python 2 remains a supported platform for the core development >> team, and that commercial support will remain available well after upstream >> maintenance ends. However, in the absence of this network security >> enhancement policy, that position is difficult to justify when it comes to >> software that operates over the public internet. Just as many developers >> consider it too difficult to develop truly secure modern networked software >> in C/C++ (largely due to the challenges associated with manual >> memory management), I consider it too difficult to develop truly secure >> modern networked software using the Python 2 series without introducing this >> network security enhancement policy, as doing so would mean reimplementing >> substantial portions of the standard library as third party modules to gain >> access to the newer underlying network security protocols and mechanisms, >> and then injecting those into the module namespace to override their >> standard library counterparts. >> >> Requiring that latent defects in an application's Unicode correctness be >> addressed in order to migrate to Python 3 is not a reasonable alternative >> recommendation, especially given the likely existence of legacy code that >> lacks the kind of automated regression test suite needed to help support >> a migration from Python 2 to Python 3. The key point of this PEP is that >> those situations affect more people than just the developers and users of >> the affected application: their existence becomes something that developers >> of secure networked services need to take into account as part of their >> security design. By making it more feasible to enhance the security of the >> Python 2 series, we can help contribute to the evolution of a more secure >> internet for all concerned. >> >> For the Python 2 series, this proposal will most obviously impact the >> remaining maintenance releases of Python 2.7. However, commercial >> redistributors that continue to offer full support for Python 2.6, and will >> also do so for Python 2.7 after upstream maintenance ends, may choose >> to take this policy into account when deciding what changes to backport to >> their own maintenance updates. Providing commercial redistributors such >> as ActiveState, Attachmate, Canonical, Continuum Analytics, Enthought, and >> Red Hat that option is one of the benefits offered by requiring feature >> detection based forward compatibility for the modules covered by this policy. >> >> >> Open Questions >> ============== >> >> * What are the risks associated with allowing OpenSSL to be updated to >> new feature versions in the Windows and Mac OS X binary installers for >> maintenance releases? >> >> * Are there any other security relevant modules that should be covered >> by either a blanket or conditional exemption? >> >> * Should we enumerate a specific list of "other networking modules" rather >> than leaving it implicit? >> >> >> Acknowledgement >> =============== >> >> Thanks to Christian Heimes for his recent efforts on greatly improving >> Python's SSL support in the Python 3 series, and a variety of members of >> the Python community for helping me to better understand the implications >> of the default settings we provide in our SSL modules, and the impact that >> tolerating the use of SSL infrastructure that was defined in 2010 >> (Python 2.7) or even 2008 (Python 2.6) potentially has for the security >> of the web as a whole. >> >> Christian and Donald Stufft also provided valuable feedback on a preliminary >> draft of this proposal. >> >> >> Copyright >> ========= >> >> This document has been placed in the public domain. >> >> >> >> .. >> Local Variables: >> mode: indented-text >> indent-tabs-mode: nil >> sentence-end-double-space: t >> fill-column: 70 >> coding: utf-8 >> End: >> >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ben%40bendarnell.com > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Mar 23 00:08:29 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Mar 2014 09:08:29 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: Message-ID: On 23 March 2014 08:53, Ben Darnell wrote: > I agree wholeheartedly with the sentiment behind this PEP, but I have > concerns about the implementation. If we introduce new APIs into the ssl > module then we will see packages and applications that depend on Python > 2.7.7+, just like with the introduction of bool in 2.2.1. This will be a > mess unless adoption of new Python minor versions is swift and consistent. > The PEP says that new features should be testable with feature detection, > but how far does that go? The Python 3 NPN interface is a method of > SSLContext, which didn't exist in 2.7. Do we backport the entire SSLContext > interface in a minor release, or introduce a new NPN interface that is not > based on SSLContext just for older Pythons? The whole thing. The problem at the moment is that too many applications just silently run without proper security settings on older Python release, as users don't completely want to drop support for that version, but also decide it's too hard to enable the right security settings properly. The idea here is to make it easy for them to enable the settings if they're available, but still fall back to their old behaviour if they're not. The "fail or fall back" decision will depend on the library and its assessment of security compatibility vs breadth of support for legacy versions. The issue isn't really the ssl module itself - it's the other modules that *depend* on the ssl module (like httplib, urllib, poplib, ftplib, imaplib). You could technically try to monkeypatch or shadow the stdlib ssl module from a third party library, but that's a *really* dangerous approach to security (since it makes it hard to do proper integration testing). > In my experience most deployments of Python apps have better systems in > place to deal with packages installed from PyPI than new versions of Python > itself. Therefore it might be better to follow the lead of the unittest2 > module and introduce a new ssl2 module that tracks the most recent version > of Python's ssl module. > > My counter-proposal is: > * Introduce a new module 'ssl2' which is essentially a backport of the > current release's ssl module for all old versions of Python that anyone > cares to support. It's not just ssl2: you needs os2, random2, ftplib2, poplib2, imaplib2, smtplib2. You also run headlong into the problem of a complex external dependency on OpenSSL (which the Linux distro repackagers currently handle for us). This is what the """reimplementing substantial portions of the standard library as third party modules to gain access to the newer underlying network security protocols and mechanisms, and then injecting those into the module namespace to override their standard library counterparts.""" comment is about, but I guess I should expand on that for the benefit of readers not as familiar with the relevant implementation details. > * Third-party applications and libraries targeting anything but the latest > version of Python are advised to use ssl2 instead of ssl (this might also be > a good opportunity to bless certifi or an equivalent). > * Minor releases of Python are allowed to make any security-related change > that does not change any public interfaces (e.g. linking against a new > version of openssl, or disabling SSLv2) New versions of OpenSSL are nowhere near as useful when their constants aren't made available in the Python API. > * New interfaces in security-related modules are allowed only if those new > interfaces are used by other stdlib modules (since stdlib modules are the > only place where ssl2 is off-limits). For example, a backport of > match_hostname could go in if httplib or urllib used it. NPN/ALPN would not > be allowed under this rule since there is no stdlib code that uses them; > third-party SPDY or HTTP/2 code would have to use ssl2 to get this > interface. > > In this world, libraries like requests or tornado that want to use advanced > ssl features can simply declare a dependency on ssl2, instead of saying "we > require Python 2.7.7+, 3.2.6+, etc". There's a reason the developers of those libraries haven't already done that: it's genuinely hard. It's not that I didn't think of this approach, it's that I think it would actually be more difficult and less effective than the approach currently in the PEP. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From martin at v.loewis.de Sun Mar 23 00:08:48 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 23 Mar 2014 00:08:48 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: Message-ID: <532E1800.3060205@v.loewis.de> Am 22.03.14 23:39, schrieb Ned Deily: > Regarding the python.org binary installers, I think past practice has > been for us to update third-party libraries as necessary in maintenance > releases when there is good cause and with the concurrence of the > release manager, so I don't see this as a big issue. For the OS X > binary installer, the issue for OpenSSL has been that we dynamically > link to the system-supplied OpenSSL libraries and that, for various > reasons, Apple has deprecated (and frozen at non-current OpenSSL > releases) the use of those libraries in favor of their own security > frameworks. So, for multiple reasons, including the risk that OpenSSL > may be dropped from an upcoming major release of OS X, we need to start > supplying our own version with all OS X binary installers. That's the > plan regardless of the outcome of this PEP. The Windows case is similar but different. I also update OpenSSL releases between Python bug fix releases, but only to OpenSSL bug fix releases (likewise for all other third party software). So 2.7 uses 0.9.8 (currently 0.9.8y), 3.3 and 3.4 use 1.0.1 (currently 1.0.1e). I still wish it was possible to drop OpenSSL altogether on both OSX and Windows, and use the vendor TLS libraries instead. This would reduce the maintenance burden, and might even improve performance. However, it's getting difficult since the ssl module exposes more and more OpenSSL API as-is (rather than trying to provide an abstraction). For Windows, this approach was unfortunately hopeless as long as we had to support XP. Now that Vista becomes the baseline, I hope I can try to provide much of the ssl module natively using CryptoAPI. Regards, Martin From martin at v.loewis.de Sun Mar 23 00:15:35 2014 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 23 Mar 2014 00:15:35 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <1395529356.22828.97686905.3239BCC3@webmail.messagingengine.com> References: <532E0BA8.5070709@v.loewis.de> <532E1175.5090902@v.loewis.de> <1395529356.22828.97686905.3239BCC3@webmail.messagingengine.com> Message-ID: <532E1997.2010602@v.loewis.de> Am 23.03.14 00:02, schrieb Benjamin Peterson: > As (I believe) previously discussed and documented in PEP 373, this date > currently will be May 2015. Ah! Thanks for reminding me. I think PEP 466 then needs to take a position on that, as well. By the past schedule, this probably means two more bug fix releases for 2.7 (which is fine with me). However, I expect this to be a problem for the PEP, since it is unrealistic to expect that all aspects of the desired features are complete by 2015 (i.e. with no need for further updates in the coming years). Regards, Martin From p.f.moore at gmail.com Sun Mar 23 00:23:02 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 22 Mar 2014 23:23:02 +0000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <2C00F7E7-D1AD-47AD-8FCE-E3A72F25F7CF@stufft.io> References: <2C00F7E7-D1AD-47AD-8FCE-E3A72F25F7CF@stufft.io> Message-ID: On 22 March 2014 23:07, Donald Stufft wrote: > As someone who is deeply biased towards improving the packaging tool chain > and getting people to use it I think that most people will simply use the > Stdlib even if a more secure alternative exists. Infact one does exist and I > still see almost everyone using the stdlib ssl instead of pyopenssl. At best > they have an optional dependency on it which many people who aren't security > conscious won't even realize why they should install it. Windows users typically will not be able to use something like pyopenssl. It's a complex binary dependency with no wheel on PyPI. There are no easily locatable wininst installers, even - and those are messy to use in a virtualenv. While the stdlib modules may have issues, "depend on pyopenssl" is not a practical solution for many people. Paul From ncoghlan at gmail.com Sun Mar 23 00:24:59 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Mar 2014 09:24:59 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <1395523976.4237.97669609.00EDB3FA@webmail.messagingengine.com> References: <1395523976.4237.97669609.00EDB3FA@webmail.messagingengine.com> Message-ID: On 23 March 2014 07:32, Benjamin Peterson wrote: > On Sat, Mar 22, 2014, at 14:11, Nick Coghlan wrote: >> Folks, >> >> I have just posted a proposal to change the way we treat enhancements >> that relate to Python's support for network security enhancements. > > I think the PEP should also address "security-mode" releases. Do the > same exceptions apply? > > Does anyone really want to backport features to Python 3.1? As I see it, RHEL is the primary culprit for people still caring about Python 2.6. With the RHEL7 beta published late last year and hence CentOS 7 and RHEL 7 itself presumably landing well before Python 2.7 enters security fix only mode, I think it's OK to leave this aspect in the hands of the distro vendors and other commercial redistributors. We create the problem, we can bear the cost of dealing with it on behalf of our users. In Red Hat's specific case, we're also finally starting to offer users better solutions that make it easier to switch to a newer language runtime without having to upgrade the entire underlying OS or run in an unsupported configuration. In the longer term, this will hopefully mean that the expectation for upstream projects to maintain compatibility with the system Python in RHEL will become limited to just those projects that we actually include as part of the distro. However, I agree the PEP needs to have a dedicated section discussing security fix only releases. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Mar 23 00:20:41 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Mar 2014 09:20:41 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <2C00F7E7-D1AD-47AD-8FCE-E3A72F25F7CF@stufft.io> References: <2C00F7E7-D1AD-47AD-8FCE-E3A72F25F7CF@stufft.io> Message-ID: On 23 March 2014 09:07, Donald Stufft wrote: > As someone who is deeply biased towards improving the packaging tool chain > and getting people to use it I think that most people will simply use the > Stdlib even if a more secure alternative exists. Infact one does exist and I > still see almost everyone using the stdlib ssl instead of pyopenssl. At best > they have an optional dependency on it which many people who aren't security > conscious won't even realize why they should install it. I should probably mention explicitly in the PEP that security related packages almost always involve somewhat tricky binary dependencies, and while we're working on it, our packaging ecosystem will still sometimes deliver a poor user experience on that front. There's also an inherent divide between "using an old version of Python" and "willing to use new dependencies from pip". They're not disjoint sets (plenty of folks use the CentOS system Python + pip), but I expect there's a large enough population of conservative corporate users on older versions that won't use a pip based solution even if its available, thus not really solving the problem. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sun Mar 23 00:33:07 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 Mar 2014 00:33:07 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements References: <2C00F7E7-D1AD-47AD-8FCE-E3A72F25F7CF@stufft.io> Message-ID: <20140323003307.3472575c@fsol> On Sat, 22 Mar 2014 19:07:51 -0400 Donald Stufft wrote: > As someone who is deeply biased towards improving the packaging tool chain and getting people to use it I think that most people will simply use the Stdlib even if a more secure alternative exists. Infact one does exist and I still see almost everyone using the stdlib ssl instead of pyopenssl. pyopenssl exposes a different API from ssl, though. It's not just a matter of flipping an import statement. Regards Antoine. From solipsis at pitrou.net Sun Mar 23 00:34:37 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 Mar 2014 00:34:37 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements References: Message-ID: <20140323003437.2810072c@fsol> On Sun, 23 Mar 2014 09:08:29 +1000 Nick Coghlan wrote: > On 23 March 2014 08:53, Ben Darnell wrote: > > I agree wholeheartedly with the sentiment behind this PEP, but I have > > concerns about the implementation. If we introduce new APIs into the ssl > > module then we will see packages and applications that depend on Python > > 2.7.7+, just like with the introduction of bool in 2.2.1. This will be a > > mess unless adoption of new Python minor versions is swift and consistent. > > The PEP says that new features should be testable with feature detection, > > but how far does that go? The Python 3 NPN interface is a method of > > SSLContext, which didn't exist in 2.7. Do we backport the entire SSLContext > > interface in a minor release, or introduce a new NPN interface that is not > > based on SSLContext just for older Pythons? > > The whole thing. Have you first tried to do it and appreciate the amount of disruption it would cause? > It's not just ssl2: you needs os2, random2, ftplib2, poplib2, > imaplib2, smtplib2. You also run headlong into the problem of a > complex external dependency on OpenSSL (which the Linux distro > repackagers currently handle for us). So are you proposing to backport changes to all these modules too? This sounds crazy. Regards Antoine. From benjamin at python.org Sun Mar 23 00:36:59 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 22 Mar 2014 16:36:59 -0700 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <20140323003437.2810072c@fsol> References: <20140323003437.2810072c@fsol> Message-ID: <1395531419.28996.97692325.264C439C@webmail.messagingengine.com> On Sat, Mar 22, 2014, at 16:34, Antoine Pitrou wrote: > On Sun, 23 Mar 2014 09:08:29 +1000 > Nick Coghlan wrote: > > On 23 March 2014 08:53, Ben Darnell wrote: > > > I agree wholeheartedly with the sentiment behind this PEP, but I have > > > concerns about the implementation. If we introduce new APIs into the ssl > > > module then we will see packages and applications that depend on Python > > > 2.7.7+, just like with the introduction of bool in 2.2.1. This will be a > > > mess unless adoption of new Python minor versions is swift and consistent. > > > The PEP says that new features should be testable with feature detection, > > > but how far does that go? The Python 3 NPN interface is a method of > > > SSLContext, which didn't exist in 2.7. Do we backport the entire SSLContext > > > interface in a minor release, or introduce a new NPN interface that is not > > > based on SSLContext just for older Pythons? > > > > The whole thing. > > Have you first tried to do it and appreciate the amount of disruption > it would cause? > > > It's not just ssl2: you needs os2, random2, ftplib2, poplib2, > > imaplib2, smtplib2. You also run headlong into the problem of a > > complex external dependency on OpenSSL (which the Linux distro > > repackagers currently handle for us). > > So are you proposing to backport changes to all these modules too? This > sounds crazy. If one reads the PEP, one will notice it indeed proposes a "conditional" exception for "networking related modules that depend on one or more of the network security related modules [ssl, hashlib,...]". From ben at bendarnell.com Sun Mar 23 00:42:02 2014 From: ben at bendarnell.com (Ben Darnell) Date: Sat, 22 Mar 2014 19:42:02 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: Message-ID: On Sat, Mar 22, 2014 at 7:08 PM, Nick Coghlan wrote: > > The issue isn't really the ssl module itself - it's the other modules > that *depend* on the ssl module (like httplib, urllib, poplib, ftplib, > imaplib). You could technically try to monkeypatch or shadow the > stdlib ssl module from a third party library, but that's a *really* > dangerous approach to security (since it makes it hard to do proper > integration testing). > I suppose the difference is that the applications I've worked with have *already* abandoned stdlib modules like httplib or urllib in favor of libraries like requests and tornado, so I see this as mainly a question of whether the stdlib provides a suitable foundation for those packages. The argument about the difficulty of integration testing cuts both ways - I can use tox to test Tornado with several versions of a third-party package, but cannot (as long as https://launchpad.net/~fkrull/+archive/deadsnakesonly provides one package per x.y release) easily test with both Python 2.7.6 and 2.7.7. > > > In my experience most deployments of Python apps have better systems in > > place to deal with packages installed from PyPI than new versions of > Python > > itself. Therefore it might be better to follow the lead of the unittest2 > > module and introduce a new ssl2 module that tracks the most recent > version > > of Python's ssl module. > > > > My counter-proposal is: > > * Introduce a new module 'ssl2' which is essentially a backport of the > > current release's ssl module for all old versions of Python that anyone > > cares to support. > > It's not just ssl2: you needs os2, random2, ftplib2, poplib2, > imaplib2, smtplib2. You also run headlong into the problem of a > complex external dependency on OpenSSL (which the Linux distro > repackagers currently handle for us). > Just os2 and random2; as I said a few bullet points down the stdlib libraries that use ssl would be sufficient justification for additions to the ssl module's interface. > > This is what the """reimplementing substantial portions of the > standard library as third party modules to gain access to the newer > underlying network security protocols and mechanisms, and then > injecting those into the module namespace to override their standard > library counterparts.""" comment is about, but I guess I should expand > on that for the benefit of readers not as familiar with the relevant > implementation details. > In my proposal there is no monkey-patching; the newer ssl interfaces would come solely from the ssl2 module. > > > * Third-party applications and libraries targeting anything but the > latest > > version of Python are advised to use ssl2 instead of ssl (this might > also be > > a good opportunity to bless certifi or an equivalent). > > * Minor releases of Python are allowed to make any security-related > change > > that does not change any public interfaces (e.g. linking against a new > > version of openssl, or disabling SSLv2) > > New versions of OpenSSL are nowhere near as useful when their > constants aren't made available in the Python API. > > > * New interfaces in security-related modules are allowed only if those > new > > interfaces are used by other stdlib modules (since stdlib modules are the > > only place where ssl2 is off-limits). For example, a backport of > > match_hostname could go in if httplib or urllib used it. NPN/ALPN would > not > > be allowed under this rule since there is no stdlib code that uses them; > > third-party SPDY or HTTP/2 code would have to use ssl2 to get this > > interface. > > > > In this world, libraries like requests or tornado that want to use > advanced > > ssl features can simply declare a dependency on ssl2, instead of saying > "we > > require Python 2.7.7+, 3.2.6+, etc". > > There's a reason the developers of those libraries haven't already > done that: it's genuinely hard. It's not that I didn't think of this > approach, it's that I think it would actually be more difficult and > less effective than the approach currently in the PEP. > One of the reasons it's genuinely hard is that no one has provided an interface compatible with the stdlib ssl module. An ssl2 module that backported Python 3.4's ssl module would be a *much* more attractive porting target than e.g. PyOpenSSL. -Ben > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Mar 23 00:49:30 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 22 Mar 2014 19:49:30 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: Message-ID: <5063D16D-0E47-451F-B5D8-5EE27A677ACC@stufft.io> In the case of requests they already have an optional dependency on pyopenssl. It's just many people either don't know they should use it, are unable to use it, or unwilling to use the python packaging tool chain because of its current flaws. > On Mar 22, 2014, at 7:42 PM, Ben Darnell wrote: > >> On Sat, Mar 22, 2014 at 7:08 PM, Nick Coghlan wrote: >> >> The issue isn't really the ssl module itself - it's the other modules >> that *depend* on the ssl module (like httplib, urllib, poplib, ftplib, >> imaplib). You could technically try to monkeypatch or shadow the >> stdlib ssl module from a third party library, but that's a *really* >> dangerous approach to security (since it makes it hard to do proper >> integration testing). > > I suppose the difference is that the applications I've worked with have *already* abandoned stdlib modules like httplib or urllib in favor of libraries like requests and tornado, so I see this as mainly a question of whether the stdlib provides a suitable foundation for those packages. > > The argument about the difficulty of integration testing cuts both ways - I can use tox to test Tornado with several versions of a third-party package, but cannot (as long as https://launchpad.net/~fkrull/+archive/deadsnakes only provides one package per x.y release) easily test with both Python 2.7.6 and 2.7.7. > >> >> > In my experience most deployments of Python apps have better systems in >> > place to deal with packages installed from PyPI than new versions of Python >> > itself. Therefore it might be better to follow the lead of the unittest2 >> > module and introduce a new ssl2 module that tracks the most recent version >> > of Python's ssl module. >> > >> > My counter-proposal is: >> > * Introduce a new module 'ssl2' which is essentially a backport of the >> > current release's ssl module for all old versions of Python that anyone >> > cares to support. >> >> It's not just ssl2: you needs os2, random2, ftplib2, poplib2, >> imaplib2, smtplib2. You also run headlong into the problem of a >> complex external dependency on OpenSSL (which the Linux distro >> repackagers currently handle for us). > > Just os2 and random2; as I said a few bullet points down the stdlib libraries that use ssl would be sufficient justification for additions to the ssl module's interface. > >> >> This is what the """reimplementing substantial portions of the >> standard library as third party modules to gain access to the newer >> underlying network security protocols and mechanisms, and then >> injecting those into the module namespace to override their standard >> library counterparts.""" comment is about, but I guess I should expand >> on that for the benefit of readers not as familiar with the relevant >> implementation details. > > In my proposal there is no monkey-patching; the newer ssl interfaces would come solely from the ssl2 module. > >> >> > * Third-party applications and libraries targeting anything but the latest >> > version of Python are advised to use ssl2 instead of ssl (this might also be >> > a good opportunity to bless certifi or an equivalent). >> > * Minor releases of Python are allowed to make any security-related change >> > that does not change any public interfaces (e.g. linking against a new >> > version of openssl, or disabling SSLv2) >> >> New versions of OpenSSL are nowhere near as useful when their >> constants aren't made available in the Python API. >> >> > * New interfaces in security-related modules are allowed only if those new >> > interfaces are used by other stdlib modules (since stdlib modules are the >> > only place where ssl2 is off-limits). For example, a backport of >> > match_hostname could go in if httplib or urllib used it. NPN/ALPN would not >> > be allowed under this rule since there is no stdlib code that uses them; >> > third-party SPDY or HTTP/2 code would have to use ssl2 to get this >> > interface. >> > >> > In this world, libraries like requests or tornado that want to use advanced >> > ssl features can simply declare a dependency on ssl2, instead of saying "we >> > require Python 2.7.7+, 3.2.6+, etc". >> >> There's a reason the developers of those libraries haven't already >> done that: it's genuinely hard. It's not that I didn't think of this >> approach, it's that I think it would actually be more difficult and >> less effective than the approach currently in the PEP. > > One of the reasons it's genuinely hard is that no one has provided an interface compatible with the stdlib ssl module. An ssl2 module that backported Python 3.4's ssl module would be a *much* more attractive porting target than e.g. PyOpenSSL. > > -Ben > >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Mar 23 00:54:12 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 22 Mar 2014 23:54:12 +0000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <5063D16D-0E47-451F-B5D8-5EE27A677ACC@stufft.io> References: <5063D16D-0E47-451F-B5D8-5EE27A677ACC@stufft.io> Message-ID: On 22 March 2014 23:49, Donald Stufft wrote: > In the case of requests they already have an optional dependency on > pyopenssl. It's just many people either don't know they should use it, are > unable to use it, or unwilling to use the python packaging tool chain > because of its current flaws. Do they use the new features in the Python 3.x ssl module when it's available to give the same level of security as having pyopenssl would, or do they use a "lowest common denominator" (i.e., 2.x compatible) level of security when using the stdlib? If the latter, that would be very, very sad. Paul From donald at stufft.io Sun Mar 23 00:56:10 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 22 Mar 2014 19:56:10 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <5063D16D-0E47-451F-B5D8-5EE27A677ACC@stufft.io> References: <5063D16D-0E47-451F-B5D8-5EE27A677ACC@stufft.io> Message-ID: Also important to remember that pip itself uses the OpenSSL binding in the ssl module so there is a chicken and egg problem. > On Mar 22, 2014, at 7:49 PM, Donald Stufft wrote: > > In the case of requests they already have an optional dependency on pyopenssl. It's just many people either don't know they should use it, are unable to use it, or unwilling to use the python packaging tool chain because of its current flaws. > >> On Mar 22, 2014, at 7:42 PM, Ben Darnell wrote: >> >>> On Sat, Mar 22, 2014 at 7:08 PM, Nick Coghlan wrote: >>> >>> The issue isn't really the ssl module itself - it's the other modules >>> that *depend* on the ssl module (like httplib, urllib, poplib, ftplib, >>> imaplib). You could technically try to monkeypatch or shadow the >>> stdlib ssl module from a third party library, but that's a *really* >>> dangerous approach to security (since it makes it hard to do proper >>> integration testing). >> >> I suppose the difference is that the applications I've worked with have *already* abandoned stdlib modules like httplib or urllib in favor of libraries like requests and tornado, so I see this as mainly a question of whether the stdlib provides a suitable foundation for those packages. >> >> The argument about the difficulty of integration testing cuts both ways - I can use tox to test Tornado with several versions of a third-party package, but cannot (as long as https://launchpad.net/~fkrull/+archive/deadsnakes only provides one package per x.y release) easily test with both Python 2.7.6 and 2.7.7. >> >>> >>> > In my experience most deployments of Python apps have better systems in >>> > place to deal with packages installed from PyPI than new versions of Python >>> > itself. Therefore it might be better to follow the lead of the unittest2 >>> > module and introduce a new ssl2 module that tracks the most recent version >>> > of Python's ssl module. >>> > >>> > My counter-proposal is: >>> > * Introduce a new module 'ssl2' which is essentially a backport of the >>> > current release's ssl module for all old versions of Python that anyone >>> > cares to support. >>> >>> It's not just ssl2: you needs os2, random2, ftplib2, poplib2, >>> imaplib2, smtplib2. You also run headlong into the problem of a >>> complex external dependency on OpenSSL (which the Linux distro >>> repackagers currently handle for us). >> >> Just os2 and random2; as I said a few bullet points down the stdlib libraries that use ssl would be sufficient justification for additions to the ssl module's interface. >> >>> >>> This is what the """reimplementing substantial portions of the >>> standard library as third party modules to gain access to the newer >>> underlying network security protocols and mechanisms, and then >>> injecting those into the module namespace to override their standard >>> library counterparts.""" comment is about, but I guess I should expand >>> on that for the benefit of readers not as familiar with the relevant >>> implementation details. >> >> In my proposal there is no monkey-patching; the newer ssl interfaces would come solely from the ssl2 module. >> >>> >>> > * Third-party applications and libraries targeting anything but the latest >>> > version of Python are advised to use ssl2 instead of ssl (this might also be >>> > a good opportunity to bless certifi or an equivalent). >>> > * Minor releases of Python are allowed to make any security-related change >>> > that does not change any public interfaces (e.g. linking against a new >>> > version of openssl, or disabling SSLv2) >>> >>> New versions of OpenSSL are nowhere near as useful when their >>> constants aren't made available in the Python API. >>> >>> > * New interfaces in security-related modules are allowed only if those new >>> > interfaces are used by other stdlib modules (since stdlib modules are the >>> > only place where ssl2 is off-limits). For example, a backport of >>> > match_hostname could go in if httplib or urllib used it. NPN/ALPN would not >>> > be allowed under this rule since there is no stdlib code that uses them; >>> > third-party SPDY or HTTP/2 code would have to use ssl2 to get this >>> > interface. >>> > >>> > In this world, libraries like requests or tornado that want to use advanced >>> > ssl features can simply declare a dependency on ssl2, instead of saying "we >>> > require Python 2.7.7+, 3.2.6+, etc". >>> >>> There's a reason the developers of those libraries haven't already >>> done that: it's genuinely hard. It's not that I didn't think of this >>> approach, it's that I think it would actually be more difficult and >>> less effective than the approach currently in the PEP. >> >> One of the reasons it's genuinely hard is that no one has provided an interface compatible with the stdlib ssl module. An ssl2 module that backported Python 3.4's ssl module would be a *much* more attractive porting target than e.g. PyOpenSSL. >> >> -Ben >> >>> >>> Cheers, >>> Nick. >>> >>> -- >>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Mar 23 00:57:55 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 22 Mar 2014 19:57:55 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <5063D16D-0E47-451F-B5D8-5EE27A677ACC@stufft.io> Message-ID: <9725F0A2-81DB-4E8C-91D0-AD082D2A8322@stufft.io> They detect for ssl to have the SSLContext and use it if it's available. > On Mar 22, 2014, at 7:54 PM, Paul Moore wrote: > >> On 22 March 2014 23:49, Donald Stufft wrote: >> In the case of requests they already have an optional dependency on >> pyopenssl. It's just many people either don't know they should use it, are >> unable to use it, or unwilling to use the python packaging tool chain >> because of its current flaws. > > Do they use the new features in the Python 3.x ssl module when it's > available to give the same level of security as having pyopenssl > would, or do they use a "lowest common denominator" (i.e., 2.x > compatible) level of security when using the stdlib? If the latter, > that would be very, very sad. > > Paul From rosuav at gmail.com Sun Mar 23 00:59:22 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 23 Mar 2014 10:59:22 +1100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <20140323003307.3472575c@fsol> References: <2C00F7E7-D1AD-47AD-8FCE-E3A72F25F7CF@stufft.io> <20140323003307.3472575c@fsol> Message-ID: On Sun, Mar 23, 2014 at 10:33 AM, Antoine Pitrou wrote: > On Sat, 22 Mar 2014 19:07:51 -0400 > Donald Stufft wrote: >> As someone who is deeply biased towards improving the packaging tool chain and getting people to use it I think that most people will simply use the Stdlib even if a more secure alternative exists. Infact one does exist and I still see almost everyone using the stdlib ssl instead of pyopenssl. > > pyopenssl exposes a different API from ssl, though. It's not just a > matter of flipping an import statement. How hard would it be to create a wrapper module named "ssl.py", which imports pyopenssl (thus taking advantage of any updates it gets), and could be placed earlier in the search path than the stdlib one? I'm +1 on the overall concept, but not sure about 2.6 and older. 2.7 is already a bit of an odd duck; it's being supported for longer than its official Windows compiler, it's going to be supported by Red Hat until eternity and beyond, and it wouldn't in the least surprise me if 2.7.10 introduces a bit of stuff, backported, in ways that are fundamentally security fixes. Is it really that bad to have something depend on "2.7.10" rather than "2.7.0"? If a program will bomb without some bugfix, then it's not unreasonable to say that it depends on a version of Python that includes that fix. As long as any program written for 2.7.0 will run on 2.7.x unchanged (the normal expectations with semantic versioning), I don't see a problem with some of the "bug fixes" actually adding attributes; imagine if some feature attribute were simply misspelled in the previous versions, which could surely be fixed in a point release; it's not going to break anything. ChrisA From solipsis at pitrou.net Sun Mar 23 01:01:38 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 Mar 2014 01:01:38 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements References: Message-ID: <20140323010138.48bcafe4@fsol> On Sun, 23 Mar 2014 07:11:07 +1000 Nick Coghlan wrote: > > It is similar to the previous IDLE policy exception PEP, where we > decided that cross version consistency of IDLE superseded the general > policy against backporting enhancements to maintenance branches. But the consequences are different: the stdlib promises API stability between bugfix releases, while IDLE is a standalone GUI application. > This topic > involves a complex balance between encouraging and supporting good > security practices and limiting the risk of failures for users > upgrading to new maintenance releases, so I'd ask that folks take time > to read and consider the implications of the full PEP in the broader > context of today's internet before posting comments on specific > details, or indicating a preference one way or the other in terms of > the overall proposal. The fact that it involves a "complex balance" implies IMO that a blanket exemption is the wrong mechanism, and instead the exemption should probably be granted on a case-by-case basis. > This PEP does *not* grant any general exemptions to the usual backwards > compatibility policy for maintenance releases. Instead, it is designed > to make it easier to provide more "secure by default" behaviour in future > feature releases, while still limiting the risk of breaking currently > working software when upgrading to a new maintenance release. But enforcing "secure by default" can by construction break backwards compatibility, which is the very reason we are so conservative with such changes. > Thus the focus on > network security protocols and related cryptographic infrastructure - This is a bit limited. There are remotely-exploitable security issues which are still open on the bug tracker; they are not cryptography-related, but that shouldn't make a difference. (for example the dreaded XML issues have never been properly fixed, AFAICT) > My position on the ongoing transition from Python 2 to Python 3 has long > been that Python 2 remains a supported platform for the core development > team, and that commercial support will remain available well after upstream > maintenance ends. If this is about the difficulty of migrating to Python 3, then this PEP should focus specifically on that problem, and restrict the policy to Python 2.7. Upgrading from 3.X to 3.X+1 is easy, and so is backporting patches from 3.X to 3.X-1, conversely. > * Are there any other security relevant modules that should be covered > by either a blanket or conditional exemption? Every module can be remotely "security relevant", unfortunately. Regards Antoine. From solipsis at pitrou.net Sun Mar 23 01:08:37 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 Mar 2014 01:08:37 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> Message-ID: <20140323010837.45776044@fsol> On Sat, 22 Mar 2014 23:54:37 +0100 Thomas Wouters wrote: > > Or not rely on the standard library for their security. Much as I realize > it is necessary for rudimentary SSL support (for example) to exist in the > standard library, Unfortunately, "rudimentary SSL support" is worse than nothing. Regards Antoine. From solipsis at pitrou.net Sun Mar 23 01:05:11 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 Mar 2014 01:05:11 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements References: <2C00F7E7-D1AD-47AD-8FCE-E3A72F25F7CF@stufft.io> <20140323003307.3472575c@fsol> Message-ID: <20140323010511.464eda37@fsol> On Sun, 23 Mar 2014 10:59:22 +1100 Chris Angelico wrote: > > Is it really that bad to have something depend on "2.7.10" rather than > "2.7.0"? It makes our whole release and versioning scheme very confusing (hopefully people will not have to read Nick's PEP to understand it :-)). Regards Antoine. From christian at python.org Sun Mar 23 01:15:52 2014 From: christian at python.org (Christian Heimes) Date: Sun, 23 Mar 2014 01:15:52 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <20140323010138.48bcafe4@fsol> References: <20140323010138.48bcafe4@fsol> Message-ID: On 23.03.2014 01:01, Antoine Pitrou wrote: > This is a bit limited. There are remotely-exploitable security issues > which are still open on the bug tracker; they are not > cryptography-related, but that shouldn't make a difference. > > (for example the dreaded XML issues have never been properly fixed, > AFAICT) True, you may blame me for the situation. Only a handful of people were interested in the XML issues. I ran out of steam and moved to more sapid topics, too. Christian From ncoghlan at gmail.com Sun Mar 23 01:33:08 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Mar 2014 10:33:08 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <20140323010138.48bcafe4@fsol> Message-ID: On 23 Mar 2014 10:18, "Christian Heimes" wrote: > > On 23.03.2014 01:01, Antoine Pitrou wrote: > > This is a bit limited. There are remotely-exploitable security issues > > which are still open on the bug tracker; they are not > > cryptography-related, but that shouldn't make a difference. > > > > (for example the dreaded XML issues have never been properly fixed, > > AFAICT) > > True, you may blame me for the situation. Only a handful of people were > interested in the XML issues. I ran out of steam and moved to more sapid > topics, too. The key difference there is we're not reliant on defusedxml to download defusedxml :) Anyway, folks have given me plenty of good feedback to chew over. I'm going to go run some errands and then try to get a new revision posted this afternoon. Cheers, Nick. > > Christian > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Sun Mar 23 01:48:46 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 22 Mar 2014 20:48:46 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <20140323010138.48bcafe4@fsol> References: <20140323010138.48bcafe4@fsol> Message-ID: <20140323004847.4DF4B250030@webabinitio.net> On Sun, 23 Mar 2014 01:01:38 +0100, Antoine Pitrou wrote: > On Sun, 23 Mar 2014 07:11:07 +1000 > Nick Coghlan wrote: > > This PEP does *not* grant any general exemptions to the usual backwards > > compatibility policy for maintenance releases. Instead, it is designed > > to make it easier to provide more "secure by default" behaviour in future > > feature releases, while still limiting the risk of breaking currently > > working software when upgrading to a new maintenance release. > > But enforcing "secure by default" can by construction break backwards > compatibility, which is the very reason we are so conservative with > such changes. Yeah, I couldn't figure out what this paragraph meant. What does backporting features in maintenance releases have to do with making feature releases secure by default? If we are not relaxing the backward compatibility rules we can't do secure by default changes in a maintenance release...and if we don't do that, thinking that very much legacy software is going to get upgraded to use the new "more secure" capabilities is, I suspect (but cannot prove, obviously) mostly wishful thinking. An exception might be be commercial packages, but even there I wouldn't bet very many bucks. It seems to me that shops who are being security concious enough to be candidates for modifying currently deployed packages to be more secure are also likely to have already addressed the security issues in some other way. Now, requests and other libraries could use these new features, and since they have faster release cycles, they could presumably make the increased security be the default sooner. So that's what I see as the argument in favor...but it is a much narrower impact area than "most python software that uses ssl", and certainly doesn't cover poplib, smtplib, etc. But perhaps I am being overly cynical. Regardless, like Martin, I prefer to focus on Python3. --David From martin at v.loewis.de Sun Mar 23 01:40:32 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 23 Mar 2014 01:40:32 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <20140323010138.48bcafe4@fsol> Message-ID: <532E2D80.1050504@v.loewis.de> Am 23.03.14 01:15, schrieb Christian Heimes: > On 23.03.2014 01:01, Antoine Pitrou wrote: >> This is a bit limited. There are remotely-exploitable security issues >> which are still open on the bug tracker; they are not >> cryptography-related, but that shouldn't make a difference. >> >> (for example the dreaded XML issues have never been properly fixed, >> AFAICT) > > True, you may blame me for the situation. Only a handful of people were > interested in the XML issues. I ran out of steam and moved to more sapid > topics, too I don't think anybody wanted to assign blame (although I can sympathize with your urge to accept the blame). The fact is that this is a volunteer project: we do what we can and have fun doing. Regards, Martin From ncoghlan at gmail.com Sun Mar 23 01:55:11 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Mar 2014 10:55:11 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <20140323010837.45776044@fsol> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> Message-ID: On 23 March 2014 10:08, Antoine Pitrou wrote: > On Sat, 22 Mar 2014 23:54:37 +0100 > Thomas Wouters wrote: >> >> Or not rely on the standard library for their security. Much as I realize >> it is necessary for rudimentary SSL support (for example) to exist in the >> standard library, > > Unfortunately, "rudimentary SSL support" is worse than nothing. I'm going to try to find a way to steal that line and get it into the PEP. I'm not sure yet if my proposal is the *right* answer, but this observation gets right to the heart of the problem that actually needs to be solved. I'd actually welcome a competing PEP that seriously explored separating out the relevant parts of the Python 3.4 standard library as a PyPI project and attempted to tackle the problem by making that available at least as far back as 2.7 and then just deprecating the leaking batteries in the 2.7 standard library. Just because I think that approach would be even more work and even more disruptive than what I propose in PEP 466, doesn't mean it isn't a better idea overall. What we have essentially found is that where we could basically get away with an 18 month update cycle for improved network security support (extended out to a few years by certain major platform vendors), that approach *isn't* working when it comes to putting a feature release into long term maintenance mode. I don't think the situation isn't critical yet, but it's getting close, and I think we need to deal with it within the 12 months (and preferably sooner than that). My PEP proposes one way of addressing the challenge: allowing a wider variety of changes in maintenance releases to reduce the variance in capability in this area. Moving the affected modules out of the standard library proper and bundling the critical ones along with pip instead is indeed another alternative. However, that approach introduces additional issues of its own - I'll cover some of them in the next PEP update, but it would be good to have someone explicitly trying to make the case that a PyPI backport would be simpler for the overall ecosystem than my suggested approach. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Mar 23 02:07:26 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Mar 2014 11:07:26 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <532E2D80.1050504@v.loewis.de> References: <20140323010138.48bcafe4@fsol> <532E2D80.1050504@v.loewis.de> Message-ID: On 23 March 2014 10:40, "Martin v. L?wis" wrote: > Am 23.03.14 01:15, schrieb Christian Heimes: >> On 23.03.2014 01:01, Antoine Pitrou wrote: >>> This is a bit limited. There are remotely-exploitable security issues >>> which are still open on the bug tracker; they are not >>> cryptography-related, but that shouldn't make a difference. >>> >>> (for example the dreaded XML issues have never been properly fixed, >>> AFAICT) >> >> True, you may blame me for the situation. Only a handful of people were >> interested in the XML issues. I ran out of steam and moved to more sapid >> topics, too > > I don't think anybody wanted to assign blame (although I can sympathize > with your urge to accept the blame). The fact is that this is a > volunteer project: we do what we can and have fun doing. Agreed completely - what I'm trying to do here is set up a plan that is at least *acceptable* to the upstream community, so we can then seek corporate support for actually putting it into practice (and I'd advise against us accepting *any* proposal to resolve the situation without receiving binding commitments to provide ongoing maintenance support - while I think this proposal is important, I'm under no illusions that actually implementing it will be fun, and it's not appropriate to ask people to do that in their own time). However, we have a *lot* of downstream users and redistributors that have been taking CPython core development for granted, and by so doing, they have allowed a situation to develop that has some rather negative implications for the overall security of networked communications in the Python ecosystem. Since some of those same corporate redistributors are a key enabler allowing users to stay on those old releases that are no longer supported upstream, and others are likely to be being conservative in their own Python 3 migrations, I believe they share a lot of the responsibility for helping to resolve it, either by facilitating the migration to Python 3, helping to improve the networking security situation in Python 2, or, preferably, both. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ben at bendarnell.com Sun Mar 23 02:17:09 2014 From: ben at bendarnell.com (Ben Darnell) Date: Sat, 22 Mar 2014 21:17:09 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> Message-ID: On Sat, Mar 22, 2014 at 8:55 PM, Nick Coghlan wrote: > What we have essentially found is that where we could basically get > away with an 18 month update cycle for improved network security > support (extended out to a few years by certain major platform > vendors), that approach *isn't* working when it comes to putting a > feature release into long term maintenance mode. I don't think the > situation isn't critical yet, but it's getting close, and I think we > need to deal with it within the 12 months (and preferably sooner than > that). > This PEP as written applies to both Python 2.x and 3.x, but the two situations are very different. 3.x is on a ~18 month update cycle, so why isn't the status quo acceptable there? Python 2.x has less than 18 months of support left, so could it get by with a single exceptional release instead of a general relaxing of the rules? (if it were up to me, I'd call that release Python 2.8 instead of 2.7.7) If this PEP is mainly about a one-shot update to the security components of Python 2.x, I'd like to see an explicit list of what is in scope for the update. -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Mar 23 02:27:21 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 Mar 2014 02:27:21 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements References: <20140323010138.48bcafe4@fsol> <532E2D80.1050504@v.loewis.de> Message-ID: <20140323022721.0ee96c53@fsol> On Sun, 23 Mar 2014 01:40:32 +0100 "Martin v. L?wis" wrote: > Am 23.03.14 01:15, schrieb Christian Heimes: > > On 23.03.2014 01:01, Antoine Pitrou wrote: > >> This is a bit limited. There are remotely-exploitable security issues > >> which are still open on the bug tracker; they are not > >> cryptography-related, but that shouldn't make a difference. > >> > >> (for example the dreaded XML issues have never been properly fixed, > >> AFAICT) > > > > True, you may blame me for the situation. Only a handful of people were > > interested in the XML issues. I ran out of steam and moved to more sapid > > topics, too > > I don't think anybody wanted to assign blame (although I can sympathize > with your urge to accept the blame). The fact is that this is a > volunteer project: we do what we can and have fun doing. Indeed this wasn't meant to blame anyone, simply to point out that improving security is often hampered by lack of manpower or motivation. Regards Antoine. From donald at stufft.io Sun Mar 23 02:30:17 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 22 Mar 2014 21:30:17 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> Message-ID: On Mar 22, 2014, at 8:55 PM, Nick Coghlan wrote: > Moving the affected modules out of the standard library proper and > bundling the critical ones along with pip instead is indeed another > alternative. However, that approach introduces additional issues of > its own - I'll cover some of them in the next PEP update, but it would > be good to have someone explicitly trying to make the case that a PyPI > backport would be simpler for the overall ecosystem than my suggested > approach. FWIW pip as of right now has a policy of no C dependencies outside of the stdlib. CPython isn?t our only target and C dependencies don?t work very well on PyPy (if at all) and it makes the situation much more difficult on platforms where there are no compiler toolchains (Windows). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From guido at python.org Sun Mar 23 02:37:24 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 22 Mar 2014 18:37:24 -0700 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> Message-ID: I'm a bit under the weather and I'm not sure what to think of this yet. With that provision, and trying to be brief: I agree that there are security concerns about Python 2.7 that can't be addressed by recommending Python 3.4 instead. I also agree that the ban on new features in old releases can be stifling. But I also worry about breaking working code. I am a bit worried that writing a PEP (at least this specific PEP) would become less about arguing for a specific technical (or even political) idea, and more about putting "weasel words" in an official document so that it can be used to justify future actions without due process. I wonder whether there is a specific narrower proposal (e.g. add SSLContext to Python 2.7.7?) that would make the discussion more concrete. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Sun Mar 23 02:33:25 2014 From: bcannon at gmail.com (Brett Cannon) Date: Sun, 23 Mar 2014 01:33:25 +0000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> Message-ID: On Sat Mar 22 2014 at 8:55:36 PM, Nick Coghlan wrote: > On 23 March 2014 10:08, Antoine Pitrou wrote: > > On Sat, 22 Mar 2014 23:54:37 +0100 > > Thomas Wouters wrote: > >> > >> Or not rely on the standard library for their security. Much as I > realize > >> it is necessary for rudimentary SSL support (for example) to exist in > the > >> standard library, > > > > Unfortunately, "rudimentary SSL support" is worse than nothing. > > I'm going to try to find a way to steal that line and get it into the > PEP. I'm not sure yet if my proposal is the *right* answer, but this > observation gets right to the heart of the problem that actually needs > to be solved. > > I'd actually welcome a competing PEP that seriously explored > separating out the relevant parts of the Python 3.4 standard library > as a PyPI project and attempted to tackle the problem by making that > available at least as far back as 2.7 and then just deprecating the > leaking batteries in the 2.7 standard library. > Now I have been reading this thread on my phone and I only have cursory understanding of what failure ssl has had as of late, so this might be stupid, but what if in Python 3.5 we made it so people passed in an explicit SSL object into the relevant APIs and threw an exception if one wasn't provided when trying to access something requiring SSL? Then the ssl module would start existing on PyPI to make sure it can evolve faster than every 18 months? I know this doesn't help older APIs but it would make it rather explicit going forward that if you want security you need the latest thing and that's found on PyPI so it would at least solve the issue for the future which is better than nothing. We could try to provide matching APIs that monkey-patch older versions of Python if that's so desired to make this backwards-compatible, but as we all know that can be rather brittle. -Brett > > Just because I think that approach would be even more work and even > more disruptive than what I propose in PEP 466, doesn't mean it isn't > a better idea overall. > > What we have essentially found is that where we could basically get > away with an 18 month update cycle for improved network security > support (extended out to a few years by certain major platform > vendors), that approach *isn't* working when it comes to putting a > feature release into long term maintenance mode. I don't think the > situation isn't critical yet, but it's getting close, and I think we > need to deal with it within the 12 months (and preferably sooner than > that). > > My PEP proposes one way of addressing the challenge: allowing a wider > variety of changes in maintenance releases to reduce the variance in > capability in this area. > > Moving the affected modules out of the standard library proper and > bundling the critical ones along with pip instead is indeed another > alternative. However, that approach introduces additional issues of > its own - I'll cover some of them in the next PEP update, but it would > be good to have someone explicitly trying to make the case that a PyPI > backport would be simpler for the overall ecosystem than my suggested > approach. > > Cheers, > Nick. > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Mar 23 02:41:00 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 22 Mar 2014 21:41:00 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> Message-ID: <88AFE467-CD0F-4EA1-A966-226AB7EC2268@stufft.io> On Mar 22, 2014, at 9:17 PM, Ben Darnell wrote: > On Sat, Mar 22, 2014 at 8:55 PM, Nick Coghlan wrote: > What we have essentially found is that where we could basically get > away with an 18 month update cycle for improved network security > support (extended out to a few years by certain major platform > vendors), that approach *isn't* working when it comes to putting a > feature release into long term maintenance mode. I don't think the > situation isn't critical yet, but it's getting close, and I think we > need to deal with it within the 12 months (and preferably sooner than > that). > > This PEP as written applies to both Python 2.x and 3.x, but the two situations are very different. 3.x is on a ~18 month update cycle, so why isn't the status quo acceptable there? Python 2.x has less than 18 months of support left, so could it get by with a single exceptional release instead of a general relaxing of the rules? (if it were up to me, I'd call that release Python 2.8 instead of 2.7.7) If this PEP is mainly about a one-shot update to the security components of Python 2.x, I'd like to see an explicit list of what is in scope for the update. > > -Ben > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io As one of the instigators who convinced Nick to write this PEP I?m far less concerned about back porting things to earlier 3.x releases than I am about getting things into 2.7 at this point. Going from 3.2 to 3.4 is not terribly difficult if it requires any work at all. Going from 2.7 to 3.4 is often times a significant investment in resources that has to be taken by *every* network using project. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sun Mar 23 02:47:27 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 22 Mar 2014 21:47:27 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> Message-ID: <5AA3E8B5-9AE0-4EAA-A9DF-6A0CEB96DD4D@stufft.io> On Mar 22, 2014, at 9:33 PM, Brett Cannon wrote: > > > On Sat Mar 22 2014 at 8:55:36 PM, Nick Coghlan wrote: > On 23 March 2014 10:08, Antoine Pitrou wrote: > > On Sat, 22 Mar 2014 23:54:37 +0100 > > Thomas Wouters wrote: > >> > >> Or not rely on the standard library for their security. Much as I realize > >> it is necessary for rudimentary SSL support (for example) to exist in the > >> standard library, > > > > Unfortunately, "rudimentary SSL support" is worse than nothing. > > I'm going to try to find a way to steal that line and get it into the > PEP. I'm not sure yet if my proposal is the *right* answer, but this > observation gets right to the heart of the problem that actually needs > to be solved. > > I'd actually welcome a competing PEP that seriously explored > separating out the relevant parts of the Python 3.4 standard library > as a PyPI project and attempted to tackle the problem by making that > available at least as far back as 2.7 and then just deprecating the > leaking batteries in the 2.7 standard library. > > Now I have been reading this thread on my phone and I only have cursory understanding of what failure ssl has had as of late, so this might be stupid, but what if in Python 3.5 we made it so people passed in an explicit SSL object into the relevant APIs and threw an exception if one wasn't provided when trying to access something requiring SSL? Then the ssl module would start existing on PyPI to make sure it can evolve faster than every 18 months? It?s not really a failure so much as what was concerned secure in 2010 isn?t the case in 2014. I don?t want to sounds like I?m trying to say that pip should dictate Python policy or anything but pip more or less depends on the fact that ssl is available in the stdlib at this point in time in order to be able to handle secure downloads. Every 18 months isn?t that big of a deal tbh, the larger concern is Python 2.7 which has had nearly 4 years since it?s release and is likely to exist for far more years past that. > > I know this doesn't help older APIs but it would make it rather explicit going forward that if you want security you need the latest thing and that's found on PyPI so it would at least solve the issue for the future which is better than nothing. We could try to provide matching APIs that monkey-patch older versions of Python if that's so desired to make this backwards-compatible, but as we all know that can be rather brittle. > > -Brett > > > Just because I think that approach would be even more work and even > more disruptive than what I propose in PEP 466, doesn't mean it isn't > a better idea overall. > > What we have essentially found is that where we could basically get > away with an 18 month update cycle for improved network security > support (extended out to a few years by certain major platform > vendors), that approach *isn't* working when it comes to putting a > feature release into long term maintenance mode. I don't think the > situation isn't critical yet, but it's getting close, and I think we > need to deal with it within the 12 months (and preferably sooner than > that). > > My PEP proposes one way of addressing the challenge: allowing a wider > variety of changes in maintenance releases to reduce the variance in > capability in this area. > > Moving the affected modules out of the standard library proper and > bundling the critical ones along with pip instead is indeed another > alternative. However, that approach introduces additional issues of > its own - I'll cover some of them in the next PEP update, but it would > be good to have someone explicitly trying to make the case that a PyPI > backport would be simpler for the overall ecosystem than my suggested > approach. > > Cheers, > Nick. > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Sun Mar 23 04:18:02 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Mar 2014 13:18:02 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <5AA3E8B5-9AE0-4EAA-A9DF-6A0CEB96DD4D@stufft.io> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <5AA3E8B5-9AE0-4EAA-A9DF-6A0CEB96DD4D@stufft.io> Message-ID: On 23 Mar 2014 11:47, "Donald Stufft" wrote: > > > On Mar 22, 2014, at 9:33 PM, Brett Cannon wrote: > >> >> >> On Sat Mar 22 2014 at 8:55:36 PM, Nick Coghlan wrote: >>> >>> On 23 March 2014 10:08, Antoine Pitrou wrote: >>> > On Sat, 22 Mar 2014 23:54:37 +0100 >>> > Thomas Wouters wrote: >>> >> >>> >> Or not rely on the standard library for their security. Much as I realize >>> >> it is necessary for rudimentary SSL support (for example) to exist in the >>> >> standard library, >>> > >>> > Unfortunately, "rudimentary SSL support" is worse than nothing. >>> >>> I'm going to try to find a way to steal that line and get it into the >>> PEP. I'm not sure yet if my proposal is the *right* answer, but this >>> observation gets right to the heart of the problem that actually needs >>> to be solved. >>> >>> I'd actually welcome a competing PEP that seriously explored >>> separating out the relevant parts of the Python 3.4 standard library >>> as a PyPI project and attempted to tackle the problem by making that >>> available at least as far back as 2.7 and then just deprecating the >>> leaking batteries in the 2.7 standard library. >> >> >> Now I have been reading this thread on my phone and I only have cursory understanding of what failure ssl has had as of late, so this might be stupid, but what if in Python 3.5 we made it so people passed in an explicit SSL object into the relevant APIs and threw an exception if one wasn't provided when trying to access something requiring SSL? Then the ssl module would start existing on PyPI to make sure it can evolve faster than every 18 months? > > > It's not really a failure so much as what was concerned secure in 2010 isn't the case in 2014. > > I don't want to sounds like I'm trying to say that pip should dictate Python policy or anything > but pip more or less depends on the fact that ssl is available in the stdlib at this point in time > in order to be able to handle secure downloads. > > Every 18 months isn't that big of a deal tbh, the larger concern is Python 2.7 which has had > nearly 4 years since it's release and is likely to exist for far more years past that. Yes, I think it's worth narrowing the scope to just Python 2.7, and leaving the process unchanged for 3.x. Cheers, Nick. > >> >> I know this doesn't help older APIs but it would make it rather explicit going forward that if you want security you need the latest thing and that's found on PyPI so it would at least solve the issue for the future which is better than nothing. We could try to provide matching APIs that monkey-patch older versions of Python if that's so desired to make this backwards-compatible, but as we all know that can be rather brittle. >> >> -Brett >> >>> >>> >>> Just because I think that approach would be even more work and even >>> more disruptive than what I propose in PEP 466, doesn't mean it isn't >>> a better idea overall. >>> >>> What we have essentially found is that where we could basically get >>> away with an 18 month update cycle for improved network security >>> support (extended out to a few years by certain major platform >>> vendors), that approach *isn't* working when it comes to putting a >>> feature release into long term maintenance mode. I don't think the >>> situation isn't critical yet, but it's getting close, and I think we >>> need to deal with it within the 12 months (and preferably sooner than >>> that). >>> >>> My PEP proposes one way of addressing the challenge: allowing a wider >>> variety of changes in maintenance releases to reduce the variance in >>> capability in this area. >>> >>> Moving the affected modules out of the standard library proper and >>> bundling the critical ones along with pip instead is indeed another >>> alternative. However, that approach introduces additional issues of >>> its own - I'll cover some of them in the next PEP update, but it would >>> be good to have someone explicitly trying to make the case that a PyPI >>> backport would be simpler for the overall ecosystem than my suggested >>> approach. >>> >>> Cheers, >>> Nick. >>> >>> >>> -- >>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io > > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Mar 23 05:31:36 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 23 Mar 2014 00:31:36 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> Message-ID: <532E63A8.9020705@udel.edu> On 3/22/2014 8:55 PM, Nick Coghlan wrote: >> Unfortunately, "rudimentary SSL support" is worse than nothing. > > I'm going to try to find a way to steal that line and get it into the > PEP. I'm not sure yet if my proposal is the *right* answer, but this > observation gets right to the heart of the problem that actually needs > to be solved. > > I'd actually welcome a competing PEP that seriously explored > separating out the relevant parts of the Python 3.4 standard library > as a PyPI project and attempted to tackle the problem by making that > available at least as far back as 2.7 and then just deprecating the > leaking batteries in the 2.7 standard library. I have a competing proposal that is a variation on yours. First, restrict attention to 2.7. The problem, as you defined it in the PEP, is that a) 2.7 is currently not an adequate base for public internet servers, coupled with b) some companies are determined to continue using 2.x for their servers, and c) condition b) is not going to end in a year or two. Your proposal is to enhance 2.7 by backporting some indefinite number of server security features from 3.x to 2.7. The first problem with this idea is that the result will not be Python 2.7 as currently defined. So lets not break our naming and stability policy and and engender legitimate objections by labeling Server Security Python as Python 2.7.z, where z is a single digit. The second problem is that enhancement will not, I strongly suspect, be a one time event. So there will need to be several enhancement releases. Producing a generally enhanced 2.8, 2.9, 2.10, etc, for the next decade would solve both problems -- but we are adamantly not going to do that. Instead, I think the PEP should propose a special series of server enhancement releases that are based on the final 2.7 maintenance release (2.7.8 or 2.7.9) but which have have a different application-specific enhancement policy. It would start with something like 'ServerPython 2.7.10', with the numbers moving upward from 10. I think one justification for this is that if PSF does not sanction such a series, it will happen anyway, but possibly with different and incompatible releases from different vendors or distribution families. As I understand the proposal, a subset of core developers would set the specification while some commercial group would have to fund it. The download page for the final 2.7.z maintenance release could say something like "We recommend that you move to the most recent Python 3 version if at all possible. If you cannot do that and you want to use Python to run a server on the public internet, we urge you to instead use the latest version of ServerPython 2.7.1s. This series is based on Python 2.7.z but has been and will continue to be enhanced with security features backported from Python 3." Part of the idea above is that 2.7 development would end or mostly end after 5 years, the difference being that the last 3 years have not been restricted only to security bugs. Since security bugs are nearly always server problems, they would be fixed thereafter in ServerPython. If ServerPython is kept in the CPython repository, it would be a separate branch. I would, of course, also need enhancements needed to build sensibly on current systems. On Windows, for instance, it would be built with a current compiler. There are still details to be filled in or altered, but this is my general idea. -- Terry Jan Reedy From ncoghlan at gmail.com Sun Mar 23 08:07:24 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Mar 2014 17:07:24 +1000 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 Message-ID: Several significant changes in this revision: - scope narrowed to just Python 2.7 plus permission for commercial redistributors to use the same strategy in their long term support releases - far more explicit that this is about inviting potential corporate contributors to address the situation for the benefit of the overall Python ecosystem, not offering to fix it for them for free - clarified that third party integration testing services would need to be updated to support testing against multiple Python 2.7 minor releases - explicit sections on why I don't think the status quo is sustainable, why I don't think Python 2.8 would actually solve the problem, and why I think a PyPI based solution not only wouldn't solve the problem, but would be rather difficult to get working in the first place - be completely explicit that I am *not* speaking on behalf of Red Hat at this point and have no authority to make commitments on their behalf. Instead, I'm looking for upstream consensus that 1) this is a genuine problem that needs to be solved; 2) we're open to corporate assistance in solving it; and 3) we have a pretty good idea what help we actually want. If all that happens, *then* I can take up the issue internally to try to get us some help in maintaining the proposed solution (hopefully other folks with corporate influence can do the same, and we may actually get some ongoing assistance with upstream maintenance out of this, rather than having our downstream redistributors continue to take us for granted). Diff: http://hg.python.org/peps/rev/2e82209dda21 Updated web version: http://www.python.org/dev/peps/pep-0466/ Advance warning: while I was able to get this revision turned around pretty quickly, future revisions are likely to take a fair bit longer. It was already a rather busy month before I decided to start this discussion on top of everything else :) Cheers, Nick. PEP: 466 Title: Network Security Enhancement Exception for Python 2.7 Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan , Status: Draft Type: Informational Content-Type: text/x-rst Created: 23-Mar-2014 Post-History: 23-Mar-2014 Abstract ======== Most CPython tracker issues are classified as errors in behaviour or proposed enhancements. Most patches to fix behavioural errors are applied to all active maintenance branches. Enhancement patches are restricted to the default branch that becomes the next Python version. This cadence works reasonably well during Python's normal 18-24 month feature release cycle, which is still applicable to the Python 3 series. However, the age of the standard library in Python 2 has now reached a point where it is sufficiently far behind the state of the art in network security protocols for it to be causing real problems in commercial use cases where upgrading to Python 3 in the near term may not be practical. Accordingly, this PEP relaxes the normal restrictions by allowing enhancements to be applied in Python 2.7 maintenance releases for standard library components that have implications for the overall security of the internet. In particular, the exception will apply to: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the ``sha`` module (Python 2 only) * the components of other networking modules that make use of these modules * the components of the ``random`` and ``os`` modules that are relevant to cryptographic applications * the version of OpenSSL bundled with the binary installers Proposed backports for these modules will still need to undergo normal backwards compatibility assessments, but new features will be permitted where appropriate, making it easier to implement secure networked software in Python, even for software that needs to remain compatible with older feature releases of Python. While this PEP does not make any changes to the core development team's handling of security-fix-only branches that are no longer in active maintenance, it *does* recommend that commercial redistributors providing extended support periods for the Python standard library either adopt a similar approach to ensuring that the secure networking infrastructure keeps pace with the evolution of the internet, or else disclaim support for the use of older versions in roles that involving connecting directly to the public internet. Exemption Policy ================ Under this policy, the following network security related modules are granted a blanket exemption to the restriction against adding new features in maintenance releases, for the purpose of keeping their APIs aligned with their counterparts in the latest feature release of Python 3: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the ``sha`` module (Python 2 only) This exemption applies to *all* proposals to backport backwards compatible changes in these modules to Python 2.7 maintenance releases. This choice is made deliberately to ensure that the "feature or fix?" argument isn't simply replaced by a "security related or not?" argument. These particular modules are inherently security related, and all enhancements to them improve Python's capabilities as a platform for development of secure networked software. As part of this policy, permission is also granted to upgrade to newer feature releases of OpenSSL when preparing the binary installers for new maintenance releases of CPython. In addition to the above blanket exemption, a conditional exemption is granted for these modules that may include some network security related features: * the ``os`` module (specifically ``os.urandom``) * the ``random`` module * networking related modules that depend on one or more of the network security related modules listed above This more limited exemption for these modules requires that the *specific* enhancement being proposed for backporting needs to be justified as being network security related. If the enhancement under discussion is designed to take advantage of a new feature in one of the network security related modules, then that will be taken as implying that the enhancement is security related. Backwards Compatibility Considerations ====================================== This PEP does *not* grant any general exemptions to the usual backwards compatibility policy for maintenance releases. Instead, by explicitly encouraging the use of feature based checks and explicitly opting in to less secure configurations, it is designed to make it easier to provide more "secure by default" behaviour in future feature releases, while still limiting the risk of breaking currently working software when upgrading to a new maintenance release. In *all* cases where this policy is applied to backport enhancements to maintenance releases, it MUST be possible to write cross-version compatible code that operates by "feature detection" (for example, checking for particular attributes in the module), without needing to explicitly check the Python version. It is then up to library and framework code to provide an appropriate warning and fallback behaviour if a desired feature is found to be missing. While some especially security sensitive software MAY fail outright if a desired security feature is unavailable, most software SHOULD instead continue operating using a slightly degraded security configuration. Affected APIs SHOULD be designed to allow library and application code to perform the following actions after detecting the presence of a relevant network security related feature: * explicitly opt in to more secure settings (to allow the use of enhanced security features in older maintenance releases of Python) * explicitly opt in to less secure settings (to allow the use of newer Python feature releases in lower security environments) * determine the default setting for the feature (this MAY require explicit Python version checks to determine the Python feature release, but MUST NOT require checking for a specific maintenance release) Security related changes to other modules (such as data format processing libraries) will continue to be made available as backports and new modules on the Python Package Index, as independent distribution remains the preferred approach to handling software that needs to evolve faster than the standard library. Refer to the `Motivation and Rationale`_ section for a review of the characteristics that make the secure networking infrastructure worthy of special consideration. Other Considerations ==================== Maintainability --------------- This policy does NOT represent a commitment by volunteer contributors to actually backport network security related changes from the Python 3 series to the Python 2 series. Rather, it is intended to send a clear signal to potential corporate contributors that the core development team are willing to review and merge corporate contributions that put this policy into effect. Backporting security related fixes and enhancements to earlier versions is a common service for commercial redistributors to offer to their customers. This policy represents an explicit invitation to contribute some of those changes back to the upstream community in cases where they are likely to have a broad impact that helps improve the security of the internet as a whole. Documentation ------------- All modules that take advantage of this policy to backport network security related enhancements to earlier Python versions MUST include a "Security Considerations" section in their documentation. In addition to any other module specific contents, this section MUST enumerate key security enhancements and fixes (with CVE identifiers where applicable), along with the feature and maintenance releases that first included them. Security releases ----------------- This PEP does not propose any changes to the handling of security releases - those will continue to be source only releases that include only critical security fixes. However, the recommendations for library and application developers are deliberately designed to accommodate commercial redistributors applying this policy to any Python release series that is either in security fix only mode, or has been declared "end of life" by the core development team. Whether or not redistributors choose to exercise that option will be up to the redistributor. Integration testing ------------------- Third party integration testing services would likely need to start offering users a choice of multiple Python 2.7.x versions to test against, to ensure that the application is correctly degrading gracefully if it attempts to use newer networking features on maintenance releases that are too old to provide them. Evolution of this Policy ======================== The key requirement for a module to be considered for inclusion in this policy (whether under a blanket or conditional exemption) is that it must have security implications *beyond* the specific application that is written in Python and the system that application is running on. Thus the focus on network security protocols and related cryptographic infrastructure - Python is a popular choice for the development of web services and clients, and thus the capabilities of widely used Python versions have implications for the security design of other services that may be using newer versions of Python or other development languages. The intent behind this requirement is to minimise any impact that the introduction of this policy may have on the stability and compatibility of maintenance releases. It would be thoroughly counterproductive if end users became as cautious about updating to new Python maintenance releases as they are about updating to new feature releases. Motivation and Rationale ======================== This PEP can be seen as a more targeted version of the "faster standard library release cycle" proposals discussed in PEP 407 and PEP 413, focusing specifically on those areas which have implications beyond the Python community. Background ---------- The creation of this PEP was prompted primarily by the aging SSL support in the Python 2 series. As of March 2014, the Python 2.7 SSL module is approaching four years of age, and the SSL support in the still popular Python 2.6 release had its feature set locked six years ago. These are simply too old to provide a foundation that can be recommended in good conscience for secure networking software that operates over the public internet, especially in an era where it is becoming quite clearly evident that advanced persistent security threats are even more widespread and more indiscriminate in their targeting than had previously been understood. While they represented reasonable security infrastructure in their time, the state of the art has moved on, and we need to investigate mechanisms for effectively providing more up to date network security infrastructure for users that, for whatever reason, are not currently in a position to migrate to Python 3. While the use of the system OpenSSL installation addresses many of these concerns on Linux platforms, it doesn't address all of them, and in the case of the binary installers for Windows and Mac OS X that are published on python.org, the version of OpenSSL used is entirely within the control of the Python core development team, and currently limited to OpenSSL maintenance releases for the version initially shipped with the corresponding Python feature release. With increased popularity comes increased responsibility, and this policy aims to acknowledge the fact that Python's popularity and adoption has now reached a level where some of our design and policy decisions have significant implications beyond the Python development community. As one example, the Python 2 ``ssl`` module does not support the Server Name Identification standard. While it is possible to obtain SNI support by using the third party ``requests`` client library, actually doing so currently requires using not only ``requests`` and its embedded dependencies, but also half a dozen or more additional libraries. The lack of support in the Python 2 series thus serves as an impediment to making effective use of SNI on servers, as Python 2 clients will frequently fail to handle it correctly. Another more critical example is the lack of SSL hostname matching in the Python 2 standard library - it is currently necessary to rely on a third party library, such as ``requests`` or ``backports.ssl_match_hostname`` to obtain that functionality in Python 2. The Python 2 series also remains more vulnerable to remote timing attacks on security sensitive comparisons than the Python 3 series, as it lacks a standard library equivalent to the timing attack resistant ``hmac.compare_digest()`` function. While appropriate secure comparison functions can be implemented in third party extensions, may users don't even consider the problem and use ordinary equality comparisons instead - while a standard library solution doesn't automatically fix that problem, it *does* make the barrier to resolution much lower once the problem is pointed out. My position on the ongoing transition from Python 2 to Python 3 has long been that Python 2 remains a supported platform for the core development team, and that commercial support will remain available well after upstream maintenance ends. However, in the absence of this network security enhancement policy, that position is difficult to justify when it comes to software that operates over the public internet. Just as many developers consider it too difficult to develop truly secure modern networked software in C/C++ (largely due to the challenges associated with manual memory management), I anticipate that in the not too distant future, it will be considered too difficult to develop truly secure modern networked software using the Python 2 series (some developers would argue that we have already reached that point). Alternative: advise developers of networked software to migrate to Python 3 --------------------------------------------------------------------------- This alternative represents the status quo. Unfortunately, it has proven to be unworkable in practice, as the backwards compatibility implications mean that this is a non-trivial migration process for large applications and integration projects. Now that we're fully aware of the impact the limitations in Python 2 may be having on the evolution of internet security standards, I no longer believe that it is reasonable to expect platform and application developers to resolve all of the latent defects in an application's Unicode correctness solely in order to gain access to the network security enhancements available in Python 3. While (as far as I am aware) Ubuntu has successfully switched to Python 3.4 as its main Python interpreter for its 14.04 LTS release, Fedora still has a lot of work to do to migrate, and it will take a non-trivial amount of time to migrate the relevant infrastructure components. While Red Hat are also actively working to make it easier for users to use more recent versions of Python on our stable platforms, it's going to take time for those efforts to start having an impact on end users' choice of version, and those changes won't affect the core tools regardless. The OpenStack migration to Python 3 is also still in its infancy, and even though that's a project with an extensive and relatively robust automated test suite, it's large enough that it is going to take quite some time to migrate. And that's just three of the highest profile open source projects that make heavy use of Python. Given the likely existence of large amounts of legacy code that lacks the kind of automated regression test suite needed to help support a migration from Python 2 to Python 3. The key point of this PEP is that those situations affect more people than just the developers and users of the affected application: their existence becomes something that developers of secure networked services need to take into account as part of their security design. As Terry Reedy noted, if we try to persist with the status quo, the likely outcome is that commercial redistributors will attempt to do something like this on behalf of their customers *anyway*, but in a potentially inconsistent and ad hoc manner. By drawing the scope definition process into the upstream project we are in a better position to influence the approach taken to address the situation and to help ensure some consistency across redistributors. The problem is real, so *something* needs to change, and this PEP describes my currently preferred approach to addressing the situation. Alternative: create and release Python 2.8 ------------------------------------------ With sufficient corporate support, it likely *would* be possible to create and release Python 2.8 (it's highly unlikely such a project would garner enough interest to be achievable with only volunteers). However, this wouldn't actually solve the problem, as the aim is to provide a *relatively low impact* way to incorporate enhanced security features into integrated products and deployments that make use of Python 2. Upgrading to a new Python feature release would mean both more work for the core development team, as well as a more disruptive update that most potential end users would likely just skip entirely. Attempting to create a Python 2.8 release would also bring in suggestions to backport many additional features from Python 3 (such as ``tracemalloc`` and the improved coroutine support). This is not a recommended approach, as it would involve substantial additional work for a result that is actually less effective as a solution to the original problem (the widespread use of the aging network security infrastructure in Python 2). Alternative: distribute the security enhancements via PyPI ---------------------------------------------------------- While it initially appears to be an attractive and easier to manage approach, there are actually several significant problems with this idea. Firstly, this PEP encompasses a non-trivial portion of the standard library. It's not just the underlying SSL support, but also the libraries for other network protocols like HTTP, FTP, IMAP, and POP3 that integrate with the SSL infrastructure to provide secure links, and that's just the protocols in the standard library. Even if an API compatible ``ssl2`` module was made available, it would need to be imported and injected into ``sys.modules`` as ``ssl`` before importing any other module that needed it. Secondly, this is complex, low level, cross-platform code that integrates with the underlying operating system across a variety of POSIX platforms (including Mac OS X) and Windows. The CPython BuildBot fleet is already set up to handle continuous integration in that context, but most of the freely available continuous integration services just offer Linux, and perhaps paid access to Windows. Those services work reasonably well for software that largely runs on the abstraction layers offered by Python and other dynamic languages, but won't suffice for the kind of code involved here. The OpenSSL dependency for the network security support also qualifies as the kind of "complex binary dependency" that isn't yet handled well by the ``pip`` based software distribution ecosystem. Relying on a binary dependency also creates potential compatibility problems for ``pip`` when running on other interpreters like ``PyPy``. Another practical problem with the idea is the fact that ``pip`` itself relies on the ``ssl`` support in the standard library (with some additional support from a bundled copy of ``requests``, which in turn bundles ``backport.ssl_match_hostname``), and hence would require any replacement module to also be bundled within ``pip``. This wouldn't pose any insurmountable difficulties (it's just another dependency to vendor), but it *would* mean yet another copy of OpenSSL to keep up to date. This approach also has the same flaw as all other "improve security by renaming things" approaches: they completely miss the users who most need help, and raise significant barriers against being able to encourage users to do the right thing when their infrastructure supports it (since "use this other module" is a much higher impact change than "turn on this higher security setting"). Deprecating the aging SSL infrastructure in the standard library in favour of an external module would be even more user hostile than taking the risk of trying to upgrade it in place. Last, but certainly not least, this approach suffers from the same problem as the idea of doing a Python 2.8 release: likely not solving the actual problem. Commercial redistributors of Python are set up to redistribute *Python*, and a pre-existing set of additional packages. Getting new packages added to the pre-existing set *can* be done, but means going around to each and every redistributor and asking them to update their repackaging process accordingly. By contrast, the approach described in this PEP would require redistributors to *opt out* of the security enhancements, which most of them are unlikely to do. Open Questions ============== * What are the risks associated with allowing OpenSSL to be updated to new feature versions in the Windows and Mac OS X binary installers for maintenance releases? Currently we just upgrade to the appropriate OpenSSL maintenance releases, rather than switching to the latest feature release. In particular, is it possible Windows C extensions may be linking against the Python provided OpenSSL module? * Are there any other security relevant modules that should be covered by either a blanket or conditional exemption? Disclosure of Interest ====================== The author of this PEP currently works for Red Hat on test automation tools. If this proposal is accepted, I will be strongly encouraging Red Hat to take advantage of the resulting opportunity to help improve the overall security of the Python ecosystem. However, I do not speak for Red Hat in this matter, and cannot make any commitments on Red Hat's behalf. Acknowledgements ================ Thanks to Christian Heimes for his recent efforts on greatly improving Python's SSL support in the Python 3 series, and a variety of members of the Python community for helping me to better understand the implications of the default settings we provide in our SSL modules, and the impact that tolerating the use of SSL infrastructure that was defined in 2010 (Python 2.7) or even 2008 (Python 2.6) potentially has for the security of the web as a whole. Christian and Donald Stufft also provided valuable feedback on a preliminary draft of this proposal. Thanks also to participants in the python-dev mailing list thread [1]_ References ========== .. [1] https://mail.python.org/pipermail/python-dev/2014-March/133334.html Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sun Mar 23 08:16:51 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Mar 2014 03:16:51 -0400 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: References: Message-ID: <27ED9ED5-A130-49EA-A515-0AC29FBABE0E@stufft.io> On Mar 23, 2014, at 3:07 AM, Nick Coghlan wrote: > Several significant changes in this revision: > > - scope narrowed to just Python 2.7 plus permission for commercial > redistributors to use the same strategy in their long term support > releases > - far more explicit that this is about inviting potential corporate > contributors to address the situation for the benefit of the overall > Python ecosystem, not offering to fix it for them for free > - clarified that third party integration testing services would need > to be updated to support testing against multiple Python 2.7 minor > releases For what it?s worth, I have an outstanding PR against Travis CI that would make this trivial for them at least. They are a pretty popular CI service especially for OSS projects. I made that PR for unrelated reasons but it could at least serve as a template for other projects to do the same thing. > - explicit sections on why I don't think the status quo is > sustainable, why I don't think Python 2.8 would actually solve the > problem, and why I think a PyPI based solution not only wouldn't solve > the problem, but would be rather difficult to get working in the first > place > - be completely explicit that I am *not* speaking on behalf of Red Hat > at this point and have no authority to make commitments on their > behalf. Instead, I'm looking for upstream consensus that 1) this is a > genuine problem that needs to be solved; 2) we're open to corporate > assistance in solving it; and 3) we have a pretty good idea what help > we actually want. If all that happens, *then* I can take up the issue > internally to try to get us some help in maintaining the proposed > solution (hopefully other folks with corporate influence can do the > same, and we may actually get some ongoing assistance with upstream > maintenance out of this, rather than having our downstream > redistributors continue to take us for granted). > > Diff: http://hg.python.org/peps/rev/2e82209dda21 > Updated web version: http://www.python.org/dev/peps/pep-0466/ > > Advance warning: while I was able to get this revision turned around > pretty quickly, future revisions are likely to take a fair bit longer. > It was already a rather busy month before I decided to start this > discussion on top of everything else :) > > Cheers, > Nick. > > > PEP: 466 > Title: Network Security Enhancement Exception for Python 2.7 > Version: $Revision$ > Last-Modified: $Date$ > Author: Nick Coghlan , > Status: Draft > Type: Informational > Content-Type: text/x-rst > Created: 23-Mar-2014 > Post-History: 23-Mar-2014 > > > Abstract > ======== > > Most CPython tracker issues are classified as errors in behaviour or > proposed enhancements. Most patches to fix behavioural errors are > applied to all active maintenance branches. Enhancement patches are > restricted to the default branch that becomes the next Python version. > > This cadence works reasonably well during Python's normal 18-24 month > feature release cycle, which is still applicable to the Python 3 series. > However, the age of the standard library in Python 2 has now reached a point > where it is sufficiently far behind the state of the art in network security > protocols for it to be causing real problems in commercial use cases > where upgrading to Python 3 in the near term may not be practical. > > Accordingly, this PEP relaxes the normal restrictions by allowing > enhancements to be applied in Python 2.7 maintenance releases for standard > library components that have implications for the overall security of the > internet. In particular, the exception will apply to: > > * the ``ssl`` module > * the ``hashlib`` module > * the ``hmac`` module > * the ``sha`` module (Python 2 only) > * the components of other networking modules that make use of these modules > * the components of the ``random`` and ``os`` modules that are relevant to > cryptographic applications > * the version of OpenSSL bundled with the binary installers > > Proposed backports for these modules will still need to undergo normal > backwards compatibility assessments, but new features will be permitted where > appropriate, making it easier to implement secure networked software in > Python, even for software that needs to remain compatible with older feature > releases of Python. > > While this PEP does not make any changes to the core development team's > handling of security-fix-only branches that are no longer in active > maintenance, it *does* recommend that commercial redistributors providing > extended support periods for the Python standard library either adopt a > similar approach to ensuring that the secure networking infrastructure > keeps pace with the evolution of the internet, or else disclaim support > for the use of older versions in roles that involving connecting > directly to the public internet. > > > Exemption Policy > ================ > > Under this policy, the following network security related modules are > granted a blanket exemption to the restriction against adding new features > in maintenance releases, for the purpose of keeping their APIs aligned with > their counterparts in the latest feature release of Python 3: > > * the ``ssl`` module > * the ``hashlib`` module > * the ``hmac`` module > * the ``sha`` module (Python 2 only) > > This exemption applies to *all* proposals to backport backwards compatible > changes in these modules to Python 2.7 maintenance releases. This choice is > made deliberately to ensure that the "feature or fix?" argument isn't simply > replaced by a "security related or not?" argument. These particular modules > are inherently security related, and all enhancements to them improve > Python's capabilities as a platform for development of secure networked > software. > > As part of this policy, permission is also granted to upgrade to newer > feature releases of OpenSSL when preparing the binary installers > for new maintenance releases of CPython. > > In addition to the above blanket exemption, a conditional exemption is > granted for these modules that may include some network security related > features: > > * the ``os`` module (specifically ``os.urandom``) > * the ``random`` module > * networking related modules that depend on one or more of the network > security related modules listed above > > This more limited exemption for these modules requires that the *specific* > enhancement being proposed for backporting needs to be justified as being > network security related. If the enhancement under discussion is designed > to take advantage of a new feature in one of the network security related > modules, then that will be taken as implying that the enhancement is > security related. > > > Backwards Compatibility Considerations > ====================================== > > This PEP does *not* grant any general exemptions to the usual backwards > compatibility policy for maintenance releases. Instead, by explicitly > encouraging the use of feature based checks and explicitly opting in to > less secure configurations, it is designed to make it easier to provide > more "secure by default" behaviour in future feature releases, while still > limiting the risk of breaking currently working software when upgrading to > a new maintenance release. > > In *all* cases where this policy is applied to backport enhancements to > maintenance releases, it MUST be possible to write cross-version compatible > code that operates by "feature detection" (for example, checking for > particular attributes in the module), without needing to explicitly check > the Python version. > > It is then up to library and framework code to provide an appropriate warning > and fallback behaviour if a desired feature is found to be missing. While > some especially security sensitive software MAY fail outright if a desired > security feature is unavailable, most software SHOULD instead continue > operating using a slightly degraded security configuration. > > Affected APIs SHOULD be designed to allow library and application code to > perform the following actions after detecting the presence of a relevant > network security related feature: > > * explicitly opt in to more secure settings (to allow the use of enhanced > security features in older maintenance releases of Python) > * explicitly opt in to less secure settings (to allow the use of newer Python > feature releases in lower security environments) > * determine the default setting for the feature (this MAY require explicit > Python version checks to determine the Python feature release, but MUST > NOT require checking for a specific maintenance release) > > Security related changes to other modules (such as data format processing > libraries) will continue to be made available as backports and new modules > on the Python Package Index, as independent distribution remains the > preferred approach to handling software that needs to evolve faster than > the standard library. Refer to the `Motivation and Rationale`_ section for > a review of the characteristics that make the secure networking > infrastructure worthy of special consideration. > > > Other Considerations > ==================== > > Maintainability > --------------- > > This policy does NOT represent a commitment by volunteer contributors to > actually backport network security related changes from the Python 3 series > to the Python 2 series. Rather, it is intended to send a clear signal to > potential corporate contributors that the core development team are willing > to review and merge corporate contributions that put this policy into > effect. > > Backporting security related fixes and enhancements to earlier versions is > a common service for commercial redistributors to offer to their customers. > This policy represents an explicit invitation to contribute some of those > changes back to the upstream community in cases where they are likely to > have a broad impact that helps improve the security of the internet as a > whole. > > > Documentation > ------------- > > All modules that take advantage of this policy to backport network > security related enhancements to earlier Python versions MUST include > a "Security Considerations" section in their documentation. > > In addition to any other module specific contents, this section MUST > enumerate key security enhancements and fixes (with CVE identifiers where > applicable), along with the feature and maintenance releases that first > included them. > > > Security releases > ----------------- > > This PEP does not propose any changes to the handling of security > releases - those will continue to be source only releases that > include only critical security fixes. > > However, the recommendations for library and application developers are > deliberately designed to accommodate commercial redistributors applying > this policy to any Python release series that is either in security > fix only mode, or has been declared "end of life" by the core development > team. > > Whether or not redistributors choose to exercise that option will be up > to the redistributor. > > > Integration testing > ------------------- > > Third party integration testing services would likely need to start > offering users a choice of multiple Python 2.7.x versions to test against, > to ensure that the application is correctly degrading gracefully if it > attempts to use newer networking features on maintenance releases that > are too old to provide them. > > > Evolution of this Policy > ======================== > > The key requirement for a module to be considered for inclusion in this > policy (whether under a blanket or conditional exemption) is that it must > have security implications *beyond* the specific application that is written > in Python and the system that application is running on. Thus the focus on > network security protocols and related cryptographic infrastructure - Python > is a popular choice for the development of web services and clients, and > thus the capabilities of widely used Python versions have implications for > the security design of other services that may be using newer versions of > Python or other development languages. > > The intent behind this requirement is to minimise any impact that the > introduction of this policy may have on the stability and compatibility of > maintenance releases. It would be thoroughly counterproductive if end > users became as cautious about updating to new Python maintenance releases > as they are about updating to new feature releases. > > > Motivation and Rationale > ======================== > > This PEP can be seen as a more targeted version of the "faster standard > library release cycle" proposals discussed in PEP 407 and PEP 413, > focusing specifically on those areas which have implications beyond the > Python community. > > > Background > ---------- > > The creation of this PEP was prompted primarily by the aging SSL support in > the Python 2 series. As of March 2014, the Python 2.7 SSL module is > approaching four years of age, and the SSL support in the still popular > Python 2.6 release had its feature set locked six years ago. > > These are simply too old to provide a foundation that can be recommended > in good conscience for secure networking software that operates over the > public internet, especially in an era where it is becoming quite clearly > evident that advanced persistent security threats are even more widespread > and more indiscriminate in their targeting than had previously been > understood. While they represented reasonable security infrastructure in > their time, the state of the art has moved on, and we need to investigate > mechanisms for effectively providing more up to date network security > infrastructure for users that, for whatever reason, are not currently in > a position to migrate to Python 3. > > While the use of the system OpenSSL installation addresses many of these > concerns on Linux platforms, it doesn't address all of them, and in the > case of the binary installers for Windows and Mac OS X that are published > on python.org, the version of OpenSSL used is entirely within the control > of the Python core development team, and currently limited to OpenSSL > maintenance releases for the version initially shipped with the corresponding > Python feature release. > > With increased popularity comes increased responsibility, and this policy > aims to acknowledge the fact that Python's popularity and adoption has now > reached a level where some of our design and policy decisions have > significant implications beyond the Python development community. > > As one example, the Python 2 ``ssl`` module does not support the Server > Name Identification standard. While it is possible to obtain SNI support > by using the third party ``requests`` client library, actually doing so > currently requires using not only ``requests`` and its embedded dependencies, > but also half a dozen or more additional libraries. The lack of support > in the Python 2 series thus serves as an impediment to making effective > use of SNI on servers, as Python 2 clients will frequently fail to handle > it correctly. > > Another more critical example is the lack of SSL hostname matching in the > Python 2 standard library - it is currently necessary to rely on a third > party library, such as ``requests`` or ``backports.ssl_match_hostname`` to > obtain that functionality in Python 2. > > The Python 2 series also remains more vulnerable to remote timing attacks > on security sensitive comparisons than the Python 3 series, as it lacks a > standard library equivalent to the timing attack resistant > ``hmac.compare_digest()`` function. While appropriate secure comparison > functions can be implemented in third party extensions, may users don't > even consider the problem and use ordinary equality comparisons instead > - while a standard library solution doesn't automatically fix that problem, > it *does* make the barrier to resolution much lower once the problem is > pointed out. > > My position on the ongoing transition from Python 2 to Python 3 has long > been that Python 2 remains a supported platform for the core development > team, and that commercial support will remain available well after upstream > maintenance ends. However, in the absence of this network security > enhancement policy, that position is difficult to justify when it comes to > software that operates over the public internet. Just as many developers > consider it too difficult to develop truly secure modern networked software > in C/C++ (largely due to the challenges associated with manual > memory management), I anticipate that in the not too distant future, it > will be considered too difficult to develop truly secure modern networked > software using the Python 2 series (some developers would argue that we > have already reached that point). > > > Alternative: advise developers of networked software to migrate to Python 3 > --------------------------------------------------------------------------- > > This alternative represents the status quo. Unfortunately, it has proven > to be unworkable in practice, as the backwards compatibility implications > mean that this is a non-trivial migration process for large applications > and integration projects. > > Now that we're fully aware of the impact the limitations in Python 2 may be > having on the evolution of internet security standards, I no longer believe > that it is reasonable to expect platform and application developers to > resolve all of the latent defects in an application's Unicode correctness > solely in order to gain access to the network security enhancements > available in Python 3. > > While (as far as I am aware) Ubuntu has successfully switched to Python 3.4 > as its main Python interpreter for its 14.04 LTS release, Fedora still > has a lot of work to do to migrate, and it will take a non-trivial amount > of time to migrate the relevant infrastructure components. While Red Hat > are also actively working to make it easier for users to use more recent > versions of Python on our stable platforms, it's going to take time for > those efforts to start having an impact on end users' choice of version, > and those changes won't affect the core tools regardless. > > The OpenStack migration to Python 3 is also still in its infancy, and even > though that's a project with an extensive and relatively robust automated > test suite, it's large enough that it is going to take quite some time > to migrate. > > And that's just three of the highest profile open source projects that > make heavy use of Python. Given the likely existence of large amounts of > legacy code that lacks the kind of automated regression test suite needed > to help support a migration from Python 2 to Python 3. The key point of > this PEP is that those situations affect more people than just the > developers and users of the affected application: their existence becomes > something that developers of secure networked services need to take into > account as part of their security design. > > As Terry Reedy noted, if we try to persist with the status quo, the likely > outcome is that commercial redistributors will attempt to do something > like this on behalf of their customers *anyway*, but in a potentially > inconsistent and ad hoc manner. By drawing the scope definition process > into the upstream project we are in a better position to influence the > approach taken to address the situation and to help ensure some consistency > across redistributors. > > The problem is real, so *something* needs to change, and this PEP describes > my currently preferred approach to addressing the situation. > > > Alternative: create and release Python 2.8 > ------------------------------------------ > > With sufficient corporate support, it likely *would* be possible to create > and release Python 2.8 (it's highly unlikely such a project would garner > enough interest to be achievable with only volunteers). However, this > wouldn't actually solve the problem, as the aim is to provide a *relatively > low impact* way to incorporate enhanced security features into integrated > products and deployments that make use of Python 2. Upgrading to a new > Python feature release would mean both more work for the core development > team, as well as a more disruptive update that most potential end users > would likely just skip entirely. > > Attempting to create a Python 2.8 release would also bring in suggestions > to backport many additional features from Python 3 (such as ``tracemalloc`` > and the improved coroutine support). > > This is not a recommended approach, as it would involve substantial > additional work for a result that is actually less effective as a solution > to the original problem (the widespread use of the aging network security > infrastructure in Python 2). > > > Alternative: distribute the security enhancements via PyPI > ---------------------------------------------------------- > > While it initially appears to be an attractive and easier to manage > approach, there are actually several significant problems with this > idea. > > Firstly, this PEP encompasses a non-trivial portion of the standard library. > It's not just the underlying SSL support, but also the libraries for other > network protocols like HTTP, FTP, IMAP, and POP3 that integrate with the > SSL infrastructure to provide secure links, and that's just the protocols > in the standard library. Even if an API compatible ``ssl2`` module was > made available, it would need to be imported and injected > into ``sys.modules`` as ``ssl`` before importing any other module that > needed it. > > Secondly, this is complex, low level, cross-platform code that integrates > with the underlying operating system across a variety of POSIX platforms > (including Mac OS X) and Windows. The CPython BuildBot fleet is already set > up to handle continuous integration in that context, but most of the > freely available continuous integration services just offer Linux, and > perhaps paid access to Windows. Those services work reasonably well for > software that largely runs on the abstraction layers offered by Python and > other dynamic languages, but won't suffice for the kind of code involved > here. > > The OpenSSL dependency for the network security support also qualifies as > the kind of "complex binary dependency" that isn't yet handled well by the > ``pip`` based software distribution ecosystem. Relying on a binary > dependency also creates potential compatibility problems for ``pip`` when > running on other interpreters like ``PyPy``. > > Another practical problem with the idea is the fact that ``pip`` itself > relies on the ``ssl`` support in the standard library (with some additional > support from a bundled copy of ``requests``, which in turn bundles > ``backport.ssl_match_hostname``), and hence would require any replacement > module to also be bundled within ``pip``. This wouldn't pose any > insurmountable difficulties (it's just another dependency to vendor), but > it *would* mean yet another copy of OpenSSL to keep up to > date. > > This approach also has the same flaw as all other "improve security by > renaming things" approaches: they completely miss the users who most need > help, and raise significant barriers against being able to encourage users > to do the right thing when their infrastructure supports it (since > "use this other module" is a much higher impact change than "turn on this > higher security setting"). Deprecating the aging SSL infrastructure in the > standard library in favour of an external module would be even more user > hostile than taking the risk of trying to upgrade it in place. > > Last, but certainly not least, this approach suffers from the same problem > as the idea of doing a Python 2.8 release: likely not solving the actual > problem. Commercial redistributors of Python are set up to redistribute > *Python*, and a pre-existing set of additional packages. Getting new > packages added to the pre-existing set *can* be done, but means going > around to each and every redistributor and asking them to update their > repackaging process accordingly. By contrast, the approach described in > this PEP would require redistributors to *opt out* of the security > enhancements, which most of them are unlikely to do. > > > Open Questions > ============== > > * What are the risks associated with allowing OpenSSL to be updated to > new feature versions in the Windows and Mac OS X binary installers for > maintenance releases? Currently we just upgrade to the appropriate > OpenSSL maintenance releases, rather than switching to the latest > feature release. In particular, is it possible Windows C extensions may > be linking against the Python provided OpenSSL module? > > * Are there any other security relevant modules that should be covered > by either a blanket or conditional exemption? > > > Disclosure of Interest > ====================== > > The author of this PEP currently works for Red Hat on test automation tools. > If this proposal is accepted, I will be strongly encouraging Red Hat to take > advantage of the resulting opportunity to help improve the overall security > of the Python ecosystem. However, I do not speak for Red Hat in this matter, > and cannot make any commitments on Red Hat's behalf. > > > Acknowledgements > ================ > > Thanks to Christian Heimes for his recent efforts on greatly improving > Python's SSL support in the Python 3 series, and a variety of members of > the Python community for helping me to better understand the implications > of the default settings we provide in our SSL modules, and the impact that > tolerating the use of SSL infrastructure that was defined in 2010 > (Python 2.7) or even 2008 (Python 2.6) potentially has for the security > of the web as a whole. > > Christian and Donald Stufft also provided valuable feedback on a preliminary > draft of this proposal. > > Thanks also to participants in the python-dev mailing list thread [1]_ > > > References > ========== > > .. [1] https://mail.python.org/pipermail/python-dev/2014-March/133334.html > > > Copyright > ========= > > This document has been placed in the public domain. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rosuav at gmail.com Sun Mar 23 08:30:30 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 23 Mar 2014 18:30:30 +1100 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: References: Message-ID: On Sun, Mar 23, 2014 at 6:07 PM, Nick Coghlan wrote: > And that's just three of the highest profile open source projects that > make heavy use of Python. Given the likely existence of large amounts of > legacy code that lacks the kind of automated regression test suite needed > to help support a migration from Python 2 to Python 3. The key point of > this PEP is that those situations affect more people than just the > developers and users of the affected application: their existence becomes > something that developers of secure networked services need to take into > account as part of their security design. Grammatical point: The sentence beginning "Given..." doesn't seem complete. ChrisA From jurko.gospodnetic at pke.hr Sun Mar 23 09:02:16 2014 From: jurko.gospodnetic at pke.hr (=?windows-1250?Q?Jurko_Gospodneti=E6?=) Date: Sun, 23 Mar 2014 09:02:16 +0100 Subject: [Python-Dev] PEP 453 (Explicit bootstrapping of pip in Python installations) - slight typo Message-ID: Hi. Not really sure where to report this - missing closing parentheses in the PEP text at the end of the second paragraph in section 'Implementation strategy' http://legacy.python.org/dev/peps/pep-0453/#id35 > and would not try to contact PyPI (instead installing directly > from the private wheel files. Hope this helps. Best regards, Jurko Gospodneti? From martin at v.loewis.de Sun Mar 23 09:42:34 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 23 Mar 2014 09:42:34 +0100 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: References: Message-ID: <532E9E7A.7060405@v.loewis.de> Am 23.03.14 08:07, schrieb Nick Coghlan: > Several significant changes in this revision: > > - scope narrowed to just Python 2.7 plus permission for commercial > redistributors to use the same strategy in their long term support > releases Thanks; the rationale is now much clearer, and also indicates yet another alternative path: fork Python. The PEP indicates that vendors are likely to fork Python anyway (as they always did, in a small scale). This could become more official and coordinated. Create an "enhanced TLS" clone of cpython, and start applying changes to 2.6, 2.7, and any other branches you consider relevant. Keep it merged with the cpython code base. This model has worked for many years for Stackless Python. Then, vendors have the choice of either releasing from the official CPython repository, or from the enhanced TLS one. All reasoning on application-level feature testing still applies: applications would have to feature-test whether they run on python.org release, or on an enhanced release. For Windows in particular, it would be up to ActiveState to decide whether they build binaries from python.org, or from the enhanced TLS repo. They could actually opt to provide both, leaving the choice to users. Even if this notion of forking is not acceptable, I suggest that you could still start working on the code in a separate clone, and the decision on the PEP could be deferred until a proposed implementation is ready. I see it as a risk of the PEP that the implementation might not be available before May 2015, in which case the PEP would become irrelevant. Regards, Martin From christian at python.org Sun Mar 23 10:48:33 2014 From: christian at python.org (Christian Heimes) Date: Sun, 23 Mar 2014 10:48:33 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> Message-ID: On 23.03.2014 02:33, Brett Cannon wrote: > Now I have been reading this thread on my phone and I only have cursory > understanding of what failure ssl has had as of late, so this might be > stupid, but what if in Python 3.5 we made it so people passed in an > explicit SSL object into the relevant APIs and threw an exception if one > wasn't provided when trying to access something requiring SSL? Then the > ssl module would start existing on PyPI to make sure it can evolve > faster than every 18 months? I want to deprecate all SSL-related arguments except for the context argument. Several modules still except key and cert parameters. Do you want to make the context parameter a required parameter? From victor.stinner at gmail.com Sun Mar 23 10:48:57 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 23 Mar 2014 10:48:57 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: Message-ID: Hi, 2014-03-22 22:11 GMT+01:00 Nick Coghlan : > In particular, the exception will apply to: > > * the ``ssl`` module > * the ``hashlib`` module > * the ``hmac`` module > * the ``sha`` module (Python 2 only) > * the components of other networking modules that make use of these modules > * the components of the ``random`` and ``os`` modules that are relevant to > cryptographic applications > * the version of OpenSSL bundled with the binary installers The current Python development cycle is too slow to track latest security recommandations. Python 2.7 is now very far from Python 3.4, so backportng so many features is very complex and error-prone. IMO A third-party module backporting all these security enhancements would be easy to develop. There are already M2Crypto, cryptography and PyOpenSSL modules for example. You may reuse them. The drawback is that applications would be benefit immediatly from this work, they should be modified to use the new module. But usually, developers who care of security are able to do these modifications. The PEP is very generic. Can you try to list all missing security features of old Python versions? Victor From martin at v.loewis.de Sun Mar 23 11:17:07 2014 From: martin at v.loewis.de (martin at v.loewis.de) Date: Sun, 23 Mar 2014 11:17:07 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: Message-ID: <20140323111707.Horde.gUJfhKxhwldwV6hsKnBncA1@webmail.df.eu> Quoting Victor Stinner : > The drawback is that applications would be benefit immediatly from > this work, they should be modified to use the new module. But usually, > developers who care of security are able to do these modifications. I think asking developers to make significant modifications to their code is besides the point of the PEP. However, if they are willing to make changes, I'd still recommend that they port their code to Python 3, as that is the better long-term investment. Regards, Martin From victor.stinner at gmail.com Sun Mar 23 11:31:12 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 23 Mar 2014 11:31:12 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <20140323111707.Horde.gUJfhKxhwldwV6hsKnBncA1@webmail.df.eu> References: <20140323111707.Horde.gUJfhKxhwldwV6hsKnBncA1@webmail.df.eu> Message-ID: Hi, 2014-03-23 11:17 GMT+01:00 : > Quoting Victor Stinner : >> The drawback is that applications would be benefit immediatly from >> this work, they should be modified to use the new module. But usually, >> developers who care of security are able to do these modifications. > > I think asking developers to make significant modifications to their > code is besides the point of the PEP. I expect something like replacing "import ssl" with "from safestdlib import ssl", same for other modules. FYI I backported some Python 3.3 modules for Python 2.7 for my Trollius project, and it was not so hard. For ssl.SSLContext, it's an empty class, it's just a wrapper to ssl.wrap_socket() which raises errors when unsupported features are used. http://trollius.readthedocs.org/#backports I wrote Trollius to port OpenStack to Python 3, not to defer the port :-) > However, if they are willing > to make changes, I'd still recommend that they port their code to > Python 3, as that is the better long-term investment. It's not always possible. For example, the OpenStack projet still have +30 dependencies which are not Python 3 compatible yet. https://wiki.openstack.org/wiki/Python3#Dependencies We are porting these dependencies, but it may still take one year to port the whole OpenStack project to Python 3. Come to help us :) Sorry, it's maybe not fair to take the worst example (OpenStack) :-) Victor From ncoghlan at gmail.com Sun Mar 23 13:07:35 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Mar 2014 22:07:35 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <20140323111707.Horde.gUJfhKxhwldwV6hsKnBncA1@webmail.df.eu> Message-ID: On 23 Mar 2014 20:33, "Victor Stinner" wrote: > > Sorry, it's maybe not fair to take the worst example (OpenStack) :-) I suspect the Fedora/RHEL/CentOS ecosystem is going to make OpenStack look like a relatively simple port, and backwards compatibility constraints mean that already defined RHEL/CentOS releases simply won't be forward ported at all. I confess I hadn't fully connected the relevant dots until after writing the PEP this weekend, so I think I'm going to be having some interesting conversations with colleagues about that over the next couple of weeks and at PyCon :) Cheers, Nick. > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Mar 23 13:16:00 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Mar 2014 22:16:00 +1000 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: <532E9E7A.7060405@v.loewis.de> References: <532E9E7A.7060405@v.loewis.de> Message-ID: On 23 Mar 2014 18:42, Martin v. L?wis wrote: > > Am 23.03.14 08:07, schrieb Nick Coghlan: > > Several significant changes in this revision: > > > > - scope narrowed to just Python 2.7 plus permission for commercial > > redistributors to use the same strategy in their long term support > > releases > > Thanks; the rationale is now much clearer, and also indicates yet > another alternative path: fork Python. > > The PEP indicates that vendors are likely to fork Python anyway > (as they always did, in a small scale). This could become more official > and coordinated. Create an "enhanced TLS" clone of cpython, and start > applying changes to 2.6, 2.7, and any other branches you consider > relevant. Keep it merged with the cpython code base. This model > has worked for many years for Stackless Python. > > Then, vendors have the choice of either releasing from the official > CPython repository, or from the enhanced TLS one. All reasoning on > application-level feature testing still applies: applications would > have to feature-test whether they run on python.org release, or > on an enhanced release. > > For Windows in particular, it would be up to ActiveState to decide > whether they build binaries from python.org, or from the enhanced > TLS repo. They could actually opt to provide both, leaving the choice > to users. > > Even if this notion of forking is not acceptable, I suggest > that you could still start working on the code in a separate clone, > and the decision on the PEP could be deferred until a proposed > implementation is ready. I see it as a risk of the PEP that the > implementation might not be available before May 2015, in which > case the PEP would become irrelevant. Yes, a 2.7-enhanced-tls clone is definitely a good notion. Your suggestion to recast the entire PEP along those lines, so we always retain the option of choosing between them, also sounds plausible. Cheers, Nick. > > Regards, > Martin > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Sun Mar 23 14:00:02 2014 From: skip at pobox.com (Skip Montanaro) Date: Sun, 23 Mar 2014 08:00:02 -0500 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <532E63A8.9020705@udel.edu> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> Message-ID: On Sat, Mar 22, 2014 at 11:31 PM, Terry Reedy wrote: > The download page for the final 2.7.z maintenance release could say > something like "We recommend that you move to the most recent Python 3 > version if at all possible. If you cannot do that and you want to use Python > to run a server on the public internet, we urge you to instead use the > latest version of ServerPython 2.7.1s. This series is based on Python 2.7.z > but has been and will continue to be enhanced with security features > backported from Python 3." I'm unclear how this would be better than just biting the bullet and making a 2.8 release. On the one hand, the 2.7.x number suggests (based on the existing release protocol) that it should be a drop-in replacement for earlier 2.7 micro releases. On the other hand, calling it something like "ServerPython" implies that it's not necessary for network client applications, when, if I read the PEP correctly, it most certainly would be. If you create a 2.8 release which is restricted to just the topic areas of the PEP (that is, no other stuff backported from 3.x, no requirement to add other non-security bug fixes, etc), the incremented minor version number tells people that a bit of extra care is required to upgrade. The lack of change in the code base outside the security apparatus should make update pretty trivial for most every non-networked application. If the PEP or something like it is approved, the work is still going to have to be done, no matter what you call it. Why not be transparent about it? Skip From solipsis at pitrou.net Sun Mar 23 14:13:44 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 Mar 2014 14:13:44 +0100 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 References: Message-ID: <20140323141344.1b3d078a@fsol> On Sun, 23 Mar 2014 17:07:24 +1000 Nick Coghlan wrote: > Another more critical example is the lack of SSL hostname matching in the > Python 2 standard library - it is currently necessary to rely on a third > party library, such as ``requests`` or ``backports.ssl_match_hostname`` to > obtain that functionality in Python 2. Do note that match_hostname() is a pure Python function and is easy to paste into your own code (if you don't want to pull in a dependency). It doesn't need SSLContext or any other recent stuff, just a certificate dict which Python 2.x is already able to provide (SSLSocket.getpeercert()). > Firstly, this PEP encompasses a non-trivial portion of the standard library. > It's not just the underlying SSL support, but also the libraries for other > network protocols like HTTP, FTP, IMAP, and POP3 that integrate with the > SSL infrastructure to provide secure links, and that's just the protocols > in the standard library. It's still not obvious what you are proposing to do with these other libraries. If you are proposing to validate certs against system CAs and check hostnames by default - you are going to break compatibility for a lot of current uses. As Martin I think it would be easier to reason about a concrete backport proposal. Regards Antoine. From p.f.moore at gmail.com Sun Mar 23 14:23:04 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 23 Mar 2014 13:23:04 +0000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: Message-ID: On 22 March 2014 21:11, Nick Coghlan wrote: > Full PEP included inline below, and available in more readable form at > http://www.python.org/dev/peps/pep-0466/ Disclaimer: I pretty much don't use 2.x, or write server-type software, so my practical requirement for the changes proposed here is negligible. Having thought about the points in the discussion so far, it seems to me: 1. The only real issue is 2.7, so the PEP should limit itself to "how do we deal with 2.7". 2. The PEP talks vaguely about commercial support for the security changes. I'm not quite sure what's being said here - I suspect Nick has some concrete input from Red Hat here, but he's trying to keep from being too specific in the PEP. I don't think that's necessarily helpful. If the proposal is "Companies who want backported security features can provide patches and this is how python-dev will evaluate them" then let's say that. At the moment, there's a feel in the PEP that python-dev are proposing to do the backports, and I'm not sure that's true. As written, the PEP risks exposing python-dev to accusations of not having delivered the improved security that the PEP promises. 3. If companies want specific security enhancements, there's nothing to stop them developing a patch, writing a PEP and proposing it to python-dev. The first such patch would trigger pretty much the current debate (is it OK to add an enhancement to 2.7, because it is for security reasons?) but with the benefit of being about something specific. Further proposals along the same lines could reference the first PEP as justification for allowing subsequent security enhancements. 4. A process like that described in (3) could happen right now. Why isn't it? Presumably the need for such enhancements is there (or there'd be no need for this PEP) so is it simply that companies like Red Hat don't know that they have this option? Would it be better simply to publish something explaining "How commercial vendors can work with python-dev to deliver long-term support of Python 2.7".? 5. If the driver for this is Linux vendor support for 2.7, who is going to provide the Windows/OSX/Solaris/etc implementations of new features? Is "nobody" (implying that new features could end up being Linux-only) an acceptable answer? Will the burden of adding cross-platform support be on python-dev? Will patches be rejected unless they cover all supported platforms (and will this be an unacceptable burden for companies like Red Hat to take on)? The situation with Python 2.7 has been understood for many years now. Individual end users may not have thought through the implications and we have a responsibility to support them. But companies like Red Hat[1] are fully aware of their own release cycles, and the position with Python 2.x is public knowledge. They have to take some share of the blame for not planning for this situation (and they have paying customers who expect them to handle this, unlike python-dev). One problem here is that it's not at all clear what counts as "dealing with a commercial vendor" means in the context of enhancements (security or otherwise) to Python. We already have a route that's open to anyone, so what else is needed? And why? Most of the above is directed at the "commercial vendors can help out" aspect of the PEP. With regard to the wider suggestion that security enhancements be allowed for 2.7, we've been debating similar questions in various forms for *ages*. There's been talk of a 2.8 release. There have been ongoing discussions about helping people move to Python 3. There have been many discussions about whether it's OK for applications to remain on Python 2.7 (all of which said "yes, it's fine" - maybe the security fix aspect makes that statement a little less clear-cut). The fact that we haven't felt the need thus far to change policy on 2.7 says that ("commercial vendors need it" aspects aside) this is just another cycle of that same debate. Paul [1] I seem to be picking on Red Hat here. I'm not - it's just that the RHEL support cycle is the most-often quoted example of commercial support for Python 2.x, so I'm using that as my example. From p.f.moore at gmail.com Sun Mar 23 14:26:27 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 23 Mar 2014 13:26:27 +0000 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: References: Message-ID: On 23 March 2014 07:07, Nick Coghlan wrote: > Advance warning: while I was able to get this revision turned around > pretty quickly, future revisions are likely to take a fair bit longer. > It was already a rather busy month before I decided to start this > discussion on top of everything else :) Ha - you produced this update while I was still thinking about the first draft, and it arrived (unnoticed) while I was writing my response. Sorry if some or all of my points in the other email are now irrelevant as a result. I'll try to read and assess this version before responding - so feel free to take longer over the next revision:-) Paul From cory at lukasa.co.uk Sun Mar 23 08:29:07 2014 From: cory at lukasa.co.uk (Cory Benfield) Date: Sun, 23 Mar 2014 07:29:07 +0000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <532E63A8.9020705@udel.edu> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> Message-ID: On 23 March 2014 at 04:32:17, Terry Reedy (tjreedy at udel.edu(mailto:tjreedy at udel.edu)) wrote: > Instead, I think the PEP should propose a special series of server > enhancement releases that are based on the final 2.7 maintenance release > (2.7.8 or 2.7.9) but which have have a different application-specific > enhancement policy. This is an interesting idea. My biggest problem with it is that, at least with the ssl library, these aren?t server-only problems. If we suggest that they are, we end up in the same position we?re in right now (that is, hurting the internet). For example, Python 2.7?s ssl module lacks the OP_NO_COMPRESSION option for OpenSSL, meaning that the application is at the mercy of the server to determine whether it?s vulnerable to the CRIME attack. Given that all modern browsers already disable TLS compression, we can assume that lots of server admins haven?t bothered disabling it on their end. This leaves pretty much anyone using HTTPS, client or server, on Python 2.7 at risk of the CRIME attack. This isn?t a server-only problem, so I feel like limiting the fixes to a ?server? release is not good enough. From solipsis at pitrou.net Sun Mar 23 16:34:40 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 Mar 2014 16:34:40 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> Message-ID: <20140323163440.712f98be@fsol> On Sun, 23 Mar 2014 07:29:07 +0000 Cory Benfield wrote: > On 23 March 2014 at 04:32:17, Terry Reedy (tjreedy at udel.edu(mailto:tjreedy at udel.edu)) wrote: > > Instead, I think the PEP should propose a special series of server > > enhancement releases that are based on the final 2.7 maintenance release > > (2.7.8 or 2.7.9) but which have have a different application-specific > > enhancement policy. > > This is an interesting idea. My biggest problem with it is that, at least > with the ssl library, these aren?t server-only problems. If we suggest that > they are, we end up in the same position we?re in right now (that is, hurting > the internet). > > For example, Python 2.7?s ssl module lacks the OP_NO_COMPRESSION option for > OpenSSL, This is easy to change in a bugfix release, though. Someone just has to open an issue and write a patch. Regards Antoine. From donald at stufft.io Sun Mar 23 16:37:25 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Mar 2014 11:37:25 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <20140323163440.712f98be@fsol> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323163440.712f98be@fsol> Message-ID: <21535320-998D-4DDD-9EB2-31A30AEB72E5@stufft.io> On Mar 23, 2014, at 11:34 AM, Antoine Pitrou wrote: > On Sun, 23 Mar 2014 07:29:07 +0000 > Cory Benfield wrote: >> On 23 March 2014 at 04:32:17, Terry Reedy (tjreedy at udel.edu(mailto:tjreedy at udel.edu)) wrote: >>> Instead, I think the PEP should propose a special series of server >>> enhancement releases that are based on the final 2.7 maintenance release >>> (2.7.8 or 2.7.9) but which have have a different application-specific >>> enhancement policy. >> >> This is an interesting idea. My biggest problem with it is that, at least >> with the ssl library, these aren?t server-only problems. If we suggest that >> they are, we end up in the same position we?re in right now (that is, hurting >> the internet). >> >> For example, Python 2.7?s ssl module lacks the OP_NO_COMPRESSION option for >> OpenSSL, > > This is easy to change in a bugfix release, though. Someone just has to > open an issue and write a patch. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io I already did open an issue and write a patch :) There?s someone on that issue saying that flipping that without a way to flip it back would break their application. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From eric at trueblade.com Sun Mar 23 16:40:57 2014 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 23 Mar 2014 11:40:57 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <21535320-998D-4DDD-9EB2-31A30AEB72E5@stufft.io> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323163440.712f98be@fsol> <21535320-998D-4DDD-9EB2-31A30AEB72E5@stufft.io> Message-ID: <532F0089.3030009@trueblade.com> On 3/23/2014 11:37 AM, Donald Stufft wrote: > > On Mar 23, 2014, at 11:34 AM, Antoine Pitrou wrote: > >> On Sun, 23 Mar 2014 07:29:07 +0000 >> Cory Benfield wrote: >>> This is an interesting idea. My biggest problem with it is that, at least >>> with the ssl library, these aren?t server-only problems. If we suggest that >>> they are, we end up in the same position we?re in right now (that is, hurting >>> the internet). >>> >>> For example, Python 2.7?s ssl module lacks the OP_NO_COMPRESSION option for >>> OpenSSL, >> >> This is easy to change in a bugfix release, though. Someone just has to >> open an issue and write a patch. > > I already did open an issue and write a patch :) > > There?s someone on that issue saying that flipping that without a way to flip it back > would break their application. http://bugs.python.org/issue20994, if anyone is looking for it. Eric. From solipsis at pitrou.net Sun Mar 23 16:46:39 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 Mar 2014 16:46:39 +0100 Subject: [Python-Dev] OP_NO_COMPRESSION In-Reply-To: <21535320-998D-4DDD-9EB2-31A30AEB72E5@stufft.io> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323163440.712f98be@fsol> <21535320-998D-4DDD-9EB2-31A30AEB72E5@stufft.io> Message-ID: <20140323164639.76025040@fsol> On Sun, 23 Mar 2014 11:37:25 -0400 Donald Stufft wrote: > > I already did open an issue and write a patch :) > > There?s someone on that issue saying that flipping that without a way to flip it back > would break their application. You're right, I had forgotten about that :-) I'd be surprised if disabling compression could altogether break an application. It may make it less efficient, though, if it relied on TLS to compress data for it (which doesn't sound like a good thing to rely on, anyway). Regards Antoine. From donald at stufft.io Sun Mar 23 16:52:37 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Mar 2014 11:52:37 -0400 Subject: [Python-Dev] OP_NO_COMPRESSION In-Reply-To: <20140323164639.76025040@fsol> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323163440.712f98be@fsol> <21535320-998D-4DDD-9EB2-31A30AEB72E5@stufft.io> <20140323164639.76025040@fsol> Message-ID: <801F6EB8-8817-4FB2-A6CC-DDA5F185DC2F@stufft.io> On Mar 23, 2014, at 11:46 AM, Antoine Pitrou wrote: > On Sun, 23 Mar 2014 11:37:25 -0400 > Donald Stufft wrote: >> >> I already did open an issue and write a patch :) >> >> There?s someone on that issue saying that flipping that without a way to flip it back >> would break their application. > > You're right, I had forgotten about that :-) > > I'd be surprised if disabling compression could altogether break an > application. It may make it less efficient, though, if it relied on TLS > to compress data for it (which doesn't sound like a good thing to rely > on, anyway). > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io I tend to agree about that, but I was mostly echoing the fact that they?ve expressed that sentiment. Without SSLContext I couldn?t think of a reasonable way to enable them to turn it on/off that didn?t involve a whole new API just for some versions of 2.7. This seemed to me like a wholly worse idea than back porting SSLContext :/ ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sun Mar 23 16:55:25 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Mar 2014 11:55:25 -0400 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: <20140323141344.1b3d078a@fsol> References: <20140323141344.1b3d078a@fsol> Message-ID: <30987DAE-F09B-4056-9171-CFA630C792EF@stufft.io> On Mar 23, 2014, at 9:13 AM, Antoine Pitrou wrote: > On Sun, 23 Mar 2014 17:07:24 +1000 > Nick Coghlan wrote: >> Another more critical example is the lack of SSL hostname matching in the >> Python 2 standard library - it is currently necessary to rely on a third >> party library, such as ``requests`` or ``backports.ssl_match_hostname`` to >> obtain that functionality in Python 2. > > Do note that match_hostname() is a pure Python function and is easy to > paste into your own code (if you don't want to pull in a dependency). > It doesn't need SSLContext or any other recent stuff, just a > certificate dict which Python 2.x is already able to provide > (SSLSocket.getpeercert()). So the problem with match_hostname is that it?s a security sensitive function, there have already been at least one fix to it because of it doing something incorrectly. Advocating users to copy it into their own code case typically means that it?ll get copied once and forgotten. So for any security updates in the future they are unlikely to get those. It seems like the danger of _adding_ things like that is pretty minimal. > >> Firstly, this PEP encompasses a non-trivial portion of the standard library. >> It's not just the underlying SSL support, but also the libraries for other >> network protocols like HTTP, FTP, IMAP, and POP3 that integrate with the >> SSL infrastructure to provide secure links, and that's just the protocols >> in the standard library. > > It's still not obvious what you are proposing to do with these other > libraries. If you are proposing to validate certs against system CAs and > check hostnames by default - you are going to break compatibility for a > lot of current uses. > > As Martin I think it would be easier to reason about a concrete backport > proposal. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From breamoreboy at yahoo.co.uk Sun Mar 23 16:55:12 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sun, 23 Mar 2014 15:55:12 +0000 Subject: [Python-Dev] OP_NO_COMPRESSION In-Reply-To: <20140323164639.76025040@fsol> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323163440.712f98be@fsol> <21535320-998D-4DDD-9EB2-31A30AEB72E5@stufft.io> <20140323164639.76025040@fsol> Message-ID: On 23/03/2014 15:46, Antoine Pitrou wrote: > On Sun, 23 Mar 2014 11:37:25 -0400 > Donald Stufft wrote: >> >> I already did open an issue and write a patch :) >> >> There?s someone on that issue saying that flipping that without a way to flip it back >> would break their application. > > You're right, I had forgotten about that :-) > > I'd be surprised if disabling compression could altogether break an > application. It may make it less efficient, though, if it relied on TLS > to compress data for it (which doesn't sound like a good thing to rely > on, anyway). > > Regards > > Antoine. I've looked at the issue and can't find any such comment. The closest is from http://bugs.python.org/issue20994#msg214246 "Also some users will absolutely want to manually re-enable compression, please don't disable it entirely." Chinese whispers? :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com From donald at stufft.io Sun Mar 23 16:58:38 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Mar 2014 11:58:38 -0400 Subject: [Python-Dev] OP_NO_COMPRESSION In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323163440.712f98be@fsol> <21535320-998D-4DDD-9EB2-31A30AEB72E5@stufft.io> <20140323164639.76025040@fsol> Message-ID: <311DDE28-F412-4349-865D-C4DE09348E2D@stufft.io> On Mar 23, 2014, at 11:55 AM, Mark Lawrence wrote: > On 23/03/2014 15:46, Antoine Pitrou wrote: >> On Sun, 23 Mar 2014 11:37:25 -0400 >> Donald Stufft wrote: >>> >>> I already did open an issue and write a patch :) >>> >>> There?s someone on that issue saying that flipping that without a way to flip it back >>> would break their application. >> >> You're right, I had forgotten about that :-) >> >> I'd be surprised if disabling compression could altogether break an >> application. It may make it less efficient, though, if it relied on TLS >> to compress data for it (which doesn't sound like a good thing to rely >> on, anyway). >> >> Regards >> >> Antoine. > > I've looked at the issue and can't find any such comment. The closest is from http://bugs.python.org/issue20994#msg214246 "Also some users will absolutely want to manually re-enable compression, please don't disable it entirely." Chinese whispers? :) > > -- > My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. > > Mark Lawrence > > --- > This email is free from viruses and malware because avast! Antivirus protection is active. > http://www.avast.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Ah, maybe he told me that in IRC then, I actually know him and was arguing with him about it there. Sorry I got my streams crossed. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From guido at python.org Sun Mar 23 17:22:05 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 23 Mar 2014 09:22:05 -0700 Subject: [Python-Dev] On porting to Python 3 as the answer Message-ID: This really upset me: On Sun, Mar 23, 2014 at 3:17 AM, wrote: > I think asking developers to make significant modifications to their > code is besides the point of the PEP. However, if they are willing > to make changes, I'd still recommend that they port their code to > Python 3, as that is the better long-term investment. > This is a completely unrealistic form of wishful thinking, and repeating it won't make it more true. At Dropbox I work with a large group of very capable developers on several large code bases that are currently in 2.7. We are constantly changing our code to make it more secure (there are several teams specifically in charge of that). And yet porting to Python 3 is completely out of scope, for a variety of reasons. Please stop your wishful thinking. (TBH, I expect that none of the changes to Python 2.7 under consideration would make any difference for the security of Dropbox. But neither would switching to Python 3.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Sun Mar 23 17:33:40 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 23 Mar 2014 17:33:40 +0100 Subject: [Python-Dev] On porting to Python 3 as the answer In-Reply-To: References: Message-ID: <532F0CE4.6000200@v.loewis.de> Am 23.03.14 17:22, schrieb Guido van Rossum: > At Dropbox I work with a large group of very capable developers on > several large code bases that are currently in 2.7. We are constantly > changing our code to make it more secure (there are several teams > specifically in charge of that). And yet porting to Python 3 is > completely out of scope, for a variety of reasons. > > Please stop your wishful thinking. I can stop expressing it; I don't think I can stop wishing it :-) If it's really unrealistic that Dropbox will ever port the code to Python 3, would you then think that Python 3 is a doomed project, since it won't ever see significant usage? Regards, Martin From donald at stufft.io Sun Mar 23 18:08:35 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Mar 2014 13:08:35 -0400 Subject: [Python-Dev] On porting to Python 3 as the answer In-Reply-To: <532F0CE4.6000200@v.loewis.de> References: <532F0CE4.6000200@v.loewis.de> Message-ID: <1284082C-56A3-4D13-BB54-4987667BDAA2@stufft.io> On Mar 23, 2014, at 12:33 PM, Martin v. L?wis wrote: > Am 23.03.14 17:22, schrieb Guido van Rossum: >> At Dropbox I work with a large group of very capable developers on >> several large code bases that are currently in 2.7. We are constantly >> changing our code to make it more secure (there are several teams >> specifically in charge of that). And yet porting to Python 3 is >> completely out of scope, for a variety of reasons. >> >> Please stop your wishful thinking. > > I can stop expressing it; I don't think I can stop wishing it :-) > > If it's really unrealistic that Dropbox will ever port the code > to Python 3, would you then think that Python 3 is a doomed project, > since it won't ever see significant usage? I think expecting every production instance of Python 2.7 to port is unrealistic. New projects will start to get written in Python 3, some existing projects will get ported over time. But I doubt there is ever going to be some mass exodus where everyone suddenly starts moving to Python 3. For people/companies/projects where Python 2 is currently working fine and porting would be a significant effort they have to decide between adding more value to their project through bug fixes, new features, etc or essentially sinking cost into Porting to Python 3 which their end users will not benefit from in a very large way. It?s hard to make that business case for existing code, especially given that for a lot of the truly hard parts of porting to Python 3 there?s no way to port module by module so you have to basically do an entire code base at once. This adds extra churn, increases the risk for new bugs, and is quite an annoying process. That doesn?t mean Python 3 is bad or is inherently a doomed project, but it?s essentially a new, but very similar, language. > > Regards, > Martin > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From guido at python.org Sun Mar 23 18:45:27 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 23 Mar 2014 10:45:27 -0700 Subject: [Python-Dev] On porting to Python 3 as the answer In-Reply-To: <532F0CE4.6000200@v.loewis.de> References: <532F0CE4.6000200@v.loewis.de> Message-ID: On Sun, Mar 23, 2014 at 9:33 AM, "Martin v. L?wis" wrote: > Am 23.03.14 17:22, schrieb Guido van Rossum: > > At Dropbox I work with a large group of very capable developers on > > several large code bases that are currently in 2.7. We are constantly > > changing our code to make it more secure (there are several teams > > specifically in charge of that). And yet porting to Python 3 is > > completely out of scope, for a variety of reasons. > > > > Please stop your wishful thinking. > > I can stop expressing it; I don't think I can stop wishing it :-) > > If it's really unrealistic that Dropbox will ever port the code > to Python 3, would you then think that Python 3 is a doomed project, > since it won't ever see significant usage? > No, it's just that the timescale is drastically different. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Mar 23 21:43:14 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 Mar 2014 21:43:14 +0100 Subject: [Python-Dev] cpython: #20145: assert[Raises|Warns]Regex now raise TypeError on bad regex. References: <3fsTlN5YKTz7LjS@mail.python.org> Message-ID: <20140323214314.3de42c4f@fsol> On Sun, 23 Mar 2014 20:47:28 +0100 (CET) r.david.murray wrote: > http://hg.python.org/cpython/rev/ec556e45641a > changeset: 89936:ec556e45641a > user: R David Murray > date: Sun Mar 23 15:08:43 2014 -0400 > summary: > #20145: assert[Raises|Warns]Regex now raise TypeError on bad regex. > > Previously a non-string, non-regex second argument could cause the test > to always pass. It seems like this would be useful to fix in 3.4 too. Regards Antoine. From ncoghlan at gmail.com Sun Mar 23 23:35:26 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Mar 2014 08:35:26 +1000 Subject: [Python-Dev] On porting to Python 3 as the answer In-Reply-To: References: <532F0CE4.6000200@v.loewis.de> Message-ID: On 24 Mar 2014 03:48, "Guido van Rossum" wrote: > > On Sun, Mar 23, 2014 at 9:33 AM, "Martin v. L?wis" wrote: >> >> Am 23.03.14 17:22, schrieb Guido van Rossum: >> > At Dropbox I work with a large group of very capable developers on >> > several large code bases that are currently in 2.7. We are constantly >> > changing our code to make it more secure (there are several teams >> > specifically in charge of that). And yet porting to Python 3 is >> > completely out of scope, for a variety of reasons. >> > >> > Please stop your wishful thinking. >> >> I can stop expressing it; I don't think I can stop wishing it :-) >> >> If it's really unrealistic that Dropbox will ever port the code >> to Python 3, would you then think that Python 3 is a doomed project, >> since it won't ever see significant usage? > > > No, it's just that the timescale is drastically different. Yep, and Paul was ultimately right in guessing that part of my motivation here was seeing trouble on the horizon in terms of RH's ability to affordably sustain our own long term support commitments for the Python 2 series. That said, I didn't actually realise the full implications for us until *after* writing, publishing and receiving feedback on the PEP, though - otherwise I would likely have done things in a different order. Isn't open source fun? :) So I'm now going to switch my focus on this topic to the Fedora community for a while: I think the second draft of the PEP is in a pretty reasonable state, and from my perspective, anything we come up with is going to have to at least pass muster with the Fedora Engineering Steering Committee for it to actually solve the upcoming supportability problem on that side of things. It would be good if folks engaged with other downstream redistributors could also get them to take a look and provide feedback. A plan we like, but key redistributors choose not to go along with, would be rather missing the point... Cheers, Nick. > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Mar 24 00:24:29 2014 From: barry at python.org (Barry Warsaw) Date: Sun, 23 Mar 2014 19:24:29 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <20140323010138.48bcafe4@fsol> References: <20140323010138.48bcafe4@fsol> Message-ID: <20140323192429.0c9bb343@limelight.wooz.org> On Mar 23, 2014, at 01:01 AM, Antoine Pitrou wrote: >But enforcing "secure by default" can by construction break backwards >compatibility, which is the very reason we are so conservative with >such changes. Also, many developers who are stuck on Python 2 have already evaluated, designed, and implemented workarounds for security issues in ancient stdlib code. You have to be very careful that any changes in some future 2.7 stdlib secure-by-default release doesn't break those workarounds. That's a "trick question" too because you can't know all of them. I didn't read the PEP until just now, so I never saw the first draft. As written it still makes me uncomfortable because as Antoine says, lots of changes could be classified as "security related" and we definitely don't want this PEP to be used as a wedge to make a wink-wink-nudge-nudge release of Python 2.8. I think the key point for consumers of Python has to be *predictability*. -Barry From tjreedy at udel.edu Mon Mar 24 00:33:05 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 23 Mar 2014 19:33:05 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> Message-ID: On 3/23/2014 3:29 AM, Cory Benfield wrote: > On 23 March 2014 at 04:32:17, Terry Reedy (tjreedy at udel.edu(mailto:tjreedy at udel.edu)) wrote: >> Instead, I think the PEP should propose a special series of server >> enhancement releases that are based on the final 2.7 maintenance release >> (2.7.8 or 2.7.9) but which have have a different application-specific >> enhancement policy. > > This is an interesting idea. My biggest problem with it is that, at least > with the ssl library, these aren?t server-only problems. If we suggest that > they are, we end up in the same position we?re in right now (that is, hurting > the internet). If I understood the problem and pep too narrowly, substitute 'security' or whatever phrase you want for 'server'. My last sentence was an admission that some details of what I said would need to be changed. The key idea is to admit the what is being proposed is a minor fork of 2.7 (somewhat like Stackless 2.7) that might break code and which would require separate code testing -- Terry Jan Reedy From rdmurray at bitdance.com Mon Mar 24 00:44:42 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 23 Mar 2014 19:44:42 -0400 Subject: [Python-Dev] cpython: #20145: assert[Raises|Warns]Regex now raise TypeError on bad regex. In-Reply-To: <20140323214314.3de42c4f@fsol> References: <3fsTlN5YKTz7LjS@mail.python.org> <20140323214314.3de42c4f@fsol> Message-ID: <20140323234443.0D120250049@webabinitio.net> On Sun, 23 Mar 2014 21:43:14 +0100, Antoine Pitrou wrote: > On Sun, 23 Mar 2014 20:47:28 +0100 (CET) > r.david.murray wrote: > > http://hg.python.org/cpython/rev/ec556e45641a > > changeset: 89936:ec556e45641a > > user: R David Murray > > date: Sun Mar 23 15:08:43 2014 -0400 > > summary: > > #20145: assert[Raises|Warns]Regex now raise TypeError on bad regex. > > > > Previously a non-string, non-regex second argument could cause the test > > to always pass. > > It seems like this would be useful to fix in 3.4 too. You will note that the change got a porting note in What's New. It will only break tests (that are currently no-ops), not code, so I'd be fine with backporting it, but the original issue did not call for that. If we fix it in 3.4, should we fix it in 2.7 as well? --David From solipsis at pitrou.net Mon Mar 24 00:47:54 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 24 Mar 2014 00:47:54 +0100 Subject: [Python-Dev] cpython: #20145: assert[Raises|Warns]Regex now raise TypeError on bad regex. In-Reply-To: <20140323234443.0D120250049@webabinitio.net> References: <3fsTlN5YKTz7LjS@mail.python.org> <20140323214314.3de42c4f@fsol> <20140323234443.0D120250049@webabinitio.net> Message-ID: <20140324004754.2449e7fb@fsol> On Sun, 23 Mar 2014 19:44:42 -0400 "R. David Murray" wrote: > On Sun, 23 Mar 2014 21:43:14 +0100, Antoine Pitrou wrote: > > On Sun, 23 Mar 2014 20:47:28 +0100 (CET) > > r.david.murray wrote: > > > http://hg.python.org/cpython/rev/ec556e45641a > > > changeset: 89936:ec556e45641a > > > user: R David Murray > > > date: Sun Mar 23 15:08:43 2014 -0400 > > > summary: > > > #20145: assert[Raises|Warns]Regex now raise TypeError on bad regex. > > > > > > Previously a non-string, non-regex second argument could cause the test > > > to always pass. > > > > It seems like this would be useful to fix in 3.4 too. > > You will note that the change got a porting note in What's New. It will > only break tests (that are currently no-ops), not code, so I'd be fine > with backporting it, but the original issue did not call for that. Exposing buggy tests sounds like a good thing to me :-) > If we fix it in 3.4, should we fix it in 2.7 as well? If the *Regex methods are there, yes, IMO. Regards Antoine. From ncoghlan at gmail.com Mon Mar 24 00:48:06 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Mar 2014 09:48:06 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <20140323192429.0c9bb343@limelight.wooz.org> References: <20140323010138.48bcafe4@fsol> <20140323192429.0c9bb343@limelight.wooz.org> Message-ID: On 24 Mar 2014 09:27, "Barry Warsaw" wrote: > > On Mar 23, 2014, at 01:01 AM, Antoine Pitrou wrote: > > >But enforcing "secure by default" can by construction break backwards > >compatibility, which is the very reason we are so conservative with > >such changes. > > Also, many developers who are stuck on Python 2 have already evaluated, > designed, and implemented workarounds for security issues in ancient stdlib > code. You have to be very careful that any changes in some future 2.7 stdlib > secure-by-default release doesn't break those workarounds. That's a "trick > question" too because you can't know all of them. > > I didn't read the PEP until just now, so I never saw the first draft. As > written it still makes me uncomfortable because as Antoine says, lots of > changes could be classified as "security related" and we definitely don't want > this PEP to be used as a wedge to make a wink-wink-nudge-nudge release of > Python 2.8. Hence why the proposal mostly *isn't* scoped by "security related". It is scoped by "affects the security design of other systems, including systems not written in Python". The core problem to be solved is ensuring that the "cold, dead hand" of the Python 2 standard library doesn't hold back the evolution of internet security standards. I'm starting to think it makes sense to drop the higher level abstractions from the proposal, though. Once the core SSL support issue is addressed, it's comparatively straightforward to handle any other cases as PyPI modules if needed. pip, for example, already bundles requests to handle HTTPS, but that still relies on the stdlib SSL support. > I think the key point for consumers of Python has to be *predictability*. Agreed. That's a key part of why the proposal is mainly about syncing certain key modules with their Python 3 counterparts, rather than piecemeal backports. That way, all you need to know is "the SSL, hashlib and hmac modules are kept in sync with Python 3 feature releases, but use the same default settings as the original Python 2.7 release". A partial backport is actually *harder* for users to deal with (another reason I'm quite amenable to dropping that part of the proposal). Cheers, Nick. > > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Mar 24 00:54:20 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 23 Mar 2014 19:54:20 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> Message-ID: On 3/23/2014 9:00 AM, Skip Montanaro wrote: > On Sat, Mar 22, 2014 at 11:31 PM, Terry Reedy wrote: >> The download page for the final 2.7.z maintenance release could say >> something like "We recommend that you move to the most recent Python 3 >> version if at all possible. If you cannot do that and you want to use Python >> to run a server on the public internet, we urge you to instead use the >> latest version of ServerPython 2.7.1s. This series is based on Python 2.7.z >> but has been and will continue to be enhanced with security features >> backported from Python 3." > > I'm unclear how this would be better than just biting the bullet and > making a 2.8 release. The first step is to 'bite the bullet' and admit that the proposal is for a fork of 2.7. (Martin use that word today and pointed out the analogy with Stackless Python.) I decided not to suggest '2.8' because a) we said there would be no 2.8 (though that could be reversed); b) once we say '2.8', there would be a mountain of proposals to change this and that; c) people will expect the change from 2.7 to 2.8 to be something like previous deltas, but it will not be; d) the security fork might have a change that would normally require a deprecation period, but there might not be; and e) once a 2.8 were released, it too would be closed to enhancements, so that 2.9, 2.10, 2.11.. would eventually be needed. I mentioned this as one possible solution, but one I do not like. In summary, "2.8 is to 2.7 as 2.7 is to 2.6, etcetera" would not be true, so lets not pretend or mislead people that is would be. The fork series, not being a regular CPython series, would not be subject to the regular CPython rule of needing a new minor version number for each set of enhancements and spacing the sets 18 to 24 months apart. > On the one hand, the 2.7.x number suggests > (based on the existing release protocol) that it should be a drop-in > replacement for earlier 2.7 micro releases. No CPython x.y.z has a two-digit number for z. I know that this is a subtle hint, which is why I also propose a new name. I do not think that anyone expects Stackless Python 2.7 to be an exact drop-in replacement for any 2.7.z, although it is for code that is not affected by its specialized alterations. > On the other hand, calling > it something like "ServerPython" implies that it's not necessary for > network client applications, when, if I read the PEP correctly, it > most certainly would be. Consider 'ServerPython' a placeholder for a better name. In my penultimate sentence "There are still details to be filled in or altered," I was specifically thinking of this. > If you create a 2.8 release which is restricted to just the topic > areas of the PEP (that is, no other stuff backported from 3.x, no > requirement to add other non-security bug fixes, etc), the incremented > minor version number tells people that a bit of extra care is required > to upgrade. The lack of change in the code base outside the security > apparatus should make update pretty trivial for most every > non-networked application. If the PEP or something like it is > approved, the work is still going to have to be done, no matter what > you call it. Why not be transparent about it? We agree on being transparent, even if we have different ideas on exactly what needs to be made clear. -- Terry Jan Reedy From barry at python.org Mon Mar 24 01:03:12 2014 From: barry at python.org (Barry Warsaw) Date: Sun, 23 Mar 2014 20:03:12 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> Message-ID: <20140323200312.33b6b0b1@limelight.wooz.org> On Mar 23, 2014, at 08:00 AM, Skip Montanaro wrote: >I'm unclear how this would be better than just biting the bullet and >making a 2.8 release. On the one hand, the 2.7.x number suggests >(based on the existing release protocol) that it should be a drop-in >replacement for earlier 2.7 micro releases. On the other hand, calling >it something like "ServerPython" implies that it's not necessary for >network client applications, when, if I read the PEP correctly, it >most certainly would be. It seems to me that the problem the PEP addresses can largely be confined to the Python 2.7 standard library and the fact that it's bundled with the language. I want to stick to our no-Python-2.8 pledge, but I wonder if there isn't a middle ground where we fork the stdlib into a separate branch, and apply security specific changes there. Python 2.7.x will always be the "standard stdlib". We would never release a specific Python 2.7 + "security stdlib" release, but downstream developers would be able to overlay this forked stdlib on top of the standard one. Volunteers or corporate sponsors could distribute binary installers with this combination of pure Python 2.7 language + "security enhanced stdlib", and Linux distros could do the necessary building and distributing for their own platforms if they so desired. The trick is what do you call this new combination, how do you invoke it, and how do you keep it distinct and independent of the system's standard Python 2.7? Maybe this is some scent of Python 2.8 by another name, but let's never call it Python 2.8. -Barry From jnoller at gmail.com Mon Mar 24 01:31:36 2014 From: jnoller at gmail.com (Jesse Noller) Date: Sun, 23 Mar 2014 19:31:36 -0500 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <20140323200312.33b6b0b1@limelight.wooz.org> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> Message-ID: <49C342BC-2199-476B-A94D-1C171B7F3F49@gmail.com> > On Mar 23, 2014, at 7:03 PM, Barry Warsaw wrote: > >> On Mar 23, 2014, at 08:00 AM, Skip Montanaro wrote: >> >> I'm unclear how this would be better than just biting the bullet and >> making a 2.8 release. On the one hand, the 2.7.x number suggests >> (based on the existing release protocol) that it should be a drop-in >> replacement for earlier 2.7 micro releases. On the other hand, calling >> it something like "ServerPython" implies that it's not necessary for >> network client applications, when, if I read the PEP correctly, it >> most certainly would be. > > It seems to me that the problem the PEP addresses can largely be confined to > the Python 2.7 standard library and the fact that it's bundled with the > language. I want to stick to our no-Python-2.8 pledge, but I wonder if there > isn't a middle ground where we fork the stdlib into a separate branch, and > apply security specific changes there. > > Python 2.7.x will always be the "standard stdlib". We would never release a > specific Python 2.7 + "security stdlib" release, but downstream developers > would be able to overlay this forked stdlib on top of the standard one. > Volunteers or corporate sponsors could distribute binary installers with this > combination of pure Python 2.7 language + "security enhanced stdlib", and > Linux distros could do the necessary building and distributing for their own > platforms if they so desired. > > The trick is what do you call this new combination, how do you invoke it, and > how do you keep it distinct and independent of the system's standard Python > 2.7? > > Maybe this is some scent of Python 2.8 by another name, but let's never call > it Python 2.8. > It seems like this and the fork idea are both jumping through hoops to avoid the inevitable - a 2.7 security release or a 2.8 security only release. Especially as I know nick isn't the only one looking to hire FTEs for this specific work as it's actively hurting service providers/distributions and others. If we call it a security only release, and scope/review all patches to that it seems that it's pretty clear what the release is for. 2.x is going to be in the wild for at least another decade as these large legacy codebases are functioning, profitable and dedicating teams of engineers to port them to 3.x doesn't make sense to the business. As new services and things come online we'll see the gradual movement to 3 accelerate especially as people realize clients and service talk better on an interface such as http rather than large monolithic tragedies. > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com From rosuav at gmail.com Mon Mar 24 01:38:01 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 24 Mar 2014 11:38:01 +1100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <20140323200312.33b6b0b1@limelight.wooz.org> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> Message-ID: On Mon, Mar 24, 2014 at 11:03 AM, Barry Warsaw wrote: > Python 2.7.x will always be the "standard stdlib". We would never release a > specific Python 2.7 + "security stdlib" release, but downstream developers > would be able to overlay this forked stdlib on top of the standard one. > Volunteers or corporate sponsors could distribute binary installers with this > combination of pure Python 2.7 language + "security enhanced stdlib", and > Linux distros could do the necessary building and distributing for their own > platforms if they so desired. > > The trick is what do you call this new combination, how do you invoke it, and > how do you keep it distinct and independent of the system's standard Python > 2.7? Easy. Just set PYTHONPATH to import the SEPython [1] lib ahead of the standard lib. Then you can go back to the standard 2.7 (if you want to) by unsetting PYTHONPATH. It'd be nice if SEPython defined a modified sys.version for clarity, but otherwise, it'd be a vanilla Python 2.7. ChrisA [1] By analogy with SELinux From tjreedy at udel.edu Mon Mar 24 01:43:09 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 23 Mar 2014 20:43:09 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <20140323200312.33b6b0b1@limelight.wooz.org> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> Message-ID: On 3/23/2014 8:03 PM, Barry Warsaw wrote: > On Mar 23, 2014, at 08:00 AM, Skip Montanaro wrote: > >> I'm unclear how this would be better than just biting the bullet and >> making a 2.8 release. On the one hand, the 2.7.x number suggests >> (based on the existing release protocol) that it should be a drop-in >> replacement for earlier 2.7 micro releases. On the other hand, calling >> it something like "ServerPython" implies that it's not necessary for >> network client applications, when, if I read the PEP correctly, it >> most certainly would be. > It seems to me that the problem the PEP addresses can largely be confined to > the Python 2.7 standard library and the fact that it's bundled with the > language. I want to stick to our no-Python-2.8 pledge, but I wonder if there > isn't a middle ground where we fork the stdlib into a separate branch, and > apply security specific changes there. > > Python 2.7.x will always be the "standard stdlib". We would never release a > specific Python 2.7 + "security stdlib" release, but downstream developers > would be able to overlay this forked stdlib on top of the standard one. > Volunteers or corporate sponsors could distribute binary installers with this > combination of pure Python 2.7 language + "security enhanced stdlib", and > Linux distros could do the necessary building and distributing for their own > platforms if they so desired. > > The trick is what do you call this new combination, I have already agree that 'ServerPython' is just a placeholder. > how do you invoke it, and how do you keep it distinct and > independent of the system's standard Python 2.7? > Maybe this is some scent of Python 2.8 by another name, but let's never call > it Python 2.8. I agree, no only because of the 'pledge', but because it is the wrong model. Nick's latest paraphrase of where he thinks his proposal is heading is "the SSL, hashlib and hmac modules are kept in sync with Python 3 feature releases, but use the same default settings as the original Python 2.7 release". '2.8' would imply a broader focus than just 3 modules, and it would only work for the first of a series of enhancement releases. -- Terry Jan Reedy From tjreedy at udel.edu Mon Mar 24 01:50:52 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 23 Mar 2014 20:50:52 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <20140323010138.48bcafe4@fsol> <20140323192429.0c9bb343@limelight.wooz.org> Message-ID: On 3/23/2014 7:48 PM, Nick Coghlan wrote: > Agreed. That's a key part of why the proposal is mainly about syncing > certain key modules with their Python 3 counterparts, rather than > piecemeal backports. That way, all you need to know is "the SSL, hashlib > and hmac modules are kept in sync with Python 3 feature releases, but > use the same default settings as the original Python 2.7 release". If you restrict the proposal to just those three modules, then the proposal could be to just add three new modules to 2.7: ssl3, hashlib3, and hmac3. Each would be regularly updated backports of the 3.x modules with two differences: defaults for settings that exist in 2.7 would remain as in 2.7 and would not be updated*, and each module would get a .py3_ver x.y.z attribute so code could be conditioned on which backport version it is running with. * though users would be encouraged to use the most secure settings. -- Terry Jan Reedy From donald at stufft.io Mon Mar 24 01:57:21 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Mar 2014 20:57:21 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <49C342BC-2199-476B-A94D-1C171B7F3F49@gmail.com> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <49C342BC-2199-476B-A94D-1C171B7F3F49@gmail.com> Message-ID: <18738740-69F3-4361-8C28-0474B1B198A4@stufft.io> On Mar 23, 2014, at 8:31 PM, Jesse Noller wrote: > > >> On Mar 23, 2014, at 7:03 PM, Barry Warsaw wrote: >> >>> On Mar 23, 2014, at 08:00 AM, Skip Montanaro wrote: >>> >>> I'm unclear how this would be better than just biting the bullet and >>> making a 2.8 release. On the one hand, the 2.7.x number suggests >>> (based on the existing release protocol) that it should be a drop-in >>> replacement for earlier 2.7 micro releases. On the other hand, calling >>> it something like "ServerPython" implies that it's not necessary for >>> network client applications, when, if I read the PEP correctly, it >>> most certainly would be. >> >> It seems to me that the problem the PEP addresses can largely be confined to >> the Python 2.7 standard library and the fact that it's bundled with the >> language. I want to stick to our no-Python-2.8 pledge, but I wonder if there >> isn't a middle ground where we fork the stdlib into a separate branch, and >> apply security specific changes there. >> >> Python 2.7.x will always be the "standard stdlib". We would never release a >> specific Python 2.7 + "security stdlib" release, but downstream developers >> would be able to overlay this forked stdlib on top of the standard one. >> Volunteers or corporate sponsors could distribute binary installers with this >> combination of pure Python 2.7 language + "security enhanced stdlib", and >> Linux distros could do the necessary building and distributing for their own >> platforms if they so desired. >> >> The trick is what do you call this new combination, how do you invoke it, and >> how do you keep it distinct and independent of the system's standard Python >> 2.7? >> >> Maybe this is some scent of Python 2.8 by another name, but let's never call >> it Python 2.8. >> > > It seems like this and the fork idea are both jumping through hoops to avoid the inevitable - a 2.7 security release or a 2.8 security only release. Especially as I know nick isn't the only one looking to hire FTEs for this specific work as it's actively hurting service providers/distributions and others. > > If we call it a security only release, and scope/review all patches to that it seems that it's pretty clear what the release is for. > > 2.x is going to be in the wild for at least another decade as these large legacy codebases are functioning, profitable and dedicating teams of engineers to port them to 3.x doesn't make sense to the business. As new services and things come online we'll see the gradual movement to 3 accelerate especially as people realize clients and service talk better on an interface such as http rather than large monolithic tragedies. > > >> -Barry >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io I agree, the bulk of the alternative suggestions feel more like trying to adhere to a policy for policy?s sake rather than actually figure out what is best for the users. Adding new APIs to 2.7 feels to me like a pretty backwards compatible thing to be honest. The biggest contenders are things like SSLContext and match_hostname. The PEP still says things must be backwards compatible so we can't change existing APIs in a way that the backported things aren't opt in. I mean how likely is it that there is code out there relying on the fact that SSLContext doesn't exist other than as a detection to determine if they can use SSLContext? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From tjreedy at udel.edu Mon Mar 24 01:58:21 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 23 Mar 2014 20:58:21 -0400 Subject: [Python-Dev] cpython: #20145: assert[Raises|Warns]Regex now raise TypeError on bad regex. In-Reply-To: <20140324004754.2449e7fb@fsol> References: <3fsTlN5YKTz7LjS@mail.python.org> <20140323214314.3de42c4f@fsol> <20140323234443.0D120250049@webabinitio.net> <20140324004754.2449e7fb@fsol> Message-ID: On 3/23/2014 7:47 PM, Antoine Pitrou wrote: > On Sun, 23 Mar 2014 19:44:42 -0400 > "R. David Murray" wrote: > >> On Sun, 23 Mar 2014 21:43:14 +0100, Antoine Pitrou wrote: >>> On Sun, 23 Mar 2014 20:47:28 +0100 (CET) >>> r.david.murray wrote: ... >>>> Previously a non-string, non-regex second argument could cause the test >>>> to always pass. >>> >>> It seems like this would be useful to fix in 3.4 too. >> >> You will note that the change got a porting note in What's New. It will >> only break tests (that are currently no-ops), not code, so I'd be fine >> with backporting it, but the original issue did not call for that. > > Exposing buggy tests sounds like a good thing to me :-) > >> If we fix it in 3.4, should we fix it in 2.7 as well? > > If the *Regex methods are there, yes, IMO. I agree. Since I and others am writing tests for all current versions of Idle, I would like a buggy test to fail no matter where it is first tested. Currently, I would tend to start with 3.4. -- Terry Jan Reedy From barry at python.org Mon Mar 24 02:31:12 2014 From: barry at python.org (Barry Warsaw) Date: Sun, 23 Mar 2014 21:31:12 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> Message-ID: <20140323213112.1b7d2e32@limelight.wooz.org> On Mar 24, 2014, at 11:38 AM, Chris Angelico wrote: >Easy. Just set PYTHONPATH to import the SEPython [1] lib ahead of the >standard lib. Then you can go back to the standard 2.7 (if you want >to) by unsetting PYTHONPATH. > >It'd be nice if SEPython defined a modified sys.version for clarity, >but otherwise, it'd be a vanilla Python 2.7. That's certainly more in the direction of what I think is an appropriate upstream solution. The thing is, there isn't one single "what's best for users" resolution. There are many, many competing requirements and I think it will be difficult to satisfy everyone. I'm particularly sensitive to complaints of unexpected changes between micro releases. -Barry From donald at stufft.io Mon Mar 24 03:34:56 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Mar 2014 22:34:56 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <20140323213112.1b7d2e32@limelight.wooz.org> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> Message-ID: On Mar 23, 2014, at 9:31 PM, Barry Warsaw wrote: > On Mar 24, 2014, at 11:38 AM, Chris Angelico wrote: > >> Easy. Just set PYTHONPATH to import the SEPython [1] lib ahead of the >> standard lib. Then you can go back to the standard 2.7 (if you want >> to) by unsetting PYTHONPATH. >> >> It'd be nice if SEPython defined a modified sys.version for clarity, >> but otherwise, it'd be a vanilla Python 2.7. > > That's certainly more in the direction of what I think is an appropriate > upstream solution. > > The thing is, there isn't one single "what's best for users" resolution. > There are many, many competing requirements and I think it will be difficult > to satisfy everyone. I'm particularly sensitive to complaints of unexpected > changes between micro releases. If it?s too much for a micro release call it 2.8 or 2.9 or something and give it a super limited scope. Proposals like modifying the sys.path to opt into the backported things or having an official unofficial stdlib just adds more things downstream users have to actually test against and more possible configurations or Pythons. Right now users have a singular method for determining what the runtime environment looks like for Python, the version. There are processes around selecting different Python versions for things, upgrading etc. This isn?t a new thing for users. Suddenly having to mess with PYTHONPATH or having to figure out if their distro used the ?official stdlib? or the ?official unofficial stdlib? just adds massive confusion. Things like ssl3 etc are ?ok? but less than optimal IMO because one of the benefits of adding SSLContext and such to Python 2.x is that you can write code that works in 2.x and 3.x without having to special case 2.7.7 or 2.8 or whatever. However if you just add ssl3 then now you have a third path through the code that you have to deal with. > > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rosuav at gmail.com Mon Mar 24 06:24:12 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 24 Mar 2014 16:24:12 +1100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> Message-ID: On Mon, Mar 24, 2014 at 1:34 PM, Donald Stufft wrote: > Right now users have a singular method for determining what the runtime > environment looks like for Python, the version. There are processes around > selecting different Python versions for things, upgrading etc. This isn?t > a new thing for users. Suddenly having to mess with PYTHONPATH or > having to figure out if their distro used the ?official stdlib? or the ?official > unofficial stdlib? just adds massive confusion. Various distros of Linux ship modified versions of things. On my current system (Debian Wheezy), the apt-gettable Python is tagged "2.7.3-4+deb7u1", so there are already at least some patches that aren't in vanilla 2.7.3. Presumably the distro could happily just ship the enhanced stdlib, and you don't need to mess with PYTHONPATH; and a modified sys.version or other detection method could be used to probe if you need to be sure. As has already been pointed out, this can already happen, but in an ad-hoc way. Making it official or semi-official would mean that a script written for Debian's "Python 2.7.10" would run on Red Hat's "Python 2.7.10", which would surely be an advantage. ChrisA From theller at ctypes.org Mon Mar 24 08:35:13 2014 From: theller at ctypes.org (Thomas Heller) Date: Mon, 24 Mar 2014 08:35:13 +0100 Subject: [Python-Dev] python 3.4 and pywin32 In-Reply-To: References: Message-ID: Am 22.03.2014 00:25, schrieb Nick Coghlan: > On 22 March 2014 05:46, Thomas Heller wrote: >> With python 3.4 and pywin32 version 218 it is only possible >> to import win32com or win32api when pywintypes has been imported before. >> >> I have no idea if this is a bug in pywin32 or in Python 3.4. >> Does anyone know more? > > Do you have a traceback for the failed imports? There were certainly > non-trivial changes to the import system in 3.4, but without knowing > more details about the failures, it's hard to say what the culprits > might be. I have created http://bugs.python.org/issue21050. Thomas From ncoghlan at gmail.com Mon Mar 24 09:44:30 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Mar 2014 18:44:30 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> Message-ID: On 24 Mar 2014 15:25, "Chris Angelico" wrote: > As has already been pointed out, this can already happen, but in an > ad-hoc way. Making it official or semi-official would mean that a > script written for Debian's "Python 2.7.10" would run on Red Hat's > "Python 2.7.10", which would surely be an advantage. And having it break on the official Windows and Mac OS X binaries would benefit end users, how? The position I am coming to is that the "enhanced security" release should be the default one that we publish binary installers for, but we should also ensure that downstream redistributors can easily do "Python 2.7 with legacy SSL" releases if they so choose. I'm happier forcing end users to rely on a redistributor to opt in to a lower security option than I am to knowingly publish too many additional releases with network security infrastructure that is (at best) rapidly approaching its use by date. Cheers, Nick. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Mar 24 10:02:21 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 24 Mar 2014 20:02:21 +1100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> Message-ID: On Mon, Mar 24, 2014 at 7:44 PM, Nick Coghlan wrote: > > On 24 Mar 2014 15:25, "Chris Angelico" wrote: > >> As has already been pointed out, this can already happen, but in an >> ad-hoc way. Making it official or semi-official would mean that a >> script written for Debian's "Python 2.7.10" would run on Red Hat's >> "Python 2.7.10", which would surely be an advantage. > > And having it break on the official Windows and Mac OS X binaries would > benefit end users, how? It wouldn't. And if python.org doesn't publish a Windows binary, then either the exact same thing will happen, or there'll be one unofficial SEPython-like build. With only a semi-official SEPython 2.7.10, there could still be installers for all platforms - if nothing better, a two-stage installer that fires off the unchanged one and then plonks its own files elsewhere. > The position I am coming to is that the "enhanced security" release should > be the default one that we publish binary installers for, but we should also > ensure that downstream redistributors can easily do "Python 2.7 with legacy > SSL" releases if they so choose. I'm happier forcing end users to rely on a > redistributor to opt in to a lower security option than I am to knowingly > publish too many additional releases with network security infrastructure > that is (at best) rapidly approaching its use by date. That's how it'd be with an official 2.7 security-enhanced release, and that would be fine. But even not going that far would have its benefit; as long as there's a clear definition of what "SEPython 2.7.10" (or whatever it's called) is, a source-only release of those files would at least help to unify what would otherwise be a mess. But going the whole way and making a binary installer for an enhanced-security Python would definitely unify it a lot more. ChrisA From mal at egenix.com Mon Mar 24 10:10:18 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 24 Mar 2014 10:10:18 +0100 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: References: Message-ID: <532FF67A.3010206@egenix.com> On 23.03.2014 08:07, Nick Coghlan wrote: > Open Questions > ============== > > * What are the risks associated with allowing OpenSSL to be updated to > new feature versions in the Windows and Mac OS X binary installers for > maintenance releases? Currently we just upgrade to the appropriate > OpenSSL maintenance releases, rather than switching to the latest > feature release. In particular, is it possible Windows C extensions may > be linking against the Python provided OpenSSL module? Python's _ssl/_hashlib modules link statically against OpenSSL in Python 2.7, so the OpenSSL DLLs are not exposed to other extensions. The OpenSSL version used for 2.7.6 is 0.9.8y. Upgrading to 1.0.0 or 1.0.1 will likely need a few minor tweaks, but not cause general breakage - at least that's my experience with the egenix-pyopenssl distribution. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 24 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-03-29: PythonCamp 2014, Cologne, Germany ... 5 days to go 2014-04-09: PyCon 2014, Montreal, Canada ... 16 days to go 2014-04-29: Python Meeting Duesseldorf ... 36 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Mon Mar 24 13:33:00 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 24 Mar 2014 13:33:00 +0100 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: <532FF67A.3010206@egenix.com> References: <532FF67A.3010206@egenix.com> Message-ID: Le 24/03/2014 10:10, M.-A. Lemburg a ?crit : > On 23.03.2014 08:07, Nick Coghlan wrote: >> Open Questions >> ============== >> >> * What are the risks associated with allowing OpenSSL to be updated to >> new feature versions in the Windows and Mac OS X binary installers for >> maintenance releases? Currently we just upgrade to the appropriate >> OpenSSL maintenance releases, rather than switching to the latest >> feature release. In particular, is it possible Windows C extensions may >> be linking against the Python provided OpenSSL module? > > Python's _ssl/_hashlib modules link statically against OpenSSL in > Python 2.7, so the OpenSSL DLLs are not exposed to other extensions. I suppose you mean under Windows. Under Linux (and probably OS X too), the _ssl module is linked dynamically with OpenSSL: $ ldd build/lib.linux-x86_64-2.7-pydebug/_ssl.so linux-vdso.so.1 => (0x00007fff3f1de000) libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fd8853ea000) libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007fd885010000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd884df1000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd884a2b000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd884827000) /lib64/ld-linux-x86-64.so.2 (0x00007fd885868000) Regards Antoine. From mal at egenix.com Mon Mar 24 13:39:07 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 24 Mar 2014 13:39:07 +0100 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: References: <532FF67A.3010206@egenix.com> Message-ID: <5330276B.4080600@egenix.com> On 24.03.2014 13:33, Antoine Pitrou wrote: > Le 24/03/2014 10:10, M.-A. Lemburg a ?crit : >> On 23.03.2014 08:07, Nick Coghlan wrote: >>> Open Questions >>> ============== >>> >>> * What are the risks associated with allowing OpenSSL to be updated to >>> new feature versions in the Windows and Mac OS X binary installers for >>> maintenance releases? Currently we just upgrade to the appropriate >>> OpenSSL maintenance releases, rather than switching to the latest >>> feature release. In particular, is it possible Windows C extensions may >>> be linking against the Python provided OpenSSL module? >> >> Python's _ssl/_hashlib modules link statically against OpenSSL in >> Python 2.7, so the OpenSSL DLLs are not exposed to other extensions. > > I suppose you mean under Windows. Yes. Should have included that detail in the email :-) > Under Linux (and probably OS X too), the _ssl module is linked > dynamically with OpenSSL: > > $ ldd build/lib.linux-x86_64-2.7-pydebug/_ssl.so > linux-vdso.so.1 => (0x00007fff3f1de000) > libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fd8853ea000) > libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007fd885010000) > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd884df1000) > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd884a2b000) > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd884827000) > /lib64/ld-linux-x86-64.so.2 (0x00007fd885868000) Right, and it's using the system library, not a private copy - which can be both good and bad depending on how recent the system's library version is. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 24 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-03-29: PythonCamp 2014, Cologne, Germany ... 5 days to go 2014-04-09: PyCon 2014, Montreal, Canada ... 16 days to go 2014-04-29: Python Meeting Duesseldorf ... 36 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Mon Mar 24 13:51:50 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Mar 2014 22:51:50 +1000 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: <5330276B.4080600@egenix.com> References: <532FF67A.3010206@egenix.com> <5330276B.4080600@egenix.com> Message-ID: On 24 March 2014 22:39, M.-A. Lemburg wrote: > On 24.03.2014 13:33, Antoine Pitrou wrote: >> Under Linux (and probably OS X too), the _ssl module is linked >> dynamically with OpenSSL: >> >> $ ldd build/lib.linux-x86_64-2.7-pydebug/_ssl.so >> linux-vdso.so.1 => (0x00007fff3f1de000) >> libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fd8853ea000) >> libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007fd885010000) >> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd884df1000) >> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd884a2b000) >> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd884827000) >> /lib64/ld-linux-x86-64.so.2 (0x00007fd885868000) > > Right, and it's using the system library, not a private copy - which > can be both good and bad depending on how recent the system's library > version is. Even if *we* statically linked OpenSSL on Linux, you can bet distro vendors would switch it back to dynamic linking. Hence the comment in the PEP about vendor provided OpenSSL updates mitigating some of the concerns on Linux (defaulting not all of them though - it's still far too easy for developers to make mistakes and too hard from them to do the right thing from a security perspective). You also reminded me that I need to dig around for and reference Ned's email about the status of OS X and reference that (OpenSSL upgrades were a casualty of Apple's anti-GPL crusade, so the OS X installers were switched to static linking somewhere along the line). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Mar 24 14:43:29 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Mar 2014 23:43:29 +1000 Subject: [Python-Dev] PEP 466 (round 3): Python 2 network security enhancements Message-ID: And time for round 3 :) Notable changes: - removed the higher level networking modules from the scope of the PEP. They made people nervous, and aren't actually needed to achieved the desired result (at the higher levels, it's much easier for third party pure Python modules like requests to step in and fill the gap, as long as the core SSL infrastructure has been updated) - dropped the sha module from the exemption and explicitly noted that both the sha and md5 modules have been deprecated since Python 2.5 - conditional exemptions are now only granted for things that the security infrastructure depends on, rather than the other way around - answered the OpenSSL question based on MAL and Ned's feedback - added the "2.7-legacy-ssl" branch to the proposal (along with some pointed comments about the development priorities implied by preferring this branch to the one with the improved security infrastructure) - made it even clearer that I consider the situation commercial redistributor's responsibility to clean up, based on the support commitments they have made to their users. The PEP is primarily about changing one specific aspect of our maintenance policies so that the onus is clearly pushed back on them to fill the gap between what their customers want and what upstream volunteers are prepared to do for free - we shouldn't give them the opportunity to hide behind the excuse that we won't let them fix it upstream. Cheers, Nick. ==================================== PEP: 466 Title: Network Security Enhancement Exception for Python 2.7 Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan , Status: Draft Type: Informational Content-Type: text/x-rst Created: 23-Mar-2014 Post-History: 23-Mar-2014 Abstract ======== Most CPython tracker issues are classified as errors in behaviour or proposed enhancements. Most patches to fix behavioural errors are applied to all active maintenance branches. Enhancement patches are restricted to the default branch that becomes the next Python version. This cadence works reasonably well during Python's normal 18-24 month feature release cycle, which is still applicable to the Python 3 series. However, the age of the standard library in Python 2 has now reached a point where it is sufficiently far behind the state of the art in network security protocols for it to be causing real problems in commercial use cases where upgrading to Python 3 in the near term may not be practical. Accordingly, this PEP relaxes the normal restrictions by allowing enhancements to be applied in Python 2.7 maintenance releases for standard library components that have implications for the overall security of the internet. In particular, the exception will apply to: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the components of the ``random`` and ``os`` modules that the above modules rely on for cryptographic randomness * the version of OpenSSL bundled with the binary installers for Windows and Mac OS X Changes to these modules will still need to undergo normal backwards compatibility assessments, but otherwise they will be kept entirely aligned with the latest feature release of their Python 3 counterparts. This is designed to make it easier to implement secure networked software in Python, even for software that currently needs to remain compatible with the Python 2 series (which includes a lot of network infrastructure software). The development branches will be arranged in such a way that even though any further Python 2.7 releases published by the core development team provide updated network security infrastructure, it will remain possible for downstream redistributors to publish "Python 2.7 with legacy SSL infrastructure" if they choose to do so. While this PEP does not make any changes to the core development team's handling of security-fix-only branches that are no longer in active maintenance, it *does* recommend that commercial redistributors providing extended support periods for the Python standard library either adopt a similar approach to ensuring that the secure networking infrastructure keeps pace with the evolution of the internet, or else disclaim support for the use of older versions in roles that involving connecting directly to the public internet. Exemption Policy ================ Under this policy, the following network security related modules are granted a blanket exemption to the restriction against adding new features in maintenance releases, for the purpose of keeping their APIs aligned with their counterparts in the latest feature release of Python 3: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module This exemption applies to *all* proposals to backport backwards compatible changes in these modules to Python 2.7 maintenance releases. This choice is made deliberately to ensure that the "feature or fix?" argument isn't simply replaced by a "security related or not?" argument. These particular modules are inherently security related, and all enhancements to them improve Python's capabilities as a platform for development of secure networked software. As part of this policy, permission is also granted to upgrade to newer feature releases of OpenSSL when preparing the binary installers for new maintenance releases of CPython. Note that the ``sha`` and ``md5`` modules are not covered by this policy, but have been deprecated in favour of ``hashlib`` since Python 2.5 and have been removed entirely in the Python 3 series. In addition to the above blanket exemption, a conditional exemption is granted for these modules that may include some network security related features: * the ``os`` module (specifically ``os.urandom``) * the ``random`` module This more limited exemption for these modules requires that the *specific* enhancement being proposed for backporting needs to be justified as being network security related. This is generally restricted to cases where the feature in question is needed by an update to one of the modules covered by the blanket exemption. Backwards Compatibility Considerations ====================================== This PEP does *not* grant any general exemptions to the usual backwards compatibility policy for maintenance releases. Instead, by explicitly encouraging the use of feature based checks and explicitly opting in to less secure configurations, it is designed to make it easier to provide more "secure by default" behaviour in future feature releases, while still limiting the risk of breaking currently working software when upgrading to a new maintenance release. In *all* cases where this policy is applied to backport enhancements to maintenance releases, it MUST be possible to write cross-version compatible code that operates by "feature detection" (for example, checking for particular attributes in the module), without needing to explicitly check the Python version. It is then up to library and framework code to provide an appropriate warning and fallback behaviour if a desired feature is found to be missing. While some especially security sensitive software MAY fail outright if a desired security feature is unavailable, most software SHOULD instead emit a warning and continue operating using a slightly degraded security configuration. Affected APIs SHOULD be designed to allow library and application code to perform the following actions after detecting the presence of a relevant network security related feature: * explicitly opt in to more secure settings (to allow the use of enhanced security features in older maintenance releases of Python) * explicitly opt in to less secure settings (to allow the use of newer Python feature releases in lower security environments) * determine the default setting for the feature (this MAY require explicit Python version checks to determine the Python feature release, but MUST NOT require checking for a specific maintenance release) Security related changes to other modules (such as higher level networking libraries and data format processing libraries) will continue to be made available as backports and new modules on the Python Package Index, as independent distribution remains the preferred approach to handling software that must continue to evolve to handle changing development requirements independently of the Python 2 standard library. Refer to the `Motivation and Rationale`_ section for a review of the characteristics that make the secure networking infrastructure worthy of special consideration. OpenSSL compatibility --------------------- Under this policy, OpenSSL may be upgraded to more recent feature releases in Python 2.7 maintenance releases. On Linux and most other POSIX systems, the specific version of OpenSSL used already varies, as Python dynamically links to the system provided OpenSSL library by default. For the Windows binary installers, the ``_ssl`` and ``_hashlib`` modules are statically linked with OpenSSL and the associated symbols are not exported. Marc-Andre Lemburg indicates that updating to newer OpenSSL releases in the ``egenix-pyopenssl`` binaries has not resulted in any reported compatibility issues [3]_ The Mac OS X binary installers historically followed the same policy as other POSIX installations and dynamically linked to the Apple provided OpenSSL libraries. However, Apple has now ceased updating these cross-platform libraries, instead requiring that even cross-platform developers adopt Mac OS X specific interfaces to access up to date security infrastructure on their platform. Accordingly, and independently of this PEP, the Mac OS X binary installers were already going to be switched to statically linker newer versions of OpenSSL [4]_ Other Considerations ==================== Maintainability --------------- This policy does NOT represent a commitment by volunteer contributors to actually backport network security related changes from the Python 3 series to the Python 2 series. Rather, it is intended to send a clear signal to potential corporate contributors that the core development team are willing to accept offers of corporate assistance in putting this policy into effect and handling the resulting increase in the Python 2 maintenance load. Backporting security related fixes and enhancements to earlier versions is a common service for commercial redistributors to offer to their customers. This policy represents an explicit invitation to contribute some of those changes back to the upstream community in cases where they are likely to have a broad impact that helps improve the security of the internet as a whole, rather than sitting back and waiting for unpaid volunteers to do it for them. Documentation ------------- All modules covered by this policy MUST include a "Security Considerations" section in their documentation. In addition to any other module specific contents, this section MUST enumerate key security enhancements and fixes (with CVE identifiers where applicable), along with the feature and maintenance releases that first included them. Security releases ----------------- This PEP does not propose any changes to the handling of security releases - those will continue to be source only releases that include only critical security fixes. However, the recommendations for library and application developers are deliberately designed to accommodate commercial redistributors that choose to apply this policy to additional Python release series that are either in security fix only mode, or have been declared "end of life" by the core development team. Whether or not redistributors choose to exercise that option will be up to the individual redistributor. Integration testing ------------------- Third party integration testing services should offer users the ability to test against both the latest Python 2.7 maintenance release and the latest "Python 2.7 with legacy SSL infrastructure" release, to ensure that libraries, frameworks and applications handle the legacy security infrastructure correctly (either failing or degrading gracefully, depending on the security sensitivity of the software). Handling lower security environments with low risk tolerance ------------------------------------------------------------ For better or for worse (mostly worse), there are some environments where the risk of latent security defects is more tolerated than the risk of regressions in maintenance releases. This policy largely excludes these environments from consideration where the modules covered by the exemption are concerned. However, one concession is made for the benefit of such environments: while the main ``2.7`` branch in Mercurial will include the updated network security infrastructure, the development process will be updated to include a ``2.7-legacy-ssl`` branch with the more limited feature set of the original 2.7 network security infrastructure, allowing downstream redistributors to continue to provide Python 2.7 with the legacy SSL infrastructure if they choose to do so. This branch will be tested on the CPython continuous integration infrastructure, but no releases will be made from it by the core development team. As noted above, corporate redistributors and users are expected to provide the additional development effort needed to make this practical. It is not acceptable to expect volunteer contributors to resolve a problem created largely by conservative corporate development practices. Evolution of this Policy ======================== The key requirement for a module to be considered for inclusion in this policy (whether under a blanket or conditional exemption) is that it must have security implications *beyond* the specific application that is written in Python and the system that application is running on. Thus the focus on network security protocols and related cryptographic infrastructure - Python is a popular choice for the development of web services and clients, and thus the capabilities of widely used Python versions have implications for the security design of other services that may themselves be using newer versions of Python or other development languages, but need to interoperate with clients or servers written using older versions of Python. The intent behind this requirement is to minimise any impact that the introduction of this policy may have on the stability and compatibility of maintenance releases. It would be thoroughly counterproductive if end users became as cautious about updating to new Python maintenance releases as they are about updating to new feature releases. Motivation and Rationale ======================== This PEP can be seen as a more targeted version of the "faster standard library release cycle" proposals discussed in PEP 407 and PEP 413, focusing specifically on those areas which have implications beyond the Python community. Background ---------- The creation of this PEP was prompted primarily by the aging SSL support in the Python 2 series. As of March 2014, the Python 2.7 SSL module is approaching four years of age, and the SSL support in the still popular Python 2.6 release had its feature set locked six years ago. These are simply too old to provide a foundation that can be recommended in good conscience for secure networking software that operates over the public internet, especially in an era where it is becoming quite clearly evident that advanced persistent security threats are even more widespread and more indiscriminate in their targeting than had previously been understood. While they represented reasonable security infrastructure in their time, the state of the art has moved on, and we need to investigate mechanisms for effectively providing more up to date network security infrastructure for users that, for whatever reason, are not currently in a position to migrate to Python 3. While the use of the system OpenSSL installation addresses many of these concerns on Linux platforms, it doesn't address all of them, and in the case of the binary installers for Windows and Mac OS X that are published on python.org, the version of OpenSSL used is entirely within the control of the Python core development team, and currently limited to OpenSSL maintenance releases for the version initially shipped with the corresponding Python feature release. With increased popularity comes increased responsibility, and this policy aims to acknowledge the fact that Python's popularity and adoption has now reached a level where some of our design and policy decisions have significant implications beyond the Python development community. As one example, the Python 2 ``ssl`` module does not support the Server Name Identification standard. While it is possible to obtain SNI support by using the third party ``requests`` client library, actually doing so currently requires using not only ``requests`` and its embedded dependencies, but also half a dozen or more additional libraries. The lack of support in the Python 2 series thus serves as an impediment to making effective use of SNI on servers, as Python 2 clients will frequently fail to handle it correctly. Another more critical example is the lack of SSL hostname matching in the Python 2 standard library - it is currently necessary to rely on a third party library, such as ``requests`` or ``backports.ssl_match_hostname`` to obtain that functionality in Python 2. The Python 2 series also remains more vulnerable to remote timing attacks on security sensitive comparisons than the Python 3 series, as it lacks a standard library equivalent to the timing attack resistant ``hmac.compare_digest()`` function. While appropriate secure comparison functions can be implemented in third party extensions, may users don't even consider the issue and use ordinary equality comparisons instead - while a standard library solution doesn't automatically fix that problem, it *does* make the barrier to resolution much lower once the problem is pointed out. My position on the ongoing transition from Python 2 to Python 3 has long been that Python 2 remains a supported platform for the core development team, and that commercial support will remain available well after upstream maintenance ends. However, in the absence of this network security enhancement policy, that position is difficult to justify when it comes to software that operates over the public internet. Just as many developers consider it too difficult to develop truly secure modern networked software in C/C++ (largely due to the challenges associated with manual memory management), I anticipate that in the not too distant future, it will be considered too difficult to develop truly secure modern networked software using the Python 2 series (some developers would argue that we have already reached that point). Alternative: advise developers of networked software to migrate to Python 3 --------------------------------------------------------------------------- This alternative represents the status quo. Unfortunately, it has proven to be unworkable in practice, as the backwards compatibility implications mean that this is a non-trivial migration process for large applications and integration projects. While the tools for migration have evolved to a point where it is possible to migrate even large applications opportunistically and incrementally (rather than all at once) by updating code to run in the large common subset of Python 2 and Python 3, using the most recent technology often isn't a priority in commercial environments. Previously, this was considered an acceptable harm, as while it was an unfortunate problem for the affected developers to have to face, it was seen as an issue between them and their management chain to make the case for infrastructure modernisation, and this case would become naturally more compelling as the Python 3 series evolved. However, now that we're fully aware of the impact the limitations of the Python 2 standard library may be having on the evolution of internet security standards, I no longer believe that it is reasonable to expect platform and application developers to resolve all of the latent defects in an application's Unicode correctness solely in order to gain access to the network security enhancements already available in Python 3. While Ubuntu (and to some extent Debian as well) are committed to porting all default system services and scripts to Python 3, and to removing Python 2 from its default distribution images (but not from its archives), this is a mammoth task and won't be completed for the Ubuntu 14.04 LTS release (at least for the desktop image - it may be achieved for the mobile and server images). Fedora has even more work to do to migrate, and it will take a non-trivial amount of time to migrate the relevant infrastructure components. While Red Hat are also actively working to make it easier for users to use more recent versions of Python on our stable platforms, it's going to take time for those efforts to start having an impact on end users' choice of version, and any such changes also don't benefit the core platform infrastructure that runs in the integrated system Python by necessity. The OpenStack migration to Python 3 is also still in its infancy, and even though that's a project with an extensive and relatively robust automated test suite, it's still large enough that it is going to take quite some time to migrate. And that's just three of the highest profile open source projects that make heavy use of Python. Given the likely existence of large amounts of legacy code that lacks the kind of automated regression test suite needed to help support a migration from Python 2 to Python 3, there are likely to be many cases where reimplementation (perhaps even in Python 3) proves easier than migration. The key point of this PEP is that those situations affect more people than just the developers and users of the affected application: the existence of clients and servers with outdated network security infrastructure becomes something that developers of secure networked services need to take into account as part of their security design, and that's a problem that inhibits the adoption of better security standards. As Terry Reedy noted, if we try to persist with the status quo, the likely outcome is that commercial redistributors will attempt to do something like this on behalf of their customers *anyway*, but in a potentially inconsistent and ad hoc manner. By drawing the scope definition process into the upstream project we are in a better position to influence the approach taken to address the situation and to help ensure some consistency across redistributors. The problem is real, so *something* needs to change, and this PEP describes my currently preferred approach to addressing the situation. Alternative: create and release Python 2.8 ------------------------------------------ With sufficient corporate support, it likely *would* be possible to create and release Python 2.8 (it's highly unlikely such a project would garner enough interest to be achievable with only volunteers). However, this wouldn't actually solve the problem, as the aim is to provide a *relatively low impact* way to incorporate enhanced security features into integrated products and deployments that make use of Python 2. Upgrading to a new Python feature release would mean both more work for the core development team, as well as a more disruptive update that most potential end users would likely just skip entirely. Attempting to create a Python 2.8 release would also bring in suggestions to backport many additional features from Python 3 (such as ``tracemalloc`` and the improved coroutine support), making the migration from Python 2.7 to this hypothetical 2.8 release even riskier and more disruptive. This is not a recommended approach, as it would involve substantial additional work for a result that is actually less effective in achieving the original aim (which is to eliminate the current widespread use of the aging network security infrastructure in the Python 2 series). Alternative: distribute the security enhancements via PyPI ---------------------------------------------------------- While this initially appears to be an attractive and easier to manage approach, it actually suffers from several significant problems. Firstly, this is complex, low level, cross-platform code that integrates with the underlying operating system across a variety of POSIX platforms (including Mac OS X) and Windows. The CPython BuildBot fleet is already set up to handle continuous integration in that context, but most of the freely available continuous integration services just offer Linux, and perhaps paid access to Windows. Those services work reasonably well for software that largely runs on the abstraction layers offered by Python and other dynamic languages, as well as the more comprehensive abstraction offered by the JVM, but won't suffice for the kind of code involved here. The OpenSSL dependency for the network security support also qualifies as the kind of "complex binary dependency" that isn't yet handled well by the ``pip`` based software distribution ecosystem. Relying on a third party binary dependency also creates potential compatibility problems for ``pip`` when running on other interpreters like ``PyPy``. Another practical problem with the idea is the fact that ``pip`` itself relies on the ``ssl`` support in the standard library (with some additional support from a bundled copy of ``requests``, which in turn bundles ``backport.ssl_match_hostname``), and hence would require any replacement module to also be bundled within ``pip``. This wouldn't pose any insurmountable difficulties (it's just another dependency to vendor), but it *would* mean yet another copy of OpenSSL to keep up to date. This approach also has the same flaw as all other "improve security by renaming things" approaches: they completely miss the users who most need help, and raise significant barriers against being able to encourage users to do the right thing when their infrastructure supports it (since "use this other module" is a much higher impact change than "turn on this higher security setting"). Deprecating the aging SSL infrastructure in the standard library in favour of an external module would be even more user hostile than accepting the slightly increased risk of regressions associated with upgrading it in place. Last, but certainly not least, this approach suffers from the same problem as the idea of doing a Python 2.8 release: likely not solving the actual problem. Commercial redistributors of Python are set up to redistribute *Python*, and a pre-existing set of additional packages. Getting new packages added to the pre-existing set *can* be done, but means approaching each and every redistributor and asking them to update their repackaging process accordingly. By contrast, the approach described in this PEP would require redistributors to deliberately *opt out* of the security enhancements (by switching to redistributing directly from the ``2.7-legacy-ssl`` branch rather than the main ``2.7`` branch), which most of them are unlikely to do. Open Questions ============== * I believe I've addressed all the technical and scope questions I had, or others raised. That just leaves the question of "If we agree to this plan, who is actually going to handle all the extra work involved?" :) Disclosure of Interest ====================== The author of this PEP currently works for Red Hat on test automation tools. If this proposal is accepted, I will be strongly encouraging Red Hat to take advantage of the resulting opportunity to help improve the overall security of the Python ecosystem. However, I do not speak for Red Hat in this matter, and cannot make any commitments on Red Hat's behalf. Acknowledgements ================ Thanks to Christian Heimes for his recent efforts on greatly improving Python's SSL support in the Python 3 series, and a variety of members of the Python community for helping me to better understand the implications of the default settings we provide in our SSL modules, and the impact that tolerating the use of SSL infrastructure that was defined in 2010 (Python 2.7) or even 2008 (Python 2.6) potentially has for the security of the web as a whole. Christian and Donald Stufft also provided valuable feedback on a preliminary draft of this proposal. Thanks also to participants in the python-dev mailing list threads [1,2]_ References ========== .. [1] https://mail.python.org/pipermail/python-dev/2014-March/133334.html .. [2] https://mail.python.org/pipermail/python-dev/2014-March/133389.html .. [3] https://mail.python.org/pipermail/python-dev/2014-March/133438.html .. [4] https://mail.python.org/pipermail/python-dev/2014-March/133347.html Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From skip at pobox.com Mon Mar 24 14:49:15 2014 From: skip at pobox.com (Skip Montanaro) Date: Mon, 24 Mar 2014 08:49:15 -0500 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <20140323200312.33b6b0b1@limelight.wooz.org> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> Message-ID: On Sun, Mar 23, 2014 at 7:03 PM, Barry Warsaw wrote: > I want to stick to our no-Python-2.8 pledge .... I don't understand this dogmatic adherence to a "no 2.8 pledge." Back when it was made, I don't think the issue of SSL vulnerabilities was understood (at least not to the current level). Now we're in a pickle. We need to find some way to release something that brings the SSL (and related) feature set up-to-date. In this thread and Nick's PEP have read about a few ways this might be accomplished (I probably missed a couple): * separate the security-dependent bits out into a separate "semi"-stdlib which would be served from PyPI * continue with 2.7.x but admit that there will be some possible backward incompatibilities with earlier 2.7 versions in the affected modules * change the name somehow and jump to two-digit micro version numbers It seems to me the PyPI option should be a non-starter, as it's unlikely that you will get the bulk of the 2.7 installations to update to that version, and if people do install it, they wind up with possible backward incompatibilities. The second option goes against the drop-in history of micro releases. The third seems just like "weasel words" to avoid saying "2.8." Also, a careful reading of the Zen of Python suggests these solutions might violate a number of its aphorisms (explicit is better than implicit, simple is better than complex, practicality beats purity). So what's the big deal? Why can't we be pragmatic and call this thing (whatever it turns out to be) 2.8? Is this pledge and its rationale written down in a PEP somewhere, so I can study the reasons behind what appears at this point to be blind adherence? Did someone administer a blood oath at a recent PyCon? Pledge-be-damned-ly y'rs, Skip From ncoghlan at gmail.com Mon Mar 24 15:11:28 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Mar 2014 00:11:28 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> Message-ID: On 24 March 2014 23:49, Skip Montanaro wrote: > So what's the big deal? Why can't we be pragmatic and call this thing > (whatever it turns out to be) 2.8? Is this pledge and its rationale > written down in a PEP somewhere, so I can study the reasons behind > what appears at this point to be blind adherence? There's a ton of work involved in creating a new feature release, and there's no way we're going to go through that much drudgery for the legacy Python 2 series without someone paying for it. Even the more limited network security infrastructure update proposal in this PEP is conditional on corporate contributors agreeing to do the heavy lifting. > Did someone > administer a blood oath at a recent PyCon? As discussed in the PEP, a Python 2.8 release wouldn't actually solve this problem, so there's no reason for commercial redistributors to contribute to making it happen. For example, RHEL7 and derivatives are already locked in to 2.7 until 2024, RHEL6 and derivatives are locked in to 2.6 until 2020. The only way to keep those combination of RHEL and the Python 2 standard library from holding back the evolution of internet security standards is to find a way solve the problem *within* the 2.7 line in such a way that I can then make the case for also backporting it to 2.6 in a RHEL6 point release. I think the proposal currently in round 3 of the PEP is something I can sell to the Platform Engineering team as a necessary update to make in a relatively timely fashion (I don't think the situation is critical yet, but give it another year or two and I'll be a lot more concerned). By contrast, trying to handle this via a Python 2.8 release would mean I would still have to advocate for us to adopt the policy in the PEP independently of upstream, and that's not a good outcome for anyone. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From skip at pobox.com Mon Mar 24 15:18:49 2014 From: skip at pobox.com (Skip Montanaro) Date: Mon, 24 Mar 2014 09:18:49 -0500 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> Message-ID: On Mon, Mar 24, 2014 at 9:11 AM, Nick Coghlan wrote: > For example, RHEL7 and derivatives are already locked in to 2.7 until > 2024, RHEL6 and derivatives are locked in to 2.6 until 2020. The only > way to keep those combination of RHEL and the Python 2 standard > library from holding back the evolution of internet security standards > is to find a way solve the problem *within* the 2.7 line in such a way > that I can then make the case for also backporting it to 2.6 in a > RHEL6 point release. Thanks for the explanation. I'm still a bit confused though. If there are backward compatibility issues with the proposed changes (whatever they turn out to be), are the commercial redistributors still going to balk at releasing these changes to their customers? From the reading I've done (this thread and your second iteration of the PEP), it seems like application developers will have to make some changes to take advantage of these updated security bits. Is there some path forward that really makes everything a drop-in improvement, requiring no change to application code, and breaking nothing that already works? Skip From rdmurray at bitdance.com Mon Mar 24 15:21:50 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 24 Mar 2014 10:21:50 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <20140323213112.1b7d2e32@limelight.wooz.org> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> Message-ID: <20140324142151.6F1D1250049@webabinitio.net> On Sun, 23 Mar 2014 21:31:12 -0400, Barry Warsaw wrote: > On Mar 24, 2014, at 11:38 AM, Chris Angelico wrote: > > >Easy. Just set PYTHONPATH to import the SEPython [1] lib ahead of the > >standard lib. Then you can go back to the standard 2.7 (if you want > >to) by unsetting PYTHONPATH. > > > >It'd be nice if SEPython defined a modified sys.version for clarity, > >but otherwise, it'd be a vanilla Python 2.7. > > That's certainly more in the direction of what I think is an appropriate > upstream solution. > > The thing is, there isn't one single "what's best for users" resolution. > There are many, many competing requirements and I think it will be difficult > to satisfy everyone. I'm particularly sensitive to complaints of unexpected > changes between micro releases. In the context of that last sentence, I think it is worth noting the stance that 3.4 is taking[*] about security backward compatibility, since many people may not be aware of it (we only just finished making the documentation clear). If you use create_default_context() to get your context object, you get a "best practices" level of security *that may change between maintenance releases*. If you want things to not change between maintenance releases, you create your own context object and set its controls appropriately. In other words, the programmer opts in to maintenance release security improvements by using create_default_context. I presume that whatever comes out of this PEP will use the same approach. Note: thanks again to Christian Heimes and Antoine Pitrou for this work. Without Christian's work, I think we wouldn't even be having this conversation. Antoine's earlier work laid essential groundwork, but by itself I'm not sure that would have been enough to result in calls for a backport. It took both of them, with some help from others as well. --David [*] I actually don't know if this was discussed on python-dev previously because I've got a backlog of messages I'm not caught up on. If it hasn't been, then doubly good to mention it now, since the first 3.4 maintenance release hasn't happened yet :) From ncoghlan at gmail.com Mon Mar 24 15:36:28 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Mar 2014 00:36:28 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> Message-ID: On 25 March 2014 00:18, Skip Montanaro wrote: > On Mon, Mar 24, 2014 at 9:11 AM, Nick Coghlan wrote: >> For example, RHEL7 and derivatives are already locked in to 2.7 until >> 2024, RHEL6 and derivatives are locked in to 2.6 until 2020. The only >> way to keep those combination of RHEL and the Python 2 standard >> library from holding back the evolution of internet security standards >> is to find a way solve the problem *within* the 2.7 line in such a way >> that I can then make the case for also backporting it to 2.6 in a >> RHEL6 point release. > > Thanks for the explanation. I'm still a bit confused though. If there > are backward compatibility issues with the proposed changes (whatever > they turn out to be), are the commercial redistributors still going to > balk at releasing these changes to their customers? From the reading > I've done (this thread and your second iteration of the PEP), it seems > like application developers will have to make some changes to take > advantage of these updated security bits. Is there some path forward > that really makes everything a drop-in improvement, requiring no > change to application code, and breaking nothing that already works? The PEP does not permit backwards compatibility breaks in maintenance releases, thus people are solely concerned about the increased risk of regressions created by backporting all of the Python 3 enhancements to these modules to Python 2.7 and then ensuring that the default behaviour remains the same as earlier Python 2.7 releases. I think those folks have their priorities back to front, hence http://www.python.org/dev/peps/pep-0466/#handling-lower-security-environments-with-low-risk-tolerance :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Mar 24 15:40:59 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Mar 2014 00:40:59 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> Message-ID: On 25 March 2014 00:36, Nick Coghlan wrote: > On 25 March 2014 00:18, Skip Montanaro wrote: > The PEP does not permit backwards compatibility breaks in maintenance > releases Well, ssl.create_default_context() will use the same "this is a dynamic best practices API" policy as it does in 3.4. But everything that already exists in 2.7.6 will remain compatible with that behaviour by default. The policy change in the PEP won't automagically make networked software secure - it will just make it a heck of a lot *easier* to write in Python 2 (or the common subset of Python 2 and Python 3) by backporting the Python 3.4 enhancements to a 2.7 maintenance release, and, depending on the relative timing of the releases, likely the Python 3.5 enhancements to a later maintenance release. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From cory at lukasa.co.uk Mon Mar 24 10:28:08 2014 From: cory at lukasa.co.uk (Cory Benfield) Date: Mon, 24 Mar 2014 09:28:08 +0000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> Message-ID: On 24 March 2014 08:44, Nick Coghlan wrote: > The position I am coming to is that the "enhanced security" release should > be the default one that we publish binary installers for, but we should also > ensure that downstream redistributors can easily do "Python 2.7 with legacy > SSL" releases if they so choose. I'm happier forcing end users to rely on a > redistributor to opt in to a lower security option than I am to knowingly > publish too many additional releases with network security infrastructure > that is (at best) rapidly approaching its use by date. I'm with you here Nick, the default should really be "Python 2.7 with working TLS", not "Python 2.7 with broken TLS" (my only-slightly tongue-in-cheek proposed names for the releases). My concern with having Chris' proposal of an opt-in SEPython release is about discovery. I suspect most people will never find out about the SEPython release, meaning that most people will remain on insecure Python 2.7 distributions. If that happens, requests is essentially required to continue to support our current TLS model for 2.7 for as long as we support 2.7, and this PEP would give us no benefit at all. We're obviously not the only stakeholder here, but it's worth noting. Cory From solipsis at pitrou.net Mon Mar 24 17:10:23 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 24 Mar 2014 17:10:23 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <20140324142151.6F1D1250049@webabinitio.net> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> <20140324142151.6F1D1250049@webabinitio.net> Message-ID: Le 24/03/2014 15:21, R. David Murray a ?crit : > > In the context of that last sentence, I think it is worth noting the > stance that 3.4 is taking[*] about security backward compatibility, > since many people may not be aware of it (we only just finished making > the documentation clear). > > If you use create_default_context() to get your context object, you get a > "best practices" level of security *that may change between maintenance > releases*. If you want things to not change between maintenance releases, > you create your own context object and set its controls appropriately. Indeed. Note that this works because create_default_context() is a new API, hence it was ok to choose this particular maintenance policy. Maintenance policy of 3.4 as a whole (i.e. all other APIs) hasn't changed. (but some other aspects of SSL configuration, e.g. the default cipher list, is also amenable to changes in bugfix releases, as Donald's latest commits exemplify; in this case it should stay within the limits of reasonable backwards compatibility, i.e. not break any common use case) Regards Antoine. From nad at acm.org Mon Mar 24 18:23:37 2014 From: nad at acm.org (Ned Deily) Date: Mon, 24 Mar 2014 10:23:37 -0700 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 References: <532FF67A.3010206@egenix.com> <5330276B.4080600@egenix.com> Message-ID: In article , Nick Coghlan wrote: > You also reminded me that I need to dig around for and reference Ned's > email about the status of OS X and reference that (OpenSSL upgrades > were a casualty of Apple's anti-GPL crusade, so the OS X installers > were switched to static linking somewhere along the line). AFAIK, Apple's decision to deprecate OpenSSL has nothing to do with the GPL, since OpenSSL isn't GPL-licensed, but rather with OpenSSL API compatibility issues: http://rentzsch.tumblr.com/post/33696323211/wherein-i-write-apples-techno te-about-openssl-on-os-x -- Ned Deily, nad at acm.org From mal at egenix.com Mon Mar 24 18:36:02 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 24 Mar 2014 18:36:02 +0100 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: References: <532FF67A.3010206@egenix.com> <5330276B.4080600@egenix.com> Message-ID: <53306D02.8050702@egenix.com> On 24.03.2014 18:23, Ned Deily wrote: > In article > , > Nick Coghlan wrote: >> You also reminded me that I need to dig around for and reference Ned's >> email about the status of OS X and reference that (OpenSSL upgrades >> were a casualty of Apple's anti-GPL crusade, so the OS X installers >> were switched to static linking somewhere along the line). > > AFAIK, Apple's decision to deprecate OpenSSL has nothing to do with the > GPL, since OpenSSL isn't GPL-licensed, but rather with OpenSSL API > compatibility issues: > > http://rentzsch.tumblr.com/post/33696323211/wherein-i-write-apples-techno > te-about-openssl-on-os-x What a strange reasoning. Do they really believe that ABIs don't change when bumping the library version from 0.9.8 to 1.0.0 ? OpenSSL's history w/r to backwards compatibility up until 0.9.7 wasn't the greatest, but since 0.9.8 it has gotten to a level that's very reliable. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 24 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-03-29: PythonCamp 2014, Cologne, Germany ... 5 days to go 2014-04-09: PyCon 2014, Montreal, Canada ... 16 days to go 2014-04-29: Python Meeting Duesseldorf ... 36 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From Nikolaus at rath.org Mon Mar 24 18:56:05 2014 From: Nikolaus at rath.org (Nikolaus Rath) Date: Mon, 24 Mar 2014 10:56:05 -0700 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: (Nick Coghlan's message of "Sun, 23 Mar 2014 17:07:24 +1000") References: Message-ID: <874n2n5vm2.fsf@rath.org> Nick Coghlan writes: > Maintainability > --------------- > > This policy does NOT represent a commitment by volunteer contributors to > actually backport network security related changes from the Python 3 series > to the Python 2 series. Rather, it is intended to send a clear signal to > potential corporate contributors that the core development team are willing > to review and merge corporate contributions that put this policy into > effect. As I understand, at least for smaller patches it is actually more work to apply a patch than than to write it. With that in mind, are there really sufficient volunteer resources available to review and merge these corporate contributions if they come? The issue tracker certainly does not lack issues with unreviewed and/or unapplied patches... Best, -Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ?Time flies like an arrow, fruit flies like a Banana.? From kristjan at ccpgames.com Mon Mar 24 10:25:58 2014 From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=) Date: Mon, 24 Mar 2014 09:25:58 +0000 Subject: [Python-Dev] cpython (3.3): Make the various iterators' "setstate" sliently and consistently clip the In-Reply-To: <532F8B5E.4080003@hastings.org> References: <532F8B5E.4080003@hastings.org> Message-ID: Hi there. I didn?t see the original email in python-dev, sorry about that. The ?setstate? of the iterators is primarily used when unpickling them. This is code that was added during the PyCon sprints 2012, IIRC. Some iterators already did the silent clipping. One did not (rangeiter), it raised a valueerror, but it did so at the wrong index, so that an iterator could not be set to the ?exhausted? state. Others did no checking, allowing the value to be set to an state that would cause undefined behavior. The change is to prevent the last case. It is there purely for paranoid reasons. There should be no reason why a iterator should be unpickled such that its range and position would be mismatching and no reason to bloat the code with diagnostic error code for that, but still, guarding us from undefined states is essential. If you think I should be adding exceptions for this, then I can do that. The reason this didn?t go through the tracker is that this is code from myself and the Stackless sprint that didn?t itself go through the tracker at the time. There really Is no one more qualified to verify this code than myself ? K From: Larry Hastings [mailto:larry at midwinter.com] On Behalf Of Larry Hastings Sent: 24. mars 2014 01:33 To: Kristj?n Valur J?nsson Subject: Fwd: Re: [Python-Dev] cpython (3.3): Make the various iterators' "setstate" sliently and consistently clip the Still no reply on this...? I'd like to see your answer too. /arry -------- Original Message -------- Subject: Re: [Python-Dev] cpython (3.3): Make the various iterators' "setstate" sliently and consistently clip the Date: Sat, 08 Mar 2014 08:01:23 +0100 From: Georg Brandl To: python-dev at python.org Am 06.03.2014 09:02, schrieb Serhiy Storchaka: > 05.03.14 17:24, kristjan.jonsson ???????(??): >> http://hg.python.org/cpython/rev/3b2c28061184 >> changeset: 89477:3b2c28061184 >> branch: 3.3 >> parent: 89475:24d4e52f4f87 >> user: Kristj?n Valur J?nsson >> date: Wed Mar 05 13:47:57 2014 +0000 >> summary: >> Make the various iterators' "setstate" sliently and consistently clip the >> index. This avoids the possibility of setting an iterator to an invalid >> state. > > Why indexes are silently clipped instead of raising an exception? > >> files: >> Lib/test/test_range.py | 12 ++++++++++ >> Modules/arraymodule.c | 2 + >> Objects/bytearrayobject.c | 10 ++++++-- >> Objects/bytesobject.c | 10 ++++++-- >> Objects/listobject.c | 2 + >> Objects/rangeobject.c | 31 +++++++++++++++++++++++--- >> Objects/tupleobject.c | 4 +- >> Objects/unicodeobject.c | 10 ++++++-- >> 8 files changed, 66 insertions(+), 15 deletions(-) > > And it would be better first discuss and review such large changes on > the bugtracker. Agreed. Kristjan, could you please explain a bit more about this change and use the tracker in the future? Georg _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/larry%40hastings.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Mar 24 20:37:41 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 25 Mar 2014 06:37:41 +1100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> Message-ID: On Mon, Mar 24, 2014 at 8:28 PM, Cory Benfield wrote: > On 24 March 2014 08:44, Nick Coghlan wrote: >> The position I am coming to is that the "enhanced security" release should >> be the default one that we publish binary installers for, but we should also >> ensure that downstream redistributors can easily do "Python 2.7 with legacy >> SSL" releases if they so choose. I'm happier forcing end users to rely on a >> redistributor to opt in to a lower security option than I am to knowingly >> publish too many additional releases with network security infrastructure >> that is (at best) rapidly approaching its use by date. > > I'm with you here Nick, the default should really be "Python 2.7 with > working TLS", not "Python 2.7 with broken TLS" (my only-slightly > tongue-in-cheek proposed names for the releases). My concern with > having Chris' proposal of an opt-in SEPython release is about > discovery. I suspect most people will never find out about the > SEPython release, meaning that most people will remain on insecure > Python 2.7 distributions. If that happens, requests is essentially > required to continue to support our current TLS model for 2.7 for as > long as we support 2.7, and this PEP would give us no benefit at all. > We're obviously not the only stakeholder here, but it's worth noting. The opting in could be done at the distro level. Red Hat could ship a thing called /usr/bin/python that runs SEPython, and that package could be identified and numbered in such a way that it's clearly a drop-in replacement for vanilla Python. If backward compatibility is done carefully (which, from everything I'm seeing here, is the way it's to be done), there should be absolutely no downside to using SEPython, except for portability (because now you're depending on it for security), and corner cases like testing. Alternatively, it can be done as a separate package: install "python" to get Python, and install "python-security-patches" or "python-tls" to get the tweaks - once that's installed, /usr/bin/python becomes SEPython. Programs can depend on that package to ensure security. That does have a partial discoverability problem (possibly even worse, as you could test on a computer that has x, y, z installed, and then deploy on a brand new computer, and nothing there ever depended on SEPython, so you're vulnerable without knowing why), but that could be ameliorated by simply advising people to type "yum install python-tls" if they want to run any Python programs that use TLS. But I'd be inclined to the first option; its biggest downside is that installing "python" installs something other than the "standard stdlib", which IMO isn't a big problem if (a) it's 100% backward compatible, and (b) it's semi-official anyway. ChrisA From ncoghlan at gmail.com Mon Mar 24 22:38:16 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Mar 2014 07:38:16 +1000 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: <874n2n5vm2.fsf@rath.org> References: <874n2n5vm2.fsf@rath.org> Message-ID: On 25 Mar 2014 04:00, "Nikolaus Rath" wrote: > > Nick Coghlan writes: > > Maintainability > > --------------- > > > > This policy does NOT represent a commitment by volunteer contributors to > > actually backport network security related changes from the Python 3 series > > to the Python 2 series. Rather, it is intended to send a clear signal to > > potential corporate contributors that the core development team are willing > > to review and merge corporate contributions that put this policy into > > effect. > > As I understand, at least for smaller patches it is actually more work > to apply a patch than than to write it. With that in mind, are there > really sufficient volunteer resources available to review and merge > these corporate contributions if they come? The issue tracker certainly > does not lack issues with unreviewed and/or unapplied patches... At least to start, this would likely be about seeking more upstream time for existing core contributors. Beyond that, PEP 462 covers another way for corporate users to give back - if they want to build massive commercial enterprises on our software, they can help maintain and upgrade the infrastructure that makes it possible in the first place. It's potentially worth reading some of the board candidate statements for this year, particularly mine and Van's: https://wiki.python.org/moin/PythonSoftwareFoundation/BoardCandidates2014 The lack of paid development time for CPython compared to similarly critical projects like the Linux kernel and OpenStack is of grave concern to me personally from a volunteer burnout perspective, and it was a problem at least Van and I were already specifically wanting to address over the next year or so. Over the course of writing the PEP I realised that the situation with the Python 2 network security modules is a perfect example of the kinds of problems that the current lack of upstream engagement and investment can cause. Cheers, Nick. > > > Best, > -Nikolaus > > -- > Encrypted emails preferred. > PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C > > ?Time flies like an arrow, fruit flies like a Banana.? > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Mar 25 00:04:29 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 24 Mar 2014 19:04:29 -0400 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: References: <874n2n5vm2.fsf@rath.org> Message-ID: On Mar 24, 2014, at 5:38 PM, Nick Coghlan wrote: > > On 25 Mar 2014 04:00, "Nikolaus Rath" wrote: > > > > Nick Coghlan writes: > > > Maintainability > > > --------------- > > > > > > This policy does NOT represent a commitment by volunteer contributors to > > > actually backport network security related changes from the Python 3 series > > > to the Python 2 series. Rather, it is intended to send a clear signal to > > > potential corporate contributors that the core development team are willing > > > to review and merge corporate contributions that put this policy into > > > effect. > > > > As I understand, at least for smaller patches it is actually more work > > to apply a patch than than to write it. With that in mind, are there > > really sufficient volunteer resources available to review and merge > > these corporate contributions if they come? The issue tracker certainly > > does not lack issues with unreviewed and/or unapplied patches... > > At least to start, this would likely be about seeking more upstream time for existing core contributors. > > Beyond that, PEP 462 covers another way for corporate users to give back - if they want to build massive commercial enterprises on our software, they can help maintain and upgrade the infrastructure that makes it possible in the first place. > > It's potentially worth reading some of the board candidate statements for this year, particularly mine and Van's: > > https://wiki.python.org/moin/PythonSoftwareFoundation/BoardCandidates2014 > > The lack of paid development time for CPython compared to similarly critical projects like the Linux kernel and OpenStack is of grave concern to me personally from a volunteer burnout perspective, and it was a problem at least Van and I were already specifically wanting to address over the next year or so. Over the course of writing the PEP I realised that the situation with the Python 2 network security modules is a perfect example of the kinds of problems that the current lack of upstream engagement and investment can cause. > > Cheers, > Nick. > > > > > > > Best, > > -Nikolaus > > > > -- > > Encrypted emails preferred. > > PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C > > > > ?Time flies like an arrow, fruit flies like a Banana.? > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io I'd like to just go on a brief tangent here. While I totally agree that it would be incredibly awesome if more companies put dedicated time into developing and maintaining CPython I don't think pushing all the blame on to them is accurate. The attitude towards security issues and backwards compatibility has a somewhat equal share in the causes of the aging security infrastructure of the 2.x line. Now this PEP, if accepted, does a lot to resolve the largest offenders of this policy (and there has been some signs lately that perhaps going forward this will be better) but I think it is not doing anyone a favor if we just point fingers *over there* and claim the fault lies with someone else doing or not doing something. I *don't* want to disparage anyone or anything of that like, mostly to say that while of course increased resources from corporate users would help the situation immensely but that additionally there is a reasonably sized contingent of influential members who still want to treat Python as a hobbyist project and not a critical piece of the infrastructure of the Internet as a whole. I *don't* want to get help from downstream users, especially on important but "boring" or hard issues such as security, and then have them feel shutdown and unable to actually get anything done as others who have attempted to resolve some of these issues in the past have had happen to them. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From AHettinger at Prominic.NET Mon Mar 24 23:51:34 2014 From: AHettinger at Prominic.NET (Andrew M. Hettinger) Date: Mon, 24 Mar 2014 17:51:34 -0500 Subject: [Python-Dev] RFE 20469: ssl.getpeercert() should include extensions Message-ID: I thought I'd wait until the 3.4 release before I bothered asking about this: http://bugs.python.org/issue20469 I don't think I'm qualified to actually be writing code for the ssl module, but is there anything else that I can do to help? I could probably put together a demonstration-case if that would be helpful. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Mar 25 01:13:00 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 24 Mar 2014 20:13:00 -0400 Subject: [Python-Dev] RFE 20469: ssl.getpeercert() should include extensions In-Reply-To: References: Message-ID: On 3/24/2014 6:51 PM, Andrew M. Hettinger wrote: > I thought I'd wait until the 3.4 release before I bothered asking about > this: http://bugs.python.org/issue20469 > > I don't think I'm qualified to actually be writing code for the ssl > module, but is there anything else that I can do to help? > > I could probably put together a demonstration-case if that would be > helpful. Yes. Comment on issue. 'test needed' is intentionally listed as the first stage. -- Terry Jan Reedy From tjreedy at udel.edu Tue Mar 25 01:58:10 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 24 Mar 2014 20:58:10 -0400 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: References: <874n2n5vm2.fsf@rath.org> Message-ID: On 3/24/2014 7:04 PM, Donald Stufft wrote: > > On Mar 24, 2014, at 5:38 PM, Nick Coghlan > wrote: >> Beyond that, PEP 462 covers another way for corporate users to give >> back - if they want to build massive commercial enterprises on our >> software, they can help maintain and upgrade the infrastructure that >> makes it possible in the first place. >> >> It's potentially worth reading some of the board candidate statements >> for this year, particularly mine and Van's: >> >> https://wiki.python.org/moin/PythonSoftwareFoundation/BoardCandidates2014 I read all of them. >> The lack of paid development time for CPython compared to similarly >> critical projects like the Linux kernel and OpenStack is of grave >> concern to me personally from a volunteer burnout perspective, I am glad to read that. Some of the expert professional core developers scoff at me being burned out from News Merge Hell and push race losses. >> and it >> was a problem at least Van and I were already specifically wanting to >> address over the next year or so. Over the course of writing the PEP I >> realised that the situation with the Python 2 network security modules >> is a perfect example of the kinds of problems that the current lack of >> upstream engagement and investment can cause. > I'd like to just go on a brief tangent here. > > While I totally agree that it would be incredibly awesome if more > companies put > dedicated time into developing and maintaining CPython I don't think pushing > all the blame on to them is accurate. For all I know, PSF has not yet asked in the right way, whatever that would be. > will be better) but I think it is not doing anyone a favor if we just point > fingers *over there* and claim the fault lies with someone else doing or not > doing something. I agree that we should better figure out what to go going forward. > I *don't* want to disparage anyone or anything of that like, mostly to > say that > while of course increased resources from corporate users would help the > situation > immensely but that additionally there is a reasonably sized contingent of > influential members who still want to treat Python as a hobbyist project and > not a critical piece of the infrastructure of the Internet as a whole. I find that surprising as I do not personally know any such people. To me, Python is both. My only objection is to corporatists who want to exclude amateur and hobbyist projects, for instance from PyPI (which I believe started as a hobbyist project). I personally would like someone paid full-time to upgrade the commit infrastructure, as soon possible. to make current committers more productive and make becoming a committer more attractive. Then I would like 2 people paid, one for doc issues, one to code, to work on the backlog of contributed patches. I know that are people who are not contributing any more because their previous contributions have sat unattended to. > I > *don't* want to get help from downstream users, especially on important but > "boring" or hard issues such as security, and then have them feel > shutdown and > unable to actually get anything done as others who have attempted to resolve > some of these issues in the past have had happen to them. Just from reading pydev, I am not familiar with such events and cannot comment. -- Terry Jan Reedy From ben at bendarnell.com Tue Mar 25 02:29:57 2014 From: ben at bendarnell.com (Ben Darnell) Date: Mon, 24 Mar 2014 21:29:57 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> Message-ID: On Mon, Mar 24, 2014 at 4:44 AM, Nick Coghlan wrote: > > On 24 Mar 2014 15:25, "Chris Angelico" wrote: > > > As has already been pointed out, this can already happen, but in an > > ad-hoc way. Making it official or semi-official would mean that a > > script written for Debian's "Python 2.7.10" would run on Red Hat's > > "Python 2.7.10", which would surely be an advantage. > > And having it break on the official Windows and Mac OS X binaries would > benefit end users, how? > > The position I am coming to is that the "enhanced security" release should > be the default one that we publish binary installers for, but we should > also ensure that downstream redistributors can easily do "Python 2.7 with > legacy SSL" releases if they so choose. I'm happier forcing end users to > rely on a redistributor to opt in to a lower security option than I am to > knowingly publish too many additional releases with network security > infrastructure that is (at best) rapidly approaching its use by date. > I am strongly against allowing downstream distributors that choice. Unless the security-enhanced variant of Python 2.7 quickly and completely overtakes all previous versions, we will be creating a compatibility problem between the two variants of Python 2.7. I believe that the changes motivating this PEP can be made with minimal backwards-incompatibility risk and (if the PEP is accepted) we should use all the leverage at our disposal to drive adoption. The risk is not backwards incompatibility, it is ambiguity of what Python 2.7 means. If changes under this PEP would result in any distributors rationally remaining at Python 2.7.6, then the result of any such changes should be labelled Python 2.8. -Ben > Cheers, > Nick. > > > > > ChrisA > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ben%40bendarnell.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Tue Mar 25 03:25:45 2014 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 25 Mar 2014 02:25:45 +0000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> Message-ID: <5330E929.2070607@mrabarnett.plus.com> On 2014-03-25 01:29, Ben Darnell wrote: > On Mon, Mar 24, 2014 at 4:44 AM, Nick Coghlan > wrote: > > > On 24 Mar 2014 15:25, "Chris Angelico" > wrote: > > > As has already been pointed out, this can already happen, but in an > > ad-hoc way. Making it official or semi-official would mean that a > > script written for Debian's "Python 2.7.10" would run on Red Hat's > > "Python 2.7.10", which would surely be an advantage. > > And having it break on the official Windows and Mac OS X binaries > would benefit end users, how? > > The position I am coming to is that the "enhanced security" release > should be the default one that we publish binary installers for, but > we should also ensure that downstream redistributors can easily do > "Python 2.7 with legacy SSL" releases if they so choose. I'm happier > forcing end users to rely on a redistributor to opt in to a lower > security option than I am to knowingly publish too many additional > releases with network security infrastructure that is (at best) > rapidly approaching its use by date. > > > I am strongly against allowing downstream distributors that choice. > Unless the security-enhanced variant of Python 2.7 quickly and > completely overtakes all previous versions, we will be creating a > compatibility problem between the two variants of Python 2.7. I believe > that the changes motivating this PEP can be made with minimal > backwards-incompatibility risk and (if the PEP is accepted) we should > use all the leverage at our disposal to drive adoption. The risk is not > backwards incompatibility, it is ambiguity of what Python 2.7 means. If > changes under this PEP would result in any distributors rationally > remaining at Python 2.7.6, then the result of any such changes should be > labelled Python 2.8. > I think that calling it Python 2.8 would be a bad idea for the reasons that have already been stated. Perhaps it should just be called Python 2.7 Enhanced Security ("Python 2.7 ES"). From tjreedy at udel.edu Tue Mar 25 03:40:24 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 24 Mar 2014 22:40:24 -0400 Subject: [Python-Dev] PEP 466 (round 3): Python 2 network security enhancements In-Reply-To: References: Message-ID: On 3/24/2014 9:43 AM, Nick Coghlan wrote: > And time for round 3 :) And round 3 of my response: contrary to what I said before, I now think that the base proposal should be the simplest possible: selectively (and minimally) waive the 'no-enhancement in maintenance release policy' for future 2.7 releases because certain internet security features have become dangerously obsolete and socially irresponsible and because 2.7 is exceptional in not having a followup version and will be exceptional in its amount and length of use. When we do a brown bag release in 1 to 4 weeks, we break the normal maintenance interval. We create a nuisance for those who already downloaded the replaced release. We create a nuisance for those who test with each maintenance release. But the reason we do that is because we also have a no-regression policy and we decide that the nuisance of a quick release is over-ridden by the nuisance of regression -- even if doing so increases the net user pain over not doing the quick release. (I personally have not been affected by regressions so far but have been affected by the new-release nuisance.) In the area of internet security, standing still for too long is a form of regression -- in terms of effectiveness. An enhanced version of 2.7 will be a bit of a nuisance, but only for the people who use the enhancements. The decreasing effectiveness of static security modules will also be a nuisance. -- Terry Jan Reedy From ncoghlan at gmail.com Tue Mar 25 08:51:51 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Mar 2014 17:51:51 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: <5330E929.2070607@mrabarnett.plus.com> References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> <5330E929.2070607@mrabarnett.plus.com> Message-ID: On 25 March 2014 12:25, MRAB wrote: > On 2014-03-25 01:29, Ben Darnell wrote: >> >> On Mon, Mar 24, 2014 at 4:44 AM, Nick Coghlan > > wrote: >> >> >> On 24 Mar 2014 15:25, "Chris Angelico" > > wrote: >> >> > As has already been pointed out, this can already happen, but in an >> > ad-hoc way. Making it official or semi-official would mean that a >> > script written for Debian's "Python 2.7.10" would run on Red Hat's >> > "Python 2.7.10", which would surely be an advantage. >> >> And having it break on the official Windows and Mac OS X binaries >> would benefit end users, how? >> >> The position I am coming to is that the "enhanced security" release >> should be the default one that we publish binary installers for, but >> we should also ensure that downstream redistributors can easily do >> "Python 2.7 with legacy SSL" releases if they so choose. I'm happier >> forcing end users to rely on a redistributor to opt in to a lower >> security option than I am to knowingly publish too many additional >> releases with network security infrastructure that is (at best) >> rapidly approaching its use by date. >> >> >> I am strongly against allowing downstream distributors that choice. >> Unless the security-enhanced variant of Python 2.7 quickly and >> completely overtakes all previous versions, we will be creating a >> compatibility problem between the two variants of Python 2.7. I believe >> that the changes motivating this PEP can be made with minimal >> backwards-incompatibility risk and (if the PEP is accepted) we should >> use all the leverage at our disposal to drive adoption. The risk is not >> backwards incompatibility, it is ambiguity of what Python 2.7 means. If >> changes under this PEP would result in any distributors rationally >> remaining at Python 2.7.6, then the result of any such changes should be >> labelled Python 2.8. >> > I think that calling it Python 2.8 would be a bad idea for the reasons > that have already been stated. > > Perhaps it should just be called Python 2.7 Enhanced Security ("Python > 2.7 ES"). The PEP currently calls the proposed unmodified fork of 2.7 "Python 2.7 with Legacy SSL". I suspect we could potentially ask the PSF to enforce that from a trademark perspective (that is, redistributors wouldn't be allowed to call versions with the legacy infrastructure "Python 2.7", they'd have to include the "with Legacy SSL" qualifier - that would also encompass all redistributions of 2.7.6 and below). I'm actually personally OK with just making vendors do all the work if they're really so worried about a slightly increased chance of undetected regressions that they prefer to keep using older SSL infrastructure. I think persisting with the old SSL infrastructure for too much longer would be a fundamentally bad idea, so I don't mind at all making it more difficult for downstream redistributors to do so. As Ben notes, allowing them this option increases the chance of confusion about what "Python 2.7" means, and once an upgrade Python 2.7 release was published, the "Python 2.7 with Legacy SSL" moniker would apply just as well to Python 2.7.6 and earlier as it would to a hypothetical additional branch that would impose an ongoing maintenance burden upstream. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Mar 25 09:11:49 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Mar 2014 18:11:49 +1000 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: References: <874n2n5vm2.fsf@rath.org> Message-ID: On 25 March 2014 09:04, Donald Stufft wrote: > On Mar 24, 2014, at 5:38 PM, Nick Coghlan wrote: > While I totally agree that it would be incredibly awesome if more companies > put > dedicated time into developing and maintaining CPython I don't think pushing > all the blame on to them is accurate. > > The attitude towards security issues and backwards compatibility has a > somewhat > equal share in the causes of the aging security infrastructure of the 2.x > line. > Now this PEP, if accepted, does a lot to resolve the largest offenders of > this > policy (and there has been some signs lately that perhaps going forward this > will be better) but I think it is not doing anyone a favor if we just point > fingers *over there* and claim the fault lies with someone else doing or not > doing something. > > I *don't* want to disparage anyone or anything of that like, mostly to say > that > while of course increased resources from corporate users would help the > situation > immensely but that additionally there is a reasonably sized contingent of > influential members who still want to treat Python as a hobbyist project and > not a critical piece of the infrastructure of the Internet as a whole. I > *don't* want to get help from downstream users, especially on important but > "boring" or hard issues such as security, and then have them feel shutdown > and > unable to actually get anything done as others who have attempted to resolve > some of these issues in the past have had happen to them. I actually agree with this (hence why I wrote the PEP in the first place), I just became really, really, really, annoyed with certain organisations over the course of writing the PEP drafts and that is reflected in the tone of the latest draft. However, in deliberately not naming names, I now realise I've left it open to *other* organisations thinking "Does he mean us? How is this our fault?". For clarification: if an org is guessing whether or not I was referring to them in particular while drafting the PEP, then no, I'm not. The specific organisations concerned are in absolutely no doubt as to the fact I'm genuinely angry with them. That said, while it certainly made me feel better at the time, I agree some of the current phrasing is not actually helpful in resolving the situation amicably for the benefit of all concerned, so I'll revise the offending sections of the PEP :) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From g.brandl at gmx.net Tue Mar 25 09:21:16 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 25 Mar 2014 09:21:16 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> <5330E929.2070607@mrabarnett.plus.com> Message-ID: Am 25.03.2014 08:51, schrieb Nick Coghlan: >> I think that calling it Python 2.8 would be a bad idea for the reasons >> that have already been stated. >> >> Perhaps it should just be called Python 2.7 Enhanced Security ("Python >> 2.7 ES"). > > The PEP currently calls the proposed unmodified fork of 2.7 "Python > 2.7 with Legacy SSL". I suspect we could potentially ask the PSF to > enforce that from a trademark perspective (that is, redistributors > wouldn't be allowed to call versions with the legacy infrastructure > "Python 2.7", they'd have to include the "with Legacy SSL" qualifier - > that would also encompass all redistributions of 2.7.6 and below). I don't know. It still feels like a source of confusion all round to have two different (C)Pythons not distinguished by version number. I haven't followed all of this thread, so forgive me if this suggestion has come up already: Since we know the EOL of 2.7, can't we say there won't be any more "non-secure" bugfix releases than up to 2.7.9, and the namespace 2.7.10 (yeah I know, but still way better than 2.8) and above is free for the "new SSL" versions. This also works from a version requirement point of view: if you require Python >= 2.7.10 you know you'll get the new features. If you don't, you shouldn't be using (or carefully checking) the new opt-in features. > I'm actually personally OK with just making vendors do all the work if > they're really so worried about a slightly increased chance of > undetected regressions that they prefer to keep using older SSL > infrastructure. I think persisting with the old SSL infrastructure for > too much longer would be a fundamentally bad idea, so I don't mind at > all making it more difficult for downstream redistributors to do so. I agree, if no other solution can be found we should err on the secure side (as opposed to the safe side). Georg From rosuav at gmail.com Tue Mar 25 09:26:08 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 25 Mar 2014 19:26:08 +1100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> Message-ID: On Tue, Mar 25, 2014 at 7:12 PM, Cory Benfield wrote: > On 24 March 2014 19:37, Chris Angelico wrote: >> The opting in could be done at the distro level. Red Hat could ship a >> thing called /usr/bin/python that runs SEPython, and that package >> could be identified and numbered in such a way that it's clearly a >> drop-in replacement for vanilla Python. If backward compatibility is >> done carefully (which, from everything I'm seeing here, is the way >> it's to be done), there should be absolutely no downside to using >> SEPython, except for portability (because now you're depending on it >> for security), and corner cases like testing. > > What's your solution for OS X, Windows et al? My concern is that if > you have a release called 'Python' and a release called 'Python with > security stuff', a surprisingly large number of people will download > the first, especially if the notes for the security release say 'this > may cause some minor compatibility problems'. IMHO, I'd rather have > good security be the default for everyone, and require an explicit > opt-out to get the bad security release. Exactly the same. If someone wants to distribute SEPython (that someone might be python.org itself, or ActiveState, or anyone else who has an interest in it), they're welcome to do so; and it could be done either as an all-in-one that packages all of CPython, or as an add-on; either way would work just as well, but the former would be cleaner. ChrisA From christian at python.org Tue Mar 25 09:36:59 2014 From: christian at python.org (Christian Heimes) Date: Tue, 25 Mar 2014 09:36:59 +0100 Subject: [Python-Dev] RFE 20469: ssl.getpeercert() should include extensions In-Reply-To: References: Message-ID: On 24.03.2014 23:51, Andrew M. Hettinger wrote: > I thought I'd wait until the 3.4 release before I bothered asking about > this: http://bugs.python.org/issue20469 > > I don't think I'm qualified to actually be writing code for the ssl > module, but is there anything else that I can do to help? > > I could probably put together a demonstration-case if that would be > helpful. I already have some code for the task. It wasn't ready for 3.4 but I'm going to add it to 3.5. I'm planing to include a X.509 type in order to wrap X.509 certificates. From rosuav at gmail.com Tue Mar 25 10:01:42 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 25 Mar 2014 20:01:42 +1100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> Message-ID: On Tue, Mar 25, 2014 at 7:37 PM, Cory Benfield wrote: > On 25 March 2014 08:26, Chris Angelico wrote: >> Exactly the same. If someone wants to distribute SEPython (that >> someone might be python.org itself, or ActiveState, or anyone else who >> has an interest in it), they're welcome to do so; and it could be done >> either as an all-in-one that packages all of CPython, or as an add-on; >> either way would work just as well, but the former would be cleaner. > > Reading this I suspect we're mostly in agreement but having trouble > communicating. My understanding of your point is simply that you don't > want python-dev to 'bless' either of the 2.7 releases proposed as > _the_ 2.7 release, instead pushing that choice on to people > distributing Python. I can get behind that plan so long as the source > code releases are named in such a way that people are either a) forced > to make a choice; or b) defaulted to secure 2.7. I'd like python.org / python-dev to bless, if not some specific version, at least some specific structure. I think that's something like what has been in the PEP at some point, though I haven't dug into the current version deeply enough to be sure. But if you take current 2.7 as a baseline, every new feature would be implemented by creating a new attribute of either the ssl module or some class in it; if the attribute is there, you can use it (eg a constant/enum value that's a parameter to something else), and if it's not, you can't. As long as the names are consistent, it'd be easy for a program to either probe and use what it can get, or just use what it wants and bomb if you don't give it a sufficiently-secure Python. So by that model, current 2.7 is fully compliant, and anything that doesn't actively conflict with that is also compliant. Any script that is written for the current 2.7 is guaranteed also to run on any compliant SEPython; and anything written for SEPython has to gracefully handle (which might mean cleanly bombing) anything down to and including current 2.7. Does that make sense? ChrisA From martin at v.loewis.de Tue Mar 25 11:34:34 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 25 Mar 2014 11:34:34 +0100 Subject: [Python-Dev] 3.4 buildbots available In-Reply-To: <1395522184.31368.97664617.2CD6986F@webmail.messagingengine.com> References: <20140322191057.1bc34e4d@fsol> <1395522184.31368.97664617.2CD6986F@webmail.messagingengine.com> Message-ID: <53315BBA.6060403@v.loewis.de> Am 22.03.14 22:03, schrieb Benjamin Peterson: > > On Sat, Mar 22, 2014, at 11:10, Antoine Pitrou wrote: >> >> Hello, >> >> I've created the 3.4 category in the buildbots setup: >> http://buildbot.python.org/all/waterfall?category=3.4.stable >> >> I've also retired 3.2 and 3.3 buildbots. Someone will have to update >> the text and URLs at https://www.python.org/dev/buildbot/. > > We probably want a http://buildbot.python.org/3.4.stable/ redirect, too, > then. > Done! Martin From barry at python.org Tue Mar 25 13:48:28 2014 From: barry at python.org (Barry Warsaw) Date: Tue, 25 Mar 2014 08:48:28 -0400 Subject: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7 In-Reply-To: References: <874n2n5vm2.fsf@rath.org> Message-ID: <20140325084828.107e73db@limelight.wooz.org> On Mar 25, 2014, at 06:11 PM, Nick Coghlan wrote: >I actually agree with this (hence why I wrote the PEP in the first >place), I just became really, really, really, annoyed with certain >organisations over the course of writing the PEP drafts and that is >reflected in the tone of the latest draft. However, in deliberately >not naming names, I now realise I've left it open to *other* >organisations thinking "Does he mean us? How is this our fault?". For >clarification: if an org is guessing whether or not I was referring to >them in particular while drafting the PEP, then no, I'm not. The >specific organisations concerned are in absolutely no doubt as to the >fact I'm genuinely angry with them. > >That said, while it certainly made me feel better at the time, I agree >some of the current phrasing is not actually helpful in resolving the >situation amicably for the benefit of all concerned, so I'll revise >the offending sections of the PEP :) Anger management through PEP writing! That's novel, but I can show you some more effective techniques at Pycon. :) -Barry From ncoghlan at gmail.com Tue Mar 25 14:09:45 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Mar 2014 23:09:45 +1000 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements Message-ID: Hmm, should probably have more entries in the post history... Anyway: * As Ben Darnell noted, the legacy SSL branch thing was a fundamentally bad idea. We can't stop downstream redistributors doing that if they really want to, but we don't have to encourage them * I also explained why I think selective backports are a bad idea * I've given a lot more context on how Red Hat actually manages to handle such long term support cycles, and that what I'm proposing here really isn't that radical from that perspective (and is, in fact, about the only available solution that can be effectively slotted into that model) * I've been a lot more explicit about the fact that long term maintenance releases are genuinely *different* from CPython's normal release lifetimes * more details on the specific support I think this proposal would need to become a practical reality * removed some unhelpful side comments and obliquely acknowledged the fact that it mat conceivably have been difficult for outsiders to see that it might actually be possible to make this case effectively :) Cheers, Nick. Diff: http://hg.python.org/peps/rev/4eb8de143801 Web: http://www.python.org/dev/peps/pep-0466/ =================================== PEP: 466 Title: Network Security Enhancement Exception for Python 2.7 Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan , Status: Draft Type: Informational Content-Type: text/x-rst Created: 23-Mar-2014 Post-History: 23-Mar-2014 Abstract ======== Most CPython tracker issues are classified as errors in behaviour or proposed enhancements. Most patches to fix behavioural errors are applied to all active maintenance branches. Enhancement patches are restricted to the default branch that becomes the next Python version. This cadence works reasonably well during Python's normal 18-24 month feature release cycle, which is still applicable to the Python 3 series. However, the age of the standard library in Python 2 has now reached a point where it is sufficiently far behind the state of the art in network security protocols for it to be causing real problems in commercial use cases where upgrading to Python 3 in the near term may not be practical. In recognition of the additional practical considerations that have arisen during the 4+ year maintenance cycle for Python 2.7, this PEP allows Python 2.7 standard library components that have implications for the overall security of the internet to be updated in line with the corresponding Python 3 feature releases. Specifically, the exception will apply to: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the components of the ``random`` and ``os`` modules that the above modules rely on for cryptographic randomness * the version of OpenSSL bundled with the binary installers for Windows and Mac OS X Changes to these modules will still need to undergo normal backwards compatibility assessments to ensure their default behaviour remains consistent with earlier Python 2.7 releases, but otherwise they will be kept entirely aligned with the latest feature release of their Python 3 counterparts. This is designed to make it easier to implement secure networked software in Python, even for software that currently needs to remain compatible with the Python 2 series (which includes a lot of network infrastructure software). While this PEP does not make any changes to the core development team's handling of security-fix-only branches that are no longer in active maintenance, it *does* recommend that commercial redistributors providing extended support periods for the Python standard library either adopt a similar approach to ensuring that the secure networking infrastructure keeps pace with the evolution of the internet, or else explicitly disclaim support for the use of older versions in roles that involve connecting directly to the public internet. Exemption Policy ================ Under this policy, the following network security related modules are granted a blanket exemption to the normal restriction against adding new features in Python 2.7 maintenance releases, for the purpose of keeping their APIs aligned with their counterparts in the latest feature release of Python 3: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module Under this exemption, these modules are updated to provide identical functionality to their Python 3 counterparts after every new Python 3 feature release. The default behaviour of the backported modules will be adjusted as appropriate to meet the backwards compatibility requirements of a Python 2.7 maintenance release. As part of this policy, permission is also granted to upgrade to newer feature releases of OpenSSL when preparing the binary installers for new maintenance releases of Python 2.7. Note that the ``sha`` and ``md5`` modules are not covered by this policy, but have been deprecated in favour of ``hashlib`` since Python 2.5 and have been removed entirely in the Python 3 series. In addition to the above blanket exemption, a conditional exemption is granted for these modules that may include some network security related features: * the ``os`` module (specifically ``os.urandom``) * the ``random`` module This more limited exemption for these modules requires that the *specific* enhancement being proposed for backporting needs to be justified as being network security related. This is generally restricted to cases where the feature in question is needed by an update to one of the modules covered by the blanket exemption. Backwards Compatibility Considerations ====================================== This PEP does *not* grant Python 2.7 any general exemptions to the usual backwards compatibility policy for maintenance releases. Instead, by explicitly encouraging the use of feature based checks and explicitly opting in to less secure configurations, it is designed to make it easier to write more secure cross-version compatible Python software, while still limiting the risk of breaking currently working software when upgrading to a new Python 2.7 maintenance release. In *all* cases where this policy is applied to backport enhancements to Python 2.7 maintenance releases, it MUST be possible to write cross-version compatible code that operates by "feature detection" (for example, checking for particular attributes in the module), without needing to explicitly check the Python version. It is then up to library and framework code to provide an appropriate warning and fallback behaviour if a desired feature is found to be missing. While some especially security sensitive software MAY fail outright if a desired security feature is unavailable, most software SHOULD instead emit a warning and continue operating using a slightly degraded security configuration. Affected APIs SHOULD be designed to allow library and application code to perform the following actions after detecting the presence of a relevant network security related feature: * explicitly opt in to more secure settings (to allow the use of enhanced security features in older maintenance releases of Python) * explicitly opt in to less secure settings (to allow the use of newer Python feature releases in lower security environments) * determine the default setting for the feature (this MAY require explicit Python version checks to determine the Python feature release, but MUST NOT require checking for a specific maintenance release) Security related changes to other modules (such as higher level networking libraries and data format processing libraries) will continue to be made available as backports and new modules on the Python Package Index, as independent distribution remains the preferred approach to handling software that must continue to evolve to handle changing development requirements independently of the Python 2 standard library. Refer to the `Motivation and Rationale`_ section for a review of the characteristics that make the secure networking infrastructure worthy of special consideration. OpenSSL compatibility --------------------- Under this policy, OpenSSL may be upgraded to more recent feature releases in Python 2.7 maintenance releases. On Linux and most other POSIX systems, the specific version of OpenSSL used already varies, as CPython dynamically links to the system provided OpenSSL library by default. For the Windows binary installers, the ``_ssl`` and ``_hashlib`` modules are statically linked with OpenSSL and the associated symbols are not exported. Marc-Andre Lemburg indicates that updating to newer OpenSSL releases in the ``egenix-pyopenssl`` binaries has not resulted in any reported compatibility issues [3]_ The Mac OS X binary installers historically followed the same policy as other POSIX installations and dynamically linked to the Apple provided OpenSSL libraries. However, Apple has now ceased updating these cross-platform libraries, instead requiring that even cross-platform developers adopt Mac OS X specific interfaces to access up to date security infrastructure on their platform. Accordingly, and independently of this PEP, the Mac OS X binary installers were already going to be switched to statically linker newer versions of OpenSSL [4]_ Other Considerations ==================== Maintainability --------------- This policy does NOT represent a commitment by volunteer contributors to actually backport network security related changes from the Python 3 series to the Python 2 series. Rather, it is intended to send a clear signal to potential corporate contributors that the core development team are willing to accept offers of corporate assistance in putting this policy into effect and handling the resulting increase in the Python 2 maintenance load. Backporting security related fixes and enhancements to earlier versions is a common service for commercial redistributors to offer to their customers. This policy represents an explicit invitation to contribute some of those changes back to the upstream Python community in cases where they are likely to have a broad impact that helps improve the security of the internet as a whole, with the assurance that the existing core development team not only won't object to such contributions, but will actively encourage their incorporation into the next Python 2.7 maintenance release. Documentation ------------- All modules covered by this policy MUST include a "Security Considerations" section in their documentation in order to take advantage of this policy. In addition to any other module specific contents, this section MUST enumerate key security enhancements and fixes (with CVE identifiers where applicable), along with the feature and maintenance releases that first included them. Security releases ----------------- This PEP does not propose any changes to the handling of security releases - those will continue to be source only releases that include only critical security fixes. However, the recommendations for library and application developers are deliberately designed to accommodate commercial redistributors that choose to apply this policy to additional Python release series that are either in security fix only mode, or have been declared "end of life" by the core development team. Whether or not redistributors choose to exercise that option will be up to the individual redistributor. Integration testing ------------------- Third party integration testing services should offer users the ability to test against specific Python 2.7 maintenance releases, to ensure that libraries, frameworks and applications can still test their handling of the legacy security infrastructure correctly (either failing or degrading gracefully, depending on the security sensitivity of the software), even after the latest Python 2.7 maintenance release has been synchronised with a new Python 3 feature release for the modules covered by this policy. Handling lower security environments with low risk tolerance ------------------------------------------------------------ For better or for worse (mostly worse), there are some environments where the risk of latent security defects is more tolerated than even a slightly increased risk of regressions in maintenance releases. This policy largely excludes these environments from consideration where the modules covered by the exemption are concerned - this approach is entirely inappropriate for software connected to the public internet, and defence in depth security principles suggest that it is not appropriate for most private networks either. Downstream redistributors may still choose to cater to such environments, but they will need to handle the process of downgrading the security related modules and doing the associated regression testing themselves. The main CPython continuous integration infrastructure will not cover this scenario. Evolution of this Policy ======================== The key requirement for a module to be considered for inclusion in this policy (whether under a blanket or conditional exemption) is that it must have security implications *beyond* the specific application that is written in Python and the system that application is running on. Thus the focus on network security protocols and related cryptographic infrastructure - Python is a popular choice for the development of web services and clients, and thus the capabilities of widely used Python versions have implications for the security design of other services that may themselves be using newer versions of Python or other development languages, but need to interoperate with clients or servers written using older versions of Python. The intent behind this requirement is to minimise any impact that the introduction of this policy may have on the stability and compatibility of maintenance releases. It would be thoroughly counterproductive if end users became as cautious about updating to new Python 2.7 maintenance releases as they are about updating to new feature releases within the same release series. Motivation and Rationale ======================== This PEP can be seen as a more targeted version of the "faster standard library release cycle" proposals discussed in PEP 407 and PEP 413, focusing specifically on those areas which have implications beyond the Python community. Background ---------- The creation of this PEP was prompted primarily by the aging SSL support in the Python 2 series. As of March 2014, the Python 2.7 SSL module is approaching four years of age, and the SSL support in the still popular Python 2.6 release had its feature set locked six years ago. These are simply too old to provide a foundation that can be recommended in good conscience for secure networking software that operates over the public internet, especially in an era where it is becoming quite clearly evident that advanced persistent security threats are even more widespread and more indiscriminate in their targeting than had previously been understood. While they represented reasonable security infrastructure in their time, the state of the art has moved on, and we need to investigate mechanisms for effectively providing more up to date network security infrastructure for users that, for whatever reason, are not currently in a position to migrate to Python 3. While the use of the system OpenSSL installation addresses many of these concerns on Linux platforms, it doesn't address all of them (in particular, it is still difficult for sotware to explicitly require some higher level security settings). In the case of the binary installers for Windows and Mac OS X that are published on python.org, the version of OpenSSL used is entirely within the control of the Python core development team, but is currently limited to OpenSSL maintenance releases for the version initially shipped with the corresponding Python feature release. With increased popularity comes increased responsibility, and this policy aims to acknowledge the fact that Python's popularity and adoption has now reached a level where some of our design and policy decisions have significant implications beyond the Python development community. As one example, the Python 2 ``ssl`` module does not support the Server Name Identification standard. While it is possible to obtain SNI support by using the third party ``requests`` client library, actually doing so currently requires using not only ``requests`` and its embedded dependencies, but also half a dozen or more additional libraries. The lack of support in the Python 2 series thus serves as an impediment to making effective use of SNI on servers, as Python 2 clients will frequently fail to handle it correctly. Another more critical example is the lack of SSL hostname matching in the Python 2 standard library - it is currently necessary to rely on a third party library, such as ``requests`` or ``backports.ssl_match_hostname`` to obtain that functionality in Python 2. The Python 2 series also remains more vulnerable to remote timing attacks on security sensitive comparisons than the Python 3 series, as it lacks a standard library equivalent to the timing attack resistant ``hmac.compare_digest()`` function. While appropriate secure comparison functions can be implemented in third party extensions, many users don't even consider the issue and use ordinary equality comparisons instead - while a standard library solution doesn't automatically fix that problem, it *does* make the barrier to resolution much lower once the problem is pointed out. My position on the ongoing transition from Python 2 to Python 3 has long been that Python 2 remains a supported platform for the core development team, and that commercial support will remain available well after upstream maintenance ends. However, in the absence of this network security enhancement policy, that position is difficult to justify when it comes to software that operates over the public internet. Just as many developers consider it too difficult to develop truly secure modern networked software in C/C++ (largely due to the challenges associated with manual memory management), I anticipate that in the not too distant future, it will be considered too difficult to develop truly secure modern networked software using the Python 2 series (some developers would argue that we have already reached that point). Python 2.7 represents the only long term maintenance release the core development team has provided, and it is natural that there will be things that worked over a historically shorter maintenance lifespan that don't work over this longer support period. In the specific case of the problem described in this PEP, the simplest available solution is to acknowledge that long term maintenance of network security related modules *requires* the ability to add new features, even while retaining backwards compatibility for existing interfaces. It is worth comparing the approach described in this PEP with Red Hat's handling of its long term support commitments: it isn't the RHEL 6.0 release itself that receives 10 years worth of support, but the overall RHEL 6 *series*. The individual RHEL 6.x point releases within the series then receive a wide variety of new features, including security enhancements, all while meeting strict backwards compatibility guarantees for existing software. Subscribers *are* able to continue using a given point release after the next point release has become available, but a corresponding add-on subscription for `Extended Update Support `__ is needed to cover the additional backporting work involved. To date, downstream redistributors have respected our upstream policy of "no new features in Python maintenance releases". This PEP explicitly accepts that a more nuanced policy is appropriate in the case of network security related features, and the specific one it describes is deliberately designed such that it at least has some chance of being applied to Red Hat Enterprise Linux and its downstream derivatives. Alternative: advise developers of networked software to migrate to Python 3 --------------------------------------------------------------------------- This alternative represents the status quo. Unfortunately, it has proven to be unworkable in practice, as the backwards compatibility implications mean that this is a non-trivial migration process for large applications and integration projects. While the tools for migration have evolved to a point where it is possible to migrate even large applications opportunistically and incrementally (rather than all at once) by updating code to run in the large common subset of Python 2 and Python 3, using the most recent technology often isn't a priority in commercial environments. Previously, this was considered an acceptable harm, as while it was an unfortunate problem for the affected developers to have to face, it was seen as an issue between them and their management chain to make the case for infrastructure modernisation, and this case would become naturally more compelling as the Python 3 series evolved. However, now that we're fully aware of the impact the limitations of the Python 2 standard library may be having on the evolution of internet security standards, I no longer believe that it is reasonable to expect platform and application developers to resolve all of the latent defects in an application's Unicode correctness solely in order to gain access to the network security enhancements already available in Python 3. While Ubuntu (and to some extent Debian as well) are committed to porting all default system services and scripts to Python 3, and to removing Python 2 from its default distribution images (but not from its archives), this is a mammoth task and won't be completed for the Ubuntu 14.04 LTS release (at least for the desktop image - it may be achieved for the mobile and server images). Fedora has even more work to do to migrate, and it will take a non-trivial amount of time to migrate the relevant infrastructure components. While Red Hat are also actively working to make it easier for users to use more recent versions of Python on our stable platforms, it's going to take time for those efforts to start having an impact on end users' choice of version, and any such changes also don't benefit the core platform infrastructure that runs in the integrated system Python by necessity. The OpenStack migration to Python 3 is also still in its infancy, and even though that's a project with an extensive and relatively robust automated test suite, it's still large enough that it is going to take quite some time to migrate. And that's just three of the highest profile open source projects that make heavy use of Python. Given the likely existence of large amounts of legacy code that lacks the kind of automated regression test suite needed to help support a migration from Python 2 to Python 3, there are likely to be many cases where reimplementation (perhaps even in Python 3) proves easier than migration. The key point of this PEP is that those situations affect more people than just the developers and users of the affected application: the existence of clients and servers with outdated network security infrastructure becomes something that developers of secure networked services need to take into account as part of their security design, and that's a problem that inhibits the adoption of better security standards. As Terry Reedy noted, if we try to persist with the status quo, the likely outcome is that commercial redistributors will attempt to do something like this on behalf of their customers *anyway*, but in a potentially inconsistent and ad hoc manner. By drawing the scope definition process into the upstream project we are in a better position to influence the approach taken to address the situation and to help ensure some consistency across redistributors. The problem is real, so *something* needs to change, and this PEP describes my preferred approach to addressing the situation. Alternative: create and release Python 2.8 ------------------------------------------ With sufficient corporate support, it likely *would* be possible to create and release Python 2.8 (it's highly unlikely such a project would garner enough interest to be achievable with only volunteers). However, this wouldn't actually solve the problem, as the aim is to provide a *relatively low impact* way to incorporate enhanced security features into integrated products and deployments that make use of Python 2. Upgrading to a new Python feature release would mean both more work for the core development team, as well as a more disruptive update that most potential end users would likely just skip entirely. Attempting to create a Python 2.8 release would also bring in suggestions to backport many additional features from Python 3 (such as ``tracemalloc`` and the improved coroutine support), making the migration from Python 2.7 to this hypothetical 2.8 release even riskier and more disruptive. This is not a recommended approach, as it would involve substantial additional work for a result that is actually less effective in achieving the original aim (which is to eliminate the current widespread use of the aging network security infrastructure in the Python 2 series). Furthermore, while I can't make any commitments to actually addressing this issue on Red Hat platforms, I *can* categorically rule out the idea of a Python 2.8 being of any use to me in even attempting to get it addressed. Alternative: distribute the security enhancements via PyPI ---------------------------------------------------------- While this initially appears to be an attractive and easier to manage approach, it actually suffers from several significant problems. Firstly, this is complex, low level, cross-platform code that integrates with the underlying operating system across a variety of POSIX platforms (including Mac OS X) and Windows. The CPython BuildBot fleet is already set up to handle continuous integration in that context, but most of the freely available continuous integration services just offer Linux, and perhaps paid access to Windows. Those services work reasonably well for software that largely runs on the abstraction layers offered by Python and other dynamic languages, as well as the more comprehensive abstraction offered by the JVM, but won't suffice for the kind of code involved here. The OpenSSL dependency for the network security support also qualifies as the kind of "complex binary dependency" that isn't yet handled well by the ``pip`` based software distribution ecosystem. Relying on a third party binary dependency also creates potential compatibility problems for ``pip`` when running on other interpreters like ``PyPy``. Another practical problem with the idea is the fact that ``pip`` itself relies on the ``ssl`` support in the standard library (with some additional support from a bundled copy of ``requests``, which in turn bundles ``backport.ssl_match_hostname``), and hence would require any replacement module to also be bundled within ``pip``. This wouldn't pose any insurmountable difficulties (it's just another dependency to vendor), but it *would* mean yet another copy of OpenSSL to keep up to date. This approach also has the same flaw as all other "improve security by renaming things" approaches: they completely miss the users who most need help, and raise significant barriers against being able to encourage users to do the right thing when their infrastructure supports it (since "use this other module" is a much higher impact change than "turn on this higher security setting"). Deprecating the aging SSL infrastructure in the standard library in favour of an external module would be even more user hostile than accepting the slightly increased risk of regressions associated with upgrading it in place. Last, but certainly not least, this approach suffers from the same problem as the idea of doing a Python 2.8 release: likely not solving the actual problem. Commercial redistributors of Python are set up to redistribute *Python*, and a pre-existing set of additional packages. Getting new packages added to the pre-existing set *can* be done, but means approaching each and every redistributor and asking them to update their repackaging process accordingly. By contrast, the approach described in this PEP would require redistributors to deliberately *opt out* of the security enhancements by deliberately downgrading the provided network security infrastructure, which most of them are unlikely to do. Alternative: provide a "legacy SSL infrastructure" branch --------------------------------------------------------- Earlier versions of this PEP included the concept of a ``2.7-legacy-ssl`` branch that preserved the exact feature set of the Python 2.7.6 network security infrastructure. It is the opinion of the PEP author that anyone that actually wants this is almost certainly making a mistake, and if they insist they really do want it in their specific situation, they're welcome to either make it themselves or arrange for a downstream redistributor to make it for them. If they are made publicly available, any such rebuilds should be referred to as "Python 2.7 with Legacy SSL" to clearly distinguish them from the official Python 2.7 releases that include more up to date network security infrastructure. After the first Python 2.7 maintenance release that has the security infrastructure updated to match Python 3.4, it would also be appropriate to refer to Python 2.7.6 and earlier releases as "Python 2.7 with Legacy SSL". Alternative: selectively backport particular APIs ------------------------------------------------- An instinctively minimalist reaction to this proposal is to only backport particular APIs in the affected modules that are judged to be "security critical". However, this ends up providing a worse end user experience, as well as a worse developer experience. For end users, the selective backporting approach means learning not only the legacy Python 2.7 API and the current Python 3 APIs, but also the hybrid API created by the selective backporting process. It is much simpler to just learn the APIs of the original Python 2.7 feature release and the relevant Python 3 features releases, without trying to define a new hybrid API that only exists in later Python 2.7 maintenance branches. For developers, the main benefit of the "just align the available functionality with Python 3" approach is that it reduces the amount of design work needing when updating Python 2.7 with new security features. The "feature or fix?" and "security related or not?" debates are both handled in advance by this policy, leaving only the matter of ensuring that backwards compatibility is preserved as needed by adjusting the default behaviour in the Python 2.7 backport when appropriate. Open Questions ============== * MvL has indicated he is not prepared to tackle the task of trying to integrate a newer OpenSSL into the also aging Python 2.7 build infrastructure on Windows (unfortunately, we've looked into upgrading that build infrastructure, and the backwards compatibility issues appear to be effectively insurmountable). We would require a commitment from another trusted contributor to handle at least this task, and potentially also taking over the task of creating the official Python 2.7 Windows installers for the remaining Python 2.7 maintenance releases. * We would need commitments to create and review full backports of the components covered by this policy from Python 3.4 to Python 2.7, as well as support for handling any more specific security issues affecting these modules. * I believe I've addressed all the technical and scope questions I had, or others raised. That just leaves the question of "Do we agree this plan actually makes sense, given the constraints on downstream redistributors that would also like to see this problem solved?" :) Disclosure of Interest ====================== The author of this PEP currently works for Red Hat on test automation tools. If this proposal is accepted, I will be strongly encouraging Red Hat to take advantage of the resulting opportunity to help improve the overall security of the Python ecosystem. However, I do not speak for Red Hat in this matter, and cannot make any commitments on Red Hat's behalf. Acknowledgements ================ Thanks to Christian Heimes for his recent efforts on greatly improving Python's SSL support in the Python 3 series, and a variety of members of the Python community for helping me to better understand the implications of the default settings we provide in our SSL modules, and the impact that tolerating the use of SSL infrastructure that was defined in 2010 (Python 2.7) or even 2008 (Python 2.6) potentially has for the security of the web as a whole. Christian and Donald Stufft also provided valuable feedback on a preliminary draft of this proposal. Thanks also to participants in the python-dev mailing list threads [1,2,5]_ References ========== .. [1] https://mail.python.org/pipermail/python-dev/2014-March/133334.html .. [2] https://mail.python.org/pipermail/python-dev/2014-March/133389.html .. [3] https://mail.python.org/pipermail/python-dev/2014-March/133438.html .. [4] https://mail.python.org/pipermail/python-dev/2014-March/133347.html .. [5] https://mail.python.org/pipermail/python-dev/2014-March/133442.html -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Tue Mar 25 14:29:25 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 25 Mar 2014 13:29:25 +0000 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: References: Message-ID: On 25 March 2014 13:09, Nick Coghlan wrote: > * MvL has indicated he is not prepared to tackle the task of trying to > integrate a newer OpenSSL into the also aging Python 2.7 build > infrastructure on Windows (unfortunately, we've looked into upgrading > that build infrastructure, and the backwards compatibility issues > appear to be effectively insurmountable). We would require a commitment > from another trusted contributor to handle at least this task, and > potentially also taking over the task of creating the official > Python 2.7 Windows installers for the remaining Python 2.7 maintenance > releases. One issue that strikes me is that much of the focus of this PEP is on supporting Linux distributions. This is entirely reasonable, as they are the ones with the sort of long-term support commitments that result in this issue (in the Windows world, possibly ActiveState offer formal support for Python 2.7, but otherwise I'm not aware of actual paid support options that might be relevant on Windows). With that in mind, is it reasonable to expect Linux vendors to support delivery of updated Windows builds of Python 2.7? If not, is it acceptable to python-dev to release a Python 2.7 maintenance release with backported security enhancements only available for Linux? (The same questions can be asked of OSX or Solaris support - this isn't solely a Windows issue). I think the PEP needs to be explicit here about what python-dev expect in terms of cross-platform support. I would assume that the expectation is that we deliver exactly the same level of cross-platform support as for 3.x, but commercial vendors could quite easily miss that implication if it is not spelled out. Paul From ncoghlan at gmail.com Tue Mar 25 14:47:04 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Mar 2014 23:47:04 +1000 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: References: Message-ID: On 25 Mar 2014 23:29, "Paul Moore" wrote: > > On 25 March 2014 13:09, Nick Coghlan wrote: > > * MvL has indicated he is not prepared to tackle the task of trying to > > integrate a newer OpenSSL into the also aging Python 2.7 build > > infrastructure on Windows (unfortunately, we've looked into upgrading > > that build infrastructure, and the backwards compatibility issues > > appear to be effectively insurmountable). We would require a commitment > > from another trusted contributor to handle at least this task, and > > potentially also taking over the task of creating the official > > Python 2.7 Windows installers for the remaining Python 2.7 maintenance > > releases. > > One issue that strikes me is that much of the focus of this PEP is on > supporting Linux distributions. This is entirely reasonable, as they > are the ones with the sort of long-term support commitments that > result in this issue (in the Windows world, possibly ActiveState offer > formal support for Python 2.7, but otherwise I'm not aware of actual > paid support options that might be relevant on Windows). With that in > mind, is it reasonable to expect Linux vendors to support delivery of > updated Windows builds of Python 2.7? If not, is it acceptable to > python-dev to release a Python 2.7 maintenance release with backported > security enhancements only available for Linux? (The same questions > can be asked of OSX or Solaris support - this isn't solely a Windows > issue). > > I think the PEP needs to be explicit here about what python-dev expect > in terms of cross-platform support. I would assume that the > expectation is that we deliver exactly the same level of > cross-platform support as for 3.x, but commercial vendors could quite > easily miss that implication if it is not spelled out. The PEP says to sync with Python 3, and that has full cross platform support. The Linux focus just comes from the fact that Linux is where the problem is most evident. It's not like we're going to just be giving the PEP to vendors as a spec and leaving them to it - it's largely an invitation to participate more directly upstream to help resolve a particularly thorny problem, not a Statement of Work :) Cheers, Nick. > > Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Mar 25 15:25:29 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 25 Mar 2014 14:25:29 +0000 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: References: Message-ID: On 25 March 2014 13:47, Nick Coghlan wrote: > It's not like we're going to just be giving the PEP to vendors as a spec and > leaving them to it - it's largely an invitation to participate more directly > upstream to help resolve a particularly thorny problem, not a Statement of > Work :) :-) I don't really know the APIs involved, but AIUI one of the 3.4 enhancements is exposing the SSLContext. Is the code to do this compatible with the version of OpenSSL bundled with Python 2.7 on Windows? If not, suppose that Red Hat provide resources that work on backporting the code, but they don't have Windows experts so no-one deals with integrating the new OpenSSL into the Windows binaries. Would the backport be blocked until someone is found to do the Windows work? This "I've written a patch but it hasn't been applied" is the type of scenario that puts people off contributing. If it's likely to happen, I think Red Hat have a right to know that in advance. And I don't know that it's something they would appreciate without python-dev pointing it out. If we're reasonably sure (not necessarily certain, there's always grey areas) that this isn't going to be an issue, then that's also fine. We can simply say that. That's all I'm saying. Not trying to require anything of contributors, just trying to be open and explicit about the criteria that will apply to accepting contributions. Paul. From solipsis at pitrou.net Tue Mar 25 16:10:54 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 25 Mar 2014 16:10:54 +0100 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements References: Message-ID: <20140325161054.69fc75a7@fsol> On Tue, 25 Mar 2014 14:25:29 +0000 Paul Moore wrote: > On 25 March 2014 13:47, Nick Coghlan wrote: > > It's not like we're going to just be giving the PEP to vendors as a spec and > > leaving them to it - it's largely an invitation to participate more directly > > upstream to help resolve a particularly thorny problem, not a Statement of > > Work :) > > :-) > > I don't really know the APIs involved, but AIUI one of the 3.4 > enhancements is exposing the SSLContext. No, it was done as soon as 3.2. Note that exposing SSLContext doesn't achieve anything *in itself*, until you also add relevant APIs to query and influence the SSLContext's internal state (which was done gradually in 3.2, 3.3 and 3.4). Separate enhancements have to do with strengthening the default SSL options and accepted SSL ciphers; when they didn't add new APIs or threatened to break legitimate use case, they have even been applied to bugfix branches. > Is the code to do this > compatible with the version of OpenSSL bundled with Python 2.7 on > Windows? It is. Actually, we have (or had) buildbots with older OpenSSL versions! The problem with backporting SSLContext is really that 1) you are adding a non-minimal new API set to the 2.7 feature set; 2) you must make it so that other stdlib modules take advantage of the new APIs (otherwise why bother?). > This "I've written a patch but it hasn't been applied" is the type of > scenario that puts people off contributing. In the case of ssl, I think I've tried to apply most interesting and finished patches. This is how you get features such as NPN protocols or server-side SNI. Regards Antoine. From cory at lukasa.co.uk Tue Mar 25 09:12:51 2014 From: cory at lukasa.co.uk (Cory Benfield) Date: Tue, 25 Mar 2014 08:12:51 +0000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> Message-ID: On 24 March 2014 19:37, Chris Angelico wrote: > The opting in could be done at the distro level. Red Hat could ship a > thing called /usr/bin/python that runs SEPython, and that package > could be identified and numbered in such a way that it's clearly a > drop-in replacement for vanilla Python. If backward compatibility is > done carefully (which, from everything I'm seeing here, is the way > it's to be done), there should be absolutely no downside to using > SEPython, except for portability (because now you're depending on it > for security), and corner cases like testing. What's your solution for OS X, Windows et al? My concern is that if you have a release called 'Python' and a release called 'Python with security stuff', a surprisingly large number of people will download the first, especially if the notes for the security release say 'this may cause some minor compatibility problems'. IMHO, I'd rather have good security be the default for everyone, and require an explicit opt-out to get the bad security release. From cory at lukasa.co.uk Tue Mar 25 09:37:52 2014 From: cory at lukasa.co.uk (Cory Benfield) Date: Tue, 25 Mar 2014 08:37:52 +0000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> Message-ID: On 25 March 2014 08:26, Chris Angelico wrote: > Exactly the same. If someone wants to distribute SEPython (that > someone might be python.org itself, or ActiveState, or anyone else who > has an interest in it), they're welcome to do so; and it could be done > either as an all-in-one that packages all of CPython, or as an add-on; > either way would work just as well, but the former would be cleaner. Reading this I suspect we're mostly in agreement but having trouble communicating. My understanding of your point is simply that you don't want python-dev to 'bless' either of the 2.7 releases proposed as _the_ 2.7 release, instead pushing that choice on to people distributing Python. I can get behind that plan so long as the source code releases are named in such a way that people are either a) forced to make a choice; or b) defaulted to secure 2.7. From cory at lukasa.co.uk Tue Mar 25 10:35:34 2014 From: cory at lukasa.co.uk (Cory Benfield) Date: Tue, 25 Mar 2014 09:35:34 +0000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> Message-ID: On 25 March 2014 09:01, Chris Angelico wrote: > So by that model, current 2.7 is fully compliant, and anything that > doesn't actively conflict with that is also compliant. Any script that > is written for the current 2.7 is guaranteed also to run on any > compliant SEPython; and anything written for SEPython has to > gracefully handle (which might mean cleanly bombing) anything down to > and including current 2.7. Does that make sense? Absolutely. =) My additional concern on top of that is wanting users to fall into a pit of success by making it overwhelmingly more likely that users will accidentally end up with the safe version if they aren't paying attention. I'm not hugely bothered about how that's done: I'd just like not to have to field Requests bug reports about lack of security that boil down to a user having grabbed the insecure version by accident. From bcannon at gmail.com Tue Mar 25 16:10:02 2014 From: bcannon at gmail.com (Brett Cannon) Date: Tue, 25 Mar 2014 15:10:02 +0000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> <5330E929.2070607@mrabarnett.plus.com> Message-ID: On Tue Mar 25 2014 at 4:21:51 AM, Georg Brandl wrote: > Am 25.03.2014 08:51, schrieb Nick Coghlan: > > >> I think that calling it Python 2.8 would be a bad idea for the reasons > >> that have already been stated. > >> > >> Perhaps it should just be called Python 2.7 Enhanced Security ("Python > >> 2.7 ES"). > > > > The PEP currently calls the proposed unmodified fork of 2.7 "Python > > 2.7 with Legacy SSL". I suspect we could potentially ask the PSF to > > enforce that from a trademark perspective (that is, redistributors > > wouldn't be allowed to call versions with the legacy infrastructure > > "Python 2.7", they'd have to include the "with Legacy SSL" qualifier - > > that would also encompass all redistributions of 2.7.6 and below). > > I don't know. It still feels like a source of confusion all round to > have two different (C)Pythons not distinguished by version number. > > I haven't followed all of this thread, so forgive me if this suggestion > has come up already: > > Since we know the EOL of 2.7, can't we say there won't be any more > "non-secure" > bugfix releases than up to 2.7.9, and the namespace 2.7.10 (yeah I know, > but > still way better than 2.8) and above is free for the "new SSL" versions. > > This also works from a version requirement point of view: if you require > Python > >= 2.7.10 you know you'll get the new features. If you don't, you > shouldn't > be using (or carefully checking) the new opt-in features. > Or if this is such a big deal we start with 2.7.6 and not postpone until 2.7.10 (which I guess could happen immediately after 2.7.9 and have nothing more than the upgraded modules). People have been making grandiose statements about how the security of the internet is hampered by Python 2.7 in this discussion. If these statements are actually not over-stated then we should do the fix sooner *and* add the incentive people to switch over by getting more bug fixes. It's not like Python 2.7 is getting a ton of fixes at this point anyway. > > > I'm actually personally OK with just making vendors do all the work if > > they're really so worried about a slightly increased chance of > > undetected regressions that they prefer to keep using older SSL > > infrastructure. I think persisting with the old SSL infrastructure for > > too much longer would be a fundamentally bad idea, so I don't mind at > > all making it more difficult for downstream redistributors to do so. > > I agree, if no other solution can be found we should err on the secure > side (as opposed to the safe side). > As long as we make it clear we have chosen to change our backwards-compatibility guarantees in the name of security and have a link to the last backwards-compatible release then I agree as well. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Mar 25 16:17:16 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 25 Mar 2014 08:17:16 -0700 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: <20140325161054.69fc75a7@fsol> References: <20140325161054.69fc75a7@fsol> Message-ID: On Tue, Mar 25, 2014 at 8:10 AM, Antoine Pitrou wrote: > The problem with backporting SSLContext is really that 1) you are adding > a non-minimal new API set to the 2.7 feature set; 2) you must make it so > that other stdlib modules take advantage of the new APIs (otherwise > why bother?). > I actually worry about another scenario. Suppose we backport SSLContext to 2.7.7. There is plenty of software out there nowadays that uses a single source for Python 2 and 3 compatibility. Given that SSLContext until now has only been available in Python 3, and given the recommendation to use dynamic feature detection, not version checks, it's at least *conceivable* that some library detects SSLContext and switches to some code that uses Python-3-only features (of the variety that don't cause syntax errors in Python 2). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Tue Mar 25 16:31:56 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 25 Mar 2014 15:31:56 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?PEP_466_=28round_4=29=3A_Python_2=2E7_netw?= =?utf-8?q?ork_security=09enhancements?= References: <20140325161054.69fc75a7@fsol> Message-ID: A casual glance at https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/ util.py#L610 which is probably the most widely used consumer of these APIs, outside the stdlib itself, looks to me like if these names were to suddenly show up, everything would continue to work just fine, with the advance of being able to explicitly specify some options. All of which is to say: I don't think this is a real concern. Alex From guido at python.org Tue Mar 25 17:35:16 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 25 Mar 2014 09:35:16 -0700 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: References: <20140325161054.69fc75a7@fsol> Message-ID: On Tue, Mar 25, 2014 at 8:31 AM, Alex Gaynor wrote: > A casual glance at > > https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/ > util.py#L610 > which is probably the most widely used consumer of these APIs, outside the > stdlib itself, looks to me like if these names were to suddenly show up, > everything would continue to work just fine, with the advance of being > able to > explicitly specify some options. > > All of which is to say: I don't think this is a real concern. That would be great, because I have no other major beef with the PEP (but I still need to read in in full -- it's long and half of it still feels like weasel words to me, so I can't apply my usual skimming tactics). I would like the PEP (or perhaps a companion PEP?) spell out the set of enhancements that we would *currently* like to see backported from Python 3.4 to 2.7, without the implication that these would be the *only* enhancements -- such a list would serve as an example and to focus the understanding. The PEP currently doesn't even name SSLContext! I wouldn't be totally surprised to find that there are some details of some API added to Python 3.4 that simply cannot be backported due to some important difference between Python 2 and 3 (e.g. because of differences in Unicode handling, or a missing socket method). I don't think such things would be showstoppers, they would just have to be worked around carefully; but it would be better to know about them now rather than having to figure out how to comply with the PEP's insistence of a full backport. I do note that the PEP seems to have some weasel-words about breaking backward compatibility in the name of security. The phrase "This PEP does *not* grant Python 2.7 any general exemptions to the usual backwards compatibility policy for maintenance releases" *could* be interpreted to imply that the PEP grants some *specific* exemptions (regardless of whether that was Nick's intention when he wrote that sentence). I'd like clarity on this; IIRC we've had to make some compatibility-breaking changes in the past for security reasons, but I don't recall the details or how that worked out (whether much code broke and whether that was considered a good or a bad thing). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Mar 25 17:46:48 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 25 Mar 2014 12:46:48 -0400 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: References: <20140325161054.69fc75a7@fsol> Message-ID: On Mar 25, 2014, at 12:35 PM, Guido van Rossum wrote: > On Tue, Mar 25, 2014 at 8:31 AM, Alex Gaynor wrote: > A casual glance at > https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/ > util.py#L610 > which is probably the most widely used consumer of these APIs, outside the > stdlib itself, looks to me like if these names were to suddenly show up, > everything would continue to work just fine, with the advance of being able to > explicitly specify some options. > > All of which is to say: I don't think this is a real concern. > > That would be great, because I have no other major beef with the PEP (but I still need to read in in full -- it's long and half of it still feels like weasel words to me, so I can't apply my usual skimming tactics). > > I would like the PEP (or perhaps a companion PEP?) spell out the set of enhancements that we would *currently* like to see backported from Python 3.4 to 2.7, without the implication that these would be the *only* enhancements -- such a list would serve as an example and to focus the understanding. The PEP currently doesn't even name SSLContext! > > I wouldn't be totally surprised to find that there are some details of some API added to Python 3.4 that simply cannot be backported due to some important difference between Python 2 and 3 (e.g. because of differences in Unicode handling, or a missing socket method). I don't think such things would be showstoppers, they would just have to be worked around carefully; but it would be better to know about them now rather than having to figure out how to comply with the PEP's insistence of a full backport. > > I do note that the PEP seems to have some weasel-words about breaking backward compatibility in the name of security. The phrase "This PEP does not grant Python 2.7 any general exemptions to the usual backwards compatibility policy for maintenance releases" *could* be interpreted to imply that the PEP grants some specific exemptions (regardless of whether that was Nick's intention when he wrote that sentence). I'd like clarity on this; IIRC we've had to make some compatibility-breaking changes in the past for security reasons, but I don't recall the details or how that worked out (whether much code broke and whether that was considered a good or a bad thing). I?m pretty sure Nick was just trying to say that the changes made under this PEP still have to be backwards compatible in the sense that APIs can?t change their default behavior and such. In other words we can?t suddenly flip on hostname checking or anything like that. > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From am.nine10 at i.softbank.jp Tue Mar 25 17:22:22 2014 From: am.nine10 at i.softbank.jp (april) Date: Wed, 26 Mar 2014 01:22:22 +0900 Subject: [Python-Dev] 3.3 branch is now in security fix mode Message-ID: <40550269-B4B2-45FE-A434-8DF37B099B8E@i.softbank.jp> Sent from my iPhone From guido at python.org Tue Mar 25 18:16:21 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 25 Mar 2014 10:16:21 -0700 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: References: <20140325161054.69fc75a7@fsol> Message-ID: On Tue, Mar 25, 2014 at 9:46 AM, Donald Stufft wrote: > > On Mar 25, 2014, at 12:35 PM, Guido van Rossum wrote: > [...] > > I do note that the PEP seems to have some weasel-words about breaking > backward compatibility in the name of security. The phrase "This PEP does > *not* grant Python 2.7 any general exemptions to the usual backwards > compatibility policy for maintenance releases" *could* be interpreted to > imply that the PEP grants some *specific* exemptions (regardless of > whether that was Nick's intention when he wrote that sentence). I'd like > clarity on this; IIRC we've had to make some compatibility-breaking changes > in the past for security reasons, but I don't recall the details or how > that worked out (whether much code broke and whether that was considered a > good or a bad thing). > > > I'm pretty sure Nick was just trying to say that the changes made under > this PEP still have to be backwards compatible in the sense that APIs can't > change their default behavior and such. In other words we can't suddenly > flip on hostname checking or anything like that. > Then the words should be clarified (maybe by removing 'general'?). This PEP invites interpretation by future generations so it should be as clear as possible on the intent, to avoid scholarly arguments. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Mar 25 18:18:15 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 25 Mar 2014 13:18:15 -0400 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: References: <20140325161054.69fc75a7@fsol> Message-ID: On Mar 25, 2014, at 1:16 PM, Guido van Rossum wrote: > On Tue, Mar 25, 2014 at 9:46 AM, Donald Stufft wrote: > > On Mar 25, 2014, at 12:35 PM, Guido van Rossum wrote: > [...] >> >> I do note that the PEP seems to have some weasel-words about breaking backward compatibility in the name of security. The phrase "This PEP does not grant Python 2.7 any general exemptions to the usual backwards compatibility policy for maintenance releases" *could* be interpreted to imply that the PEP grants some specific exemptions (regardless of whether that was Nick's intention when he wrote that sentence). I'd like clarity on this; IIRC we've had to make some compatibility-breaking changes in the past for security reasons, but I don't recall the details or how that worked out (whether much code broke and whether that was considered a good or a bad thing). > > I?m pretty sure Nick was just trying to say that the changes made under this PEP still have to be backwards compatible in the sense that APIs can?t change their default behavior and such. In other words we can?t suddenly flip on hostname checking or anything like that. > > Then the words should be clarified (maybe by removing 'general'?). This PEP invites interpretation by future generations so it should be as clear as possible on the intent, to avoid scholarly arguments. > > -- > --Guido van Rossum (python.org/~guido) Yea I agree, was just stating what I understand the PEP to be proposing :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From martin at v.loewis.de Tue Mar 25 19:29:30 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 25 Mar 2014 19:29:30 +0100 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: References: Message-ID: <5331CB0A.6080603@v.loewis.de> Am 25.03.14 14:47, schrieb Nick Coghlan: > The PEP says to sync with Python 3, and that has full cross platform > support. The Linux focus just comes from the fact that Linux is where > the problem is most evident. However, it fails to address a critical detail: the upcoming maintenance end for 2.7. This makes it less platform-agnostic than you say: For Linux, many users rely on the system vendor to provide Python binary, and they may chose to continue improving the backported APIs even after python-dev's maintenance ends. OTOH, for Windows, most users rely on the binaries from python.org, which will stop evolving in May 2015. Of course, there is still ActiveState, but there isn't the same kind of vendor choice that you see for Linux. So I think the PEP should make it clear whether: a) you expect that the maintenance period for Python 2.7 is to be extended beyond 2015, b) you expect that the backports will evolve in the years to come, or whether this will be a one-shot activity c) and if the answer to b was "will evolve", whether you expect this will create forks of 2.7 after maintenance has ended (as the then-security-only 2.7 branch won't see any further improvement to the backport, according to the PEP) Regards, Martin From turnbull at sk.tsukuba.ac.jp Tue Mar 25 19:53:30 2014 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 26 Mar 2014 03:53:30 +0900 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: References: Message-ID: <87eh1qqfdh.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > Changes to these modules will still need to undergo normal backwards > compatibility assessments to ensure their default behaviour remains > consistent with earlier Python 2.7 releases, I find this wording confusing. Does this mean that (possibly imaginary example) where Python 3.n enables a new "host match" check by default that Python 2.7 doesn't have, Python 2.7 gets the ability to check "host match", but it defaults to off? > While this PEP does not make any changes to the core development team's > handling of security-fix-only branches that are no longer in active > maintenance, I wonder if it would be possible to generalize the approach to 2.7 in such a way that our repos can collect backports of security *enhancements* which we have reviewed, even though they wouldn't be distributed in a release? That way the distros can share their patches to 2.x, x < 7. > infrastructure on their platform. Accordingly, and independently of this > PEP, the Mac OS X binary installers were already going to be switched to > statically linker newer versions of OpenSSL [4]_ Typo: statically linker -> statically-linked > For better or for worse (mostly worse), there are some environments where Is that "For better or for worse (mostly worse)" really necessary? Consenting adults and all that. > the risk of latent security defects is more tolerated than even a slightly > increased risk of regressions in maintenance releases. This policy largely > excludes these environments from consideration where the modules covered by > the exemption are concerned Again, I think you can put a period here. The people making such decisions know what they're doing, at least they think they do. Just tell them they're on they're own now, and their complaints, if any, will fall on deaf ears. > Downstream redistributors may still choose to cater to such environments, > but they will need to handle the process of downgrading the security > related modules and doing the associated regression testing themselves. > The main CPython continuous integration infrastructure will not cover this > scenario. This is how to write it, I think. > Motivation and Rationale > ======================== > > This PEP can be seen as a more targeted version of the "faster standard > library release cycle" proposals discussed in PEP 407 and PEP 413, > focusing specifically on those areas which have implications beyond the > Python community. FWIW, I don't see it that way at all. > It is worth comparing the approach described in this PEP with Red Hat's > handling of its long term support commitments: it isn't the RHEL 6.0 release > itself that receives 10 years worth of support, but the overall RHEL 6 > *series*. The individual RHEL 6.x point releases within the series then > receive a wide variety of new features, including security enhancements, > all while meeting strict backwards compatibility guarantees for existing > software. I don't understand this comparison. I think it requires too much familiarity with Red Hat policy to be useful to the great majority of readers, ie, it tells them a lot more about Red Hat then it does about Python. > However, now that we're fully aware of the impact the limitations of the > Python 2 standard library may be having on the evolution of internet > security standards, I suggest s/evolution/implementation/ here. > As Terry Reedy noted, URL? > Alternative: create and release Python 2.8 > ------------------------------------------ I don't see how this can possibly be viable. It would have to involve essentially the same restrictions as this PEP, or it completely fails to satisfy the requirements of downstreams. Even then, it would be a huge, uphill PR battle for downstreams trying to enhance their current 2.7 with the content of this PEP. > > With sufficient corporate support, it likely *would* be possible to create > and release Python 2.8 (it's highly unlikely such a project would garner > enough interest to be achievable with only volunteers). However, this > wouldn't actually solve the problem, as the aim is to provide a *relatively > low impact* way to incorporate enhanced security features into integrated > products and deployments that make use of Python 2. > > Upgrading to a new Python feature release would mean both more work for the > core development team, as well as a more disruptive update that most > potential end users would likely just skip entirely. > > Attempting to create a Python 2.8 release would also bring in suggestions > to backport many additional features from Python 3 (such as ``tracemalloc`` > and the improved coroutine support), making the migration from Python 2.7 > to this hypothetical 2.8 release even riskier and more disruptive. > > This is not a recommended approach, as it would involve substantial > additional work for a result that is actually less effective in achieving > the original aim (which is to eliminate the current widespread use of the > aging network security infrastructure in the Python 2 series). > > Furthermore, while I can't make any commitments to actually addressing > this issue on Red Hat platforms, I *can* categorically rule out the idea > of a Python 2.8 being of any use to me in even attempting to get it > addressed. > > > Alternative: distribute the security enhancements via PyPI > ---------------------------------------------------------- > > While this initially appears to be an attractive and easier to manage > approach, it actually suffers from several significant problems. > > Firstly, this is complex, low level, cross-platform code that integrates > with the underlying operating system across a variety of POSIX platforms > (including Mac OS X) and Windows. The CPython BuildBot fleet is already set > up to handle continuous integration in that context, but most of the > freely available continuous integration services just offer Linux, and > perhaps paid access to Windows. Those services work reasonably well for > software that largely runs on the abstraction layers offered by Python and > other dynamic languages, as well as the more comprehensive abstraction > offered by the JVM, but won't suffice for the kind of code involved here. > > The OpenSSL dependency for the network security support also qualifies as > the kind of "complex binary dependency" that isn't yet handled well by the > ``pip`` based software distribution ecosystem. Relying on a third party > binary dependency also creates potential compatibility problems for ``pip`` > when running on other interpreters like ``PyPy``. > > Another practical problem with the idea is the fact that ``pip`` itself > relies on the ``ssl`` support in the standard library (with some additional > support from a bundled copy of ``requests``, which in turn bundles > ``backport.ssl_match_hostname``), and hence would require any replacement > module to also be bundled within ``pip``. This wouldn't pose any > insurmountable difficulties (it's just another dependency to vendor), but > it *would* mean yet another copy of OpenSSL to keep up to date. > > This approach also has the same flaw as all other "improve security by > renaming things" approaches: they completely miss the users who most need > help, and raise significant barriers against being able to encourage users > to do the right thing when their infrastructure supports it (since > "use this other module" is a much higher impact change than "turn on this > higher security setting"). Deprecating the aging SSL infrastructure in the > standard library in favour of an external module would be even more user > hostile than accepting the slightly increased risk of regressions associated > with upgrading it in place. > > Last, but certainly not least, this approach suffers from the same problem > as the idea of doing a Python 2.8 release: likely not solving the actual > problem. Commercial redistributors of Python are set up to redistribute > *Python*, and a pre-existing set of additional packages. Getting new > packages added to the pre-existing set *can* be done, but means approaching > each and every redistributor and asking them to update their > repackaging process accordingly. By contrast, the approach described in > this PEP would require redistributors to deliberately *opt out* of the > security enhancements by deliberately downgrading the provided network > security infrastructure, which most of them are unlikely to do. > > > Alternative: provide a "legacy SSL infrastructure" branch > --------------------------------------------------------- > > Earlier versions of this PEP included the concept of a ``2.7-legacy-ssl`` > branch that preserved the exact feature set of the Python 2.7.6 network > security infrastructure. > > It is the opinion of the PEP author that anyone that actually wants this is > almost certainly making a mistake, and if they insist they really do want > it in their specific situation, they're welcome to either make it themselves > or arrange for a downstream redistributor to make it for them. > > If they are made publicly available, any such rebuilds should be referred to > as "Python 2.7 with Legacy SSL" to clearly distinguish them from the official > Python 2.7 releases that include more up to date network security > infrastructure. > > After the first Python 2.7 maintenance release that has the security > infrastructure updated to match Python 3.4, it would also be appropriate to > refer to Python 2.7.6 and earlier releases as "Python 2.7 with Legacy SSL". > > > Alternative: selectively backport particular APIs > ------------------------------------------------- > > An instinctively minimalist reaction to this proposal is to only backport > particular APIs in the affected modules that are judged to be "security > critical". However, this ends up providing a worse end user experience, > as well as a worse developer experience. > > For end users, the selective backporting approach means learning not only > the legacy Python 2.7 API and the current Python 3 APIs, but also the > hybrid API created by the selective backporting process. It is much > simpler to just learn the APIs of the original Python 2.7 feature release > and the relevant Python 3 features releases, without trying to define a new > hybrid API that only exists in later Python 2.7 maintenance branches. > > For developers, the main benefit of the "just align the available > functionality with Python 3" approach is that it reduces the amount of > design work needing when updating Python 2.7 with new security features. > The "feature or fix?" and "security related or not?" debates are both > handled in advance by this policy, leaving only the matter of ensuring > that backwards compatibility is preserved as needed by adjusting the > default behaviour in the Python 2.7 backport when appropriate. > > > Open Questions > ============== > > * MvL has indicated he is not prepared to tackle the task of trying to > integrate a newer OpenSSL into the also aging Python 2.7 build > infrastructure on Windows (unfortunately, we've looked into upgrading > that build infrastructure, and the backwards compatibility issues > appear to be effectively insurmountable). We would require a commitment > from another trusted contributor to handle at least this task, and > potentially also taking over the task of creating the official > Python 2.7 Windows installers for the remaining Python 2.7 maintenance > releases. > > * We would need commitments to create and review full backports of the > components covered by this policy from Python 3.4 to Python 2.7, as well > as support for handling any more specific security issues affecting these > modules. > > * I believe I've addressed all the technical and scope questions I had, or > others raised. That just leaves the question of "Do we agree this plan > actually makes sense, given the constraints on downstream redistributors > that would also like to see this problem solved?" :) > > > Disclosure of Interest > ====================== > > The author of this PEP currently works for Red Hat on test automation tools. > If this proposal is accepted, I will be strongly encouraging Red Hat to take > advantage of the resulting opportunity to help improve the overall security > of the Python ecosystem. However, I do not speak for Red Hat in this matter, > and cannot make any commitments on Red Hat's behalf. > > > Acknowledgements > ================ > > Thanks to Christian Heimes for his recent efforts on greatly improving > Python's SSL support in the Python 3 series, and a variety of members of > the Python community for helping me to better understand the implications > of the default settings we provide in our SSL modules, and the impact that > tolerating the use of SSL infrastructure that was defined in 2010 > (Python 2.7) or even 2008 (Python 2.6) potentially has for the security > of the web as a whole. > > Christian and Donald Stufft also provided valuable feedback on a preliminary > draft of this proposal. > > Thanks also to participants in the python-dev mailing list threads [1,2,5]_ > > > References > ========== > > .. [1] https://mail.python.org/pipermail/python-dev/2014-March/133334.html > .. [2] https://mail.python.org/pipermail/python-dev/2014-March/133389.html > .. [3] https://mail.python.org/pipermail/python-dev/2014-March/133438.html > .. [4] https://mail.python.org/pipermail/python-dev/2014-March/133347.html > .. [5] https://mail.python.org/pipermail/python-dev/2014-March/133442.html > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/stephen%40xemacs.org From stephen at xemacs.org Tue Mar 25 20:09:21 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 26 Mar 2014 04:09:21 +0900 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: References: Message-ID: <87d2haqen2.fsf@uwakimon.sk.tsukuba.ac.jp> Urk. Premature send. As it turns out, I had no specific comments to make after the one comment on > Alternative: create and release Python 2.8 > ------------------------------------------ My apologies to anybody who read to the bottom for wasting their time. I do have one generic comment on style: I think at this point the statements tagged with "I (Nick Coghlan) think" should either be edited to remove the reference to the author, or moved to the "Caveat: I work for RH" section or the "Open Issues" section. If they're not pretty much consensus by now, they probably don't belong in the main body of the PEP. From ncoghlan at gmail.com Tue Mar 25 23:15:29 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Mar 2014 08:15:29 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E0BA8.5070709@v.loewis.de> <2BDDCC2A-9A07-4DB4-9727-3ECFB3D49931@stufft.io> <532E0E22.6080304@v.loewis.de> <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> <5330E929.2070607@mrabarnett.plus.com> Message-ID: On 26 Mar 2014 01:19, "Brett Cannon" wrote: > As long as we make it clear we have chosen to change our backwards-compatibility guarantees in the name of security and have a link to the last backwards-compatible release then I agree as well. I am not sure how this meme got started, but let me be clear: the proposed policy DOES NOT provide blanket permission to break backwards compatibility in the affected modules. It only allows ADDING new features to bring these modules into line with their Python 3 counterparts, making it easier for third party packages like requests to do the right thing in a cross-version compatible way. The "use feature detection, not version checks" guidelines in the PEP are designed to deal with the concerns around subsequently missing features in the existing Python 2.7 releases. The remaining concern appears to be largely around the slightly increased chance of regressions that comes with making larger changes to these modules in order to incorporate the new features. Given our regression test suite, and those of other projects like OpenStack and components of the Linux distributions, I now consider that concern to be entirely misplaced. The only "backwards compatibility breaks allowed" general exemption applies to the new "ssl.create_default_context()" function, which *is* defined as allowing backwards incompatible changes to keep up with evolving security requirements. That exemption is in the documentation of that API, though - the only impact of this PEP would be to also make that API available in 2.7.7+. Regards, Nick. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Mar 25 23:19:47 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 25 Mar 2014 23:19:47 +0100 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen Message-ID: <20140325231947.5ec9ad63@fsol> Hi, On core-mentorship someone asked about PEP 3145 - Asynchronous I/O for subprocess.popen. I answered that asyncio now has subprocess support (including non-blocking I/O on the three standard stream pipes), so it's not obvious anything else is needed. Should we change the PEP's status to Rejected or Superseded? Regards Antoine. From g.brandl at gmx.net Tue Mar 25 23:30:22 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 25 Mar 2014 23:30:22 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> <5330E929.2070607@mrabarnett.plus.com> Message-ID: Am 25.03.2014 23:15, schrieb Nick Coghlan: > > On 26 Mar 2014 01:19, "Brett Cannon" > wrote: >> As long as we make it clear we have chosen to change our > backwards-compatibility guarantees in the name of security and have a link to > the last backwards-compatible release then I agree as well. > > I am not sure how this meme got started, but let me be clear: the proposed > policy DOES NOT provide blanket permission to break backwards compatibility in > the affected modules. It only allows ADDING new features to bring these modules > into line with their Python 3 counterparts, making it easier for third party > packages like requests to do the right thing in a cross-version compatible way. We know. That's what we mean by that. Georg From solipsis at pitrou.net Tue Mar 25 23:33:55 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 25 Mar 2014 23:33:55 +0100 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements References: Message-ID: <20140325233355.14cba72c@fsol> On Tue, 25 Mar 2014 23:09:45 +1000 Nick Coghlan wrote: > > Alternative: selectively backport particular APIs > ------------------------------------------------- > > An instinctively minimalist reaction to this proposal is to only backport > particular APIs in the affected modules that are judged to be "security > critical". However, this ends up providing a worse end user experience, > as well as a worse developer experience. > > For end users, the selective backporting approach means learning not only > the legacy Python 2.7 API and the current Python 3 APIs, but also the > hybrid API created by the selective backporting process. I think this is a strawman, since you are also advocating for a "feature detection" approach to writing cross-version code. It is already required, actually, if wanting to write code compatible from 3.2 to 3.4 (for example, SSLContext exists in 3.2 but create_default_context appears in 3.4 while OP_NO_COMPRESSION appears in 3.3). I would much rather selectively backport a minimal set of APIs than the whole range of ssl APIs. There are things there (RAND_bytes, RAND_pseudo_bytes) that are not even useful for network security, or only in a rather uncommon manner (such as channel bindings). Regards Antoine. From ncoghlan at gmail.com Tue Mar 25 23:35:37 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Mar 2014 08:35:37 +1000 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: References: Message-ID: On 26 Mar 2014 00:25, "Paul Moore" wrote: > > On 25 March 2014 13:47, Nick Coghlan wrote: > > It's not like we're going to just be giving the PEP to vendors as a spec and > > leaving them to it - it's largely an invitation to participate more directly > > upstream to help resolve a particularly thorny problem, not a Statement of > > Work :) > > :-) > > I don't really know the APIs involved, but AIUI one of the 3.4 > enhancements is exposing the SSLContext. Is the code to do this > compatible with the version of OpenSSL bundled with Python 2.7 on > Windows? If not, suppose that Red Hat provide resources that work on > backporting the code, but they don't have Windows experts so no-one > deals with integrating the new OpenSSL into the Windows binaries. > Would the backport be blocked until someone is found to do the Windows > work? We'll get it done. For example, while *I* definitely approach the problem from a Linux vendor perspective (and that's reflected in the PEP), I also know several folks at Rackspace have expressed concern about the status quo, and the client side of OpenStack is cross platform. > This "I've written a patch but it hasn't been applied" is the type of > scenario that puts people off contributing. If it's likely to happen, > I think Red Hat have a right to know that in advance. And I don't know > that it's something they would appreciate without python-dev pointing > it out. If we're reasonably sure (not necessarily certain, there's > always grey areas) that this isn't going to be an issue, then that's > also fine. We can simply say that. Yeah, I think we can make sure the right folks are involved to make it happen. The PEP is about me getting agreement in advance that we actually want to see the problem fixed, and the constraints we want to impose on the solution. Once we have that agreement, I won't be sitting around idly *waiting* for assistance to magically appear - I'll go looking for it, and I know there are others that will do the same :) > That's all I'm saying. Not trying to require anything of contributors, > just trying to be open and explicit about the criteria that will apply > to accepting contributions. I won't expect Linux folks to fix Windows problems (as that rarely works well). If the PEP is accepted, I *will* ensure we get the policy implemented on all supported platforms for 2.7.7+ by getting appropriate people involved (and will also work on securing the appropriate longer term support commitments). Cheers, Nick. > > Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Mar 26 00:03:52 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Mar 2014 09:03:52 +1000 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: <20140325233355.14cba72c@fsol> References: <20140325233355.14cba72c@fsol> Message-ID: On 26 Mar 2014 08:35, "Antoine Pitrou" wrote: > > On Tue, 25 Mar 2014 23:09:45 +1000 > Nick Coghlan wrote: > > > > Alternative: selectively backport particular APIs > > ------------------------------------------------- > > > > An instinctively minimalist reaction to this proposal is to only backport > > particular APIs in the affected modules that are judged to be "security > > critical". However, this ends up providing a worse end user experience, > > as well as a worse developer experience. > > > > For end users, the selective backporting approach means learning not only > > the legacy Python 2.7 API and the current Python 3 APIs, but also the > > hybrid API created by the selective backporting process. > > I think this is a strawman, since you are also advocating for a > "feature detection" approach to writing cross-version code. It is > already required, actually, if wanting to write code compatible from > 3.2 to 3.4 (for example, SSLContext exists in 3.2 but > create_default_context appears in 3.4 while OP_NO_COMPRESSION appears > in 3.3). > > I would much rather selectively backport a minimal set of APIs than the > whole range of ssl APIs. There are things there (RAND_bytes, > RAND_pseudo_bytes) that are not even useful for network security, or > only in a rather uncommon manner (such as channel bindings). Yeah, I think this is a valid point, and, as Guido noted, we also want the option to skip backporting things if they depend on other aspects of Python 3 that we decide can't be backported. So a feature-by-feature decision making process actually does make more sense than a blanket exemption. Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Mar 26 00:06:47 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Mar 2014 09:06:47 +1000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> <5330E929.2070607@mrabarnett.plus.com> Message-ID: On 26 Mar 2014 08:32, "Georg Brandl" wrote: > > Am 25.03.2014 23:15, schrieb Nick Coghlan: > > > > On 26 Mar 2014 01:19, "Brett Cannon" > > wrote: > >> As long as we make it clear we have chosen to change our > > backwards-compatibility guarantees in the name of security and have a link to > > the last backwards-compatible release then I agree as well. > > > > I am not sure how this meme got started, but let me be clear: the proposed > > policy DOES NOT provide blanket permission to break backwards compatibility in > > the affected modules. It only allows ADDING new features to bring these modules > > into line with their Python 3 counterparts, making it easier for third party > > packages like requests to do the right thing in a cross-version compatible way. > > We know. That's what we mean by that. That's not what Brett said - he called 2.7.6 the "last backwards compatible release". That's not correct, as even under my proposal, 2.7.7+ will still be backwards compatible. Cheers, Nick. > > Georg > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Mar 26 00:14:02 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Mar 2014 09:14:02 +1000 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: <20140325231947.5ec9ad63@fsol> References: <20140325231947.5ec9ad63@fsol> Message-ID: On 26 Mar 2014 08:22, "Antoine Pitrou" wrote: > > > Hi, > > On core-mentorship someone asked about PEP 3145 - Asynchronous I/O for > subprocess.popen. I answered that asyncio now has subprocess support > (including non-blocking I/O on the three standard stream pipes), so > it's not obvious anything else is needed. > > Should we change the PEP's status to Rejected or Superseded? Yes. I think we'd typically use Rejected in this case, as Superseded normally relates to the evolution of interface definition PEPs. Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Mar 26 00:14:04 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 25 Mar 2014 16:14:04 -0700 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: <20140325231947.5ec9ad63@fsol> References: <20140325231947.5ec9ad63@fsol> Message-ID: That would be a rather strong unilateral decision. Why don't you ask the authors? In theory the PEP's proposals could serve in situations where asyncio isn't appropriate, and asyncio's subprocess I/O isn't the smoothest API imaginable. (In practice I'm not sure if the PEP would have been written with asyncio subprocess support in place.) On Tue, Mar 25, 2014 at 3:19 PM, Antoine Pitrou wrote: > > Hi, > > On core-mentorship someone asked about PEP 3145 - Asynchronous I/O for > subprocess.popen. I answered that asyncio now has subprocess support > (including non-blocking I/O on the three standard stream pipes), so > it's not obvious anything else is needed. > > Should we change the PEP's status to Rejected or Superseded? > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Mar 26 00:24:06 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 25 Mar 2014 19:24:06 -0400 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: References: <20140325233355.14cba72c@fsol> Message-ID: On Mar 25, 2014, at 7:03 PM, Nick Coghlan wrote: > > On 26 Mar 2014 08:35, "Antoine Pitrou" wrote: > > > > On Tue, 25 Mar 2014 23:09:45 +1000 > > Nick Coghlan wrote: > > > > > > Alternative: selectively backport particular APIs > > > ------------------------------------------------- > > > > > > An instinctively minimalist reaction to this proposal is to only backport > > > particular APIs in the affected modules that are judged to be "security > > > critical". However, this ends up providing a worse end user experience, > > > as well as a worse developer experience. > > > > > > For end users, the selective backporting approach means learning not only > > > the legacy Python 2.7 API and the current Python 3 APIs, but also the > > > hybrid API created by the selective backporting process. > > > > I think this is a strawman, since you are also advocating for a > > "feature detection" approach to writing cross-version code. It is > > already required, actually, if wanting to write code compatible from > > 3.2 to 3.4 (for example, SSLContext exists in 3.2 but > > create_default_context appears in 3.4 while OP_NO_COMPRESSION appears > > in 3.3). > > > > I would much rather selectively backport a minimal set of APIs than the > > whole range of ssl APIs. There are things there (RAND_bytes, > > RAND_pseudo_bytes) that are not even useful for network security, or > > only in a rather uncommon manner (such as channel bindings). > > Yeah, I think this is a valid point, and, as Guido noted, we also want the option to skip backporting things if they depend on other aspects of Python 3 that we decide can't be backported. > > So a feature-by-feature decision making process actually does make more sense than a blanket exemption. > > Looking at the ssl module, just about the only thing that wouldn?t be helpful to have backported is the rand functions that AP mentioned (and those mostly because everyone should just use os.urandom for everything ever). The NPN stuff isn?t important for security related stuff either though it?d probably be more work to rip it out and more disruptive as well. Looking at the hmac module, the only thing that really matters for a backport would be the constant time compare. Lokoing at hashlib, the guaranteed and supported algorithms would be nice to have but not really security sensitive. Adding pbkdf2_hmac would be really nice for the security of web services though. Looking at os.urandom, back porting the lazily opened and held file descriptor would be a nice to have, but not strictly required. (Not sure if that would be considered a backwards incompat change). Nothing in random.py really stands out to me looking at it?s docs. > Cheers, > Nick. > > > > > Regards > > > > Antoine. > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From solipsis at pitrou.net Wed Mar 26 00:24:56 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 Mar 2014 00:24:56 +0100 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: <20140326002456.0f150a3f@fsol> On Tue, 25 Mar 2014 16:14:04 -0700 Guido van Rossum wrote: > That would be a rather strong unilateral decision. Why don't you ask the > authors? In theory the PEP's proposals could serve in situations where > asyncio isn't appropriate, and asyncio's subprocess I/O isn't the smoothest > API imaginable. (In practice I'm not sure if the PEP would have been > written with asyncio subprocess support in place.) That's a good point. I now have e-mailed Eric Pruitt and Josiah Carlson (I couldn't find an e-mail for Charles R. McCreary). Regards Antoine. From ethan at stoneleaf.us Tue Mar 25 23:37:11 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 25 Mar 2014 15:37:11 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 Message-ID: <53320517.3010101@stoneleaf.us> Okay, I included that last round of comments (from late February). Barring typos, this should be the final version. Final comments? ----------------------------------------------------------------------------- PEP: 461 Title: Adding % formatting to bytes and bytearray Version: $Revision$ Last-Modified: $Date$ Author: Ethan Furman Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2014-01-13 Python-Version: 3.5 Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25 Resolution: Abstract ======== This PEP proposes adding % formatting operations similar to Python 2's ``str`` type to ``bytes`` and ``bytearray`` [1]_ [2]_. Rationale ========= While interpolation is usually thought of as a string operation, there are cases where interpolation on ``bytes`` or ``bytearrays`` make sense, and the work needed to make up for this missing functionality detracts from the overall readability of the code. Motivation ========== With Python 3 and the split between ``str`` and ``bytes``, one small but important area of programming became slightly more difficult, and much more painful -- wire format protocols [3]_. This area of programming is characterized by a mixture of binary data and ASCII compatible segments of text (aka ASCII-encoded text). Bringing back a restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in writing new wire format code, and in porting Python 2 wire format code. Common use-cases include ``dbf`` and ``pdf`` file formats, ``email`` formats, and ``FTP`` and ``HTTP`` communications, among many others. Proposed semantics for ``bytes`` and ``bytearray`` formatting ============================================================= %-interpolation --------------- All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, ``%g``, etc.) will be supported, and will work as they do for str, including the padding, justification and other related modifiers. The only difference will be that the results from these codes will be ASCII-encoded text, not unicode. In other words, for any numeric formatting code `%x`:: b"%x" % val is equivalent to ("%x" % val).encode("ascii") Examples:: >>> b'%4x' % 10 b' a' >>> b'%#4x' % 10 ' 0xa' >>> b'%04X' % 10 '000A' ``%c`` will insert a single byte, either from an ``int`` in range(256), or from a ``bytes`` argument of length 1, not from a ``str``. Examples:: >>> b'%c' % 48 b'0' >>> b'%c' % b'a' b'a' ``%s`` is included for two reasons: 1) `b` is already a format code for ``format`` numerics (binary), and 2) it will make 2/3 code easier as Python 2.x code uses ``%s``; however, it is restricted in what it will accept:: - input type supports ``Py_buffer`` [6]_? use it to collect the necessary bytes - input type is something else? use its ``__bytes__`` method [7]_ ; if there isn't one, raise a ``TypeError`` In particular, ``%s`` will not accept numbers (use a numeric format code for that), nor ``str`` (encode it to ``bytes``). Examples:: >>> b'%s' % b'abc' b'abc' >>> b'%s' % 'some string'.encode('utf8') b'some string' >>> b'%s' % 3.14 Traceback (most recent call last): ... TypeError: b'%s' does not accept numbers, use a numeric code instead >>> b'%s' % 'hello world!' Traceback (most recent call last): ... TypeError: b'%s' does not accept 'str', it must be encoded to `bytes` ``%a`` will call ``ascii()`` on the interpolated value. This is intended as a debugging aid, rather than something that should be used in production. Non-ASCII values will be encoded to either ``\xnn`` or ``\unnnn`` representation. Use cases include developing a new protocol and writing landmarks into the stream; debugging data going into an existing protocol to see if the problem is the protocol itself or bad data; a fall-back for a serialization format; or even a rudimentary serialization format when defining ``__bytes__`` would not be appropriate [8]. .. note:: If a ``str`` is passed into ``%a``, it will be surrounded by quotes. Unsupported codes ----------------- ``%r`` (which calls ``__repr__`` and returns a ``str``) is not supported. Proposed variations =================== It was suggested to let ``%s`` accept numbers, but since numbers have their own format codes this idea was discarded. It has been suggested to use ``%b`` for bytes as well as ``%s``. This was rejected as not adding any value either in clarity or simplicity. It has been proposed to automatically use ``.encode('ascii','strict')`` for ``str`` arguments to ``%s``. - Rejected as this would lead to intermittent failures. Better to have the operation always fail so the trouble-spot can be correctly fixed. It has been proposed to have ``%s`` return the ascii-encoded repr when the value is a ``str`` (b'%s' % 'abc' --> b"'abc'"). - Rejected as this would lead to hard to debug failures far from the problem site. Better to have the operation always fail so the trouble-spot can be easily fixed. Originally this PEP also proposed adding format-style formatting, but it was decided that format and its related machinery were all strictly text (aka ``str``) based, and it was dropped. Various new special methods were proposed, such as ``__ascii__``, ``__format_bytes__``, etc.; such methods are not needed at this time, but can be visited again later if real-world use shows deficiencies with this solution. Objections ========== The objections raised against this PEP were mainly variations on two themes:: - the ``bytes`` and ``bytearray`` types are for pure binary data, with no assumptions about encodings - offering %-interpolation that assumes an ASCII encoding will be an attractive nuisance and lead us back to the problems of the Python 2 ``str``/``unicode`` text model As was seen during the discussion, ``bytes`` and ``bytearray`` are also used for mixed binary data and ASCII-compatible segments: file formats such as ``dbf`` and ``pdf``, network protocols such as ``ftp`` and ``email``, etc. ``bytes`` and ``bytearray`` already have several methods which assume an ASCII compatible encoding. ``upper()``, ``isalpha()``, and ``expandtabs()`` to name just a few. %-interpolation, with its very restricted mini-language, will not be any more of a nuisance than the already existing methods. Some have objected to allowing the full range of numeric formatting codes with the claim that decimal alone would be sufficient. However, at least two formats (dbf and pdf) make use of non-decimal numbers. Footnotes ========= .. [1] http://docs.python.org/2/library/stdtypes.html#string-formatting .. [2] neither string.Template, format, nor str.format are under consideration .. [3] https://mail.python.org/pipermail/python-dev/2014-January/131518.html .. [4] to use a str object in a bytes interpolation, encode it first .. [5] %c is not an exception as neither of its possible arguments are str .. [6] http://docs.python.org/3/c-api/buffer.html examples: ``memoryview``, ``array.array``, ``bytearray``, ``bytes`` .. [7] http://docs.python.org/3/reference/datamodel.html#object.__bytes__ .. [8] https://mail.python.org/pipermail/python-dev/2014-February/132750.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From alex.gaynor at gmail.com Wed Mar 26 00:47:38 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 25 Mar 2014 23:47:38 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?PEP_466_=28round_4=29=3A_Python_2=2E7_netw?= =?utf-8?q?ork_security=09enhancements?= References: <20140325233355.14cba72c@fsol> Message-ID: At this I think this PEP has become a little too vague and abstract, and I think we'd probably be better served by getting more concrete: Problem: Some of Python 2's modules which are fundamentally necessary for interop with the broader internet, and the security thereof, are missing really important features. Right now Python 2 has a policy of getting absolutely new features. Solution: We're going to ignore that policy for a couple of pretty important features to that end. Here's my proposed list of such featuers: * hmac * constant_time_compare * os * Persisant FD for os.urandom() * ssl * SNI * SSLContext * A giant suite of constants from OpenSSL * The functions for checking a hostname against a certificate * The functions for finding the platform's certificate store Alex From Nikolaus at rath.org Wed Mar 26 01:54:09 2014 From: Nikolaus at rath.org (Nikolaus Rath) Date: Tue, 25 Mar 2014 17:54:09 -0700 Subject: [Python-Dev] SSLSocket.send() for non-blocking sockets Message-ID: <87mwgdye32.fsf@vostro.rath.org> Hello, I'd like to hear some more opinions on http://bugs.python.org/issue20951. I think the possible courses of action are: 1. Document the current behavior. This has the drawback that (a) it still remains rather counterintuitive, and (b) one still needs extra code to distinguish between a zero-return because the write would block, and a zero-return because the peer has closed the connection. I'm not sure about the best way to do this check. I would probably try to run select() on the underlying raw socket. Is there a better way? 2. Change the behavior immediately, potentially breaking some applications that worked around it, but unbreaking others that relied on the documented behavior. 3. Deprecate the current behavior, and introduce a transition period where the new behavior is controlled by a flag. Drawbacks: no direct breakage of anything, but additional complexity may indirectly cause bugs. Thoughts? Best, -Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ?Time flies like an arrow, fruit flies like a Banana.? From rosuav at gmail.com Wed Mar 26 02:27:31 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 26 Mar 2014 12:27:31 +1100 Subject: [Python-Dev] SSLSocket.send() for non-blocking sockets In-Reply-To: <87mwgdye32.fsf@vostro.rath.org> References: <87mwgdye32.fsf@vostro.rath.org> Message-ID: On Wed, Mar 26, 2014 at 11:54 AM, Nikolaus Rath wrote: > 2. Change the behavior immediately, potentially breaking some > applications that worked around it, but unbreaking others that relied > on the documented behavior. If it's a functionality change that's likely to break code, would it be worth changing it only in 3.5, and documenting it as broken in 3.4 and earlier? ChrisA From ethan at stoneleaf.us Wed Mar 26 02:43:30 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 25 Mar 2014 18:43:30 -0700 Subject: [Python-Dev] C code: %s vs %U Message-ID: <533230C2.60302@stoneleaf.us> %s is a string. %U is unicode? If so, then %s should only be used when it is certain the string in question has no unicode in it? -- ~Ethan~ From dholth at gmail.com Wed Mar 26 03:10:57 2014 From: dholth at gmail.com (Daniel Holth) Date: Tue, 25 Mar 2014 22:10:57 -0400 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <53320517.3010101@stoneleaf.us> References: <53320517.3010101@stoneleaf.us> Message-ID: I love it. On Tue, Mar 25, 2014 at 6:37 PM, Ethan Furman wrote: > Okay, I included that last round of comments (from late February). > > Barring typos, this should be the final version. > > Final comments? > > ----------------------------------------------------------------------------- > PEP: 461 > Title: Adding % formatting to bytes and bytearray > Version: $Revision$ > Last-Modified: $Date$ > Author: Ethan Furman > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 2014-01-13 > Python-Version: 3.5 > Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25 > Resolution: > > > Abstract > ======== > > This PEP proposes adding % formatting operations similar to Python 2's > ``str`` > type to ``bytes`` and ``bytearray`` [1]_ [2]_. > > > Rationale > ========= > > While interpolation is usually thought of as a string operation, there are > cases where interpolation on ``bytes`` or ``bytearrays`` make sense, and the > work needed to make up for this missing functionality detracts from the > overall > readability of the code. > > > Motivation > ========== > > With Python 3 and the split between ``str`` and ``bytes``, one small but > important area of programming became slightly more difficult, and much more > painful -- wire format protocols [3]_. > > This area of programming is characterized by a mixture of binary data and > ASCII compatible segments of text (aka ASCII-encoded text). Bringing back a > restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in > writing new wire format code, and in porting Python 2 wire format code. > > Common use-cases include ``dbf`` and ``pdf`` file formats, ``email`` > formats, and ``FTP`` and ``HTTP`` communications, among many others. > > > Proposed semantics for ``bytes`` and ``bytearray`` formatting > ============================================================= > > %-interpolation > --------------- > > All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, > ``%g``, etc.) will be supported, and will work as they do for str, including > the padding, justification and other related modifiers. The only difference > will be that the results from these codes will be ASCII-encoded text, not > unicode. In other words, for any numeric formatting code `%x`:: > > b"%x" % val > > is equivalent to > > ("%x" % val).encode("ascii") > > Examples:: > > >>> b'%4x' % 10 > b' a' > > >>> b'%#4x' % 10 > ' 0xa' > > >>> b'%04X' % 10 > '000A' > > ``%c`` will insert a single byte, either from an ``int`` in range(256), or > from > a ``bytes`` argument of length 1, not from a ``str``. > > Examples:: > > >>> b'%c' % 48 > b'0' > > >>> b'%c' % b'a' > b'a' > > ``%s`` is included for two reasons: 1) `b` is already a format code for > ``format`` numerics (binary), and 2) it will make 2/3 code easier as Python > 2.x > code uses ``%s``; however, it is restricted in what it will accept:: > > - input type supports ``Py_buffer`` [6]_? > use it to collect the necessary bytes > > - input type is something else? > use its ``__bytes__`` method [7]_ ; if there isn't one, raise a > ``TypeError`` > > In particular, ``%s`` will not accept numbers (use a numeric format code for > that), nor ``str`` (encode it to ``bytes``). > > Examples:: > > >>> b'%s' % b'abc' > b'abc' > > >>> b'%s' % 'some string'.encode('utf8') > b'some string' > > >>> b'%s' % 3.14 > Traceback (most recent call last): > ... > TypeError: b'%s' does not accept numbers, use a numeric code instead > > >>> b'%s' % 'hello world!' > Traceback (most recent call last): > ... > TypeError: b'%s' does not accept 'str', it must be encoded to `bytes` > > > ``%a`` will call ``ascii()`` on the interpolated value. This is intended > as a debugging aid, rather than something that should be used in production. > Non-ASCII values will be encoded to either ``\xnn`` or ``\unnnn`` > representation. Use cases include developing a new protocol and writing > landmarks into the stream; debugging data going into an existing protocol > to see if the problem is the protocol itself or bad data; a fall-back for a > serialization format; or even a rudimentary serialization format when > defining ``__bytes__`` would not be appropriate [8]. > > .. note:: > > If a ``str`` is passed into ``%a``, it will be surrounded by quotes. > > > Unsupported codes > ----------------- > > ``%r`` (which calls ``__repr__`` and returns a ``str``) is not supported. > > > Proposed variations > =================== > > It was suggested to let ``%s`` accept numbers, but since numbers have their > own > format codes this idea was discarded. > > It has been suggested to use ``%b`` for bytes as well as ``%s``. This was > rejected as not adding any value either in clarity or simplicity. > > It has been proposed to automatically use ``.encode('ascii','strict')`` for > ``str`` arguments to ``%s``. > > - Rejected as this would lead to intermittent failures. Better to have > the > operation always fail so the trouble-spot can be correctly fixed. > > It has been proposed to have ``%s`` return the ascii-encoded repr when the > value is a ``str`` (b'%s' % 'abc' --> b"'abc'"). > > - Rejected as this would lead to hard to debug failures far from the > problem > site. Better to have the operation always fail so the trouble-spot can > be > easily fixed. > > Originally this PEP also proposed adding format-style formatting, but it was > decided that format and its related machinery were all strictly text (aka > ``str``) based, and it was dropped. > > Various new special methods were proposed, such as ``__ascii__``, > ``__format_bytes__``, etc.; such methods are not needed at this time, but > can > be visited again later if real-world use shows deficiencies with this > solution. > > > Objections > ========== > > The objections raised against this PEP were mainly variations on two > themes:: > > - the ``bytes`` and ``bytearray`` types are for pure binary data, with no > assumptions about encodings > - offering %-interpolation that assumes an ASCII encoding will be an > attractive nuisance and lead us back to the problems of the Python 2 > ``str``/``unicode`` text model > > As was seen during the discussion, ``bytes`` and ``bytearray`` are also used > for mixed binary data and ASCII-compatible segments: file formats such as > ``dbf`` and ``pdf``, network protocols such as ``ftp`` and ``email``, etc. > > ``bytes`` and ``bytearray`` already have several methods which assume an > ASCII > compatible encoding. ``upper()``, ``isalpha()``, and ``expandtabs()`` to > name > just a few. %-interpolation, with its very restricted mini-language, will > not > be any more of a nuisance than the already existing methods. > > Some have objected to allowing the full range of numeric formatting codes > with > the claim that decimal alone would be sufficient. However, at least two > formats (dbf and pdf) make use of non-decimal numbers. > > > Footnotes > ========= > > .. [1] http://docs.python.org/2/library/stdtypes.html#string-formatting > .. [2] neither string.Template, format, nor str.format are under > consideration > .. [3] https://mail.python.org/pipermail/python-dev/2014-January/131518.html > .. [4] to use a str object in a bytes interpolation, encode it first > .. [5] %c is not an exception as neither of its possible arguments are str > .. [6] http://docs.python.org/3/c-api/buffer.html > examples: ``memoryview``, ``array.array``, ``bytearray``, ``bytes`` > .. [7] http://docs.python.org/3/reference/datamodel.html#object.__bytes__ > .. [8] > https://mail.python.org/pipermail/python-dev/2014-February/132750.html > > > Copyright > ========= > > This document has been placed in the public domain. > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com From tjreedy at udel.edu Wed Mar 26 04:59:51 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 25 Mar 2014 23:59:51 -0400 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <20140323010837.45776044@fsol> <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> <5330E929.2070607@mrabarnett.plus.com> Message-ID: On 3/25/2014 6:15 PM, Nick Coghlan wrote: > I am not sure how this meme got started, but let me be clear: the > proposed policy DOES NOT provide blanket permission to break backwards > compatibility in the affected modules. It only allows ADDING new > features to bring these modules into line with their Python 3 > counterparts, making it easier for third party packages like requests to > do the right thing in a cross-version compatible way. I think the problem is that 'no new features' covers two quite different types of 'new' lumped together: change-new and new-new. New-new adds something completely new and does not break any sensible behavior that we worry about. We freely add new-new features with each new version. We do not add them to maintenance releases because then maintenance releases would be new versions. There are also considerations of incubation time, and the increased possibility of regressions with candidate-release versus alpha-beta-candidate-release schedules. Change-new additions changing existing behavior (like changing a default) and can therefore break code that we care about. Even for new versions, we are wary them and may require deprecation. One of the ironies of change-new is that it is much more similar to change-fix, which is allowed as maintenance, than new-new is. The tracker controversies are about issues on the borderline between change-fix and change-new. The original version of the PEP reference 434 as precedent. While partly valid, the reference could have fed the confusion because 434 does not draw the distinction given above. Indeed, part of the point of the PEP is that separating change-fix from change-new is difficult. As far as regressions go, a big new-new patch touching several untested modules worries me more than a small change-??? patch tweaking something in only one module, especially after writing tests for the module. -- Terry Jan Reedy From Nikolaus at rath.org Wed Mar 26 05:37:16 2014 From: Nikolaus at rath.org (Nikolaus Rath) Date: Tue, 25 Mar 2014 21:37:16 -0700 Subject: [Python-Dev] SSLSocket.send() for non-blocking sockets In-Reply-To: (Chris Angelico's message of "Wed, 26 Mar 2014 12:27:31 +1100") References: <87mwgdye32.fsf@vostro.rath.org> Message-ID: <87k3bhy3r7.fsf@vostro.rath.org> Chris Angelico writes: > On Wed, Mar 26, 2014 at 11:54 AM, Nikolaus Rath wrote: >> 2. Change the behavior immediately, potentially breaking some >> applications that worked around it, but unbreaking others that relied >> on the documented behavior. > > If it's a functionality change that's likely to break code, would it > be worth changing it only in 3.5, and documenting it as broken in 3.4 > and earlier? Yes, that's what I meant. I don't think changing it in 3.4 is an option at all. Best, -Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ?Time flies like an arrow, fruit flies like a Banana.? From g.brandl at gmx.net Wed Mar 26 07:15:07 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 26 Mar 2014 07:15:07 +0100 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements In-Reply-To: References: <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> <5330E929.2070607@mrabarnett.plus.com> Message-ID: Am 26.03.2014 00:06, schrieb Nick Coghlan: > > On 26 Mar 2014 08:32, "Georg Brandl" > wrote: >> >> Am 25.03.2014 23:15, schrieb Nick Coghlan: >> > >> > On 26 Mar 2014 01:19, "Brett Cannon" >> > >> wrote: >> >> As long as we make it clear we have chosen to change our >> > backwards-compatibility guarantees in the name of security and have a link to >> > the last backwards-compatible release then I agree as well. >> > >> > I am not sure how this meme got started, but let me be clear: the proposed >> > policy DOES NOT provide blanket permission to break backwards compatibility in >> > the affected modules. It only allows ADDING new features to bring these modules >> > into line with their Python 3 counterparts, making it easier for third party >> > packages like requests to do the right thing in a cross-version compatible way. >> >> We know. That's what we mean by that. > > That's not what Brett said - he called 2.7.6 the "last backwards compatible > release". That's not correct, as even under my proposal, 2.7.7+ will still be > backwards compatible. Yeah, I took "backwards-compatibility guarantees" to also mean the "no new features" guarantee, but you're right that the two can be separated. Georg From storchaka at gmail.com Wed Mar 26 09:14:06 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 26 Mar 2014 10:14:06 +0200 Subject: [Python-Dev] C code: %s vs %U In-Reply-To: <533230C2.60302@stoneleaf.us> References: <533230C2.60302@stoneleaf.us> Message-ID: 26.03.14 03:43, Ethan Furman ???????(??): > %s is a string. > > %U is unicode? > > If so, then %s should only be used when it is certain the string in > question has no unicode in it? %s is UTF-8 encoded string. From victor.stinner at gmail.com Wed Mar 26 11:10:14 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 26 Mar 2014 11:10:14 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <53320517.3010101@stoneleaf.us> References: <53320517.3010101@stoneleaf.us> Message-ID: 2014-03-25 23:37 GMT+01:00 Ethan Furman : > ``%a`` will call ``ascii()`` on the interpolated value. I'm not sure that I understood correctly: is the "%a" format supported? The result of ascii() is a Unicode string. Does it mean that ("%a" % obj) should give the same result than ascii(obj).encode('ascii', 'strict')? Would it be possible to add a table or list to summarize supported format characters? I found: - single byte: %c - integer: %d, %u, %i, %o, %x, %X, %f, %g, "etc." (can you please complete "etc." ?) - bytes and __bytes__ method: %s - ascii(): %a I guess that the implementation of %a can avoid a conversion from ASCII ("PyUnicode_DecodeASCII" in the following code) and then a conversion to ASCII again (in bytes%args): PyObject * PyObject_ASCII(PyObject *v) { PyObject *repr, *ascii, *res; repr = PyObject_Repr(v); if (repr == NULL) return NULL; if (PyUnicode_IS_ASCII(repr)) return repr; /* repr is guaranteed to be a PyUnicode object by PyObject_Repr */ ascii = _PyUnicode_AsASCIIString(repr, "backslashreplace"); Py_DECREF(repr); if (ascii == NULL) return NULL; res = PyUnicode_DecodeASCII( <==== HERE PyBytes_AS_STRING(ascii), PyBytes_GET_SIZE(ascii), NULL); Py_DECREF(ascii); return res; } > This is intended > as a debugging aid, rather than something that should be used in production. I don't understand the purpose of this sentence. Does it mean that %a must not be used? IMO this sentence can be removed. > Non-ASCII values will be encoded to either ``\xnn`` or ``\unnnn`` > representation. Unicode is larger than that! print(ascii(chr(0x10ffff))) => '\U0010ffff' > Use cases include developing a new protocol and writing > landmarks into the stream; debugging data going into an existing protocol > to see if the problem is the protocol itself or bad data; a fall-back for a > serialization format; or even a rudimentary serialization format when > defining ``__bytes__`` would not be appropriate [8]. I understand the debug use case. I'm not convinced by the serialization idea :-) > .. note:: > > If a ``str`` is passed into ``%a``, it will be surrounded by quotes. And: - bytes gets a "b" prefix and surrounded by quotes as well (b'...') - the quote ' is escaped as \' if the string contains quotes ' and " Can you also please add examples for %a? >>> b"%a" % 123 b'123' >>> b"%s" % ascii(b"bytes") b"b'bytes'" >>> b"%s" % "text" # hum, it's not easy to see surrounding quotes with this examples b"'text'" The following more complex examples are maybe not needed: >>> b"%a" % "euro:?" b"'euro:\\u20ac'" >>> b"%a" % """quotes >'"<""" b'\'quotes >\\\'"<\'' > Proposed variations > =================== > It would be fair to mention also a whole different PEP, Antoine's PEP 460! Victor From solipsis at pitrou.net Wed Mar 26 11:48:44 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 Mar 2014 11:48:44 +0100 Subject: [Python-Dev] C code: %s vs %U References: <533230C2.60302@stoneleaf.us> Message-ID: <20140326114844.0422b5f3@fsol> On Tue, 25 Mar 2014 18:43:30 -0700 Ethan Furman wrote: > %s is a string. > > %U is unicode? What is the context? From solipsis at pitrou.net Wed Mar 26 12:02:39 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 Mar 2014 12:02:39 +0100 Subject: [Python-Dev] C code: %s vs %U References: <533230C2.60302@stoneleaf.us> <20140326114844.0422b5f3@fsol> Message-ID: <20140326120239.2a415976@fsol> On Wed, 26 Mar 2014 11:48:44 +0100 Antoine Pitrou wrote: > On Tue, 25 Mar 2014 18:43:30 -0700 > Ethan Furman wrote: > > %s is a string. > > > > %U is unicode? > > What is the context? Ok, I suppose it's PyUnicode_Format? :-) (I was initially thinking PyArg_ParseTuple, but it doesn't use "%") Regards Antoine. From victor.stinner at gmail.com Wed Mar 26 12:46:23 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 26 Mar 2014 12:46:23 +0100 Subject: [Python-Dev] C code: %s vs %U In-Reply-To: <20140326120239.2a415976@fsol> References: <533230C2.60302@stoneleaf.us> <20140326114844.0422b5f3@fsol> <20140326120239.2a415976@fsol> Message-ID: 2014-03-26 12:02 GMT+01:00 Antoine Pitrou : > Ok, I suppose it's PyUnicode_Format? :-) I don't think that PyUnicode_Format supports %U. For PyUnicode_FromFormatV(), %U expects a Python Unicode object, whereas "%s" expects a ASCII (or latin1?) encoded byte string "const char*". For PyArg_ParseTuple, "s" accepts str and bytes and encodes str to UTF-8, whereas "U" expects str. Victor From victor.stinner at gmail.com Wed Mar 26 12:55:47 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 26 Mar 2014 12:55:47 +0100 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: <20140325231947.5ec9ad63@fsol> References: <20140325231947.5ec9ad63@fsol> Message-ID: Hi, For your information, asyncio.subprocess.Process is limited. It's not possible yet to connect pipes between two processes. Something like "cat | wc -l" where the cat stdin comes from Python. It's possible to enhance the API to implement that, but the timeframe was too short to implement it before Python 3.4. Victor 2014-03-25 23:19 GMT+01:00 Antoine Pitrou : > > Hi, > > On core-mentorship someone asked about PEP 3145 - Asynchronous I/O for > subprocess.popen. I answered that asyncio now has subprocess support > (including non-blocking I/O on the three standard stream pipes), so > it's not obvious anything else is needed. > > Should we change the PEP's status to Rejected or Superseded? > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From ncoghlan at gmail.com Wed Mar 26 13:00:16 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Mar 2014 22:00:16 +1000 Subject: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7 Message-ID: Guido and Antoine persuaded me that selective backports would be a better idea for the network security enhancements than the wholesale module backports previously suggested, while Alex and Donald provided the necessary additional details, so here's a revised version of the PEP. Despite making it more explicit, I deleted more lines than I added, strongly suggesting that switching to selective backports was the right call :) I dealt with the SSL module the way Donald suggested: excluding the RAND_* functions, rather than listing everything else. I also changed the headings to make it clear the listed alternatives were rejected ideas, made the footnotes a bit more readable, and tidied up the wording in a few places. Diff: http://hg.python.org/peps/rev/8527f6e2beb0 Web: http://www.python.org/dev/peps/pep-0466/ ========================== PEP: 466 Title: Network Security Enhancement Exception for Python 2.7 Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan , Status: Draft Type: Informational Content-Type: text/x-rst Created: 23-Mar-2014 Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014 Abstract ======== Most CPython tracker issues are classified as errors in behaviour or proposed enhancements. Most patches to fix behavioural errors are applied to all active maintenance branches. Enhancement patches are restricted to the default branch that becomes the next Python version. This cadence works reasonably well during Python's normal 18-24 month feature release cycle, which is still applicable to the Python 3 series. However, the age of the standard library in Python 2 has now reached a point where it is sufficiently far behind the state of the art in network security protocols for it to be causing real problems in use cases where upgrading to Python 3 in the near term may not be feasible. In recognition of the additional practical considerations that have arisen during the 4+ year maintenance cycle for Python 2.7, this PEP allows Python 2.7 standard library components that have implications for the overall security of the internet to be updated in line with the corresponding Python 3 feature releases. Specifically, the exception allows a critical set of network security related features to be backported from Python 3.4 to the upcoming Python 2.7.7 maintenance release. While this PEP does not make any changes to the core development team's handling of security-fix-only branches that are no longer in active maintenance, it *does* recommend that commercial redistributors providing extended support periods for the Python standard library either backport these features to their supported versions, or else explicitly disclaim support for the use of older versions in roles that involve connecting directly to the public internet. Exemption Policy ================ Under this policy, the following features SHOULD be backported from Python 3.4 to the upcoming Python 2.7.7 maintenance release: * in the ``os`` module: * persistent file descriptor for ``os.urandom()``. * in the ``hmac`` module: * constant time comparison function (``hmac.compare_digest()``). * in the ``hashlib`` module: * password hashing function (``hashlib.pbkdf2_hmac()``). * details of hash algorithm availability (``hashlib.algorithms_guaranteed`` and ``hashlib.algorithms_guaranteed``). * in the ``ssl`` module: * this module is almost entirely synchronised with its Python 3 counterpart, bringing TLSv2, SSLContext manipulation, Server Name Identification, access to platform certificate stores, standard library support for peer hostname validation and more to the Python 2 series. * the only ``ssl`` module features *not* backported under this policy are the ``ssl.RAND_*`` functions that provide access to OpenSSL's random number generation capabilities - use ``os.urandom()`` instead. As part of this policy, permission is also granted to upgrade to newer feature releases of OpenSSL when preparing the binary installers for new maintenance releases of Python 2.7. Backwards Compatibility Considerations ====================================== As in the Python 3 series, the backported ``ssl.create_default_context()`` API is granted a backwards compatibility exemption that permits the protocol, options, cipher and other settings of the created SSL context to be made This PEP does *not* grant any exemptions to the usual backwards compatibility policy for maintenance releases. Instead, by explicitly encouraging the use of feature based checks, it is designed to make it easier to write more secure cross-version compatible Python software, while still limiting the risk of breaking currently working software when upgrading to a new Python 2.7 maintenance release. In all cases where this policy allows new features to be backported to the Python 2.7 release series, it is possible to write cross-version compatible code that operates by "feature detection" (for example, checking for particular attributes in a module), without needing to explicitly check the Python version. It is then up to library and framework code to provide an appropriate warning and fallback behaviour if a desired feature is found to be missing. While some especially security sensitive software MAY fail outright if a desired security feature is unavailable, most software SHOULD instead emit a warning and continue operating using a slightly degraded security configuration. The backported APIs allow library and application code to perform the following actions after detecting the presence of a relevant network security related feature: * explicitly opt in to more secure settings (to allow the use of enhanced security features in older maintenance releases of Python with less secure default behaviour) * explicitly opt in to less secure settings (to allow the use of newer Python feature releases in lower security environments) * determine the default setting for the feature (this MAY require explicit Python version checks to determine the Python feature release, but DOES NOT require checking for a specific maintenance release) Security related changes to other modules (such as higher level networking libraries and data format processing libraries) will continue to be made available as backports and new modules on the Python Package Index, as independent distribution remains the preferred approach to handling software that must continue to evolve to handle changing development requirements independently of the Python 2 standard library. Refer to the `Motivation and Rationale`_ section for a review of the characteristics that make the secure networking infrastructure worthy of special consideration. OpenSSL compatibility --------------------- Under this policy, OpenSSL may be upgraded to more recent feature releases in Python 2.7 maintenance releases. On Linux and most other POSIX systems, the specific version of OpenSSL used already varies, as CPython dynamically links to the system provided OpenSSL library by default. For the Windows binary installers, the ``_ssl`` and ``_hashlib`` modules are statically linked with OpenSSL and the associated symbols are not exported. Marc-Andre Lemburg indicates that updating to newer OpenSSL releases in the ``egenix-pyopenssl`` binaries has not resulted in any reported compatibility issues [3]_ The Mac OS X binary installers historically followed the same policy as other POSIX installations and dynamically linked to the Apple provided OpenSSL libraries. However, Apple has now ceased updating these cross-platform libraries, instead requiring that even cross-platform developers adopt Mac OS X specific interfaces to access up to date security infrastructure on their platform. Accordingly, and independently of this PEP, the Mac OS X binary installers were already going to be switched to statically linker newer versions of OpenSSL [4]_ Other Considerations ==================== Maintainability --------------- This policy does NOT represent a commitment by volunteer contributors to actually backport network security related changes from the Python 3 series to the Python 2 series. Rather, it is intended to send a clear signal to potential corporate contributors that the core development team are willing to accept offers of corporate assistance in putting this policy into effect and handling the resulting increase in the Python 2 maintenance load. Backporting security related fixes and enhancements to earlier versions is a common service for commercial redistributors to offer to their customers. This policy represents an explicit invitation to implement those changes in the core development tree in cases where they are likely to have a broad impact that helps improve the security of the internet as a whole, with the assurance that the existing core development team not only won't object to such contributions, but will actively encourage their incorporation into the next Python 2.7 maintenance release. Documentation ------------- All modules covered by this policy MUST include a "Security Considerations" section in their documentation in order to take advantage of this policy. In addition to any other module specific contents, this section SHOULD enumerate key security enhancements and fixes (with CVE identifiers where applicable), along with the feature and maintenance releases that first included them. Security releases ----------------- This PEP does not propose any changes to the handling of security releases - those will continue to be source only releases that include only critical security fixes. However, the recommendations for library and application developers are deliberately designed to accommodate commercial redistributors that choose to apply this policy to additional Python release series that are either in security fix only mode, or have been declared "end of life" by the core development team. Whether or not redistributors choose to exercise that option will be up to the individual redistributor. Integration testing ------------------- Third party integration testing services should offer users the ability to test against specific Python 2.7 maintenance releases, to ensure that libraries, frameworks and applications can still test their handling of the legacy security infrastructure correctly (either failing or degrading gracefully, depending on the security sensitivity of the software), even after the features covered in this policy have been backported to the Python 2.7 series. Handling lower security environments with low risk tolerance ------------------------------------------------------------ For better or for worse (mostly worse), there are some environments where the risk of latent security defects is more tolerated than even a slightly increased risk of regressions in maintenance releases. This policy largely excludes these environments from consideration where the modules covered by the exemption are concerned - this approach is entirely inappropriate for software connected to the public internet, and defence in depth security principles suggest that it is not appropriate for most private networks either. Downstream redistributors may still choose to cater to such environments, but they will need to handle the process of downgrading the security related modules and doing the associated regression testing themselves. The main CPython continuous integration infrastructure will not cover this scenario. Evolution of this Policy ======================== The key requirement for a feature to be considered for inclusion in this policy is that it must have security implications *beyond* the specific application that is written in Python and the system that application is running on. Thus the focus on network security protocols, password storage and related cryptographic infrastructure - Python is a popular choice for the development of web services and clients, and thus the capabilities of widely used Python versions have implications for the security design of other services that may themselves be using newer versions of Python or other development languages, but need to interoperate with clients or servers written using older versions of Python. The intent behind this requirement is to minimise any impact that the introduction of this policy may have on the stability and compatibility of maintenance releases. It would be thoroughly counterproductive if end users became as cautious about updating to new Python 2.7 maintenance releases as they are about updating to new feature releases within the same release series. Motivation and Rationale ======================== The creation of this PEP was prompted primarily by the aging SSL support in the Python 2 series. As of March 2014, the Python 2.7 SSL module is approaching four years of age, and the SSL support in the still popular Python 2.6 release had its feature set locked six years ago. These are simply too old to provide a foundation that can be recommended in good conscience for secure networking software that operates over the public internet, especially in an era where it is becoming quite clearly evident that advanced persistent security threats are even more widespread and more indiscriminate in their targeting than had previously been understood. While they represented reasonable security infrastructure in their time, the state of the art has moved on, and we need to investigate mechanisms for effectively providing more up to date network security infrastructure for users that, for whatever reason, are not currently in a position to migrate to Python 3. While the use of the system OpenSSL installation addresses many of these concerns on Linux platforms, it doesn't address all of them (in particular, it is still difficult for sotware to explicitly require some higher level security settings). In the case of the binary installers for Windows and Mac OS X that are published on python.org, the version of OpenSSL used is entirely within the control of the Python core development team, but is currently limited to OpenSSL maintenance releases for the version initially shipped with the corresponding Python feature release. With increased popularity comes increased responsibility, and this policy aims to acknowledge the fact that Python's popularity and adoption is at a sufficiently high level that some of our design and policy decisions have significant implications beyond the Python development community. As one example, the Python 2 ``ssl`` module does not support the Server Name Identification standard. While it is possible to obtain SNI support by using the third party ``requests`` client library, actually doing so currently requires using not only ``requests`` and its embedded dependencies, but also half a dozen or more additional libraries. The lack of support in the Python 2 series thus serves as an impediment to making effective use of SNI on servers, as Python 2 clients will frequently fail to handle it correctly. Another more critical example is the lack of SSL hostname matching in the Python 2 standard library - it is currently necessary to rely on a third party library, such as ``requests`` or ``backports.ssl_match_hostname`` to obtain that functionality in Python 2. The Python 2 series also remains more vulnerable to remote timing attacks on security sensitive comparisons than the Python 3 series, as it lacks a standard library equivalent to the timing attack resistant ``hmac.compare_digest()`` function. While appropriate secure comparison functions can be implemented in third party extensions, many users don't even consider the issue and use ordinary equality comparisons instead - while a standard library solution doesn't automatically fix that problem, it *does* make the barrier to resolution much lower once the problem is pointed out. My position on the ongoing transition from Python 2 to Python 3 has long been that Python 2 remains a supported platform for the core development team, and that commercial support will remain available well after upstream maintenance ends. However, in the absence of this network security enhancement policy, that position is difficult to justify when it comes to software that operates over the public internet. Just as many developers consider it too difficult to develop truly secure modern networked software in C/C++ (largely due to the challenges associated with manual memory management), I anticipate that in the not too distant future, it will be considered too difficult to develop truly secure modern networked software using the Python 2 series (some developers would argue that we have already reached that point). Python 2.7 represents the only long term maintenance release the core development team has provided, and it is natural that there will be things that worked over a historically shorter maintenance lifespan that don't work over this longer support period. In the specific case of the problem described in this PEP, the simplest available solution is to acknowledge that long term maintenance of network security related modules *requires* the ability to add new features, even while retaining backwards compatibility for existing interfaces. It is worth comparing the approach described in this PEP with Red Hat's handling of its long term support commitments: it isn't the RHEL 6.0 release itself that receives 10 years worth of support, but the overall RHEL 6 *series*. The individual RHEL 6.x point releases within the series then receive a wide variety of new features, including security enhancements, all while meeting strict backwards compatibility guarantees for existing software. The policy described in this PEP brings our approach to long term maintenance more into line with this precedent - we retain our strict backwards compatibility requirements, but slightly relax the restrictions against adding new features. To date, downstream redistributors have respected our upstream policy of "no new features in Python maintenance releases". This PEP explicitly accepts that a more nuanced policy is appropriate in the case of network security related features, and the specific one it describes is deliberately designed such that it at least has some chance of being applied to Red Hat Enterprise Linux and its downstream derivatives. Rejected alternative: just advise developers to migrate to Python 3 ------------------------------------------------------------------- This alternative represents the status quo. Unfortunately, it has proven to be unworkable in practice, as the backwards compatibility implications mean that this is a non-trivial migration process for large applications and integration projects. While the tools for migration have evolved to a point where it is possible to migrate even large applications opportunistically and incrementally (rather than all at once) by updating code to run in the large common subset of Python 2 and Python 3, using the most recent technology often isn't a priority in commercial environments. Previously, this was considered an acceptable harm, as while it was an unfortunate problem for the affected developers to have to face, it was seen as an issue between them and their management chain to make the case for infrastructure modernisation, and this case would become naturally more compelling as the Python 3 series evolved. However, now that we're fully aware of the impact the limitations of the Python 2 standard library may be having on the evolution of internet security standards, I no longer believe that it is reasonable to expect platform and application developers to resolve all of the latent defects in an application's Unicode correctness solely in order to gain access to the network security enhancements already available in Python 3. While Ubuntu (and to some extent Debian as well) are committed to porting all default system services and scripts to Python 3, and to removing Python 2 from its default distribution images (but not from its archives), this is a mammoth task and won't be completed for the Ubuntu 14.04 LTS release (at least for the desktop image - it may be achieved for the mobile and server images). Fedora has even more work to do to migrate, and it will take a non-trivial amount of time to migrate the relevant infrastructure components. While Red Hat are also actively working to make it easier for users to use more recent versions of Python on our stable platforms, it's going to take time for those efforts to start having an impact on end users' choice of version, and any such changes also don't benefit the core platform infrastructure that runs in the integrated system Python by necessity. The OpenStack migration to Python 3 is also still in its infancy, and even though that's a project with an extensive and relatively robust automated test suite, it's still large enough that it is going to take quite some time to migrate fully to a Python 2/3 compatible code base. And that's just three of the highest profile open source projects that make heavy use of Python. Given the likely existence of large amounts of legacy code that lacks the kind of automated regression test suite needed to help support a migration from Python 2 to Python 3, there are likely to be many cases where reimplementation (perhaps even in Python 3) proves easier than migration. The key point of this PEP is that those situations affect more people than just the developers and users of the affected application: the existence of clients and servers with outdated network security infrastructure becomes something that developers of secure networked services need to take into account as part of their security design, and that's a problem that inhibits the adoption of better security standards. As Terry Reedy noted, if we try to persist with the status quo, the likely outcome is that commercial redistributors will attempt to do something like this on behalf of their customers *anyway*, but in a potentially inconsistent and ad hoc manner. By drawing the scope definition process into the upstream project we are in a better position to influence the approach taken to address the situation and to help ensure some consistency across redistributors. The problem is real, so *something* needs to change, and this PEP describes my preferred approach to addressing the situation. Rejected alternative: create and release Python 2.8 --------------------------------------------------- With sufficient corporate support, it likely *would* be possible to create and release Python 2.8 (it's highly unlikely such a project would garner enough interest to be achievable with only volunteers). However, this wouldn't actually solve the problem, as the aim is to provide a *relatively low impact* way to incorporate enhanced security features into integrated products and deployments that make use of Python 2. Upgrading to a new Python feature release would mean both more work for the core development team, as well as a more disruptive update that most potential end users would likely just skip entirely. Attempting to create a Python 2.8 release would also bring in suggestions to backport many additional features from Python 3 (such as ``tracemalloc`` and the improved coroutine support), making the migration from Python 2.7 to this hypothetical 2.8 release even riskier and more disruptive. This is not a recommended approach, as it would involve substantial additional work for a result that is actually less effective in achieving the original aim (which is to eliminate the current widespread use of the aging network security infrastructure in the Python 2 series). Furthermore, while I can't make any commitments to actually addressing this issue on Red Hat platforms, I *can* categorically rule out the idea of a Python 2.8 being of any use to me in even attempting to get it addressed. Rejected alternative: distribute the security enhancements via PyPI ------------------------------------------------------------------- While this initially appears to be an attractive and easier to manage approach, it actually suffers from several significant problems. Firstly, this is complex, low level, cross-platform code that integrates with the underlying operating system across a variety of POSIX platforms (including Mac OS X) and Windows. The CPython BuildBot fleet is already set up to handle continuous integration in that context, but most of the freely available continuous integration services just offer Linux, and perhaps paid access to Windows. Those services work reasonably well for software that largely runs on the abstraction layers offered by Python and other dynamic languages, as well as the more comprehensive abstraction offered by the JVM, but won't suffice for the kind of code involved here. The OpenSSL dependency for the network security support also qualifies as the kind of "complex binary dependency" that isn't yet handled well by the ``pip`` based software distribution ecosystem. Relying on a third party binary dependency also creates potential compatibility problems for ``pip`` when running on other interpreters like ``PyPy``. Another practical problem with the idea is the fact that ``pip`` itself relies on the ``ssl`` support in the standard library (with some additional support from a bundled copy of ``requests``, which in turn bundles ``backport.ssl_match_hostname``), and hence would require any replacement module to also be bundled within ``pip``. This wouldn't pose any insurmountable difficulties (it's just another dependency to vendor), but it *would* mean yet another copy of OpenSSL to keep up to date. This approach also has the same flaw as all other "improve security by renaming things" approaches: they completely miss the users who most need help, and raise significant barriers against being able to encourage users to do the right thing when their infrastructure supports it (since "use this other module" is a much higher impact change than "turn on this higher security setting"). Deprecating the aging SSL infrastructure in the standard library in favour of an external module would be even more user hostile than accepting the slightly increased risk of regressions associated with upgrading it in place. Last, but certainly not least, this approach suffers from the same problem as the idea of doing a Python 2.8 release: likely not solving the actual problem. Commercial redistributors of Python are set up to redistribute *Python*, and a pre-existing set of additional packages. Getting new packages added to the pre-existing set *can* be done, but means approaching each and every redistributor and asking them to update their repackaging process accordingly. By contrast, the approach described in this PEP would require redistributors to deliberately *opt out* of the security enhancements by deliberately downgrading the provided network security infrastructure, which most of them are unlikely to do. Rejected variant: provide a "legacy SSL infrastructure" branch -------------------------------------------------------------- Earlier versions of this PEP included the concept of a ``2.7-legacy-ssl`` branch that preserved the exact feature set of the Python 2.7.6 network security infrastructure. It is the opinion of the PEP author that anyone that actually wants this is almost certainly making a mistake, and if they insist they really do want it in their specific situation, they're welcome to either make it themselves or arrange for a downstream redistributor to make it for them. If they are made publicly available, any such rebuilds should be referred to as "Python 2.7 with Legacy SSL" to clearly distinguish them from the official Python 2.7 releases that include more up to date network security infrastructure. After the first Python 2.7 maintenance release that has the security infrastructure updated to match Python 3.4, it would also be appropriate to refer to Python 2.7.6 and earlier releases as "Python 2.7 with Legacy SSL". Rejected variant: synchronise particular modules entirely with Python 3 ----------------------------------------------------------------------- Earlier versions of this PEP suggested synchronising the ``hmac``, ``hashlib`` and ``ssl`` modules entirely with their Python 3 counterparts. This approach proved too vague to build a compelling case for the exception, and has thus been replaced by the current more explicit proposal. Open Questions ============== * MvL has indicated he is not prepared to tackle the task of trying to integrate a newer OpenSSL into the also aging Python 2.7 build infrastructure on Windows (unfortunately, we've looked into upgrading that build infrastructure, and the backwards compatibility issues appear to be effectively insurmountable). We would require a commitment from another trusted contributor to handle at least this task, and potentially also taking over the task of creating the official Python 2.7 Windows installers for the remaining Python 2.7 maintenance releases. * We would need commitments to create and review full backports of the components covered by this policy from Python 3.4 to Python 2.7, as well as support for handling any more specific security issues affecting these modules. * Did I miss anything important in the switch to a more restrictive proposal? Disclosure of Interest ====================== The author of this PEP currently works for Red Hat on test automation tools. If this proposal is accepted, I will be strongly encouraging Red Hat to take advantage of the resulting opportunity to help improve the overall security of the Python ecosystem. However, I do not speak for Red Hat in this matter, and cannot make any commitments on Red Hat's behalf. Acknowledgements ================ Thanks to Christian Heimes and other for their efforts in greatly improving Python's SSL support in the Python 3 series, and a variety of members of the Python community for helping me to better understand the implications of the default settings we provide in our SSL modules, and the impact that tolerating the use of SSL infrastructure that was defined in 2010 (Python 2.7) or even 2008 (Python 2.6) potentially has for the security of the web as a whole. Thanks to Donald Stufft and Alex Gaynor for identifying a more limited set of essential security features that allowed the proposal to be made more fine-grained than backporting entire modules from Python 3.4 [7,8]_. Christian and Donald also provided valuable feedback on a preliminary draft of this proposal. Thanks also to participants in the python-dev mailing list threads [1,2,5,6]_ References ========== .. [1] PEP 466 discussion (round 1) (https://mail.python.org/pipermail/python-dev/2014-March/133334.html) .. [2] PEP 466 discussion (round 2) (https://mail.python.org/pipermail/python-dev/2014-March/133389.html) .. [3] Marc-Andre Lemburg's OpenSSL feedback for Windows (https://mail.python.org/pipermail/python-dev/2014-March/133438.html) .. [4] Ned Deily's OpenSSL feedback for Mac OS X (https://mail.python.org/pipermail/python-dev/2014-March/133347.html) .. [5] PEP 466 discussion (round 3) (https://mail.python.org/pipermail/python-dev/2014-March/133442.html) .. [6] PEP 466 discussion (round 4) (https://mail.python.org/pipermail/python-dev/2014-March/133472.html) .. [7] Donald Stufft's recommended set of backported features (https://mail.python.org/pipermail/python-dev/2014-March/133500.html) .. [8] Alex Gaynor's recommended set of backported features (https://mail.python.org/pipermail/python-dev/2014-March/133503.html) Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed Mar 26 13:05:06 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 26 Mar 2014 08:05:06 -0400 Subject: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7 In-Reply-To: References: Message-ID: Typo I think: As in the Python 3 series, the backported ssl.create_default_context() API is granted a backwards compatibility exemption that permits the protocol, options, cipher and other settings of the created SSL context to be made made what? On Mar 26, 2014, at 8:00 AM, Nick Coghlan wrote: > Guido and Antoine persuaded me that selective backports would be a > better idea for the network security enhancements than the wholesale > module backports previously suggested, while Alex and Donald provided > the necessary additional details, so here's a revised version of the > PEP. Despite making it more explicit, I deleted more lines than I > added, strongly suggesting that switching to selective backports was > the right call :) > > I dealt with the SSL module the way Donald suggested: excluding the > RAND_* functions, rather than listing everything else. > > I also changed the headings to make it clear the listed alternatives > were rejected ideas, made the footnotes a bit more readable, and > tidied up the wording in a few places. > > Diff: http://hg.python.org/peps/rev/8527f6e2beb0 > Web: http://www.python.org/dev/peps/pep-0466/ > > ========================== > PEP: 466 > Title: Network Security Enhancement Exception for Python 2.7 > Version: $Revision$ > Last-Modified: $Date$ > Author: Nick Coghlan , > Status: Draft > Type: Informational > Content-Type: text/x-rst > Created: 23-Mar-2014 > Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014 > > > Abstract > ======== > > Most CPython tracker issues are classified as errors in behaviour or > proposed enhancements. Most patches to fix behavioural errors are > applied to all active maintenance branches. Enhancement patches are > restricted to the default branch that becomes the next Python version. > > This cadence works reasonably well during Python's normal 18-24 month > feature release cycle, which is still applicable to the Python 3 series. > However, the age of the standard library in Python 2 has now reached a point > where it is sufficiently far behind the state of the art in network security > protocols for it to be causing real problems in use cases where upgrading to > Python 3 in the near term may not be feasible. > > In recognition of the additional practical considerations that have arisen > during the 4+ year maintenance cycle for Python 2.7, this PEP allows > Python 2.7 standard library components that have implications for the > overall security of the internet to be updated in line with the > corresponding Python 3 feature releases. > > Specifically, the exception allows a critical set of network security > related features to be backported from Python 3.4 to the upcoming Python > 2.7.7 maintenance release. > > While this PEP does not make any changes to the core development team's > handling of security-fix-only branches that are no longer in active > maintenance, it *does* recommend that commercial redistributors providing > extended support periods for the Python standard library either backport > these features to their supported versions, or else explicitly disclaim > support for the use of older versions in roles that involve connecting > directly to the public internet. > > > Exemption Policy > ================ > > Under this policy, the following features SHOULD be backported from Python > 3.4 to the upcoming Python 2.7.7 maintenance release: > > * in the ``os`` module: > > * persistent file descriptor for ``os.urandom()``. > > * in the ``hmac`` module: > > * constant time comparison function (``hmac.compare_digest()``). > > * in the ``hashlib`` module: > > * password hashing function (``hashlib.pbkdf2_hmac()``). > * details of hash algorithm availability (``hashlib.algorithms_guaranteed`` > and ``hashlib.algorithms_guaranteed``). > > * in the ``ssl`` module: > > * this module is almost entirely synchronised with its Python 3 > counterpart, bringing TLSv2, SSLContext manipulation, Server Name > Identification, access to platform certificate stores, standard > library support for peer hostname validation and more to the Python 2 > series. > * the only ``ssl`` module features *not* backported under this policy are > the ``ssl.RAND_*`` functions that provide access to OpenSSL's random > number generation capabilities - use ``os.urandom()`` instead. > > As part of this policy, permission is also granted to upgrade to newer > feature releases of OpenSSL when preparing the binary installers > for new maintenance releases of Python 2.7. > > > Backwards Compatibility Considerations > ====================================== > > As in the Python 3 series, the backported ``ssl.create_default_context()`` > API is granted a backwards compatibility exemption that permits the > protocol, options, cipher and other settings of the created SSL context to > be made > > This PEP does *not* grant any exemptions to the usual backwards > compatibility policy for maintenance releases. Instead, by explicitly > encouraging the use of feature based checks, it is designed to make it easier > to write more secure cross-version compatible Python software, while still > limiting the risk of breaking currently working software when upgrading to > a new Python 2.7 maintenance release. > > In all cases where this policy allows new features to be backported to > the Python 2.7 release series, it is possible to write cross-version > compatible code that operates by "feature detection" (for example, checking > for particular attributes in a module), without needing to explicitly check > the Python version. > > It is then up to library and framework code to provide an appropriate warning > and fallback behaviour if a desired feature is found to be missing. While > some especially security sensitive software MAY fail outright if a desired > security feature is unavailable, most software SHOULD instead emit a warning > and continue operating using a slightly degraded security configuration. > > The backported APIs allow library and application code to perform the > following actions after detecting the presence of a relevant > network security related feature: > > * explicitly opt in to more secure settings (to allow the use of enhanced > security features in older maintenance releases of Python with less > secure default behaviour) > * explicitly opt in to less secure settings (to allow the use of newer Python > feature releases in lower security environments) > * determine the default setting for the feature (this MAY require explicit > Python version checks to determine the Python feature release, but DOES > NOT require checking for a specific maintenance release) > > Security related changes to other modules (such as higher level networking > libraries and data format processing libraries) will continue to be made > available as backports and new modules on the Python Package Index, as > independent distribution remains the preferred approach to handling > software that must continue to evolve to handle changing development > requirements independently of the Python 2 standard library. Refer to > the `Motivation and Rationale`_ section for a review of the characteristics > that make the secure networking infrastructure worthy of special > consideration. > > > OpenSSL compatibility > --------------------- > > Under this policy, OpenSSL may be upgraded to more recent feature releases > in Python 2.7 maintenance releases. On Linux and most other POSIX systems, > the specific version of OpenSSL used already varies, as CPython dynamically > links to the system provided OpenSSL library by default. > > For the Windows binary installers, the ``_ssl`` and ``_hashlib`` modules are > statically linked with OpenSSL and the associated symbols are not exported. > Marc-Andre Lemburg indicates that updating to newer OpenSSL releases in the > ``egenix-pyopenssl`` binaries has not resulted in any reported compatibility > issues [3]_ > > The Mac OS X binary installers historically followed the same policy as > other POSIX installations and dynamically linked to the Apple provided > OpenSSL libraries. However, Apple has now ceased updating these > cross-platform libraries, instead requiring that even cross-platform > developers adopt Mac OS X specific interfaces to access up to date security > infrastructure on their platform. Accordingly, and independently of this > PEP, the Mac OS X binary installers were already going to be switched to > statically linker newer versions of OpenSSL [4]_ > > > Other Considerations > ==================== > > Maintainability > --------------- > > This policy does NOT represent a commitment by volunteer contributors to > actually backport network security related changes from the Python 3 series > to the Python 2 series. Rather, it is intended to send a clear signal to > potential corporate contributors that the core development team are willing > to accept offers of corporate assistance in putting this policy into > effect and handling the resulting increase in the Python 2 maintenance > load. > > Backporting security related fixes and enhancements to earlier versions is > a common service for commercial redistributors to offer to their customers. > This policy represents an explicit invitation to implement those changes > in the core development tree in cases where they are likely to have a broad > impact that helps improve the security of the internet as a whole, with the > assurance that the existing core development team not only won't object to > such contributions, but will actively encourage their incorporation into > the next Python 2.7 maintenance release. > > > Documentation > ------------- > > All modules covered by this policy MUST include a "Security Considerations" > section in their documentation in order to take advantage of this policy. > > In addition to any other module specific contents, this section SHOULD > enumerate key security enhancements and fixes (with CVE identifiers where > applicable), along with the feature and maintenance releases that first > included them. > > > Security releases > ----------------- > > This PEP does not propose any changes to the handling of security > releases - those will continue to be source only releases that > include only critical security fixes. > > However, the recommendations for library and application developers are > deliberately designed to accommodate commercial redistributors that choose > to apply this policy to additional Python release series that are either in > security fix only mode, or have been declared "end of life" by the core > development team. > > Whether or not redistributors choose to exercise that option will be up > to the individual redistributor. > > > Integration testing > ------------------- > > Third party integration testing services should offer users the ability > to test against specific Python 2.7 maintenance releases, to ensure that > libraries, frameworks and applications can still test their handling of the > legacy security infrastructure correctly (either failing or degrading > gracefully, depending on the security sensitivity of the software), even > after the features covered in this policy have been backported to the > Python 2.7 series. > > > Handling lower security environments with low risk tolerance > ------------------------------------------------------------ > > For better or for worse (mostly worse), there are some environments where > the risk of latent security defects is more tolerated than even a slightly > increased risk of regressions in maintenance releases. This policy largely > excludes these environments from consideration where the modules covered by > the exemption are concerned - this approach is entirely inappropriate for > software connected to the public internet, and defence in depth security > principles suggest that it is not appropriate for most private networks > either. > > Downstream redistributors may still choose to cater to such environments, > but they will need to handle the process of downgrading the security > related modules and doing the associated regression testing themselves. > The main CPython continuous integration infrastructure will not cover this > scenario. > > > Evolution of this Policy > ======================== > > The key requirement for a feature to be considered for inclusion in this > policy is that it must have security implications *beyond* the specific > application that is written in Python and the system that application is > running on. Thus the focus on network security protocols, password storage > and related cryptographic infrastructure - Python is a popular choice for > the development of web services and clients, and thus the capabilities of > widely used Python versions have implications for the security design of > other services that may themselves be using newer versions of Python or > other development languages, but need to interoperate with clients or > servers written using older versions of Python. > > The intent behind this requirement is to minimise any impact that the > introduction of this policy may have on the stability and compatibility of > maintenance releases. It would be thoroughly counterproductive if end > users became as cautious about updating to new Python 2.7 maintenance > releases as they are about updating to new feature releases within the > same release series. > > > Motivation and Rationale > ======================== > > The creation of this PEP was prompted primarily by the aging SSL support in > the Python 2 series. As of March 2014, the Python 2.7 SSL module is > approaching four years of age, and the SSL support in the still popular > Python 2.6 release had its feature set locked six years ago. > > These are simply too old to provide a foundation that can be recommended > in good conscience for secure networking software that operates over the > public internet, especially in an era where it is becoming quite clearly > evident that advanced persistent security threats are even more widespread > and more indiscriminate in their targeting than had previously been > understood. While they represented reasonable security infrastructure in > their time, the state of the art has moved on, and we need to investigate > mechanisms for effectively providing more up to date network security > infrastructure for users that, for whatever reason, are not currently in > a position to migrate to Python 3. > > While the use of the system OpenSSL installation addresses many of these > concerns on Linux platforms, it doesn't address all of them (in particular, > it is still difficult for sotware to explicitly require some higher level > security settings). In the case of the binary installers for Windows and > Mac OS X that are published on python.org, the version of OpenSSL used is > entirely within the control of the Python core development team, but is > currently limited to OpenSSL maintenance releases for the version initially > shipped with the corresponding Python feature release. > > With increased popularity comes increased responsibility, and this policy > aims to acknowledge the fact that Python's popularity and adoption is at a > sufficiently high level that some of our design and policy decisions have > significant implications beyond the Python development community. > > As one example, the Python 2 ``ssl`` module does not support the Server > Name Identification standard. While it is possible to obtain SNI support > by using the third party ``requests`` client library, actually doing so > currently requires using not only ``requests`` and its embedded dependencies, > but also half a dozen or more additional libraries. The lack of support > in the Python 2 series thus serves as an impediment to making effective > use of SNI on servers, as Python 2 clients will frequently fail to handle > it correctly. > > Another more critical example is the lack of SSL hostname matching in the > Python 2 standard library - it is currently necessary to rely on a third > party library, such as ``requests`` or ``backports.ssl_match_hostname`` to > obtain that functionality in Python 2. > > The Python 2 series also remains more vulnerable to remote timing attacks > on security sensitive comparisons than the Python 3 series, as it lacks a > standard library equivalent to the timing attack resistant > ``hmac.compare_digest()`` function. While appropriate secure comparison > functions can be implemented in third party extensions, many users don't > even consider the issue and use ordinary equality comparisons instead > - while a standard library solution doesn't automatically fix that problem, > it *does* make the barrier to resolution much lower once the problem is > pointed out. > > My position on the ongoing transition from Python 2 to Python 3 has long > been that Python 2 remains a supported platform for the core development > team, and that commercial support will remain available well after upstream > maintenance ends. However, in the absence of this network security > enhancement policy, that position is difficult to justify when it comes to > software that operates over the public internet. Just as many developers > consider it too difficult to develop truly secure modern networked software > in C/C++ (largely due to the challenges associated with manual > memory management), I anticipate that in the not too distant future, it > will be considered too difficult to develop truly secure modern networked > software using the Python 2 series (some developers would argue that we > have already reached that point). > > Python 2.7 represents the only long term maintenance release the core > development team has provided, and it is natural that there will be things > that worked over a historically shorter maintenance lifespan that don't work > over this longer support period. In the specific case of the problem > described in this PEP, the simplest available solution is to acknowledge > that long term maintenance of network security related modules *requires* > the ability to add new features, even while retaining backwards compatibility > for existing interfaces. > > It is worth comparing the approach described in this PEP with Red Hat's > handling of its long term support commitments: it isn't the RHEL 6.0 release > itself that receives 10 years worth of support, but the overall RHEL 6 > *series*. The individual RHEL 6.x point releases within the series then > receive a wide variety of new features, including security enhancements, > all while meeting strict backwards compatibility guarantees for existing > software. The policy described in this PEP brings our approach to long term > maintenance more into line with this precedent - we retain our strict > backwards compatibility requirements, but slightly relax the restrictions > against adding new features. > > To date, downstream redistributors have respected our upstream policy of > "no new features in Python maintenance releases". This PEP explicitly > accepts that a more nuanced policy is appropriate in the case of network > security related features, and the specific one it describes is deliberately > designed such that it at least has some chance of being applied to Red Hat > Enterprise Linux and its downstream derivatives. > > > Rejected alternative: just advise developers to migrate to Python 3 > ------------------------------------------------------------------- > > This alternative represents the status quo. Unfortunately, it has proven > to be unworkable in practice, as the backwards compatibility implications > mean that this is a non-trivial migration process for large applications > and integration projects. While the tools for migration have evolved to > a point where it is possible to migrate even large applications > opportunistically and incrementally (rather than all at once) by updating > code to run in the large common subset of Python 2 and Python 3, using the > most recent technology often isn't a priority in commercial environments. > > Previously, this was considered an acceptable harm, as while it was an > unfortunate problem for the affected developers to have to face, it was > seen as an issue between them and their management chain to make the case > for infrastructure modernisation, and this case would become naturally > more compelling as the Python 3 series evolved. > > However, now that we're fully aware of the impact the limitations of the > Python 2 standard library may be having on the evolution of internet > security standards, I no longer believe that it is reasonable to expect > platform and application developers to resolve all of the latent defects > in an application's Unicode correctness solely in order to gain access to > the network security enhancements already available in Python 3. > > While Ubuntu (and to some extent Debian as well) are committed to porting all > default system services and scripts to Python 3, and to removing Python 2 > from its default distribution images (but not from its archives), this is > a mammoth task and won't be completed for the Ubuntu 14.04 LTS release > (at least for the desktop image - it may be achieved for the mobile and > server images). > > Fedora has even more work to do to migrate, and it will take a non-trivial > amount of time to migrate the relevant infrastructure components. While > Red Hat are also actively working to make it easier for users to use more > recent versions of Python on our stable platforms, it's going to take time > for those efforts to start having an impact on end users' choice of version, > and any such changes also don't benefit the core platform infrastructure > that runs in the integrated system Python by necessity. > > The OpenStack migration to Python 3 is also still in its infancy, and even > though that's a project with an extensive and relatively robust automated > test suite, it's still large enough that it is going to take quite some time > to migrate fully to a Python 2/3 compatible code base. > > And that's just three of the highest profile open source projects that > make heavy use of Python. Given the likely existence of large amounts of > legacy code that lacks the kind of automated regression test suite needed > to help support a migration from Python 2 to Python 3, there are likely to > be many cases where reimplementation (perhaps even in Python 3) proves > easier than migration. The key point of this PEP is that those situations > affect more people than just the developers and users of the affected > application: the existence of clients and servers with outdated network > security infrastructure becomes something that developers of secure > networked services need to take into account as part of their security > design, and that's a problem that inhibits the adoption of better security > standards. > > As Terry Reedy noted, if we try to persist with the status quo, the likely > outcome is that commercial redistributors will attempt to do something > like this on behalf of their customers *anyway*, but in a potentially > inconsistent and ad hoc manner. By drawing the scope definition process > into the upstream project we are in a better position to influence the > approach taken to address the situation and to help ensure some consistency > across redistributors. > > The problem is real, so *something* needs to change, and this PEP describes > my preferred approach to addressing the situation. > > > Rejected alternative: create and release Python 2.8 > --------------------------------------------------- > > With sufficient corporate support, it likely *would* be possible to create > and release Python 2.8 (it's highly unlikely such a project would garner > enough interest to be achievable with only volunteers). However, this > wouldn't actually solve the problem, as the aim is to provide a *relatively > low impact* way to incorporate enhanced security features into integrated > products and deployments that make use of Python 2. > > Upgrading to a new Python feature release would mean both more work for the > core development team, as well as a more disruptive update that most > potential end users would likely just skip entirely. > > Attempting to create a Python 2.8 release would also bring in suggestions > to backport many additional features from Python 3 (such as ``tracemalloc`` > and the improved coroutine support), making the migration from Python 2.7 > to this hypothetical 2.8 release even riskier and more disruptive. > > This is not a recommended approach, as it would involve substantial > additional work for a result that is actually less effective in achieving > the original aim (which is to eliminate the current widespread use of the > aging network security infrastructure in the Python 2 series). > > Furthermore, while I can't make any commitments to actually addressing > this issue on Red Hat platforms, I *can* categorically rule out the idea > of a Python 2.8 being of any use to me in even attempting to get it > addressed. > > > Rejected alternative: distribute the security enhancements via PyPI > ------------------------------------------------------------------- > > While this initially appears to be an attractive and easier to manage > approach, it actually suffers from several significant problems. > > Firstly, this is complex, low level, cross-platform code that integrates > with the underlying operating system across a variety of POSIX platforms > (including Mac OS X) and Windows. The CPython BuildBot fleet is already set > up to handle continuous integration in that context, but most of the > freely available continuous integration services just offer Linux, and > perhaps paid access to Windows. Those services work reasonably well for > software that largely runs on the abstraction layers offered by Python and > other dynamic languages, as well as the more comprehensive abstraction > offered by the JVM, but won't suffice for the kind of code involved here. > > The OpenSSL dependency for the network security support also qualifies as > the kind of "complex binary dependency" that isn't yet handled well by the > ``pip`` based software distribution ecosystem. Relying on a third party > binary dependency also creates potential compatibility problems for ``pip`` > when running on other interpreters like ``PyPy``. > > Another practical problem with the idea is the fact that ``pip`` itself > relies on the ``ssl`` support in the standard library (with some additional > support from a bundled copy of ``requests``, which in turn bundles > ``backport.ssl_match_hostname``), and hence would require any replacement > module to also be bundled within ``pip``. This wouldn't pose any > insurmountable difficulties (it's just another dependency to vendor), but > it *would* mean yet another copy of OpenSSL to keep up to date. > > This approach also has the same flaw as all other "improve security by > renaming things" approaches: they completely miss the users who most need > help, and raise significant barriers against being able to encourage users > to do the right thing when their infrastructure supports it (since > "use this other module" is a much higher impact change than "turn on this > higher security setting"). Deprecating the aging SSL infrastructure in the > standard library in favour of an external module would be even more user > hostile than accepting the slightly increased risk of regressions associated > with upgrading it in place. > > Last, but certainly not least, this approach suffers from the same problem > as the idea of doing a Python 2.8 release: likely not solving the actual > problem. Commercial redistributors of Python are set up to redistribute > *Python*, and a pre-existing set of additional packages. Getting new > packages added to the pre-existing set *can* be done, but means approaching > each and every redistributor and asking them to update their > repackaging process accordingly. By contrast, the approach described in > this PEP would require redistributors to deliberately *opt out* of the > security enhancements by deliberately downgrading the provided network > security infrastructure, which most of them are unlikely to do. > > > Rejected variant: provide a "legacy SSL infrastructure" branch > -------------------------------------------------------------- > > Earlier versions of this PEP included the concept of a ``2.7-legacy-ssl`` > branch that preserved the exact feature set of the Python 2.7.6 network > security infrastructure. > > It is the opinion of the PEP author that anyone that actually wants this is > almost certainly making a mistake, and if they insist they really do want > it in their specific situation, they're welcome to either make it themselves > or arrange for a downstream redistributor to make it for them. > > If they are made publicly available, any such rebuilds should be referred to > as "Python 2.7 with Legacy SSL" to clearly distinguish them from the official > Python 2.7 releases that include more up to date network security > infrastructure. > > After the first Python 2.7 maintenance release that has the security > infrastructure updated to match Python 3.4, it would also be appropriate to > refer to Python 2.7.6 and earlier releases as "Python 2.7 with Legacy SSL". > > > Rejected variant: synchronise particular modules entirely with Python 3 > ----------------------------------------------------------------------- > > Earlier versions of this PEP suggested synchronising the ``hmac``, > ``hashlib`` and ``ssl`` modules entirely with their Python 3 counterparts. > > This approach proved too vague to build a compelling case for the exception, > and has thus been replaced by the current more explicit proposal. > > > Open Questions > ============== > > * MvL has indicated he is not prepared to tackle the task of trying to > integrate a newer OpenSSL into the also aging Python 2.7 build > infrastructure on Windows (unfortunately, we've looked into upgrading > that build infrastructure, and the backwards compatibility issues > appear to be effectively insurmountable). We would require a commitment > from another trusted contributor to handle at least this task, and > potentially also taking over the task of creating the official > Python 2.7 Windows installers for the remaining Python 2.7 maintenance > releases. > > * We would need commitments to create and review full backports of the > components covered by this policy from Python 3.4 to Python 2.7, as well > as support for handling any more specific security issues affecting these > modules. > > * Did I miss anything important in the switch to a more restrictive > proposal? > > > Disclosure of Interest > ====================== > > The author of this PEP currently works for Red Hat on test automation tools. > If this proposal is accepted, I will be strongly encouraging Red Hat to take > advantage of the resulting opportunity to help improve the overall security > of the Python ecosystem. However, I do not speak for Red Hat in this matter, > and cannot make any commitments on Red Hat's behalf. > > > Acknowledgements > ================ > > Thanks to Christian Heimes and other for their efforts in greatly improving > Python's SSL support in the Python 3 series, and a variety of members of > the Python community for helping me to better understand the implications > of the default settings we provide in our SSL modules, and the impact that > tolerating the use of SSL infrastructure that was defined in 2010 > (Python 2.7) or even 2008 (Python 2.6) potentially has for the security > of the web as a whole. > > Thanks to Donald Stufft and Alex Gaynor for identifying a more limited set > of essential security features that allowed the proposal to be made more > fine-grained than backporting entire modules from Python 3.4 [7,8]_. > > Christian and Donald also provided valuable feedback on a preliminary > draft of this proposal. > > Thanks also to participants in the python-dev mailing list threads > [1,2,5,6]_ > > > References > ========== > > .. [1] PEP 466 discussion (round 1) > (https://mail.python.org/pipermail/python-dev/2014-March/133334.html) > > .. [2] PEP 466 discussion (round 2) > (https://mail.python.org/pipermail/python-dev/2014-March/133389.html) > > .. [3] Marc-Andre Lemburg's OpenSSL feedback for Windows > (https://mail.python.org/pipermail/python-dev/2014-March/133438.html) > > .. [4] Ned Deily's OpenSSL feedback for Mac OS X > (https://mail.python.org/pipermail/python-dev/2014-March/133347.html) > > .. [5] PEP 466 discussion (round 3) > (https://mail.python.org/pipermail/python-dev/2014-March/133442.html) > > .. [6] PEP 466 discussion (round 4) > (https://mail.python.org/pipermail/python-dev/2014-March/133472.html) > > .. [7] Donald Stufft's recommended set of backported features > (https://mail.python.org/pipermail/python-dev/2014-March/133500.html) > > .. [8] Alex Gaynor's recommended set of backported features > (https://mail.python.org/pipermail/python-dev/2014-March/133503.html) > > > > Copyright > ========= > > This document has been placed in the public domain. > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Mar 26 13:09:42 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Mar 2014 22:09:42 +1000 Subject: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7 In-Reply-To: References: Message-ID: On 26 March 2014 22:05, Donald Stufft wrote: > Typo I think: > > As in the Python 3 series, the backported ssl.create_default_context() API > is granted a backwards compatibility exemption that permits the protocol, > options, cipher and other settings of the created SSL context to be made > > made what? Incomplete edit that I didn't notice until after posting. Fixed in the web version now, along with the copy & paste error in the list of hashlib attributes to be backported. The first two backwards compatibility paragraphs: ======================= As in the Python 3 series, the backported ssl.create_default_context() API is granted a backwards compatibility exemption that permits the protocol, options, cipher and other settings of the created SSL context to be updated in maintenance releases to use higher default security settings. This allows them to appropriately balance compatibility and security at the time of the maintenance release, rather than at the time of the original feature release. This PEP does not grant any other exemptions to the usual backwards compatibility policy for maintenance releases. Instead, by explicitly encouraging the use of feature based checks, it is designed to make it easier to write more secure cross-version compatible Python software, while still limiting the risk of breaking currently working software when upgrading to a new Python 2.7 maintenance release. ======================= Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed Mar 26 13:20:01 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 26 Mar 2014 08:20:01 -0400 Subject: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7 In-Reply-To: References: Message-ID: On Mar 26, 2014, at 8:00 AM, Nick Coghlan wrote: > Guido and Antoine persuaded me that selective backports would be a > better idea for the network security enhancements than the wholesale > module backports previously suggested, while Alex and Donald provided > the necessary additional details, so here's a revised version of the > PEP. Despite making it more explicit, I deleted more lines than I > added, strongly suggesting that switching to selective backports was > the right call :) > > I dealt with the SSL module the way Donald suggested: excluding the > RAND_* functions, rather than listing everything else. > > I also changed the headings to make it clear the listed alternatives > were rejected ideas, made the footnotes a bit more readable, and > tidied up the wording in a few places. > > Diff: http://hg.python.org/peps/rev/8527f6e2beb0 > Web: http://www.python.org/dev/peps/pep-0466/ > > ========================== > PEP: 466 > Title: Network Security Enhancement Exception for Python 2.7 > Version: $Revision$ > Last-Modified: $Date$ > Author: Nick Coghlan , > Status: Draft > Type: Informational > Content-Type: text/x-rst > Created: 23-Mar-2014 > Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014 > > This looks reasonable to me still and still solves the major problems that trying to securely use the 2.7 series has. +1 From me. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From martin at v.loewis.de Wed Mar 26 14:22:13 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 26 Mar 2014 14:22:13 +0100 Subject: [Python-Dev] C code: %s vs %U In-Reply-To: <533230C2.60302@stoneleaf.us> References: <533230C2.60302@stoneleaf.us> Message-ID: <5332D485.1050605@v.loewis.de> [Assuming you are talking about PyUnicode_FromFormatV] > %s is a string. No. %s is a char*; C does not have a "string" type. The string behind the pointer should be UTF-8 encoded; other encodings are tolerated through the "replace" error handler. > %U is unicode? No. This is a PyObject* whose Python type is 'str' (i.e. an object for which PyUnicode_Check succeeds) > If so, then %s should only be used when it is certain the string in > question has no unicode in it? No. If you have a char*, use %s; using %U would crash. Regards, Martin From ethan at stoneleaf.us Wed Mar 26 14:06:28 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 26 Mar 2014 06:06:28 -0700 Subject: [Python-Dev] C code: %s vs %U In-Reply-To: <20140326114844.0422b5f3@fsol> References: <533230C2.60302@stoneleaf.us> <20140326114844.0422b5f3@fsol> Message-ID: <5332D0D4.8070109@stoneleaf.us> On 03/26/2014 03:48 AM, Antoine Pitrou wrote: > On Tue, 25 Mar 2014 18:43:30 -0700 > Ethan Furman wrote: >> %s is a string. >> >> %U is unicode? > > What is the context? A patch I'm adapting has these lines: static PyObject* module_getattr(PyObject *m, PyObject *name) { char *mod_name_str; ... PyErr_Format(PyExc_AttributeError, "module '%s' has no attribute '%U'", mod_name_str, name); So it looks like %s is referring to a simple string, and %U is referring to a PyObject, but I was hoping for verification. -- ~Ethan~ From d4n1h4ck at gmail.com Wed Mar 26 14:50:52 2014 From: d4n1h4ck at gmail.com (Daniel Pimentel (d4n1)) Date: Wed, 26 Mar 2014 10:50:52 -0300 Subject: [Python-Dev] [RELEASED] Python 3.4.0 In-Reply-To: References: <53269663.9050408@hastings.org> Message-ID: Good job men :D 2014-03-17 14:18 GMT-03:00 Alioune Dia : > yeah , asyncio is a great module, congrat for all jobs you are doing > --Ad | Dakar > > > 2014-03-17 18:11 GMT+01:00 Giampaolo Rodola' : > >> The what's new looks truly amazing, with pathlib and asyncio being my >> favourite additions. >> Thanks for all the hard work. >> >> >> On Mon, Mar 17, 2014 at 5:57 PM, Ryan Gonzalez wrote: >> >>> YES!!! +1 to the authors of the statistics and pathlib modules. >>> >>> >>> On Mon, Mar 17, 2014 at 1:29 AM, Larry Hastings wrote: >>> >>>> >>>> On behalf of the Python development team, I'm thrilled to announce >>>> the official release of Python 3.4. >>>> >>>> >>>> Python 3.4 includes a range of improvements of the 3.x series, including >>>> hundreds of small improvements and bug fixes. Major new features and >>>> changes in the 3.4 release series include: >>>> >>>> * PEP 428, a "pathlib" module providing object-oriented filesystem paths >>>> * PEP 435, a standardized "enum" module >>>> * PEP 436, a build enhancement that will help generate introspection >>>> information for builtins >>>> * PEP 442, improved semantics for object finalization >>>> * PEP 443, adding single-dispatch generic functions to the standard >>>> library >>>> * PEP 445, a new C API for implementing custom memory allocators >>>> * PEP 446, changing file descriptors to not be inherited by default >>>> in subprocesses >>>> * PEP 450, a new "statistics" module >>>> * PEP 451, standardizing module metadata for Python's module import >>>> system >>>> * PEP 453, a bundled installer for the *pip* package manager >>>> * PEP 454, a new "tracemalloc" module for tracing Python memory >>>> allocations >>>> * PEP 456, a new hash algorithm for Python strings and binary data >>>> * PEP 3154, a new and improved protocol for pickled objects >>>> * PEP 3156, a new "asyncio" module, a new framework for asynchronous I/O >>>> >>>> >>>> To download Python 3.4.0 visit: >>>> >>>> http://www.python.org/download/releases/3.4.0/ >>>> >>>> >>>> This is a production release. Please report any issues you notice to: >>>> >>>> http://bugs.python.org/ >>>> >>>> >>>> Enjoy! >>>> >>>> >>>> -- >>>> Larry Hastings, Release Manager >>>> larry at hastings.org >>>> (on behalf of the entire python-dev team and 3.4's contributors) >>>> _______________________________________________ >>>> Python-Dev mailing list >>>> Python-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/python-dev >>>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ >>>> rymg19%40gmail.com >>>> >>> >>> >>> >>> -- >>> Ryan >>> If anybody ever asks me why I prefer C++ to C, my answer will be simple: >>> "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was >>> nul-terminated." >>> >>> >>> -- >>> https://mail.python.org/mailman/listinfo/python-list >>> >>> >> >> >> -- >> Giampaolo - http://grodola.blogspot.com >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/dia.aliounes%40gmail.com >> >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/d4n1h4ck%40gmail.com > > -- Msc. Daniel Pimentel (d4n1 ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Mar 26 14:57:47 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Mar 2014 23:57:47 +1000 Subject: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7 In-Reply-To: References: Message-ID: On 26 Mar 2014 23:12, "Cory Benfield" wrote: > > Nick, > > On 26 March 2014 12:00, Nick Coghlan wrote: > > As one example, the Python 2 ``ssl`` module does not support the Server > > Name Identification standard. > > Tiny note: SNI is 'Server Name Indication', not 'Identification'. =) Hah, you know, I've never actually looked that up, because I assumed I knew what it stood for based on context :) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Mar 26 15:03:02 2014 From: barry at python.org (Barry Warsaw) Date: Wed, 26 Mar 2014 10:03:02 -0400 Subject: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7 In-Reply-To: References: Message-ID: <20140326100302.499c6006@anarchist.wooz.org> On Mar 26, 2014, at 10:00 PM, Nick Coghlan wrote: >Guido and Antoine persuaded me that selective backports would be a >better idea for the network security enhancements than the wholesale >module backports previously suggested, while Alex and Donald provided >the necessary additional details, so here's a revised version of the >PEP. Despite making it more explicit, I deleted more lines than I >added, strongly suggesting that switching to selective backports was >the right call :) Thanks. I think the narrower scope and details greatly improves this PEP and resolves most of the concerns I had about it. It maintains the backward compatibility pledge but relaxes the new feature backport prohibition where security concerns trump. Cheers, -Barry From bcannon at gmail.com Wed Mar 26 15:01:00 2014 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 26 Mar 2014 14:01:00 +0000 Subject: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements References: <532E63A8.9020705@udel.edu> <20140323200312.33b6b0b1@limelight.wooz.org> <20140323213112.1b7d2e32@limelight.wooz.org> <5330E929.2070607@mrabarnett.plus.com> Message-ID: On Wed Mar 26 2014 at 2:15:39 AM, Georg Brandl wrote: > Am 26.03.2014 00:06, schrieb Nick Coghlan: > > > > On 26 Mar 2014 08:32, "Georg Brandl" > > wrote: > >> > >> Am 25.03.2014 23:15, schrieb Nick Coghlan: > >> > > >> > On 26 Mar 2014 01:19, "Brett Cannon" > > >> > >> wrote: > >> >> As long as we make it clear we have chosen to change our > >> > backwards-compatibility guarantees in the name of security and have a > link to > >> > the last backwards-compatible release then I agree as well. > >> > > >> > I am not sure how this meme got started, but let me be clear: the > proposed > >> > policy DOES NOT provide blanket permission to break backwards > compatibility in > >> > the affected modules. It only allows ADDING new features to bring > these modules > >> > into line with their Python 3 counterparts, making it easier for > third party > >> > packages like requests to do the right thing in a cross-version > compatible way. > >> > >> We know. That's what we mean by that. > > > > That's not what Brett said - he called 2.7.6 the "last backwards > compatible > > release". That's not correct, as even under my proposal, 2.7.7+ will > still be > > backwards compatible. > > Yeah, I took "backwards-compatibility guarantees" to also mean the "no new > features" guarantee, but you're right that the two can be separated. > That's also the way I took it; not that code would be inherently broken from 2.7.0 -> 2.7.x, but that the "let's stay conservative and add *nothing*" would stop being followed since something in 2.7.x wouldn't work in 2.7.0. IOW I expect (except maybe in extreme edge cases) that unmodified code would work across 2.7 as long as it works in 2.7.0 no matter what happens with this PEP. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Wed Mar 26 15:26:18 2014 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 26 Mar 2014 14:26:18 +0000 Subject: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7 References: Message-ID: On Wed Mar 26 2014 at 8:02:08 AM, Nick Coghlan wrote: > Guido and Antoine persuaded me that selective backports would be a > better idea for the network security enhancements than the wholesale > module backports previously suggested, while Alex and Donald provided > the necessary additional details, so here's a revised version of the > PEP. Despite making it more explicit, I deleted more lines than I > added, strongly suggesting that switching to selective backports was > the right call :) > > I dealt with the SSL module the way Donald suggested: excluding the > RAND_* functions, rather than listing everything else. > > I also changed the headings to make it clear the listed alternatives > were rejected ideas, made the footnotes a bit more readable, and > tidied up the wording in a few places. > > Diff: http://hg.python.org/peps/rev/8527f6e2beb0 > Web: http://www.python.org/dev/peps/pep-0466/ > > ========================== > PEP: 466 > Title: Network Security Enhancement Exception for Python 2.7 > Version: $Revision$ > Last-Modified: $Date$ > Author: Nick Coghlan , > Status: Draft > Type: Informational > Content-Type: text/x-rst > Created: 23-Mar-2014 > Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014 > [SNIP] > Exemption Policy > ================ > > Under this policy, the following features SHOULD be backported from Python > 3.4 to the upcoming Python 2.7.7 maintenance release: > > * in the ``os`` module: > > * persistent file descriptor for ``os.urandom()``. > > * in the ``hmac`` module: > > * constant time comparison function (``hmac.compare_digest()``). > > * in the ``hashlib`` module: > > * password hashing function (``hashlib.pbkdf2_hmac()``). > * details of hash algorithm availability (``hashlib.algorithms_ > guaranteed`` > and ``hashlib.algorithms_guaranteed``). > You said algorithms_guaranteed twice. I assume that wasn't for emphasis. =) I'm +1 on this version of the PEP. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Mar 26 15:07:45 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 26 Mar 2014 07:07:45 -0700 Subject: [Python-Dev] C code: %s vs %U In-Reply-To: <5332D485.1050605@v.loewis.de> References: <533230C2.60302@stoneleaf.us> <5332D485.1050605@v.loewis.de> Message-ID: <5332DF31.60609@stoneleaf.us> On 03/26/2014 06:22 AM, ? wrote: > [Assuming you are talking about PyUnicode_FromFormatV] >> %s is a string. > > No. %s is a char*; C does not have a "string" type. > The string behind the pointer should be UTF-8 encoded; > other encodings are tolerated through the "replace" error > handler. > >> %U is unicode? > > No. This is a PyObject* whose Python type is 'str' > (i.e. an object for which PyUnicode_Check succeeds) > >> If so, then %s should only be used when it is certain the string in >> question has no unicode in it? > > No. If you have a char*, use %s; using %U would crash. Many thanks! -- ~Ethan~ From cory at lukasa.co.uk Wed Mar 26 14:12:10 2014 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 26 Mar 2014 13:12:10 +0000 Subject: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7 In-Reply-To: References: Message-ID: Nick, On 26 March 2014 12:00, Nick Coghlan wrote: > As one example, the Python 2 ``ssl`` module does not support the Server > Name Identification standard. Tiny note: SNI is 'Server Name Indication', not 'Identification'. =) Otherwise, I'm +1 on this as well. From ethan at stoneleaf.us Wed Mar 26 15:35:52 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 26 Mar 2014 07:35:52 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> Message-ID: <5332E5C8.1010506@stoneleaf.us> On 03/26/2014 03:10 AM, Victor Stinner wrote: > 2014-03-25 23:37 GMT+01:00 Ethan Furman: >> >> ``%a`` will call ``ascii()`` on the interpolated value. > > I'm not sure that I understood correctly: is the "%a" format > supported? The result of ascii() is a Unicode string. Does it mean > that ("%a" % obj) should give the same result than > ascii(obj).encode('ascii', 'strict')? Changed to: ------------------------------------------------------------------------------- ``%a`` will give the equivalent of ``repr(some_obj).encode('ascii', 'backslashreplace')`` on the interpolated value. Use cases include developing a new protocol and writing landmarks into the stream; debugging data going into an existing protocol to see if the problem is the protocol itself or bad data; a fall-back for a serialization format; or any situation where defining ``__bytes__`` would not be appropriate but a readable/informative representation is needed [8]. ------------------------------------------------------------------------------- > Would it be possible to add a table or list to summarize supported > format characters? I found: > > - single byte: %c > - integer: %d, %u, %i, %o, %x, %X, %f, %g, "etc." (can you please > complete "etc." ?) > - bytes and __bytes__ method: %s > - ascii(): %a Changed to: ------------------------------------------------------------------------------- %-interpolation --------------- All the numeric formatting codes (``d``, ``i``, ``o``, ``u``, ``x``, ``X``, ``e``, ``E'', ``f``, ``F``, ``g``, ``G``, and any that are subsequently added to Python 3) will be supported, and will work as they do for str, including the padding, justification and other related modifiers (currently ``#``, ``0``, ``-``, `` `` (space), and ``+`` (plus any added to Python 3)). The only non-numeric codes allowed are ``c``, ``s``, and ``a``. For the numeric codes, the only difference between ``str`` and ``bytes`` (or ``bytearray``) interpolation is that the results from these codes will be ASCII-encoded text, not unicode. In other words, for any numeric formatting code `%x`:: ------------------------------------------------------------------------------- > I don't understand the purpose of this sentence. Does it mean that %a > must not be used? IMO this sentence can be removed. The sentence about %a being for debugging has been removed. >> Non-ASCII values will be encoded to either ``\xnn`` or ``\unnnn`` >> representation. > > Unicode is larger than that! print(ascii(chr(0x10ffff))) => '\U0010ffff' Removed. With the explicit reference to the 'backslashreplace' error handler any who want to know what it might look like can refer to that. >> .. note:: >> >> If a ``str`` is passed into ``%a``, it will be surrounded by quotes. > > And: > > - bytes gets a "b" prefix and surrounded by quotes as well (b'...') > - the quote ' is escaped as \' if the string contains quotes ' and " Shouldn't be an issue now with the new definition which no longer references the ascii() function. > Can you also please add examples for %a? ------------------------------------------------------------------------------- Examples:: >>> b'%a' % 3.14 b'3.14' >>> b'%a' % b'abc' b'abc' >>> b'%a' % 'def' b"'def'" ------------------------------------------------------------------------------- >> Proposed variations >> =================== >> > > It would be fair to mention also a whole different PEP, Antoine's PEP 460! My apologies for the omission. ------------------------------------------------------------------------------- A competing PEP, ``PEP 460 Add binary interpolation and formatting`` [9], also exists. .. [9] http://python.org/dev/peps/pep-0460/ ------------------------------------------------------------------------------- Thank you, Victor. From thomas at python.org Wed Mar 26 16:14:00 2014 From: thomas at python.org (Thomas Wouters) Date: Wed, 26 Mar 2014 16:14:00 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <53320517.3010101@stoneleaf.us> References: <53320517.3010101@stoneleaf.us> Message-ID: On Tue, Mar 25, 2014 at 11:37 PM, Ethan Furman wrote: > In particular, ``%s`` will not accept numbers (use a numeric format code > for > that), nor ``str`` (encode it to ``bytes``). > I don't understand this restriction, and there isn't a rationale for it in the PEP (other than "you can already use numeric formats", which doesn't explain why it's undesirable to have it anyway.) It is extremely common in existing 2.x code to use %s for anything, just like people use {} for anything with str.format. Not supporting this feels like it would be problematic for porting code. Did this come up in the earlier discussions? -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Wed Mar 26 16:26:36 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 26 Mar 2014 15:26:36 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?PEP_466_=28round_5=29=3A_selected_network_?= =?utf-8?q?security=09enhancements_for_Python_2=2E7?= References: Message-ID: This mostly looks good to me, however I'm not sure I understand the point of this sentence: "Rather, it is intended to send a clear signal to potential corporate contributors that the core development team are willing to accept offers of corporate assistance in putting this policy into effect [...]". It's fairly evident to me that the folks most likely to actually do the work of implementing this are myself and Donald. This PEP really has nothing to do with corporate contribution, so I think this sentence ought to be removed. Alex From ethan at stoneleaf.us Wed Mar 26 16:20:21 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 26 Mar 2014 08:20:21 -0700 Subject: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7 In-Reply-To: References: Message-ID: <5332F035.3060207@stoneleaf.us> On 03/26/2014 05:00 AM, Nick Coghlan wrote: > The Mac OS X binary installers historically followed the same policy as > other POSIX installations and dynamically linked to the Apple provided > OpenSSL libraries. However, Apple has now ceased updating these > cross-platform libraries, instead requiring that even cross-platform > developers adopt Mac OS X specific interfaces to access up to date security > infrastructure on their platform. Accordingly, and independently of this > PEP, the Mac OS X binary installers were already going to be switched to > statically linker newer versions of OpenSSL [4]_ Should be 'link', not 'linker'. Looks good. +1 -- ~Ethan~ From ethan at stoneleaf.us Wed Mar 26 17:47:25 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 26 Mar 2014 09:47:25 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> Message-ID: <5333049D.5050107@stoneleaf.us> On 03/26/2014 08:14 AM, Thomas Wouters wrote: > On Tue, Mar 25, 2014 at 11:37 PM, Ethan Furman wrote: > > In particular, ``%s`` will not accept numbers (use a numeric format code for > that), nor ``str`` (encode it to ``bytes``). > > > I don't understand this restriction, and there isn't a rationale for it in the PEP (other than "you can already use > numeric formats", which doesn't explain why it's undesirable to have it anyway.) It is extremely common in existing 2.x > code to use %s for anything And that's the problem -- in 2.x %s works always, but 3.x for bytes and bytearray %s will fail in numerous situations. It seems to me the main reason for using %s instead of %d is that 'some_var' may have a number, or it may have the textual representation of that number; in 3.x the first would succeed, the second would fail. That's the kind of intermittent failure we do not want. The PEP is not designed to make it so 2.x code can be ported as-is, but rather that 2.x code can be cleaned up (if necessary) and then run the same in both 2.x and 3.x (at least as far as byte and bytearray %-formatting is concerned). > Did this come up in the earlier discussions? https://mail.python.org/pipermail/python-dev/2014-January/131576.html -- ~Ethan~ From marko at pacujo.net Wed Mar 26 21:31:56 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Wed, 26 Mar 2014 22:31:56 +0200 Subject: [Python-Dev] collections.sortedtree Message-ID: <8738i4og5f.fsf@elektro.pacujo.net> I have made a full implementation of a balanced tree and would like to know what the process is to have it considered for inclusion in Python 3. To summarize, the implementation closely parallels dict() features and resides in _collectionsmodule.c under the name collections.sortedtree. It uses solely the "<" operator to compare keys. I have chosen the AVL tree as an implementation technique. The views support a number of optional arguments: sorteddict.keys(reversed=False, start=unspecified, inclusive=True) The primary objective of having a balanced tree in the standard library is to support ordered access in an efficient manner. The typical applications would include timers (networking), aging (cache) and prefix patterns (routing). Marko From ethan at stoneleaf.us Wed Mar 26 21:53:41 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 26 Mar 2014 13:53:41 -0700 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <8738i4og5f.fsf@elektro.pacujo.net> References: <8738i4og5f.fsf@elektro.pacujo.net> Message-ID: <53333E55.6040209@stoneleaf.us> On 03/26/2014 01:31 PM, Marko Rauhamaa wrote: > > I have made a full implementation of a balanced tree and would like to > know what the process is to have it considered for inclusion in Python > 3. Open a ticket on the tracker [1], post your code to that ticket, sign the CLA [2], answer questions, etc., that come up in the code review. I believe Raymond Hettinger is the Guardian of the collections module; add him as nosy. -- ~Ethan~ [1] http://http://bugs.python.org [2] https://www.python.org/psf/contrib/contrib-form From benjamin at python.org Wed Mar 26 21:55:00 2014 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 26 Mar 2014 13:55:00 -0700 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <8738i4og5f.fsf@elektro.pacujo.net> References: <8738i4og5f.fsf@elektro.pacujo.net> Message-ID: <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> On Wed, Mar 26, 2014, at 13:31, Marko Rauhamaa wrote: > > I have made a full implementation of a balanced tree and would like to > know what the process is to have it considered for inclusion in Python > 3. It's not a bad idea. (I believe others have proposed an red-black tree.) Certainly, it requires a PEP and a few months of bikesheding, though. From guido at python.org Wed Mar 26 21:59:06 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Mar 2014 13:59:06 -0700 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <53333E55.6040209@stoneleaf.us> References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> Message-ID: Actually, the first step is publish it on PyPI, the second is to get a fair number of happy users there. The bar for getting something included into the stdlib is pretty high -- you need to demonstrate that there is a need *and* that having it as a 3rd party module is a problem. And that once it's in, (a) it will be stable, and (b) someone who cares about it and knows the code thoroughly is available maintain it for years. On Wed, Mar 26, 2014 at 1:53 PM, Ethan Furman wrote: > On 03/26/2014 01:31 PM, Marko Rauhamaa wrote: > >> >> I have made a full implementation of a balanced tree and would like to >> know what the process is to have it considered for inclusion in Python >> 3. >> > > Open a ticket on the tracker [1], post your code to that ticket, sign the > CLA [2], answer questions, etc., that come up in the code review. > > I believe Raymond Hettinger is the Guardian of the collections module; add > him as nosy. > > -- > ~Ethan~ > > [1] http://http://bugs.python.org > [2] https://www.python.org/psf/contrib/contrib-form > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Mar 26 22:25:12 2014 From: barry at python.org (Barry Warsaw) Date: Wed, 26 Mar 2014 17:25:12 -0400 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> Message-ID: <20140326172512.22e1f21a@anarchist.wooz.org> On Mar 26, 2014, at 01:55 PM, Benjamin Peterson wrote: >It's not a bad idea. (I believe others have proposed an red-black tree.) >Certainly, it requires a PEP and a few months of bikesheding, though. Generally, PEPs aren't necessary for simple or relatively uncontroversial additions to existing modules or the stdlib. If you're proposing an addition to an existing module, file a tracker issue and nosy the module expert (for collections, it's Raymond Hettinger[1]). Others may see the new issue and decide to nosy themselves in if they want to provide additional feedback and input. If you're proposing an entirely new module, it's best to publish it first on PyPI, get lots of users, and listen to their feedback as you evolve the library. Once you feel like it's had plenty of time to bake as a third party module *and* the API is sufficiently stable that you won't mind the Python stdlib 18 month development cycle, then you can propose that it be included in Python. Cheers, -Barry [1] http://docs.python.org/devguide/experts.html From solipsis at pitrou.net Wed Mar 26 22:26:28 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 Mar 2014 22:26:28 +0100 Subject: [Python-Dev] collections.sortedtree References: <8738i4og5f.fsf@elektro.pacujo.net> Message-ID: <20140326222628.24336818@fsol> On Wed, 26 Mar 2014 22:31:56 +0200 Marko Rauhamaa wrote: > > The primary objective of having a balanced tree in the standard library > is to support ordered access in an efficient manner. The typical > applications would include timers (networking), aging (cache) Wouldn't a heapq work as well for those two? Regards Antoine. From ncoghlan at gmail.com Wed Mar 26 22:27:22 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Mar 2014 07:27:22 +1000 Subject: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7 In-Reply-To: References: Message-ID: On 27 Mar 2014 01:28, "Alex Gaynor" wrote: > > This mostly looks good to me, however I'm not sure I understand the point of > this sentence: "Rather, it is intended to send a clear signal to potential > corporate contributors that the core development team are willing to accept > offers of corporate assistance in putting this policy into effect [...]". It's > fairly evident to me that the folks most likely to actually do the work of > implementing this are myself and Donald. This PEP really has nothing to do with > corporate contribution, so I think this sentence ought to be removed. Sure, I can reword that bit. I think I've made my point on that front by now, anyway, so preserving it in the web version isn't really necessary :) Cheers, Nick. > > Alex > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Wed Mar 26 22:27:54 2014 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 26 Mar 2014 14:27:54 -0700 Subject: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"] In-Reply-To: <20140326172512.22e1f21a@anarchist.wooz.org> References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> <20140326172512.22e1f21a@anarchist.wooz.org> Message-ID: <1395869274.5751.99327085.56513D6F@webmail.messagingengine.com> On Wed, Mar 26, 2014, at 14:25, Barry Warsaw wrote: > On Mar 26, 2014, at 01:55 PM, Benjamin Peterson wrote: > > >It's not a bad idea. (I believe others have proposed an red-black tree.) > >Certainly, it requires a PEP and a few months of bikesheding, though. > > Generally, PEPs aren't necessary for simple or relatively uncontroversial > additions to existing modules or the stdlib. I would have said that, too, several years ago, but I think we've been requiring (or using anyway) PEPs for a lot more things now. OrderedDict had a PEP for example. I'm not sure if that's a good thing or not. From barry at python.org Wed Mar 26 22:30:30 2014 From: barry at python.org (Barry Warsaw) Date: Wed, 26 Mar 2014 17:30:30 -0400 Subject: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"] In-Reply-To: <1395869274.5751.99327085.56513D6F@webmail.messagingengine.com> References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> <20140326172512.22e1f21a@anarchist.wooz.org> <1395869274.5751.99327085.56513D6F@webmail.messagingengine.com> Message-ID: <20140326173030.2ee5dc11@anarchist.wooz.org> On Mar 26, 2014, at 02:27 PM, Benjamin Peterson wrote: >I would have said that, too, several years ago, but I think we've been >requiring (or using anyway) PEPs for a lot more things now. OrderedDict >had a PEP for example. > >I'm not sure if that's a good thing or not. Hmm, me neither! I guess if someone *wants* to go through the PEP gauntlet, I won't stop them. It builds character. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From donald at stufft.io Wed Mar 26 22:37:40 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 26 Mar 2014 17:37:40 -0400 Subject: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"] In-Reply-To: <20140326173030.2ee5dc11@anarchist.wooz.org> References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> <20140326172512.22e1f21a@anarchist.wooz.org> <1395869274.5751.99327085.56513D6F@webmail.messagingengine.com> <20140326173030.2ee5dc11@anarchist.wooz.org> Message-ID: <447D6458-D0EB-45F4-88F3-65A771B530A5@stufft.io> On Mar 26, 2014, at 5:30 PM, Barry Warsaw wrote: > On Mar 26, 2014, at 02:27 PM, Benjamin Peterson wrote: > >> I would have said that, too, several years ago, but I think we've been >> requiring (or using anyway) PEPs for a lot more things now. OrderedDict >> had a PEP for example. >> >> I'm not sure if that's a good thing or not. > > Hmm, me neither! I guess if someone *wants* to go through the PEP gauntlet, I > won't stop them. It builds character. > > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Is that what it?s called? ?character? >:] ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From marko at pacujo.net Wed Mar 26 22:36:02 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Wed, 26 Mar 2014 23:36:02 +0200 Subject: [Python-Dev] collections.sortedtree In-Reply-To: (Guido van Rossum's message of "Wed, 26 Mar 2014 13:59:06 -0700") References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> Message-ID: <87vbv0mym5.fsf@elektro.pacujo.net> Guido van Rossum : > Actually, the first step is publish it on PyPI, the second is to get a > fair number of happy users there. The bar for getting something > included into the stdlib is pretty high -- you need to demonstrate > that there is a need *and* that having it as a 3rd party module is a > problem. I hear you about the process. About the need part, I'm wondering if you haven't felt it in implementing the timers for asyncio. I have had that need in several network programming projects and have ended up using my AVL tree implementation (C and Python). Well, time will tell if frequent canceled timers end up piling up the heap queue. Marko From tjreedy at udel.edu Wed Mar 26 22:37:58 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 26 Mar 2014 17:37:58 -0400 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> Message-ID: On 3/26/2014 4:59 PM, Guido van Rossum wrote: > Actually, the first step is publish it on PyPI, the second is to get a > fair number of happy users there. The bar for getting something included > into the stdlib is pretty high -- you need to demonstrate that there is > a need *and* that having it as a 3rd party module is a problem. And that > once it's in, (a) it will be stable, and (b) someone who cares about it > and knows the code thoroughly is available maintain it for years. Perhaps the collections doc should mention that there are other specializes container types available on PyPI and either list some or point to a wiki page listing some. There must be at least 10 that could be included in such a list. -- Terry Jan Reedy From victor.stinner at gmail.com Wed Mar 26 22:41:12 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 26 Mar 2014 22:41:12 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <5332E5C8.1010506@stoneleaf.us> References: <53320517.3010101@stoneleaf.us> <5332E5C8.1010506@stoneleaf.us> Message-ID: 2014-03-26 15:35 GMT+01:00 Ethan Furman : > ------------------------------------------------------------------------------- > Examples:: > > >>> b'%a' % 3.14 > b'3.14' > > >>> b'%a' % b'abc' > b'abc' This one is wrong: >>> repr(b'abc').encode('ascii', 'backslashreplace') b"b'abc'" Victor From guido at python.org Wed Mar 26 22:42:07 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Mar 2014 14:42:07 -0700 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <87vbv0mym5.fsf@elektro.pacujo.net> References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <87vbv0mym5.fsf@elektro.pacujo.net> Message-ID: I haven't felt it, heapq feels natural to me for this use case. :-) I'm aware of the issue of frequent cancelled timers, but chose to wait and see rather than preemptively fix it (only so many hours in a day). IIRC pyftplib has a clever cleanup algorithm that we can easily add if that usage pattern becomes popular. On Wed, Mar 26, 2014 at 2:36 PM, Marko Rauhamaa wrote: > Guido van Rossum : > > > Actually, the first step is publish it on PyPI, the second is to get a > > fair number of happy users there. The bar for getting something > > included into the stdlib is pretty high -- you need to demonstrate > > that there is a need *and* that having it as a 3rd party module is a > > problem. > > I hear you about the process. > > About the need part, I'm wondering if you haven't felt it in > implementing the timers for asyncio. I have had that need in several > network programming projects and have ended up using my AVL tree > implementation (C and Python). > > Well, time will tell if frequent canceled timers end up piling up the > heap queue. > > > Marko > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Mar 26 22:44:15 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Mar 2014 07:44:15 +1000 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> Message-ID: On 27 Mar 2014 07:02, "Guido van Rossum" wrote: > > Actually, the first step is publish it on PyPI, the second is to get a fair number of happy users there. The bar for getting something included into the stdlib is pretty high -- you need to demonstrate that there is a need *and* that having it as a 3rd party module is a problem. And that once it's in, (a) it will be stable, and (b) someone who cares about it and knows the code thoroughly is available maintain it for years. The "why not a third party module?" bar also got a fair bit higher with Python 3.4 - by bundling pip, we have deliberately made third party modules easier to consume, thus weakening the convenience argument that applies to stdlib inclusion. The Python Packaging Authority are also working to reduce the barriers to distribution and consumption of C extensions, which will again weaken the argument for stdlib inclusion of third party C extensions. The metadata 2.0 efforts are also designed to streamline the process of conversion to platform specific formats like RPM. It's not that I don't see a sorted tree as valuable - I do. I just also see it as a fairly specialised type. The main case I could see being made for inclusion is if it made a major difference to the implementation of something else that was already in the stdlib. I believe that's where most other additions to the collections and types modules have come from in recent releases - we wanted to *use* them, and it seemed better to do the work to design an appropriate public API rather than keep them private (enum and generic function support arrived the same way, although we haven't actually redesigned pprint to rely on the latter yet). Cheers, Nick. > > > On Wed, Mar 26, 2014 at 1:53 PM, Ethan Furman wrote: >> >> On 03/26/2014 01:31 PM, Marko Rauhamaa wrote: >>> >>> >>> I have made a full implementation of a balanced tree and would like to >>> know what the process is to have it considered for inclusion in Python >>> 3. >> >> >> Open a ticket on the tracker [1], post your code to that ticket, sign the CLA [2], answer questions, etc., that come up in the code review. >> >> I believe Raymond Hettinger is the Guardian of the collections module; add him as nosy. >> >> -- >> ~Ethan~ >> >> [1] http://http://bugs.python.org >> [2] https://www.python.org/psf/contrib/contrib-form >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko at pacujo.net Wed Mar 26 22:41:42 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Wed, 26 Mar 2014 23:41:42 +0200 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <20140326222628.24336818@fsol> (Antoine Pitrou's message of "Wed, 26 Mar 2014 22:26:28 +0100") References: <8738i4og5f.fsf@elektro.pacujo.net> <20140326222628.24336818@fsol> Message-ID: <87ppl8mycp.fsf@elektro.pacujo.net> Antoine Pitrou : > Wouldn't a heapq work as well for those two? In my experience, networking entities typically start a timer at each interaction and cancel the pending one. So you have numerous timers that virtually never expire. You might have 100 interactions per second, each canceling and restarting a 10-minute timer. I don't know first hand if that causes heap queues to cause measurable heap or CPU pressure. Marko From solipsis at pitrou.net Wed Mar 26 22:46:58 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 Mar 2014 22:46:58 +0100 Subject: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"] References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> <20140326172512.22e1f21a@anarchist.wooz.org> <1395869274.5751.99327085.56513D6F@webmail.messagingengine.com> <20140326173030.2ee5dc11@anarchist.wooz.org> <447D6458-D0EB-45F4-88F3-65A771B530A5@stufft.io> Message-ID: <20140326224658.52a7d8a8@fsol> On Wed, 26 Mar 2014 17:37:40 -0400 Donald Stufft wrote: > > On Mar 26, 2014, at 5:30 PM, Barry Warsaw wrote: > > > On Mar 26, 2014, at 02:27 PM, Benjamin Peterson wrote: > > > >> I would have said that, too, several years ago, but I think we've been > >> requiring (or using anyway) PEPs for a lot more things now. OrderedDict > >> had a PEP for example. > >> > >> I'm not sure if that's a good thing or not. > > > > Hmm, me neither! I guess if someone *wants* to go through the PEP gauntlet, I > > won't stop them. It builds character. > > > > -Barry > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io > > Is that what it?s called? ?character? >:] Preferably unicode. Regards Antoine. From solipsis at pitrou.net Wed Mar 26 22:48:48 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 Mar 2014 22:48:48 +0100 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <87ppl8mycp.fsf@elektro.pacujo.net> References: <8738i4og5f.fsf@elektro.pacujo.net> <20140326222628.24336818@fsol> <87ppl8mycp.fsf@elektro.pacujo.net> Message-ID: <20140326224848.1e457671@fsol> On Wed, 26 Mar 2014 23:41:42 +0200 Marko Rauhamaa wrote: > Antoine Pitrou : > > > Wouldn't a heapq work as well for those two? > > In my experience, networking entities typically start a timer at each > interaction and cancel the pending one. So you have numerous timers that > virtually never expire. You might have 100 interactions per second, each > canceling and restarting a 10-minute timer. > > I don't know first hand if that causes heap queues to cause measurable > heap or CPU pressure. Each individual heapq operation (push or pop) will be O(log n). That's not different from a balanced search tree (although of course the constant multiplier may vary). Regards Antoine. From marko at pacujo.net Wed Mar 26 22:51:13 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Wed, 26 Mar 2014 23:51:13 +0200 Subject: [Python-Dev] collections.sortedtree In-Reply-To: (Terry Reedy's message of "Wed, 26 Mar 2014 17:37:58 -0400") References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> Message-ID: <87ha6kmxwu.fsf@elektro.pacujo.net> Terry Reedy : > Perhaps the collections doc should mention that there are other > specializes container types available on PyPI and either list some or > point to a wiki page listing some. There must be at least 10 that > could be included in such a list. There *is* a relatively high threshold of importing C extensions from an external source. If I build an application making use of them and advise coworkers to use it, they would likely balk at having to compile them. Not all machines have a development toolkit. Furthermore: # which pip /usr/bin/which: no pip in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:\ /usr/sbin:/usr/bin:/root/bin) # yum install pip No package pip available. Marko From marko at pacujo.net Wed Mar 26 22:57:27 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Wed, 26 Mar 2014 23:57:27 +0200 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <20140326224848.1e457671@fsol> (Antoine Pitrou's message of "Wed, 26 Mar 2014 22:48:48 +0100") References: <8738i4og5f.fsf@elektro.pacujo.net> <20140326222628.24336818@fsol> <87ppl8mycp.fsf@elektro.pacujo.net> <20140326224848.1e457671@fsol> Message-ID: <87a9ccmxmg.fsf@elektro.pacujo.net> Antoine Pitrou : > Marko Rauhamaa wrote: >> In my experience, networking entities typically start a timer at each >> interaction and cancel the pending one. So you have numerous timers >> that virtually never expire. You might have 100 interactions per >> second, each canceling and restarting a 10-minute timer. > > Each individual heapq operation (push or pop) will be O(log n). That's > not different from a balanced search tree (although of course the > constant multiplier may vary). Yes, but if I have 1000 connections with one active timer each. The size of my sorted tree is 1000 timer objects. There are typically no expiries to react to. If the canceled timer lingers in the heapq till its expiry (in 10 minutes), the size is 100 * 10 * 60 = 60,000. The CPU has to wake up constantly to clear the expired timers. In practice, none of that might matter. Marko From solipsis at pitrou.net Wed Mar 26 23:02:47 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 Mar 2014 23:02:47 +0100 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <87a9ccmxmg.fsf@elektro.pacujo.net> References: <8738i4og5f.fsf@elektro.pacujo.net> <20140326222628.24336818@fsol> <87ppl8mycp.fsf@elektro.pacujo.net> <20140326224848.1e457671@fsol> <87a9ccmxmg.fsf@elektro.pacujo.net> Message-ID: <20140326230247.084e422d@fsol> On Wed, 26 Mar 2014 23:57:27 +0200 Marko Rauhamaa wrote: > Antoine Pitrou : > > > Marko Rauhamaa wrote: > >> In my experience, networking entities typically start a timer at each > >> interaction and cancel the pending one. So you have numerous timers > >> that virtually never expire. You might have 100 interactions per > >> second, each canceling and restarting a 10-minute timer. > > > > Each individual heapq operation (push or pop) will be O(log n). That's > > not different from a balanced search tree (although of course the > > constant multiplier may vary). > > Yes, but if I have 1000 connections with one active timer each. The size > of my sorted tree is 1000 timer objects. There are typically no expiries > to react to. > > If the canceled timer lingers in the heapq till its expiry (in 10 > minutes), the size is 100 * 10 * 60 = 60,000. The CPU has to wake up > constantly to clear the expired timers. Ideally, I think you should be able to replace the cancelled item with the last item in the heap and then fix the heap in logarithmic time, but the heapq API doesn't seem to provide a way to do this. Regards Antoine. From ben at bendarnell.com Wed Mar 26 23:14:19 2014 From: ben at bendarnell.com (Ben Darnell) Date: Wed, 26 Mar 2014 18:14:19 -0400 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <20140326230247.084e422d@fsol> References: <8738i4og5f.fsf@elektro.pacujo.net> <20140326222628.24336818@fsol> <87ppl8mycp.fsf@elektro.pacujo.net> <20140326224848.1e457671@fsol> <87a9ccmxmg.fsf@elektro.pacujo.net> <20140326230247.084e422d@fsol> Message-ID: On Wed, Mar 26, 2014 at 6:02 PM, Antoine Pitrou wrote: > On Wed, 26 Mar 2014 23:57:27 +0200 > Marko Rauhamaa wrote: > > Antoine Pitrou : > > > > > Marko Rauhamaa wrote: > > >> In my experience, networking entities typically start a timer at each > > >> interaction and cancel the pending one. So you have numerous timers > > >> that virtually never expire. You might have 100 interactions per > > >> second, each canceling and restarting a 10-minute timer. > > > > > > Each individual heapq operation (push or pop) will be O(log n). That's > > > not different from a balanced search tree (although of course the > > > constant multiplier may vary). > > > > Yes, but if I have 1000 connections with one active timer each. The size > > of my sorted tree is 1000 timer objects. There are typically no expiries > > to react to. > > > > If the canceled timer lingers in the heapq till its expiry (in 10 > > minutes), the size is 100 * 10 * 60 = 60,000. The CPU has to wake up > > constantly to clear the expired timers. > > Ideally, I think you should be able to replace the cancelled item with > the last item in the heap and then fix the heap in logarithmic time, > but the heapq API doesn't seem to provide a way to do this. > Heaps provide easy access to the first item, but you can't find the last element without scanning the whole thing. With a heap, I think the best approach to the timer-cancellation problem is to periodically rebuild the heap from the non-cancelled items (Tornado keeps a count of cancellations and rebuilds the heap when the number of cancelled timeouts is half of the total. If cancellations are infrequent then they're just left in the heap until their time comes due). -Ben > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ben%40bendarnell.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Mar 26 23:16:03 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Mar 2014 15:16:03 -0700 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <20140326230247.084e422d@fsol> References: <8738i4og5f.fsf@elektro.pacujo.net> <20140326222628.24336818@fsol> <87ppl8mycp.fsf@elektro.pacujo.net> <20140326224848.1e457671@fsol> <87a9ccmxmg.fsf@elektro.pacujo.net> <20140326230247.084e422d@fsol> Message-ID: Yeah, so the pyftp fix is to keep track of how many timers were cancelled, and if the number exceeds a threshold it just recreates the heap, something like heap = [x for x in heap if not x.cancelled] heapify(heap) On Wed, Mar 26, 2014 at 3:02 PM, Antoine Pitrou wrote: > On Wed, 26 Mar 2014 23:57:27 +0200 > Marko Rauhamaa wrote: > > Antoine Pitrou : > > > > > Marko Rauhamaa wrote: > > >> In my experience, networking entities typically start a timer at each > > >> interaction and cancel the pending one. So you have numerous timers > > >> that virtually never expire. You might have 100 interactions per > > >> second, each canceling and restarting a 10-minute timer. > > > > > > Each individual heapq operation (push or pop) will be O(log n). That's > > > not different from a balanced search tree (although of course the > > > constant multiplier may vary). > > > > Yes, but if I have 1000 connections with one active timer each. The size > > of my sorted tree is 1000 timer objects. There are typically no expiries > > to react to. > > > > If the canceled timer lingers in the heapq till its expiry (in 10 > > minutes), the size is 100 * 10 * 60 = 60,000. The CPU has to wake up > > constantly to clear the expired timers. > > Ideally, I think you should be able to replace the cancelled item with > the last item in the heap and then fix the heap in logarithmic time, > but the heapq API doesn't seem to provide a way to do this. > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Mar 26 23:18:36 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 Mar 2014 23:18:36 +0100 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <20140326222628.24336818@fsol> <87ppl8mycp.fsf@elektro.pacujo.net> <20140326224848.1e457671@fsol> <87a9ccmxmg.fsf@elektro.pacujo.net> <20140326230247.084e422d@fsol> Message-ID: <20140326231836.13e40e07@fsol> On Wed, 26 Mar 2014 18:14:19 -0400 Ben Darnell wrote: > > > > > > If the canceled timer lingers in the heapq till its expiry (in 10 > > > minutes), the size is 100 * 10 * 60 = 60,000. The CPU has to wake up > > > constantly to clear the expired timers. > > > > Ideally, I think you should be able to replace the cancelled item with > > the last item in the heap and then fix the heap in logarithmic time, > > but the heapq API doesn't seem to provide a way to do this. > > Heaps provide easy access to the first item, but you can't find the last > element without scanning the whole thing. I was talking about the last element, not the largest. The point of replacing with the last element is that shrinking is O(1). Regards Antoine. From ethan at stoneleaf.us Wed Mar 26 22:53:13 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 26 Mar 2014 14:53:13 -0700 Subject: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"] In-Reply-To: <20140326224658.52a7d8a8@fsol> References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> <20140326172512.22e1f21a@anarchist.wooz.org> <1395869274.5751.99327085.56513D6F@webmail.messagingengine.com> <20140326173030.2ee5dc11@anarchist.wooz.org> <447D6458-D0EB-45F4-88F3-65A771B530A5@stufft.io> <20140326224658.52a7d8a8@fsol> Message-ID: <53334C49.8080000@stoneleaf.us> On 03/26/2014 02:46 PM, Antoine Pitrou wrote: > On Wed, 26 Mar 2014 17:37:40 -0400 > Donald Stufft wrote: >> On Mar 26, 2014, at 5:30 PM, Barry Warsaw wrote: >>> I guess if someone *wants* to go through the PEP gauntlet, I >>> won't stop them. It builds character. >> >> Is that what it?s called? ?character? >:] > > Preferably unicode. Indeed, a lot more swear symbols available in Unicode! ;) -- ~Ethan~ From drsalists at gmail.com Wed Mar 26 23:47:31 2014 From: drsalists at gmail.com (Dan Stromberg) Date: Wed, 26 Mar 2014 15:47:31 -0700 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> Message-ID: On Wed, Mar 26, 2014 at 1:55 PM, Benjamin Peterson wrote: > On Wed, Mar 26, 2014, at 13:31, Marko Rauhamaa wrote: >> >> I have made a full implementation of a balanced tree and would like to >> know what the process is to have it considered for inclusion in Python >> 3. > > It's not a bad idea. (I believe others have proposed an red-black tree.) > Certainly, it requires a PEP and a few months of bikesheding, though. I'd recommend a plain binary tree (for random keys), red-black tree and treap (both for ordered or mostly-ordered keys, where red-black tree gives low operation duration standard deviation, and treap gives low average operation duration). https://en.wikipedia.org/wiki/Binary_search_tree#Performance_comparisons http://stromberg.dnsalias.org/~strombrg/python-tree-and-heap-comparison/ It'd likely make sense to have either a pure python implementation, or pure python and C-extended, so that Pypy and Jython can share the feature with CPython. From drsalists at gmail.com Thu Mar 27 00:02:06 2014 From: drsalists at gmail.com (Dan Stromberg) Date: Wed, 26 Mar 2014 16:02:06 -0700 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <87wqfginck.fsf@elektro.pacujo.net> References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> <87wqfginck.fsf@elektro.pacujo.net> Message-ID: On Wed, Mar 26, 2014 at 3:52 PM, Marko Rauhamaa wrote: > Dan Stromberg : > >> It'd likely make sense to have either a pure python implementation, or >> pure python and C-extended, so that Pypy and Jython can share the >> feature with CPython. > > Jython can build directly on Java's native SortedMap implementation. The > API should not tie it to a tree. Optimizations and refactorings should > be allowed. Only O(log N) worst-case behavior should be mandated. Rare worst cases should be fine for a good reason - see python dict's. From marko at pacujo.net Wed Mar 26 23:52:59 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Thu, 27 Mar 2014 00:52:59 +0200 Subject: [Python-Dev] collections.sortedtree In-Reply-To: (Dan Stromberg's message of "Wed, 26 Mar 2014 15:47:31 -0700") References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> Message-ID: <87wqfginck.fsf@elektro.pacujo.net> Dan Stromberg : > It'd likely make sense to have either a pure python implementation, or > pure python and C-extended, so that Pypy and Jython can share the > feature with CPython. Jython can build directly on Java's native SortedMap implementation. The API should not tie it to a tree. Optimizations and refactorings should be allowed. Only O(log N) worst-case behavior should be mandated. (And now I notice I named this thread wrong; I named my thingy collections.sorteddict.) Marko From ethan at stoneleaf.us Wed Mar 26 23:47:05 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 26 Mar 2014 15:47:05 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <5332E5C8.1010506@stoneleaf.us> Message-ID: <533358E9.3060107@stoneleaf.us> On 03/26/2014 02:41 PM, Victor Stinner wrote: > 2014-03-26 15:35 GMT+01:00 Ethan Furman : >> ------------------------------------------------------------------------------- >> Examples:: >> >> >>> b'%a' % 3.14 >> b'3.14' >> >> >>> b'%a' % b'abc' >> b'abc' > > This one is wrong: > >>>> repr(b'abc').encode('ascii', 'backslashreplace') > b"b'abc'" Fixed, thanks. -- ~Ethan~ From stutzbach at google.com Thu Mar 27 01:15:21 2014 From: stutzbach at google.com (Daniel Stutzbach) Date: Wed, 26 Mar 2014 17:15:21 -0700 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <8738i4og5f.fsf@elektro.pacujo.net> References: <8738i4og5f.fsf@elektro.pacujo.net> Message-ID: On Wed, Mar 26, 2014 at 1:31 PM, Marko Rauhamaa wrote: > sorteddict.keys(reversed=False, start=unspecified, inclusive=True) > FYI - there are already several packages on PyPi that provide a sorteddict: https://pypi.python.org/pypi?%3Aaction=search&term=sorteddict&submit=search -- Daniel Stutzbach (I wrote one of them) -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Thu Mar 27 03:11:49 2014 From: eliben at gmail.com (Eli Bendersky) Date: Wed, 26 Mar 2014 19:11:49 -0700 Subject: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"] In-Reply-To: <1395869274.5751.99327085.56513D6F@webmail.messagingengine.com> References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> <20140326172512.22e1f21a@anarchist.wooz.org> <1395869274.5751.99327085.56513D6F@webmail.messagingengine.com> Message-ID: On Wed, Mar 26, 2014 at 2:27 PM, Benjamin Peterson wrote: > > > On Wed, Mar 26, 2014, at 14:25, Barry Warsaw wrote: > > On Mar 26, 2014, at 01:55 PM, Benjamin Peterson wrote: > > > > >It's not a bad idea. (I believe others have proposed an red-black tree.) > > >Certainly, it requires a PEP and a few months of bikesheding, though. > > > > Generally, PEPs aren't necessary for simple or relatively uncontroversial > > additions to existing modules or the stdlib. > > I would have said that, too, several years ago, but I think we've been > requiring (or using anyway) PEPs for a lot more things now. OrderedDict > had a PEP for example. > This is probably a natural outcome of the rising popularity of Python in the last few years. Much more users, more core developers, more at stake... > > I'm not sure if that's a good thing or not. > YMMV but IMHO this is a good thing. PEPs provide a single point of reference to a discussion that would otherwise be spread over multiple centi-threads (not that PEPs don't create centi-threads, but they outlive them in a way). Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Mar 27 03:27:51 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 26 Mar 2014 19:27:51 -0700 Subject: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"] In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> <20140326172512.22e1f21a@anarchist.wooz.org> <1395869274.5751.99327085.56513D6F@webmail.messagingengine.com> Message-ID: <53338CA7.6000604@stoneleaf.us> On 03/26/2014 07:11 PM, Eli Bendersky wrote: > On Wed, Mar 26, 2014 at 2:27 PM, Benjamin Peterson wrote: >> >> I'm not sure if that's a good thing or not. > > YMMV but IMHO this is a good thing. PEPs provide a single point of reference to a discussion that would otherwise be > spread over multiple centi-threads (not that PEPs don't create centi-threads, but they outlive them in a way). Plus the PEP can help prevent multiple mega-threads as the same idea is revisited again and again and again... -- ~Ethan~ From stephen at xemacs.org Thu Mar 27 09:02:36 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 27 Mar 2014 17:02:36 +0900 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: References: <20140325233355.14cba72c@fsol> Message-ID: <8761n0qdb7.fsf@uwakimon.sk.tsukuba.ac.jp> Alex Gaynor writes: > Here's my proposed list of such featuers: And suppose that list grows over time? After all, it once was []. If we go for a feature-by-feature list, that has two more-or-less hidden costs. (1) Python-Dev has to specify which ones, and either risks a new specification debate in the future, or needs to spend time now describing criteria and processes for extending the list. (2) Users may need to worry about the list. (OTOH, as long as the list is restricted to features in certain modules, users can choose to assume anything in those modules may have changed behavior and that's no different from Nick's proposal for them.) From stephen at xemacs.org Thu Mar 27 09:11:42 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 27 Mar 2014 17:11:42 +0900 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> Message-ID: <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > On 27 Mar 2014 07:02, "Guido van Rossum" wrote: >> Actually, the first step is publish it on PyPI, the second is to >> get a fair number of happy users there. The bar for getting something >> included into the stdlib is pretty high > The "why not a third party module?" bar also got a fair bit higher > with Python 3.4 - by bundling pip, we have deliberately made third > party modules easier to consume, thus weakening the convenience > argument that applies to stdlib inclusion. Maybe. That depends on if you care about the convenience of folks who have to get new modules past Corporate Security, but it's easier to get an upgrade of the whole shebang. I don't think it's ever really been resolved whether they're a "typical case that won't go away" or a special group whose special needs should be considered. Steve From fijall at gmail.com Thu Mar 27 09:16:08 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 27 Mar 2014 10:16:08 +0200 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Thu, Mar 27, 2014 at 10:11 AM, Stephen J. Turnbull wrote: > Nick Coghlan writes: > > > On 27 Mar 2014 07:02, "Guido van Rossum" wrote: > >> Actually, the first step is publish it on PyPI, the second is to > >> get a fair number of happy users there. The bar for getting something > >> included into the stdlib is pretty high > > > The "why not a third party module?" bar also got a fair bit higher > > with Python 3.4 - by bundling pip, we have deliberately made third > > party modules easier to consume, thus weakening the convenience > > argument that applies to stdlib inclusion. > > Maybe. That depends on if you care about the convenience of folks who > have to get new modules past Corporate Security, but it's easier to > get an upgrade of the whole shebang. I don't think it's ever really > been resolved whether they're a "typical case that won't go away" or a > special group whose special needs should be considered. > > Steve And random pieces of C included in the standard library can be shuffled under the carpet under the disguise of upgrade or what are you suggesting? From stephen at xemacs.org Thu Mar 27 09:19:32 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 27 Mar 2014 17:19:32 +0900 Subject: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"] In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> <20140326172512.22e1f21a@anarchist.wooz.org> <1395869274.5751.99327085.56513D6F@webmail.messagingengine.com> Message-ID: <8738i4qciz.fsf@uwakimon.sk.tsukuba.ac.jp> Eli Bendersky writes: > On Wed, Mar 26, 2014 at 2:27 PM, Benjamin Peterson wrote: >> I would have said that, too, several years ago, but I think we've >> been requiring (or using anyway) PEPs for a lot more things now. > YMMV but IMHO this is a good thing. FWIW I was just talking to Matz yesterday, and asked him about it. He said he thought PEPs worked great for Python (in the context of explain that his (and Ruby's) style is different). Just-ask-the-nearest-independent-opinion-ly y'rs, From p.f.moore at gmail.com Thu Mar 27 10:07:27 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 27 Mar 2014 09:07:27 +0000 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 27 March 2014 08:16, Maciej Fijalkowski wrote: > And random pieces of C included in the standard library can be > shuffled under the carpet under the disguise of upgrade or what are > you suggesting? The sort of thing that happens is that the relevant approvers will accept python-dev as a "trusted supplier" and then Python upgrades are acceptable subject to review of the changes, etc. For a new module, there is a whole other level of questions around how do we trust the person who developed the code, do we need to do a full code review, etc? It's a bit unfair to describe the process as "random pieces of C" being "shuffled under the carpet". (Although there probably are environments where that is uncomfortably close to the truth :-() Paul From fijall at gmail.com Thu Mar 27 10:10:52 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 27 Mar 2014 11:10:52 +0200 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Thu, Mar 27, 2014 at 11:07 AM, Paul Moore wrote: > On 27 March 2014 08:16, Maciej Fijalkowski wrote: >> And random pieces of C included in the standard library can be >> shuffled under the carpet under the disguise of upgrade or what are >> you suggesting? > > The sort of thing that happens is that the relevant approvers will > accept python-dev as a "trusted supplier" and then Python upgrades are > acceptable subject to review of the changes, etc. For a new module, > there is a whole other level of questions around how do we trust the > person who developed the code, do we need to do a full code review, > etc? > > It's a bit unfair to describe the process as "random pieces of C" > being "shuffled under the carpet". (Although there probably are > environments where that is uncomfortably close to the truth :-() > > Paul I just find "my company is stupid so let's work around it by putting stuff to python standard library" unacceptable argument for python-dev and all the python community. From kristjan at ccpgames.com Thu Mar 27 10:25:05 2014 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 27 Mar 2014 09:25:05 +0000 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <20140326222628.24336818@fsol> <87ppl8mycp.fsf@elektro.pacujo.net> <20140326224848.1e457671@fsol> <87a9ccmxmg.fsf@elektro.pacujo.net> <20140326230247.084e422d@fsol> Message-ID: True. I've long since added a heapdel() to our local fork. a heappop(idx=0) extension would do the same I can provide a patch if there is interest. K Ideally, I think you should be able to replace the cancelled item with the last item in the heap and then fix the heap in logarithmic time, but the heapq API doesn't seem to provide a way to do this. Regards Antoine. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Mar 27 10:32:03 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Mar 2014 19:32:03 +1000 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: <8761n0qdb7.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20140325233355.14cba72c@fsol> <8761n0qdb7.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 27 March 2014 18:02, Stephen J. Turnbull wrote: > Alex Gaynor writes: > > > Here's my proposed list of such featuers: > > And suppose that list grows over time? After all, it once was []. > > If we go for a feature-by-feature list, that has two more-or-less > hidden costs. (1) Python-Dev has to specify which ones, and either > risks a new specification debate in the future, or needs to spend > time now describing criteria and processes for extending the list. It's not a hidden cost - it's a deliberately chosen one. Guido was wary of an open-ended agreement, so by enumerating the precise set of missing features in Python 2.7 that are causing concern for the network security folks, we get to address the immediate problem, without granting permission to backport further arbitrary features without additional discussion. > (2) Users may need to worry about the list. (OTOH, as long as the > list is restricted to features in certain modules, users can choose to > assume anything in those modules may have changed behavior and that's > no different from Nick's proposal for them.) The PEP already specifically advises that cross-version compatible code use feature detection rather than version checks. For network security, it's recommended to avoid using the low level modules directly, anyway - it's much better to use a higher level library like requests, to reduce the number of places where we need to get the security design right. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Mar 27 10:43:39 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Mar 2014 19:43:39 +1000 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 27 March 2014 18:11, Stephen J. Turnbull wrote: > Nick Coghlan writes: > > > On 27 Mar 2014 07:02, "Guido van Rossum" wrote: > >> Actually, the first step is publish it on PyPI, the second is to > >> get a fair number of happy users there. The bar for getting something > >> included into the stdlib is pretty high > > > The "why not a third party module?" bar also got a fair bit higher > > with Python 3.4 - by bundling pip, we have deliberately made third > > party modules easier to consume, thus weakening the convenience > > argument that applies to stdlib inclusion. > > Maybe. That depends on if you care about the convenience of folks who > have to get new modules past Corporate Security, but it's easier to > get an upgrade of the whole shebang. I don't think it's ever really > been resolved whether they're a "typical case that won't go away" or a > special group whose special needs should be considered. I'm all too well aware of this particular problem (although thankfully don't have to deal with it myself any more), and it's a key part of why I said "a fair bit higher" rather than "insurmountable" :) Such environments *do* usually have procedures to get additional open source software approved, it's just a pain. That pain then becomes another factor to take into account deciding between using a simpler solution, rolling your own custom solution, or tackling your organisation's module approval process. That said, getting approval is definitely easier when the request is to trust "the Python Software Foundation" vs "J. Random Programmer's account on GitHub", so yes, it still counts in favour of sufficiently compelling stdlib additions. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Mar 27 10:58:35 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Mar 2014 19:58:35 +1000 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 27 March 2014 19:10, Maciej Fijalkowski wrote: > On Thu, Mar 27, 2014 at 11:07 AM, Paul Moore wrote: >> On 27 March 2014 08:16, Maciej Fijalkowski wrote: >>> And random pieces of C included in the standard library can be >>> shuffled under the carpet under the disguise of upgrade or what are >>> you suggesting? >> >> The sort of thing that happens is that the relevant approvers will >> accept python-dev as a "trusted supplier" and then Python upgrades are >> acceptable subject to review of the changes, etc. For a new module, >> there is a whole other level of questions around how do we trust the >> person who developed the code, do we need to do a full code review, >> etc? >> >> It's a bit unfair to describe the process as "random pieces of C" >> being "shuffled under the carpet". (Although there probably are >> environments where that is uncomfortably close to the truth :-() >> >> Paul > > I just find "my company is stupid so let's work around it by putting > stuff to python standard library" unacceptable argument for python-dev > and all the python community. Due diligence and prudent risk management are not stupid - most open source projects and small companies just don't have the luxury of worrying about them, as they're so far down the list of concerns that the additional risk of using arbitrary code downloaded off the internet doesn't even register. As organisations get larger, they have more to lose, so they typically start worrying about that kind of thing. Properly vetting software for licensing and potential security issues is expensive though, so they usually prefer to outsource that task to trusted providers (this is one of the key concepts that gets me paid). Contractual arrangements and brand reputations start to replace blind trust in upstream developers. Is this less efficient than full open collaboration? Yes, it is - the "verify" part of "trust, but verify" comes at a real cost in time and money. However, there are plenty of good reasons that phrase is "trust, but verify" rather than "trust unconditionally". You may choose not to care about those more cautious users, and that's fine - but they're still worth taking into account when making design decisions if you want to cross the marketing chasm from "early adopters" to everyone else. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Mar 27 11:32:02 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Mar 2014 20:32:02 +1000 Subject: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"] In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> <20140326172512.22e1f21a@anarchist.wooz.org> <1395869274.5751.99327085.56513D6F@webmail.messagingengine.com> Message-ID: On 27 March 2014 12:11, Eli Bendersky wrote: > On Wed, Mar 26, 2014 at 2:27 PM, Benjamin Peterson > wrote: >> I'm not sure if that's a good thing or not. > > YMMV but IMHO this is a good thing. PEPs provide a single point of reference > to a discussion that would otherwise be spread over multiple centi-threads > (not that PEPs don't create centi-threads, but they outlive them in a way). >From my point of view, the primary purpose of the PEP process is to provide a way for us to finally say "yes" to controversial proposals that have valid arguments against them. When things are obviously good ideas that don't impose a big maintenance burden, nobody really objects if we skip the PEP process (that isn't always a good thing - directory and zipfile execution flew under the radar for years because it was such a neat idea that Guido approved it directly in the issue, and then we forgot to mention it in the 2.6 What's New). Some ideas aren't obviously good, or a suitable API isn't obvious, or they impose a major additional maintenance burden, or they require a change to our development policies. In those cases, the PEP process allows us to collectively ask the question "Is this worth the hassle?". Cases like the restoration of binary interpolation support, or my proposal to backport network security features, also showcase how the PEP process can be used to refine the *question* so the PEP champion is forced to figure out what they *really* want, and propose a solution that clearly solves that specific problem, rather than overreaching and asking for more than is needed. (This is also reflected in the relative fates of the current matrix multiplication proposal and previous more general proposals) With the introduction of the BDFL-Delegate system, and then the decision last year to give the "Discussions-To" header a bit more force and allow groups like the Python Packaging Authority to make use of the PEP process independently of python-dev, the PEP process is also becoming more streamlined, making it more effective in its role as a tool for establishing consensus - there's less need to convince someone to drop a veto without a PEP, as the PEP process itself is getting less painful. Most of the time when I hear people say "the PEP process is too difficult", I eventually find that what they really mean is "learning the kinds of things that python-dev are likely to be worried about, and ensuring that the PEP adequately addresses their concerns, and listening to feedback, and reconsidering what I actually want, and revising my proposal, such that they eventually say yes is too time consuming". Helping people to learn exactly how to navigate that process is actually one of the main roles of python-ideas these days, although we don't do a good job (at all) of advertising that fact. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Thu Mar 27 11:38:28 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 27 Mar 2014 21:38:28 +1100 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Thu, Mar 27, 2014 at 8:58 PM, Nick Coghlan wrote: >On 27 March 2014 19:10, Maciej Fijalkowski wrote: >> I just find "my company is stupid so let's work around it by putting >> stuff to python standard library" unacceptable argument for python-dev >> and all the python community. > > Due diligence and prudent risk management are not stupid - most open > source projects and small companies just don't have the luxury of > worrying about them, as they're so far down the list of concerns that > the additional risk of using arbitrary code downloaded off the > internet doesn't even register. I don't think anyone's saying it's stupid to be cautious, but more that it's stupid to blindly accept the latest python.org release and *not* accept something from another source. And if that's stupid, well, I'm stupid too - blindly accepting a whole lot of binary package updates because they're on ftp.au.debian.org, for instance. Why do I trust that, and not random sites on the internet? Because I trust that the Debian package maintainers to check what goes through, and I trust that there are people with reputations at stake, who won't want to send something dodgy through. It's not perfect, but it's a whole lot easier than checking every single package that goes through. ChrisA From victor.stinner at gmail.com Thu Mar 27 11:47:56 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 27 Mar 2014 11:47:56 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <533358E9.3060107@stoneleaf.us> References: <53320517.3010101@stoneleaf.us> <5332E5C8.1010506@stoneleaf.us> <533358E9.3060107@stoneleaf.us> Message-ID: The PEP 461 looks good to me. It's a nice addition to Python 3.5 and the PEP is well defined. I can help to implement it. Maybe, it would be nice to provide an implementation as a third-party party module on PyPI for Python 2.6-3.4. Note: I fixed a typo in your PEP (reST syntax). Victor 2014-03-26 23:47 GMT+01:00 Ethan Furman : >> This one is wrong: >> >>>>> repr(b'abc').encode('ascii', 'backslashreplace') >> >> b"b'abc'" > > > Fixed, thanks. From ncoghlan at gmail.com Thu Mar 27 12:10:09 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Mar 2014 21:10:09 +1000 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 27 March 2014 20:38, Chris Angelico wrote: > On Thu, Mar 27, 2014 at 8:58 PM, Nick Coghlan wrote: >>On 27 March 2014 19:10, Maciej Fijalkowski wrote: >>> I just find "my company is stupid so let's work around it by putting >>> stuff to python standard library" unacceptable argument for python-dev >>> and all the python community. >> >> Due diligence and prudent risk management are not stupid - most open >> source projects and small companies just don't have the luxury of >> worrying about them, as they're so far down the list of concerns that >> the additional risk of using arbitrary code downloaded off the >> internet doesn't even register. > > I don't think anyone's saying it's stupid to be cautious, but more > that it's stupid to blindly accept the latest python.org release and > *not* accept something from another source. And if that's stupid, > well, I'm stupid too - blindly accepting a whole lot of binary package > updates because they're on ftp.au.debian.org, for instance. Why do I > trust that, and not random sites on the internet? Because I trust that > the Debian package maintainers to check what goes through, and I trust > that there are people with reputations at stake, who won't want to > send something dodgy through. It's not perfect, but it's a whole lot > easier than checking every single package that goes through. Right - trusting the PSF and the Debian package review process are reasonable trade-offs at a personal level. However, it highlights why this is still a benefit of bringing things into the standard library: a lot of the overhead in review and audit processes is incurred based on the *number of components to be reviewed*, rather than the amount of code they contain. Other aspects of the overhead are incurred *per organisation trusted*. Inclusion in the standard library means not only bringing something under the PSF's "trust umbrella", but also within the core Python component. While we usually use corporate environments as our example, it applies equally to Linux distros - they generally have Python packaged already, and will eventually inherit standard library improvements. For third party packages, it requires someone to do the work to get them installed. Extending our trust to include a new component isn't to be done lightly, but it *does* genuinely save work in the long run for a whole lot of other people whenever we choose to do so, and that is the point Stephen was making. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Thu Mar 27 12:14:17 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 27 Mar 2014 22:14:17 +1100 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Thu, Mar 27, 2014 at 10:10 PM, Nick Coghlan wrote: > Extending our trust to include a new component isn't to be done > lightly, but it *does* genuinely save work in the long run for a whole > lot of other people whenever we choose to do so, and that is the point > Stephen was making. Exactly so. Blessing a module as part of the stdlib is an important mark of trust, and one that is most definitely not stupid to respect. (In case my previous message was ambiguous, I am fully in favour of people trusting python.org in this way, and fully understanding of people NOT trusting other sources, or at least not automatically.) ChrisA From solipsis at pitrou.net Thu Mar 27 12:24:49 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 27 Mar 2014 12:24:49 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 References: <53320517.3010101@stoneleaf.us> Message-ID: <20140327122449.71a50ea2@fsol> On Tue, 25 Mar 2014 15:37:11 -0700 Ethan Furman wrote: > > ``%a`` will call ``ascii()`` on the interpolated value. This is intended > as a debugging aid, rather than something that should be used in production. > Non-ASCII values will be encoded to either ``\xnn`` or ``\unnnn`` > representation. Use cases include developing a new protocol and writing > landmarks into the stream; debugging data going into an existing protocol > to see if the problem is the protocol itself or bad data; a fall-back for a > serialization format; or even a rudimentary serialization format when > defining ``__bytes__`` would not be appropriate [8]. The "use cases" you are enumerating for "%a" are chimeric. Did you *actually* do those things in real life, or are you inventing them for the PEP? Regards Antoine. From ncoghlan at gmail.com Thu Mar 27 12:26:50 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Mar 2014 21:26:50 +1000 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <5332E5C8.1010506@stoneleaf.us> <533358E9.3060107@stoneleaf.us> Message-ID: On 27 March 2014 20:47, Victor Stinner wrote: > The PEP 461 looks good to me. It's a nice addition to Python 3.5 and > the PEP is well defined. +1 from me as well. One minor request is that I don't think the rationale for rejecting numbers from "%s" is incomplete - IIRC, the problem there is that the normal path for handling those is the coercion via str() and this proposal deliberately *doesn't* allow that path. That means supporting numbers would mean writing a lot of *additional* code, and that isn't needed since 2/3 compatible code can just be adjusted to use an appropriate numeric code. > Note: I fixed a typo in your PEP (reST syntax). I also committed a couple of markup tweaks, since it seemed easier to just fix them than explain what was broken. However, there are also two dead footnotes (4 & 5), which I have left alone - I'm not sure if the problem is a missing reference, or if the footnote can go away now. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Thu Mar 27 12:30:01 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 27 Mar 2014 12:30:01 +0100 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <20140326222628.24336818@fsol> <87ppl8mycp.fsf@elektro.pacujo.net> <20140326224848.1e457671@fsol> <87a9ccmxmg.fsf@elektro.pacujo.net> <20140326230247.084e422d@fsol> Message-ID: <20140327123001.3c1aea06@fsol> On Thu, 27 Mar 2014 09:25:05 +0000 Kristj?n Valur J?nsson wrote: > True. > I've long since added a heapdel() to our local fork. > a heappop(idx=0) extension would do the same > I can provide a patch if there is interest. I think either of them would be cool. I don't know it would be approved by Raymond or whoever else. Regards Antoine. From ncoghlan at gmail.com Thu Mar 27 12:42:14 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Mar 2014 21:42:14 +1000 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <20140327122449.71a50ea2@fsol> References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> Message-ID: On 27 March 2014 21:24, Antoine Pitrou wrote: > On Tue, 25 Mar 2014 15:37:11 -0700 > Ethan Furman wrote: >> >> ``%a`` will call ``ascii()`` on the interpolated value. This is intended >> as a debugging aid, rather than something that should be used in production. >> Non-ASCII values will be encoded to either ``\xnn`` or ``\unnnn`` >> representation. Use cases include developing a new protocol and writing >> landmarks into the stream; debugging data going into an existing protocol >> to see if the problem is the protocol itself or bad data; a fall-back for a >> serialization format; or even a rudimentary serialization format when >> defining ``__bytes__`` would not be appropriate [8]. > > The "use cases" you are enumerating for "%a" are chimeric. Did you > *actually* do those things in real life, or are you inventing them for > the PEP? I'm the one that raised the "discourage misuse of __bytes__" concern, so I'd like %a to stay in at least for that reason. %a is a perfectly well defined format code (albeit one you'd only be likely to use while messing about with serialisation protocols, as the PEP describes - for example, if a %b code was ending up producing wrong data, you might switch to %a temporarily to get a better idea of where the bad data was coming from), while using __bytes__ to make %s behave the way %a is defined in the PEP would just be wrong in most cases. I consider %a the preemptive PEP 308 of binary interpolation format codes - in the absence of %a, I'm certain that users would end up abusing __bytes__ and %s to get the same effect, just as they used the known bug magnet that was the and/or hack for a long time in the absence of PEP 308. I also seem to recall Guido saying he liked it, which flipped the discussion from "do we have a good rationale for including it?" to "do we have a good rationale for the BDFL to ignore his instincts?". However, it would be up to Guido to confirm that recollection, and if "Guido likes it" is part of the reason for inclusion of the %a code, the PEP should mention that explicitly. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Thu Mar 27 13:07:59 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 27 Mar 2014 14:07:59 +0200 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <20140326222628.24336818@fsol> <87ppl8mycp.fsf@elektro.pacujo.net> <20140326224848.1e457671@fsol> <87a9ccmxmg.fsf@elektro.pacujo.net> <20140326230247.084e422d@fsol> Message-ID: 27.03.14 00:16, Guido van Rossum ???????(??): > Yeah, so the pyftp fix is to keep track of how many timers were > cancelled, and if the number exceeds a threshold it just recreates the > heap, something like > > heap = [x for x in heap if not x.cancelled] > heapify(heap) See also http://bugs.python.org/issue13451 which proposes such approach for the sched module. From thomas at python.org Thu Mar 27 13:16:42 2014 From: thomas at python.org (Thomas Wouters) Date: Thu, 27 Mar 2014 13:16:42 +0100 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> Message-ID: On Wed, Mar 26, 2014 at 9:55 PM, Benjamin Peterson wrote: > On Wed, Mar 26, 2014, at 13:31, Marko Rauhamaa wrote: > > > > I have made a full implementation of a balanced tree and would like to > > know what the process is to have it considered for inclusion in Python > > 3. > > It's not a bad idea. (I believe others have proposed an red-black tree.) > Certainly, it requires a PEP and a few months of bikesheding, though. Not to mention discussion about whether it shouldn't just be an existing PyPI package, like http://pypi.python.org/pypi/blist, rather than a new implementation. -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Thu Mar 27 14:48:54 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 27 Mar 2014 09:48:54 -0400 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <20140327122449.71a50ea2@fsol> References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> Message-ID: <20140327134854.DCC8025005B@webabinitio.net> On Thu, 27 Mar 2014 12:24:49 +0100, Antoine Pitrou wrote: > On Tue, 25 Mar 2014 15:37:11 -0700 > Ethan Furman wrote: > > > > ``%a`` will call ``ascii()`` on the interpolated value. This is intended > > as a debugging aid, rather than something that should be used in production. > > Non-ASCII values will be encoded to either ``\xnn`` or ``\unnnn`` > > representation. Use cases include developing a new protocol and writing > > landmarks into the stream; debugging data going into an existing protocol > > to see if the problem is the protocol itself or bad data; a fall-back for a > > serialization format; or even a rudimentary serialization format when > > defining ``__bytes__`` would not be appropriate [8]. > > The "use cases" you are enumerating for "%a" are chimeric. Did you > *actually* do those things in real life, or are you inventing them for > the PEP? The use cases came from someone else (Jim Jewett?) so you should be asking him, not Ethan :) As for the "did you actually do those things in real life", I know I've done the "dump the repr into the data (protocol) stream to see what I've really got here" debug trick in the string context, so I have no doubt that I will want to do it in the bytes context as well. In fact, it is probably somewhat more likely in the bytes context, since I know I've been in situations with data exchange protocols where I couldn't get console output and setting up logging was much more painful than just dumping the debug data into into the data stream. Or where doing so made it much clearer what was going on than separate logging would. I've done the 'landmark' thing as well, in the string context; that can be very useful when doing incremental test driven development. (Granted, you could do that with __bytes__; you might well be writing a __bytes__ method anyway as the next step, but it *is* more overhead/boilerplate than just starting with %a...and it gets people used to reaching for __bytes__ for the "wrong" purpose, which is Nick's concern). In theory I can see using %a for serialization in certain limited contexts (I've done that with string repr in private utility scripts), but in practice I doubt that would happen in a binary context, since those are much more likely to be actually going over a "wire" of some sort (ie: places you really don't want to use eval even when it would work). So yeah, I think %a has *practical* utility. --David From marko at pacujo.net Thu Mar 27 15:02:17 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Thu, 27 Mar 2014 16:02:17 +0200 Subject: [Python-Dev] collections.sortedtree In-Reply-To: (Thomas Wouters's message of "Thu, 27 Mar 2014 13:16:42 +0100") References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> Message-ID: <87y4zvwxhy.fsf@elektro.pacujo.net> Thomas Wouters : > Not to mention discussion about whether it shouldn't just be an existing > PyPI package, like http://pypi.python.org/pypi/blist, rather than a new > implementation. I'm fine with any implementation as long as it is in the standard library. Marko From guido at python.org Thu Mar 27 16:37:34 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Mar 2014 08:37:34 -0700 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Surely you can show empathy and still explain why it's not that easy. On Mar 27, 2014 2:11 AM, "Maciej Fijalkowski" wrote: > On Thu, Mar 27, 2014 at 11:07 AM, Paul Moore wrote: > > On 27 March 2014 08:16, Maciej Fijalkowski wrote: > >> And random pieces of C included in the standard library can be > >> shuffled under the carpet under the disguise of upgrade or what are > >> you suggesting? > > > > The sort of thing that happens is that the relevant approvers will > > accept python-dev as a "trusted supplier" and then Python upgrades are > > acceptable subject to review of the changes, etc. For a new module, > > there is a whole other level of questions around how do we trust the > > person who developed the code, do we need to do a full code review, > > etc? > > > > It's a bit unfair to describe the process as "random pieces of C" > > being "shuffled under the carpet". (Although there probably are > > environments where that is uncomfortably close to the truth :-() > > > > Paul > > I just find "my company is stupid so let's work around it by putting > stuff to python standard library" unacceptable argument for python-dev > and all the python community. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stutzbach at google.com Thu Mar 27 16:50:01 2014 From: stutzbach at google.com (Daniel Stutzbach) Date: Thu, 27 Mar 2014 08:50:01 -0700 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <20140326230247.084e422d@fsol> References: <8738i4og5f.fsf@elektro.pacujo.net> <20140326222628.24336818@fsol> <87ppl8mycp.fsf@elektro.pacujo.net> <20140326224848.1e457671@fsol> <87a9ccmxmg.fsf@elektro.pacujo.net> <20140326230247.084e422d@fsol> Message-ID: On Wed, Mar 26, 2014 at 3:02 PM, Antoine Pitrou wrote: > > Ideally, I think you should be able to replace the cancelled item with > the last item in the heap and then fix the heap in logarithmic time, > but the heapq API doesn't seem to provide a way to do this. Due to way the heapq is implemented, it can't provide an efficient API for removing an arbitrary item. Swapping with the last element allows you to efficiently remove the item at a particular index, but you first need to find the current index of an item, which requires a O(n) scan. To provide efficient cancellation and removal, a heap implementation needs some way to efficiently answer "What is the current index of this item?". There are a couple of ways to achieve that, but they all require more storage than heapq's list-based approach. -- Daniel Stutzbach -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Mar 27 16:53:15 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 27 Mar 2014 16:53:15 +0100 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <20140326222628.24336818@fsol> <87ppl8mycp.fsf@elektro.pacujo.net> <20140326224848.1e457671@fsol> <87a9ccmxmg.fsf@elektro.pacujo.net> <20140326230247.084e422d@fsol> Message-ID: <20140327165315.41c8f894@fsol> On Thu, 27 Mar 2014 08:50:01 -0700 Daniel Stutzbach wrote: > > Due to way the heapq is implemented, it can't provide an efficient API for > removing an arbitrary item. Swapping with the last element allows you to > efficiently remove the item at a particular index, but you first need to > find the current index of an item, which requires a O(n) scan. > > To provide efficient cancellation and removal, a heap implementation needs > some way to efficiently answer "What is the current index of this item?". > There are a couple of ways to achieve that, but they all require more > storage than heapq's list-based approach. You are right. I was assuming the index is already known. Regards Antoine. From guido at python.org Thu Mar 27 16:54:02 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Mar 2014 08:54:02 -0700 Subject: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements In-Reply-To: References: <20140325233355.14cba72c@fsol> <8761n0qdb7.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: BTW, my intention was also to set a precedent for future additions to the list. "We did this before" is a great argument to help smooth the path, without bypassing deliberations altogether. On Mar 27, 2014 2:33 AM, "Nick Coghlan" wrote: > On 27 March 2014 18:02, Stephen J. Turnbull wrote: > > Alex Gaynor writes: > > > > > Here's my proposed list of such featuers: > > > > And suppose that list grows over time? After all, it once was []. > > > > If we go for a feature-by-feature list, that has two more-or-less > > hidden costs. (1) Python-Dev has to specify which ones, and either > > risks a new specification debate in the future, or needs to spend > > time now describing criteria and processes for extending the list. > > It's not a hidden cost - it's a deliberately chosen one. Guido was > wary of an open-ended agreement, so by enumerating the precise set of > missing features in Python 2.7 that are causing concern for the > network security folks, we get to address the immediate problem, > without granting permission to backport further arbitrary features > without additional discussion. > > > (2) Users may need to worry about the list. (OTOH, as long as the > > list is restricted to features in certain modules, users can choose to > > assume anything in those modules may have changed behavior and that's > > no different from Nick's proposal for them.) > > The PEP already specifically advises that cross-version compatible > code use feature detection rather than version checks. For network > security, it's recommended to avoid using the low level modules > directly, anyway - it's much better to use a higher level library like > requests, to reduce the number of places where we need to get the > security design right. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Mar 27 17:08:03 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 27 Mar 2014 09:08:03 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <20140327122449.71a50ea2@fsol> References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> Message-ID: <53344CE3.4060305@stoneleaf.us> On 03/27/2014 04:24 AM, Antoine Pitrou wrote: > On Tue, 25 Mar 2014 15:37:11 -0700 Ethan Furman wrote: >> >> ``%a`` will call ``ascii()`` on the interpolated value. This is intended >> as a debugging aid, rather than something that should be used in production. >> Non-ASCII values will be encoded to either ``\xnn`` or ``\unnnn`` >> representation. Use cases include developing a new protocol and writing >> landmarks into the stream; debugging data going into an existing protocol >> to see if the problem is the protocol itself or bad data; a fall-back for a >> serialization format; or even a rudimentary serialization format when >> defining ``__bytes__`` would not be appropriate [8]. > > The "use cases" you are enumerating for "%a" are chimeric. Cool word! Haven't seen it a long time. :) > Did you *actually* do those things in real life, or are you inventing them > for the PEP? The examples came from Jim Jewett, but I can easily see myself using them. -- ~Ethan~ From yorik.sar at gmail.com Thu Mar 27 17:19:12 2014 From: yorik.sar at gmail.com (Yuriy Taraday) Date: Thu, 27 Mar 2014 20:19:12 +0400 Subject: [Python-Dev] multiprocessing BaseManager's "serializer" argument Message-ID: Hello. Is there any reason BaseManager's "serializer" argument has never been documented? We're going to use BaseManager for simple secure local RPC and for the "secure" part we can't use pickle. Can we hope that argument won't go away? Should I create a bug and provide some patch for the docs to add mention of this argument? -- Kind regards, Yuriy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Thu Mar 27 17:42:11 2014 From: bcannon at gmail.com (Brett Cannon) Date: Thu, 27 Mar 2014 16:42:11 +0000 Subject: [Python-Dev] multiprocessing BaseManager's "serializer" argument References: Message-ID: Creating the bug is the best bet as then the reason can be discussed there. If there are no objections then you can create a patch to make sure it's tested and documented. On Thu Mar 27 2014 at 12:20:00 PM, Yuriy Taraday wrote: > Hello. > > Is there any reason BaseManager's "serializer" argument has never been > documented? > We're going to use BaseManager for simple secure local RPC and for the > "secure" part we can't use pickle. Can we hope that argument won't go away? > Should I create a bug and provide some patch for the docs to add mention of > this argument? > > -- > > Kind regards, Yuriy. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Mar 27 16:58:44 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 27 Mar 2014 08:58:44 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> Message-ID: <53344AB4.50904@stoneleaf.us> On 03/27/2014 04:42 AM, Nick Coghlan wrote: > > I also seem to recall Guido saying he liked it [%a], which flipped the > discussion from "do we have a good rationale for including it?" to "do > we have a good rationale for the BDFL to ignore his instincts?". > However, it would be up to Guido to confirm that recollection, and if > "Guido likes it" is part of the reason for inclusion of the %a code, > the PEP should mention that explicitly. I checked Guido's posts (Subject contains PEP 461, From contains guido) and did not see anything to that effect. -- ~Ethan~ From solipsis at pitrou.net Thu Mar 27 17:48:16 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 27 Mar 2014 17:48:16 +0100 Subject: [Python-Dev] cpython: inspect.Signature: Add 'Signature.from_callable' classmethod. Closes #17373 References: <3fvrmy6cvPz7Ll5@mail.python.org> Message-ID: <20140327174816.25e87c40@fsol> On Thu, 27 Mar 2014 17:12:02 +0100 (CET) yury.selivanov wrote: > > + .. classmethod:: Signature.from_callable(obj) > + > + Return a :class:`Signature` (or its subclass) object for a given callable > + ``obj``. This method simplifies subclassing of :class:`Signature`: > + > + :: > + > + class MySignature(Signature): > + pass > + sig = MySignature.from_callable(min) > + assert isinstance(sig, MySignature) > + This needs a "versionadded" tag. Regards Antoine. From solipsis at pitrou.net Thu Mar 27 17:48:52 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 27 Mar 2014 17:48:52 +0100 Subject: [Python-Dev] cpython: asyncio.tests: Autodiscover asyncio tests. Patch by Vajrasky Kok. Closes #20668 References: <3fvs0T1bpLz7LlP@mail.python.org> Message-ID: <20140327174852.3ccd6408@fsol> On Thu, 27 Mar 2014 17:22:01 +0100 (CET) yury.selivanov wrote: > http://hg.python.org/cpython/rev/bcc77493249c > changeset: 89995:bcc77493249c > user: Yury Selivanov > date: Thu Mar 27 12:21:20 2014 -0400 > summary: > asyncio.tests: Autodiscover asyncio tests. Patch by Vajrasky Kok. Closes #20668 > > files: > Lib/test/test_asyncio/__init__.py | 22 ++++++++---------- > Misc/NEWS | 3 ++ > 2 files changed, 13 insertions(+), 12 deletions(-) Perhaps the tests.txt file should be removed from the repo? Regards Antoine. From yorik.sar at gmail.com Thu Mar 27 17:53:47 2014 From: yorik.sar at gmail.com (Yuriy Taraday) Date: Thu, 27 Mar 2014 20:53:47 +0400 Subject: [Python-Dev] multiprocessing BaseManager's "serializer" argument In-Reply-To: References: Message-ID: On Thu, Mar 27, 2014 at 8:42 PM, Brett Cannon wrote: > Creating the bug is the best bet as then the reason can be discussed > there. If there are no objections then you can create a patch to make sure > it's tested and documented. Created http://bugs.python.org/issue21078 -- Kind regards, Yuriy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Mar 27 18:20:14 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Mar 2014 10:20:14 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <53344AB4.50904@stoneleaf.us> References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> Message-ID: Actually, I had ignored this discussion for so long that I was surprised by the outcome. My main use case isn't printing a number that may already be a string (I understand why that isn't reasonable when the output is expected to be bytes); it's printing a usually numeric value that may sometimes be None. It's a little surprising to have to use %a for this, but I guess I can live with it. On Thu, Mar 27, 2014 at 8:58 AM, Ethan Furman wrote: > On 03/27/2014 04:42 AM, Nick Coghlan wrote: > >> >> I also seem to recall Guido saying he liked it [%a], which flipped the >> >> discussion from "do we have a good rationale for including it?" to "do >> we have a good rationale for the BDFL to ignore his instincts?". >> However, it would be up to Guido to confirm that recollection, and if >> "Guido likes it" is part of the reason for inclusion of the %a code, >> the PEP should mention that explicitly. >> > > I checked Guido's posts (Subject contains PEP 461, From contains guido) > and did not see anything to that effect. > > -- > ~Ethan~ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Mar 27 18:29:54 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Mar 2014 10:29:54 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> Message-ID: I also don't understand why we can't use %b instead of %s. AFAIK %b currently doesn't mean anything and I somehow don't expect we're likely to add it for other reasons (unless there's a proposal I'm missing?). Just like we use %a instead of %r to remind people that it's not quite the same (since it applies .encode('ascii', 'backslashreplace')), shouldn't we use anything *but* %s to remind people that that is also not the same (not at all, in fact)? The PEP's argument against %b ("rejected as not adding any value either in clarity or simplicity") is hardly a good reason. On Thu, Mar 27, 2014 at 10:20 AM, Guido van Rossum wrote: > Actually, I had ignored this discussion for so long that I was surprised > by the outcome. My main use case isn't printing a number that may already > be a string (I understand why that isn't reasonable when the output is > expected to be bytes); it's printing a usually numeric value that may > sometimes be None. It's a little surprising to have to use %a for this, but > I guess I can live with it. > > > On Thu, Mar 27, 2014 at 8:58 AM, Ethan Furman wrote: > >> On 03/27/2014 04:42 AM, Nick Coghlan wrote: >> >>> >>> I also seem to recall Guido saying he liked it [%a], which flipped the >>> >>> discussion from "do we have a good rationale for including it?" to "do >>> we have a good rationale for the BDFL to ignore his instincts?". >>> However, it would be up to Guido to confirm that recollection, and if >>> "Guido likes it" is part of the reason for inclusion of the %a code, >>> the PEP should mention that explicitly. >>> >> >> I checked Guido's posts (Subject contains PEP 461, From contains guido) >> and did not see anything to that effect. >> >> -- >> ~Ethan~ >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ >> guido%40python.org >> > > > > -- > --Guido van Rossum (python.org/~guido) > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Thu Mar 27 18:31:55 2014 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 27 Mar 2014 17:31:55 +0000 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <53344AB4.50904@stoneleaf.us> References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> Message-ID: <5334608B.1050907@mrabarnett.plus.com> On 2014-03-27 15:58, Ethan Furman wrote: > On 03/27/2014 04:42 AM, Nick Coghlan wrote: >> >> I also seem to recall Guido saying he liked it [%a], which flipped the >> discussion from "do we have a good rationale for including it?" to "do >> we have a good rationale for the BDFL to ignore his instincts?". >> However, it would be up to Guido to confirm that recollection, and if >> "Guido likes it" is part of the reason for inclusion of the %a code, >> the PEP should mention that explicitly. > > I checked Guido's posts (Subject contains PEP 461, From contains guido) and did not see anything to that effect. > Date: Mon, 13 Jan 2014 12:09:23 -0800 Subject: Re: [Python-Dev] PEP 460 reboot """If we have %b for strictly interpolating bytes, I'm fine with adding %a for calling ascii() on the argument and then interpolating the result after ASCII-encoding it.""" From guido at python.org Thu Mar 27 18:36:18 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Mar 2014 10:36:18 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <5334608B.1050907@mrabarnett.plus.com> References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <5334608B.1050907@mrabarnett.plus.com> Message-ID: Wow. I'm pretty consistent. I still like that. :-) On Thu, Mar 27, 2014 at 10:31 AM, MRAB wrote: > On 2014-03-27 15:58, Ethan Furman wrote: > >> On 03/27/2014 04:42 AM, Nick Coghlan wrote: >> >>> >>> I also seem to recall Guido saying he liked it [%a], which flipped the >>> discussion from "do we have a good rationale for including it?" to "do >>> we have a good rationale for the BDFL to ignore his instincts?". >>> However, it would be up to Guido to confirm that recollection, and if >>> "Guido likes it" is part of the reason for inclusion of the %a code, >>> the PEP should mention that explicitly. >>> >> >> I checked Guido's posts (Subject contains PEP 461, From contains guido) >> and did not see anything to that effect. >> >> Date: Mon, 13 Jan 2014 12:09:23 -0800 > Subject: Re: [Python-Dev] PEP 460 reboot > > """If we have %b for strictly interpolating bytes, I'm fine with adding > %a for calling ascii() on the argument and then interpolating the > result after ASCII-encoding it.""" > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Mar 27 18:55:15 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 27 Mar 2014 10:55:15 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> Message-ID: <53346603.1000402@stoneleaf.us> On 03/27/2014 10:29 AM, Guido van Rossum wrote: > > I also don't understand why we can't use %b instead of %s. AFAIK %b currently doesn't mean anything and I somehow don't > expect we're likely to add it for other reasons (unless there's a proposal I'm missing?). Just like we use %a instead of > %r to remind people that it's not quite the same (since it applies .encode('ascii', 'backslashreplace')), shouldn't we > use anything *but* %s to remind people that that is also not the same (not at all, in fact)? The PEP's argument against > %b ("rejected as not adding any value either in clarity or simplicity") is hardly a good reason. The biggest reason to use %s is to support a common code base for 2/3 endeavors. The biggest reason to not include %b is that it means binary number in format(); given that each type can invent it's own mini-language, this probably isn't a very strong argument against it. I have moderate feelings for keeping %s as a synonym for %b for backwards compatibility with Py2 code (when it's appropriate). -- ~Ethan~ From guido at python.org Thu Mar 27 19:24:12 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Mar 2014 11:24:12 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <53346603.1000402@stoneleaf.us> References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> Message-ID: On Thu, Mar 27, 2014 at 10:55 AM, Ethan Furman wrote: > On 03/27/2014 10:29 AM, Guido van Rossum wrote: > >> >> I also don't understand why we can't use %b instead of %s. AFAIK %b >> currently doesn't mean anything and I somehow don't >> expect we're likely to add it for other reasons (unless there's a >> proposal I'm missing?). Just like we use %a instead of >> %r to remind people that it's not quite the same (since it applies >> .encode('ascii', 'backslashreplace')), shouldn't we >> use anything *but* %s to remind people that that is also not the same >> (not at all, in fact)? The PEP's argument against >> %b ("rejected as not adding any value either in clarity or simplicity") >> is hardly a good reason. >> > > The biggest reason to use %s is to support a common code base for 2/3 > endeavors. But it's mostly useless for that purpose. In Python 2, in practice %s doesn't mean "string". It means "use the default formatting just as if I was using print." And in theory it also means that -- in fact "call __str__()" is the formal definition, and print is also defined as using __str__, and this is all intentional. (I also intended __str__ to be *mostly* the same as __repr__, with a specific exception for the str type itself. In practice some frameworks have adopted a different interpretation, making __repr__ produce something *more* "user friendly" than __str__ but including newlines, because some people believe the main use case for __repr__ is the interactive prompt. I believe this causes problems for some *other* uses of __repr__, such as for producing an "unambiguous" representation useful for e.g. logging -- but I don't want to be too bitter about it. :-) The biggest reason to not include %b is that it means binary number in > format(); given that each type can invent it's own mini-language, this > probably isn't a very strong argument against it. > Especially since I can't imagine the spelling in format() includes '%'. > I have moderate feelings for keeping %s as a synonym for %b for backwards > compatibility with Py2 code (when it's appropriate). > I think it's mere existence (with the restrictions currently in the PEP) would cause more confusion than that is worth. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Mar 27 19:04:12 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 27 Mar 2014 11:04:12 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <53346603.1000402@stoneleaf.us> References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> Message-ID: <5334681C.40904@stoneleaf.us> On 03/27/2014 10:55 AM, Ethan Furman wrote: > On 03/27/2014 10:29 AM, Guido van Rossum wrote: >> >> I also don't understand why we can't use %b instead of %s. AFAIK %b currently doesn't mean anything and I somehow don't >> expect we're likely to add it for other reasons (unless there's a proposal I'm missing?). Just like we use %a instead of >> %r to remind people that it's not quite the same (since it applies .encode('ascii', 'backslashreplace')), shouldn't we >> use anything *but* %s to remind people that that is also not the same (not at all, in fact)? The PEP's argument against >> %b ("rejected as not adding any value either in clarity or simplicity") is hardly a good reason. > > The biggest reason to use %s is to support a common code base for 2/3 endeavors. The biggest reason to not include %b > is that it means binary number in format(); given that each type can invent it's own mini-language, this probably isn't > a very strong argument against it. > > I have moderate feelings for keeping %s as a synonym for %b for backwards compatibility with Py2 code (when it's > appropriate). Changed to: ---------------------------------------------------------------------------------- ``%b`` will insert a series of bytes. These bytes are collected in one of two ways: - input type supports ``Py_buffer`` [4]_? use it to collect the necessary bytes - input type is something else? use its ``__bytes__`` method [5]_ ; if there isn't one, raise a ``TypeError`` In particular, ``%b`` will not accept numbers nor ``str``. ``str`` is rejected as the string to bytes conversion requires an encoding, and we are refusing to guess; numbers are rejected because: - what makes a number is fuzzy (float? Decimal? Fraction? some user type?) - allowing numbers would lead to ambiguity between numbers and textual representations of numbers (3.14 vs '3.14') - given the nature of wire formats, explicit is definitely better than implicit ``%s`` is included as a synonym for ``%b`` for the sole purpose of making 2/3 code bases easier to maintain. Python 3 only code should use ``%b``. Examples:: >>> b'%b' % b'abc' b'abc' >>> b'%b' % 'some string'.encode('utf8') b'some string' >>> b'%b' % 3.14 Traceback (most recent call last): ... TypeError: b'%b' does not accept 'float' >>> b'%b' % 'hello world!' Traceback (most recent call last): ... TypeError: b'%b' does not accept 'str' ---------------------------------------------------------------------------------- -- ~Ethan~ From guido at python.org Thu Mar 27 19:41:45 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Mar 2014 11:41:45 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <5334681C.40904@stoneleaf.us> References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> Message-ID: Much better, but I'm still not happy with including %s at all. Otherwise it's accept-worthy. (How's that for pressure. :-) On Thu, Mar 27, 2014 at 11:04 AM, Ethan Furman wrote: > On 03/27/2014 10:55 AM, Ethan Furman wrote: > >> On 03/27/2014 10:29 AM, Guido van Rossum wrote: >> >>> >>> I also don't understand why we can't use %b instead of %s. AFAIK %b >>> currently doesn't mean anything and I somehow don't >>> expect we're likely to add it for other reasons (unless there's a >>> proposal I'm missing?). Just like we use %a instead of >>> %r to remind people that it's not quite the same (since it applies >>> .encode('ascii', 'backslashreplace')), shouldn't we >>> use anything *but* %s to remind people that that is also not the same >>> (not at all, in fact)? The PEP's argument against >>> %b ("rejected as not adding any value either in clarity or simplicity") >>> is hardly a good reason. >>> >> >> The biggest reason to use %s is to support a common code base for 2/3 >> endeavors. The biggest reason to not include %b >> is that it means binary number in format(); given that each type can >> invent it's own mini-language, this probably isn't >> a very strong argument against it. >> >> I have moderate feelings for keeping %s as a synonym for %b for backwards >> compatibility with Py2 code (when it's >> appropriate). >> > > Changed to: > ------------------------------------------------------------ > ---------------------- > ``%b`` will insert a series of bytes. These bytes are collected in one of > two > ways: > > - input type supports ``Py_buffer`` [4]_? > > use it to collect the necessary bytes > > - input type is something else? > use its ``__bytes__`` method [5]_ ; if there isn't one, raise a > ``TypeError`` > > In particular, ``%b`` will not accept numbers nor ``str``. ``str`` is > rejected > as the string to bytes conversion requires an encoding, and we are > refusing to > guess; numbers are rejected because: > > - what makes a number is fuzzy (float? Decimal? Fraction? some user > type?) > > - allowing numbers would lead to ambiguity between numbers and textual > representations of numbers (3.14 vs '3.14') > > - given the nature of wire formats, explicit is definitely better than > implicit > > ``%s`` is included as a synonym for ``%b`` for the sole purpose of making > 2/3 code > bases easier to maintain. Python 3 only code should use ``%b``. > > Examples:: > > >>> b'%b' % b'abc' > b'abc' > > >>> b'%b' % 'some string'.encode('utf8') > b'some string' > > >>> b'%b' % 3.14 > > Traceback (most recent call last): > ... > TypeError: b'%b' does not accept 'float' > > >>> b'%b' % 'hello world!' > > Traceback (most recent call last): > ... > TypeError: b'%b' does not accept 'str' > ------------------------------------------------------------ > ---------------------- > > > -- > ~Ethan~ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Thu Mar 27 19:47:59 2014 From: bcannon at gmail.com (Brett Cannon) Date: Thu, 27 Mar 2014 18:47:59 +0000 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> Message-ID: On Thu Mar 27 2014 at 2:42:40 PM, Guido van Rossum wrote: > Much better, but I'm still not happy with including %s at all. Otherwise > it's accept-worthy. (How's that for pressure. :-) > But if we only add %b and leave out %s then how is this going to lead to Python 2/3 compatible code since %b is not in Python 2? Or am I misunderstanding you? -Brett > > > On Thu, Mar 27, 2014 at 11:04 AM, Ethan Furman wrote: > >> On 03/27/2014 10:55 AM, Ethan Furman wrote: >> >>> On 03/27/2014 10:29 AM, Guido van Rossum wrote: >>> >>>> >>>> I also don't understand why we can't use %b instead of %s. AFAIK %b >>>> currently doesn't mean anything and I somehow don't >>>> expect we're likely to add it for other reasons (unless there's a >>>> proposal I'm missing?). Just like we use %a instead of >>>> %r to remind people that it's not quite the same (since it applies >>>> .encode('ascii', 'backslashreplace')), shouldn't we >>>> use anything *but* %s to remind people that that is also not the same >>>> (not at all, in fact)? The PEP's argument against >>>> %b ("rejected as not adding any value either in clarity or simplicity") >>>> is hardly a good reason. >>>> >>> >>> The biggest reason to use %s is to support a common code base for 2/3 >>> endeavors. The biggest reason to not include %b >>> is that it means binary number in format(); given that each type can >>> invent it's own mini-language, this probably isn't >>> a very strong argument against it. >>> >>> I have moderate feelings for keeping %s as a synonym for %b for >>> backwards compatibility with Py2 code (when it's >>> appropriate). >>> >> >> Changed to: >> ------------------------------------------------------------ >> ---------------------- >> ``%b`` will insert a series of bytes. These bytes are collected in one >> of two >> ways: >> >> - input type supports ``Py_buffer`` [4]_? >> >> use it to collect the necessary bytes >> >> - input type is something else? >> use its ``__bytes__`` method [5]_ ; if there isn't one, raise a >> ``TypeError`` >> >> In particular, ``%b`` will not accept numbers nor ``str``. ``str`` is >> rejected >> as the string to bytes conversion requires an encoding, and we are >> refusing to >> guess; numbers are rejected because: >> >> - what makes a number is fuzzy (float? Decimal? Fraction? some user >> type?) >> >> - allowing numbers would lead to ambiguity between numbers and textual >> representations of numbers (3.14 vs '3.14') >> >> - given the nature of wire formats, explicit is definitely better than >> implicit >> >> ``%s`` is included as a synonym for ``%b`` for the sole purpose of making >> 2/3 code >> bases easier to maintain. Python 3 only code should use ``%b``. >> >> Examples:: >> >> >>> b'%b' % b'abc' >> b'abc' >> >> >>> b'%b' % 'some string'.encode('utf8') >> b'some string' >> >> >>> b'%b' % 3.14 >> >> Traceback (most recent call last): >> ... >> TypeError: b'%b' does not accept 'float' >> >> >>> b'%b' % 'hello world!' >> >> Traceback (most recent call last): >> ... >> TypeError: b'%b' does not accept 'str' >> ------------------------------------------------------------ >> ---------------------- >> > >> >> -- >> ~Ethan~ >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ >> guido%40python.org >> > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Mar 27 19:52:41 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 27 Mar 2014 14:52:41 -0400 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> Message-ID: I feel not including %s is nuts. Should I write .replace('%b', '%s')? All I desperately need are APIs that provide enough unicode / str type safety that I get an exception when mixing them accidentally... in my own code, dynamic typing is usually a bug. As has been endlessly discussed, %s for bytes is a bit like exposing sprintf()... On Thu, Mar 27, 2014 at 2:41 PM, Guido van Rossum wrote: > Much better, but I'm still not happy with including %s at all. Otherwise > it's accept-worthy. (How's that for pressure. :-) > > > On Thu, Mar 27, 2014 at 11:04 AM, Ethan Furman wrote: >> >> On 03/27/2014 10:55 AM, Ethan Furman wrote: >>> >>> On 03/27/2014 10:29 AM, Guido van Rossum wrote: >>>> >>>> >>>> I also don't understand why we can't use %b instead of %s. AFAIK %b >>>> currently doesn't mean anything and I somehow don't >>>> expect we're likely to add it for other reasons (unless there's a >>>> proposal I'm missing?). Just like we use %a instead of >>>> %r to remind people that it's not quite the same (since it applies >>>> .encode('ascii', 'backslashreplace')), shouldn't we >>>> use anything *but* %s to remind people that that is also not the same >>>> (not at all, in fact)? The PEP's argument against >>>> %b ("rejected as not adding any value either in clarity or simplicity") >>>> is hardly a good reason. >>> >>> >>> The biggest reason to use %s is to support a common code base for 2/3 >>> endeavors. The biggest reason to not include %b >>> is that it means binary number in format(); given that each type can >>> invent it's own mini-language, this probably isn't >>> a very strong argument against it. >>> >>> I have moderate feelings for keeping %s as a synonym for %b for backwards >>> compatibility with Py2 code (when it's >>> appropriate). >> >> >> Changed to: >> >> ---------------------------------------------------------------------------------- >> ``%b`` will insert a series of bytes. These bytes are collected in one of >> two >> ways: >> >> - input type supports ``Py_buffer`` [4]_? >> >> use it to collect the necessary bytes >> >> - input type is something else? >> use its ``__bytes__`` method [5]_ ; if there isn't one, raise a >> ``TypeError`` >> >> In particular, ``%b`` will not accept numbers nor ``str``. ``str`` is >> rejected >> as the string to bytes conversion requires an encoding, and we are >> refusing to >> guess; numbers are rejected because: >> >> - what makes a number is fuzzy (float? Decimal? Fraction? some user >> type?) >> >> - allowing numbers would lead to ambiguity between numbers and textual >> representations of numbers (3.14 vs '3.14') >> >> - given the nature of wire formats, explicit is definitely better than >> implicit >> >> ``%s`` is included as a synonym for ``%b`` for the sole purpose of making >> 2/3 code >> bases easier to maintain. Python 3 only code should use ``%b``. >> >> Examples:: >> >> >>> b'%b' % b'abc' >> b'abc' >> >> >>> b'%b' % 'some string'.encode('utf8') >> b'some string' >> >> >>> b'%b' % 3.14 >> >> Traceback (most recent call last): >> ... >> TypeError: b'%b' does not accept 'float' >> >> >>> b'%b' % 'hello world!' >> >> Traceback (most recent call last): >> ... >> TypeError: b'%b' does not accept 'str' >> >> ---------------------------------------------------------------------------------- >> >> >> -- >> ~Ethan~ >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > From guido at python.org Thu Mar 27 19:53:50 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Mar 2014 11:53:50 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> Message-ID: So what's the use case for Python 2/3 compatible code? IMO the main use case for the PEP is simply to be able to construct bytes from a combination of a template and some input that may include further bytes and numbers. E.g. in asyncio when you write an HTTP client or server you have to construct bytes to write to the socket, and I'd be happy if I could write b'HTTP/1.0 %d %b\r\n' % (status, message) rather than having to use str(status).encode('ascii') and concatenation or join(). On Thu, Mar 27, 2014 at 11:47 AM, Brett Cannon wrote: > > > On Thu Mar 27 2014 at 2:42:40 PM, Guido van Rossum > wrote: > >> Much better, but I'm still not happy with including %s at all. Otherwise >> it's accept-worthy. (How's that for pressure. :-) >> > > But if we only add %b and leave out %s then how is this going to lead to > Python 2/3 compatible code since %b is not in Python 2? Or am I > misunderstanding you? > > -Brett > > >> >> >> On Thu, Mar 27, 2014 at 11:04 AM, Ethan Furman wrote: >> >>> On 03/27/2014 10:55 AM, Ethan Furman wrote: >>> >>>> On 03/27/2014 10:29 AM, Guido van Rossum wrote: >>>> >>>>> >>>>> I also don't understand why we can't use %b instead of %s. AFAIK %b >>>>> currently doesn't mean anything and I somehow don't >>>>> expect we're likely to add it for other reasons (unless there's a >>>>> proposal I'm missing?). Just like we use %a instead of >>>>> %r to remind people that it's not quite the same (since it applies >>>>> .encode('ascii', 'backslashreplace')), shouldn't we >>>>> use anything *but* %s to remind people that that is also not the same >>>>> (not at all, in fact)? The PEP's argument against >>>>> %b ("rejected as not adding any value either in clarity or >>>>> simplicity") is hardly a good reason. >>>>> >>>> >>>> The biggest reason to use %s is to support a common code base for 2/3 >>>> endeavors. The biggest reason to not include %b >>>> is that it means binary number in format(); given that each type can >>>> invent it's own mini-language, this probably isn't >>>> a very strong argument against it. >>>> >>>> I have moderate feelings for keeping %s as a synonym for %b for >>>> backwards compatibility with Py2 code (when it's >>>> appropriate). >>>> >>> >>> Changed to: >>> ------------------------------------------------------------ >>> ---------------------- >>> ``%b`` will insert a series of bytes. These bytes are collected in one >>> of two >>> ways: >>> >>> - input type supports ``Py_buffer`` [4]_? >>> >>> use it to collect the necessary bytes >>> >>> - input type is something else? >>> use its ``__bytes__`` method [5]_ ; if there isn't one, raise a >>> ``TypeError`` >>> >>> In particular, ``%b`` will not accept numbers nor ``str``. ``str`` is >>> rejected >>> as the string to bytes conversion requires an encoding, and we are >>> refusing to >>> guess; numbers are rejected because: >>> >>> - what makes a number is fuzzy (float? Decimal? Fraction? some user >>> type?) >>> >>> - allowing numbers would lead to ambiguity between numbers and textual >>> representations of numbers (3.14 vs '3.14') >>> >>> - given the nature of wire formats, explicit is definitely better than >>> implicit >>> >>> ``%s`` is included as a synonym for ``%b`` for the sole purpose of >>> making 2/3 code >>> bases easier to maintain. Python 3 only code should use ``%b``. >>> >>> Examples:: >>> >>> >>> b'%b' % b'abc' >>> b'abc' >>> >>> >>> b'%b' % 'some string'.encode('utf8') >>> b'some string' >>> >>> >>> b'%b' % 3.14 >>> >>> Traceback (most recent call last): >>> ... >>> TypeError: b'%b' does not accept 'float' >>> >>> >>> b'%b' % 'hello world!' >>> >>> Traceback (most recent call last): >>> ... >>> TypeError: b'%b' does not accept 'str' >>> ------------------------------------------------------------ >>> ---------------------- >>> >> >>> >>> -- >>> ~Ethan~ >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ >>> guido%40python.org >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ >> brett%40python.org >> > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Mar 27 19:34:24 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 27 Mar 2014 11:34:24 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> Message-ID: <53346F30.6050708@stoneleaf.us> On 03/27/2014 11:24 AM, Guido van Rossum wrote: > On Thu, Mar 27, 2014 at 10:55 AM, Ethan Furman wrote: >> >> The biggest reason to use %s is to support a common code base for 2/3 endeavors. > > But it's mostly useless for that purpose. In Python 2, in practice %s doesn't mean "string". [...] In Python 2 if one is using 'str' as a 'bytes' container, and doing interpolation, %s is the only choice available for other 'bytes' (aka other 'str's). Note that I'm happy to be proven wrong on this point. :) -- ~Ethan~ From guido at python.org Thu Mar 27 19:59:11 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Mar 2014 11:59:11 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> Message-ID: On Thu, Mar 27, 2014 at 11:52 AM, Daniel Holth wrote: > I feel not including %s is nuts. Should I write .replace('%b', '%s')? > I assume you meant .replace('%s', '%b') (unless you're converting Python 3 code to Python 2, which would mean you really are nuts :-). But that's not going to help for the majority of code using %s -- as I am trying to argue, %s doesn't mean "expect the argument to be a str" and neither is that how it's commonly used (although it's *possible* that that is how *you* use it exclusively -- that doesn't make you nuts, just more strict than most people). > All I desperately need are APIs that provide enough unicode / str type > safety that I get an exception when mixing them accidentally... in my > own code, dynamic typing is usually a bug. As has been endlessly > discussed, %s for bytes is a bit like exposing sprintf()... > I don't understand that last claim (I can't figure out whether in this context is exposing sprintf() is considered good or bad). But apart from that, can you give some specific examples? PS. I am not trying to be difficult. I honestly don't understand the use case yet, and the PEP doesn't do much to support it. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Mar 27 20:06:22 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Mar 2014 12:06:22 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <53346F30.6050708@stoneleaf.us> References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <53346F30.6050708@stoneleaf.us> Message-ID: On Thu, Mar 27, 2014 at 11:34 AM, Ethan Furman wrote: > On 03/27/2014 11:24 AM, Guido van Rossum wrote: > >> On Thu, Mar 27, 2014 at 10:55 AM, Ethan Furman wrote: >> >>> >>> The biggest reason to use %s is to support a common code base for 2/3 >>> endeavors. >>> >> >> But it's mostly useless for that purpose. In Python 2, in practice %s >> doesn't mean "string". [...] >> > > In Python 2 if one is using 'str' as a 'bytes' container, and doing > interpolation, %s is the only choice available for other 'bytes' (aka other > 'str's). Note that I'm happy to be proven wrong on this point. :) > That is true. And we can't change Python 2. I still have this idea in my head that *most* cases where %s is used in Python 2 will break in Python 3 under the PEP's rules, but perhaps they are not the majority of situations where the context is manipulating bytes. And I suppose that *very* few internet protocols are designed to accept either an integer or the literal string None, so that use case (which I brought up) isn't very realistic -- in fact it may be better to raise an exception rather than sending a protocol violation. So, I think you have changed my mind. I still like the idea of promoting %b in pure Python 3 code to emphasize that it really behaves very differently from %s; but I now have peace with %s as an alias. (It might also benefit cases where somehow there's a symmetry in some Python 3 code between bytes and str.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Thu Mar 27 20:10:34 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 27 Mar 2014 12:10:34 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> Message-ID: <533477AA.1070205@g.nevcal.com> On 3/27/2014 11:59 AM, Guido van Rossum wrote: > On Thu, Mar 27, 2014 at 11:52 AM, Daniel Holth > wrote: > > I feel not including %s is nuts. Should I write .replace('%b', '%s')? > > > I assume you meant .replace('%s', '%b') (unless you're converting > Python 3 code to Python 2, which would mean you really are nuts :-). > > But that's not going to help for the majority of code using %s -- as I > am trying to argue, %s doesn't mean "expect the argument to be a str" > and neither is that how it's commonly used (although it's *possible* > that that is how *you* use it exclusively -- that doesn't make you > nuts, just more strict than most people). That _is_ how it is commonly used in Py2 when dealing with binary data in mixed ASCII/binary protocols, is what I've been hearing in this discussion, and what small use I've made of Py2 when some unported module forced me to use it (I started Python about the time Py3 was released)... the expected argument is a (Py2) str containing binary data (would be bytes in Py3). While there are many other reasons to use %s in other coding situations, this is the only way to do bytes interpolations using %. And there is no %b in Py2, so for Py2/3 compatibility, %s needs to do bytes interpolations in Py3. And if it does, there is no need for %b in Py3 %, because they would be identical and redundant. > All I desperately need are APIs that provide enough unicode / str type > safety that I get an exception when mixing them accidentally... in my > own code, dynamic typing is usually a bug. As has been endlessly > discussed, %s for bytes is a bit like exposing sprintf()... > > > I don't understand that last claim (I can't figure out whether in this > context is exposing sprintf() is considered good or bad). But apart > from that, can you give some specific examples? > > PS. I am not trying to be difficult. I honestly don't understand the > use case yet, and the PEP doesn't do much to support it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Mar 27 20:18:41 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 27 Mar 2014 15:18:41 -0400 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> Message-ID: On Thu, Mar 27, 2014 at 2:53 PM, Guido van Rossum wrote: > So what's the use case for Python 2/3 compatible code? IMO the main use case > for the PEP is simply to be able to construct bytes from a combination of a > template and some input that may include further bytes and numbers. E.g. in > asyncio when you write an HTTP client or server you have to construct bytes > to write to the socket, and I'd be happy if I could write b'HTTP/1.0 %d > %b\r\n' % (status, message) rather than having to use > str(status).encode('ascii') and concatenation or join(). It seems to be notoriously difficult to understand or explain why Unicode can still be very hard in Python 3 or in code that is in the middle of being ported or has to run in both interpreters. As far as I can tell part of it is when a symbol has type(str or bytes) depending (declared as if we had a static type system with union types); some of it is because incorrect mixing can happen without an exception, only to be discovered later and far away in space and time from the error (worse of all in a serialized file), and part of it is all of the not easily checkable "types" a particular Unicode object has depending on whether it contains surrogates or codes > n. Sometimes you might simply disagree about whether an API should be returning bytes or Unicode in mildly ambiguous cases like base64 encoding. Sometimes Unicode is just intrinsically complicated. For me this PEP holds the promise of being able to do work in the bytes domain, with no accidental mixing ever, when I *really* want bytes. For 2+3 I would get exceptions sometimes in Python 2 and exceptions all the time in Python 3 for mistakes. I hope this is less error prone in strict domains than for example u"string processing".encode('latin1'). And I hope that there is very little type(str or int) in HTTP for example or other "legitimate" bytes domains but I don't know; I suspect that if you have a lot of problems with bytes' %s then it's a clue you should use (u"%s" % (argument)).encode() instead. sprintf()'s version of %s just takes a char* and puts it in without doing any type conversion of course. IANACL (I am not a C lawyer). From ethan at stoneleaf.us Thu Mar 27 20:01:10 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 27 Mar 2014 12:01:10 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> Message-ID: <53347576.6010309@stoneleaf.us> On 03/27/2014 11:53 AM, Guido van Rossum wrote: > > So what's the use case for Python 2/3 compatible code? IMO the main use case for the PEP is simply to be able to > construct bytes from a combination of a template and some input that may include further bytes and numbers. E.g. in > asyncio when you write an HTTP client or server you have to construct bytes to write to the socket, and I'd be happy if > I could write b'HTTP/1.0 %d %b\r\n' % (status, message) rather than having to use str(status).encode('ascii') and > concatenation or join(). My own dbf module [1] would make use of this feature, and I'm sure some of the pdf modules would as well (I recall somebody chiming in about their own pdf module). -- ~Ethan~ [1] https://pypi.python.org/pypi/dbf From ethan at stoneleaf.us Thu Mar 27 20:11:03 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 27 Mar 2014 12:11:03 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> Message-ID: <533477C7.6020109@stoneleaf.us> On 03/27/2014 11:59 AM, Guido van Rossum wrote: > > PS. I am not trying to be difficult. I honestly don't understand the use case yet, and the PEP doesn't do much to > support it. How's this? ---------------------------------------------------------------------------- Compatibility with Python 2 =========================== As noted above, ``%s`` is being included solely to help ease migration from, and/or have a single code base with, Python 2. This is important as there are modules both in the wild and behind closed doors that currently use the Python 2 ``str`` type as a ``bytes`` container, and hence are using ``%s`` as a bytes interpolator. However, ``%b`` should be used in new, Python 3 only code, so ``%s`` will immediately be deprecated, but not removed until the next major Python release. ---------------------------------------------------------------------------- -- ~Ethan~ From guido at python.org Thu Mar 27 21:00:08 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Mar 2014 13:00:08 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <533477C7.6020109@stoneleaf.us> References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> <533477C7.6020109@stoneleaf.us> Message-ID: I love it! On Thu, Mar 27, 2014 at 12:11 PM, Ethan Furman wrote: > On 03/27/2014 11:59 AM, Guido van Rossum wrote: > >> >> PS. I am not trying to be difficult. I honestly don't understand the use >> case yet, and the PEP doesn't do much to >> support it. >> > > How's this? > ------------------------------------------------------------ > ---------------- > Compatibility with Python 2 > =========================== > > As noted above, ``%s`` is being included solely to help ease migration > from, > and/or have a single code base with, Python 2. This is important as there > are modules both in the wild and behind closed doors that currently use the > Python 2 ``str`` type as a ``bytes`` container, and hence are using ``%s`` > as a bytes interpolator. > > However, ``%b`` should be used in new, Python 3 only code, so ``%s`` will > immediately be deprecated, but not removed until the next major Python > release. > > ------------------------------------------------------------ > ---------------- > > -- > ~Ethan~ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Mar 27 21:07:05 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 27 Mar 2014 13:07:05 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Request for Pronouncement Message-ID: <533484E9.3090300@stoneleaf.us> Requesting pronouncement on PEP 461. Full text below. =============================================================================== PEP: 461 Title: Adding % formatting to bytes and bytearray Version: $Revision$ Last-Modified: $Date$ Author: Ethan Furman Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2014-01-13 Python-Version: 3.5 Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25, 2014-03-27 Resolution: Abstract ======== This PEP proposes adding % formatting operations similar to Python 2's ``str`` type to ``bytes`` and ``bytearray`` [1]_ [2]_. Rationale ========= While interpolation is usually thought of as a string operation, there are cases where interpolation on ``bytes`` or ``bytearrays`` make sense, and the work needed to make up for this missing functionality detracts from the overall readability of the code. Motivation ========== With Python 3 and the split between ``str`` and ``bytes``, one small but important area of programming became slightly more difficult, and much more painful -- wire format protocols [3]_. This area of programming is characterized by a mixture of binary data and ASCII compatible segments of text (aka ASCII-encoded text). Bringing back a restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in writing new wire format code, and in porting Python 2 wire format code. Common use-cases include ``dbf`` and ``pdf`` file formats, ``email`` formats, and ``FTP`` and ``HTTP`` communications, among many others. Proposed semantics for ``bytes`` and ``bytearray`` formatting ============================================================= %-interpolation --------------- All the numeric formatting codes (``d``, ``i``, ``o``, ``u``, ``x``, ``X``, ``e``, ``E``, ``f``, ``F``, ``g``, ``G``, and any that are subsequently added to Python 3) will be supported, and will work as they do for str, including the padding, justification and other related modifiers (currently ``#``, ``0``, ``-``, `` `` (space), and ``+`` (plus any added to Python 3)). The only non-numeric codes allowed are ``c``, ``b``, ``a``, and ``s`` (which is a synonym for b). For the numeric codes, the only difference between ``str`` and ``bytes`` (or ``bytearray``) interpolation is that the results from these codes will be ASCII-encoded text, not unicode. In other words, for any numeric formatting code `%x`:: b"%x" % val is equivalent to:: ("%x" % val).encode("ascii") Examples:: >>> b'%4x' % 10 b' a' >>> b'%#4x' % 10 ' 0xa' >>> b'%04X' % 10 '000A' ``%c`` will insert a single byte, either from an ``int`` in range(256), or from a ``bytes`` argument of length 1, not from a ``str``. Examples:: >>> b'%c' % 48 b'0' >>> b'%c' % b'a' b'a' ``%b`` will insert a series of bytes. These bytes are collected in one of two ways: - input type supports ``Py_buffer`` [4]_? use it to collect the necessary bytes - input type is something else? use its ``__bytes__`` method [5]_ ; if there isn't one, raise a ``TypeError`` In particular, ``%b`` will not accept numbers nor ``str``. ``str`` is rejected as the string to bytes conversion requires an encoding, and we are refusing to guess; numbers are rejected because: - what makes a number is fuzzy (float? Decimal? Fraction? some user type?) - allowing numbers would lead to ambiguity between numbers and textual representations of numbers (3.14 vs '3.14') - given the nature of wire formats, explicit is definitely better than implicit ``%s`` is included as a synonym for ``%b`` for the sole purpose of making 2/3 code bases easier to maintain. Python 3 only code should use ``%b``. Examples:: >>> b'%b' % b'abc' b'abc' >>> b'%b' % 'some string'.encode('utf8') b'some string' >>> b'%b' % 3.14 Traceback (most recent call last): ... TypeError: b'%b' does not accept 'float' >>> b'%b' % 'hello world!' Traceback (most recent call last): ... TypeError: b'%b' does not accept 'str' ``%a`` will give the equivalent of ``repr(some_obj).encode('ascii', 'backslashreplace')`` on the interpolated value. Use cases include developing a new protocol and writing landmarks into the stream; debugging data going into an existing protocol to see if the problem is the protocol itself or bad data; a fall-back for a serialization format; or any situation where defining ``__bytes__`` would not be appropriate but a readable/informative representation is needed [6]_. Examples:: >>> b'%a' % 3.14 b'3.14' >>> b'%a' % b'abc' b"b'abc'" >>> b'%a' % 'def' b"'def'" Unsupported codes ----------------- ``%r`` (which calls ``__repr__`` and returns a ``str``) is not supported. Compatibility with Python 2 =========================== As noted above, ``%s`` is being included solely to help ease migration from, and/or have a single code base with, Python 2. This is important as there are modules both in the wild and behind closed doors that currently use the Python 2 ``str`` type as a ``bytes`` container, and hence are using ``%s`` as a bytes interpolator. However, ``%b`` should be used in new, Python 3 only code, so ``%s`` will immediately be deprecated, but not removed until the next major Python release. Proposed variations =================== It has been proposed to automatically use ``.encode('ascii','strict')`` for ``str`` arguments to ``%b``. - Rejected as this would lead to intermittent failures. Better to have the operation always fail so the trouble-spot can be correctly fixed. It has been proposed to have ``%b`` return the ascii-encoded repr when the value is a ``str`` (b'%b' % 'abc' --> b"'abc'"). - Rejected as this would lead to hard to debug failures far from the problem site. Better to have the operation always fail so the trouble-spot can be easily fixed. Originally this PEP also proposed adding format-style formatting, but it was decided that format and its related machinery were all strictly text (aka ``str``) based, and it was dropped. Various new special methods were proposed, such as ``__ascii__``, ``__format_bytes__``, etc.; such methods are not needed at this time, but can be visited again later if real-world use shows deficiencies with this solution. A competing PEP, ``PEP 460 Add binary interpolation and formatting`` [7]_, also exists. Objections ========== The objections raised against this PEP were mainly variations on two themes: - the ``bytes`` and ``bytearray`` types are for pure binary data, with no assumptions about encodings - offering %-interpolation that assumes an ASCII encoding will be an attractive nuisance and lead us back to the problems of the Python 2 ``str``/``unicode`` text model As was seen during the discussion, ``bytes`` and ``bytearray`` are also used for mixed binary data and ASCII-compatible segments: file formats such as ``dbf`` and ``pdf``, network protocols such as ``ftp`` and ``email``, etc. ``bytes`` and ``bytearray`` already have several methods which assume an ASCII compatible encoding. ``upper()``, ``isalpha()``, and ``expandtabs()`` to name just a few. %-interpolation, with its very restricted mini-language, will not be any more of a nuisance than the already existing methods. Some have objected to allowing the full range of numeric formatting codes with the claim that decimal alone would be sufficient. However, at least two formats (dbf and pdf) make use of non-decimal numbers. Footnotes ========= .. [1] http://docs.python.org/2/library/stdtypes.html#string-formatting .. [2] neither string.Template, format, nor str.format are under consideration .. [3] https://mail.python.org/pipermail/python-dev/2014-January/131518.html .. [4] http://docs.python.org/3/c-api/buffer.html examples: ``memoryview``, ``array.array``, ``bytearray``, ``bytes`` .. [5] http://docs.python.org/3/reference/datamodel.html#object.__bytes__ .. [6] https://mail.python.org/pipermail/python-dev/2014-February/132750.html .. [7] http://python.org/dev/peps/pep-0460/ Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From guido at python.org Thu Mar 27 21:44:10 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Mar 2014 13:44:10 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Request for Pronouncement In-Reply-To: <533484E9.3090300@stoneleaf.us> References: <533484E9.3090300@stoneleaf.us> Message-ID: Accepted. Congrats with marshalling yet another quite contentious discussion, and putting up with my last-minute block-headedness! If you're going to commit another change, may I suggest to add, to the section stating that %r is not supported, that %a is usually a suitable replacement for %r? On Thu, Mar 27, 2014 at 1:07 PM, Ethan Furman wrote: > Requesting pronouncement on PEP 461. Full text below. > > ============================================================ > =================== > PEP: 461 > Title: Adding % formatting to bytes and bytearray > Version: $Revision$ > Last-Modified: $Date$ > Author: Ethan Furman > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 2014-01-13 > Python-Version: 3.5 > Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25, > 2014-03-27 > Resolution: > > > Abstract > ======== > > This PEP proposes adding % formatting operations similar to Python 2's > ``str`` > type to ``bytes`` and ``bytearray`` [1]_ [2]_. > > > Rationale > ========= > > While interpolation is usually thought of as a string operation, there are > cases where interpolation on ``bytes`` or ``bytearrays`` make sense, and > the > work needed to make up for this missing functionality detracts from the > overall > readability of the code. > > > Motivation > ========== > > With Python 3 and the split between ``str`` and ``bytes``, one small but > important area of programming became slightly more difficult, and much more > painful -- wire format protocols [3]_. > > This area of programming is characterized by a mixture of binary data and > ASCII compatible segments of text (aka ASCII-encoded text). Bringing back > a > restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in > writing new wire format code, and in porting Python 2 wire format code. > > Common use-cases include ``dbf`` and ``pdf`` file formats, ``email`` > formats, and ``FTP`` and ``HTTP`` communications, among many others. > > > Proposed semantics for ``bytes`` and ``bytearray`` formatting > ============================================================= > > %-interpolation > --------------- > > All the numeric formatting codes (``d``, ``i``, ``o``, ``u``, ``x``, ``X``, > ``e``, ``E``, ``f``, ``F``, ``g``, ``G``, and any that are subsequently > added > to Python 3) will be supported, and will work as they do for str, including > the padding, justification and other related modifiers (currently ``#``, > ``0``, > ``-``, `` `` (space), and ``+`` (plus any added to Python 3)). The only > non-numeric codes allowed are ``c``, ``b``, ``a``, and ``s`` (which is a > synonym for b). > > For the numeric codes, the only difference between ``str`` and ``bytes`` > (or > ``bytearray``) interpolation is that the results from these codes will be > ASCII-encoded text, not unicode. In other words, for any numeric > formatting > code `%x`:: > > b"%x" % val > > is equivalent to:: > > ("%x" % val).encode("ascii") > > Examples:: > > >>> b'%4x' % 10 > b' a' > > >>> b'%#4x' % 10 > ' 0xa' > > >>> b'%04X' % 10 > '000A' > > ``%c`` will insert a single byte, either from an ``int`` in range(256), or > from > a ``bytes`` argument of length 1, not from a ``str``. > > Examples:: > > >>> b'%c' % 48 > b'0' > > >>> b'%c' % b'a' > b'a' > > ``%b`` will insert a series of bytes. These bytes are collected in one of > two > ways: > > - input type supports ``Py_buffer`` [4]_? > use it to collect the necessary bytes > > - input type is something else? > use its ``__bytes__`` method [5]_ ; if there isn't one, raise a > ``TypeError`` > > In particular, ``%b`` will not accept numbers nor ``str``. ``str`` is > rejected > as the string to bytes conversion requires an encoding, and we are > refusing to > guess; numbers are rejected because: > > - what makes a number is fuzzy (float? Decimal? Fraction? some user > type?) > > - allowing numbers would lead to ambiguity between numbers and textual > representations of numbers (3.14 vs '3.14') > > - given the nature of wire formats, explicit is definitely better than > implicit > > ``%s`` is included as a synonym for ``%b`` for the sole purpose of making > 2/3 code > bases easier to maintain. Python 3 only code should use ``%b``. > > Examples:: > > >>> b'%b' % b'abc' > b'abc' > > >>> b'%b' % 'some string'.encode('utf8') > b'some string' > > >>> b'%b' % 3.14 > Traceback (most recent call last): > ... > TypeError: b'%b' does not accept 'float' > > >>> b'%b' % 'hello world!' > Traceback (most recent call last): > ... > TypeError: b'%b' does not accept 'str' > > > ``%a`` will give the equivalent of > ``repr(some_obj).encode('ascii', 'backslashreplace')`` on the interpolated > value. Use cases include developing a new protocol and writing landmarks > into the stream; debugging data going into an existing protocol to see if > the problem is the protocol itself or bad data; a fall-back for a > serialization > format; or any situation where defining ``__bytes__`` would not be > appropriate > but a readable/informative representation is needed [6]_. > > Examples:: > > >>> b'%a' % 3.14 > b'3.14' > > >>> b'%a' % b'abc' > b"b'abc'" > > >>> b'%a' % 'def' > b"'def'" > > > Unsupported codes > ----------------- > > ``%r`` (which calls ``__repr__`` and returns a ``str``) is not supported. > > > Compatibility with Python 2 > =========================== > > As noted above, ``%s`` is being included solely to help ease migration > from, > and/or have a single code base with, Python 2. This is important as there > are modules both in the wild and behind closed doors that currently use the > Python 2 ``str`` type as a ``bytes`` container, and hence are using ``%s`` > as a bytes interpolator. > > However, ``%b`` should be used in new, Python 3 only code, so ``%s`` will > immediately be deprecated, but not removed until the next major Python > release. > > > Proposed variations > =================== > > It has been proposed to automatically use ``.encode('ascii','strict')`` for > ``str`` arguments to ``%b``. > > - Rejected as this would lead to intermittent failures. Better to have > the > operation always fail so the trouble-spot can be correctly fixed. > > It has been proposed to have ``%b`` return the ascii-encoded repr when the > value is a ``str`` (b'%b' % 'abc' --> b"'abc'"). > > - Rejected as this would lead to hard to debug failures far from the > problem > site. Better to have the operation always fail so the trouble-spot > can be > easily fixed. > > Originally this PEP also proposed adding format-style formatting, but it > was > decided that format and its related machinery were all strictly text (aka > ``str``) based, and it was dropped. > > Various new special methods were proposed, such as ``__ascii__``, > ``__format_bytes__``, etc.; such methods are not needed at this time, but > can > be visited again later if real-world use shows deficiencies with this > solution. > > A competing PEP, ``PEP 460 Add binary interpolation and formatting`` [7]_, > also exists. > > > Objections > ========== > > The objections raised against this PEP were mainly variations on two > themes: > > - the ``bytes`` and ``bytearray`` types are for pure binary data, with no > assumptions about encodings > > - offering %-interpolation that assumes an ASCII encoding will be an > attractive nuisance and lead us back to the problems of the Python 2 > ``str``/``unicode`` text model > > As was seen during the discussion, ``bytes`` and ``bytearray`` are also > used > for mixed binary data and ASCII-compatible segments: file formats such as > ``dbf`` and ``pdf``, network protocols such as ``ftp`` and ``email``, etc. > > ``bytes`` and ``bytearray`` already have several methods which assume an > ASCII > compatible encoding. ``upper()``, ``isalpha()``, and ``expandtabs()`` to > name > just a few. %-interpolation, with its very restricted mini-language, will > not > be any more of a nuisance than the already existing methods. > > Some have objected to allowing the full range of numeric formatting codes > with > the claim that decimal alone would be sufficient. However, at least two > formats (dbf and pdf) make use of non-decimal numbers. > > > Footnotes > ========= > > .. [1] http://docs.python.org/2/library/stdtypes.html#string-formatting > .. [2] neither string.Template, format, nor str.format are under > consideration > .. [3] https://mail.python.org/pipermail/python-dev/2014- > January/131518.html > .. [4] http://docs.python.org/3/c-api/buffer.html > examples: ``memoryview``, ``array.array``, ``bytearray``, ``bytes`` > .. [5] http://docs.python.org/3/reference/datamodel.html#object.__bytes__ > .. [6] https://mail.python.org/pipermail/python-dev/2014- > February/132750.html > .. [7] http://python.org/dev/peps/pep-0460/ > > > Copyright > ========= > > This document has been placed in the public domain. > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Mar 27 19:57:35 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 27 Mar 2014 11:57:35 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> Message-ID: <5334749F.8070703@stoneleaf.us> On 03/27/2014 11:41 AM, Guido van Rossum wrote: > Much better, but I'm still not happy with including %s at all. Otherwise it's accept-worthy. (How's that for pressure. :-) FWIW, I feel the same, but the need for compatible 2/3 code bases is real. Hey, how's this? We'll let %s in, but immediately deprecate it. ;) Of course, we won't remove it until Python IV. -- ~Ethan~ From ethan at stoneleaf.us Thu Mar 27 22:21:06 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 27 Mar 2014 14:21:06 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Request for Pronouncement In-Reply-To: References: <533484E9.3090300@stoneleaf.us> Message-ID: <53349642.1030301@stoneleaf.us> On 03/27/2014 01:44 PM, Guido van Rossum wrote: > > Accepted. Yay! > If you're going to commit another change, may I suggest to add, to the section stating that %r is not supported, that %a > is usually a suitable replacement for %r? Done. -- ~Ethan~ From josiah.carlson at gmail.com Thu Mar 27 22:52:59 2014 From: josiah.carlson at gmail.com (Josiah Carlson) Date: Thu, 27 Mar 2014 14:52:59 -0700 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: Hopping in to give my take on this, which I've expressed to Antoine off-list. When I first built the functionality about 8.5-9 years ago, I personally just wanted to be able to build something that could replace some of Expect: http://expect.sourceforge.net/ . The original and perhaps current API of the GSoC patch were inspired by my experience with asyncore (hence send() and recv() methods), but I never made an effort to get it practically working with asyncore - primarily because such would be functionally impossible on Windows without a lot of work to pull in a chunk of what was pywin32 libraries (at the time, Windows was a primary target). On the *nix side of things, performing the integration would be arguably trivial as select, poll, epoll, etc., all deal with pipes the same way as any other file handles (on-disk files, domain sockets, network sockets, etc.), with OS X being the exception. A little work would have been necessary to handle the two readable file handles and one writable file handle, but it's not that much different than building a proxy. But I digress. At this point I still believe that the functionality is useful from a scriptable interaction perspective, regardless of platform. I don't believe that being able to natively support the piping of output from one process to another is necessary, but a convenient future feature. That said, discussions about the quality of the existing GSoC patch and its API basically mean that the existing code to implement async subprocesses within the subprocess module precludes it from an easy or short acceptance process. And without substantial efforts from one or more people would doom the feature request and PEP to rejection. As an alternative, easily 95% of what most people would use this for can be written as an example using the asyncio module and included in the docs just after (or replacing) http://docs.python.org/3/library/asyncio-subprocess.html#example . Adding a reference to the subprocess module docs to point off to the asyncio subprocess example docs would get people a copy/paste snippet that they can include and update to their heart's content. Benefits to updating the docs: * It can happen at any time and doesn't need to wait for a 3.5 release (it can also happily wait) * No one likes maintaining code, but everyone loves docs (especially if it documents likely use-cases) * Because it is example docs, maybe a multi-week bikeshedding discussion about API doesn't need to happen (as long as "read line", "read X bytes", "read what is available", and "write this data" - all with timeouts - are shown, people can build everything else they want/need) * An example using asyncio is shorter than the modifications to the subprocess module * I would celebrate the closing of a feature request I opened in 2005 Aside from discarding code (Eric's and my own), not supporting Python-side chained pipes, and potentially angering some purists who *needed* this to be based on the subprocess module, I'm not sure I can think of any drawbacks. And arguably 2/3 of those drawbacks are imagined. Let me know your thoughts. If it gets an "okay", I'll come up with some example code, update the docs, and post a link to the code review in this thread. - Josiah On Wed, Mar 26, 2014 at 4:55 AM, Victor Stinner wrote: > Hi, > > For your information, asyncio.subprocess.Process is limited. It's not > possible yet to connect pipes between two processes. Something like > "cat | wc -l" where the cat stdin comes from Python. > > It's possible to enhance the API to implement that, but the timeframe > was too short to implement it before Python 3.4. > > Victor > > 2014-03-25 23:19 GMT+01:00 Antoine Pitrou : > > > > Hi, > > > > On core-mentorship someone asked about PEP 3145 - Asynchronous I/O for > > subprocess.popen. I answered that asyncio now has subprocess support > > (including non-blocking I/O on the three standard stream pipes), so > > it's not obvious anything else is needed. > > > > Should we change the PEP's status to Rejected or Superseded? > > > > Regards > > > > Antoine. > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/josiah.carlson%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Mar 27 23:05:55 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 27 Mar 2014 23:05:55 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> Message-ID: <20140327230555.571227d2@fsol> On Thu, 27 Mar 2014 18:47:59 +0000 Brett Cannon wrote: > On Thu Mar 27 2014 at 2:42:40 PM, Guido van Rossum wrote: > > > Much better, but I'm still not happy with including %s at all. Otherwise > > it's accept-worthy. (How's that for pressure. :-) > > > > But if we only add %b and leave out %s then how is this going to lead to > Python 2/3 compatible code since %b is not in Python 2? Or am I > misunderstanding you? I think we have reached a point where adding porting-related facilities in 3.5 may actually slow down the pace of porting, rather than accelerate it (because people will then wait for 3.5 to start porting stuff). Regards Antoine. From solipsis at pitrou.net Thu Mar 27 23:08:31 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 27 Mar 2014 23:08:31 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> <5334749F.8070703@stoneleaf.us> Message-ID: <20140327230831.3fe93056@fsol> On Thu, 27 Mar 2014 11:57:35 -0700 Ethan Furman wrote: > On 03/27/2014 11:41 AM, Guido van Rossum wrote: > > Much better, but I'm still not happy with including %s at all. Otherwise it's accept-worthy. (How's that for pressure. :-) > > FWIW, I feel the same, but the need for compatible 2/3 code bases is real. > > Hey, how's this? We'll let %s in, but immediately deprecate it. ;) Of course, we won't remove it until Python IV. I vote for an environment variable-controlled feature activation (or with a registry key under Windows) ;) Regards Antoine. From victor.stinner at gmail.com Fri Mar 28 00:11:00 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 28 Mar 2014 00:11:00 +0100 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: Hi, 2014-03-27 22:52 GMT+01:00 Josiah Carlson : > ... but I never made an effort to get it practically working > with asyncore - primarily because such would be functionally impossible on > Windows without a lot of work to pull in a chunk of what was pywin32 > libraries (at the time, Windows was a primary target). On the *nix side of > things, performing the integration would be arguably trivial as select, > poll, epoll, etc., all deal with pipes the same way as any other file > handles (on-disk files, domain sockets, network sockets, etc.), with OS X > being the exception. You should definitively take a look at asyncio. It handles sockets *and* pipes on all platforms, and even character devices (PTY) on some platforms. (The current status is still unclear to me, especially regarding to the "non blocking" flag of the PTY.) On Windows, asyncio uses IOCP. asyncio.subprocess solves also an old issue related to polling: subprocess.wait(timeout) uses polling because it was not possible to register an handler for SIGCHLD handler without breaking backward compatibility. asyncio supports also signals as well. > As an alternative, easily 95% of what most people would use this for can be > written as an example using the asyncio module and included in the docs just > after (or replacing) > http://docs.python.org/3/library/asyncio-subprocess.html#example . Adding a > reference to the subprocess module docs to point off to the asyncio > subprocess example docs would get people a copy/paste snippet that they can > include and update to their heart's content. Yeah, a link should be added from the subprocess module to the asyncio.subprocess module (the module, not the example). FYI the asyncore doc now has this note: "Note: This module exists for backwards compatibility only. For new code we recommend using asyncio." I opened the following issue for the "ls | wc -l" feature request: http://bugs.python.org/issue21080 Victor From victor.stinner at gmail.com Fri Mar 28 00:24:34 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 28 Mar 2014 00:24:34 +0100 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: 2014-03-27 22:52 GMT+01:00 Josiah Carlson : > * Because it is example docs, maybe a multi-week bikeshedding discussion > about API doesn't need to happen (as long as "read line", "read X bytes", > "read what is available", and "write this data" - all with timeouts - are > shown, people can build everything else they want/need) I don't understand this point. Using asyncio, you can read and write a single byte or a whole line. Using functions like asyncio.wait_for(), it's easy to add a timeout on such operation. Victor From kristjan at ccpgames.com Thu Mar 27 10:36:10 2014 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 27 Mar 2014 09:36:10 +0000 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <87vbv0mym5.fsf@elektro.pacujo.net> Message-ID: for our stackless socket framework we have the same issue. Windows provides an opaque timer system where a timer can be cancelled by handle. But on linux one has to craft one's own. One thing with this particular use case is that a heapq is overkill for network timer. . For network timers a granularity of one second should typically be sufficient. Thus, one can implement a map of future times (in quantisized time, e.g. whole seconds) to sets of timers. A timer is then keyed by its quantisized due time plus its callback. Cancellation can then be O(1). From: Python-Dev [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Guido van Rossum Sent: 26. mars 2014 21:42 To: Marko Rauhamaa Cc: Python-Dev Subject: Re: [Python-Dev] collections.sortedtree I haven't felt it, heapq feels natural to me for this use case. :-) I'm aware of the issue of frequent cancelled timers, but chose to wait and see rather than preemptively fix it (only so many hours in a day). IIRC pyftplib has a clever cleanup algorithm that we can easily add if that usage pattern becomes popular. On Wed, Mar 26, 2014 at 2:36 PM, Marko Rauhamaa > wrote: Guido van Rossum >: > Actually, the first step is publish it on PyPI, the second is to get a > fair number of happy users there. The bar for getting something > included into the stdlib is pretty high -- you need to demonstrate > that there is a need *and* that having it as a 3rd party module is a > problem. I hear you about the process. About the need part, I'm wondering if you haven't felt it in implementing the timers for asyncio. I have had that need in several network programming projects and have ended up using my AVL tree implementation (C and Python). Well, time will tell if frequent canceled timers end up piling up the heap queue. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josiah.carlson at gmail.com Fri Mar 28 02:16:50 2014 From: josiah.carlson at gmail.com (Josiah Carlson) Date: Thu, 27 Mar 2014 18:16:50 -0700 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: You don't understand the point because you don't understand the feature request or PEP. That is probably my fault for not communicating the intent better in the past. The feature request and PEP were written to offer something like the below (or at least enough that the below could be built with minimal effort): def do_login(...): proc = subprocess.Popen(...) current = proc.recv(timeout=5) last_line = current.rstrip().rpartition('\n')[-1] if last_line.endswith('login:'): proc.send(username) if proc.readline(timeout=5).rstrip().endswith('password:'): proc.send(password) if 'welcome' in proc.recv(timeout=5).lower(): return proc proc.kill() The API above can be very awkward (as shown :P ), but that's okay. From those building blocks a (minimally) enterprising user would add functionality to suit their needs. The existing subprocess module only offers two methods for *any* amount of communication over pipes with the subprocess: check_output() and communicate(), only the latter of which supports sending data (once, limited by system-level pipe buffer lengths). Neither allow for nontrivial interactions from a single subprocess.Popen() invocation. The purpose was to be able to communicate in a bidirectional manner with a subprocess without blocking, or practically speaking, blocking with a timeout. That's where the "async" term comes from. Again, there was never any intent to have the functionality be part of asyncore or any other asynchronous sockets framework, which is why there are no handle_*() methods, readable(), writable(), etc. Your next questions will be: But why bother at all? Why not just build the piece you need *inside* asyncio? Why does this need anything more? The answer to those questions are wants and needs. If I'm a user that needs interactive subprocess handling, I want to be able to do something like the code snippet above. The last thing I need is to have to rewrite the way my application/script/whatever handles *everything* just because a new asynchronous IO library has been included in the Python standard library - it's a bit like selling you a $300 bicycle when you need a $20 wheel for your scooter. That there *now* exists the ability to have async subprocesses as part of asyncio is a fortunate happenstance, as the necessary underlying tools for building the above now exist in the standard library. It's a matter of properly embedding the asyncio-related bits inside a handful of functions to provide something like the above, which is what I was offering to write. But why not keep working on the subprocess module? Yep. Tried that. Coming up on 9 years since I created the feature request and original Activestate recipe. To go that route is going to be 2-3 times as much work as has already been dedicated to get somewhere remotely acceptable for inclusion in Python 3.5, but more likely, subsequent rejection for similar reasons why it has been in limbo. But here's the thing: I can build enough using asyncio in 30-40 lines of Python to offer something like the above API. The problem is that it really has no natural home. It uses asyncio, so makes no sense to put in subprocess. It doesn't fit the typical asyncio behavior, so doesn't make sense to put in asyncio. The required functionality isn't big enough to warrant a submodule anywhere. Heck, it's even way too small to toss into an external PyPI module. But in the docs? It would show an atypical, but not wholly unreasonable use of asyncio (the existing example already shows what I would consider to be an atypical use of asyncio). It would provide a good starting point for someone who just wants/needs something like the snippet above. It is *yet another* use-case for asyncio. And it could spawn a larger library for offering a more fleshed-out subprocess-related API, though that is probably more wishful thinking on my part than anything. - Josiah On Thu, Mar 27, 2014 at 4:24 PM, Victor Stinner wrote: > 2014-03-27 22:52 GMT+01:00 Josiah Carlson : > > * Because it is example docs, maybe a multi-week bikeshedding discussion > > about API doesn't need to happen (as long as "read line", "read X bytes", > > "read what is available", and "write this data" - all with timeouts - are > > shown, people can build everything else they want/need) > > I don't understand this point. Using asyncio, you can read and write a > single byte or a whole line. Using functions like asyncio.wait_for(), > it's easy to add a timeout on such operation. > > Victor > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Mar 27 23:10:21 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 28 Mar 2014 11:10:21 +1300 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <20140327134854.DCC8025005B@webabinitio.net> References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <20140327134854.DCC8025005B@webabinitio.net> Message-ID: <5334A1CD.1010305@canterbury.ac.nz> R. David Murray wrote: > I've done the 'landmark' thing as well, in the string context; that can be > very useful when doing incremental test driven development. (Granted, you > could do that with __bytes__; Can't you do it more easily just by wrapping ascii() around the argument? That seems sufficient for debugging purposes to me. -- Greg From ethan at stoneleaf.us Fri Mar 28 03:09:58 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 27 Mar 2014 19:09:58 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <5334A1CD.1010305@canterbury.ac.nz> References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <20140327134854.DCC8025005B@webabinitio.net> <5334A1CD.1010305@canterbury.ac.nz> Message-ID: <5334D9F6.6010703@stoneleaf.us> On 03/27/2014 03:10 PM, Greg Ewing wrote: > R. David Murray wrote: >> I've done the 'landmark' thing as well, in the string context; that can be >> very useful when doing incremental test driven development. (Granted, you >> could do that with __bytes__; > > Can't you do it more easily just by wrapping ascii() > around the argument? That seems sufficient for debugging > purposes to me. The problem there is ascii() still returns unicode (okay, okay, str), so you still have to encode it. -- ~Ethan~ From tjreedy at udel.edu Fri Mar 28 03:18:10 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 27 Mar 2014 22:18:10 -0400 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: On 3/27/2014 9:16 PM, Josiah Carlson wrote: > You don't understand the point because you don't understand the feature > request or PEP. That is probably my fault for not communicating the > intent better in the past. The feature request and PEP were written to > offer something like the below (or at least enough that the below could > be built with minimal effort): > > def do_login(...): > proc = subprocess.Popen(...) > current = proc.recv(timeout=5) > last_line = current.rstrip().rpartition('\n')[-1] > if last_line.endswith('login:'): > proc.send(username) > if proc.readline(timeout=5).rstrip().endswith('password:'): > proc.send(password) > if 'welcome' in proc.recv(timeout=5).lower(): > return proc > proc.kill() > > The API above can be very awkward (as shown :P ), but that's okay. From > those building blocks a (minimally) enterprising user would add > functionality to suit their needs. The existing subprocess module only > offers two methods for *any* amount of communication over pipes with the > subprocess: check_output() and communicate(), only the latter of which > supports sending data (once, limited by system-level pipe buffer > lengths). Neither allow for nontrivial interactions from a single > subprocess.Popen() invocation. According to my reading of the doc, one should (in the absence of deadlocks, and without having timeouts) be able to use proc.stdin.write and proc.stdout.read. Do those not actually work? The purpose was to be able to communicate > in a bidirectional manner with a subprocess without blocking, or > practically speaking, blocking with a timeout. That's where the "async" > term comes from. Again, there was never any intent to have the > functionality be part of asyncore or any other asynchronous sockets > framework, which is why there are no handle_*() methods, readable(), > writable(), etc. > > Your next questions will be: But why bother at all? Why not just build > the piece you need *inside* asyncio? Why does this need anything more? > The answer to those questions are wants and needs. If I'm a user that > needs interactive subprocess handling, I want to be able to do something > like the code snippet above. The last thing I need is to have to rewrite > the way my application/script/whatever handles *everything* just because > a new asynchronous IO library has been included in the Python standard > library - it's a bit like selling you a $300 bicycle when you need a $20 > wheel for your scooter. > > That there *now* exists the ability to have async subprocesses as part > of asyncio is a fortunate happenstance, as the necessary underlying > tools for building the above now exist in the standard library. It's a > matter of properly embedding the asyncio-related bits inside a handful > of functions to provide something like the above, which is what I was > offering to write. But why not keep working on the subprocess module? > Yep. Tried that. Coming up on 9 years since I created the feature > request and original Activestate recipe. To go that route is going to be > 2-3 times as much work as has already been dedicated to get somewhere > remotely acceptable for inclusion in Python 3.5, but more likely, > subsequent rejection for similar reasons why it has been in limbo. > > But here's the thing: I can build enough using asyncio in 30-40 lines of > Python to offer something like the above API. The problem is that it > really has no natural home. It uses asyncio, so makes no sense to put in > subprocess. It doesn't fit the typical asyncio behavior, so doesn't make > sense to put in asyncio. The required functionality isn't big enough to > warrant a submodule anywhere. Heck, it's even way too small to toss into > an external PyPI module. But in the docs? It would show an atypical, but > not wholly unreasonable use of asyncio (the existing example already > shows what I would consider to be an atypical use of asyncio). It would > provide a good starting point for someone who just wants/needs something > like the snippet above. It is *yet another* use-case for asyncio. And it > could spawn a larger library for offering a more fleshed-out > subprocess-related API, though that is probably more wishful thinking on > my part than anything. > > - Josiah > > > > On Thu, Mar 27, 2014 at 4:24 PM, Victor Stinner > > wrote: > > 2014-03-27 22:52 GMT+01:00 Josiah Carlson >: > > * Because it is example docs, maybe a multi-week bikeshedding > discussion > > about API doesn't need to happen (as long as "read line", "read X > bytes", > > "read what is available", and "write this data" - all with > timeouts - are > > shown, people can build everything else they want/need) > > I don't understand this point. Using asyncio, you can read and write a > single byte or a whole line. Using functions like asyncio.wait_for(), > it's easy to add a timeout on such operation. > > Victor > > > > -- Terry Jan Reedy From stephen at xemacs.org Fri Mar 28 05:00:26 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 28 Mar 2014 13:00:26 +0900 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87ob0rotut.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > On 27 March 2014 18:11, Stephen J. Turnbull wrote: > > I don't think it's ever really been resolved whether they're a > > "typical case that won't go away" or a special group whose > > special needs should be considered. > That said, getting approval is definitely easier when the request is > to trust "the Python Software Foundation" vs "J. Random Programmer's > account on GitHub", so yes, it still counts in favour of sufficiently > compelling stdlib additions. I note that this *still* doesn't resolve the alternatives presented above. I hasten to add, it doesn't need to be done *now*. Yes, I do think it matters. Steve From stephen at xemacs.org Fri Mar 28 05:06:58 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 28 Mar 2014 13:06:58 +0900 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87mwgbotjx.fsf@uwakimon.sk.tsukuba.ac.jp> Maciej Fijalkowski writes: > On Thu, Mar 27, 2014 at 10:11 AM, Stephen J. Turnbull > wrote: > > Maybe. That depends on if you care about the convenience of folks who > > have to get new modules past Corporate Security, but it's easier to > > get an upgrade of the whole shebang. I don't think it's ever really > > been resolved whether they're a "typical case that won't go away" or a > > special group whose special needs should be considered. > > > > Steve > > And random pieces of C included in the standard library can be > shuffled under the carpet under the disguise of upgrade or what are > you suggesting? CPython doesn't do stuff like that; C code in CPython gets careful review, and can hardly be called "random". PyPI packages do do stuff like that. Not all of them by far, but to tell the difference you need to review line-by-line. So I'm simply saying (as Nick did) that it is often easier to get changes past a corporate bureaucracy if they are certified by inclusion in CPython (not to exclude PyPy or Jython, but the corporate consideration is about a specific distribution), than if it's a random package on PyPI. From stephen at xemacs.org Fri Mar 28 05:26:21 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 28 Mar 2014 13:26:21 +0900 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87lhvvosnm.fsf@uwakimon.sk.tsukuba.ac.jp> Maciej Fijalkowski writes: > I just find "my company is stupid I find labeling companies as "stupid", merely because they are cautious about what external code they allow their developers to depend on, unacceptable. And I'll go to the trouble of explaining why. > so let's work around it by putting stuff to python standard > library" unacceptable argument for python-dev and all the python > community. Then you're completely missing the point. There are two issues in including code in the Python standard library. The first is technical excellence. You need to hurdle a certain bar, or the code won't go in. Furthermore, this bar includes a comprehensive set of regression tests. This implies that (1) human review of the code is on average much easier for CPython stdlib code than for PyPI code, and (2) a claim that no relevant-to-the-company behavior has actually changed is (again, on average) much more verifiable and plausible for CPython than for PyPI code. This is at least some convenience for *all* users, and a near necessity for some strictly controlled environments. In the latter, the "gatekeeper" is all too likely to say "PyPI? Bring us a CTO signoff that 'this module is essential', or forget it". The second is a purely economic tradeoff: is the value of Python aggregated across all our users better enhanced by restricting additions to the stdlib, and thus reducing future maintenance effort and (presumably) increasing the rate at which Python code is improved, or by improving the batteries that are included (with the opposite effects)? That's a judgment call. From rosuav at gmail.com Fri Mar 28 05:36:42 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 28 Mar 2014 15:36:42 +1100 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <87lhvvosnm.fsf@uwakimon.sk.tsukuba.ac.jp> References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> <87lhvvosnm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Fri, Mar 28, 2014 at 3:26 PM, Stephen J. Turnbull wrote: > This is at least some convenience for *all* > users, and a near necessity for some strictly controlled environments. > In the latter, the "gatekeeper" is all too likely to say "PyPI? Bring > us a CTO signoff that 'this module is essential', or forget it". Don't forget, of course, that there is a middle ground. Something that's really REALLY awesome on PyPI but isn't in the stdlib might be packaged by various Linux distros. If you use Debian, typing "apt-cache pkgnames python-" (or using tab completion on an "apt-get install") will show you a veritable ton of packages, many of which are originally off PyPI. Even if you don't actually use that distribution, it might be worth telling the gatekeeper that Debian and/or Red Hat have indicated trust for some particular version of some particular package, which might help get over that hump. But it's still going to be a much bigger hump than "it's in CPython 3.4". ChrisA From josiah.carlson at gmail.com Fri Mar 28 06:09:47 2014 From: josiah.carlson at gmail.com (Josiah Carlson) Date: Thu, 27 Mar 2014 22:09:47 -0700 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: By digging into the internals of a subprocess produced by Popen(), you can write in a blocking manner to the stdin pipe, and read in a blocking manner from the stdout/stderr pipe(s). For scripting most command-line operations, the lack of timeouts and the ability to *stop* trying to read is as important as being able to spawn an external process. It kind-of kills that side of the usefulness of Python as a tool for scripting. The question is not whether or not a user of Python can dig into the internals, make some calls, then get it to be non-blocking - the existence of two different patches to do so (the most recent of which is from 4 1/2 years ago) shows that it *can* be done. The question is whether or not the desire for the functionality warrants having functions or methods to perform these operations in the standard library. I and others have claimed that it should go into the standard library. Heck, there was enough of a push that Eric got paid to write his version of the functionality for a GSoC project in 2009. There has even been activity on the bug itself unrelated to deferring discussions as recently as May 2012 (after which activity seems to have paused for reasons I don't know). Some people have raised reasonable questions about the API and implementation, but no one is willing to offer an alternative API that they think would be better, so discussions about implementation of a non-existent API for inclusion are moot. But honestly, I have approximately zero faith that what I say or do will lead to the inclusion of any changes to the subprocess module. Which is why I'm offering to write a short example that uses asyncio for inclusion in the docs. It's not what I've wanted for almost 9 years, but at least it has a chance of actually happening. I'll take a chance at updating the docs instead of a 3 to 9 month bikeshedding just to lead to rejection any day. So yeah. Someone want to make a decision? Tell me to write the docs, I will. Tell me to go take a long walk off a short pier, I'll thank you for your time and leave you alone. - Josiah On Thu, Mar 27, 2014 at 7:18 PM, Terry Reedy wrote: > On 3/27/2014 9:16 PM, Josiah Carlson wrote: > >> You don't understand the point because you don't understand the feature >> request or PEP. That is probably my fault for not communicating the >> intent better in the past. The feature request and PEP were written to >> offer something like the below (or at least enough that the below could >> be built with minimal effort): >> >> def do_login(...): >> proc = subprocess.Popen(...) >> current = proc.recv(timeout=5) >> last_line = current.rstrip().rpartition('\n')[-1] >> if last_line.endswith('login:'): >> proc.send(username) >> if proc.readline(timeout=5).rstrip().endswith('password:'): >> proc.send(password) >> if 'welcome' in proc.recv(timeout=5).lower(): >> return proc >> proc.kill() >> >> The API above can be very awkward (as shown :P ), but that's okay. From >> those building blocks a (minimally) enterprising user would add >> functionality to suit their needs. The existing subprocess module only >> offers two methods for *any* amount of communication over pipes with the >> subprocess: check_output() and communicate(), only the latter of which >> supports sending data (once, limited by system-level pipe buffer >> lengths). Neither allow for nontrivial interactions from a single >> subprocess.Popen() invocation. >> > > According to my reading of the doc, one should (in the absence of > deadlocks, and without having timeouts) be able to use proc.stdin.write and > proc.stdout.read. Do those not actually work? > > > > The purpose was to be able to communicate > >> in a bidirectional manner with a subprocess without blocking, or >> practically speaking, blocking with a timeout. That's where the "async" >> term comes from. Again, there was never any intent to have the >> functionality be part of asyncore or any other asynchronous sockets >> framework, which is why there are no handle_*() methods, readable(), >> writable(), etc. >> >> Your next questions will be: But why bother at all? Why not just build >> the piece you need *inside* asyncio? Why does this need anything more? >> The answer to those questions are wants and needs. If I'm a user that >> needs interactive subprocess handling, I want to be able to do something >> like the code snippet above. The last thing I need is to have to rewrite >> the way my application/script/whatever handles *everything* just because >> a new asynchronous IO library has been included in the Python standard >> library - it's a bit like selling you a $300 bicycle when you need a $20 >> wheel for your scooter. >> >> That there *now* exists the ability to have async subprocesses as part >> of asyncio is a fortunate happenstance, as the necessary underlying >> tools for building the above now exist in the standard library. It's a >> matter of properly embedding the asyncio-related bits inside a handful >> of functions to provide something like the above, which is what I was >> offering to write. But why not keep working on the subprocess module? >> Yep. Tried that. Coming up on 9 years since I created the feature >> request and original Activestate recipe. To go that route is going to be >> 2-3 times as much work as has already been dedicated to get somewhere >> remotely acceptable for inclusion in Python 3.5, but more likely, >> subsequent rejection for similar reasons why it has been in limbo. >> >> But here's the thing: I can build enough using asyncio in 30-40 lines of >> Python to offer something like the above API. The problem is that it >> really has no natural home. It uses asyncio, so makes no sense to put in >> subprocess. It doesn't fit the typical asyncio behavior, so doesn't make >> sense to put in asyncio. The required functionality isn't big enough to >> warrant a submodule anywhere. Heck, it's even way too small to toss into >> an external PyPI module. But in the docs? It would show an atypical, but >> not wholly unreasonable use of asyncio (the existing example already >> shows what I would consider to be an atypical use of asyncio). It would >> provide a good starting point for someone who just wants/needs something >> like the snippet above. It is *yet another* use-case for asyncio. And it >> could spawn a larger library for offering a more fleshed-out >> subprocess-related API, though that is probably more wishful thinking on >> my part than anything. >> >> - Josiah >> >> >> >> On Thu, Mar 27, 2014 at 4:24 PM, Victor Stinner >> > wrote: >> >> 2014-03-27 22:52 GMT+01:00 Josiah Carlson > >: >> >> > * Because it is example docs, maybe a multi-week bikeshedding >> discussion >> > about API doesn't need to happen (as long as "read line", "read X >> bytes", >> > "read what is available", and "write this data" - all with >> timeouts - are >> > shown, people can build everything else they want/need) >> >> I don't understand this point. Using asyncio, you can read and write a >> single byte or a whole line. Using functions like asyncio.wait_for(), >> it's easy to add a timeout on such operation. >> >> Victor >> >> >> >> >> > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > josiah.carlson%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Fri Mar 28 08:52:59 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 28 Mar 2014 16:52:59 +0900 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> <87lhvvosnm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87ha6ipxno.fsf@uwakimon.sk.tsukuba.ac.jp> Chris Angelico writes: > Don't forget, of course, that there is a middle ground. Something > that's really REALLY awesome on PyPI but isn't in the stdlib might be > packaged by various Linux distros. Oh, agreed, and any organization that cares that much will already have the RHEL or Ubuntu LTS (etc, those are the ones my buddies use) contract. CPython is not the only certification that is trusted (to whatever degree) by users of CPython. But forget Debian and Gentoo at least -- all it takes get in to those distros is a license (free) and a maintainer (at application time). And they'll even wink at the license given sufficient user demand. Steve From p.f.moore at gmail.com Fri Mar 28 09:09:45 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 28 Mar 2014 08:09:45 +0000 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: On 28 March 2014 05:09, Josiah Carlson wrote: > So yeah. Someone want to make a decision? Tell me to write the docs, I will. > Tell me to go take a long walk off a short pier, I'll thank you for your > time and leave you alone. I had a need for this a few years ago. It's messy to do on Windows (ctypes callouts to PeekNamedPipe to check if you can read from the process without blocking). So I would like to see a recipe for this, (even if it's likely to be another few years before I ever need it again :-)). Paul From victor.stinner at gmail.com Fri Mar 28 11:20:14 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 28 Mar 2014 11:20:14 +0100 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: 2014-03-28 2:16 GMT+01:00 Josiah Carlson : > def do_login(...): > proc = subprocess.Popen(...) > current = proc.recv(timeout=5) > last_line = current.rstrip().rpartition('\n')[-1] > if last_line.endswith('login:'): > proc.send(username) > if proc.readline(timeout=5).rstrip().endswith('password:'): > proc.send(password) > if 'welcome' in proc.recv(timeout=5).lower(): > return proc > proc.kill() I don't understand this example. How is it "asynchronous"? It looks like blocking calls. In my definition, asynchronous means that you can call this function twice on two processes, and they will run in parallel. Using greenlet/eventlet, you can write code which looks blocking, but runs asynchronously. But I don't think that you are using greenlet or eventlet here. I take a look at the implementation: http://code.google.com/p/subprocdev/source/browse/subprocess.py It doesn't look portable. On Windows, WriteFile() is used. This function is blocking, or I missed something huge :-) It's much better if a PEP is portable. Adding time.monotonic() only to Linux would make the PEP 418 much shorter (4 sentences instead of 10 pages? :-))! The implementation doesn't look reliable: def get_conn_maxsize(self, which, maxsize): # Not 100% certain if I get how this works yet. if maxsize is None: maxsize = 1024 ... This constant 1024 looks arbitrary. On UNIX, a write into a pipe may block with less bytes (512 bytes). asyncio has a completly different design. On Windows, it uses overlapped operations with IOCP event loop. Such operation can be cancelled. Windows cares of the buffering. On UNIX, non-blocking mode is used with select() (or something faster like epoll) and asyncio retries to write more data when the pipe (or any file descriptor used for process stdin/stdoud/stderr) becomes ready (for reading/writing). asyncio design is more reliable and portable. I don't see how you can implement asynchronous communication with a subprocess without the complex machinery of an event loop. > The API above can be very awkward (as shown :P ), but that's okay. From > those building blocks a (minimally) enterprising user would add > functionality to suit their needs. The existing subprocess module only > offers two methods for *any* amount of communication over pipes with the > subprocess: check_output() and communicate(), only the latter of which > supports sending data (once, limited by system-level pipe buffer lengths). As I wrote, it's complex to handle non-blocking file descriptors. You have to catch EWOULDBLOCK and retries later when the file descriptor becomes ready. The main thread has to watch for such event on the file descriptor, or you need a dedicated thread. By the way, subprocess.communicate() is currently implemented using threads on Windows. > Neither allow for nontrivial interactions from a single subprocess.Popen() > invocation. The purpose was to be able to communicate in a bidirectional > manner with a subprocess without blocking, or practically speaking, blocking > with a timeout. That's where the "async" term comes from. I call this "non-blocking functions", not "async functions". It's quite simple to check if a read will block on not on UNIX. It's more complex to implement it on Windows. And even more complex to handle to add a buffer to write(). > Your next questions will be: But why bother at all? Why not just build the > piece you need *inside* asyncio? Why does this need anything more? The > answer to those questions are wants and needs. If I'm a user that needs > interactive subprocess handling, I want to be able to do something like the > code snippet above. The last thing I need is to have to rewrite the way my > application/script/whatever handles *everything* just because a new > asynchronous IO library has been included in the Python standard library - > it's a bit like selling you a $300 bicycle when you need a $20 wheel for > your scooter. You don't have to rewrite your whole application. If you only want to use asyncio event loop in a single function, you can use loop.run_until_complete(do_login) which blocks until the function completes. The "function" is an asynchronous coroutine in fact. Full example of asynchronous communication with a subprocess (the python interactive interpreter) using asyncio high-level API: --- import asyncio.subprocess import time import sys @asyncio.coroutine def eval_python_async(command, encoding='ascii', loop=None): proc = yield from asyncio.subprocess.create_subprocess_exec( sys.executable, "-u", "-i", stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT, loop=loop) # wait for the prompt buffer = bytearray() while True: data = yield from proc.stdout.read(100) buffer.extend(data) if buffer.endswith(b'>>> '): break proc.stdin.write(command.encode(encoding) + b"\n") yield from proc.stdin.drain() proc.stdin.close() output = yield from proc.stdout.read() output = output.decode(encoding) output = output.rstrip() if output.endswith('>>>'): output = output[:-3].rstrip() return output def eval_python(command, timeout=None): loop = asyncio.get_event_loop() task = asyncio.Task(eval_python_async(command, loop=loop), loop=loop) return loop.run_until_complete(asyncio.wait_for(task, timeout)) def test_sequential(nproc, command): t0 = time.monotonic() for index in range(nproc): eval_python(command) return time.monotonic() - t0 def test_parallel(nproc, command): loop = asyncio.get_event_loop() tasks = [asyncio.Task(eval_python_async(command, loop=loop), loop=loop) for index in range(nproc)] t0 = time.monotonic() loop.run_until_complete(asyncio.wait(tasks)) return time.monotonic() - t0 print("1+1 = %r" % eval_python("1+1", timeout=1.0)) slow_code = "import math; print(str(math.factorial(20000)).count('7'))" dt = test_sequential(10, slow_code) print("Run 10 tasks in sequence: %.1f sec" % dt) dt2 = test_parallel(10, slow_code) print("Run 10 tasks in parallel: %.1f sec (speed=%.1f)" % (dt2, dt/dt2)) # cleanup asyncio asyncio.get_event_loop().close() --- Output: --- 1+1 = '2' Run 10 tasks in sequence: 2.8 sec Run 10 tasks in parallel: 0.6 sec (speed=4.6) --- (My CPU has 8 cores, the speed may be lower on other computers with fewer cores.) Even if eval_python_async() is asynchronous, eval_python() function is blocking so you can write: print("1+1 = %r" % eval_python("1+1")) without callback nor "yield from". Running tasks in parallel is faster than running them in sequence (almost 5 times faster on my PC). The syntax in eval_python_async() is close to the API you proposed, except that you have to add "yield from" in front of "blocking" functions like read() or drain() (it's the function to flush the stdin buffer, I'm not sure that it is needed in this example). The timeout is on the whole eval_python_async(), but you can as well using finer timeout on each read/write. > But here's the thing: I can build enough using asyncio in 30-40 lines of > Python to offer something like the above API. The problem is that it really > has no natural home. I agree that writing explicit asynchronous code is more complex than using eventlet. Asynchronous programming is hard. > But in the docs? It would show an atypical, but not > wholly unreasonable use of asyncio (the existing example already shows what > I would consider to be an atypical use of asyncio). The asyncio documentation is still a work-in-progress. I tried to document all APIs, but there are too few examples and the documentation is still focused on the API instead of being oriented to the user of the API. Don't hesitate to contribute to the documentation! We can probably write a simple example showing how to interact with an interactive program like Python. Victor From p.f.moore at gmail.com Fri Mar 28 11:28:30 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 28 Mar 2014 10:28:30 +0000 Subject: [Python-Dev] Jython site now seems to redirect to docs.python.org??? Message-ID: I'm not sure if this is a result of the recent website reorg, but www.jython.org seems to be redirecting to docs.python.org for me. Presumably this is an error - where do I report it and/or is it a known issue? Thanks, Paul. From ncoghlan at gmail.com Fri Mar 28 11:49:54 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 Mar 2014 20:49:54 +1000 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: On 28 March 2014 20:20, Victor Stinner wrote: > 2014-03-28 2:16 GMT+01:00 Josiah Carlson : >> def do_login(...): >> proc = subprocess.Popen(...) >> current = proc.recv(timeout=5) >> last_line = current.rstrip().rpartition('\n')[-1] >> if last_line.endswith('login:'): >> proc.send(username) >> if proc.readline(timeout=5).rstrip().endswith('password:'): >> proc.send(password) >> if 'welcome' in proc.recv(timeout=5).lower(): >> return proc >> proc.kill() > > I don't understand this example. How is it "asynchronous"? It looks > like blocking calls. In my definition, asynchronous means that you can > call this function twice on two processes, and they will run in > parallel. Without reading all the reference from PEP 3145 again, I now seem to recall the problem it was aimed at was the current deadlock warnings in the subprocess docs - if you're not careful to make sure you keep reading from the stdout and stderr pipes while writing to stdin, you can fill up the kernel buffers and deadlock while communicating with the subprocess. So the "asynchronous" part is to be able to happily write large amounts of data to a stdin pipe without fear of deadlock with a subprocess that has just written large amounts of data to the stdout or stderr pipes. So, from the perspective of the user, it behaves like a synchronous blocking operation, but on the backend it needs to use asynchronous read and write operations to avoid deadlock. I suspect it would likely be a relatively thin wrapper around run_until_complete(). Also, as far as where such functionality should live in the standard library could go, it's entirely possible for it to live in its natural home of "subprocess". To make that work, the core subprocess.Popen functionality would need to be moved to a _subprocess module, and then both subprocess and asyncio would depend on that, allowing subprocess to also depend on asyncio without creating a circular import. So I'll go back on my original comment - assuming I've now remembered its intended effects correctly PEP 3145 remains a valid proposal, independent of (but potentially relying on) asyncio, as the problem it is designed to solve is all those notes like "Do not use stdout=PIPE or stderr=PIPE with this function. As the pipes are not being read in the current process, the child process may block if it generates enough output to a pipe to fill up the OS pipe buffer." in the current subprocess module by using an asynchronous backend while still presenting a synchronous API. And rather than adding a new API, I'd hope it could propose just getting rid of those warnings by reimplementing the current deadlock prone APIs on top of "run_until_complete()" and exploring the potential consequences for backwards compatibility. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Mar 28 11:53:57 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 Mar 2014 20:53:57 +1000 Subject: [Python-Dev] Jython site now seems to redirect to docs.python.org??? In-Reply-To: References: Message-ID: I'm not sure where the responsibilities of the redesign team end and those of the infrastructure team start, but since the switch to the new site I've been adding anything related to the website parts of python.org to https://github.com/python/pythondotorg/issues If that's not the right place, then I'd also like to know the answer to Paul's "where do I report it?" question :) Cheers, Nick. From p.f.moore at gmail.com Fri Mar 28 11:57:25 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 28 Mar 2014 10:57:25 +0000 Subject: [Python-Dev] Jython site now seems to redirect to docs.python.org??? In-Reply-To: References: Message-ID: On 28 March 2014 10:53, Nick Coghlan wrote: > I'm not sure where the responsibilities of the redesign team end and > those of the infrastructure team start, but since the switch to the > new site I've been adding anything related to the website parts of > python.org to https://github.com/python/pythondotorg/issues Thanks, Nick - I've just raised an issue on there. Paul From steve at pearwood.info Fri Mar 28 12:03:42 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 28 Mar 2014 22:03:42 +1100 Subject: [Python-Dev] Negative timedelta strings Message-ID: <20140328110342.GI16526@ando> There's a furious discussion going on at the python-list mailing list, about negative timedelta strings: py> str(timedelta(0, -1)) '-1 day, 23:59:59' This is documented. It's even documented as being "somewhat unusual". I found a tracker item for it, back in July 2010: http://bugs.python.org/issue9430 and that refers back to a discussion on #python-dev, but (as far as I know) that's not archived anywhere. I had a look on the python-dev mailing list around that time, and couldn't find anything. Does anyone remember the rationale for this behaviour? Is it open to debate or is it now cast in stone? (Personally, I'm not looking to reopen the debate, and if I did, it would be on python-ideas.) Any links to the earlier discussion would be helpful. Thanks, -- Steve From solipsis at pitrou.net Fri Mar 28 12:06:40 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 28 Mar 2014 12:06:40 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <20140327134854.DCC8025005B@webabinitio.net> <5334A1CD.1010305@canterbury.ac.nz> <5334D9F6.6010703@stoneleaf.us> Message-ID: <20140328120640.41ec69fd@fsol> On Thu, 27 Mar 2014 19:09:58 -0700 Ethan Furman wrote: > On 03/27/2014 03:10 PM, Greg Ewing wrote: > > R. David Murray wrote: > >> I've done the 'landmark' thing as well, in the string context; that can be > >> very useful when doing incremental test driven development. (Granted, you > >> could do that with __bytes__; > > > > Can't you do it more easily just by wrapping ascii() > > around the argument? That seems sufficient for debugging > > purposes to me. > > The problem there is ascii() still returns unicode (okay, okay, str), so you still have to encode it. ascii(my_object).encode() Regards Antoine. From solipsis at pitrou.net Fri Mar 28 12:12:07 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 28 Mar 2014 12:12:07 +0100 Subject: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"] References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> <20140326172512.22e1f21a@anarchist.wooz.org> <1395869274.5751.99327085.56513D6F@webmail.messagingengine.com> Message-ID: <20140328121207.7dab5075@fsol> On Thu, 27 Mar 2014 20:32:02 +1000 Nick Coghlan wrote: > > Most of the time when I hear people say "the PEP process is too > difficult", I eventually find that what they really mean is "learning > the kinds of things that python-dev are likely to be worried about, > and ensuring that the PEP adequately addresses their concerns, and > listening to feedback, and reconsidering what I actually want, and > revising my proposal, such that they eventually say yes is too time > consuming". Well, the PEP process *is* difficult and not only because you have to learn the kinds of things that python-dev are likely to be worried about. Getting a PEP accepted for a feature is much more work than getting a feature accepted in the bug tracker. Regards Antoine. From ncoghlan at gmail.com Fri Mar 28 12:18:58 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 Mar 2014 21:18:58 +1000 Subject: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"] In-Reply-To: <20140328121207.7dab5075@fsol> References: <8738i4og5f.fsf@elektro.pacujo.net> <1395867300.29596.99313629.4B080CF9@webmail.messagingengine.com> <20140326172512.22e1f21a@anarchist.wooz.org> <1395869274.5751.99327085.56513D6F@webmail.messagingengine.com> <20140328121207.7dab5075@fsol> Message-ID: On 28 March 2014 21:12, Antoine Pitrou wrote: > On Thu, 27 Mar 2014 20:32:02 +1000 > Nick Coghlan wrote: >> >> Most of the time when I hear people say "the PEP process is too >> difficult", I eventually find that what they really mean is "learning >> the kinds of things that python-dev are likely to be worried about, >> and ensuring that the PEP adequately addresses their concerns, and >> listening to feedback, and reconsidering what I actually want, and >> revising my proposal, such that they eventually say yes is too time >> consuming". > > Well, the PEP process *is* difficult and not only because you have to > learn the kinds of things that python-dev are likely to be worried > about. Getting a PEP accepted for a feature is much more work than > getting a feature accepted in the bug tracker. Oh, agreed. It's only the "too" qualifier that I question - I'm not sure how much easier we could make it before it ceased to serve its filtering purpose. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Fri Mar 28 12:24:23 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 28 Mar 2014 07:24:23 -0400 Subject: [Python-Dev] Jython site now seems to redirect to docs.python.org??? In-Reply-To: References: Message-ID: <2F7A7239-EF60-4382-AB64-124816FD9206@stufft.io> On Mar 28, 2014, at 6:57 AM, Paul Moore wrote: > On 28 March 2014 10:53, Nick Coghlan wrote: >> I'm not sure where the responsibilities of the redesign team end and >> those of the infrastructure team start, but since the switch to the >> new site I've been adding anything related to the website parts of >> python.org to https://github.com/python/pythondotorg/issues > > Thanks, Nick - I've just raised an issue on there. > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Probably infrastructure-staff at python.org ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Fri Mar 28 12:37:57 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 28 Mar 2014 11:37:57 +0000 Subject: [Python-Dev] Jython site now seems to redirect to docs.python.org??? In-Reply-To: <2F7A7239-EF60-4382-AB64-124816FD9206@stufft.io> References: <2F7A7239-EF60-4382-AB64-124816FD9206@stufft.io> Message-ID: On 28 March 2014 11:24, Donald Stufft wrote: > Probably infrastructure-staff at python.org OK, I've emailed them as well. Paul From kristjan at ccpgames.com Fri Mar 28 10:20:35 2014 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Fri, 28 Mar 2014 09:20:35 +0000 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <20140327165315.41c8f894@fsol> References: <8738i4og5f.fsf@elektro.pacujo.net> <20140326222628.24336818@fsol> <87ppl8mycp.fsf@elektro.pacujo.net> <20140326224848.1e457671@fsol> <87a9ccmxmg.fsf@elektro.pacujo.net> <20140326230247.084e422d@fsol> <20140327165315.41c8f894@fsol> Message-ID: > -----Original Message----- > From: Python-Dev [mailto:python-dev- > bounces+kristjan=ccpgames.com at python.org] On Behalf Of Antoine Pitrou > Sent: 27. mars 2014 15:53 > To: python-dev at python.org > Subject: Re: [Python-Dev] collections.sortedtree > > On Thu, 27 Mar 2014 08:50:01 -0700 > Daniel Stutzbach wrote: > > To provide efficient cancellation and removal, a heap implementation > > needs some way to efficiently answer "What is the current index of this > item?". > > There are a couple of ways to achieve that, but they all require more > > storage than heapq's list-based approach. > > You are right. I was assuming the index is already known. > > Regards > > Antoine. Yes. But for rare occurrances, it is annoying that heap.remove(item) is more expensive than it needs to be. It is a reasonable operation, just like list.remove() is. I'll be willing to experiment with extending the heapq. methods to take an optional "map" argument. 'map' would be a dict, mapping objects in the heap to indices. If provided, each of the heapq methouds would take care to update the map of any objects that it touches with the current index of the object. Usage could be something like: heap = [] map = {} def insert(i): heapq.heappush(heap, I, map) def pop(i): return heapq.heappop(heap, map) def remove(i): heapq.heapdel(heap, map[i], map) K From benjamin at python.org Fri Mar 28 14:46:06 2014 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 28 Mar 2014 06:46:06 -0700 Subject: [Python-Dev] Jython site now seems to redirect to docs.python.org??? In-Reply-To: References: Message-ID: <1396014366.3065.100025309.6E711DD2@webmail.messagingengine.com> On Fri, Mar 28, 2014, at 3:28, Paul Moore wrote: > I'm not sure if this is a result of the recent website reorg, but > www.jython.org seems to be redirecting to docs.python.org for me. > Presumably this is an error - where do I report it and/or is it a > known issue? This was my fault, though maybe apache 2.2's weird virtual host selection rules can share some of the blame. Fixed now. From p.f.moore at gmail.com Fri Mar 28 14:58:55 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 28 Mar 2014 13:58:55 +0000 Subject: [Python-Dev] Jython site now seems to redirect to docs.python.org??? In-Reply-To: <1396014366.3065.100025309.6E711DD2@webmail.messagingengine.com> References: <1396014366.3065.100025309.6E711DD2@webmail.messagingengine.com> Message-ID: On 28 March 2014 13:46, Benjamin Peterson wrote: > This was my fault, though maybe apache 2.2's weird virtual host > selection rules can share some of the blame. Fixed now. Thanks Paul From andrew.svetlov at gmail.com Fri Mar 28 15:29:58 2014 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Fri, 28 Mar 2014 16:29:58 +0200 Subject: [Python-Dev] cpython: inspect.Signature: Add 'Signature.from_callable' classmethod. Closes #17373 In-Reply-To: <20140327174816.25e87c40@fsol> References: <3fvrmy6cvPz7Ll5@mail.python.org> <20140327174816.25e87c40@fsol> Message-ID: And probably the block should be deindented On Thu, Mar 27, 2014 at 6:48 PM, Antoine Pitrou wrote: > On Thu, 27 Mar 2014 17:12:02 +0100 (CET) > yury.selivanov wrote: >> >> + .. classmethod:: Signature.from_callable(obj) >> + >> + Return a :class:`Signature` (or its subclass) object for a given callable >> + ``obj``. This method simplifies subclassing of :class:`Signature`: >> + >> + :: >> + >> + class MySignature(Signature): >> + pass >> + sig = MySignature.from_callable(min) >> + assert isinstance(sig, MySignature) >> + > > This needs a "versionadded" tag. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com -- Thanks, Andrew Svetlov From tseaver at palladion.com Fri Mar 28 15:45:01 2014 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 28 Mar 2014 10:45:01 -0400 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/27/2014 09:16 PM, Josiah Carlson wrote: > But here's the thing: I can build enough using asyncio in 30-40 lines > of Python to offer something like the above API. The problem is that > it really has no natural home. It uses asyncio, so makes no sense to > put in subprocess. It doesn't fit the typical asyncio behavior, so > doesn't make sense to put in asyncio. The required functionality isn't > big enough to warrant a submodule anywhere. Heck, it's even way too > small to toss into an external PyPI module. Seems perfect for the Cheesehop to me. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlM1iucACgkQ+gerLs4ltQ4RygCfQOjBD7jTZU5ILub/sKxGYqH8 8v8AoKkv2ePkRn3X43CpGBQNeB9uNufQ =xgSe -----END PGP SIGNATURE----- From tseaver at palladion.com Fri Mar 28 16:01:45 2014 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 28 Mar 2014 11:01:45 -0400 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/27/2014 04:11 AM, Stephen J. Turnbull wrote: > Maybe. That depends on if you care about the convenience of folks > who have to get new modules past Corporate Security, but it's easier > to get an upgrade of the whole shebang. I don't think it's ever > really been resolved whether they're a "typical case that won't go > away" or a special group whose special needs should be considered. ISTM that such concerns should be addressed via some kind of paid support contract (a la RHEL), and not used to drive decisions for the underlying FLOSS project. The existence of aggregated resources from such a support organization would then be relevant to the "include / exclude" discussion: presumably the support organization could fund the maintenance of the otherwise-excluded module based on its customers' paid-for needs. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlM1jtQACgkQ+gerLs4ltQ5OOgCdHeOjBjpfJ1w5mkAWZsajflWK U3wAmgIxnc7BFIaoouQ0kdkCgoF+lMr3 =yhYm -----END PGP SIGNATURE----- From steve at pearwood.info Fri Mar 28 16:02:59 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 29 Mar 2014 02:02:59 +1100 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <20140326222628.24336818@fsol> <87ppl8mycp.fsf@elektro.pacujo.net> <20140326224848.1e457671@fsol> <87a9ccmxmg.fsf@elektro.pacujo.net> <20140326230247.084e422d@fsol> <20140327165315.41c8f894@fsol> Message-ID: <20140328150258.GJ16526@ando> On Fri, Mar 28, 2014 at 09:20:35AM +0000, Kristj?n Valur J?nsson wrote: > I'll be willing to experiment with extending the heapq. methods to take an optional "map" argument. > 'map' would be a dict, mapping objects in the heap to indices. If provided, each of the heapq methouds would > take care to update the map of any objects that it touches with the current index of the object. > > Usage could be something like: > > heap = [] > map = {} > def insert(i): > heapq.heappush(heap, I, map) > > def pop(i): > return heapq.heappop(heap, map) > > def remove(i): > heapq.heapdel(heap, map[i], map) If you're going to make heapq more complex, wouldn't it be better to encapsulate the logic into a Heap class? heapq should remain the same, and Heap could (if accepted) move into collections. And should this discussion move to python-ideas? -- Steven From rdmurray at bitdance.com Fri Mar 28 16:08:02 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 28 Mar 2014 11:08:02 -0400 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: <20140328150802.A9CAE250CDD@webabinitio.net> On Fri, 28 Mar 2014 10:45:01 -0400, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 03/27/2014 09:16 PM, Josiah Carlson wrote: > > But here's the thing: I can build enough using asyncio in 30-40 lines > > of Python to offer something like the above API. The problem is that > > it really has no natural home. It uses asyncio, so makes no sense to > > put in subprocess. It doesn't fit the typical asyncio behavior, so > > doesn't make sense to put in asyncio. The required functionality isn't > > big enough to warrant a submodule anywhere. Heck, it's even way too > > small to toss into an external PyPI module. > > Seems perfect for the Cheesehop to me. Indeed. I heard a rumor[*] that there's at least one package in the cheeseshop that consists of a one-liner. On the other hand, we have multiprocessing examples in the docs that are longer than that, so it sounds like a great asyncio example to me, especially given that Victor says we don't have enough examples yet. --David [*] https://pypi.python.org/pypi/first It's not *actually* a one liner, but you could write it as one, and the actual code isn't much heavier :) From aponomarenko at rosalab.ru Fri Mar 28 09:31:59 2014 From: aponomarenko at rosalab.ru (Andrey Ponomarenko) Date: Fri, 28 Mar 2014 12:31:59 +0400 Subject: [Python-Dev] libpython added to ABI tracker Message-ID: <5335337F.8000303@rosalab.ru> Hi, The libpython library has been added to the ABI tracker: http://upstream-tracker.org/versions/python.html The page lists library versions and changes in API/ABI. -- Andrey Ponomarenko, NTC IT ROSA. From victor.stinner at gmail.com Fri Mar 28 16:25:28 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 28 Mar 2014 16:25:28 +0100 Subject: [Python-Dev] libpython added to ABI tracker In-Reply-To: <5335337F.8000303@rosalab.ru> References: <5335337F.8000303@rosalab.ru> Message-ID: Hi, 2014-03-28 9:31 GMT+01:00 Andrey Ponomarenko : > The libpython library has been added to the ABI tracker: > http://upstream-tracker.org/versions/python.html > > The page lists library versions and changes in API/ABI. Nice! By the way, would it be possible to add a second page for the stable ABI of Python? Victor From stephen at xemacs.org Fri Mar 28 16:57:21 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 29 Mar 2014 00:57:21 +0900 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87vbuynwny.fsf@uwakimon.sk.tsukuba.ac.jp> Tres Seaver writes: > On 03/27/2014 04:11 AM, Stephen J. Turnbull wrote: > > > Maybe. That depends on if you care about the convenience of folks > > who have to get new modules past Corporate Security, but it's easier > > to get an upgrade of the whole shebang. I don't think it's ever > > really been resolved whether they're a "typical case that won't go > > away" or a special group whose special needs should be considered. > > ISTM that such concerns should be addressed via some kind of paid support > contract (a la RHEL), and not used to drive decisions for the underlying > FLOSS project. The existence of aggregated resources from such a support > organization would then be relevant to the "include / exclude" > discussion: presumably the support organization could fund the > maintenance of the otherwise-excluded module based on its customers' > paid-for needs. So, let me get this straight: you think that one user should pay Red Hat to vet the package for RHEL, and another user should pay to get it into Ubuntu, and another user to get it into SuSE? And then the distros should all pay lawyers to write contracts to make sure that nobody is paying too much for support in the stdlib when they eventually get it in? (Except the customers, of course, everybody will be happy if *they* pay more than they need to.) Seems to me that putting it into stdlib in the first place makes more sense, if it's going to end up there at all. Another way to put it is, we need a better way to fund support of "routine maintenance" (ie, the unfun parts) than negotiating it module by module. From alexander.belopolsky at gmail.com Fri Mar 28 17:07:51 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 28 Mar 2014 12:07:51 -0400 Subject: [Python-Dev] Negative timedelta strings In-Reply-To: <20140328110342.GI16526@ando> References: <20140328110342.GI16526@ando> Message-ID: On Fri, Mar 28, 2014 at 7:03 AM, Steven D'Aprano wrote: > py> str(timedelta(0, -1)) > '-1 day, 23:59:59' > .. > Does anyone remember the rationale for this behaviour? I don't recall any better rationale than what I wrote in the docs: "String representations of timedelta objects are normalized similarly to their internal representation." > Is it open to debate or is it now cast in stone? I think the barrier for changing str() is lower than that for changing repr(), but I would be against any changes in this area. (I may have had a different view if ISO 8601 syntax for timedeltas was not so ugly. :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Fri Mar 28 17:18:56 2014 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 28 Mar 2014 12:18:56 -0400 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <87vbuynwny.fsf@uwakimon.sk.tsukuba.ac.jp> References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> <87vbuynwny.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/28/2014 11:57 AM, Stephen J. Turnbull wrote: > So, let me get this straight: you think that one user should pay Red > Hat to vet the package for RHEL, and another user should pay to get > it into Ubuntu, and another user to get it into SuSE? And then the > distros should all pay lawyers to write contracts to make sure that > nobody is paying too much for support in the stdlib when they > eventually get it in? (Except the customers, of course, everybody > will be happy if *they* pay more than they need to.) No, I'm arguing that *if* the concerns you articulate represent a significant number of "corporate* customers (i.e., a "market"), their interests would be better represented by *some* organization who is paid to reflect them. RHEL and Ubuntu LTS could potentially be contributors to that pool. I'm mostly arguing the FLOSS project should feel free to ignore high-maintenance-cost commercial concerns until those concerns bring either blook (funded developer time) or treasure (pooled to pay for the same time) to the table to pay for them. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlM1oPAACgkQ+gerLs4ltQ52ZgCg06AobjcZa1lGBDtFzRk6IjEK 6DkAnj33xAkqcDUjLGaT9NP4YtZdkAos =62VL -----END PGP SIGNATURE----- From skip at pobox.com Fri Mar 28 17:19:52 2014 From: skip at pobox.com (Skip Montanaro) Date: Fri, 28 Mar 2014 11:19:52 -0500 Subject: [Python-Dev] Negative timedelta strings In-Reply-To: References: <20140328110342.GI16526@ando> Message-ID: On Fri, Mar 28, 2014 at 11:07 AM, Alexander Belopolsky wrote: >> Is it open to debate or is it now cast in stone? > > I think the barrier for changing str() is lower than that for changing > repr(), but I would be against any changes in this area. (I may have had a > different view if ISO 8601 syntax for timedeltas was not so ugly. :-) I think str() should be left alone. It's clear there is no one best str representation for timedelta objects. I think it might be worthwhile to add a method which allows the programmer to format timedelta objects in whatever way s/he sees fit. That would support the ISO 8601 syntax (*), and anything else the programmer things is better than the status quo. Skip (*) As an aside (that is, this belongs in a separate thread if you want to discuss it), in my opinion, attempting to support ISO 8601 formatting is pointless without the presence of an anchor datetime. Otherwise how would you know how far back "five months" or "seven years" was? If that's the case, then you might as well add the timedelta to your anchor datetime and just use datetime.strftime(). From tseaver at palladion.com Fri Mar 28 17:22:27 2014 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 28 Mar 2014 12:22:27 -0400 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> <87vbuynwny.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/28/2014 12:18 PM, Tres Seaver wrote: > I'm mostly arguing the FLOSS project should feel free to ignore > high-maintenance-cost commercial concerns until those concerns bring > either blook (funded developer time) or treasure (pooled to pay for > the same time) to the table to pay for them. Dangit, s/blook/blood/ Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlM1ocMACgkQ+gerLs4ltQ6k8ACgm+ycaOaqZGsefJU5iu9kL4bS 1e4AmQHq/vb4Q6GV/MNuCZQr4HKG/JER =8fLu -----END PGP SIGNATURE----- From solipsis at pitrou.net Fri Mar 28 17:25:21 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 28 Mar 2014 17:25:21 +0100 Subject: [Python-Dev] libpython added to ABI tracker References: <5335337F.8000303@rosalab.ru> Message-ID: <20140328172521.2a2abe7e@fsol> On Fri, 28 Mar 2014 12:31:59 +0400 Andrey Ponomarenko wrote: > Hi, > > The libpython library has been added to the ABI tracker: > http://upstream-tracker.org/versions/python.html > > The page lists library versions and changes in API/ABI. Thanks. Do note that most of these changes are on private (internal) APIs that normal user code should never invoke / rely on (for example all those APIs starting with an underscore). Regards Antoine. From solipsis at pitrou.net Fri Mar 28 17:30:35 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 28 Mar 2014 17:30:35 +0100 Subject: [Python-Dev] Negative timedelta strings References: <20140328110342.GI16526@ando> Message-ID: <20140328173035.53104fe2@fsol> On Fri, 28 Mar 2014 11:19:52 -0500 Skip Montanaro wrote: > On Fri, Mar 28, 2014 at 11:07 AM, Alexander Belopolsky > wrote: > >> Is it open to debate or is it now cast in stone? > > > > I think the barrier for changing str() is lower than that for changing > > repr(), but I would be against any changes in this area. (I may have had a > > different view if ISO 8601 syntax for timedeltas was not so ugly. :-) > > I think str() should be left alone. It's clear there is no one best > str representation for timedelta objects. But at least we could have one that isn't terribly confusing :-) Regards Antoine. From p.f.moore at gmail.com Fri Mar 28 17:39:45 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 28 Mar 2014 16:39:45 +0000 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> <87vbuynwny.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 28 March 2014 16:22, Tres Seaver wrote: > On 03/28/2014 12:18 PM, Tres Seaver wrote: >> I'm mostly arguing the FLOSS project should feel free to ignore >> high-maintenance-cost commercial concerns until those concerns bring >> either blook (funded developer time) or treasure (pooled to pay for >> the same time) to the table to pay for them. > > Dangit, > > s/blook/blood/ Rats. I thought I'd just learned a new word. Regardless of reality, I think "blook" should be a real word :-) Paul From alexander.belopolsky at gmail.com Fri Mar 28 17:41:41 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 28 Mar 2014 12:41:41 -0400 Subject: [Python-Dev] Negative timedelta strings In-Reply-To: References: <20140328110342.GI16526@ando> Message-ID: On Fri, Mar 28, 2014 at 12:19 PM, Skip Montanaro wrote: > (*) As an aside (that is, this belongs in a separate thread if you > want to discuss it), in my opinion, attempting to support ISO 8601 > formatting is pointless without the presence of an anchor datetime. > I meant ISO 8601 syntax for "durations" [1]. """ For example, "P3Y6M4DT12H30M5S" represents a duration of "three years, six months, four days, twelve hours, thirty minutes, and five seconds". """ And if you have to ask - no I don't want to see this anywhere near Python stdlib. :-) [1] http://en.wikipedia.org/wiki/ISO_8601#Durations -------------- next part -------------- An HTML attachment was scrubbed... URL: From josiah.carlson at gmail.com Fri Mar 28 17:45:40 2014 From: josiah.carlson at gmail.com (Josiah Carlson) Date: Fri, 28 Mar 2014 09:45:40 -0700 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: If it makes you feel any better, I spent an hour this morning building a 2-function API for Linux and Windows, both tested, not using ctypes, and not even using any part of asyncio (the Windows bits are in msvcrt and _winapi). It works in Python 3.3+. You can see it here: http://pastebin.com/0LpyQtU5 - Josiah On Fri, Mar 28, 2014 at 1:09 AM, Paul Moore wrote: > On 28 March 2014 05:09, Josiah Carlson wrote: > > So yeah. Someone want to make a decision? Tell me to write the docs, I > will. > > Tell me to go take a long walk off a short pier, I'll thank you for your > > time and leave you alone. > > I had a need for this a few years ago. It's messy to do on Windows > (ctypes callouts to PeekNamedPipe to check if you can read from the > process without blocking). So I would like to see a recipe for this, > (even if it's likely to be another few years before I ever need it > again :-)). > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Mar 28 18:07:50 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 28 Mar 2014 18:07:50 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140328170750.9EFD4560CD@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-03-21 - 2014-03-28) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4534 (+23) closed 28324 (+51) total 32858 (+74) Open issues with patches: 2090 Issues opened (61) ================== #15968: Incorporate Tcl/Tk/Tix into the Windows build process http://bugs.python.org/issue15968 reopened by haypo #16047: Tools/freeze no longer works in Python 3 http://bugs.python.org/issue16047 reopened by lemburg #21013: server-specific SSL context configuration http://bugs.python.org/issue21013 opened by pitrou #21014: `1` => `True`; for tutorial docs http://bugs.python.org/issue21014 opened by SamuelMarks #21016: trace: $prefix and $exec_prefix improperly replaced on Fedora http://bugs.python.org/issue21016 opened by mathstuf #21018: [patch] added missing documentation about escaping characters http://bugs.python.org/issue21018 opened by Arun.Persaud #21019: PyMethodDef ml_name is char* instead of const char* http://bugs.python.org/issue21019 opened by h.venev #21020: PyMethodDef ml_doc is char* instead of const char* http://bugs.python.org/issue21020 opened by h.venev #21021: PyMemberDef name is char* instead of const char* http://bugs.python.org/issue21021 opened by h.venev #21022: PyMemberDef doc is char* instead of const char* http://bugs.python.org/issue21022 opened by h.venev #21023: PyTypeObject tp_name is char* instead of const char* http://bugs.python.org/issue21023 opened by h.venev #21024: PyTypeObject tp_doc is char* instead of const char* http://bugs.python.org/issue21024 opened by h.venev #21025: if check_hostname is true, context can't be used for server pu http://bugs.python.org/issue21025 opened by pitrou #21026: Document sitecustomize.py problems with pythonw http://bugs.python.org/issue21026 opened by terry.reedy #21027: difflib new cli interface http://bugs.python.org/issue21027 opened by Claudiu.Popa #21028: ElementTree objects should support all the same methods as Ele http://bugs.python.org/issue21028 opened by rhettinger #21029: IDLE mis-coloring "print" http://bugs.python.org/issue21029 opened by rhettinger #21030: pip usable only by administrators on Windows http://bugs.python.org/issue21030 opened by Christian.Ullrich #21031: [patch] Add AlpineLinux to the platform module's supported dis http://bugs.python.org/issue21031 opened by Elizacat #21032: Socket leak if HTTPConnection.getresponse() fails http://bugs.python.org/issue21032 opened by vadmium #21033: previous trace function still invoked after sys.settrace() http://bugs.python.org/issue21033 opened by xdegaye #21034: Python docs reference the Distribute package which has been de http://bugs.python.org/issue21034 opened by Jurko.Gospodneti?? #21035: Python's HTTP server implementations hangs after 16.343 reques http://bugs.python.org/issue21035 opened by chobeiry #21037: add an AddressSanitizer build option http://bugs.python.org/issue21037 opened by neologix #21040: socketserver: use selectors module http://bugs.python.org/issue21040 opened by neologix #21041: pathlib.PurePath.parents rejects negative indexes http://bugs.python.org/issue21041 opened by akira #21042: ctypes.util.find_library() should return full pathname instead http://bugs.python.org/issue21042 opened by Hernan.Grecco #21044: tarfile does not handle file .name being an int http://bugs.python.org/issue21044 opened by seirl #21046: Document formulas used in statistics http://bugs.python.org/issue21046 opened by zach.ware #21047: html.parser.HTMLParser: convert_charrefs should become True by http://bugs.python.org/issue21047 opened by Arfrever #21048: Index 'as' in import and with statements http://bugs.python.org/issue21048 opened by terry.reedy #21052: Consider dropping ImportWarning for empty sys.path_hooks and s http://bugs.python.org/issue21052 opened by brett.cannon #21054: Improve indexing of syntax symbols http://bugs.python.org/issue21054 opened by terry.reedy #21055: Index (augmented) assignment symbols http://bugs.python.org/issue21055 opened by terry.reedy #21056: csv documentation is incorrect http://bugs.python.org/issue21056 opened by NRGunby #21057: TextIOWrapper does not support reading bytearrays or memoryvie http://bugs.python.org/issue21057 opened by nikratio #21059: idle_test.test_warning failure http://bugs.python.org/issue21059 opened by neologix #21060: Better error message for setup.py upload command without sdist http://bugs.python.org/issue21060 opened by eric.araujo #21061: Is contextlib.redirect_stdout reentrant or not? http://bugs.python.org/issue21061 opened by perey #21062: Evalute all import-related modules for best practices http://bugs.python.org/issue21062 opened by brett.cannon #21063: Touch up one-line descriptions of modules for module index http://bugs.python.org/issue21063 opened by brett.cannon #21066: The separate download version for the documentation doesn't wo http://bugs.python.org/issue21066 opened by Cristian.Baboi #21068: Make ssl.PROTOCOL_* an enum http://bugs.python.org/issue21068 opened by pitrou #21069: test_fileno of test_urllibnet intermittently fails when using http://bugs.python.org/issue21069 opened by dfarrell07 #21070: test_xmlrpc waits forever instead of reporting failure http://bugs.python.org/issue21070 opened by matejcik #21071: struct.Struct.format is bytes, but should be str http://bugs.python.org/issue21071 opened by zbysz #21072: Python docs and downloads not available for Egypt http://bugs.python.org/issue21072 opened by Leo.Butcher #21073: Py_ReprEnter potentially misbehaves during malformed thread st http://bugs.python.org/issue21073 opened by itaibn #21074: Too aggressive constant folding http://bugs.python.org/issue21074 opened by naoki #21075: fileinput should use stdin.buffer for "rb" mode http://bugs.python.org/issue21075 opened by brandon-rhodes #21076: Turn signal.SIG* constants into enums http://bugs.python.org/issue21076 opened by giampaolo.rodola #21077: Turtle Circle Speed 0 http://bugs.python.org/issue21077 opened by Garrett.Grimsley #21078: multiprocessing.managers.BaseManager.__init__'s "serializer" a http://bugs.python.org/issue21078 opened by yorik.sar #21079: EmailMessage.is_attachment == False if filename is present http://bugs.python.org/issue21079 opened by brandon-rhodes #21080: asyncio.subprocess: connect pipes of two programs http://bugs.python.org/issue21080 opened by haypo #21081: missing vietnamese codec TCVN 5712:1993 in Python http://bugs.python.org/issue21081 opened by progfou #21082: os.makedirs(exist_ok=True) is not thread-safe: umask is set te http://bugs.python.org/issue21082 opened by desrt #21083: Add get_content_disposition() to email.message.Message http://bugs.python.org/issue21083 opened by brandon-rhodes #21084: IDLE can't deal with characters above the range (U+0000-U+FFFF http://bugs.python.org/issue21084 opened by wjssz #21085: compile error Python3.3 on Cygwin http://bugs.python.org/issue21085 opened by dellair.jie #21086: Source with "# -*- coding: ASCII -*-" and UNIX EOL (\n) result http://bugs.python.org/issue21086 opened by Morten.Z Most recent 15 issues with no replies (15) ========================================== #21083: Add get_content_disposition() to email.message.Message http://bugs.python.org/issue21083 #21077: Turtle Circle Speed 0 http://bugs.python.org/issue21077 #21066: The separate download version for the documentation doesn't wo http://bugs.python.org/issue21066 #21062: Evalute all import-related modules for best practices http://bugs.python.org/issue21062 #21060: Better error message for setup.py upload command without sdist http://bugs.python.org/issue21060 #21055: Index (augmented) assignment symbols http://bugs.python.org/issue21055 #21054: Improve indexing of syntax symbols http://bugs.python.org/issue21054 #21048: Index 'as' in import and with statements http://bugs.python.org/issue21048 #21047: html.parser.HTMLParser: convert_charrefs should become True by http://bugs.python.org/issue21047 #21042: ctypes.util.find_library() should return full pathname instead http://bugs.python.org/issue21042 #21034: Python docs reference the Distribute package which has been de http://bugs.python.org/issue21034 #21033: previous trace function still invoked after sys.settrace() http://bugs.python.org/issue21033 #21032: Socket leak if HTTPConnection.getresponse() fails http://bugs.python.org/issue21032 #21031: [patch] Add AlpineLinux to the platform module's supported dis http://bugs.python.org/issue21031 #21029: IDLE mis-coloring "print" http://bugs.python.org/issue21029 Most recent 15 issues waiting for review (15) ============================================= #21082: os.makedirs(exist_ok=True) is not thread-safe: umask is set te http://bugs.python.org/issue21082 #21079: EmailMessage.is_attachment == False if filename is present http://bugs.python.org/issue21079 #21076: Turn signal.SIG* constants into enums http://bugs.python.org/issue21076 #21069: test_fileno of test_urllibnet intermittently fails when using http://bugs.python.org/issue21069 #21068: Make ssl.PROTOCOL_* an enum http://bugs.python.org/issue21068 #21063: Touch up one-line descriptions of modules for module index http://bugs.python.org/issue21063 #21057: TextIOWrapper does not support reading bytearrays or memoryvie http://bugs.python.org/issue21057 #21055: Index (augmented) assignment symbols http://bugs.python.org/issue21055 #21048: Index 'as' in import and with statements http://bugs.python.org/issue21048 #21044: tarfile does not handle file .name being an int http://bugs.python.org/issue21044 #21041: pathlib.PurePath.parents rejects negative indexes http://bugs.python.org/issue21041 #21040: socketserver: use selectors module http://bugs.python.org/issue21040 #21037: add an AddressSanitizer build option http://bugs.python.org/issue21037 #21034: Python docs reference the Distribute package which has been de http://bugs.python.org/issue21034 #21032: Socket leak if HTTPConnection.getresponse() fails http://bugs.python.org/issue21032 Top 10 most discussed issues (10) ================================= #20904: HAVE_PY_SET_53BIT_PRECISION for m68k http://bugs.python.org/issue20904 41 msgs #16047: Tools/freeze no longer works in Python 3 http://bugs.python.org/issue16047 24 msgs #21013: server-specific SSL context configuration http://bugs.python.org/issue21013 16 msgs #20210: Provide configure options to enable/disable Python modules and http://bugs.python.org/issue20210 15 msgs #21035: Python's HTTP server implementations hangs after 16.343 reques http://bugs.python.org/issue21035 14 msgs #21057: TextIOWrapper does not support reading bytearrays or memoryvie http://bugs.python.org/issue21057 11 msgs #21069: test_fileno of test_urllibnet intermittently fails when using http://bugs.python.org/issue21069 11 msgs #21079: EmailMessage.is_attachment == False if filename is present http://bugs.python.org/issue21079 11 msgs #20951: SSLSocket.send() returns 0 for non-blocking socket http://bugs.python.org/issue20951 8 msgs #21044: tarfile does not handle file .name being an int http://bugs.python.org/issue21044 8 msgs Issues closed (44) ================== #6676: expat parser throws Memory Error when parsing multiple files http://bugs.python.org/issue6676 closed by ned.deily #12209: Minor edits to faulthandler doc http://bugs.python.org/issue12209 closed by haypo #15971: Sporadic failure in test_dump_tracebacks_later_file (test_faul http://bugs.python.org/issue15971 closed by haypo #17373: Add inspect.Signature.from_callable() http://bugs.python.org/issue17373 closed by python-dev #17654: IDLE only customizes correctly for OS X when using framework b http://bugs.python.org/issue17654 closed by ned.deily #18456: Documentation for PyDict_Update is incorrect http://bugs.python.org/issue18456 closed by python-dev #19264: subprocess.Popen doesn't support unicode on Windows http://bugs.python.org/issue19264 closed by terry.reedy #19537: Fix misalignment in fastsearch_memchr_1char http://bugs.python.org/issue19537 closed by pitrou #19573: Fix the docstring of inspect.Parameter and the implementation http://bugs.python.org/issue19573 closed by python-dev #19818: tracemalloc: comments on the doc http://bugs.python.org/issue19818 closed by haypo #19995: %c, %o, %x, %X accept non-integer values instead of raising an http://bugs.python.org/issue19995 closed by ethan.furman #20145: unittest.assert*Regex functions should verify that expected_re http://bugs.python.org/issue20145 closed by r.david.murray #20163: ValueError: time data does not match format http://bugs.python.org/issue20163 closed by dellair.jie #20378: Implement `Signature.__repr__` http://bugs.python.org/issue20378 closed by python-dev #20633: SystemError: Parent module 'multiprocessing' not loaded, canno http://bugs.python.org/issue20633 closed by sbt #20668: Remove dependency on tests.txt when running test_asyncio suite http://bugs.python.org/issue20668 closed by python-dev #20726: inspect: Make Signature instances picklable http://bugs.python.org/issue20726 closed by python-dev #20816: inspect.getcallargs() attempts to iterate over None http://bugs.python.org/issue20816 closed by python-dev #20817: inspect.getcallargs() raises the wrong error if 3+ arguments a http://bugs.python.org/issue20817 closed by python-dev #20913: Standard Library documentation needs SSL security best practic http://bugs.python.org/issue20913 closed by pitrou #20935: Support building Python with Clang sanitizer rules http://bugs.python.org/issue20935 closed by neologix #20953: heap-buffer-overflow in obmalloc.c:987 http://bugs.python.org/issue20953 closed by neologix #20975: Python 3.4 build info wrong in code snippet http://bugs.python.org/issue20975 closed by python-dev #20980: In multiprocessing.pool, ExceptionWithTraceback should derive http://bugs.python.org/issue20980 closed by sbt #20990: pyflakes: undefined names, get_context() and main(), in multip http://bugs.python.org/issue20990 closed by sbt #20992: reading individual bytes of multiple binary files using the Py http://bugs.python.org/issue20992 closed by Tommy.Carstensen #20995: Use Better Default Ciphers for the SSL Module http://bugs.python.org/issue20995 closed by dstufft #21000: json.tool ought to have a help flag http://bugs.python.org/issue21000 closed by python-dev #21006: asyncio.docs : create_subprocess_exec example does not work on http://bugs.python.org/issue21006 closed by haypo #21015: support SSL_CTX_set_ecdh_auto on newer OpenSSLs http://bugs.python.org/issue21015 closed by pitrou #21017: Enable Debug Tix build on Windows http://bugs.python.org/issue21017 closed by zach.ware #21036: typo in hashtable API: _PY_HASHTABLE_ENTRY_DATA -> _Py_HASHTAB http://bugs.python.org/issue21036 closed by haypo #21038: test_epoll.TestEPoll.test_control_and_wait: remove extra asser http://bugs.python.org/issue21038 closed by haypo #21039: pathlib strips trailing slash http://bugs.python.org/issue21039 closed by pitrou #21043: Stop recommending CACert.org in the SSL documentation http://bugs.python.org/issue21043 closed by dstufft #21045: fix layout to generate documentation for Qt Assistant http://bugs.python.org/issue21045 closed by python-dev #21049: Flood of ???ImportWarning: sys.meta_path is empty??? after exc http://bugs.python.org/issue21049 closed by brett.cannon #21050: Failure to import win32api (from pywin32) http://bugs.python.org/issue21050 closed by theller #21051: incorrect utf-8 conversion with c api http://bugs.python.org/issue21051 closed by mark.dickinson #21053: Idle Crash with the ^ button http://bugs.python.org/issue21053 closed by ned.deily #21058: tempfile.NamedTemporaryFile leaks file descriptor when fdopen http://bugs.python.org/issue21058 closed by haypo #21064: Simple segfault http://bugs.python.org/issue21064 closed by ned.deily #21065: spam http://bugs.python.org/issue21065 closed by r.david.murray #21067: Support Multiple finally clauses. http://bugs.python.org/issue21067 closed by rhettinger From skip at pobox.com Fri Mar 28 18:13:25 2014 From: skip at pobox.com (Skip Montanaro) Date: Fri, 28 Mar 2014 12:13:25 -0500 Subject: [Python-Dev] ISO 8601 durations and datetime.timedelta Message-ID: Andrew wrote: > I meant ISO 8601 syntax for "durations" [1]. That's exactly what I was referring to. Consider this session: >>> now = datetime.datetime.now() >>> now datetime.datetime(2014, 3, 28, 12, 4, 38, 517110) >>> then = now - datetime.timedelta(days=57, hours=12, minutes=12, seconds=12) >>> now datetime.datetime(2014, 3, 28, 12, 4, 38, 517110) >>> then datetime.datetime(2014, 1, 29, 23, 52, 26, 517110) so, the difference is: >>> now - then datetime.timedelta(57, 43932) Given that the timedelta has more than "a month's" worth of days, how would you describe it using the ISO8601 duration notation without referencing a specific point in time? Conversely, given your example, "P3Y6M4DT12H30M5S", how would you convert that into a timedelta without knowing which exact years and months this duration refers to? Timedelta objects are very precise beasts, which is almost certainly why they don't support "years" and "months" args in their constructors. This is why I said this deserved a separate topic. Probably on python-ideas. Skip From josiah.carlson at gmail.com Fri Mar 28 18:42:03 2014 From: josiah.carlson at gmail.com (Josiah Carlson) Date: Fri, 28 Mar 2014 10:42:03 -0700 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: *This* is the type of conversation that I wanted to avoid. But I'll answer your questions because I used to do exactly the same thing. On Fri, Mar 28, 2014 at 3:20 AM, Victor Stinner wrote: > 2014-03-28 2:16 GMT+01:00 Josiah Carlson : > > def do_login(...): > > proc = subprocess.Popen(...) > > current = proc.recv(timeout=5) > > last_line = current.rstrip().rpartition('\n')[-1] > > if last_line.endswith('login:'): > > proc.send(username) > > if proc.readline(timeout=5).rstrip().endswith('password:'): > > proc.send(password) > > if 'welcome' in proc.recv(timeout=5).lower(): > > return proc > > proc.kill() > > I don't understand this example. How is it "asynchronous"? It looks > like blocking calls. In my definition, asynchronous means that you can > call this function twice on two processes, and they will run in > parallel. > In this context, async means not necessarily blocking. If you didn't provide a timeout, it would default to 0, which would return immediately with what was sent and/or received from the subprocess. If you don't believe me, that's fine, but it prevents meaningful discussion. Using greenlet/eventlet, you can write code which looks blocking, but > runs asynchronously. But I don't think that you are using greenlet or > eventlet here. > You are right. And you are talking about something that is completely out of scope. > I take a look at the implementation: > http://code.google.com/p/subprocdev/source/browse/subprocess.py > > It doesn't look portable. On Windows, WriteFile() is used. This > function is blocking, or I missed something huge :-) It's much better > if a PEP is portable. Adding time.monotonic() only to Linux would make > the PEP 418 much shorter (4 sentences instead of 10 pages? :-))! > Of course it's not portable. Windows does things differently from other platforms. That's one of the reasons why early versions required pywin32. Before you reply to another message, I would encourage you to read the bug, the pep, and perhaps the recipe I just posted: http://pastebin.com/0LpyQtU5 Or you can try to believe that I have done all of those and believe what I say, especially when I say that I don't believe that spending a lot of time worrying about the original patch/recipe and the GSoC entry. They would all require a lot of work to make reasonably sane, which is why I wrote the minimal recipe above. The implementation doesn't look reliable: > > def get_conn_maxsize(self, which, maxsize): > # Not 100% certain if I get how this works yet. > if maxsize is None: > maxsize = 1024 > ... > > This constant 1024 looks arbitrary. On UNIX, a write into a pipe may > block with less bytes (512 bytes). > Testing now I seem to be able to send non-reading subprocesses somewhat arbitrary amounts of data without leading to a block. But I can't test all Linux installations or verify that I'm correct. But whether or not this makes sense is moot, as I don't think it should be merged, and I don't believe anyone thinks it should be merged at this point. asyncio has a completly different design. On Windows, it uses > overlapped operations with IOCP event loop. Such operation can be > cancelled. Windows cares of the buffering. On UNIX, non-blocking mode > is used with select() (or something faster like epoll) and asyncio > retries to write more data when the pipe (or any file descriptor used > for process stdin/stdoud/stderr) becomes ready (for reading/writing). > > asyncio design is more reliable and portable. > More reliable, sure. More portable... only because all of the portability heavy lifting has been done and included in Python core. That's one other thing that you aren't understanding - the purpose of trying to have this in the standard library is so that people can use the functionality (async subprocesses) on multiple platforms without needing to write it themselves (poorly), ask on forums of one kind or another, copy and paste from some recipe posted to the internet, etc. It's a strict increase in the functionality and usefulness of the Python standard library and has literally zero backwards compatibility issues. This is the absolute minimum functionality necessary to make people who need this functionality happy. No, really. Absolute minimum. Sort of what asyncore was - the minimum functionality necessary to have async sockets in Python. Was it dirty? Sure. Was it difficult to use? Some people had issues. Did it work? It worked well enough that people were making money building applications based on asyncore (myself included 10 years ago). I don't see how you can implement asynchronous communication with a > subprocess without the complex machinery of an event loop. > Words can have multiple meanings. The meaning of "async" in this context is different from what you believe it to mean, which is part of your confusion. I tried to address this in my last message, but either you didn't read that part, didn't understand that part, or don't believe what I wrote. So let me write it again: In this context, "async subprocesses" means the ability to interactively interrogate a subprocess without necessarily blocking on input or output. Everyone posting questions about this on StackOverflow or other forums understands it this way. It *does not mean* that it needs to participate in an event loop, needs to be usable with asyncore, asyncio, Twisted, greenlets, gevent, or otherwise. If there is one thing that *I* need for you (and everyone else) to understand and believe in this conversation, it is the above. Do you? Yes? Okay. Now read everything that I've written again. No? Can you explain *why* you don't believe or understand me? > The API above can be very awkward (as shown :P ), but that's okay. From > > those building blocks a (minimally) enterprising user would add > > functionality to suit their needs. The existing subprocess module only > > offers two methods for *any* amount of communication over pipes with the > > subprocess: check_output() and communicate(), only the latter of which > > supports sending data (once, limited by system-level pipe buffer > lengths). > > As I wrote, it's complex to handle non-blocking file descriptors. You > have to catch EWOULDBLOCK and retries later when the file descriptor > becomes ready. The main thread has to watch for such event on the file > descriptor, or you need a dedicated thread. By the way, > subprocess.communicate() is currently implemented using threads on > Windows. > I know what it takes, I've been writing async sockets for 12 years. I used to maintain asyncore/asynchat and related libraries. Actually, you can thank me for asyncore existing in Python 2.6+ (Giampaolo has done a great job and kept asyncore alive after I stopped participating daily python-dev about 5 years ago, and I can't thank him enough for that). But to the point: stop bagging on the old patches. No one likes them. We all agree. The question is where do we go from here. > Neither allow for nontrivial interactions from a single subprocess.Popen() > > invocation. The purpose was to be able to communicate in a bidirectional > > manner with a subprocess without blocking, or practically speaking, > blocking > > with a timeout. That's where the "async" term comes from. > > I call this "non-blocking functions", not "async functions". > > It's quite simple to check if a read will block on not on UNIX. It's > more complex to implement it on Windows. And even more complex to > handle to add a buffer to write(). > Okay, call it non-blocking subprocess reads and writes. Whatever you want to call it. And yes, I know what it takes to read and write on Windows... I've done it 3 times now (the original recipe, the original patch, now the above recipe). But the other piece is that *this* doesn't necessarily need to be 100% robust - I'm not even advocating it to be in the Python standard library anywhere! I've given up on that. But a short example hanging out in the docs? Someone will use it. Someone will run into issues. They will add robustness. They will add functionality. And it will grow into something worth using before being posted to the cheeseshop. The status quo is that people don't get answers anywhere in the Python docs or the Python stdlib. Python core is noticeably absent in a source of information about how someone would go about using the subprocess module in a completely reasonable and sane manner. > Your next questions will be: But why bother at all? Why not just build the > > piece you need *inside* asyncio? Why does this need anything more? The > > answer to those questions are wants and needs. If I'm a user that needs > > interactive subprocess handling, I want to be able to do something like > the > > code snippet above. The last thing I need is to have to rewrite the way > my > > application/script/whatever handles *everything* just because a new > > asynchronous IO library has been included in the Python standard library > - > > it's a bit like selling you a $300 bicycle when you need a $20 wheel for > > your scooter. > > You don't have to rewrite your whole application. If you only want to > use asyncio event loop in a single function, you can use > loop.run_until_complete(do_login) which blocks until the function > completes. The "function" is an asynchronous coroutine in fact. > The point of this conversation is that I was offering to write the handful of wrappers that would make interactions of the form that I showed earlier easy and possible with asyncio. So that a user didn't have to write them themselves. [snip] Even if eval_python_async() is asynchronous, eval_python() function is > blocking so you can write: print("1+1 = %r" % eval_python("1+1")) > without callback nor "yield from". > > Running tasks in parallel is faster than running them in sequence > (almost 5 times faster on my PC). > This is completely unrelated to the conversation. The syntax in eval_python_async() is close to the API you proposed, > except that you have to add "yield from" in front of "blocking" > functions like read() or drain() (it's the function to flush the stdin > buffer, I'm not sure that it is needed in this example). > > The timeout is on the whole eval_python_async(), but you can as well > using finer timeout on each read/write. > > > But here's the thing: I can build enough using asyncio in 30-40 lines of > > Python to offer something like the above API. The problem is that it > really > > has no natural home. > > I agree that writing explicit asynchronous code is more complex than > using eventlet. Asynchronous programming is hard. > No, it's not hard. It just requires thinking in a different way. It's the thinking in a different way that's difficult. But I've been doing async sockets programming on and off for 13 years now, so I get it. What I'm offering is to help people *not* do that, because some people have difficulty thinking in that way. > But in the docs? It would show an atypical, but not > > wholly unreasonable use of asyncio (the existing example already shows > what > > I would consider to be an atypical use of asyncio). > > The asyncio documentation is still a work-in-progress. I tried to > document all APIs, but there are too few examples and the > documentation is still focused on the API instead of being oriented to > the user of the API. > > Don't hesitate to contribute to the documentation! > So is this the "okay" that I've been waiting with baited breath for? - Josiah We can probably write a simple example showing how to interact with an > interactive program like Python. > > Victor > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Mar 28 18:42:48 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 28 Mar 2014 13:42:48 -0400 Subject: [Python-Dev] ISO 8601 durations and datetime.timedelta In-Reply-To: References: Message-ID: On Fri, Mar 28, 2014 at 1:13 PM, Skip Montanaro wrote: > Given that the timedelta has more than "a month's" worth of days, how > would you describe it using the ISO8601 duration notation without > referencing a specific point in time? Conversely, given your example, > "P3Y6M4DT12H30M5S", how would you convert that into a timedelta > without knowing which exact years and months this duration refers to? > Timedelta objects are very precise beasts, which is almost certainly > why they don't support "years" and "months" args in their > constructors. > So why would they need to support them in str()? We could make str(timedelta(days=369, hours=12, minutes=30, seconds=5)) return "P369DT12H30M5S", but that would hardly be an improvement over >>> print timedelta(days=369, hours=12, minutes=30, seconds=5) 369 days, 12:30:05 and I don't think ISO 8601 allows for negative durations, so it gives us little guidance for the other issue either. My preference would be to leave strict ISO 8601 compliance to 3rd party libraries. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Mar 28 18:46:51 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 28 Mar 2014 10:46:51 -0700 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: On Fri, Mar 28, 2014 at 9:45 AM, Josiah Carlson wrote: > > If it makes you feel any better, I spent an hour this morning building a > 2-function API for Linux and Windows, both tested, not using ctypes, and > not even using any part of asyncio (the Windows bits are in msvcrt and > _winapi). It works in Python 3.3+. You can see it here: > http://pastebin.com/0LpyQtU5 > Seeing this makes *me* feel better. I think it's reasonable to add (some variant) of that to the subprocess module in Python 3.5. It also belongs in the Activestate cookbook. And no, the asyncio module hasn't made it obsolete. Josiah, you sound upset about the whole thing -- to the point of writing unintelligible sentences and passive-aggressive digs at everyone reading this list. I'm sorry that something happened that led you feel that way (if you indeed feel upset or frustrated) but I'm glad that you wrote that code snippet -- it is completely clear what you want and why you want it, and also what should happen next (a few rounds of code review on the tracker). But that PEP? It's just a terrible PEP. It doesn't contain a single line of example code. It doesn't specify the proposed interface, it just describes in way too many sentences how it should work, and contains a whole lot of references to various rants from which the reader is apparently meant to become enlightened. I don't know which of the three authors *really* wrote it, and I don't want to know -- I think the PEP is irrelevant to the proposed feature, which is of "put it in the bug tracker and work from there" category -- presumably the PEP was written based on the misunderstanding that having a PEP would make acceptance of the patch easier, or because during an earlier bikeshedding round someone said "please write a PEP" (someone always says that). I propose to scrap the PEP (set the status to Withdrawn) and just work on adding the methods to the subprocess module. If it were me, I'd define three methods, with longer names to clarify what they do, e.g. proc.write_nonblocking(data) data = proc.read_nonblocking() data = proc.read_stderr_nonblocking() I.e. add _nonblocking to the method names to clarify that they may return '' when there's nothing available, and have a separate method for reading stderr instead of a flag. And I'd wonder if there should be an unambiguous way to detect EOF or whether the caller should just check for proc.stdout.closed. (And what for stdin? IIRC it actually becomes writable when the other end is closed, and then the write() will fail. But maybe I forget.) But that's all bikeshedding and it can happen on the tracker or directly on the list just as easily; I don't see the need for a PEP. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From josiah.carlson at gmail.com Fri Mar 28 19:35:16 2014 From: josiah.carlson at gmail.com (Josiah Carlson) Date: Fri, 28 Mar 2014 11:35:16 -0700 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: On Fri, Mar 28, 2014 at 10:46 AM, Guido van Rossum wrote: > On Fri, Mar 28, 2014 at 9:45 AM, Josiah Carlson wrote: > >> >> If it makes you feel any better, I spent an hour this morning building a >> 2-function API for Linux and Windows, both tested, not using ctypes, and >> not even using any part of asyncio (the Windows bits are in msvcrt and >> _winapi). It works in Python 3.3+. You can see it here: >> http://pastebin.com/0LpyQtU5 >> > > Seeing this makes *me* feel better. I think it's reasonable to add (some > variant) of that to the subprocess module in Python 3.5. It also belongs in > the Activestate cookbook. And no, the asyncio module hasn't made it > obsolete. > Cool. Josiah, you sound upset about the whole thing -- to the point of writing > unintelligible sentences and passive-aggressive digs at everyone reading > this list. I'm sorry that something happened that led you feel that way (if > you indeed feel upset or frustrated) but I'm glad that you wrote that code > snippet -- it is completely clear what you want and why you want it, and > also what should happen next (a few rounds of code review on the tracker). > I'm not always a prat. Something about python-dev brings out parts of me that I thought I had discarded from my personality years ago. Toss in a bit of needing to re-explain ideas that I've been trying to explain for almost 9 years? Frustration + formerly discarded personality traits = uck. That's pretty much why I won't be rejoining the party here regularly, you are all better off without me commenting on 95% of threads like I used to. Victor, I'm sorry for being a jerk. It's hard for me to not be the guy I was when I spend time on this list. That's *my* issue, not yours. That I spent any time redirecting my frustration towards you is BS, and if I could take back the email I sent just before getting Guido's, I would. I would advise everyone to write it off as the ramblings of a surprisingly young, angry old man. Or call me an a-hole. Both are pretty accurate. :) But that PEP? It's just a terrible PEP. It doesn't contain a single line of > example code. It doesn't specify the proposed interface, it just describes > in way too many sentences how it should work, and contains a whole lot of > references to various rants from which the reader is apparently meant to > become enlightened. I don't know which of the three authors *really* wrote > it, and I don't want to know -- I think the PEP is irrelevant to the > proposed feature, which is of "put it in the bug tracker and work from > there" category -- presumably the PEP was written based on the > misunderstanding that having a PEP would make acceptance of the patch > easier, or because during an earlier bikeshedding round someone said > "please write a PEP" (someone always says that). I propose to scrap the PEP > (set the status to Withdrawn) and just work on adding the methods to the > subprocess module. > I'm not going to argue. The first I read it was 2-3 days ago. If it were me, I'd define three methods, with longer names to clarify what > they do, e.g. > > proc.write_nonblocking(data) > data = proc.read_nonblocking() > data = proc.read_stderr_nonblocking() > Easily doable. I.e. add _nonblocking to the method names to clarify that they may return > '' when there's nothing available, and have a separate method for reading > stderr instead of a flag. And I'd wonder if there should be an unambiguous > way to detect EOF or whether the caller should just check for > proc.stdout.closed. (And what for stdin? IIRC it actually becomes writable > when the other end is closed, and then the write() will fail. But maybe I > forget.) > > But that's all bikeshedding and it can happen on the tracker or directly > on the list just as easily; I don't see the need for a PEP. > Sounds good. - Josiah -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Fri Mar 28 19:49:06 2014 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 28 Mar 2014 18:49:06 +0000 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> <87vbuynwny.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <5335C422.2090601@mrabarnett.plus.com> On 2014-03-28 16:39, Paul Moore wrote: > On 28 March 2014 16:22, Tres Seaver wrote: >> On 03/28/2014 12:18 PM, Tres Seaver wrote: >>> I'm mostly arguing the FLOSS project should feel free to ignore >>> high-maintenance-cost commercial concerns until those concerns bring >>> either blook (funded developer time) or treasure (pooled to pay for >>> the same time) to the table to pay for them. >> >> Dangit, >> >> s/blook/blood/ > > Rats. I thought I'd just learned a new word. Regardless of reality, I > think "blook" should be a real word :-) > Yes, and it should mean "funded developer time on or for an open source project". From g.brandl at gmx.net Fri Mar 28 20:19:48 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 28 Mar 2014 20:19:48 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Request for Pronouncement In-Reply-To: <53349642.1030301@stoneleaf.us> References: <533484E9.3090300@stoneleaf.us> <53349642.1030301@stoneleaf.us> Message-ID: Am 27.03.2014 22:21, schrieb Ethan Furman: > On 03/27/2014 01:44 PM, Guido van Rossum wrote: >> >> Accepted. > > Yay! +1 for that Yay :) Georg From tjreedy at udel.edu Fri Mar 28 20:42:17 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 28 Mar 2014 15:42:17 -0400 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: On 3/28/2014 12:45 PM, Josiah Carlson wrote: > If it makes you feel any better, I spent an hour this morning building a > 2-function API for Linux and Windows, both tested, not using ctypes, and > not even using any part of asyncio (the Windows bits are in msvcrt and > _winapi). It works in Python 3.3+. You can see it here: > http://pastebin.com/0LpyQtU5 Thank you. The docs gave me the impression that I could simply write proc.stdin and read proc.stdout. I failed with even a simple echo server (on Windows) and your code suggests why. So it does not get lost, I attached your code to http://bugs.python.org/issue18823 My interest is with Idle. It originally ran user code in the same process as the Shell and Editor code. Then Guido added an option to os.spawn a separate process and communicate through a socket connection and the option became the default with same process (requested by -N on the command line) as a backup option. 3.2 switched to using subprocess, but still with a socket. The problem is that the socket connection intermittently fails. Firewalls are, or at least used to be one possible cause, but there are others -- unknown. (While it works, the suggestion to restart with -N is a mystery to people who have never seen a command line.) This is one of the biggest sources of complaints about Idle. A pipe connection method that always worked on Windows, *x, and Mac would be great in itself and would also allow code simplification by removing the -n option. (Roger Serwy has suggested the latter as having two modes makes patching trickier.) The current socket connection must be non-blocking. Even though the exec loop part of the Shell window waits for a response after sending a user statement, everything else is responsive. One can select text in the window, use the menus, or switch to another window. So Idle definitely needs non-blocking write and read. In my ignorance, I have no idea whether the approach in your code or that in Viktor's code is better. Either way, I will appreciate any help you give, whether by writing, reviewing, or testing, to make communication with subprocesses easier and more dependable. -- Terry Jan Reedy From josiah.carlson at gmail.com Fri Mar 28 21:26:48 2014 From: josiah.carlson at gmail.com (Josiah Carlson) Date: Fri, 28 Mar 2014 13:26:48 -0700 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: On Fri, Mar 28, 2014 at 12:42 PM, Terry Reedy wrote: > On 3/28/2014 12:45 PM, Josiah Carlson wrote: > >> If it makes you feel any better, I spent an hour this morning building a >> 2-function API for Linux and Windows, both tested, not using ctypes, and >> not even using any part of asyncio (the Windows bits are in msvcrt and >> _winapi). It works in Python 3.3+. You can see it here: >> http://pastebin.com/0LpyQtU5 >> > > Thank you. The docs gave me the impression that I could simply write > proc.stdin and read proc.stdout. I failed with even a simple echo server > (on Windows) and your code suggests why. So it does not get lost, I > attached your code to > > http://bugs.python.org/issue18823 > > My interest is with Idle. It originally ran user code in the same process > as the Shell and Editor code. Then Guido added an option to os.spawn a > separate process and communicate through a socket connection and the option > became the default with same process (requested by -N on the command line) > as a backup option. 3.2 switched to using subprocess, but still with a > socket. The problem is that the socket connection intermittently fails. > Firewalls are, or at least used to be one possible cause, but there are > others -- unknown. (While it works, the suggestion to restart with -N is a > mystery to people who have never seen a command line.) This is one of the > biggest sources of complaints about Idle. A pipe connection method that > always worked on Windows, *x, and Mac would be great in itself and would > also allow code simplification by removing the -n option. (Roger Serwy has > suggested the latter as having two modes makes patching trickier.) > > The current socket connection must be non-blocking. Even though the exec > loop part of the Shell window waits for a response after sending a user > statement, everything else is responsive. One can select text in the > window, use the menus, or switch to another window. So Idle definitely > needs non-blocking write and read. > > In my ignorance, I have no idea whether the approach in your code or that > in Viktor's code is better. Either way, I will appreciate any help you > give, whether by writing, reviewing, or testing, to make communication with > subprocesses easier and more dependable. One of my other use-cases for this was using this in *my* editor (PyPE), which I wrote (in 2003) because I lost work in Idle. This lost work was due to the same-process interpreter crashing during an interactive session. IIRC, this is partly what pushed Guido to have Idle use os.spawn() + socket. I ended up using wxPython's built-in external process support at the time, but that's obviously not useful in core Python with Idle :P This is all coming back full circle. :) - Josiah -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Fri Mar 28 21:35:31 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 28 Mar 2014 13:35:31 -0700 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: <5335DD13.5080505@g.nevcal.com> On 3/28/2014 11:35 AM, Josiah Carlson wrote: > > If it were me, I'd define three methods, with longer names to > clarify what they do, e.g. > > proc.write_nonblocking(data) > data = proc.read_nonblocking() > data = proc.read_stderr_nonblocking() > > > Easily doable. I'd appreciate being notified if you do update/test as described. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Mar 28 21:58:25 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 28 Mar 2014 16:58:25 -0400 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: On 3/28/2014 6:20 AM, Victor Stinner wrote: > Full example of asynchronous communication with a subprocess (the > python interactive interpreter) using asyncio high-level API: Thank you for writing this. As I explained in response to Josiah, Idle communicates with a python interpreter subprocess through a socket. Since making the connection is not dependable, I would like to replace the socket with the pipes. http://bugs.python.org/issue18823 However, the code below creates a subprocess for one command and one response, which can apparently be done now with subprocess.communicate. What I and others need is a continuing (non-blocking) conversion with 1 and only 1 subprocess (see my response to Josiah), and that is much more difficult. So this code does not do what he claims his will do. However it is done, I agree with the intent of the PEP to make it much easier to talk with a subprocess. Victor, if you can rewrite the below with a run_forever loop that can accept new write-read task pairs and also make each line read immediately accessible, that would be really helpful. Post it on the issue above if you prefer. Another difference from what you wrote below and what Idle does today is that the shell, defined in idlelib/PyShell.py, does not talk to the subprocess interpreter directly but to a run supervisor defined in idlelib/run.py through an rpc protocol ('cmd', 'arg string'). To use the pipes, the supervisor would grab all input from stdin (instead of the socket) and exec user code as it does today, or it could be replaced by a supervisor class with an instance with a name like _idle_supervisor_3_4_0_ that would be extremely unlikely to clash with any name created by users. > --- > import asyncio.subprocess > import time > import sys > > @asyncio.coroutine > def eval_python_async(command, encoding='ascii', loop=None): > proc = yield from asyncio.subprocess.create_subprocess_exec( > sys.executable, "-u", "-i", > stdin=asyncio.subprocess.PIPE, > stdout=asyncio.subprocess.PIPE, > stderr=asyncio.subprocess.STDOUT, > loop=loop) > > # wait for the prompt > buffer = bytearray() > while True: > data = yield from proc.stdout.read(100) > buffer.extend(data) > if buffer.endswith(b'>>> '): > break > > proc.stdin.write(command.encode(encoding) + b"\n") > yield from proc.stdin.drain() > proc.stdin.close() > > output = yield from proc.stdout.read() > > output = output.decode(encoding) > output = output.rstrip() > if output.endswith('>>>'): > output = output[:-3].rstrip() > return output > > def eval_python(command, timeout=None): > loop = asyncio.get_event_loop() > task = asyncio.Task(eval_python_async(command, loop=loop), loop=loop) > return loop.run_until_complete(asyncio.wait_for(task, timeout)) > > def test_sequential(nproc, command): > t0 = time.monotonic() > for index in range(nproc): > eval_python(command) > return time.monotonic() - t0 > > def test_parallel(nproc, command): > loop = asyncio.get_event_loop() > tasks = [asyncio.Task(eval_python_async(command, loop=loop), loop=loop) > for index in range(nproc)] > t0 = time.monotonic() > loop.run_until_complete(asyncio.wait(tasks)) > return time.monotonic() - t0 > > print("1+1 = %r" % eval_python("1+1", timeout=1.0)) > > slow_code = "import math; print(str(math.factorial(20000)).count('7'))" > > dt = test_sequential(10, slow_code) > print("Run 10 tasks in sequence: %.1f sec" % dt) > > dt2 = test_parallel(10, slow_code) > print("Run 10 tasks in parallel: %.1f sec (speed=%.1f)" % (dt2, dt/dt2)) > > # cleanup asyncio > asyncio.get_event_loop().close() -- Terry Jan Reedy From guido at python.org Fri Mar 28 22:09:19 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 28 Mar 2014 14:09:19 -0700 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: To be clear, the proposal for Idle would be to still use the RPC protocol, but run it over a pipe instead of a socket, right? On Fri, Mar 28, 2014 at 1:58 PM, Terry Reedy wrote: > On 3/28/2014 6:20 AM, Victor Stinner wrote: > > Full example of asynchronous communication with a subprocess (the >> python interactive interpreter) using asyncio high-level API: >> > > Thank you for writing this. As I explained in response to Josiah, Idle > communicates with a python interpreter subprocess through a socket. Since > making the connection is not dependable, I would like to replace the socket > with the pipes. http://bugs.python.org/issue18823 > > However, the code below creates a subprocess for one command and one > response, which can apparently be done now with subprocess.communicate. > What I and others need is a continuing (non-blocking) conversion with 1 and > only 1 subprocess (see my response to Josiah), and that is much more > difficult. So this code does not do what he claims his will do. > > However it is done, I agree with the intent of the PEP to make it much > easier to talk with a subprocess. Victor, if you can rewrite the below with > a run_forever loop that can accept new write-read task pairs and also make > each line read immediately accessible, that would be really helpful. Post > it on the issue above if you prefer. > > Another difference from what you wrote below and what Idle does today is > that the shell, defined in idlelib/PyShell.py, does not talk to the > subprocess interpreter directly but to a run supervisor defined in > idlelib/run.py through an rpc protocol ('cmd', 'arg string'). To use the > pipes, the supervisor would grab all input from stdin (instead of the > socket) and exec user code as it does today, or it could be replaced by a > supervisor class with an instance with a name like _idle_supervisor_3_4_0_ > that would be extremely unlikely to clash with any name created by users. > > > --- >> import asyncio.subprocess >> import time >> import sys >> >> @asyncio.coroutine >> def eval_python_async(command, encoding='ascii', loop=None): >> proc = yield from asyncio.subprocess.create_subprocess_exec( >> sys.executable, "-u", "-i", >> stdin=asyncio.subprocess.PIPE, >> stdout=asyncio.subprocess.PIPE, >> stderr=asyncio.subprocess.STDOUT, >> loop=loop) >> >> # wait for the prompt >> buffer = bytearray() >> while True: >> data = yield from proc.stdout.read(100) >> buffer.extend(data) >> if buffer.endswith(b'>>> '): >> break >> >> proc.stdin.write(command.encode(encoding) + b"\n") >> yield from proc.stdin.drain() >> proc.stdin.close() >> >> output = yield from proc.stdout.read() >> >> output = output.decode(encoding) >> output = output.rstrip() >> if output.endswith('>>>'): >> output = output[:-3].rstrip() >> return output >> >> def eval_python(command, timeout=None): >> loop = asyncio.get_event_loop() >> task = asyncio.Task(eval_python_async(command, loop=loop), >> loop=loop) >> return loop.run_until_complete(asyncio.wait_for(task, timeout)) >> >> def test_sequential(nproc, command): >> t0 = time.monotonic() >> for index in range(nproc): >> eval_python(command) >> return time.monotonic() - t0 >> >> def test_parallel(nproc, command): >> loop = asyncio.get_event_loop() >> tasks = [asyncio.Task(eval_python_async(command, loop=loop), >> loop=loop) >> for index in range(nproc)] >> t0 = time.monotonic() >> loop.run_until_complete(asyncio.wait(tasks)) >> return time.monotonic() - t0 >> >> print("1+1 = %r" % eval_python("1+1", timeout=1.0)) >> >> slow_code = "import math; print(str(math.factorial(20000)).count('7'))" >> >> dt = test_sequential(10, slow_code) >> print("Run 10 tasks in sequence: %.1f sec" % dt) >> >> dt2 = test_parallel(10, slow_code) >> print("Run 10 tasks in parallel: %.1f sec (speed=%.1f)" % (dt2, dt/dt2)) >> >> # cleanup asyncio >> asyncio.get_event_loop().close() >> > > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Mar 28 22:12:55 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 28 Mar 2014 22:12:55 +0100 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen References: <20140325231947.5ec9ad63@fsol> Message-ID: <20140328221255.5c923901@fsol> On Fri, 28 Mar 2014 16:58:25 -0400 Terry Reedy wrote: > On 3/28/2014 6:20 AM, Victor Stinner wrote: > > > Full example of asynchronous communication with a subprocess (the > > python interactive interpreter) using asyncio high-level API: > > Thank you for writing this. As I explained in response to Josiah, Idle > communicates with a python interpreter subprocess through a socket. > Since making the connection is not dependable, I would like to replace > the socket with the pipes. http://bugs.python.org/issue18823 > > However, the code below creates a subprocess for one command and one > response, which can apparently be done now with subprocess.communicate. > What I and others need is a continuing (non-blocking) conversion with 1 > and only 1 subprocess (see my response to Josiah), and that is much more > difficult. So this code does not do what he claims his will do. Why don't you use multiprocessing or concurrent.futures? They have everything you need for continuous conversation between processes. Regards Antoine. From greg.ewing at canterbury.ac.nz Fri Mar 28 22:19:16 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 29 Mar 2014 10:19:16 +1300 Subject: [Python-Dev] Negative timedelta strings In-Reply-To: References: <20140328110342.GI16526@ando> Message-ID: <5335E754.4010305@canterbury.ac.nz> Alexander Belopolsky wrote: > I meant ISO 8601 syntax for "durations" [1]. ISO 8601 doesn't seem to define a representation for negative durations, though, so it wouldn't solve the original problem. -- Greg From victor.stinner at gmail.com Fri Mar 28 22:24:45 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 28 Mar 2014 22:24:45 +0100 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: Le 28 mars 2014 21:59, "Terry Reedy" a ?crit : > > On 3/28/2014 6:20 AM, Victor Stinner wrote: > >> Full example of asynchronous communication with a subprocess (the >> python interactive interpreter) using asyncio high-level API: > > However, the code below creates a subprocess for one command and one response, which can apparently be done now with subprocess.communicate. What I and others need is a continuing (non-blocking) conversion with 1 and only 1 subprocess (see my response to Josiah), and that is much more difficult. So this code does not do what he claims his will do. I tried to write the shortest example showing how to read and send data and how to make the call blocking. It's different to communicate() because write occurs after the first read. It should be quite easy to enhance my example to execute more commands. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Fri Mar 28 22:29:51 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 28 Mar 2014 14:29:51 -0700 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <8738i4og5f.fsf@elektro.pacujo.net> References: <8738i4og5f.fsf@elektro.pacujo.net> Message-ID: <37C7BDAF-DE48-4303-B908-17A785534D32@gmail.com> On Mar 26, 2014, at 1:31 PM, Marko Rauhamaa wrote: > I have made a full implementation of a balanced tree and would like to > know what the process is to have it considered for inclusion in Python > 3. > > To summarize, the implementation closely parallels dict() features and > resides in _collectionsmodule.c under the name collections.sortedtree. > It uses solely the "<" operator to compare keys. I have chosen the AVL > tree as an implementation technique. FWIW, I think there may be room for a sorted collection somewhere in the standard library. As others have said, the best place to start is by putting a module on PyPi to let it mature and to compete with other approaches to the problem. Here are a few random thoughts on the over-all idea: * An AVL balanced tree isn't the only solution or necessarily the best solution to the problem. Tree nodes tend to take more space than denser structures and they have awful cache locality (these are the same reasons that deques use doubly-linked blocks rather than a plain doubly linked lists). * Another approach I have experimented with uses lazy sorting. That lets insertion be an O(1) step and incurs a one-time sorting cost upon the next lookup (and because Timsort exploits partial orderings, this can be very cheap). A lazy sorted list is dense and sequentially ordered in memory (reducing space overhead, taking full advantage of cache locality and memory controller auto-prefetch, and providing fast iteration speed by not needing to chase pointers). The lazy sort approach works best in applications that spend most of the time doing lookups and have only infrequent deletions and insertions. * The name of the tool probably should not be sortedtree. Ideally, the tool should be named for what it does, not how it does it (for the most part, users don't need to know whether the underlying implementation is a red-black tree, b-tree, judy array, sqlite database, or lazy list). That is why (I think) that Python dicts are called dicts rather than hash tables (the implementation) or an associative array (the computer science term for the abstract datatype). * There are plenty of data structures that have had good utility and popularity outside of the standard library. I that it is a good thing that blists, numpy arrays, databases, and panda dataframes all live outside the standard library. Those tools are easier to maintain externally and it is easier for you to keep control over the design. Remember the saying, "the standard library is where code goes to die" (and I would add that it should already be mature (or nearly dead) by the time it gets there). * That said, it is a reasonable possibility that the standard library would benefit from some kind sorted collection (the idea comes up from time to time). Raymond Hettinger -------------- next part -------------- An HTML attachment was scrubbed... URL: From fred at fdrake.net Fri Mar 28 22:52:24 2014 From: fred at fdrake.net (Fred Drake) Date: Fri, 28 Mar 2014 17:52:24 -0400 Subject: [Python-Dev] Negative timedelta strings In-Reply-To: <5335E754.4010305@canterbury.ac.nz> References: <20140328110342.GI16526@ando> <5335E754.4010305@canterbury.ac.nz> Message-ID: On Fri, Mar 28, 2014 at 5:19 PM, Greg Ewing wrote: > ISO 8601 doesn't seem to define a representation for > negative durations, though, so it wouldn't solve the > original problem. Aside from the horribleness of the ISO 8601 notation for a duration, it's best not to confuse the notions of duration and delta. Notionally, a delta contains more information than a duration. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From marko at pacujo.net Fri Mar 28 22:54:32 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Fri, 28 Mar 2014 23:54:32 +0200 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <37C7BDAF-DE48-4303-B908-17A785534D32@gmail.com> (Raymond Hettinger's message of "Fri, 28 Mar 2014 14:29:51 -0700") References: <8738i4og5f.fsf@elektro.pacujo.net> <37C7BDAF-DE48-4303-B908-17A785534D32@gmail.com> Message-ID: <87fvm2ng4n.fsf@elektro.pacujo.net> Raymond Hettinger : > * An AVL balanced tree isn't the only solution or necessarily the best > solution to the problem. Tree nodes tend to take more space than > denser structures and they have awful cache locality (these are the > same reasons that deques use doubly-linked blocks rather than a plain > doubly linked lists). Maybe. The key is the API. The implementation underneath should be changeable. For example, Jython would probably use SortedTree to implement it. Performance tests should help decide when an implementation is switched for a more efficient one. In some of my tests, I haven't seen any significant performance differences between RB trees and AVL trees, for example. The blist implementation, which I have taken a quick glance at, buys cache locality at the price of block copying; I have no data to decide if the tradeoff is a good one. The main thing, IMO, is to get one sorted dictionary in. > * The name of the tool probably should not be sortedtree. Correct. That was a mistake on the Subject line. In the code, it's sorteddict. > * That said, it is a reasonable possibility that the standard library > would benefit from some kind sorted collection (the idea comes up from > time to time). Yes. As a user, I have craved for an implementation, which is readily available in Java and the linux kernel, for example. Marko From marko at pacujo.net Fri Mar 28 22:57:02 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Fri, 28 Mar 2014 23:57:02 +0200 Subject: [Python-Dev] Negative timedelta strings In-Reply-To: <5335E754.4010305@canterbury.ac.nz> (Greg Ewing's message of "Sat, 29 Mar 2014 10:19:16 +1300") References: <20140328110342.GI16526@ando> <5335E754.4010305@canterbury.ac.nz> Message-ID: <87bnwqng0h.fsf@elektro.pacujo.net> Greg Ewing : > ISO 8601 doesn't seem to define a representation for > negative durations, though, so it wouldn't solve the > original problem. XSD uses ISO 8601 durations and allows a sign before the initial "P". It would appear PT1M means 60 or 61 seconds. P1D means 23, 24 or 25 hours. P1M means 28..31 days etc. Timedelta would have no option but to stick to seconds: P29389453.2345S but then, why not simply use a number: 29389453.2345 Marko From shibturn at gmail.com Fri Mar 28 23:09:11 2014 From: shibturn at gmail.com (Richard Oudkerk) Date: Fri, 28 Mar 2014 22:09:11 +0000 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: <5335F307.1070804@gmail.com> On 28/03/2014 06:35 pm, Josiah Carlson wrote: > > If it were me, I'd define three methods, with longer names to > clarify what they do, e.g. > > proc.write_nonblocking(data) > data = proc.read_nonblocking() > data = proc.read_stderr_nonblocking() > > > Easily doable. To implement write_nonblocking() on Windows, do you intend to use SetNamedPipeHandleState() with PIPE_NOWAIT? The documentation discourages using this: Note that nonblocking mode is supported for compatibility with Microsoft LAN Manager version 2.0 and should not be used to achieve asynchronous input and output (I/O) with named pipes. And I guess you will need to use a poll/sleep loop to simulate blocking or multiplexing. If you want expect-like behaviour then you need some sort of multiplexing. -- RIchard** -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Mar 28 23:47:18 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 28 Mar 2014 15:47:18 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <5332E5C8.1010506@stoneleaf.us> <533358E9.3060107@stoneleaf.us> Message-ID: <5335FBF6.2080307@stoneleaf.us> On 03/27/2014 04:26 AM, Nick Coghlan wrote: > On 27 March 2014 20:47, Victor Stinner wrote: >> The PEP 461 looks good to me. It's a nice addition to Python 3.5 and >> the PEP is well defined. > > +1 from me as well. One minor request is that I don't think the > rationale for rejecting numbers from "%s" is complete [...] Changed to --------------------------------------------------------------------------------- In particular, ``%s`` will not accept numbers nor ``str``. ``str`` is rejected as the string to bytes conversion requires an encoding, and we are refusing to guess; numbers are rejected because: - what makes a number is fuzzy (float? Decimal? Fraction? some user type?) - allowing numbers would lead to ambiguity between numbers and textual representations of numbers (3.14 vs '3.14') - given the nature of wire formats, explicit is definitely better than implicit --------------------------------------------------------------------------------- >> Note: I fixed a typo in your PEP (reST syntax). > > I also committed a couple of markup tweaks, since it seemed easier to > just fix them than explain what was broken. Thanks to both of you for that. > However, there are also > two dead footnotes (4 & 5), which I have left alone - I'm not sure if > the problem is a missing reference, or if the footnote can go away > now. Fixed. -- ~Ethan~ From marko at pacujo.net Sat Mar 29 00:38:18 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Sat, 29 Mar 2014 01:38:18 +0200 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <87fvm2ng4n.fsf@elektro.pacujo.net> (Marko Rauhamaa's message of "Fri, 28 Mar 2014 23:54:32 +0200") References: <8738i4og5f.fsf@elektro.pacujo.net> <37C7BDAF-DE48-4303-B908-17A785534D32@gmail.com> <87fvm2ng4n.fsf@elektro.pacujo.net> Message-ID: <87ha6hki6t.fsf@elektro.pacujo.net> Marko Rauhamaa : > For example, Jython would probably use SortedTree to implement it. That word just keeps coming out of my keyboard. The Java class is of course the TreeMap. Marko From ncoghlan at gmail.com Sat Mar 29 00:55:42 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 Mar 2014 09:55:42 +1000 Subject: [Python-Dev] libpython added to ABI tracker In-Reply-To: <20140328172521.2a2abe7e@fsol> References: <5335337F.8000303@rosalab.ru> <20140328172521.2a2abe7e@fsol> Message-ID: On 29 March 2014 02:25, Antoine Pitrou wrote: > On Fri, 28 Mar 2014 12:31:59 +0400 > Andrey Ponomarenko wrote: >> Hi, >> >> The libpython library has been added to the ABI tracker: >> http://upstream-tracker.org/versions/python.html >> >> The page lists library versions and changes in API/ABI. > > Thanks. Do note that most of these changes are on private (internal) > APIs that normal user code should never invoke / rely on (for example > all those APIs starting with an underscore). That's where having the stable ABI tracked as well would really come into its own, as any regressions in that *are* a big deal. I also consider it a very good sign that the maintenance releases are very stable from an ABI perspective - that's exactly what it supposed to happen :) Cheers, Nick. From ncoghlan at gmail.com Sat Mar 29 01:03:30 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 Mar 2014 10:03:30 +1000 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <87vbuynwny.fsf@uwakimon.sk.tsukuba.ac.jp> References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> <87vbuynwny.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 29 March 2014 01:57, Stephen J. Turnbull wrote: > Another way to put it is, we need a better way to fund support of > "routine maintenance" (ie, the unfun parts) than negotiating it module > by module. Yes, yes we do, and there *are* people working on that (see https://wiki.python.org/moin/PythonSoftwareFoundation/BoardCandidates2014) :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From g.rodola at gmail.com Sat Mar 29 01:24:14 2014 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sat, 29 Mar 2014 01:24:14 +0100 Subject: [Python-Dev] Fwd: shutdown concurrent.futures.ProcessPoolExecutor Message-ID: This is a follow up of: https://groups.google.com/forum/#!topic/python-tulip/91NCCqV4SFs According to the information I collected so far it seems it's not possible (or at least very hard) to cleanly shutdown a process pool and all its workers in case of KeyboardInterrupt / SIGINT. Literally, what I would like to do is this: - intercept SIGINT form the main process and/or from each worker process - send SIGINT to all workers - wait for them to finish with a timeout - finally exit Whereas it exists a solution based on multiprocessing: http://noswap.com/blog/python-multiprocessing-keyboardinterrupt ...concurrent.futures.ProcessPoolExecutor does not expose all the necessary bits to emulate it. By opening this thread I hope either to find a solution and document it or if it turns out that it is simply not possible to do this via concurrent.futures then discuss whether it's the case to expose more ProcessPoolExecutor APIs, because right now the interface it offers is pretty limited compared to multiprocessing.Pool: http://docs.python.org/3.5/library/multiprocessing.html#multiprocessing.pool.Pool In particular, it won't let you pass an initializer function to pass to multiprocessing.Pool nor terminate() the process pool (only wait() for it). ---------- Forwarded message ---------- From: Guido van Rossum Date: Sat, Mar 29, 2014 at 12:05 AM Subject: Re: [python-tulip] How to cleanly shutdown the IO loop when using run_in_executor()? To: Giampaolo Rodola' Cc: python-tulip You're going to have to move the discussion to python-dev or the python issue tracker then. On Fri, Mar 28, 2014 at 4:02 PM, Giampaolo Rodola' wrote: > > On Wed, Mar 26, 2014 at 7:35 PM, Guido van Rossum wrote: > >> Another thing to investigate might be how the executor creates the >> processes, and if anything happens to signals there. > > > After some further digging this seems to be related with the problem at > hand. > According to this: > http://noswap.com/blog/python-multiprocessing-keyboardinterrupt > ...it appears a feasible solution is to prevent workers to ever receive > KeyboardInterrupt and have the main process shutdown the pool via > pool.terminate() and finally pool.join(). > Unfortunately concurrent.futures.ProcessPoolExecutor does not expose all > the necessary bits to do that (multiprocessing.Pool(initializer=...) > argument and terminate() method). > > I also suspect that in order to emulate the suggested solution the > underlying Process instance should be daemonized, similarly to what > multiprocessing.Pool does: > > http://hg.python.org/cpython/file/3567d7ebd382/Lib/multiprocessing/pool.py#l222 > > I wonder whether concurrent.futures.ProcessPoolExecutor would be better > off exposing more APIs in order to facilitate tasks like these. > > > -- > Giampaolo - http://grodola.blogspot.com > > -- --Guido van Rossum (python.org/~guido) -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From stutzbach at google.com Sat Mar 29 01:48:05 2014 From: stutzbach at google.com (Daniel Stutzbach) Date: Fri, 28 Mar 2014 17:48:05 -0700 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <87fvm2ng4n.fsf@elektro.pacujo.net> References: <8738i4og5f.fsf@elektro.pacujo.net> <37C7BDAF-DE48-4303-B908-17A785534D32@gmail.com> <87fvm2ng4n.fsf@elektro.pacujo.net> Message-ID: On Fri, Mar 28, 2014 at 2:54 PM, Marko Rauhamaa wrote: > The blist implementation, which I have taken a quick glance at, > buys cache locality at the price of block copying; I have no data to > decide if the tradeoff is a good one. > There's actually a compile-time parameter so that you can tune the tradeoff by adjusting how wide each of blist's nodes are. Like most tradeoffs, whether it's good or bad depends on what you're trying to do. RedBlack trees are very similar to a B-tree with a node-width of 4: http://en.wikipedia.org/wiki/Red%E2%80%93black_tree#Analogy_to_B-trees_of_order_4 blist's sorteddict is written in Python on top of the blist type (which is written in C). Because of the Python layer, it's slower by a constant factor compared to pure-C implementations in microbenchmarks. -- Daniel Stutzbach -------------- next part -------------- An HTML attachment was scrubbed... URL: From drsalists at gmail.com Sat Mar 29 02:45:35 2014 From: drsalists at gmail.com (Dan Stromberg) Date: Fri, 28 Mar 2014 18:45:35 -0700 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <37C7BDAF-DE48-4303-B908-17A785534D32@gmail.com> <87fvm2ng4n.fsf@elektro.pacujo.net> Message-ID: On Fri, Mar 28, 2014 at 5:48 PM, Daniel Stutzbach wrote: > On Fri, Mar 28, 2014 at 2:54 PM, Marko Rauhamaa wrote: >> >> The blist implementation, which I have taken a quick glance at, >> >> buys cache locality at the price of block copying; I have no data to >> decide if the tradeoff is a good one. > > > There's actually a compile-time parameter so that you can tune the tradeoff > by adjusting how wide each of blist's nodes are. Like most tradeoffs, > whether it's good or bad depends on what you're trying to do. RedBlack > trees are very similar to a B-tree with a node-width of 4: > http://en.wikipedia.org/wiki/Red%E2%80%93black_tree#Analogy_to_B-trees_of_order_4 > > blist's sorteddict is written in Python on top of the blist type (which is > written in C). Because of the Python layer, it's slower by a constant > factor compared to pure-C implementations in microbenchmarks. It grieves me a bit to say this, and blist and blist's sorteddict are probably a good tool for an appropriate job, but blist's sorteddict type is quite a bit slower than several pure python implementations of dictionaries with ordered keys. In my testing blist.sorteddict was dead last for random keys, and wasn't last but was still significantly underperforming for sequential keys (outperforming only binary tree and scapegoat tree, behind all others): http://stromberg.dnsalias.org/~strombrg/python-tree-and-heap-comparison/2014-03-18/ From stephen at xemacs.org Sat Mar 29 08:46:48 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 29 Mar 2014 16:46:48 +0900 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> <87vbuynwny.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87r45lo39z.fsf@uwakimon.sk.tsukuba.ac.jp> Tres Seaver writes: > I'm mostly arguing the FLOSS project You mean "a (mostly) volunteer-supported" FLOSS project, no? > should feel free to ignore Given the above qualification, you can put a period here, as far as I'm concerned. My question is "what does *Python* *want* to ignore?", not "is it allowed to ignore?" > high-maintenance-cost What "high" maintenance cost? A stdlib addition is a marginal increase in cost. Do too many, and it's a serious burden, of course. So there needs to be *some* hurdle that any addition must clear, and that barrier becomes higher in proportion to the "code base to developers who actually do maintenance" ratio. But I think the question should be "how high?" not "can they pay?" > commercial concerns until those concerns bring either blook (funded > developer time) or treasure (pooled to pay for the same time) to > the table to pay for them. I really don't think commercial profit as the motive for a request, or ability to pay, should be an important reason to *ignore* user wants. This smacks of the proposals for "ransom software", which (a) sort of turns the principle of *volunteer* FLOSS on its head (if that doesn't bother most of the developers, no problem, but for one it bothers *me*), and (b) doesn't actually work AFAICT (it's not quite the same as "crowdfunding", which does work). It's rather the reverse: I believe we should be prepared to deal with the conflict of interest that results when *some* of the developers are offered money to provide *cater* to such concerns[1] where the community doesn't think the benefit is that high. Footnotes: [1] Should not be that hard in *this* community, but that is the issue IMO. From tjreedy at udel.edu Sat Mar 29 09:38:18 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 29 Mar 2014 04:38:18 -0400 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: On 3/28/2014 5:09 PM, Guido van Rossum wrote: > To be clear, the proposal for Idle would be to still use the RPC > protocol, but run it over a pipe instead of a socket, right? The was and is the current proposal, assuming that it is the easiest thing to do that would work. While responding to Victor, it occurred to me as a speculative idea that once pipes were working, it *might* be possible to implement the protocol as class methods, and let user code go directly to the interpreter. I would ask what you thought before seriously working on the idea. -- Terry Jan Reedy From tjreedy at udel.edu Sat Mar 29 09:44:32 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 29 Mar 2014 04:44:32 -0400 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: <20140328221255.5c923901@fsol> References: <20140325231947.5ec9ad63@fsol> <20140328221255.5c923901@fsol> Message-ID: On 3/28/2014 5:12 PM, Antoine Pitrou wrote: > On Fri, 28 Mar 2014 16:58:25 -0400 > Terry Reedy wrote: >> However, the code below creates a subprocess for one command and one >> response, which can apparently be done now with subprocess.communicate. >> What I and others need is a continuing (non-blocking) conversion with 1 >> and only 1 subprocess (see my response to Josiah), and that is much more >> difficult. So this code does not do what he claims his will do. > > Why don't you use multiprocessing or concurrent.futures? They have > everything you need for continuous conversation between processes. I have not used either and no one suggested either before, while Amaury Forgeot d'Arc and Guido suggested subprocess pipes. I added those two ideas to the issue. -- Terry Jan Reedy From solipsis at pitrou.net Sat Mar 29 16:30:25 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 29 Mar 2014 16:30:25 +0100 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen References: <20140325231947.5ec9ad63@fsol> <20140328221255.5c923901@fsol> Message-ID: <20140329163025.60ee9403@fsol> On Sat, 29 Mar 2014 04:44:32 -0400 Terry Reedy wrote: > On 3/28/2014 5:12 PM, Antoine Pitrou wrote: > > On Fri, 28 Mar 2014 16:58:25 -0400 > > Terry Reedy wrote: > > >> However, the code below creates a subprocess for one command and one > >> response, which can apparently be done now with subprocess.communicate. > >> What I and others need is a continuing (non-blocking) conversion with 1 > >> and only 1 subprocess (see my response to Josiah), and that is much more > >> difficult. So this code does not do what he claims his will do. > > > > Why don't you use multiprocessing or concurrent.futures? They have > > everything you need for continuous conversation between processes. > > I have not used either and no one suggested either before, while Amaury > Forgeot d'Arc and Guido suggested subprocess pipes. I added those two > ideas to the issue. Looking at idlelib/rpc.py, it looks largely like an uncommented (untested?) reimplementation of multiprocessing pipes, with weird architecture choices (RPCServer is actually a client?). multiprocessing should have everything you need: you can run child processes, communicate with them using Queues, Locks, Conditions, or you can even automate asynchronous execution with a process Pool. Those are cross-platform and use the most appropriate platform-specific primitives (for examples named pipes under Windows). They are also quite well-tested, and duly maintained by Richard :-) Regards Antoine. From rdmurray at bitdance.com Sat Mar 29 17:43:01 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 29 Mar 2014 12:43:01 -0400 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: <20140329163025.60ee9403@fsol> References: <20140325231947.5ec9ad63@fsol> <20140328221255.5c923901@fsol> <20140329163025.60ee9403@fsol> Message-ID: <20140329164301.7A93D250D0C@webabinitio.net> On Sat, 29 Mar 2014 16:30:25 +0100, Antoine Pitrou wrote: > On Sat, 29 Mar 2014 04:44:32 -0400 > Terry Reedy wrote: > > On 3/28/2014 5:12 PM, Antoine Pitrou wrote: > > > On Fri, 28 Mar 2014 16:58:25 -0400 > > > Terry Reedy wrote: > > > > >> However, the code below creates a subprocess for one command and one > > >> response, which can apparently be done now with subprocess.communicate. > > >> What I and others need is a continuing (non-blocking) conversion with 1 > > >> and only 1 subprocess (see my response to Josiah), and that is much more > > >> difficult. So this code does not do what he claims his will do. > > > > > > Why don't you use multiprocessing or concurrent.futures? They have > > > everything you need for continuous conversation between processes. > > > > I have not used either and no one suggested either before, while Amaury > > Forgeot d'Arc and Guido suggested subprocess pipes. I added those two > > ideas to the issue. > > Looking at idlelib/rpc.py, it looks largely like an uncommented > (untested?) reimplementation of multiprocessing pipes, with weird > architecture choices (RPCServer is actually a client?). I think instead we might call a "pre" implementation :) I'm pretty sure this Idle stuff existed before multiprocessing did. (In English, 'reimplementation' implies that multiprocessing existed already, and therefore implies someone looked at it and copied the concepts in it it badly, which is not the case as far as I'm aware.) --David From yselivanov.ml at gmail.com Sat Mar 29 18:19:08 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sat, 29 Mar 2014 13:19:08 -0400 Subject: [Python-Dev] cpython: inspect.Signature: Add 'Signature.from_callable' classmethod. Closes #17373 In-Reply-To: References: <3fvrmy6cvPz7Ll5@mail.python.org> <20140327174816.25e87c40@fsol> Message-ID: <5337008C.8080503@gmail.com> Thanks, it's fixed now. Yury On 2014-03-28, 10:29 AM, Andrew Svetlov wrote: > And probably the block should be deindented > > On Thu, Mar 27, 2014 at 6:48 PM, Antoine Pitrou wrote: >> On Thu, 27 Mar 2014 17:12:02 +0100 (CET) >> yury.selivanov wrote: >>> + .. classmethod:: Signature.from_callable(obj) >>> + >>> + Return a :class:`Signature` (or its subclass) object for a given callable >>> + ``obj``. This method simplifies subclassing of :class:`Signature`: >>> + >>> + :: >>> + >>> + class MySignature(Signature): >>> + pass >>> + sig = MySignature.from_callable(min) >>> + assert isinstance(sig, MySignature) >>> + >> This needs a "versionadded" tag. >> >> Regards >> >> Antoine. >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com > > From ethan at stoneleaf.us Sat Mar 29 18:05:51 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 29 Mar 2014 10:05:51 -0700 Subject: [Python-Dev] PyErr_ Fetch | Restore & friends Message-ID: <5336FD6F.1070900@stoneleaf.us> I'm working on issue 1615 [1] and came up with this tidbit, which works [2], but not well enough: slot_tp_getattr_hook(PyObject *self, PyObject *name) { ... + PyObject *error_type, *error_value, *error_traceback; ... + /* if an AttributeError is set, save it and call getattr; if getattr + sets an AttributeError, discard it and return the original + AttributeError */ if (res == NULL && PyErr_ExceptionMatches(PyExc_AttributeError)) { + PyErr_Fetch(&error_type, &error_value, &error_traceback); res = call_attribute(self, getattr, name); + if (res == NULL && PyErr_ExceptionMatches(PyExc_AttributeError)) + PyErr_Restore(error_type, error_value, error_traceback); } So I tried adding in a second call to PyErr_Fetch and adding the original error as context, like so: if (res == NULL && PyErr_ExceptionMatches(PyExc_AttributeError)) { PyErr_Fetch(&ctx_error_type, &ctx_error_value, &ctx_error_traceback); res = call_attribute(self, getattr, name); if (res == NULL && PyErr_ExceptionMatches(PyExc_AttributeError)) { PyErr_Fetch(&error_type, &error_value, &error_traceback); PyException_SetContext(error_value, ctx_error_value); PyErr_Restore(error_type, error_value, error_traceback); } else { Py_DECREF(ctx_error_value); } Py_DECREF(ctx_error_type); Py_DECREF(ctx_error_traceback); } This bit of code won't even finish compiling. I am not sure if my understanding of references (and how these functions create/consume them) or my understanding of when and where to call PyErr_Fetch|Restore or PyException_SetContext is at fault, but I would greatly appreciate somebody correcting my understanding. :) -- ~Ethan~ [1] http://http://bugs.python.org/issue1615 [2] works as in everything compiles, but the resulting behavior introduces a new bug (the message of a 'raise AttributeError' in a __getattr__ is ignored). From greg at krypto.org Sat Mar 29 19:53:45 2014 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 29 Mar 2014 11:53:45 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <20140327230555.571227d2@fsol> References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> <20140327230555.571227d2@fsol> Message-ID: On Thu, Mar 27, 2014 at 3:05 PM, Antoine Pitrou wrote: > On Thu, 27 Mar 2014 18:47:59 +0000 > Brett Cannon wrote: > > On Thu Mar 27 2014 at 2:42:40 PM, Guido van Rossum > wrote: > > > > > Much better, but I'm still not happy with including %s at all. > Otherwise > > > it's accept-worthy. (How's that for pressure. :-) > > > > > > > But if we only add %b and leave out %s then how is this going to lead to > > Python 2/3 compatible code since %b is not in Python 2? Or am I > > misunderstanding you? > > I think we have reached a point where adding porting-related facilities > in 3.5 may actually slow down the pace of porting, rather than > accelerate it (because people will then wait for 3.5 to start porting > stuff). > I understand that sentiment but that is an unjustified fear. It is not a good reason not to do it. Projects are already trying to port stuff today and running into roadblocks when it comes to ascii-compatible bytes formatting for real world data formats in code needing to be 2.x compatible. I'm pulling out my practicality beats purity card here. Mercurial is one of the large Python 2.4-2.7 code bases that needs this feature in order to support Python 3 in a sane manner. (+Augie Fackler to look at the latest http://legacy.python.org/dev/peps/pep-0461/ to confirm usefulness) -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Mar 29 19:59:07 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 29 Mar 2014 19:59:07 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> <20140327230555.571227d2@fsol> Message-ID: <20140329195907.0e6d40f1@fsol> On Sat, 29 Mar 2014 11:53:45 -0700 "Gregory P. Smith" wrote: > > I understand that sentiment but that is an unjustified fear. It is not a > good reason not to do it. Projects are already trying to port stuff today > and running into roadblocks when it comes to ascii-compatible bytes > formatting for real world data formats in code needing to be 2.x > compatible. I'm pulling out my practicality beats purity card here. "Roadblocks" is an unjustified term here. Important code bases such as Tornado have already achieved this a long time ago. While lack of bytes formatting does make porting harder, it is not a "roadblock" as in you can't work it around. > Mercurial is one of the large Python 2.4-2.7 code bases that needs this > feature in order to support Python 3 in a sane manner. (+Augie Fackler to > look at the latest http://legacy.python.org/dev/peps/pep-0461/ to confirm > usefulness) http://www.selenic.com/pipermail/mercurial-devel/2014-March/057474.html Regards Antoine. From greg at krypto.org Sat Mar 29 20:01:53 2014 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 29 Mar 2014 12:01:53 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <5332E5C8.1010506@stoneleaf.us> <533358E9.3060107@stoneleaf.us> Message-ID: On Thu, Mar 27, 2014 at 3:47 AM, Victor Stinner wrote: > The PEP 461 looks good to me. It's a nice addition to Python 3.5 and > the PEP is well defined. > +1 > I can help to implement it. Maybe, it would be nice to provide an > implementation as a third-party party module on PyPI for Python > 2.6-3.4. > That is possible and would enable bytes formatting on earlier 3.x versions. I'm not sure if there is any value in backporting to 2.x as those already have such formatting with Python 2's str.__mod__ % operator. Though I don't know what it'd look like as an API as a module. Brainstorming: It'd either involve function calls to format instead of % or a container class to wrap format strings in with a __mod__ method that calls the bytes formatting code instead of native str % formatting when needed. >From a 2.x-3.x compatible code standpoint the above could exist but the container class constructor would be a no-op on Python 2. if sys.version_info[0] == 2: BytesFormatter = str else: class BytesFormatter: ... def __mod__ ... -gps > Note: I fixed a typo in your PEP (reST syntax). > > Victor > > 2014-03-26 23:47 GMT+01:00 Ethan Furman : > >> This one is wrong: > >> > >>>>> repr(b'abc').encode('ascii', 'backslashreplace') > >> > >> b"b'abc'" > > > > > > Fixed, thanks. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Sat Mar 29 20:10:23 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 29 Mar 2014 12:10:23 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <5332E5C8.1010506@stoneleaf.us> <533358E9.3060107@stoneleaf.us> Message-ID: <53371A9F.9050004@g.nevcal.com> On 3/29/2014 12:01 PM, Gregory P. Smith wrote: > From a 2.x-3.x compatible code standpoint the above could exist but > the container class constructor would be a no-op on Python 2. > if sys.version_info[0] == 2: > BytesFormatter = str > else: > class BytesFormatter: ... def __mod__ ... If done as a container class, the Python 2 version should implement the same restrictions on %s for numerics, and implement %b. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Mar 29 21:04:11 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 29 Mar 2014 16:04:11 -0400 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: <20140329163025.60ee9403@fsol> References: <20140325231947.5ec9ad63@fsol> <20140328221255.5c923901@fsol> <20140329163025.60ee9403@fsol> Message-ID: On 3/29/2014 11:30 AM, Antoine Pitrou wrote: > On Sat, 29 Mar 2014 04:44:32 -0400 > Terry Reedy wrote: >> On 3/28/2014 5:12 PM, Antoine Pitrou wrote: [for Idle] >>> Why don't you use multiprocessing or concurrent.futures? They have >>> everything you need for continuous conversation between processes. >> >> I have not used either and no one suggested either before, while Amaury >> Forgeot d'Arc and Guido suggested subprocess pipes. I added those two >> ideas to the issue. > > Looking at idlelib/rpc.py, it looks largely like an uncommented Some things have comments or docstrings; I am adding some as I can. > (untested?) The test of most of Idle is that it works when a person tries to use it, which it most does. I am working on automated tests too. > reimplementation of multiprocessing pipes, with weird > architecture choices (RPCServer is actually a client?). As David said, pre-implementation. It is at least a decade old. > multiprocessing should have everything you need: you can run child > processes, communicate with them using Queues, Locks, Conditions, or > you can even automate asynchronous execution with a process Pool. Those > are cross-platform and use the most appropriate platform-specific > primitives (for examples named pipes under Windows). They are also > quite well-tested, and duly maintained by Richard :-) This is not the only thing Idle does that is or should be done elsewhere. If it is done elsewhere in the stdlib (and tested), I am happy to switch. Idle originally created calltips from code objects and docstrings. When inspect.get... did most of the job for functions coded in Python, Idle switched to using that and some calltips code was removed. Once most C coded functions work with inspect.signature (and I do hope the ArgClinic work gets done for 3.5), I will switch and delete more code and some tests. -- Terry Jan Reedy From ncoghlan at gmail.com Sat Mar 29 21:47:54 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Mar 2014 06:47:54 +1000 Subject: [Python-Dev] PyErr_ Fetch | Restore & friends In-Reply-To: <5336FD6F.1070900@stoneleaf.us> References: <5336FD6F.1070900@stoneleaf.us> Message-ID: On 30 March 2014 03:05, Ethan Furman wrote: > > This bit of code won't even finish compiling. I am not sure if my > understanding of references (and how these functions create/consume them) or > my understanding of when and where to call PyErr_Fetch|Restore or > PyException_SetContext is at fault, but I would greatly appreciate somebody > correcting my understanding. :) I had to figure this out for the codec exception chaining in Python 3.4, and the main piece that appears to be missing from your code is an appropriate call to PyErr_NormalizeException(). (The interpreter tries to avoid actually instantiating exceptions if it can, with the consequence that explicit normalisation is often needed before non-trivial operations on the current exception) This is the code that does the heavy lifting for the codec exception chaining in Python 3.4: http://hg.python.org/cpython/file/36492f927a40/Objects/exceptions.c#l2644 Sine you're raising a genuinely new exception, rather than implicitly replacing an existing one, you don't need all the checks I have in that function to make sure the implicit replacement is OK. You'd just need some more like the final section that actually creates the new exception and links it up with the old one: http://hg.python.org/cpython/file/36492f927a40/Objects/exceptions.c#l2764 Setting the context correctly for exceptions raised in C should be similar to that, just setting the context rather than the cause. (Although don't forget the bit earlier in the function that populates the traceback correctly on the original exception) I thought I had filed a "Add a public API to make properly chained exceptions easier to raise from C" RFE while working on the codec chaining, but I'm not seeing anything now... Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Sat Mar 29 22:01:31 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 29 Mar 2014 14:01:31 -0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <20140329195907.0e6d40f1@fsol> References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> <20140327230555.571227d2@fsol> <20140329195907.0e6d40f1@fsol> Message-ID: <533734AB.2030107@stoneleaf.us> On 03/29/2014 11:59 AM, Antoine Pitrou wrote: > On Sat, 29 Mar 2014 11:53:45 -0700 "Gregory P. Smith" wrote: >> >> I understand that sentiment but that is an unjustified fear. It is not a >> good reason not to do it. Projects are already trying to port stuff today >> and running into roadblocks when it comes to ascii-compatible bytes >> formatting for real world data formats in code needing to be 2.x >> compatible. I'm pulling out my practicality beats purity card here. > > "Roadblocks" is an unjustified term here. It's actually quite appropriate: to get around a physical roadblock you would have to leave the road, forge through lumpy ground and stinging nettles, and then get back on the road. A very good analogy, actually. ;) -- ~Ethan~ From ethan at stoneleaf.us Sat Mar 29 22:05:36 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 29 Mar 2014 14:05:36 -0700 Subject: [Python-Dev] PyErr_ Fetch | Restore & friends In-Reply-To: References: <5336FD6F.1070900@stoneleaf.us> Message-ID: <533735A0.6050909@stoneleaf.us> On 03/29/2014 01:47 PM, Nick Coghlan wrote: > On 30 March 2014 03:05, Ethan Furman wrote: >> >> This bit of code won't even finish compiling. I am not sure if my >> understanding of references (and how these functions create/consume them) or >> my understanding of when and where to call PyErr_Fetch|Restore or >> PyException_SetContext is at fault, but I would greatly appreciate somebody >> correcting my understanding. :) > > I had to figure this out for the codec exception chaining in Python > 3.4, and the main piece that appears to be missing from your code is > an appropriate call to PyErr_NormalizeException(). (The interpreter > tries to avoid actually instantiating exceptions if it can, with the > consequence that explicit normalisation is often needed before > non-trivial operations on the current exception)h One of the many gyrations I tried before asking for help involved calling PyErr_NormalizeException, but it didn't seem to solve the problem so I removed it before posting the minimum that I thought should work. > This is the code that does the heavy lifting for the codec exception > chaining in Python 3.4: > > http://hg.python.org/cpython/file/36492f927a40/Objects/exceptions.c#l2644 > http://hg.python.org/cpython/file/36492f927a40/Objects/exceptions.c#l2764 Thanks, I'll check those out. -- ~Ethan~ From ncoghlan at gmail.com Sat Mar 29 22:51:48 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Mar 2014 07:51:48 +1000 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: <533734AB.2030107@stoneleaf.us> References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> <20140327230555.571227d2@fsol> <20140329195907.0e6d40f1@fsol> <533734AB.2030107@stoneleaf.us> Message-ID: On 30 March 2014 07:01, Ethan Furman wrote: > On 03/29/2014 11:59 AM, Antoine Pitrou wrote: > >> On Sat, 29 Mar 2014 11:53:45 -0700 "Gregory P. Smith" wrote: >>> >>> >>> I understand that sentiment but that is an unjustified fear. It is not a >>> good reason not to do it. Projects are already trying to port stuff today >>> and running into roadblocks when it comes to ascii-compatible bytes >>> formatting for real world data formats in code needing to be 2.x >>> compatible. I'm pulling out my practicality beats purity card here. >> >> >> "Roadblocks" is an unjustified term here. > > > It's actually quite appropriate: to get around a physical roadblock you > would have to leave the road, forge through lumpy ground and stinging > nettles, and then get back on the road. > > A very good analogy, actually. ;) I tend to call them "barriers to migration". Up to Python 3.4, my focus has been more on general "barriers to entry" for Python 3 that applied as much or more to new users as they did to existing ones - hence working on getting pip incorporated, providing a better path to mastery for the codec system, helping Larry with Argument Clinic, helping Eric with the simpler import customisation, trying to help improve the integration with the POSIX text model, assorted tweaks to make the type system more accessible etc. I think Python 3.4 is now in a pretty good place on that front, particularly with Larry stating up front that he considers the ongoing rollout of Argument Clinic usage to be in scope for Python 3.4.x maintenance releases. So for 3.5, I think it makes sense to focus on those "barriers to migration" and other activities that benefit existing Python 2 users more so than users that are completely new to Python and starting directly with Python 3. Binary interpolation is a big one (thanks Ethan!), as is the proposed policy change to allow network security features to evolve within Python 2.7 maintenance releases. Our community has done a lot of work to support us in our goal of modernising and migrating a large fraction of the ecosystem to a new version of the language, even though the full implications of the revised models for binary and text data turned out to be more profound than I think any of us realised back in 2006 when Guido first turned the previously hypothetical "Py3k" into a genuine active effort to create a new revision of the language, better suited to the global nature of the 21st century. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Sat Mar 29 23:04:09 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 29 Mar 2014 18:04:09 -0400 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <20140327122449.71a50ea2@fsol> <53344AB4.50904@stoneleaf.us> <53346603.1000402@stoneleaf.us> <5334681C.40904@stoneleaf.us> <20140327230555.571227d2@fsol> Message-ID: > On Thu, Mar 27, 2014 at 3:05 PM, Antoine Pitrou > wrote: > I think we have reached a point where adding porting-related facilities AFAIK, The only porting specific feature is %s as a synonym for %b. Not pretty, but tolerable. Otherwise, I have the impression that the PEP pretty much stands on its own. > in 3.5 may actually slow down the pace of porting, rather than > accelerate it (because people will then wait for 3.5 to start porting > stuff). Or, they should download the source and compile and continue or start porting as soon as the bytes % is added. Having earlier Windows and Mac preview binaries might help a tiny bit. If you are saying that Py3 development should not be driven by Py2 concerns, I agree. -- Terry Jan Reedy From tseaver at palladion.com Sat Mar 29 23:04:34 2014 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 29 Mar 2014 18:04:34 -0400 Subject: [Python-Dev] collections.sortedtree In-Reply-To: <87r45lo39z.fsf@uwakimon.sk.tsukuba.ac.jp> References: <8738i4og5f.fsf@elektro.pacujo.net> <53333E55.6040209@stoneleaf.us> <874n2kqcw1.fsf@uwakimon.sk.tsukuba.ac.jp> <87vbuynwny.fsf@uwakimon.sk.tsukuba.ac.jp> <87r45lo39z.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/29/2014 03:46 AM, Stephen J. Turnbull wrote: > I really don't think commercial profit as the motive for a request, > or ability to pay, should be an important reason to *ignore* user > wants. We've already got corrosion on the terminals the leaky batteries-included stdlib (ssl, anyone?). I see nothing wrong with rejecting additional batteries proposed purely FBO organizations who have the kind of policies you describe, but don't contribute blood-or-treasure toward their maintenannce (*especially* because they are in the "enterprise" space, and could be expected to pay for that kind of support). This kind of "Python-in-a-tie" discussion is probably moot for any version of Python that python-dev actually cares about, BTW: by the time such organizations get around to using 3.5, it likely won't even be getting security releases from the core developers. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlM3Q2wACgkQ+gerLs4ltQ7Y5QCdGhrpenkXJnRXoseVabQDwJHX 0zcAn0tAf6iHT276oxjZS/ZhPu49wF8M =6Gci -----END PGP SIGNATURE----- From ncoghlan at gmail.com Sun Mar 30 01:28:33 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Mar 2014 10:28:33 +1000 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <5332E5C8.1010506@stoneleaf.us> <533358E9.3060107@stoneleaf.us> Message-ID: On 30 March 2014 05:01, Gregory P. Smith wrote: > > > > On Thu, Mar 27, 2014 at 3:47 AM, Victor Stinner > wrote: >> >> The PEP 461 looks good to me. It's a nice addition to Python 3.5 and >> the PEP is well defined. > > > +1 > >> >> I can help to implement it. Maybe, it would be nice to provide an >> implementation as a third-party party module on PyPI for Python >> 2.6-3.4. > > > That is possible and would enable bytes formatting on earlier 3.x versions. > I'm not sure if there is any value in backporting to 2.x as those already > have such formatting with Python 2's str.__mod__ % operator. > > Though I don't know what it'd look like as an API as a module. > Brainstorming: It'd either involve function calls to format instead of % or > a container class to wrap format strings in with a __mod__ method that calls > the bytes formatting code instead of native str % formatting when needed. The "future" project already contains a full backport of a true bytes type, rather than relying on Python 2 str objects: http://python-future.org/what_else.html#bytes It seems to me that the easiest way to make any forthcoming Python 3.5 enhancements (both binary interpolation and the other cleanups we are discussing over on Python ideas) available to single source 2/3 code bases is to commit to an API freeze for *those particular builtins* early, and then update "future" accordingly. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Sun Mar 30 03:35:02 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 29 Mar 2014 21:35:02 -0400 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3 In-Reply-To: References: <53320517.3010101@stoneleaf.us> <5332E5C8.1010506@stoneleaf.us> <533358E9.3060107@stoneleaf.us> Message-ID: On 3/29/2014 8:28 PM, Nick Coghlan wrote: > The "future" project already contains a full backport of a true bytes > type, rather than relying on Python 2 str objects: > http://python-future.org/what_else.html#bytes That project looks really nice! > It seems to me that the easiest way to make any forthcoming Python 3.5 > enhancements (both binary interpolation and the other cleanups we are > discussing over on Python ideas) available to single source 2/3 code > bases is to commit to an API freeze for *those particular builtins* > early, and then update "future" accordingly. I agree. I think syntax changes should be in by the first alpha, if not before. -- Terry Jan Reedy From josiah.carlson at gmail.com Sun Mar 30 08:58:59 2014 From: josiah.carlson at gmail.com (Josiah Carlson) Date: Sat, 29 Mar 2014 23:58:59 -0700 Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen In-Reply-To: References: <20140325231947.5ec9ad63@fsol> Message-ID: I've got a patch with partial tests and documentation that I'm holding off on upload because I believe there should be a brief discussion. Long story short, Windows needs a thread to handle writing in a non-blocking fashion, regardless of the use of asyncio or plain subprocess. If you'd like to continue following this issue and participate in the discussion, I'll see you over on http://bugs.python.org/issue1191964 . - Josiah On Fri, Mar 28, 2014 at 11:35 AM, Josiah Carlson wrote: > > On Fri, Mar 28, 2014 at 10:46 AM, Guido van Rossum wrote: > >> On Fri, Mar 28, 2014 at 9:45 AM, Josiah Carlson > > wrote: >> >>> >>> If it makes you feel any better, I spent an hour this morning building a >>> 2-function API for Linux and Windows, both tested, not using ctypes, and >>> not even using any part of asyncio (the Windows bits are in msvcrt and >>> _winapi). It works in Python 3.3+. You can see it here: >>> http://pastebin.com/0LpyQtU5 >>> >> >> Seeing this makes *me* feel better. I think it's reasonable to add (some >> variant) of that to the subprocess module in Python 3.5. It also belongs in >> the Activestate cookbook. And no, the asyncio module hasn't made it >> obsolete. >> > > Cool. > > Josiah, you sound upset about the whole thing -- to the point of writing >> unintelligible sentences and passive-aggressive digs at everyone reading >> this list. I'm sorry that something happened that led you feel that way (if >> you indeed feel upset or frustrated) but I'm glad that you wrote that code >> snippet -- it is completely clear what you want and why you want it, and >> also what should happen next (a few rounds of code review on the tracker). >> > > I'm not always a prat. Something about python-dev brings out parts of me > that I thought I had discarded from my personality years ago. Toss in a bit > of needing to re-explain ideas that I've been trying to explain for almost > 9 years? Frustration + formerly discarded personality traits = uck. That's > pretty much why I won't be rejoining the party here regularly, you are all > better off without me commenting on 95% of threads like I used to. > > Victor, I'm sorry for being a jerk. It's hard for me to not be the guy I > was when I spend time on this list. That's *my* issue, not yours. That I > spent any time redirecting my frustration towards you is BS, and if I could > take back the email I sent just before getting Guido's, I would. > > I would advise everyone to write it off as the ramblings of a surprisingly > young, angry old man. Or call me an a-hole. Both are pretty accurate. :) > > But that PEP? It's just a terrible PEP. It doesn't contain a single line >> of example code. It doesn't specify the proposed interface, it just >> describes in way too many sentences how it should work, and contains a >> whole lot of references to various rants from which the reader is >> apparently meant to become enlightened. I don't know which of the three >> authors *really* wrote it, and I don't want to know -- I think the PEP is >> irrelevant to the proposed feature, which is of "put it in the bug tracker >> and work from there" category -- presumably the PEP was written based on >> the misunderstanding that having a PEP would make acceptance of the patch >> easier, or because during an earlier bikeshedding round someone said >> "please write a PEP" (someone always says that). I propose to scrap the PEP >> (set the status to Withdrawn) and just work on adding the methods to the >> subprocess module. >> > > I'm not going to argue. The first I read it was 2-3 days ago. > > If it were me, I'd define three methods, with longer names to clarify >> what they do, e.g. >> >> proc.write_nonblocking(data) >> data = proc.read_nonblocking() >> data = proc.read_stderr_nonblocking() >> > > Easily doable. > > I.e. add _nonblocking to the method names to clarify that they may return >> '' when there's nothing available, and have a separate method for reading >> stderr instead of a flag. And I'd wonder if there should be an unambiguous >> way to detect EOF or whether the caller should just check for >> proc.stdout.closed. (And what for stdin? IIRC it actually becomes writable >> when the other end is closed, and then the write() will fail. But maybe I >> forget.) >> >> But that's all bikeshedding and it can happen on the tracker or directly >> on the list just as easily; I don't see the need for a PEP. >> > > Sounds good. > > - Josiah > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marko at pacujo.net Sun Mar 30 10:39:18 2014 From: marko at pacujo.net (Marko Rauhamaa) Date: Sun, 30 Mar 2014 11:39:18 +0300 Subject: [Python-Dev] collections.sortedtree In-Reply-To: (Guido van Rossum's message of "Wed, 26 Mar 2014 15:16:03 -0700") References: <8738i4og5f.fsf@elektro.pacujo.net> <20140326222628.24336818@fsol> <87ppl8mycp.fsf@elektro.pacujo.net> <20140326224848.1e457671@fsol> <87a9ccmxmg.fsf@elektro.pacujo.net> <20140326230247.084e422d@fsol> Message-ID: <87lhvsf5c9.fsf@elektro.pacujo.net> Guido van Rossum : > Yeah, so the pyftp fix is to keep track of how many timers were > cancelled, and if the number exceeds a threshold it just recreates the > heap, something like > > heap = [x for x in heap if not x.cancelled] > heapify(heap) I measured my target use case with a simple emulation on my linux PC. The simple test case emulates this scenario: Start N connections at frequency F and have each connection start a timer T. Then, rotate over the connections at the same frequency F restarting timer T. Stop after a duration that is much greater than T. Four different timer implementations were considered: HEAPQ: straight heapq HEAPQ*: heapq with the pyftp fix (reheapify whenever 80% of the outstanding timers have been canceled) SDICT: sorteddict (my C implementation) PyAVL: Python AVL tree (my implementation) Here are the results: N = 1000, F = 100 Hz, T = 10 min, duration 1 hr ============================================= Virt Res max len() urs sys CPU MB MB s s % ============================================= HEAPQ 22 16 60001 21.5 4.3 0.7 HEAPQ* 11 7 5000 18.4 4.2 0.6 SDICT 11 6 1000 18.2 3.9 0.6 PyAVL 11 6 1000 39.3 3.6 1.2 ============================================= N = 10000, F = 1000 Hz, T = 10 min, duration 1 hr ============================================= Virt Res max len() urs sys CPU MB MB s s % ============================================= HEAPQ 125 120 600044 223.0 25.8 6.9 HEAPQ* 21 16 50000 186.8 30.0 6.0 SDICT 15 10 10000 196.6 25.7 6.2 PyAVL 16 11 10000 412.5 22.3 12.1 ============================================= Conclusions: * The CPU load is almost identical in HEAPQ, HEAPQ* and SDICT. * HEAPQ* is better than HEAPQ because of the memory burden. * PyAVL is not all that bad compared with SDICT. Marko From martin at v.loewis.de Sun Mar 30 20:44:02 2014 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 30 Mar 2014 20:44:02 +0200 Subject: [Python-Dev] freeze build slave Message-ID: <533865F2.7050302@v.loewis.de> I have created a buildbot configuration to test freeze. At the moment, it has only one builder: http://buildbot.python.org/all/waterfall?show=AMD64%20Ubuntu%20LTS%20Freeze%203.x which currently fails as freeze doesn't actually work. The test itself works by first building Python in release mode, then installing it, then running freeze on a test program, then building the test programm (and ultimately running it). The question then is: how should that integrate with the rest of the builders? I can see three alternatives: A. (status quo) run the test on a selected subset of the Unix builders B. run the test on all Unix builders. C. integrate the test with the Unix regular Unix builder Evaluating these alternatives: B: pro: wider testing con: each such build takes the slave lock, so slaves will have to do one additional full build per commit (or two if the fix gets checked into 3.4 as well). In addition, each slave will need disk space for one additional build tree plus one Python installation, per branch. C: pro: compared to B, build time is reduced (need only to build once per branch); disk space is also reduced con: it would test a debug build, not a release build Regards, Martin From solipsis at pitrou.net Sun Mar 30 20:55:10 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 30 Mar 2014 20:55:10 +0200 Subject: [Python-Dev] freeze build slave References: <533865F2.7050302@v.loewis.de> Message-ID: <20140330205510.3cf14b9f@fsol> On Sun, 30 Mar 2014 20:44:02 +0200 "Martin v. L?wis" wrote: > I have created a buildbot configuration to test freeze. At the moment, > it has only one builder: > > http://buildbot.python.org/all/waterfall?show=AMD64%20Ubuntu%20LTS%20Freeze%203.x > > which currently fails as freeze doesn't actually work. > > The test itself works by first building Python in release mode, > then installing it, then running freeze on a test program, then > building the test programm (and ultimately running it). > > The question then is: how should that integrate with the rest > of the builders? I can see three alternatives: > A. (status quo) run the test on a selected subset of the Unix > builders > B. run the test on all Unix builders. > C. integrate the test with the Unix regular Unix builder > > Evaluating these alternatives: > B: pro: wider testing > con: each such build takes the slave lock, so slaves > will have to do one additional full build per commit > (or two if the fix gets checked into 3.4 as well). > In addition, each slave will need disk space for one > additional build tree plus one Python installation, > per branch. > C: pro: compared to B, build time is reduced (need only > to build once per branch); disk space is also reduced > con: it would test a debug build, not a release build We have at least one builder working in release (i.e. non-debug) mode. http://buildbot.python.org/all/builders/x86%20Gentoo%20Non-Debug%203.x Regards Antoine. From stefan at bytereef.org Sun Mar 30 22:37:08 2014 From: stefan at bytereef.org (Stefan Krah) Date: Sun, 30 Mar 2014 22:37:08 +0200 Subject: [Python-Dev] freeze build slave In-Reply-To: <533865F2.7050302@v.loewis.de> References: <533865F2.7050302@v.loewis.de> Message-ID: <20140330203708.GA9714@sleipnir.bytereef.org> "Martin v. L?wis" wrote: > C: pro: compared to B, build time is reduced (need only > to build once per branch); disk space is also reduced > con: it would test a debug build, not a release build It would be an option to run half of the Unix slaves (especially the ones with the more aggressive compilers) in release mode. I think that is beneficial anyway. Stefan Krah From victor.stinner at gmail.com Mon Mar 31 08:11:32 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 31 Mar 2014 08:11:32 +0200 Subject: [Python-Dev] freeze build slave In-Reply-To: <20140330203708.GA9714@sleipnir.bytereef.org> References: <533865F2.7050302@v.loewis.de> <20140330203708.GA9714@sleipnir.bytereef.org> Message-ID: I disagree. Running tests in debug code tests more things thanks to assertions, and provides more info in case of test failure or crash. Some assertions only fail on some platforms. See for example test_locale which fails with an assertion error on solaris (since Python 3.3). Adding one or two slaves should not hurt. You need maybe at least one on Windows and another on OS X. Victor Le dimanche 30 mars 2014, Stefan Krah a ?crit : > "Martin v. L?wis" > wrote: > > C: pro: compared to B, build time is reduced (need only > > to build once per branch); disk space is also reduced > > con: it would test a debug build, not a release build > > It would be an option to run half of the Unix slaves (especially the ones > with > the more aggressive compilers) in release mode. I think that is beneficial > anyway. > > > Stefan Krah > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aponomarenko at rosalab.ru Mon Mar 31 13:38:33 2014 From: aponomarenko at rosalab.ru (Andrey Ponomarenko) Date: Mon, 31 Mar 2014 15:38:33 +0400 Subject: [Python-Dev] libpython added to ABI tracker In-Reply-To: References: <5335337F.8000303@rosalab.ru> Message-ID: <533953B9.2040506@rosalab.ru> Hi, Victor Stinner wrote: > Hi, > > 2014-03-28 9:31 GMT+01:00 Andrey Ponomarenko : >> The libpython library has been added to the ABI tracker: >> http://upstream-tracker.org/versions/python.html >> >> The page lists library versions and changes in API/ABI. > Nice! > > By the way, would it be possible to add a second page for the stable > ABI of Python? > > Victor The public libpython API changes will be tracked here: http://upstream-tracker.org/versions/python_public_api.html For now I've excluded only symbols starting with an underscore. What other symbols should be excluded? Thanks. -- Andrey Ponomarenko, NTC IT ROSA. From victor.stinner at gmail.com Mon Mar 31 17:37:10 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 31 Mar 2014 17:37:10 +0200 Subject: [Python-Dev] libpython added to ABI tracker In-Reply-To: <533953B9.2040506@rosalab.ru> References: <5335337F.8000303@rosalab.ru> <533953B9.2040506@rosalab.ru> Message-ID: 2014-03-31 13:38 GMT+02:00 Andrey Ponomarenko : > The public libpython API changes will be tracked here: > http://upstream-tracker.org/versions/python_public_api.html > > For now I've excluded only symbols starting with an underscore. What other > symbols should be excluded? It's not a matter of underscore. You should define Py_LIMITED_API to 0x03020000 to test the stable ABI of Python 3.2. http://docs.python.org/dev/c-api/stable.html Victor From ericsnowcurrently at gmail.com Mon Mar 31 20:30:36 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 31 Mar 2014 12:30:36 -0600 Subject: [Python-Dev] Treating tokenize.Untokenizer as private In-Reply-To: References: Message-ID: On Tue, Feb 18, 2014 at 2:09 PM, Terry Reedy wrote: > I am working through the multiple bugs afflicting tokenize.untokenize, which > is described in the tokenize doc and has an even longer docstring. While the > function could be implemented as one 70-line function, it happens to be > implemented as a 4-line wrapper for a completely undocumented (Untokenizer > class with 4 methods. (It is unmentioned in the doc and there are currently > no docstrings.) > > I view the class as a private implementation detail and would like to treat > it as such, and perhaps even rename it _Untokenizer to make that clear. The > issue arises in #9974. It appears that a fix may require the addition of an > instance attribute or .add_whitespace parameter. If there is objection to > treating the whole class as private, I would at least like to treat > add_whitespace as the private helper that it is. There is no reason to call > it directly except for testing. Otherwise, it could just as well have been > left inline at the one call site. Is this still an open question, Terry? -eric From ericsnowcurrently at gmail.com Mon Mar 31 20:41:40 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 31 Mar 2014 12:41:40 -0600 Subject: [Python-Dev] Making proxy types easier to write and maintain In-Reply-To: <532B2BC1.6040603@hastings.org> References: <20140319194607.331336db@fsol> <20140319212242.5d25904c@fsol> <20140319230139.043aee2f@fsol> <20140320003248.GP16526@ando> <532A95F1.6060505@hastings.org> <532B2BC1.6040603@hastings.org> Message-ID: On Thu, Mar 20, 2014 at 11:56 AM, Larry Hastings wrote: > On 03/20/2014 12:49 AM, Nick Coghlan wrote: > > So long as Graham's willing to go along with it, he doesn't have to to > be the one to write the PEP. > > > PEP? Why does it need a PEP? I didn't think it'd even be a new top-level > library module, it'd be an addition / additions to functools. Would it make sense to have a new module named classtools rather than add more class-related stuff to functools. For instance, total_ordering() is the sort of thing I would expect to find in classtools rather than functools. -eric From ericsnowcurrently at gmail.com Mon Mar 31 20:53:17 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 31 Mar 2014 12:53:17 -0600 Subject: [Python-Dev] Making proxy types easier to write and maintain In-Reply-To: <20140319194607.331336db@fsol> References: <20140319194607.331336db@fsol> Message-ID: On Wed, Mar 19, 2014 at 12:46 PM, Antoine Pitrou wrote: > In http://bugs.python.org/issue19359#msg213530 I proposed to introduce a "proxy > protocol" (__proxy__ / tp_proxy) that would be used as a fallback by > _PyObject_LookupSpecial to fetch the lookup target, i.e.: > > def _PyObject_LookupSpecial(obj, name): > tp = type(obj) > try: > return getattr(tp, name) > except AttributeError: > return getattr(tp.tp_proxy(), name) > > What do you think? Regardless of the attr lookup in the interpreter, it would be nice to have a standardized explicit __proxied__ attribute on proxy objects (similar to __wrapped__). -eric From chris.barker at noaa.gov Mon Mar 31 21:09:28 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 31 Mar 2014 12:09:28 -0700 Subject: [Python-Dev] Negative timedelta strings In-Reply-To: References: <20140328110342.GI16526@ando> <5335E754.4010305@canterbury.ac.nz> Message-ID: On Fri, Mar 28, 2014 at 2:52 PM, Fred Drake wrote: > On Fri, Mar 28, 2014 at 5:19 PM, Greg Ewing > wrote: > > ISO 8601 doesn't seem to define a representation for > > negative durations, though, so it wouldn't solve the > > original problem. > > Aside from the horribleness of the ISO 8601 notation for a duration, it's > best not to confuse the notions of duration and delta. Notionally, a delta > contains more information than a duration. and less -- really it's different. A duration would be really useful actually, for things like "next month", etc,. IIRC, mxDateTime has something for this, but it's NOT the same as a timedelta. timedelta appears to be analogous to ISO 8601's " time interval", which requires 1. Start and end, such as "2007-03-01T13:00:00Z/2008-05-11T15:30:00Z" 2. Start and duration, such as "2007-03-01T13:00:00Z/P1Y2M10DT2H30M" 3. Duration and end, such as "P1Y2M10DT2H30M/2008-05-11T15:30:00Z" 4. Duration only, such as "P1Y2M10DT2H30M", with additional context information http://en.wikipedia.org/wiki/ISO_8601#Time_intervals I don't think there is or should be a any direct mapping from timedelta to ISO duration. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Mar 31 21:45:58 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 31 Mar 2014 15:45:58 -0400 Subject: [Python-Dev] Treating tokenize.Untokenizer as private In-Reply-To: References: Message-ID: On 3/31/2014 2:30 PM, Eric Snow wrote: > On Tue, Feb 18, 2014 at 2:09 PM, Terry Reedy wrote: >> I am working through the multiple bugs afflicting tokenize.untokenize, which >> is described in the tokenize doc and has an even longer docstring. While the >> function could be implemented as one 70-line function, it happens to be >> implemented as a 4-line wrapper for a completely undocumented (Untokenizer >> class with 4 methods. (It is unmentioned in the doc and there are currently >> no docstrings.) >> >> I view the class as a private implementation detail and would like to treat >> it as such, and perhaps even rename it _Untokenizer to make that clear. The >> issue arises in #9974. It appears that a fix may require the addition of an >> instance attribute or .add_whitespace parameter. If there is objection to >> treating the whole class as private, I would at least like to treat >> add_whitespace as the private helper that it is. There is no reason to call >> it directly except for testing. Otherwise, it could just as well have been >> left inline at the one call site. > > Is this still an open question, Terry? It is not currently for #9974 because after consideration of two proposed patches (one part of another issue), I decided that the conditions being guarded against either could not occur or could be checked in the caller, so that no api change was needed. However, I believe the fix for another bug, http://bugs.python.org/issue20387 will require copying the code that correctly formats indents in the old compat method to the newer method, which incorrectly assumes that indents are spaces only. I might end up wishing I could refactor the code. So this may well become a live issue again, and I would still like to know what people think. How do we do code searches (as for use of "Untokenize") these days? -- Terry Jan Reedy From raymond.hettinger at gmail.com Mon Mar 31 21:51:39 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 31 Mar 2014 12:51:39 -0700 Subject: [Python-Dev] Treating tokenize.Untokenizer as private In-Reply-To: References: Message-ID: <3B55C3AF-F294-40AC-A711-9ABAD5C7675A@gmail.com> On Feb 18, 2014, at 1:09 PM, Terry Reedy wrote: > While the function could be implemented as one 70-line function, it happens to be implemented as a 4-line wrapper for a completely undocumented (Untokenizer class with 4 methods. (It is unmentioned in the doc and there are currently no docstrings.) > > I view the class as a private implementation detail and would like to treat it as such, and perhaps even rename it _Untokenizer to make that clear. Yes, that would be reasonable. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-dev at masklinn.net Mon Mar 31 22:01:52 2014 From: python-dev at masklinn.net (Xavier Morel) Date: Mon, 31 Mar 2014 22:01:52 +0200 Subject: [Python-Dev] Negative timedelta strings In-Reply-To: References: <20140328110342.GI16526@ando> Message-ID: <6688AEE2-B77D-4F0E-BF86-3B8E9FD711CD@masklinn.net> On 2014-03-28, at 17:19 , Skip Montanaro wrote: > (*) As an aside (that is, this belongs in a separate thread if you > want to discuss it), in my opinion, attempting to support ISO 8601 > formatting is pointless without the presence of an anchor datetime. > Otherwise how would you know how far back "five months" or "seven > years" was? dateutil's relativedelta keeps the notion "abstract" until it's combined with an anchor datetime, at which point it's reified to a real duration[0]. > If that's the case, then you might as well add the timedelta to your anchor datetime and just use datetime.strftime(). You can't even express "next month" with timedelta, since the duration of a month is not a fixed number of seconds. [0] well not exactly, a relativedelta really defines a processing pipeline on its anchor date, which allows for fun stuff like "saturday the third week of next month". From stutzbach at google.com Mon Mar 31 23:01:40 2014 From: stutzbach at google.com (Daniel Stutzbach) Date: Mon, 31 Mar 2014 14:01:40 -0700 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <37C7BDAF-DE48-4303-B908-17A785534D32@gmail.com> <87fvm2ng4n.fsf@elektro.pacujo.net> Message-ID: On Fri, Mar 28, 2014 at 6:45 PM, Dan Stromberg wrote: > In my testing blist.sorteddict was dead last for random keys, and > wasn't last but was still significantly underperforming for sequential > keys (outperforming only binary tree and scapegoat tree, behind all > others): > > > http://stromberg.dnsalias.org/~strombrg/python-tree-and-heap-comparison/2014-03-18/ > Could you post the source code for your test tools, so that I can reproduce them locally and understand the results better? I think I'm confused about what you're trying to measure. It looks like the tests perform get and set operations, neither of which required a sorted dict. Wouldn't a good comparison of sorted dict types include at least one operation that relies on the sorted property? Possibly I've misunderstood how your tests work. -- Daniel Stutzbach -------------- next part -------------- An HTML attachment was scrubbed... URL: From drsalists at gmail.com Mon Mar 31 23:46:04 2014 From: drsalists at gmail.com (Dan Stromberg) Date: Mon, 31 Mar 2014 14:46:04 -0700 Subject: [Python-Dev] collections.sortedtree In-Reply-To: References: <8738i4og5f.fsf@elektro.pacujo.net> <37C7BDAF-DE48-4303-B908-17A785534D32@gmail.com> <87fvm2ng4n.fsf@elektro.pacujo.net> Message-ID: vne On Mon, Mar 31, 2014 at 2:01 PM, Daniel Stutzbach wrote: > On Fri, Mar 28, 2014 at 6:45 PM, Dan Stromberg wrote: >> >> In my testing blist.sorteddict was dead last for random keys, and >> wasn't last but was still significantly underperforming for sequential >> keys (outperforming only binary tree and scapegoat tree, behind all >> others): >> >> >> http://stromberg.dnsalias.org/~strombrg/python-tree-and-heap-comparison/2014-03-18/ > > > Could you post the source code for your test tools, so that I can reproduce > them locally and understand the results better? > > I think I'm confused about what you're trying to measure. It looks like the > tests perform get and set operations, neither of which required a sorted > dict. Wouldn't a good comparison of sorted dict types include at least one > operation that relies on the sorted property? Possibly I've misunderstood > how your tests work. The code is at http://stromberg.dnsalias.org/svn/python-tree-and-heap-comparison/trunk/ You're right, I'm not comparing find_min, find_max, generate_in_order or generate_reverse_order. The first two tend to be the same as find_arbitrary. I've been thinking about trying to import some C extension module trees, and just ignoring them if they fail to import. But I haven't. Trunk only compares pure python implementations - even for treap, which has a cython version. I tossed my changes after checking blist. Trunk does operations/second.