From stephen at xemacs.org Tue Oct 1 03:58:41 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 01 Oct 2013 10:58:41 +0900 Subject: [Python-Dev] Python for new users In-Reply-To: References: Message-ID: <871u45lpke.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Efford writes: > Widely-used and linked web resources tend to persist for a very > long time, so we shouldn't use the prevalence of Python 2 resources > as a reason for excessive caution. The key question is how much > good material is available based on Python 3 - and this has improved > significantly over the past couple of years. No, it really isn't, at least it's not more key than some other questions. Educators like you, and like me, who are in a position to teach pretty much what we want are somewhat limited by that (but it's really not that hard to take your favorite Python 2 tutorial and port it to Python 3) will certainly consider that the key question -- we can't spend all our time on one class. But there's another kind of educator, the consultant who goes into a company (or perhaps comes from their training department) who will need to teach the gory details that differ because they matter to the applications the trainees will be maintaining and extending. And of course self-taught programmers are likely to use something time-tested and recommended by their peers. I agree that the prevalence of Python 2 materials *per se* is not a reason for *excessive* caution, but I suspect if you look closely you'll discover that new ones are being produced and old ones revised and enhanced. I'll personally be interested to see what happens in the next revision of *Python Essential Reference* for example. (And so will Nori: http://turnbull.sk.tsukuba.ac.jp/media/Nori-swears-by-Python-Essential-Reference.jpg) > Encouraging the continued use of 2.7 for existing programmers is > entirely justifiable, I would disagree. Programmers should not be discouraged from using any version of Python that is needed for compatibility with existing practice, but where possible with reasonable effort, the most recent version of Python 3 should be encouraged. From barry at python.org Tue Oct 1 04:12:03 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 30 Sep 2013 22:12:03 -0400 Subject: [Python-Dev] Released: Python 2.6.9 release candidate 1 Message-ID: <20130930221203.6eed7479@anarchist> Hello Pythoneers and Pythonistas, I'm happy to announce the availability of Python 2.6.9 release candidate 1. Python 2.6.9rc1 is a security-fix source-only release for Python 2.6. This means that general bug maintenance has ended, and only critical security issues are being fixed. It also means that no installers for Windows or Mac OS X will be provided. The last binary release of Python 2.6 was 2.6.6. Python 2.6.9 final is currently scheduled for Monday, October 28, 2013. Five years after the original release of Python 2.6, the 2.6.9 final release will be the last release of the Python 2.6 series. After this, all official maintenance of any kind for Python 2.6 will cease and the series will be retired. For ongoing maintenance, please see Python 2.7. Since 2.6.9 will be the last Python 2.6 release ever, I ask that you please download Python 2.6.9rc1, build it on your favorite platforms, and test it with your favorite code. You can report bugs to the Python bug tracker: http://bugs.python.org The source can be download from: http://www.python.org/download/releases/2.6.9/ You can also see what's changed since Python 2.6.8: http://www.python.org/download/releases/2.6.9/NEWS.txt Many thanks go out to the entire Python community for their contributions and help in making Python 2.6.9 available, especially Jyrki Pulliainen for his patch contributions. Enjoy, -Barry (on behalf of the Python development community) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From yanxiaopei199 at gmail.com Tue Oct 1 04:25:58 2013 From: yanxiaopei199 at gmail.com (Thanatos xiao) Date: Tue, 1 Oct 2013 10:25:58 +0800 Subject: [Python-Dev] Released: Python 2.6.9 release candidate 1 In-Reply-To: <20130930221203.6eed7479@anarchist> References: <20130930221203.6eed7479@anarchist> Message-ID: Ooooooooooooooooooh?ths! 2013/10/1 Barry Warsaw > Hello Pythoneers and Pythonistas, > > I'm happy to announce the availability of Python 2.6.9 release candidate 1. > > Python 2.6.9rc1 is a security-fix source-only release for Python 2.6. This > means that general bug maintenance has ended, and only critical security > issues are being fixed. It also means that no installers for Windows or > Mac > OS X will be provided. The last binary release of Python 2.6 was 2.6.6. > > Python 2.6.9 final is currently scheduled for Monday, October 28, 2013. > Five > years after the original release of Python 2.6, the 2.6.9 final release > will > be the last release of the Python 2.6 series. After this, all official > maintenance of any kind for Python 2.6 will cease and the series will be > retired. > > For ongoing maintenance, please see Python 2.7. > > Since 2.6.9 will be the last Python 2.6 release ever, I ask that you please > download Python 2.6.9rc1, build it on your favorite platforms, and test it > with your favorite code. You can report bugs to the Python bug tracker: > > http://bugs.python.org > > The source can be download from: > > http://www.python.org/download/releases/2.6.9/ > > You can also see what's changed since Python 2.6.8: > > http://www.python.org/download/releases/2.6.9/NEWS.txt > > Many thanks go out to the entire Python community for their contributions > and > help in making Python 2.6.9 available, especially Jyrki Pulliainen for his > patch contributions. > > Enjoy, > -Barry > (on behalf of the Python development community) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/yanxiaopei199%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Oct 1 19:05:30 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 1 Oct 2013 19:05:30 +0200 Subject: [Python-Dev] cpython: Close #10042: functools.total_ordering now handles NotImplemented References: <3cq2H62npvz7LmW@mail.python.org> Message-ID: <20131001190530.1899b588@fsol> On Tue, 1 Oct 2013 16:02:26 +0200 (CEST) nick.coghlan wrote: > http://hg.python.org/cpython/rev/ad9f207645ab > changeset: 85913:ad9f207645ab > user: Nick Coghlan > date: Wed Oct 02 00:02:03 2013 +1000 > summary: > Close #10042: functools.total_ordering now handles NotImplemented > > (Patch by Katie Miller) Ahah, one less annoyance in the stdlib :-) Thank you! Antoine. From regebro at gmail.com Wed Oct 2 14:17:06 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 2 Oct 2013 14:17:06 +0200 Subject: [Python-Dev] PEP-431 non-status Message-ID: Hi all! If you wonder about the lack of progress reports on pep-431, this is because of a lack of progress. I simply haven't had any time to work on this. I considered make a kickstarter so I could take time off from working, but to be honest, even with that, I still have no time to realistically get this in before beta 1. Sorry about that! //Lennart From ncoghlan at gmail.com Wed Oct 2 15:07:57 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 2 Oct 2013 23:07:57 +1000 Subject: [Python-Dev] PEP-431 non-status In-Reply-To: References: Message-ID: That's a shame, but there's always 3.5 and we'll hopefully have an easier path to "pip install pytz" in 3.4 (once I get PEP 453 revised appropriately). Cheers, Nick. On 2 October 2013 22:17, Lennart Regebro wrote: > Hi all! > > If you wonder about the lack of progress reports on pep-431, this is > because of a lack of progress. I simply haven't had any time to work > on this. I considered make a kickstarter so I could take time off from > working, but to be honest, even with that, I still have no time to > realistically get this in before beta 1. > > Sorry about that! > > //Lennart > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dirkjan at ochtman.nl Wed Oct 2 15:05:15 2013 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 2 Oct 2013 15:05:15 +0200 Subject: [Python-Dev] PEP-431 non-status In-Reply-To: References: Message-ID: On Wed, Oct 2, 2013 at 2:17 PM, Lennart Regebro wrote: > If you wonder about the lack of progress reports on pep-431, this is > because of a lack of progress. I simply haven't had any time to work > on this. I considered make a kickstarter so I could take time off from > working, but to be honest, even with that, I still have no time to > realistically get this in before beta 1. Is there a list of stuff that needs to be done? I might be able to help out. Cheers, Dirkjan From regebro at gmail.com Wed Oct 2 16:42:04 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 2 Oct 2013 16:42:04 +0200 Subject: [Python-Dev] PEP-431 non-status In-Reply-To: References: Message-ID: On Wed, Oct 2, 2013 at 3:05 PM, Dirkjan Ochtman wrote: > On Wed, Oct 2, 2013 at 2:17 PM, Lennart Regebro wrote: >> If you wonder about the lack of progress reports on pep-431, this is >> because of a lack of progress. I simply haven't had any time to work >> on this. I considered make a kickstarter so I could take time off from >> working, but to be honest, even with that, I still have no time to >> realistically get this in before beta 1. > > Is there a list of stuff that needs to be done? I might be able to help out. Yes. 1. Implement the PEP. :-) I looked at it during the PyCon sprint, and I had hoped to be able to mostly move pytz into datetime, but unfortunately it wasn't that easy, because of the way datetime normalization works. The pytz normalization calls methods that would need to call the normalization... :-) //Lennart From martin at v.loewis.de Wed Oct 2 17:04:02 2013 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 02 Oct 2013 17:04:02 +0200 Subject: [Python-Dev] cpython (2.7): Add fake buildbottouch target. In-Reply-To: <20130930201149.4a034831@fsol> References: <3cpQhd3DJ5zPWd@mail.python.org> <20130930201149.4a034831@fsol> Message-ID: <524C35E2.7050407@v.loewis.de> Am 30.09.13 20:11, schrieb Antoine Pitrou: > On Mon, 30 Sep 2013 16:18:57 +0200 (CEST) > martin.v.loewis wrote: >> http://hg.python.org/cpython/rev/c1a294bbb4fa >> changeset: 85882:c1a294bbb4fa >> branch: 2.7 >> parent: 85877:dd55d54b2a15 >> user: Martin v. L?wis >> date: Mon Sep 30 16:18:31 2013 +0200 >> summary: >> Add fake buildbottouch target. > > "make touch" does exist in 2.7, so was this new target needed? Thanks for pointing this out; I must have missed that. I have now dropped the buildbottouch target again. Regards, Martin From regebro at gmail.com Wed Oct 2 18:32:35 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 2 Oct 2013 18:32:35 +0200 Subject: [Python-Dev] PEP-431 non-status In-Reply-To: References: Message-ID: That could be a possibility as well. On Wed, Oct 2, 2013 at 6:14 PM, Stuart Bishop wrote: > On Wed, Oct 2, 2013 at 9:42 PM, Lennart Regebro wrote: >> On Wed, Oct 2, 2013 at 3:05 PM, Dirkjan Ochtman wrote: >>> On Wed, Oct 2, 2013 at 2:17 PM, Lennart Regebro wrote: >>>> If you wonder about the lack of progress reports on pep-431, this is >>>> because of a lack of progress. I simply haven't had any time to work >>>> on this. I considered make a kickstarter so I could take time off from >>>> working, but to be honest, even with that, I still have no time to >>>> realistically get this in before beta 1. >>> >>> Is there a list of stuff that needs to be done? I might be able to help out. >> >> Yes. >> >> 1. Implement the PEP. >> >> :-) >> >> I looked at it during the PyCon sprint, and I had hoped to be able to >> mostly move pytz into datetime, but unfortunately it wasn't that easy, >> because of the way datetime normalization works. The pytz >> normalization calls methods that would need to call the >> normalization... :-) > > I think the first thing that needs to happen is get the is_dst flag > in. Then we can turn pytz into a new, sane API without all that awful > normalization stuff :) It doesn't all have to land in the same > release. From solipsis at pitrou.net Wed Oct 2 19:31:24 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 2 Oct 2013 19:31:24 +0200 Subject: [Python-Dev] cpython: Use cached builtins. References: <3cqjCh0tnQz7LjZ@mail.python.org> Message-ID: <20131002193124.6d516609@fsol> On Wed, 2 Oct 2013 18:16:48 +0200 (CEST) serhiy.storchaka wrote: > http://hg.python.org/cpython/rev/d48ac94e365f > changeset: 85931:d48ac94e365f > user: Serhiy Storchaka > date: Wed Oct 02 19:15:54 2013 +0300 > summary: > Use cached builtins. What's the point? I don't think it's a good idea to uglify the code if there isn't a clear benefit. Regards Antoine. From stefan_ml at behnel.de Wed Oct 2 20:58:39 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 02 Oct 2013 20:58:39 +0200 Subject: [Python-Dev] project culture: take responsibility for your commits Message-ID: Hi, I'm looking back on a rather unpleasant experience that I recently had in this developer community. Actually, twice by now. Here's what I take from it: You should take responsibility for your commits. Sounds like a simple thing - in theory. People make mistakes, that's normal. You can't always know what others do with your code, and so something that might sound like a great idea when you come up with it, may turn out to have a negative impact that you didn't, and sometimes couldn't possibly, anticipate. So, people will come to you and complain. Let's assume these complaints are reasonable (and they certainly aren't always). What do you do in that case? Do you tell them that what's in is in? Do you tell them that you are a core developer and they are not? That they can try to do better, and if they are lucky, find someone else who applies their patch? That's what happened to me recently. Here's what I'd like to see people do instead. 1) You should discuss the issue with them. If their complaints are reasonable, you may simply have been wrong. Or you may have been right, but couldn't know that there was more beyond the visible. Be open. If you're lucky, the discussion will make the problem obvious to someone else, and a patch will come for free. 2) You should invest a reasonable effort to fix the issue yourself. After all, you produced it by changing something, even if you couldn't possibly see it coming. Take responsibility for your commits. 3) If you can't fix your change, consider reverting it. No change is often better than a bad one. Maybe the time wasn't right for it. Maybe there's a better way to do it that isn't obvious yet. Taking a change back is not showing the white feather. It's showing responsibility. I know that these things aren't always easy. Sometimes people come yelling. Sometimes their complaints are plain wrong. Sometimes the huge impact they see is actually minuscule compared to the upsides. And sometimes the real size of the impact may not actually be visible to you. Be open and they will be, too. Stefan From ned at nedbatchelder.com Wed Oct 2 20:51:59 2013 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 02 Oct 2013 14:51:59 -0400 Subject: [Python-Dev] cpython: Use cached builtins. In-Reply-To: <20131002193124.6d516609@fsol> References: <3cqjCh0tnQz7LjZ@mail.python.org> <20131002193124.6d516609@fsol> Message-ID: <524C6B4F.5040108@nedbatchelder.com> On 10/2/13 1:31 PM, Antoine Pitrou wrote: > On Wed, 2 Oct 2013 18:16:48 +0200 (CEST) > serhiy.storchaka wrote: >> http://hg.python.org/cpython/rev/d48ac94e365f >> changeset: 85931:d48ac94e365f >> user: Serhiy Storchaka >> date: Wed Oct 02 19:15:54 2013 +0300 >> summary: >> Use cached builtins. > What's the point? I don't think it's a good idea to uglify the code if > there isn't a clear benefit. This type of micro-optimization seems especially unnecessary in a module intended for printing. --Ned. > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com From storchaka at gmail.com Wed Oct 2 21:11:20 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 02 Oct 2013 22:11:20 +0300 Subject: [Python-Dev] cpython: Use cached builtins. In-Reply-To: <20131002193124.6d516609@fsol> References: <3cqjCh0tnQz7LjZ@mail.python.org> <20131002193124.6d516609@fsol> Message-ID: 02.10.13 20:31, Antoine Pitrou ???????(??): > On Wed, 2 Oct 2013 18:16:48 +0200 (CEST) > serhiy.storchaka wrote: >> http://hg.python.org/cpython/rev/d48ac94e365f >> changeset: 85931:d48ac94e365f >> user: Serhiy Storchaka >> date: Wed Oct 02 19:15:54 2013 +0300 >> summary: >> Use cached builtins. > > What's the point? I don't think it's a good idea to uglify the code if > there isn't a clear benefit. These builtins already were cached. I only change recently added code to use cached values. The point was perhaps that the access to module globals is a little faster than to builtins. Fred Drake was introduced this in changeset 261a70c0aa79. You can open a separate issue for removing this caching or for adding new builtins to this set. From victor.stinner at gmail.com Wed Oct 2 21:58:20 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 2 Oct 2013 21:58:20 +0200 Subject: [Python-Dev] cpython: Use cached builtins. In-Reply-To: <20131002193124.6d516609@fsol> References: <3cqjCh0tnQz7LjZ@mail.python.org> <20131002193124.6d516609@fsol> Message-ID: I don't remember where, but I remember that I also saw things like "str=str, len=len, ...". So you keep the same name, but you use fast local lookups instead of slow builtin lookups. Victor 2013/10/2 Antoine Pitrou : > On Wed, 2 Oct 2013 18:16:48 +0200 (CEST) > serhiy.storchaka wrote: >> http://hg.python.org/cpython/rev/d48ac94e365f >> changeset: 85931:d48ac94e365f >> user: Serhiy Storchaka >> date: Wed Oct 02 19:15:54 2013 +0300 >> summary: >> Use cached builtins. > > What's the point? I don't think it's a good idea to uglify the code if > there isn't a clear benefit. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From brett at python.org Wed Oct 2 22:01:06 2013 From: brett at python.org (Brett Cannon) Date: Wed, 2 Oct 2013 16:01:06 -0400 Subject: [Python-Dev] project culture: take responsibility for your commits In-Reply-To: References: Message-ID: On Wed, Oct 2, 2013 at 2:58 PM, Stefan Behnel wrote: > Hi, > > I'm looking back on a rather unpleasant experience that I recently had in > this developer community. Actually, twice by now. Here's what I take from > it: > > You should take responsibility for your commits. > > Sounds like a simple thing - in theory. People make mistakes, that's > normal. You can't always know what others do with your code, and so > something that might sound like a great idea when you come up with it, may > turn out to have a negative impact that you didn't, and sometimes couldn't > possibly, anticipate. > > So, people will come to you and complain. Let's assume these complaints are > reasonable (and they certainly aren't always). What do you do in that case? > > Do you tell them that what's in is in? Do you tell them that you are a core > developer and they are not? That they can try to do better, and if they are > lucky, find someone else who applies their patch? That's what happened to > me recently. > I think a key point in this is what "responsibility" is. There is a constant struggle between open source contributions and volunteering, of taking your time to contribute something for free to the benefit of the world and trying to meet the requests/demands of that world. How far should someone bend over backwards to try and accommodate others in the world when what the core dev has done is viewed by them as reasonable and don't think the requested change is worth their time (assuming it's even appropriate). So are core devs "responsible" for making everyone happy? Where is the line of what is a reasonable request to consider it their responsibility to meet that request? It's all very subjective. When someone becomes a core developer it is because they code well and seem to have the community's interest at heart. This also means they are trusted to make a reasonable call as to what is a responsible response when a request comes in. In this instance, the core dev disagreed with the request but left the door open for another core dev to come in and champion the change; IOW a -0 on the proposed change (full disclaimer to the list: I know what triggered this email). If you disagreed with the decision strongly you can try to convince another core dev or all of python-dev to see if you can get another core dev to step in on your side, but is completely within a core dev's right to say "no" to a request. I think the level of responsibility also varies from project to project as hosted on hg.python.org. For instance, I do not hold devinabox to the same backwards compatibility requirement as CPython. If I change the command-line API of something like build_python.py and someone came to me and said "I disagree with that change" I feel like I have the right to say I disagree as devinabox has no released versions to be compatible against, etc. I really does depend on what your view of "responsibility" is and for what project. It's all very subjective and if you disagree that's fine and you can say so and ask for another opinion. Heck, you can even say you are worried a developer is not taking their responsibility seriously, but that's obviously a major step to not be taken lightly. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuart at stuartbishop.net Wed Oct 2 18:14:26 2013 From: stuart at stuartbishop.net (Stuart Bishop) Date: Wed, 2 Oct 2013 23:14:26 +0700 Subject: [Python-Dev] PEP-431 non-status In-Reply-To: References: Message-ID: On Wed, Oct 2, 2013 at 9:42 PM, Lennart Regebro wrote: > On Wed, Oct 2, 2013 at 3:05 PM, Dirkjan Ochtman wrote: >> On Wed, Oct 2, 2013 at 2:17 PM, Lennart Regebro wrote: >>> If you wonder about the lack of progress reports on pep-431, this is >>> because of a lack of progress. I simply haven't had any time to work >>> on this. I considered make a kickstarter so I could take time off from >>> working, but to be honest, even with that, I still have no time to >>> realistically get this in before beta 1. >> >> Is there a list of stuff that needs to be done? I might be able to help out. > > Yes. > > 1. Implement the PEP. > > :-) > > I looked at it during the PyCon sprint, and I had hoped to be able to > mostly move pytz into datetime, but unfortunately it wasn't that easy, > because of the way datetime normalization works. The pytz > normalization calls methods that would need to call the > normalization... :-) I think the first thing that needs to happen is get the is_dst flag in. Then we can turn pytz into a new, sane API without all that awful normalization stuff :) It doesn't all have to land in the same release. From ncoghlan at gmail.com Thu Oct 3 01:00:55 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 3 Oct 2013 09:00:55 +1000 Subject: [Python-Dev] project culture: take responsibility for your commits In-Reply-To: References: Message-ID: Stefan, You blew up a minor design disagreement over the new async parsing API for XML into a huge impending disaster that would destroy the XML library APIs forever. In truth, even if we had left the original commit alone it would, at worst, have resulted in a slightly inconsistent API design. I get that you're passionate about the relevant API since you need to replicate it in lxml. I even agree that the API we ended up with after I got involved as a mediator to separate out the legitimate complaints from the hyperbole is better than the one that was originally committed. But when your opening gambit is to accuse people of complete incompetence (and you repeat similar accusations multiple times over the course of the disagreement), developers are entirely within their rights to stop listening to the abuse. It's important to remember that this is a project staffed by volunteers, and some basic civility and appreciation for those efforts goes a long way in obtaining a more constructive response. In the absence of that, don't be surprised if the reaction is "There may be a valid complaint somewhere in there, but I'm not wading through the vitriol to look for it". Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Oct 3 01:21:23 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 3 Oct 2013 09:21:23 +1000 Subject: [Python-Dev] project culture: take responsibility for your commits In-Reply-To: References: Message-ID: On 3 Oct 2013 09:00, "Nick Coghlan" wrote: > > Stefan, > > You blew up a minor design disagreement over the new async parsing API for XML into a huge impending disaster that would destroy the XML library APIs forever. In truth, even if we had left the original commit alone it would, at worst, have resulted in a slightly inconsistent API design. > > I get that you're passionate about the relevant API since you need to replicate it in lxml. I even agree that the API we ended up with after I got involved as a mediator to separate out the legitimate complaints from the hyperbole is better than the one that was originally committed. > > But when your opening gambit is to accuse people of complete incompetence (and you repeat similar accusations multiple times over the course of the disagreement), developers are entirely within their rights to stop listening to the abuse. > > It's important to remember that this is a project staffed by volunteers, and some basic civility and appreciation for those efforts goes a long way in obtaining a more constructive response. In the absence of that, don't be surprised if the reaction is "There may be a valid complaint somewhere in there, but I'm not wading through the vitriol to look for it". I'll also note that the affected developers *didn't* completely ignore the concerns you raised (which would have been an entirely reasonable response), but instead asked me to help out as a neutral mediator. That's an entirely responsible thing for them to do, and one that resulted in the underlying technical concerns being resolved *despite* the unwarranted abuse. Regards, Nick. > > Regards, > Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Thu Oct 3 04:23:35 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 03 Oct 2013 11:23:35 +0900 Subject: [Python-Dev] project culture: take responsibility for your commits In-Reply-To: References: Message-ID: <87r4c3m6s8.fsf@uwakimon.sk.tsukuba.ac.jp> Stefan Behnel writes: > Hi, I'm looking back on a rather unpleasant experience that I > recently had in this developer community. Actually, twice by > now. Here's what I take from it: You should take responsibility for > your commits. I have no clue who you're addressing this advice to. If it's not yourself (from the following, I gather it's not), I think the implication of what you are saying is mistaken. Core devs (by which I mean the high-profile developers who are candidates for PEP delegate) regularly do take responsibility for their commits, just like any other committer, by changing or reverting them. That's visible on this list as well as in the commit logs. > Let's assume these complaints [about the code] are reasonable That's not sufficient. They must also be presented reasonably, by the standards of the community. Not everybody is good at doing that, and those who aren't suffer, as does the project for losing a useful contribution. Unfortunate, but digging out what matters from unclear or high-handed presentations requires an enormous amount of effort, like pyschotherapy. Good psychotherapists bill hundreds of dollars an hour. The very best "pythotherapists" bill nothing, at least not to this community. Regarding the specific core dev behavior that offended you, I can speak from my experience in another project. > What do you do in that case? Do you tell them that what's in is in? I've done that and later reversed my position. In retrospect, I believe I was correct at the time of first approach in the majority of cases, though, on the grounds of the "lesser of two evils" as I understood the issues (or occasionally that the contributor had completely misunderstood the issues). In most cases the original requester never did come up with a coherent argument, just that something unclear to me didn't work for them. Reversal in such cases was due to a third party who was able to explain the requester's requirements, and often contribute (most of) a specification of a complete fix or a good compromise. > Do you tell them that you are a core developer and they are not? I've done that. I don't know if it applies to the cases you have in mind, but invariably that was a last retort when I just wanted to shut down a conversation that had already come back to the same place twice, and polite guidance seemed to be a complete failure. Childish, I guess, but it's been effective. That's not sufficient reason to use it in Python, which has higher standards for courtesy than my other project does. Caveat: as with the next item, I have to wonder if you mistook an explanation that in such disputes, the Python default is to go with the core dev's gut feeling unless there's good reason to do otherwise, and you haven't explained well enough yet for a snotty "I am and you're not, so go away!" > That they can try to do better, and if they are lucky, find someone > else who applies their patch? Definitely, and I would advise any core developer to use exactly that response as soon as they feel the discussion is becoming unprofitable. Of course they should spin it differently, something like Well, I don't agree that reverting my patch is necessary, and something needs to be done to address the issue it handles. You are welcome to propose your own patch, but I suggest you seek out an alternative reviewer as it's hard for me to be objective about my own code once I'm convinced it's correct. Are you *sure* that your phrasing above is what the reviewers wrote, and not merely what you took from their writing? "Insult is far more frequently taken than given." In sum, I don't see what you're complaining about. Sure, you may have been mistreated in the case(s) in question, which sucks, of course. If so, you should provide details, and maybe the particular kind of case would be a lesson to all of us. Or perhaps some core devs systematically abuse their positions (but as Brett says you should be very careful about making that statement in public!) But I don't see evidence that there's a systematic failure to review "core dev" commits, or that "core devs" deny their own fallibility. Nor do I think any of the behaviors you describe *out of context* are always wrong (except "I'm a core dev and you're not" should be avoided even if effective; it's rude and ad hominem). From g.brandl at gmx.net Thu Oct 3 09:56:36 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 03 Oct 2013 09:56:36 +0200 Subject: [Python-Dev] cpython: Use cached builtins. In-Reply-To: References: <3cqjCh0tnQz7LjZ@mail.python.org> <20131002193124.6d516609@fsol> Message-ID: Am 02.10.2013 21:58, schrieb Victor Stinner: > I don't remember where, but I remember that I also saw things like > "str=str, len=len, ...". So you keep the same name, but you use fast > local lookups instead of slow builtin lookups. In this case they aren't even fast local lookups but (slightly) faster module global lookups. Not worth the effort IMO. Georg From ncoghlan at gmail.com Thu Oct 3 13:05:39 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 3 Oct 2013 21:05:39 +1000 Subject: [Python-Dev] cpython: Use cached builtins. In-Reply-To: References: <3cqjCh0tnQz7LjZ@mail.python.org> <20131002193124.6d516609@fsol> Message-ID: On 3 Oct 2013 06:00, "Victor Stinner" wrote: > > I don't remember where, but I remember that I also saw things like > "str=str, len=len, ...". So you keep the same name, but you use fast > local lookups instead of slow builtin lookups. functools uses the local binding trick in lru_cache as a speed hack (pretty sure it uses an underscore prefix, though). However lru_cache *is* likely to end up being speed critical *and* it's binding local variables , so it's actually shifting a lot more work to compile time than merely trading a builtin lookup for a global lookup does. For most code though, introducing that kind of complexity isn't worth the cost in readability. Cheers, Nick. > > Victor > > 2013/10/2 Antoine Pitrou : > > On Wed, 2 Oct 2013 18:16:48 +0200 (CEST) > > serhiy.storchaka wrote: > >> http://hg.python.org/cpython/rev/d48ac94e365f > >> changeset: 85931:d48ac94e365f > >> user: Serhiy Storchaka > >> date: Wed Oct 02 19:15:54 2013 +0300 > >> summary: > >> Use cached builtins. > > > > What's the point? I don't think it's a good idea to uglify the code if > > there isn't a clear benefit. > > > > Regards > > > > Antoine. > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Thu Oct 3 14:53:02 2013 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 3 Oct 2013 13:53:02 +0100 Subject: [Python-Dev] cpython: Use cached builtins. In-Reply-To: References: <3cqjCh0tnQz7LjZ@mail.python.org> <20131002193124.6d516609@fsol> Message-ID: On 3 Oct 2013, at 12:05, Nick Coghlan wrote: > > On 3 Oct 2013 06:00, "Victor Stinner" wrote: > > > > I don't remember where, but I remember that I also saw things like > > "str=str, len=len, ...". So you keep the same name, but you use fast > > local lookups instead of slow builtin lookups. > > functools uses the local binding trick in lru_cache as a speed hack (pretty sure it uses an underscore prefix, though). > Inside a function you *have* to use an underscore prefix (well, some alternate name anyway) - otherwise the assignment makes the name local and the lookup would fail with an unbound local error. In order to do the binding at definition time rather than call time (so only once) the assignment is often done in the function signature. This is *particularly* ugly as it not only messes up the using of the builtins but screws your function signature too. So It should really only be done where benchmarking proves it makes a difference. I've never managed to find such a case... Michael > However lru_cache *is* likely to end up being speed critical *and* it's binding local variables , so it's actually shifting a lot more work to compile time than merely trading a builtin lookup for a global lookup does. > > For most code though, introducing that kind of complexity isn't worth the cost in readability. > > Cheers, > Nick. > > > > > Victor > > > > 2013/10/2 Antoine Pitrou : > > > On Wed, 2 Oct 2013 18:16:48 +0200 (CEST) > > > serhiy.storchaka wrote: > > >> http://hg.python.org/cpython/rev/d48ac94e365f > > >> changeset: 85931:d48ac94e365f > > >> user: Serhiy Storchaka > > >> date: Wed Oct 02 19:15:54 2013 +0300 > > >> summary: > > >> Use cached builtins. > > > > > > What's the point? I don't think it's a good idea to uglify the code if > > > there isn't a clear benefit. > > > > > > Regards > > > > > > Antoine. > > > > > > > > > _______________________________________________ > > > Python-Dev mailing list > > > Python-Dev at python.org > > > https://mail.python.org/mailman/listinfo/python-dev > > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From vasilyev_igor at inbox.ru Thu Oct 3 15:09:31 2013 From: vasilyev_igor at inbox.ru (=?UTF-8?B?0JjQs9C+0YDRjCDQktCw0YHQuNC70YzQtdCy?=) Date: Thu, 03 Oct 2013 17:09:31 +0400 Subject: [Python-Dev] =?utf-8?q?summing_integer_and_class?= Message-ID: <1380805771.625450506@f364.i.mail.ru> Hi. Example test.py: class A(): ??? def __add__(self, var): ??????? print("I'm in A class") ??????? return 5 a = A() a+1 1+a Execution: python test.py I'm in A class Traceback (most recent call last): ? File "../../test.py", line 7, in ??? 1+a TypeError: unsupported operand type(s) for +: 'int' and 'instance' So adding integer to class works fine, but adding class to integer fails. I could not understand why it happens. In objects/abstact.c we have the following function: static PyObject * binary_op1(PyObject *v, PyObject *w, const int op_slot) { ??? PyObject *x; ??? binaryfunc slotv = NULL; ??? binaryfunc slotw = NULL; ??? if (v->ob_type->tp_as_number != NULL) ??????? slotv = NB_BINOP(v->ob_type->tp_as_number, op_slot); ??? if (w->ob_type != v->ob_type && ??????? w->ob_type->tp_as_number != NULL) { ??????? slotw = NB_BINOP(w->ob_type->tp_as_number, op_slot); ??????? if (slotw == slotv) ??????????? slotw = NULL; ??? } ??? if (slotv) { ??????? if (slotw && PyType_IsSubtype(w->ob_type, v->ob_type)) { ??????????? x = slotw(v, w); ??????????? if (x != Py_NotImplemented) ??????????????? return x; ??????????? Py_DECREF(x); /* can't do it */ ??????????? slotw = NULL; ??????? } ??????? x = slotv(v, w); ??????? if (x != Py_NotImplemented) ??????????? return x; ??????? Py_DECREF(x); /* can't do it */ ??? } ??? if (slotw) { ??????? x = slotw(v, w); ??????? if (x != Py_NotImplemented) ??????????? return x; ??????? Py_DECREF(x); /* can't do it */ ??? } ??? Py_RETURN_NOTIMPLEMENTED; } When we adding class to integer we have both slotv and slotw. x = slotv(v, w); -> returns Py_NotImplemented. But in this case we should execute x = slotw(v, w); and function should be completed in the same way as when we adding integer to class. Can someone advise please where I mistake. -- thanks, Igor Vasilyev -------------- next part -------------- An HTML attachment was scrubbed... URL: From igor.vasilyev at oracle.com Thu Oct 3 15:45:18 2013 From: igor.vasilyev at oracle.com (Igor Vasilyev) Date: Thu, 03 Oct 2013 17:45:18 +0400 Subject: [Python-Dev] summing integer and class Message-ID: <524D74EE.90308@oracle.com> Hi. Example test.py: class A(): def __add__(self, var): print("I'm in A class") return 5 a = A() a+1 1+a Execution: python test.py I'm in A class Traceback (most recent call last): File "../../test.py", line 7, in 1+a TypeError: unsupported operand type(s) for +: 'int' and 'instance' So adding integer to class works fine, but adding class to integer fails. I could not understand why it happens. In objects/abstact.c we have the following function: static PyObject * binary_op1(PyObject *v, PyObject *w, const int op_slot) { PyObject *x; binaryfunc slotv = NULL; binaryfunc slotw = NULL; if (v->ob_type->tp_as_number != NULL) slotv = NB_BINOP(v->ob_type->tp_as_number, op_slot); if (w->ob_type != v->ob_type && w->ob_type->tp_as_number != NULL) { slotw = NB_BINOP(w->ob_type->tp_as_number, op_slot); if (slotw == slotv) slotw = NULL; } if (slotv) { if (slotw && PyType_IsSubtype(w->ob_type, v->ob_type)) { x = slotw(v, w); if (x != Py_NotImplemented) return x; Py_DECREF(x); /* can't do it */ slotw = NULL; } x = slotv(v, w); if (x != Py_NotImplemented) return x; Py_DECREF(x); /* can't do it */ } if (slotw) { x = slotw(v, w); if (x != Py_NotImplemented) return x; Py_DECREF(x); /* can't do it */ } Py_RETURN_NOTIMPLEMENTED; } When we adding class to integer we have both slotv and slotw. x = slotv(v, w); -> returns Py_NotImplemented. But in this case we should execute x = slotw(v, w); and function should be completed in the same way as when we adding integer to class. Can someone advise please where I mistake. -- thanks, Igor Vasilyev From victor.stinner at gmail.com Thu Oct 3 17:50:06 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 3 Oct 2013 17:50:06 +0200 Subject: [Python-Dev] PEP 454: Add a new tracemalloc module (final version) Message-ID: Hi, I worked on the implementation of the tracemalloc module and its PEP 454. I consider that this third version of the PEP is ready for a final review. What should be done to finish the PEP? HTML version of the PEP 454: http://www.python.org/dev/peps/pep-0454/ Full documentation of the tracemalloc module, with examples, a short tutorial and the documentation of the command line (python -m tracemalloc): http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html I addressed all points of my own TODO list, even minor tasks. I still have some minor tasks, but only on the implementation, not on the API of the module. The API is designed to process easily data collected by tracemalloc and to be extensible. It should now be easy to write a GUI to display these data for example. API changes between versions 2 and 3 of the PEP: * Add Metric class: in version 2, a snapshot had an "user_data" attribute which was not defined, not even its type. Metrics are now formalized, displayed and can be compared. They are used to track the process memory, number of Python objects, size of memory traced by tracemalloc, etc. The API also allows you to add your own metrics. * Rewrite the "scheduler" API. I removed the restriction of a single task at the same time. tracemalloc now supports multiple tasks which have a new trigger: threshold on the traced memory. I added a new Task class. DisplayTopTask and TakeSnapshotTask classes inherit from Task, to inherit its methods and attributes. It is now possible to only schedule a task "repeat" times. * Add DisplayTop.display() method, more convinient than having to create a DisplayTopTask instance. * Remove Frame, Trace and TraceStats classes for efficiency. tracemalloc handles millions of such objects. Creating and serialization millions of such small objects could take up to 30 seconds. I was not really convinced that a class is required to store only two fields, a tuple is just fine for a debug module. * (and many other minor changes) I optimized the implementation which is now faster and releases the memory earlier thanks to a new memory pool (only used by the module). PEP: 454 Title: Add a new tracemalloc module to trace Python memory allocations Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3-September-2013 Python-Version: 3.4 Abstract ======== Add a new ``tracemalloc`` module to trace memory blocks allocated by Python. Rationale ========= Common debug tools tracing memory allocations read the C filename and line number. Using such tool to analyze Python memory allocations does not help because most memory block are allocated in the same C function, in ``PyMem_Malloc()`` for example. There are debug tools dedicated to the Python language like ``Heapy`` and ``PySizer``. These tools analyze objects type and/or content. They are useful when most memory leaks are instances of the same type and this type is only instantiated in a few functions. The problem is when the object type is very common like ``str`` or ``tuple``, and it is hard to identify where these objects are instantiated. Finding reference cycles is also a difficult problem. There are different tools to draw a diagram of all references. These tools cannot be used on large applications with thousands of objects because the diagram is too huge to be analyzed manually. Proposal ======== Using the PEP 445, it becomes easy to setup an hook on Python memory allocators. A hook can inspect Python internals to retrieve the Python tracebacks. This PEP proposes to add a new ``tracemalloc`` module. It is a debug tool to trace memory blocks allocated by Python. The module provides the following information: * Compute the differences between two snapshots to detect memory leaks * Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks * Traceback where a memory block was allocated The API of the tracemalloc module is similar to the API of the faulthandler module: ``enable()``, ``disable()`` and ``is_enabled()`` functions, an environment variable (``PYTHONFAULTHANDLER`` and ``PYTHONTRACEMALLOC``), a ``-X`` command line option (``-X faulthandler`` and ``-X tracemalloc``). See the `documentation of the faulthandler module `_. The tracemalloc module has been written for CPython. Other implementations of Python may not provide it. API === To trace most memory blocks allocated by Python, the module should be enabled as early as possible by setting the ``PYTHONTRACEMALLOC`` environment variable to ``1``, or by using ``-X tracemalloc`` command line option. The ``tracemalloc.enable()`` function can also be called to start tracing Python memory allocations. By default, a trace of an allocated memory block only stores one frame. Use the ``set_traceback_limit()`` function to store more frames. Python memory blocks allocated in the ``tracemalloc`` module are also traced by default. Use ``add_exclude_filter(tracemalloc.__file__)`` to ignore these these memory allocations. At fork, the module is automatically disabled in the child process. Main Functions -------------- ``cancel_tasks()`` function: Cancel all scheduled tasks. See also the ``get_tasks()`` function. ``clear_traces()`` function: Clear all traces and statistics on Python memory allocations, and reset the ``get_arena_size()`` and ``get_traced_memory()`` counters. ``disable()`` function: Stop tracing Python memory allocations and cancel scheduled tasks. See also ``enable()`` and ``is_enabled()`` functions. ``enable()`` function: Start tracing Python memory allocations. At fork, the module is automatically disabled in the child process. See also ``disable()`` and ``is_enabled()`` functions. ``get_stats()`` function: Get statistics on traced Python memory blocks as a dictionary ``{filename (str): {line_number (int): stats}}`` where *stats* in a ``(size: int, count: int)`` tuple, *filename* and *line_number* can be ``None``. Return an empty dictionary if the ``tracemalloc`` module is disabled. See also the ``get_traces()`` function. ``get_tasks()`` function: Get the list of scheduled tasks, list of ``Task`` instances. ``is_enabled()`` function: ``True`` if the ``tracemalloc`` module is tracing Python memory allocations, ``False`` otherwise. See also ``enable()`` and ``disable()`` functions. Trace Functions --------------- ``get_traceback_limit()`` function: Get the maximum number of frames stored in the traceback of a trace of a memory block. Use the ``set_traceback_limit()`` function to change the limit. ``get_object_address(obj)`` function: Get the address of the memory block of the specified Python object. A Python object can be composed by multiple memory blocks, the function only returns the address of the main memory block. See also ``get_object_trace()`` and ``gc.get_referrers()`` functions. ``get_object_trace(obj)`` function: Get the trace of a Python object *obj* as a ``(size: int, traceback)`` tuple where *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. The function only returns the trace of the main memory block of the object. The *size* of the trace is smaller than the total size of the object if the object is composed by more than one memory block. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the object. See also ``get_object_address()``, ``get_trace()``, ``get_traces()``, ``gc.get_referrers()`` and ``sys.getsizeof()`` functions. ``get_trace(address)`` function: Get the trace of a memory block as a ``(size: int, traceback)`` tuple where *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the memory block. See also ``get_object_trace()``, ``get_stats()`` and ``get_traces()`` functions. ``get_traces()`` function: Get all traces of Python memory allocations as a dictionary ``{address (int): trace}`` where *trace* is a ``(size: int, traceback)`` and *traceback* is a list of ``(filename: str, lineno: int)``. *traceback* can be empty, *filename* and *lineno* can be None. Return an empty dictionary if the ``tracemalloc`` module is disabled. See also ``get_object_trace()``, ``get_stats()`` and ``get_trace()`` functions. ``set_traceback_limit(nframe: int)`` function: Set the maximum number of frames stored in the traceback of a trace of a memory block. Storing the traceback of each memory allocation has an important overhead on the memory usage. Use the ``get_tracemalloc_memory()`` function to measure the overhead and the ``add_filter()`` function to select which memory allocations are traced. Use the ``get_traceback_limit()`` function to get the current limit. Filter Functions ---------------- ``add_filter(filter)`` function: Add a new filter on Python memory allocations, *filter* is a ``Filter`` instance. All inclusive filters are applied at once, a memory allocation is only ignored if no inclusive filter match its trace. A memory allocation is ignored if at least one exclusive filter matchs its trace. The new filter is not applied on already collected traces. Use the ``clear_traces()`` function to ensure that all traces match the new filter. ``add_include_filter(filename: str, lineno: int=None, traceback: bool=False)`` function: Add an inclusive filter: helper for the ``add_filter()`` method creating a ``Filter`` instance with the ``Filter.include`` attribute set to ``True``. Example: ``tracemalloc.add_include_filter(tracemalloc.__file__)`` only includes memory blocks allocated by the ``tracemalloc`` module. ``add_exclude_filter(filename: str, lineno: int=None, traceback: bool=False)`` function: Add an exclusive filter: helper for the ``add_filter()`` method creating a ``Filter`` instance with the ``Filter.include`` attribute set to ``False``. Example: ``tracemalloc.add_exclude_filter(tracemalloc.__file__)`` ignores memory blocks allocated by the ``tracemalloc`` module. ``clear_filters()`` function: Reset the filter list. See also the ``get_filters()`` function. ``get_filters()`` function: Get the filters on Python memory allocations as list of ``Filter`` instances. See also the ``clear_filters()`` function. Metric Functions ---------------- The following functions can be used to add metrics to a snapshot, see the ``Snapshot.add_metric()`` method. ``get_allocated_blocks()`` function: Get the current number of allocated memory blocks. ``get_arena_size()`` function: Get the size in bytes of traced arenas. See also the ``get_pymalloc_stats()`` function. ``get_process_memory()`` function: Get the memory usage of the current process as a ``(rss: int, vms: int)`` tuple, *rss* is the "Resident Set Size" in bytes and *vms* is the size of the virtual memory in bytes Return ``None`` if the platform is not supported. ``get_pymalloc_stats()`` function: Get statistics on the ``pymalloc`` allocator as a dictionary. +---------------------+-------------------------------------------------------+ | Key | Description | +=====================+=======================================================+ | ``alignment`` | Alignment of addresses returned to the user. | +---------------------+-------------------------------------------------------+ | ``threshold`` | Small block threshold in bytes: pymalloc uses | | | PyMem_RawMalloc() for allocation greater than | | | threshold. | +---------------------+-------------------------------------------------------+ | ``nalloc`` | Number of times object malloc called | +---------------------+-------------------------------------------------------+ | ``arena_size`` | Arena size in bytes | +---------------------+-------------------------------------------------------+ | ``total_arenas`` | Number of calls to new_arena(): total number of | | | allocated arenas, including released arenas | +---------------------+-------------------------------------------------------+ | ``max_arenas`` | Maximum number of arenas | +---------------------+-------------------------------------------------------+ | ``arenas`` | Number of arenas currently allocated | +---------------------+-------------------------------------------------------+ | ``allocated_bytes`` | Number of bytes in allocated blocks | +---------------------+-------------------------------------------------------+ | ``available_bytes`` | Number of bytes in available blocks in used pools | +---------------------+-------------------------------------------------------+ | ``pool_size`` | Pool size in bytes | +---------------------+-------------------------------------------------------+ | ``free_pools`` | Number of unused pools | +---------------------+-------------------------------------------------------+ | ``pool_headers`` | Number of bytes wasted in pool headers | +---------------------+-------------------------------------------------------+ | ``quantization`` | Number of bytes in used and full pools wasted due to | | | quantization, i.e. the necessarily leftover space at | | | the ends of used and full pools. | +---------------------+-------------------------------------------------------+ | ``arena_alignment`` | Number of bytes for arena alignment padding | +---------------------+-------------------------------------------------------+ The function is not available if Python is compiled without ``pymalloc``. See also ``get_arena_size()`` and ``sys._debugmallocstats()`` functions. ``get_traced_memory()`` function: Get the current size and maximum size of memory blocks traced by the ``tracemalloc`` module as a tuple: ``(size: int, max_size: int)``. ``get_tracemalloc_memory()`` function: Get the memory usage in bytes of the ``tracemalloc`` module as a tuple: ``(size: int, free: int)``. * *size*: total size of bytes allocated by the module, including *free* bytes * *free*: number of free bytes available to store data ``get_unicode_interned()`` function: Get the size in bytes and the length of the dictionary of Unicode interned strings as a ``(size: int, length: int)`` tuple. The size is the size of the dictionary, excluding the size of strings. DisplayTop ---------- ``DisplayTop()`` class: Display the top of allocated memory blocks. ``display(count=10, group_by="line", cumulative=False, file=None, callback=None)`` method: Take a snapshot and display the top *count* biggest allocated memory blocks grouped by *group_by*. *callback* is an optional callable object which can be used to add metrics to a snapshot. It is called with only one parameter: the newly created snapshot instance. Use the ``Snapshot.add_metric()`` method to add new metric. Return the snapshot, a ``Snapshot`` instance. ``display_snapshot(snapshot, count=10, group_by="line", cumulative=False, file=None)`` method: Display a snapshot of memory blocks allocated by Python, *snapshot* is a ``Snapshot`` instance. ``display_top_diff(top_diff, count=10, file=None)`` method: Display differences between two ``GroupedStats`` instances, *top_diff* is a ``StatsDiff`` instance. ``display_top_stats(top_stats, count=10, file=None)`` method: Display the top of allocated memory blocks grouped by the ``GroupedStats.group_by`` attribute of *top_stats*, *top_stats* is a ``GroupedStats`` instance. ``average`` attribute: If ``True`` (default value), display the average size of memory blocks. ``color`` attribute: If ``True``, always use colors. If ``False``, never use colors. The default value is ``None``: use colors if the *file* parameter is a TTY device. ``compare_to_previous`` attribute: If ``True`` (default value), compare to the previous snapshot. If ``False``, compare to the first snapshot. ``filename_parts`` attribute: Number of displayed filename parts (int, default: ``3``). Extra parts are replaced with ``'...'``. ``metrics`` attribute: If ``True`` (default value), display metrics: see ``Snapshot.metrics``. ``previous_top_stats`` attribute: Previous ``GroupedStats`` instance, or first ``GroupedStats`` instance if ``compare_to_previous`` is ``False``, used to display the differences between two snapshots. ``size`` attribute: If ``True`` (default value), display the size of memory blocks. DisplayTopTask -------------- ``DisplayTopTask(count=10, group_by="line", cumulative=False, file=sys.stdout, callback=None)`` class: Task taking temporary snapshots and displaying the top *count* memory allocations grouped by *group_by*. ``DisplayTopTask`` is based on the ``Task`` class and so inherit all attributes and methods, especially: * ``Task.cancel()`` * ``Task.schedule()`` * ``Task.set_delay()`` * ``Task.set_memory_threshold()`` Modify the ``display_top`` attribute to customize the display. ``display()`` method: Take a snapshot and display the top ``count`` biggest allocated memory blocks grouped by ``group_by`` using the ``display_top`` attribute. Return the snapshot, a ``Snapshot`` instance. ``callback`` attribute: *callback* is an optional callable object which can be used to add metrics to a snapshot. It is called with only one parameter: the newly created snapshot instance. Use the ``Snapshot.add_metric()`` method to add new metric. ``count`` attribute: Maximum number of displayed memory blocks. ``cumulative`` attribute: If ``True``, cumulate size and count of memory blocks of all frames of each trace, not only the most recent frame. The default value is ``False``. The option is ignored if the traceback limit is less than ``2``, see the ``get_traceback_limit()`` function. ``display_top`` attribute: Instance of ``DisplayTop``. ``file`` attribute: The top is written into *file*. ``group_by`` attribute: Determine how memory allocations are grouped: see ``Snapshot.top_by()`` for the available values. Filter ------ ``Filter(include: bool, pattern: str, lineno: int=None, traceback: bool=False)`` class: Filter to select which memory allocations are traced. Filters can be used to reduce the memory usage of the ``tracemalloc`` module, which can be read using the ``get_tracemalloc_memory()`` function. ``match(filename: str, lineno: int)`` method: Return ``True`` if the filter matchs the filename and line number, ``False`` otherwise. ``match_filename(filename: str)`` method: Return ``True`` if the filter matchs the filename, ``False`` otherwise. ``match_lineno(lineno: int)`` method: Return ``True`` if the filter matchs the line number, ``False`` otherwise. ``match_traceback(traceback)`` method: Return ``True`` if the filter matchs the *traceback*, ``False`` otherwise. *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples. ``include`` attribute: If *include* is ``True``, only trace memory blocks allocated in a file with a name matching filename ``pattern`` at line number ``lineno``. If *include* is ``False``, ignore memory blocks allocated in a file with a name matching filename :attr`pattern` at line number ``lineno``. ``lineno`` attribute: Line number (``int``). If is is ``None`` or less than ``1``, it matches any line number. ``pattern`` attribute: The filename *pattern* can contain one or many ``*`` joker characters which match any substring, including an empty string. The ``.pyc`` and ``.pyo`` file extensions are replaced with ``.py``. On Windows, the comparison is case insensitive and the alternative separator ``/`` is replaced with the standard separator ``\``. ``traceback`` attribute: If *traceback* is ``True``, all frames of the traceback are checked. If *traceback* is ``False``, only the most recent frame is checked. This attribute is ignored if the traceback limit is less than ``2``. See the ``get_traceback_limit()`` function. GroupedStats ------------ ``GroupedStats(timestamp: datetime.datetime, stats: dict, group_by: str, cumulative=False, metrics: dict=None)`` class: Top of allocated memory blocks grouped by *group_by* as a dictionary. The ``Snapshot.top_by()`` method creates a ``GroupedStats`` instance. ``compare_to(old_stats: GroupedStats=None)`` method: Compare to an older ``GroupedStats`` instance. Return a ``StatsDiff`` instance. The ``StatsDiff.differences`` list is not sorted: call the ``StatsDiff.sort`` method to sort the list. ``None`` values are replaced with an empty string for filenames or zero for line numbers, because ``str`` and ``int`` cannot be compared to ``None``. ``cumulative`` attribute: If ``True``, cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. ``metrics`` attribute: Dictionary storing metrics read when the snapshot was created: ``{name (str): metric}`` where *metric* type is ``Metric``. ``group_by`` attribute: Determine how memory allocations were grouped: see ``Snapshot.top_by()`` for the available values. ``stats`` attribute: Dictionary ``{key: stats}`` where the *key* type depends on the ``group_by`` attribute and *stats* is a ``(size: int, count: int)`` tuple. See the ``Snapshot.top_by()`` method. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. Metric ------ ``Metric(name: str, value: int, format: str)`` class: Value of a metric when a snapshot is created. ``name`` attribute: Name of the metric. ``value`` attribute: Value of the metric. ``format`` attribute: Format of the metric: * ``'int'``: a number * ``'percent'``: percentage, ``1.0`` means ``100%`` * ``'size'``: a size in bytes Snapshot -------- ``Snapshot(timestamp: datetime.datetime, pid: int, traces: dict=None, stats: dict=None, metrics: dict=None)`` class: Snapshot of traces and statistics on memory blocks allocated by Python. Use ``TakeSnapshotTask`` to take regulary snapshots. ``add_gc_metrics()`` method: Add a metric on garbage collector: * ``gc.objects``: total number of Python objects See the ``gc`` module. ``add_metric(name: str, value: int, format: str)`` method: Helper to add a ``Metric`` instance to ``Snapshot.metrics``. Return the newly created ``Metric`` instance. Raise an exception if the name is already present in ``Snapshot.metrics``. ``add_process_memory_metrics()`` method: Add metrics on the process memory: * ``process_memory.rss``: Resident Set Size * ``process_memory.vms``: Virtual Memory Size These metrics are only available if the ``get_process_memory()`` function is available on the platform. ``add_pymalloc_metrics()`` method: Add metrics on the Python memory allocator (``pymalloc``): * ``pymalloc.blocks``: number of allocated memory blocks * ``pymalloc.size``: size of ``pymalloc`` arenas * ``pymalloc.max_size``: maximum size of ``pymalloc`` arenas * ``pymalloc.allocated``: number of allocated bytes * ``pymalloc.free``: number of free bytes * ``pymalloc.fragmentation``: fragmentation percentage of the arenas These metrics are only available if Python is compiled in debug mode, except ``pymalloc.blocks`` which is always available. ``add_tracemalloc_metrics()`` method: Add metrics on the ``tracemalloc`` module: * ``tracemalloc.traced.size``: size of memory blocks traced by the ``tracemalloc`` module * ``tracemalloc.traced.max_size``: maximum size of memory blocks traced by the ``tracemalloc`` module * ``tracemalloc.traces``: number of traces of Python memory blocks * ``tracemalloc.module.size``: total size of bytes allocated by the ``tracemalloc`` module, including free bytes * ``tracemalloc.module.free``: number of free bytes available for the ``tracemalloc`` module * ``tracemalloc.module.fragmentation``: percentage of fragmentation of the memory allocated by the ``tracemalloc`` module * ``tracemalloc.arena_size``: size of traced arenas ``tracemalloc.traces`` metric is only present if the snapshot was created with traces. ``add_unicode_metrics()`` method: Add metrics on the Unicode interned strings: * ``unicode_interned.size``: size of the dictionary, excluding size of strings * ``unicode_interned.len``: length of the dictionary ``apply_filters(filters)`` method: Apply filters on the ``traces`` and ``stats`` dictionaries, *filters* is a list of ``Filter`` instances. ``create(traces=False, metrics=True)`` classmethod: Take a snapshot of traces and/or statistics of allocated memory blocks. If *traces* is ``True``, ``get_traces`` is called and its result is stored in the ``Snapshot.traces`` attribute. This attribute contains more information than ``Snapshot.stats`` and uses more memory and more disk space. If *traces* is ``False``, ``Snapshot.traces`` is set to ``None``. If *metrics* is ``True``, fill ``Snapshot.metrics`` with metrics using the following methods: * ``add_gc_metrics`` * ``add_process_memory_metrics`` * ``add_pymalloc_metrics`` * ``add_tracemalloc_metrics`` * ``add_unicode_metrics`` If *metrics* is ``False``, ``Snapshot.metrics`` is set to an empty dictionary. Tracebacks of traces are limited to ``traceback_limit`` frames. Call ``set_traceback_limit()`` before calling ``Snapshot.create()`` to store more frames. The ``tracemalloc`` module must be enabled to take a snapshot. See the the ``enable()`` function. ``get_metric(name, default=None)`` method: Get the value of the metric called *name*. Return *default* if the metric does not exist. ``load(filename, traces=True)`` classmethod: Load a snapshot from a file. If *traces* is ``False``, don't load traces. ``top_by(group_by: str, cumulative: bool=False)`` method: Compute top statistics grouped by *group_by* as a ``GroupedStats`` instance: ===================== ======================== ============== group_by description key type ===================== ======================== ============== ``'filename'`` filename ``str`` ``'line'`` filename and line number ``(str, int)`` ``'address'`` memory block address ``int`` ===================== ======================== ============== If *cumulative* is ``True``, cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. The *cumulative* parameter is ignored if *group_by* is ``'address'`` or if the traceback limit is less than ``2``. ``write(filename)`` method: Write the snapshot into a file. ``metrics`` attribute: Dictionary storing metrics read when the snapshot was created: ``{name (str): metric}`` where *metric* type is ``Metric``. ``pid`` attribute: Identifier of the process which created the snapshot, result of ``os.getpid()``. ``stats`` attribute: Statistics on traced Python memory, result of the ``get_stats()`` function. ``traceback_limit`` attribute: Maximum number of frames stored in a trace of a memory block allocated by Python. ``traces`` attribute: Traces of Python memory allocations, result of the ``get_traces()`` function, can be ``None``. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. StatsDiff --------- ``StatsDiff(differences, old_stats, new_stats)`` class: Differences between two ``GroupedStats`` instances. The ``GroupedStats.compare_to`` method creates a ``StatsDiff`` instance. ``sort()`` method: Sort the ``differences`` list from the biggest difference to the smallest difference. Sort by ``abs(size_diff)``, *size*, ``abs(count_diff)``, *count* and then by *key*. ``differences`` attribute: Differences between ``old_stats`` and ``new_stats`` as a list of ``(size_diff, size, count_diff, count, key)`` tuples. *size_diff*, *size*, *count_diff* and *count* are ``int``. The key type depends on the ``GroupedStats.group_by`` attribute of ``new_stats``: see the ``Snapshot.top_by()`` method. ``old_stats`` attribute: Old ``GroupedStats`` instance, can be ``None``. ``new_stats`` attribute: New ``GroupedStats`` instance. Task ---- ``Task(func, *args, **kw)`` class: Task calling ``func(*args, **kw)``. When scheduled, the task is called when the traced memory is increased or decreased by more than *threshold* bytes, or after *delay* seconds. ``call()`` method: Call ``func(*args, **kw)`` and return the result. ``cancel()`` method: Cancel the task. Do nothing if the task is not scheduled. ``get_delay()`` method: Get the delay in seconds. If the delay is ``None``, the timer is disabled. ``get_memory_threshold()`` method: Get the threshold of the traced memory. When scheduled, the task is called when the traced memory is increased or decreased by more than *threshold* bytes. The memory threshold is disabled if *threshold* is ``None``. See also the ``set_memory_threshold()`` method and the ``get_traced_memory()`` function. ``schedule(repeat: int=None)`` method: Schedule the task *repeat* times. If *repeat* is ``None``, the task is rescheduled after each call until it is cancelled. If the method is called twice, the task is rescheduled with the new *repeat* parameter. The task must have a memory threshold or a delay: see ``set_delay()`` and ``set_memory_threshold()`` methods. The ``tracemalloc`` must be enabled to schedule a task: see the ``enable`` function. The task is cancelled if the ``call()`` method raises an exception. The task can be cancelled using the ``cancel()`` method or the ``cancel_tasks()`` function. ``set_delay(seconds: int)`` method: Set the delay in seconds before the task will be called. Set the delay to ``None`` to disable the timer. The timer is based on the Python memory allocator, it is not real time. The task is called after at least *delay* seconds, it is not called exactly after *delay* seconds if no Python memory allocation occurred. The timer has a resolution of 1 second. The task is rescheduled if it was scheduled. ``set_memory_threshold(size: int)`` method: Set the threshold of the traced memory. When scheduled, the task is called when the traced memory is increased or decreased by more than *threshold* bytes. Set the threshold to ``None`` to disable it. The task is rescheduled if it was scheduled. See also the ``get_memory_threshold()`` method and the ``get_traced_memory()`` function. ``func`` attribute: Function, callable object. ``func_args`` attribute: Function arguments, ``tuple``. ``func_kwargs`` attribute: Function keyword arguments, ``dict``. It can be ``None``. TakeSnapshotTask ---------------- ``TakeSnapshotTask(filename_template: str="tracemalloc-$counter.pickle", traces: bool=False, metrics: bool=True, callback: callable=None)`` class: Task taking snapshots of Python memory allocations and writing them into files. ``TakeSnapshotTask`` is based on the ``Task`` class and so inherit all attributes and methods, especially: * ``Task.cancel()`` * ``Task.schedule()`` * ``Task.set_delay()`` * ``Task.set_memory_threshold()`` ``take_snapshot()`` method: Take a snapshot and write it into a file. Return ``(snapshot, filename)`` where *snapshot* is a ``Snapshot`` instance and filename type is ``str``. ``callback`` attribute: *callback* is an optional callable object which can be used to add metrics to a snapshot. It is called with only one parameter: the newly created snapshot instance. Use the ``Snapshot.add_metric()`` method to add new metric. ``filename_template`` attribute: Template to create a filename. The template supports the following variables: * ``$pid``: identifier of the current process * ``$timestamp``: current date and time * ``$counter``: counter starting at 1 and incremented at each snapshot, formatted as 4 decimal digits The default template is ``'tracemalloc-$counter.pickle'``. ``metrics`` attribute: Parameter passed to the ``Snapshot.create()`` function. ``traces`` attribute: Parameter passed to the ``Snapshot.create()`` function. Links ===== tracemalloc: * `#18874: Add a new tracemalloc module to trace Python memory allocations `_ * `pytracemalloc on PyPI `_ Similar projects: * `Meliae: Python Memory Usage Analyzer `_ * `Guppy-PE: umbrella package combining Heapy and GSL `_ * `PySizer `_: developed for Python 2.4 * `memory_profiler `_ * `pympler `_ * `Dozer `_: WSGI Middleware version of the CherryPy memory leak debugger * `objgraph `_ * `caulk `_ Copyright ========= This document has been placed into the public domain. From rosuav at gmail.com Thu Oct 3 17:51:46 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 4 Oct 2013 01:51:46 +1000 Subject: [Python-Dev] summing integer and class In-Reply-To: <1380805771.625450506@f364.i.mail.ru> References: <1380805771.625450506@f364.i.mail.ru> Message-ID: On Thu, Oct 3, 2013 at 11:09 PM, ????? ???????? wrote: > When we adding class to integer we have both slotv and slotw. x = slotv(v, > w); -> returns Py_NotImplemented. > But in this case we should execute x = slotw(v, w); and function should be > completed in the same way as when we adding integer to class. > > Can someone advise please where I mistake. No need to dig into the CPython source for this, the answer's pretty simple: 1+a is handled by __radd__ not __add__. >>> class A(): def __add__(self, var): print("I'm in A class") return 5 def __radd__(self, var): print("I'm in A class, too") return 6 >>> a=A() >>> a+1 I'm in A class 5 >>> 1+a I'm in A class, too 6 You could ask this sort of thing on python-list at python.org rather than python-dev. ChrisA From python-dev at masklinn.net Thu Oct 3 17:54:48 2013 From: python-dev at masklinn.net (Xavier Morel) Date: Thu, 3 Oct 2013 17:54:48 +0200 Subject: [Python-Dev] summing integer and class In-Reply-To: <524D74EE.90308@oracle.com> References: <524D74EE.90308@oracle.com> Message-ID: <5AE8F325-A71E-479E-961C-D1A7175578E9@masklinn.net> On 2013-10-03, at 15:45 , Igor Vasilyev wrote: > Hi. > > Example test.py: > > class A(): > def __add__(self, var): > print("I'm in A class") > return 5 > a = A() > a+1 > 1+a > > Execution: > python test.py > I'm in A class > Traceback (most recent call last): > File "../../test.py", line 7, in > 1+a > TypeError: unsupported operand type(s) for +: 'int' and 'instance' > > > So adding integer to class works fine, but adding class to integer fails. > I could not understand why it happens. In objects/abstact.c we have the following function: > python-dev is about developing Python itself, not about developing in Python, so that's the wrong mailing list for these kinds of question. But FWIW the answer is that Python first tries 1.__add__(a), when that fails (with NotImplemented) it uses the reflected method[0] which is a.__radd__(1). Since that does not exist, the operation is invalid. [0] http://docs.python.org/2/reference/datamodel.html#object.__radd__ From ckaynor at zindagigames.com Thu Oct 3 17:58:44 2013 From: ckaynor at zindagigames.com (Chris Kaynor) Date: Thu, 3 Oct 2013 08:58:44 -0700 Subject: [Python-Dev] summing integer and class In-Reply-To: <524D74EE.90308@oracle.com> References: <524D74EE.90308@oracle.com> Message-ID: This list is for development OF Python, not for development in python. For that reason, I will redirect this to python-list as well. My actual answer is below. On Thu, Oct 3, 2013 at 6:45 AM, Igor Vasilyev wrote: > Hi. > > Example test.py: > > class A(): > def __add__(self, var): > print("I'm in A class") > return 5 > a = A() > a+1 > 1+a > Execution: > python test.py > I'm in A class > Traceback (most recent call last): > File "../../test.py", line 7, in > 1+a > TypeError: unsupported operand type(s) for +: 'int' and 'instance' > > > So adding integer to class works fine, but adding class to integer fails. > I could not understand why it happens. In objects/abstact.c we have the > following function: > Based on the code you provided, you are only overloading the __add__ operator, which is only called when an "A" is added to something else, not when something is added to an "A". You can also override the __radd__ method to perform the swapped addition. See http://docs.python.org/2/reference/datamodel.html#object.__radd__ for the documentation (it is just below the entry on __add__). Note that for many simple cases, you could define just a single function, which then is defined as both the __add__ and __radd__ operator. For example, you could modify your "A" sample class to look like: class A(): def __add__(self, var): print("I'm in A") return 5 __radd__ = __add__ Which will produce: >>> a = A() >>> a + 1 I'm in A 5 >>> 1 + a I'm in A 5 Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Thu Oct 3 20:42:28 2013 From: christian at python.org (Christian Heimes) Date: Thu, 03 Oct 2013 20:42:28 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? Message-ID: Hi, some of you may have seen that I'm working on a PEP for a new hash API and new algorithms for hashing of bytes and str. The PEP has three major aspects. It introduces DJB's SipHash as secure hash algorithm, chances the hash API to process blocks of data instead characters and it adds an API to make the algorithm pluggable. A draft is already available [1]. Now I got some negative feedback on the 'pluggable' aspect of the new PEP on Twitter [2]. I like to get feedback from you before I finalize the PEP. The PEP proposes a pluggable hash API for a couple of reasons. I like to give users of Python a chance to replace a secure hash algorithm with a faster hash algorithm. SipHash is about as fast as FNV for common cases as our implementation of FNV process only 8 to 32 bits per cycle instead of 32 or 64. I haven't actually benchmarked how a faster hash algorithm affects the a real program, though ... I also like to make it easier to replace the hash algorithm with a different one in case a vulnerability is found. With the new API vendors and embedders have an easy and clean way to use their own hash implementation or an optimized version that is more suitable for their platform, too. For example a mobile phone vendor could provide an optimized implementation with ARM NEON intrinsics. On which level should Python support a pluggable hash algorithm? 1) Compile time option: The hash code is compiled into Python's core. Embedders have to recompile Python with different options to replace the function. 2) Library option: A hash algorithm can be added and one avaible hash algorithm can be set before Py_Initialize() is called for the first time. The approach gives embedders the chance the set their own algorithm without recompiling Python. 3) Startup options: Like 2) plus an additional environment variable and command line argument to select an algorithm. With a startup option users can select a different algorithm themselves. Christian [1] http://www.python.org/dev/peps/pep-0456/ [2] https://twitter.com/EDEADLK/status/385572395777818624 From solipsis at pitrou.net Thu Oct 3 20:50:33 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 3 Oct 2013 20:50:33 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? References: Message-ID: <20131003205033.7e5730c0@fsol> On Thu, 03 Oct 2013 20:42:28 +0200 Christian Heimes wrote: > > I haven't actually benchmarked how a faster hash algorithm > affects the a real program, though ... Chances are it doesn't. Only a "slow enough" hash algorithm might have an impact, IMHO. > On which level should Python support a pluggable hash algorithm? > > 1) Compile time option: The hash code is compiled into Python's core. > Embedders have to recompile Python with different options to replace the > function. Not much point IMHO. Embedders can patch Python if they really need this. > 2) Library option: A hash algorithm can be added and one avaible hash > algorithm can be set before Py_Initialize() is called for the first > time. Too complicated. The library option should only offer the option to replace the hash algorithm, not "add an available algorithm". > 3) Startup options: Like 2) plus an additional environment variable and > command line argument to select an algorithm. With a startup option > users can select a different algorithm themselves. -0.9. I think it's overkill. Regards Antoine. From guido at python.org Thu Oct 3 21:05:15 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 3 Oct 2013 12:05:15 -0700 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: Message-ID: Hm. I would like to stick to the philosophy that Python's hash should be as fast as it possibly can be, and should not be mistaken for a cryptographic hash. The point is to optimize dict lookups, nothing more, given typical (or even atypical) key distribution, not to thwart deliberate attacks. We already have adopted a feature that plugged most viable attacks on web apps, I think that's enough. I also agree with Antoine's response. On Thu, Oct 3, 2013 at 11:42 AM, Christian Heimes wrote: > Hi, > > some of you may have seen that I'm working on a PEP for a new hash API > and new algorithms for hashing of bytes and str. The PEP has three major > aspects. It introduces DJB's SipHash as secure hash algorithm, chances > the hash API to process blocks of data instead characters and it adds an > API to make the algorithm pluggable. A draft is already available [1]. > > Now I got some negative feedback on the 'pluggable' aspect of the new > PEP on Twitter [2]. I like to get feedback from you before I finalize > the PEP. > > The PEP proposes a pluggable hash API for a couple of reasons. I like to > give users of Python a chance to replace a secure hash algorithm with a > faster hash algorithm. SipHash is about as fast as FNV for common cases > as our implementation of FNV process only 8 to 32 bits per cycle instead > of 32 or 64. I haven't actually benchmarked how a faster hash algorithm > affects the a real program, though ... > > I also like to make it easier to replace the hash algorithm with a > different one in case a vulnerability is found. With the new API vendors > and embedders have an easy and clean way to use their own hash > implementation or an optimized version that is more suitable for their > platform, too. For example a mobile phone vendor could provide an > optimized implementation with ARM NEON intrinsics. > > > On which level should Python support a pluggable hash algorithm? > > 1) Compile time option: The hash code is compiled into Python's core. > Embedders have to recompile Python with different options to replace the > function. > > 2) Library option: A hash algorithm can be added and one avaible hash > algorithm can be set before Py_Initialize() is called for the first > time. The approach gives embedders the chance the set their own > algorithm without recompiling Python. > > 3) Startup options: Like 2) plus an additional environment variable and > command line argument to select an algorithm. With a startup option > users can select a different algorithm themselves. > > Christian > > [1] http://www.python.org/dev/peps/pep-0456/ > [2] https://twitter.com/EDEADLK/status/385572395777818624 > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Thu Oct 3 21:23:20 2013 From: christian at python.org (Christian Heimes) Date: Thu, 03 Oct 2013 21:23:20 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: Message-ID: <524DC428.40901@python.org> Am 03.10.2013 21:05, schrieb Guido van Rossum: > Hm. I would like to stick to the philosophy that Python's hash > should be as fast as it possibly can be, and should not be mistaken > for a cryptographic hash. The point is to optimize dict lookups, > nothing more, given typical (or even atypical) key distribution, > not to thwart deliberate attacks. We already have adopted a feature > that plugged most viable attacks on web apps, I think that's > enough. I also agree with Antoine's response. Python's hash is neither as fast nor as secure as it can possibly be. It's not as fast because it doesn't use the full power of modern CPUs. In most cases the code processes only 1 or 2 bytes per cycle instead of 8 bytes on 64-bit architectures. Jean-Philippe Aumasson and Daniel J. Bernstein (who are coincidentally the authors of SipHash) have shown how to recover Python randomization keys. SipHash: more secure and about same speed on most systems optimized FNV: faster but with a known issue Christian From guido at python.org Thu Oct 3 21:45:10 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 3 Oct 2013 12:45:10 -0700 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: <524DC428.40901@python.org> References: <524DC428.40901@python.org> Message-ID: On Thu, Oct 3, 2013 at 12:23 PM, Christian Heimes wrote: > Am 03.10.2013 21:05, schrieb Guido van Rossum: > > Hm. I would like to stick to the philosophy that Python's hash > > should be as fast as it possibly can be, and should not be mistaken > > for a cryptographic hash. The point is to optimize dict lookups, > > nothing more, given typical (or even atypical) key distribution, > > not to thwart deliberate attacks. We already have adopted a feature > > that plugged most viable attacks on web apps, I think that's > > enough. I also agree with Antoine's response. > > Python's hash is neither as fast nor as secure as it can possibly be. > But fixing that shouldn't need all the extra stuff you're proposing. It's not as fast because it doesn't use the full power of modern CPUs. > In most cases the code processes only 1 or 2 bytes per cycle instead > of 8 bytes on 64-bit architectures. Jean-Philippe Aumasson and Daniel > J. Bernstein (who are coincidentally the authors of SipHash) have > shown how to recover Python randomization keys. > What's a Python randomization key? > SipHash: > more secure and about same speed on most systems > Same speed as what? > optimized FNV: > faster but with a known issue > What issue? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Thu Oct 3 21:53:43 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 03 Oct 2013 22:53:43 +0300 Subject: [Python-Dev] PEP 456 In-Reply-To: References: Message-ID: Just some comments. > the first time time with a bit shift of 7 Double "time". > with a 128bit seed and 64-bit output Inconsistancy with hyphen. There are same issues in other places. > bytes_hash provides the tp_hash slot function for unicode. Typo. Should be "unicode_hash". > len = PyUnicode_GET_LENGTH(self); > switch (PyUnicode_KIND(self)) { > case PyUnicode_1BYTE_KIND: { > const Py_UCS1 *c = PyUnicode_1BYTE_DATA(self); > x = _PyHash_Func->hashfunc(c, len * sizeof(Py_UCS1)); > break; > } > case PyUnicode_2BYTE_KIND: { ... x = _PyHash_Func->hashfunc(PyUnicode_BYTE_DATA(self), PyUnicode_GET_LENGTH(self) * PyUnicode_KIND(self)); > Equal hash values result in a hash collision and therefore cause a minor speed penalty for dicts and sets with mixed keys. The cause of the collision could be removed I doubt about this. If one collects bytes and strings in one dictionary, this equality will only double the number of collisions (for DoS attack we need increase it by thousands and millions times). So it doesn't matter. On the other hand, I one deliberately uses bytes and str subclasses with overridden equality, same hash for ASCII bytes and strings can be needed. > For very short strings the setup costs for SipHash dominates its speed but it is still in the same order of magnitude as the current FNV code. We could use other algorithm for very short strings if it makes matter. > The summarized total runtime of the benchmark is within 1% of the runtime of an unmodified Python 3.4 binary. What about deviations of individual tests? From christian at python.org Thu Oct 3 21:55:20 2013 From: christian at python.org (Christian Heimes) Date: Thu, 03 Oct 2013 21:55:20 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: <524DC428.40901@python.org> Message-ID: <524DCBA8.8090305@python.org> Am 03.10.2013 21:45, schrieb Guido van Rossum: > But fixing that shouldn't need all the extra stuff you're > proposing. I have proposed some of the extra stuff for more flexibility, the rest is for testing and debugging. > What's a Python randomization key? Python's hash randomization key, the seed to randomize the output of hash() for bytes and str. > SipHash: more secure and about same speed on most systems > > Same speed as what? Same speed as the current algorithm in Python 3.3 and earlier. > optimized FNV: faster but with a known issue > > What issue? Quote from https://131002.net/siphash/#at --- Jointly with Martin Bo?let, we demonstrated weaknesses in MurmurHash (used in Ruby, Java, etc.), CityHash (used in Google), and in Python's hash. Some of the technologies affected have switched to SipHash. See this oCERT advisory, and the following resources: [...] - Python script https://131002.net/siphash/poc.py to recover the secret seed of the hash randomization in Python 2.7.3 and 3.2.3 --- It's all documented in my PEP draft, too. Christian From amk at amk.ca Thu Oct 3 21:57:47 2013 From: amk at amk.ca (A.M. Kuchling) Date: Thu, 3 Oct 2013 15:57:47 -0400 Subject: [Python-Dev] [Python-checkins] cpython: [issue19151] Fix issue number in Misc/NEWS entry. In-Reply-To: <3crNXb2Yrhz7LjS@mail.python.org> References: <3crNXb2Yrhz7LjS@mail.python.org> Message-ID: <20131003195747.GA64857@DATLANDREWK.local> On Thu, Oct 03, 2013 at 08:48:47PM +0200, eric.snow wrote: > -- Issue #19951: Fix docstring and use of _get_suppported_file_loaders() to > +- Issue #19151: Fix docstring and use of _get_suppported_file_loaders() to ^^^ likely a typo? --amk From victor.stinner at gmail.com Thu Oct 3 22:06:46 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 3 Oct 2013 22:06:46 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: Message-ID: 2013/10/3 Christian Heimes : > A hash algorithm can be added and one avaible hash > algorithm can be set before Py_Initialize() is called for the first > time. "Py_Initialize" is not the good guard. Try for example "python3 -X faulthandler": PyObject_Hash() is called before Py_Initialize() to add "faulthandler" key into sys._xoptions dictionary. Today many Python internal functions are used before Python is initialized... See the PEP 432 which proposes to improve the situation: http://www.python.org/dev/peps/pep-0432/ Victor From ericsnowcurrently at gmail.com Thu Oct 3 22:08:30 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 3 Oct 2013 14:08:30 -0600 Subject: [Python-Dev] [Python-checkins] cpython: [issue19151] Fix issue number in Misc/NEWS entry. In-Reply-To: <20131003195747.GA64857@DATLANDREWK.local> References: <3crNXb2Yrhz7LjS@mail.python.org> <20131003195747.GA64857@DATLANDREWK.local> Message-ID: On Thu, Oct 3, 2013 at 1:57 PM, A.M. Kuchling wrote: > On Thu, Oct 03, 2013 at 08:48:47PM +0200, eric.snow wrote: >> -- Issue #19951: Fix docstring and use of _get_suppported_file_loaders() to >> +- Issue #19151: Fix docstring and use of _get_suppported_file_loaders() to > ^^^ likely a typo? It's just awkward phrasing. I suppose it would be more clear as "Fix the docstring of _get_supported_file_loaders() and its use to...". If you think it's worth fixing, I'll fix it. -eric From barry at python.org Thu Oct 3 22:21:41 2013 From: barry at python.org (Barry Warsaw) Date: Thu, 3 Oct 2013 16:21:41 -0400 Subject: [Python-Dev] [Python-checkins] cpython: [issue19151] Fix issue number in Misc/NEWS entry. In-Reply-To: References: <3crNXb2Yrhz7LjS@mail.python.org> <20131003195747.GA64857@DATLANDREWK.local> Message-ID: <20131003162141.13f0729c@anarchist> On Oct 03, 2013, at 02:08 PM, Eric Snow wrote: >On Thu, Oct 3, 2013 at 1:57 PM, A.M. Kuchling wrote: >> On Thu, Oct 03, 2013 at 08:48:47PM +0200, eric.snow wrote: >>> -- Issue #19951: Fix docstring and use of _get_suppported_file_loaders() to >>> +- Issue #19151: Fix docstring and use of _get_suppported_file_loaders() to > >It's just awkward phrasing. I suppose it would be more clear as "Fix >the docstring of _get_supported_file_loaders() and its use to...". If >you think it's worth fixing, I'll fix it. PPProbably not the typppo Andrew was pppointing out. -Bary From ericsnowcurrently at gmail.com Thu Oct 3 22:38:56 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 3 Oct 2013 14:38:56 -0600 Subject: [Python-Dev] [Python-checkins] cpython: [issue19151] Fix issue number in Misc/NEWS entry. In-Reply-To: <20131003162141.13f0729c@anarchist> References: <3crNXb2Yrhz7LjS@mail.python.org> <20131003195747.GA64857@DATLANDREWK.local> <20131003162141.13f0729c@anarchist> Message-ID: On Thu, Oct 3, 2013 at 2:21 PM, Barry Warsaw wrote: > PPProbably not the typppo Andrew was pppointing out. > > -Bary Ohhh, that typppo. -eric From guido at python.org Thu Oct 3 22:47:23 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 3 Oct 2013 13:47:23 -0700 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: <524DCBA8.8090305@python.org> References: <524DC428.40901@python.org> <524DCBA8.8090305@python.org> Message-ID: On Thu, Oct 3, 2013 at 12:55 PM, Christian Heimes wrote: > Am 03.10.2013 21:45, schrieb Guido van Rossum: > > But fixing that shouldn't need all the extra stuff you're > > proposing. > > I have proposed some of the extra stuff for more flexibility, the rest > is for testing and debugging. > Hm, I don't think we need more infrastructure for this. As Antoine said, if you're hacking on this you might as well edit the source. > > What's a Python randomization key? > > Python's hash randomization key, the seed to randomize the output of > hash() for bytes and str. > Is the seed itself crypto-safe? (I.e. is it derived carefully from urandom?) > > SipHash: more secure and about same speed on most systems > > > > Same speed as what? > > Same speed as the current algorithm in Python 3.3 and earlier. > OK, then I have no objection to switching to it, *if* the security issue is really worth fixing. Otherwise it would be better to look for a hash that is *faster*, given your assertion that the current hash is inefficient. > > optimized FNV: faster but with a known issue > > > > What issue? > > Quote from https://131002.net/siphash/#at > --- > Jointly with Martin Bo?let, we demonstrated weaknesses in MurmurHash > (used in Ruby, Java, etc.), CityHash (used in Google), and in Python's > hash. Some of the technologies affected have switched to SipHash. See > this oCERT advisory, and the following resources: > > [...] > > - Python script https://131002.net/siphash/poc.py to recover > the secret seed of the hash randomization in Python 2.7.3 and > 3.2.3 > Sounds a bit like some security researchers drumming up business. If you can run the binary, presumably you can also recover the seed by looking in /proc, right? Or use ctypes or something. This demonstration seems of academic interest only. > --- > > It's all documented in my PEP draft, too. Yeah, there's lots of stuff there. I'm looking for the TL;DR version. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Thu Oct 3 22:49:13 2013 From: christian at python.org (Christian Heimes) Date: Thu, 03 Oct 2013 22:49:13 +0200 Subject: [Python-Dev] PEP 456 In-Reply-To: References: Message-ID: <524DD849.3000708@python.org> Am 03.10.2013 21:53, schrieb Serhiy Storchaka: >> the first time time with a bit shift of 7 > > Double "time". thx, fixed >> with a 128bit seed and 64-bit output > > Inconsistancy with hyphen. There are same issues in other places. I have unified the use of hyphens, thx! >> bytes_hash provides the tp_hash slot function for unicode. > > Typo. Should be "unicode_hash". Fixed > x = _PyHash_Func->hashfunc(PyUnicode_BYTE_DATA(self), > PyUnicode_GET_LENGTH(self) * PyUnicode_KIND(self)); Oh nice, that's easier to read. It's PyUnicode_DATA(). > I doubt about this. If one collects bytes and strings in one dictionary, > this equality will only double the number of collisions (for DoS attack > we need increase it by thousands and millions times). So it doesn't > matter. On the other hand, I one deliberately uses bytes and str > subclasses with overridden equality, same hash for ASCII bytes and > strings can be needed. It's not a big problem. I merely wanted to point out that there is a simple possibility for a minor optimization. That's all. :) >> For very short strings the setup costs for SipHash dominates its speed > but it is still in the same order of magnitude as the current FNV code. > > We could use other algorithm for very short strings if it makes matter. I though of that, too. The threshold is rather small, though. As far as I remember an effective hash collision DoS works with 7 or 8 chars. >> The summarized total runtime of the benchmark is within 1% of the > runtime of an unmodified Python 3.4 binary. > > What about deviations of individual tests? Here you go. http://pastebin.com/dKdnBCgb http://pastebin.com/wtfUS5Zz Christian From ncoghlan at gmail.com Thu Oct 3 23:13:35 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 4 Oct 2013 07:13:35 +1000 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: Message-ID: On 4 Oct 2013 06:08, "Victor Stinner" wrote: > > 2013/10/3 Christian Heimes : > > A hash algorithm can be added and one avaible hash > > algorithm can be set before Py_Initialize() is called for the first > > time. > > "Py_Initialize" is not the good guard. Try for example "python3 -X > faulthandler": PyObject_Hash() is called before Py_Initialize() to add > "faulthandler" key into sys._xoptions dictionary. > > Today many Python internal functions are used before Python is > initialized... See the PEP 432 which proposes to improve the > situation: > http://www.python.org/dev/peps/pep-0432/ That problem exists because our main function doesn't follow the C API usage rules, though. We require other embedding applications to be better behaved than that if they want support :) That said, while I'm mostly in favour of the PEP, I think setting the algorithm should be a private API for 3.4. I do agree that since the platform support for SipHash is slightly narrower, we need to keep the existing hash algorithm around, make it relatively easy to enable and ensure we continue to test it on the build bots. I believe that last requirement for buildbot testing is the one that should drive the design of the private configuration API. Cheers, Nick. > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Oct 3 23:17:34 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 3 Oct 2013 14:17:34 -0700 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: Message-ID: On Thu, Oct 3, 2013 at 2:13 PM, Nick Coghlan wrote: > On 4 Oct 2013 06:08, "Victor Stinner" wrote: > > > > 2013/10/3 Christian Heimes : > > > A hash algorithm can be added and one avaible hash > > > algorithm can be set before Py_Initialize() is called for the first > > > time. > > > > "Py_Initialize" is not the good guard. Try for example "python3 -X > > faulthandler": PyObject_Hash() is called before Py_Initialize() to add > > "faulthandler" key into sys._xoptions dictionary. > > > > Today many Python internal functions are used before Python is > > initialized... See the PEP 432 which proposes to improve the > > situation: > > http://www.python.org/dev/peps/pep-0432/ > > That problem exists because our main function doesn't follow the C API > usage rules, though. We require other embedding applications to be better > behaved than that if they want support :) > > That said, while I'm mostly in favour of the PEP, I think setting the > algorithm should be a private API for 3.4. > > I do agree that since the platform support for SipHash is slightly > narrower, we need to keep the existing hash algorithm around, make it > relatively easy to enable and ensure we continue to test it on the build > bots. > > I believe that last requirement for buildbot testing is the one that > should drive the design of the private configuration API. > I'll defer to Nick for approval of this PEP. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Fri Oct 4 00:44:25 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 04 Oct 2013 01:44:25 +0300 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: <524DC428.40901@python.org> <524DCBA8.8090305@python.org> Message-ID: 03.10.13 23:47, Guido van Rossum ???????(??): > On Thu, Oct 3, 2013 at 12:55 PM, Christian Heimes > wrote: > > Am 03.10.2013 21:45, schrieb Guido van Rossum: > > But fixing that shouldn't need all the extra stuff you're > > proposing. > > I have proposed some of the extra stuff for more flexibility, the rest > is for testing and debugging. > > > Hm, I don't think we need more infrastructure for this. As Antoine said, > if you're hacking on this you might as well edit the source. What we could do is to move all hash-related stuff into separated .c and .h files. > > SipHash: more secure and about same speed on most systems > > > > Same speed as what? > > Same speed as the current algorithm in Python 3.3 and earlier. > > > OK, then I have no objection to switching to it, *if* the security issue > is really worth fixing. Otherwise it would be better to look for a hash > that is *faster*, given your assertion that the current hash is inefficient. Actually same speed only for UCS1 string. For UCS2 and UCS4 strings it can be 5x to 10x slower [1]. But I don't known how it affects real programs. [1] http://bugs.python.org/issue14621#msg175048 From greg.ewing at canterbury.ac.nz Fri Oct 4 00:58:56 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 04 Oct 2013 11:58:56 +1300 Subject: [Python-Dev] summing integer and class In-Reply-To: <524D74EE.90308@oracle.com> References: <524D74EE.90308@oracle.com> Message-ID: <524DF6B0.5020802@canterbury.ac.nz> Igor Vasilyev wrote: > class A(): > def __add__(self, var): > print("I'm in A class") > return 5 > a = A() > a+1 > 1+a > > Execution: > python test.py > I'm in A class > Traceback (most recent call last): > File "../../test.py", line 7, in > 1+a > TypeError: unsupported operand type(s) for +: 'int' and 'instance' You need to define an __radd__ method for it to work as a right-hand operand. > When we adding class to integer we have both slotv and slotw. x = > slotv(v, w); -> returns Py_NotImplemented. > But in this case we should execute x = slotw(v, w); Yes, but the wrapper that gets put in the type slot calls __rxxx__ instead of __xxx__ if it's being called for the right-hand operand. -- Greg From greg at krypto.org Fri Oct 4 05:24:49 2013 From: greg at krypto.org (Gregory P. Smith) Date: Thu, 3 Oct 2013 20:24:49 -0700 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: Message-ID: On Thu, Oct 3, 2013 at 11:42 AM, Christian Heimes wrote: > Hi, > > some of you may have seen that I'm working on a PEP for a new hash API > and new algorithms for hashing of bytes and str. The PEP has three major > aspects. It introduces DJB's SipHash as secure hash algorithm, chances > the hash API to process blocks of data instead characters and it adds an > API to make the algorithm pluggable. A draft is already available [1]. > > Now I got some negative feedback on the 'pluggable' aspect of the new > PEP on Twitter [2]. I like to get feedback from you before I finalize > the PEP. > > The PEP proposes a pluggable hash API for a couple of reasons. I like to > give users of Python a chance to replace a secure hash algorithm with a > faster hash algorithm. SipHash is about as fast as FNV for common cases > as our implementation of FNV process only 8 to 32 bits per cycle instead > of 32 or 64. I haven't actually benchmarked how a faster hash algorithm > affects the a real program, though ... > > I also like to make it easier to replace the hash algorithm with a > different one in case a vulnerability is found. With the new API vendors > and embedders have an easy and clean way to use their own hash > implementation or an optimized version that is more suitable for their > platform, too. For example a mobile phone vendor could provide an > optimized implementation with ARM NEON intrinsics. > > > On which level should Python support a pluggable hash algorithm? > > 1) Compile time option: The hash code is compiled into Python's core. > Embedders have to recompile Python with different options to replace the > function. > This would be fine with me. > > 2) Library option: A hash algorithm can be added and one avaible hash > algorithm can be set before Py_Initialize() is called for the first > time. The approach gives embedders the chance the set their own > algorithm without recompiling Python. > This would be more convenient. But I only want to replace the algorithm in my code embedding CPython, not add one. So long as the performance impact of supporting this is not usefully relevant, do that (let me supply the algorithm to be used for each of bytes and str before Py_Initialize is called). > > 3) Startup options: Like 2) plus an additional environment variable and > command line argument to select an algorithm. With a startup option > users can select a different algorithm themselves. > I can't imagine any reason I or anyone else would ever want this. side note: In Python 2 and earlier the hash algorithm went to great lengths to make unicode and bytes values that were the same (in at least ascii, possibly latin-1 or utf-8 as well) hash to the same value. Is it safe to assume that very annoying performance sapping invariant is no longer required in Python 3 given that the whole default encoding for bytes to unicode comparisons is gone? (and thus the need for them to land in the same dict hash bucket) -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Fri Oct 4 05:34:20 2013 From: greg at krypto.org (Gregory P. Smith) Date: Thu, 3 Oct 2013 20:34:20 -0700 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: Message-ID: On Thu, Oct 3, 2013 at 12:05 PM, Guido van Rossum wrote: > We already have adopted a feature that plugged most viable attacks on web > apps, I think that's enough. > Actually... we did not do a very good job on that: http://bugs.python.org/issue14621 The point of allowing alternates is to let people with needs choose something else if they want without having to jump through hoops of modifying the guts of Python to do it. I don't expect python as shipped by most OS distros to use anything other than our default. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Fri Oct 4 05:37:43 2013 From: greg at krypto.org (Gregory P. Smith) Date: Thu, 3 Oct 2013 20:37:43 -0700 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: Message-ID: On Thu, Oct 3, 2013 at 1:06 PM, Victor Stinner wrote: > 2013/10/3 Christian Heimes : > > A hash algorithm can be added and one avaible hash > > algorithm can be set before Py_Initialize() is called for the first > > time. > > "Py_Initialize" is not the good guard. Try for example "python3 -X > faulthandler": PyObject_Hash() is called before Py_Initialize() to add > "faulthandler" key into sys._xoptions dictionary. > > Today many Python internal functions are used before Python is > initialized... See the PEP 432 which proposes to improve the > situation: > http://www.python.org/dev/peps/pep-0432/ then I withdraw my desire for setting it before that. compile time is fine. but how would you make that usefully easier than the existing method of replacing the two functions in our bytes and str implementations? if you want a compile time flag, perhaps just call it --enable-sip-hash and get it over with since that's what we really want. ;) -gps > > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Fri Oct 4 08:20:52 2013 From: arigo at tunes.org (Armin Rigo) Date: Fri, 4 Oct 2013 08:20:52 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: <524DC428.40901@python.org> <524DCBA8.8090305@python.org> Message-ID: Hi Guido, On Thu, Oct 3, 2013 at 10:47 PM, Guido van Rossum wrote: > Sounds a bit like some security researchers drumming up business. If you can > run the binary, presumably you can also recover the seed by looking in > /proc, right? Or use ctypes or something. This demonstration seems of > academic interest only. I'll not try to defend the opposite point of view very actively, but let me just say that, in my opinion, your objection is not valid. It is broken the same way as a different objection, which would claim that Python can be made sandbox-safe without caring about the numerous segfault cases. They are all very obscure for sure; I tried at some point to list them in Lib/test/crashers. I gave up when people started deleting the files because they no longer crashed on newer versions, just because details changed --- but not because the general crash they explained was in any way fixed... Anyway, my point is that most segfaults can, given enough effort, be transformed into a single, well-documented tool to conduct a large class of attacks. The hash issue is similar. It should be IMHO either ignored (which is fine for a huge fraction of users), or seriously fixed by people with the correctly pessimistic approach. The current hash randomization is simply not preventing anything; someone posted long ago a way to recover bit-by-bit the hash randomized used by a remote web program in Python running on a server. The only benefit of this hash randomization option (-R) was to say to the press that Python fixed very quickly the problem when it was mediatized :-/ This kind of security issues should never be classified as "academic interest only". Instead they can be classified as "it will take weeks / months / years before some crazy man manages to put together a general attack script, but likely, someone will eventually". >From this point of view I'm saluting Christian's effort, even if I prefer to stay far away from this kind of issues myself :-) A bient?t, Armin. From stefan_ml at behnel.de Fri Oct 4 09:24:11 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 04 Oct 2013 09:24:11 +0200 Subject: [Python-Dev] project culture: take responsibility for your commits In-Reply-To: <87r4c3m6s8.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87r4c3m6s8.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Stephen, thank you for your very thoughtful answer. Stephen J. Turnbull, 03.10.2013 04:23: > Stefan Behnel writes: > > > Hi, I'm looking back on a rather unpleasant experience that I > > recently had in this developer community. Actually, twice by > > now. Here's what I take from it: You should take responsibility for > > your commits. > > I have no clue who you're addressing this advice to. If it's not > yourself (from the following, I gather it's not), I think the > implication of what you are saying is mistaken. Core devs (by which I > mean the high-profile developers who are candidates for PEP delegate) > regularly do take responsibility for their commits, just like any > other committer, by changing or reverting them. That's visible on > this list as well as in the commit logs. I'm aware of that, and I apologise to those who felt offended by my post. I really didn't mean for it to read that way, but I can see in retrospect that my phrasing and its implications were bound to be read as offence, both personally and to the audience. > > Let's assume these complaints [about the code] are reasonable > > That's not sufficient. They must also be presented reasonably, by the > standards of the community. Not everybody is good at doing that, and > those who aren't suffer, as does the project for losing a useful > contribution. Unfortunate, but digging out what matters from unclear > or high-handed presentations requires an enormous amount of effort, > like pyschotherapy. Good psychotherapists bill hundreds of dollars an > hour. The very best "pythotherapists" bill nothing, at least not to > this community. I'm also aware of that. In one of the OSS projects that I lead, bad bug reports are actually quite frequent due to a broad distribution of user experience (simplicity has its drawbacks, it seems). It can sometimes take way more time than I'd have wanted to invest to decipher them and/or ask back until it's clearer. > Regarding the specific core dev behavior that offended you, I can > speak from my experience in another project. > > > What do you do in that case? Do you tell them that what's in is in? > > I've done that and later reversed my position. In retrospect, I > believe I was correct at the time of first approach in the majority of > cases, though, on the grounds of the "lesser of two evils" as I > understood the issues (or occasionally that the contributor had > completely misunderstood the issues). > > In most cases the original requester never did come up with a coherent > argument, just that something unclear to me didn't work for them. > Reversal in such cases was due to a third party who was able to > explain the requester's requirements, and often contribute (most of) a > specification of a complete fix or a good compromise. Here, you are mostly saying that it's ok to say that for illegitimate and unclear complaints. Even in that case, I'd personally be very careful with that phrase. But I guess that seconds Brett's remarks on subjectivity. > > Do you tell them that you are a core developer and they are not? > > I've done that. I don't know if it applies to the cases you have in > mind, but invariably that was a last retort when I just wanted to shut > down a conversation that had already come back to the same place > twice, and polite guidance seemed to be a complete failure. Childish, > I guess, but it's been effective. That's not sufficient reason to use > it in Python, which has higher standards for courtesy than my other > project does. I also agree here. Personally, I can't recall a situation where I ever said that in my OSS projects (and I apologise to everyone I forget here ;) > Caveat: as with the next item, I have to wonder if you mistook an > explanation that > > in such disputes, the Python default is to go with the core dev's > gut feeling unless there's good reason to do otherwise, and you > haven't explained well enough yet > > for a snotty "I am and you're not, so go away!" > > > That they can try to do better, and if they are lucky, find someone > > else who applies their patch? > > Definitely, and I would advise any core developer to use exactly that > response as soon as they feel the discussion is becoming unprofitable. The problem is that these two can go hand in hand. As a non-committer, you are always at the mercy of core developers, and it feels bad to be made aware of it. If the situation (however it was phrased) is essentially "it's committed, and now find someone else to listen", then reverting is no longer really an option. It's very hard to convince one core developer to revert a commit of another (and in fact, it should be). So, basically, by simply turning away, you are forcing the bagger into fixing it themselves, i.e. into writing the patch, into "cleaning up the mess you left", not even knowing if there will ever be someone else to then apply it. That's a very awkward situation for them. Not uncommonly, writing that patch is way more work than the original core developer invested into his/her change, both because the non-committer is likely to have less experience with that specific code, and because the core developer (being a good developer) most likely took the lazy path already, so you are left with the less lazy and more involved options. I find that it helps me (personally) as a core developer to keep that heavy unbalance in mind. That pretty much sums up my reasons for the post that triggered this thread. > Of course they should spin it differently, something like > > Well, I don't agree that reverting my patch is necessary, and > something needs to be done to address the issue it handles. > You are welcome to propose your own patch, but I suggest you > seek out an alternative reviewer as it's hard for me to be > objective about my own code once I'm convinced it's correct. > > Are you *sure* that your phrasing above is what the reviewers wrote, > and not merely what you took from their writing? "Insult is far more > frequently taken than given." I agree with both points. > In sum, I don't see what you're complaining about. Sure, you may have > been mistreated in the case(s) in question, which sucks, of course. > If so, you should provide details, and maybe the particular kind of > case would be a lesson to all of us. Or perhaps some core devs > systematically abuse their positions (but as Brett says you should be > very careful about making that statement in public!) But I don't see > evidence that there's a systematic failure to review "core dev" > commits, or that "core devs" deny their own fallibility. Nor do I > think any of the behaviors you describe *out of context* are always > wrong (except "I'm a core dev and you're not" should be avoided even > if effective; it's rude and ad hominem). I actually agree with your conclusions, and I renew my apologies to those who felt offended. Stefan From victor.stinner at gmail.com Fri Oct 4 11:15:17 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 4 Oct 2013 11:15:17 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: <524DC428.40901@python.org> <524DCBA8.8090305@python.org> Message-ID: 2013/10/4 Armin Rigo : > The current hash randomization is > simply not preventing anything; someone posted long ago a way to > recover bit-by-bit the hash randomized used by a remote web program in > Python running on a server. Oh interesting, is it public? If yes, could we please search the URL of the exploit? I'm more motivated to fix an issue if it is proved to be exploitable. I still fail to understand the real impact of a hash DoS compared to other kinds of DoS. It's like the XML bomb: the vulnerability was also known since many years, but Christian only fixed the issue recently (and the fix was implemented in a package on the Cheeseshop, not in the stblib! Is that correct?). > The only benefit of this hash > randomization option (-R) was to say to the press that Python fixed > very quickly the problem when it was mediatized :-/ The real benefit is to warn users that they should not rely on the dictionary or set order/representation (in their unit tests), and that the hash function is not deterministic :-) (So now it is much easier to replace the hash function with SipHash or anything else, without breaking new applications.) Victor From christian at python.org Fri Oct 4 11:54:55 2013 From: christian at python.org (Christian Heimes) Date: Fri, 04 Oct 2013 11:54:55 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: <524DC428.40901@python.org> <524DCBA8.8090305@python.org> Message-ID: <524E906F.1060902@python.org> Am 04.10.2013 11:15, schrieb Victor Stinner: > 2013/10/4 Armin Rigo : >> The current hash randomization is simply not preventing anything; >> someone posted long ago a way to recover bit-by-bit the hash >> randomized used by a remote web program in Python running on a >> server. > > Oh interesting, is it public? If yes, could we please search the > URL of the exploit? I'm more motivated to fix an issue if it is > proved to be exploitable. I'm intrigued, too! > I still fail to understand the real impact of a hash DoS compared > to other kinds of DoS. It's like the XML bomb: the vulnerability > was also known since many years, but Christian only fixed the issue > recently (and the fix was implemented in a package on the > Cheeseshop, not in the stblib! Is that correct?). About the XML bomb and other issues ... I kinda lost my motivation to push the fixes into the stdlib. :( The code is ready. It just needs a proper configuration interface / API. The hash DoS and XML DoS vulnerabilities have one thing in common. Both multiply the effectiveness of an attack by several orders of magnitude. You don't need 100 GBit/sec to kick a service out of existence. A simple DSL line or mobile phone with 3G/HSDPA does the same job (if done right). Nowaday Python is important, for example major parts of the Brazilian Government run on Python, Zope and Plone. There are Dropbox, Google App Engine ... > The real benefit is to warn users that they should not rely on the > dictionary or set order/representation (in their unit tests), and > that the hash function is not deterministic :-) > > (So now it is much easier to replace the hash function with SipHash > or anything else, without breaking new applications.) Thanks for your groundwork and groudbreaking work, Victor! :) Christian From solipsis at pitrou.net Fri Oct 4 11:57:39 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 4 Oct 2013 11:57:39 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? References: <524DC428.40901@python.org> <524DCBA8.8090305@python.org> Message-ID: <20131004115739.17ce98f9@pitrou.net> Le Fri, 4 Oct 2013 11:15:17 +0200, Victor Stinner a ?crit : > 2013/10/4 Armin Rigo : > > The current hash randomization is > > simply not preventing anything; someone posted long ago a way to > > recover bit-by-bit the hash randomized used by a remote web program > > in Python running on a server. > > Oh interesting, is it public? If yes, could we please search the URL > of the exploit? I'm more motivated to fix an issue if it is proved to > be exploitable. > > I still fail to understand the real impact of a hash DoS compared to > other kinds of DoS. It's like the XML bomb: the vulnerability was also > known since many years, but Christian only fixed the issue recently > (and the fix was implemented in a package on the Cheeseshop, not in > the stblib! Is that correct?). > > > The only benefit of this hash > > randomization option (-R) was to say to the press that Python fixed > > very quickly the problem when it was mediatized :-/ > > The real benefit is to warn users that they should not rely on the > dictionary or set order/representation (in their unit tests), and that > the hash function is not deterministic :-) I agree it probably had educational value. Regards Antoine. From ethan at stoneleaf.us Fri Oct 4 16:30:28 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 04 Oct 2013 07:30:28 -0700 Subject: [Python-Dev] project culture: take responsibility for your commits In-Reply-To: References: Message-ID: <524ED104.2040202@stoneleaf.us> On 10/02/2013 11:58 AM, Stefan Behnel wrote: > > I'm looking back on a rather unpleasant experience that I recently had in > this developer community. Actually, twice by now. Here's what I take from it: > > You should take responsibility for your commits. It doesn't sound like you learned anything, then, as you apparently already knew this (judging from your later post). I find it disturbing that nowhere in your two posts to this thread do you take responsibility for your part in what happened. (Disclaimer: I'm only aware of one of the incidents.) Here is what I hope you learn, as it will benefit both you, the developers you work with, and hopefully Python as well: - Be respectful - Realize that people don't always agree on the best solution - Ask for clarification on responses if you don't think your point is being understood The second and third points follow from the first, and is the one that you seemed to have the most trouble with: starting a trouble ticket with accusations that something was snuck in and done behind peoples' backs is offensive, as are continual accusations that those you are working with simply don't understand. Add to that constant complaints about writing patches yourself... well, to be brief I am not surprised you didn't have a good experience -- I don't think anybody involved with that ticket had a good experience, including myself, and I was just a bystander. -- ~Ethan~ From martin at v.loewis.de Fri Oct 4 17:13:32 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Fri, 04 Oct 2013 17:13:32 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: <524DC428.40901@python.org> <524DCBA8.8090305@python.org> Message-ID: <20131004171332.Horde.DulYqZVHTxdkY3b8IGNSHg2@webmail.df.eu> Quoting Victor Stinner : > I still fail to understand the real impact of a hash DoS compared to > other kinds of DoS. I think the key question is: how many attacking nodes do you need to control to effectively make some system deny service. A threat is bigger if you can do it in 10 requests/s from a single host, instead of needing 10,000 hosts, each making 1000 request/s. With the hash DoS, the threat is that if you manage to fill some dictionary with colliding keys, then each lookup will take a very long time, and you might arrange to put many lookups into a single HTTP request. So a single HTTP request might get very costly CPU-wise. Whether this is a serious threat or not depends on what other threats the system being attacked is vulnerable to. Maybe there is something even simpler, or maybe the hash attack is the only hope of bringing the system to its knees. IMO, the hash attack is particularly tricky since it is very easy to argue and very difficult to demonstrate. So it can result in fear and uncertainty very easily, causing people to overreact just so that they won't be accused of inactivity. Regards, Martin From solipsis at pitrou.net Fri Oct 4 17:57:11 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 4 Oct 2013 17:57:11 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? References: <524DC428.40901@python.org> <524DCBA8.8090305@python.org> <20131004171332.Horde.DulYqZVHTxdkY3b8IGNSHg2@webmail.df.eu> Message-ID: <20131004175711.36a8938b@pitrou.net> Le Fri, 04 Oct 2013 17:13:32 +0200, martin at v.loewis.de a ?crit : > > Whether this is a serious threat or not depends on what other threats > the system being attacked is vulnerable to. Maybe there is something > even simpler, or maybe the hash attack is the only hope of bringing > the system to its knees. > > IMO, the hash attack is particularly tricky since it is very easy to > argue and very difficult to demonstrate. If you know how to generate colliding hashes, it's actually relatively easy to demonstrate, assuming you know how a particular Web application processes its incoming requests (which you do if it's a standard Web application such as hgweb). Regards Antoine. From status at bugs.python.org Fri Oct 4 18:07:41 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 4 Oct 2013 18:07:41 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20131004160741.79E71560C8@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-09-27 - 2013-10-04) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4256 ( +0) closed 26682 (+58) total 30938 (+58) Open issues with patches: 1960 Issues opened (30) ================== #19066: os.execv fails with spaced names on Windows http://bugs.python.org/issue19066 reopened by techtonik #19111: 2to3 should remove from future_builtins import * http://bugs.python.org/issue19111 opened by maubp #19113: duplicate test names in Lib/ctypes/test/test_functions.py http://bugs.python.org/issue19113 opened by xdegaye #19119: duplicate test name in Lib/test/test_heapq.py http://bugs.python.org/issue19119 opened by xdegaye #19120: shlex.shlex.lineno reports a different number depending on the http://bugs.python.org/issue19120 opened by daniel-s #19121: Documentation guidelines enhancements http://bugs.python.org/issue19121 opened by techtonik #19124: os.execv executes in background on Windows http://bugs.python.org/issue19124 opened by techtonik #19129: 6.2.1. Regular Expression Syntax flags http://bugs.python.org/issue19129 opened by endoalir #19131: Broken support of compressed AIFC files http://bugs.python.org/issue19131 opened by serhiy.storchaka #19133: Transient test failure: test_with_statement (test_ftplib) http://bugs.python.org/issue19133 opened by koobs #19138: doctest.IGNORE_EXCEPTION_DETAIL doesn't match when no detail e http://bugs.python.org/issue19138 opened by jamur2 #19140: inspect.Signature.bind() inaccuracies http://bugs.python.org/issue19140 opened by epsy #19141: Windows Launcher fails to respect PATH http://bugs.python.org/issue19141 opened by gwideman #19142: Cross-compile fails trying to execute foreign pgen on build ho http://bugs.python.org/issue19142 opened by Trevor.Bowen #19143: Finding the Windows version getting messier http://bugs.python.org/issue19143 opened by tim.peters #19145: Inconsistent behaviour in itertools.repeat when using negative http://bugs.python.org/issue19145 opened by vajrasky #19146: Improvements to traceback module http://bugs.python.org/issue19146 opened by gvanrossum #19148: Minor issues with Enum docs http://bugs.python.org/issue19148 opened by Esa.Peuha #19150: IDLE shell fails: "ModifiedInterpreter instance has no attribu http://bugs.python.org/issue19150 opened by Grupobetatesting #19152: ExtensionFileLoader missing get_filename() http://bugs.python.org/issue19152 opened by eric.snow #19153: Embedding into a shared library fails again http://bugs.python.org/issue19153 opened by rinatous #19154: AttributeError: 'NoneType' in http/client.py when using select http://bugs.python.org/issue19154 opened by fviard #19156: Enum helper functions test-coverage http://bugs.python.org/issue19156 opened by CliffM #19157: ipaddress.IPv6Network.hosts function omits network and broadca http://bugs.python.org/issue19157 opened by m01 #19158: BoundedSemaphore.release() subject to races http://bugs.python.org/issue19158 opened by tim.peters #19159: 2to3 incorrectly converts two parameter unicode() constructor http://bugs.python.org/issue19159 opened by gregory.p.smith #19161: collections Counter handles nan strangely http://bugs.python.org/issue19161 opened by Adam.Davison #19164: Update uuid.UUID TypeError exception: integer should not be an http://bugs.python.org/issue19164 opened by makronized #19165: Change formatter warning to DeprecationWarning in 3.5 http://bugs.python.org/issue19165 opened by brett.cannon #19166: Unusued variable in test_keys in Lib/test/test_dict.py http://bugs.python.org/issue19166 opened by vajrasky Most recent 15 issues with no replies (15) ========================================== #19166: Unusued variable in test_keys in Lib/test/test_dict.py http://bugs.python.org/issue19166 #19165: Change formatter warning to DeprecationWarning in 3.5 http://bugs.python.org/issue19165 #19157: ipaddress.IPv6Network.hosts function omits network and broadca http://bugs.python.org/issue19157 #19156: Enum helper functions test-coverage http://bugs.python.org/issue19156 #19154: AttributeError: 'NoneType' in http/client.py when using select http://bugs.python.org/issue19154 #19140: inspect.Signature.bind() inaccuracies http://bugs.python.org/issue19140 #19138: doctest.IGNORE_EXCEPTION_DETAIL doesn't match when no detail e http://bugs.python.org/issue19138 #19133: Transient test failure: test_with_statement (test_ftplib) http://bugs.python.org/issue19133 #19131: Broken support of compressed AIFC files http://bugs.python.org/issue19131 #19129: 6.2.1. Regular Expression Syntax flags http://bugs.python.org/issue19129 #19121: Documentation guidelines enhancements http://bugs.python.org/issue19121 #19113: duplicate test names in Lib/ctypes/test/test_functions.py http://bugs.python.org/issue19113 #19102: Add tests for CLI of the tabnanny module http://bugs.python.org/issue19102 #19100: Use backslashreplace in pprint http://bugs.python.org/issue19100 #19085: Add tkinter basic options tests http://bugs.python.org/issue19085 Most recent 15 issues waiting for review (15) ============================================= #19166: Unusued variable in test_keys in Lib/test/test_dict.py http://bugs.python.org/issue19166 #19157: ipaddress.IPv6Network.hosts function omits network and broadca http://bugs.python.org/issue19157 #19156: Enum helper functions test-coverage http://bugs.python.org/issue19156 #19148: Minor issues with Enum docs http://bugs.python.org/issue19148 #19146: Improvements to traceback module http://bugs.python.org/issue19146 #19145: Inconsistent behaviour in itertools.repeat when using negative http://bugs.python.org/issue19145 #19140: inspect.Signature.bind() inaccuracies http://bugs.python.org/issue19140 #19138: doctest.IGNORE_EXCEPTION_DETAIL doesn't match when no detail e http://bugs.python.org/issue19138 #19131: Broken support of compressed AIFC files http://bugs.python.org/issue19131 #19119: duplicate test name in Lib/test/test_heapq.py http://bugs.python.org/issue19119 #19106: Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots http://bugs.python.org/issue19106 #19105: pprint doesn't use all width http://bugs.python.org/issue19105 #19104: pprint produces invalid output for long strings http://bugs.python.org/issue19104 #19103: Use pprint in displayhook http://bugs.python.org/issue19103 #19102: Add tests for CLI of the tabnanny module http://bugs.python.org/issue19102 Top 10 most discussed issues (10) ================================= #19087: bytearray front-slicing not optimized http://bugs.python.org/issue19087 29 msgs #19106: Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots http://bugs.python.org/issue19106 20 msgs #19066: os.execv fails with spaced names on Windows http://bugs.python.org/issue19066 13 msgs #19108: Benchmark runner tries to execute external Python command and http://bugs.python.org/issue19108 13 msgs #19146: Improvements to traceback module http://bugs.python.org/issue19146 11 msgs #19152: ExtensionFileLoader missing get_filename() http://bugs.python.org/issue19152 10 msgs #19099: struct.pack fails first time with unicode fmt http://bugs.python.org/issue19099 9 msgs #16040: nntplib: unlimited readline() from connection http://bugs.python.org/issue16040 8 msgs #19104: pprint produces invalid output for long strings http://bugs.python.org/issue19104 8 msgs #19124: os.execv executes in background on Windows http://bugs.python.org/issue19124 7 msgs Issues closed (55) ================== #4366: cannot find -lpythonX.X when building Python on FreeBSD with - http://bugs.python.org/issue4366 closed by pitrou #12641: Remove -mno-cygwin from distutils http://bugs.python.org/issue12641 closed by pitrou #17326: Windows build docs still referring to VS 2008 in 3.3 http://bugs.python.org/issue17326 closed by berker.peksag #17839: base64 module should use memoryview http://bugs.python.org/issue17839 closed by python-dev #18037: 2to3 passes through string literal which causes SyntaxError in http://bugs.python.org/issue18037 closed by serhiy.storchaka #18129: Fatal Python error: Cannot recover from stack overflow. http://bugs.python.org/issue18129 closed by benjamin.peterson #18594: C accelerator for collections.Counter is slow http://bugs.python.org/issue18594 closed by rhettinger #18596: enable usage of AddressSanitizer in CPython [PATCH] http://bugs.python.org/issue18596 closed by python-dev #18674: Store weak references in modules_by_index http://bugs.python.org/issue18674 closed by pitrou #18690: memoryview not considered a sequence http://bugs.python.org/issue18690 closed by python-dev #18716: Deprecate the formatter module http://bugs.python.org/issue18716 closed by brett.cannon #18857: urlencode of a None value uses the string 'None' http://bugs.python.org/issue18857 closed by orsenthil #18950: Miscellaneous fixes for the sunau module http://bugs.python.org/issue18950 closed by serhiy.storchaka #18990: Remove root attribute from XMLPullParser http://bugs.python.org/issue18990 closed by python-dev #19010: Make XMLPullParser in ElementTree inherit from XMLParser http://bugs.python.org/issue19010 closed by eli.bendersky #19014: Allow memoryview.cast() for empty views http://bugs.python.org/issue19014 closed by pitrou #19040: Problems with overriding Enum.__new__ http://bugs.python.org/issue19040 closed by ethan.furman #19053: read1() from zipfile returns empty data http://bugs.python.org/issue19053 closed by serhiy.storchaka #19069: Built-in float docstrings are not PEP-8 compatible http://bugs.python.org/issue19069 closed by rhettinger #19078: Allow reversed(memoryview), like memoryview http://bugs.python.org/issue19078 closed by python-dev #19086: Make fsum usable incrementally. http://bugs.python.org/issue19086 closed by rhettinger #19089: Windows: Make Ctrl-D exit key combination cross-platform http://bugs.python.org/issue19089 closed by eric.smith #19091: ast.parse gives wrong position for some Names when non-ascii c http://bugs.python.org/issue19091 closed by terry.reedy #19092: ExitStack.__exit__ incorrectly suppresses exceptions in __exit http://bugs.python.org/issue19092 closed by python-dev #19095: Document SSLSocket.getpeercert always returns None without do_ http://bugs.python.org/issue19095 closed by pitrou #19107: CSV output of benchmark runner tries to write to binary file http://bugs.python.org/issue19107 closed by pitrou #19109: 'TestTCPServer' is not defined http://bugs.python.org/issue19109 closed by vinay.sajip #19110: PEP-0 history link is broken http://bugs.python.org/issue19110 closed by georg.brandl #19112: tests of _TestProcess are not run by the multiprocessing test http://bugs.python.org/issue19112 closed by python-dev #19114: duplicate test names in Lib/distutils/tests/test_cmd.py http://bugs.python.org/issue19114 closed by python-dev #19115: duplicate test names in Lib/lib2to3/tests/test_fixers.py http://bugs.python.org/issue19115 closed by python-dev #19116: duplicate test names in Lib/test/test_complex.py http://bugs.python.org/issue19116 closed by python-dev #19117: duplicate test names in Lib/test/test_dis.py http://bugs.python.org/issue19117 closed by python-dev #19118: duplicate test names in Lib/test/test_ftplib.py http://bugs.python.org/issue19118 closed by python-dev #19122: duplicate test name in Lib/test/test_import.py http://bugs.python.org/issue19122 closed by python-dev #19123: duplicate test name in Lib/test/test_regrtest.py http://bugs.python.org/issue19123 closed by python-dev #19125: duplicate test name in Lib/test/test_smtplib.py http://bugs.python.org/issue19125 closed by python-dev #19126: duplicate test name in Lib/test/test_webbrowser.py http://bugs.python.org/issue19126 closed by python-dev #19127: duplicate test name in Lib/xml/dom/minidom.py http://bugs.python.org/issue19127 closed by python-dev #19128: duplicate test name in Mac/Tools/Doc/setup.py http://bugs.python.org/issue19128 closed by ned.deily #19130: PCbuild/readme.txt not up-to-date http://bugs.python.org/issue19130 closed by pitrou #19132: Add compact mode to pprint http://bugs.python.org/issue19132 closed by serhiy.storchaka #19134: test_inspect.py failed if module _multiprocessing not availabl http://bugs.python.org/issue19134 closed by brett.cannon #19135: Spam http://bugs.python.org/issue19135 closed by berker.peksag #19136: CSV, builtin open(), newline arg. Docs broken again. http://bugs.python.org/issue19136 closed by r.david.murray #19137: pprint wrongly formats set and frozenset subclasses http://bugs.python.org/issue19137 closed by serhiy.storchaka #19139: In venv, __VENV_NAME__ is the prompt, not the name http://bugs.python.org/issue19139 closed by python-dev #19144: test_inspect: failure without threads http://bugs.python.org/issue19144 closed by brett.cannon #19147: pydoc3 fcntl.flock documentation gives wrong man page section http://bugs.python.org/issue19147 closed by ned.deily #19149: python bug report http://bugs.python.org/issue19149 closed by ned.deily #19151: Docstring and WindowsRegistryFinder wrong relative to importli http://bugs.python.org/issue19151 closed by eric.snow #19155: Display stack info with color in pdb (the "w" command) http://bugs.python.org/issue19155 closed by ezio.melotti #19160: Inconsistent size for GIL release in hashlib http://bugs.python.org/issue19160 closed by python-dev #19162: datetime.datetime.min.strftime('%Y') not working http://bugs.python.org/issue19162 closed by r.david.murray #19163: PyString_Format with dict http://bugs.python.org/issue19163 closed by python-dev From raymond.hettinger at gmail.com Fri Oct 4 22:38:05 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 4 Oct 2013 13:38:05 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <523F968A.6090907@stoneleaf.us> References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> Message-ID: On Sep 22, 2013, at 6:16 PM, Ethan Furman wrote: > Are we close to asking for pronouncement? When you're ready, let me know. In the meantime, I conducting usability tests on students in Python classes and researching how well it substitutes for existing solutions for case insensitive dictionaries (the primary use case) and for other existing cases such as dictionaries with unicode normalized keys. If you want to participate in the research, I could also use help looking at what other languages do. Python is not the first language with mappings or to encounter use cases for transforming keys prior to insertion and lookup. I would like to find out what work has already been done on this problem. Another consideration is whether the problem is more general that just dictionaries. Would you want similar functionality in all mapping-like objects (i.e. a persistent dictionaries, os.environ, etc)? Would you want similar functionality for other services (i.e. case-insensitive filenames or other homomorphisms). You can also add to the discussion by trying out your own usability tests on people who haven't been exposed to this thread or the pep. My early results indicate that the API still needs work. * When shown code that uses a TransformDict, students don't seem to be able to deduce what the code does just from the context (this contrasts with something like OrderedDict and Counter where the name says what it does). * When given a description of the mechanics of a TransformDict, they don't seem to be able to figure-out what you would do with it without being given an example. * When given a example of using a TransformDict, they understand the example but don't seem to be able to come-up with other examples other than the one they were just shown. And when shown multiple examples, they can't think of other use cases where they've ever needed this in their own code. * This contrasts with the results when I show something less general like a CaseInsensitiveDict. People seem to get that right away. As you might expect, the generalized solution is harder to wrap your head around than a specific solution with a clear name. * One student asked, "why give regular dicts a key-function like sorted(), min() and max()?" I didn't have a good answer, but I haven't yet had time to read this whole thread. * Another issue is that we're accumulating too many dictionary variants and that is making it difficult to differentiate and choose between them. I haven't found anyone (even in advanced classes with very experienced pythonistas) would knew about all the variations: dict, defaultdict, Mapping, MutableMapping, mapping views, OrderedDict, Counter, ChainMap, andTransformDict. David Beazley on twitter recently proposed that we add a MinDict and MaxDict. There seems to be no shortage of ideas of things that can be done with dictionaries. Besides choosing among the dict variants, there is also confusion about other mapping topics such as 1) when to subclass from dict rather than inherit from MutableMapping, 2) the difference between defaultdict(int) and Counter's use of __missing__ to return zero, and 3) it seems that many experienced users can't even name all the existing methods on dictionaries (they forget clear(), copy(), pop(), popitem(), setdefault(), update() and the fromkeys() classmethod). Overall, my impression at this point is that key transformations are useful, but I'm not sure how to incorporate them without taking Python further away from being a language "that just fits in your head." Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri Oct 4 23:06:15 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 4 Oct 2013 23:06:15 +0200 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> Message-ID: 2013/10/4 Raymond Hettinger : > * Another issue is that we're accumulating too many dictionary > variants and that is making it difficult to differentiate and choose > between them. I haven't found anyone (even in advanced classes > with very experienced pythonistas) would knew about > all the variations: dict, defaultdict, Mapping, MutableMapping, > mapping views, OrderedDict, Counter, ChainMap, andTransformDict. Ok, but none of these classes address use cases described of the PEP 455. If it became hard to choose the best container for an use case, it's maybe a documentation issue. The PEP 455 contains a long list of existing implementations, so it means that these use cases are common (even if the Python stdlib according to the PEP). It's a good thing that Python proposes a standard implementation (efficient, well tested, documented, etc.) to answer to these use cases. I'm not convinced by your usability test. The problem is maybe the name, TransformDict. We may find a more explicit name, like TranformKeyDict or NormalizedKeyMapping. Or we can use names of the Transformers movies: OptimusPrimeDict, BumblebeeMapping, JazzDictionary, etc. (If we cannot find a better name, we may add more specialized classes: KeyInsensitiveDict and IdentiyDict. But I like the idea of using my own "transform" function.) Victor From solipsis at pitrou.net Fri Oct 4 23:14:43 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 4 Oct 2013 23:14:43 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> Message-ID: <20131004231443.55320d61@fsol> Good evening, On Fri, 4 Oct 2013 13:38:05 -0700 Raymond Hettinger wrote: > > You can also add to the discussion by trying out your own usability > tests on people who haven't been exposed to this thread or the pep. I think "usability tests" should be conducted on people who actually have a need for the API. Otherwise they simply don't make sense: if you don't need an API, then you don't have to learn / understand it either. As an example, if you conduct random usability tests about "yield from" (PEP 380, accepted) or single-dispatch generic functions (PEP 443, accepted), you'll probably get a negative outcome, especially on students. Or if you conduct usability tests about the ssl module on someone who's never done any network programming, you'll get the similar kind of negative results. > * When given a description of the mechanics of a TransformDict, > they don't seem to be able to figure-out what you would do with it > without being given an example. Well, the documentation is the place where we give examples. > * When given a example of using a TransformDict, they understand > the example but don't seem to be able to come-up with other examples > other than the one they were just shown. Is it any different for e.g. defaultdict? Because the mechanics are exactly the same: a generic construct which you can specialize for various use cases. > * This contrasts with the results when I show something less general > like a CaseInsensitiveDict. People seem to get that right away. > As you might expect, the generalized solution is harder to wrap > your head around than a specific solution with a clear name. Yet the generic solution is applicable to far many cases than the specialized one. I'm not against adding a CaseInsensitiveDict, but that would be a rather bizarre thing to do given we can add a generic construct that's far more powerful, and not significantly more difficult. > * One student asked, "why give regular dicts a key-function like > sorted(), min() and max()?" I didn't have a good answer, but I > haven't yet had time to read this whole thread. :-) The key answer is: when you want to retain the original key. > * Another issue is that we're accumulating too many dictionary > variants and that is making it difficult to differentiate and choose > between them. It shouldn't be difficult, actually, because it doesn't make sense to "choose" at all. The use cases for OrderedDict, Counter, TransformDict and defaultdict are completely different. > I haven't found anyone (even in advanced classes > with very experienced pythonistas) would knew about > all the variations: dict, defaultdict, Mapping, MutableMapping, > mapping views, OrderedDict, Counter, ChainMap, andTransformDict. Is that actually a problem? > Overall, my impression at this point is that key transformations > are useful, but I'm not sure how to incorporate them without > taking Python further away from being a language "that just fits > in your head." The language fits in your head, but the stdlib doesn't. I don't think it has done so for ages :-) I'm not proposing TransformDict as a builtin, though. Regards Antoine. From raymond.hettinger at gmail.com Fri Oct 4 23:59:34 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 4 Oct 2013 14:59:34 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> Message-ID: On Oct 4, 2013, at 2:06 PM, Victor Stinner wrote: > I'm not convinced by your usability test. You're not the one who needs to be convinced ;-) Please do conduct your own API tests and report back. This is necessary for a new class like TransformDict that was constructed from scratch and proposed for direct admission to the standard library. This contrasts with other tools like OrderedDict, ChainMap, and namedtuple which started their lives outside the standard library where we we able observe their fitness for real problems being solved by real users. None of my consulting client's have anything like a general purpose transforming dict in their utility modules, so we lack the real world experience that informed the design of the other tools in the collections module. To make up for that lack of information, we need to put it in front of users as well as do research into how other languages have tackled the use cases. In short, we need to know whether the API will make sense to people, whether their code will be more readable with a TransformDict, and whether the zoo of dict variants should continue to grow. Right now, I don't know those things. All I have to go on is that I personally think the TransformDict is a cool idea. However, that alone isn't sufficient for accepting the PEP. Raymond ?? in order to get things merged you need to solve not only just your own problem but also realize that the world is bigger than your company and try to solve things in a way where it makes sense for other people, even if primarily it is for your own situation.? -- Linus Torvalds http://www.extremeta.com/2013/09/linus-torvalds-said-linuxcon-kernel-developer-panel/390 -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Sat Oct 5 00:51:02 2013 From: larry at hastings.org (Larry Hastings) Date: Sat, 05 Oct 2013 00:51:02 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: <524DC428.40901@python.org> <524DCBA8.8090305@python.org> Message-ID: <524F4656.20807@hastings.org> On 10/04/2013 11:15 AM, Victor Stinner wrote: > 2013/10/4 Armin Rigo : >> The current hash randomization is >> simply not preventing anything; someone posted long ago a way to >> recover bit-by-bit the hash randomized used by a remote web program in >> Python running on a server. > Oh interesting, is it public? http://events.ccc.de/congress/2012/Fahrplan/events/5152.en.html Quoting the synopsis: We also describe a vulnerability of Python's new randomized hash, allowing an attacker to easily recover the 128-bit secret seed. I found all that while reading this interesting, yet moribund, bug report: http://bugs.python.org/issue14621 I guess there was enough bike shedding that people ran out of steam, or something. It happens. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Oct 5 01:14:52 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 5 Oct 2013 01:14:52 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: <524F4656.20807@hastings.org> References: <524DC428.40901@python.org> <524DCBA8.8090305@python.org> <524F4656.20807@hastings.org> Message-ID: 2013/10/5 Larry Hastings : > On 10/04/2013 11:15 AM, Victor Stinner wrote: > > 2013/10/4 Armin Rigo : > > The current hash randomization is > simply not preventing anything; someone posted long ago a way to > recover bit-by-bit the hash randomized used by a remote web program in > Python running on a server. > > Oh interesting, is it public? > > > http://events.ccc.de/congress/2012/Fahrplan/events/5152.en.html > > Quoting the synopsis: > > We also describe a vulnerability of Python's new randomized hash, allowing > an attacker to easily recover the 128-bit secret seed. SipHash homepage contains a "proof of concept" to compute the secret: https://131002.net/siphash/poc.py But the script is not an exploit on a web server, but a script running locally. It requires for example to know the hash of strings "\0" and "\0\0". I would like to know if it's possible to retrieve such information in practice. And how do you retrieve the whole hash value from an HTTP page? You may retrieve some bits using specific HTTP requests, but not directly the whole hash value. I don't know any web page displaying directly the hash value of a string coming from the user request!? I'm not saying that the hash DoS does not exist, I'm just trying to estimate the risk (compared to other DoS attacks). Changing the default hash function is also risky and has a (well, minor) impact on performances. Victor From victor.stinner at gmail.com Sat Oct 5 01:27:37 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 5 Oct 2013 01:27:37 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: <20131004171332.Horde.DulYqZVHTxdkY3b8IGNSHg2@webmail.df.eu> References: <524DC428.40901@python.org> <524DCBA8.8090305@python.org> <20131004171332.Horde.DulYqZVHTxdkY3b8IGNSHg2@webmail.df.eu> Message-ID: 2013/10/4 : > > Quoting Victor Stinner : > >> I still fail to understand the real impact of a hash DoS compared to >> other kinds of DoS. > > > I think the key question is: how many attacking nodes do you need to > control to effectively make some system deny service. A threat is bigger > if you can do it in 10 requests/s from a single host, instead of needing > 10,000 hosts, each making 1000 request/s. Correct. I know that they are some other "cheap" attacks directly at the network layer. For example, the "spamhaus/CloudFlare" attack which made a lot of noise ("300 Gbit/sec") used a DNS "trick": "The traffic is being generated primarily from DNS amplification attacks. Small requests are sent to DNS servers, generating responses from those servers that are about 50-100 times larger." http://arstechnica.com/security/2013/03/spamhaus-ddos-grows-to-internet-threatening-size/ In this case, you still need many computers to DoS a server (=> DDoS). > With the hash DoS, the threat is that if you manage to fill some dictionary > with colliding keys, then each lookup will take a very long time, and you > might arrange to put many lookups into a single HTTP request. So a single > HTTP request might get very costly CPU-wise. Ok, but why should we invest time to fix this specific DoS wheras there are other DoS like XML bomb? Why not setting a limit on the CPU time in your favorite web framework instead? I don't know the complexity of adding sandbox-like features to a web framework. (It's probably complex because we are discussing how to fix the issue directly in Python :-)) > Whether this is a serious threat or not depends on what other threats > the system being attacked is vulnerable to. Maybe there is something even > simpler, or maybe the hash attack is the only hope of bringing the system > to its knees. Popular DDoS attack are usually the simplest, like flooding the server with ping requests, flooding the DNS server, flooding with HTTP requests which take a lot of time ot process, etc. Using a botnet, you don't care of using an inefficient DoS attack, because your power is the number of zombi. I have no idea of the price of renting a botnet, it's probably expensive (and illegal as well). > IMO, the hash attack is particularly tricky since it is very easy to > argue and very difficult to demonstrate. So it can result in fear > and uncertainty very easily, causing people to overreact just so that > they won't be accused of inactivity. It would be easy to evaluate the risk with a public exploit on a real world application :-) Victor From raymond.hettinger at gmail.com Sat Oct 5 01:41:33 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 4 Oct 2013 16:41:33 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20131004231443.55320d61@fsol> References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <20131004231443.55320d61@fsol> Message-ID: <53D5BCC7-6DFC-4ED5-BE60-D429E53E0AEC@gmail.com> On Oct 4, 2013, at 2:14 PM, Antoine Pitrou wrote: > I think "usability tests" should be conducted on people who actually > have a need for the API. Otherwise they simply don't make sense: if you > don't need an API, then you don't have to learn / understand it either. You're right. Students don't make the best test subjects. It might be nice to present this at a Python meet-up or somesuch. Or some people on this list can present it at work to see how their colleagues do with it. Also, it might be nice to get feedback from existing users of IdentityDicts or CaseInsensitiveDicts to see if they are bothered by the implementation having two underlying dictionaries. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Oct 5 05:45:24 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 5 Oct 2013 13:45:24 +1000 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: <524DC428.40901@python.org> <524DCBA8.8090305@python.org> <20131004171332.Horde.DulYqZVHTxdkY3b8IGNSHg2@webmail.df.eu> Message-ID: <20131005034524.GQ7989@ando> On Sat, Oct 05, 2013 at 01:27:37AM +0200, Victor Stinner wrote: > I have no idea of the price of renting a botnet, it's probably > expensive (and illegal as well). Twelve cents per machine. Cheaper in bulk, and cheaper still for machines outside of the US. For those on a budget, you can get ten thousand zombie machines scattered all over the world for two cents each. http://threatpost.com/how-much-does-botnet-cost-022813/77573 I believe you can also rent a botnet for $2 an hour. -- Steven From ericsnowcurrently at gmail.com Sat Oct 5 07:18:51 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 4 Oct 2013 23:18:51 -0600 Subject: [Python-Dev] PEP 451: ModuleSpec Message-ID: After a few rounds on import-sig PEP 451 is really for general consumption. I also have a patch up now. HTML: http://www.python.org/dev/peps/pep-0451/ implementation: http://bugs.python.org/issue18864 Your comments would be appreciated. -eric ============================================================= PEP: 451 Title: A ModuleSpec Type for the Import System Version: $Revision$ Last-Modified: $Date$ Author: Eric Snow Discussions-To: import-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 8-Aug-2013 Python-Version: 3.4 Post-History: 8-Aug-2013, 28-Aug-2013, 18-Sep-2013, 24-Sep-2013 Resolution: Abstract ======== This PEP proposes to add a new class to importlib.machinery called "ModuleSpec". It will provide all the import-related information used to load a module and will be available without needing to load the module first. Finders will directly provide a module's spec instead of a loader (which they will continue to provide indirectly). The import machinery will be adjusted to take advantage of module specs, including using them to load modules. Terms and Concepts ================== The changes in this proposal are an opportunity to make several existing terms and concepts more clear, whereas currently they are (unfortunately) ambiguous. New concepts are also introduced in this proposal. Finally, it's worth explaining a few other existing terms with which people may not be so familiar. For the sake of context, here is a brief summary of all three groups of terms and concepts. A more detailed explanation of the import system is found at [import_system_docs]_. finder ------ A "finder" is an object that identifies the loader that the import system should use to load a module. Currently this is accomplished by calling the finder's find_module() method, which returns the loader. Finders are strictly responsible for providing the loader, which they do through their find_module() method. The import system then uses that loader to load the module. loader ------ A "loader" is an object that is used to load a module during import. Currently this is done by calling the loader's load_module() method. A loader may also provide APIs for getting information about the modules it can load, as well as about data from sources associated with such a module. Right now loaders (via load_module()) are responsible for certain boilerplate, import-related operations. These are: 1. perform some (module-related) validation; 2. create the module object; 3. set import-related attributes on the module; 4. "register" the module to sys.modules; 5. exec the module; 6. clean up in the event of failure while loading the module. This all takes place during the import system's call to Loader.load_module(). origin ------ This is a new term and concept. The idea of it exists subtly in the import system already, but this proposal makes the concept explicit. "origin" in an import context means the system (or resource within a system) from which a module originates. For the purposes of this proposal, "origin" is also a string which identifies such a resource or system. "origin" is applicable to all modules. For example, the origin for built-in and frozen modules is the interpreter itself. The import system already identifies this origin as "built-in" and "frozen", respectively. This is demonstrated in the following module repr: "". In fact, the module repr is already a relatively reliable, though implicit, indicator of a module's origin. Other modules also indicate their origin through other means, as described in the entry for "location". It is up to the loader to decide on how to interpret and use a module's origin, if at all. location -------- This is a new term. However the concept already exists clearly in the import system, as associated with the ``__file__`` and ``__path__`` attributes of modules, as well as the name/term "path" elsewhere. A "location" is a resource or "place", rather than a system at large, from which a module is loaded. It qualifies as an "origin". Examples of locations include filesystem paths and URLs. A location is identified by the name of the resource, but may not necessarily identify the system to which the resource pertains. In such cases the loader would have to identify the system itself. In contrast to other kinds of module origin, a location cannot be inferred by the loader just by the module name. Instead, the loader must be provided with a string to identify the location, usually by the finder that generates the loader. The loader then uses this information to locate the resource from which it will load the module. In theory you could load the module at a given location under various names. The most common example of locations in the import system are the files from which source and extension modules are loaded. For these modules the location is identified by the string in the ``__file__`` attribute. Although ``__file__`` isn't particularly accurate for some modules (e.g. zipped), it is currently the only way that the import system indicates that a module has a location. A module that has a location may be called "locatable". cache ----- The import system stores compiled modules in the __pycache__ directory as an optimization. This module cache that we use today was provided by PEP 3147. For this proposal, the relevant API for module caching is the ``__cache__`` attribute of modules and the cache_from_source() function in importlib.util. Loaders are responsible for putting modules into the cache (and loading out of the cache). Currently the cache is only used for compiled source modules. However, loaders may take advantage of the module cache for other kinds of modules. package ------- The concept does not change, nor does the term. However, the distinction between modules and packages is mostly superficial. Packages *are* modules. They simply have a ``__path__`` attribute and import may add attributes bound to submodules. The typically perceived difference is a source of confusion. This proposal explicitly de-emphasizes the distinction between packages and modules where it makes sense to do so. Motivation ========== The import system has evolved over the lifetime of Python. In late 2002 PEP 302 introduced standardized import hooks via finders and loaders and sys.meta_path. The importlib module, introduced with Python 3.1, now exposes a pure Python implementation of the APIs described by PEP 302, as well as of the full import system. It is now much easier to understand and extend the import system. While a benefit to the Python community, this greater accessibilty also presents a challenge. As more developers come to understand and customize the import system, any weaknesses in the finder and loader APIs will be more impactful. So the sooner we can address any such weaknesses the import system, the better...and there are a couple we can take care of with this proposal. Firstly, any time the import system needs to save information about a module we end up with more attributes on module objects that are generally only meaningful to the import system. It would be nice to have a per-module namespace in which to put future import-related information and to pass around within the import system. Secondly, there's an API void between finders and loaders that causes undue complexity when encountered. The PEP 420 (namespace packages) implementation had to work around this. The complexity surfaced again during recent efforts on a separate proposal. [ref_files_pep]_ The `finder`_ and `loader`_ sections above detail current responsibility of both. Notably, loaders are not required to provide any of the functionality of their load_module() through other methods. Thus, though the import-related information about a module is likely available without loading the module, it is not otherwise exposed. Furthermore, the requirements assocated with load_module() are common to all loaders and mostly are implemented in exactly the same way. This means every loader has to duplicate the same boilerplate code. importlib.util provides some tools that help with this, but it would be more helpful if the import system simply took charge of these responsibilities. The trouble is that this would limit the degree of customization that load_module() could easily continue to facilitate. More importantly, While a finder *could* provide the information that the loader's load_module() would need, it currently has no consistent way to get it to the loader. This is a gap between finders and loaders which this proposal aims to fill. Finally, when the import system calls a finder's find_module(), the finder makes use of a variety of information about the module that is useful outside the context of the method. Currently the options are limited for persisting that per-module information past the method call, since it only returns the loader. Popular options for this limitation are to store the information in a module-to-info mapping somewhere on the finder itself, or store it on the loader. Unfortunately, loaders are not required to be module-specific. On top of that, some of the useful information finders could provide is common to all finders, so ideally the import system could take care of those details. This is the same gap as before between finders and loaders. As an example of complexity attributable to this flaw, the implementation of namespace packages in Python 3.3 (see PEP 420) added FileFinder.find_loader() because there was no good way for find_module() to provide the namespace search locations. The answer to this gap is a ModuleSpec object that contains the per-module information and takes care of the boilerplate functionality involved with loading the module. Specification ============= The goal is to address the gap between finders and loaders while changing as little of their semantics as possible. Though some functionality and information is moved to the new ModuleSpec type, their behavior should remain the same. However, for the sake of clarity the finder and loader semantics will be explicitly identified. Here is a high-level summary of the changes described by this PEP. More detail is available in later sections. importlib.machinery.ModuleSpec (new) ------------------------------------ A specification for a module's import-system-related state. See the `ModuleSpec`_ section below for a more detailed description. * ModuleSpec(name, loader, \*, origin=None, loader_state=None, is_package=None) Attributes: * name - a string for the name of the module. * loader - the loader to use for loading. * origin - the name of the place from which the module is loaded, e.g. "builtin" for built-in modules and the filename for modules loaded from source. * submodule_search_locations - list of strings for where to find submodules, if a package (None otherwise). * loader_state - a container of extra module-specific data for use during loading. * cached (property) - a string for where the compiled module should be stored. * parent (RO-property) - the name of the package to which the module belongs as a submodule (or None). * has_location (RO-property) - a flag indicating whether or not the module's "origin" attribute refers to a location. Instance Methods: * module_repr() - provide a repr string for the spec'ed module; non-locatable modules will use their origin (e.g. "built-in"). * init_module_attrs(module) - set any of a module's import-related attributes that aren't already set. importlib.util Additions ------------------------ These are ModuleSpec factory functions, meant as a convenience for finders. See the `Factory Functions`_ section below for more detail. * spec_from_file_location(name, location, \*, loader=None, submodule_search_locations=None) - build a spec from file-oriented information and loader APIs. * from_loader(name, loader, \*, origin=None, is_package=None) - build a spec with missing information filled in by using loader APIs. Other API Additions ------------------- * importlib.find_spec(name, path=None) will work exactly the same as importlib.find_loader() (which it replaces), but return a spec instead of a loader. For loaders: * importlib.abc.Loader.exec_module(module) will execute a module in its own namespace. It replaces importlib.abc.Loader.load_module(), taking over its module execution functionality. * importlib.abc.Loader.create_module(spec) (optional) will return the module to use for loading. For modules: * Module objects will have a new attribute: ``__spec__``. API Changes ----------- * InspectLoader.is_package() will become optional. Deprecations ------------ * importlib.abc.MetaPathFinder.find_module() * importlib.abc.PathEntryFinder.find_module() * importlib.abc.PathEntryFinder.find_loader() * importlib.abc.Loader.load_module() * importlib.abc.Loader.module_repr() * The parameters and attributes of the various loaders in importlib.machinery * importlib.util.set_package() * importlib.util.set_loader() * importlib.find_loader() Removals -------- These were introduced prior to Python 3.4's release, so they can simply be removed. * importlib.abc.Loader.init_module_attrs() * importlib.util.module_to_load() Other Changes ------------- * The import system implementation in importlib will be changed to make use of ModuleSpec. * importlib.reload() will make use of ModuleSpec. * A module's import-related attributes (other than ``__spec__``) will no longer be used directly by the import system during that module's import. However, this does not impact use of those attributes (e.g. ``__path__``) when loading other modules (e.g. submodules). * Import-related attributes should no longer be added to modules directly, except by the import system. * The module type's ``__repr__()`` will be a thin wrapper around a pure Python implementation which will leverage ModuleSpec. * The spec for the ``__main__`` module will reflect the appropriate name and origin. Backward-Compatibility ---------------------- * If a finder does not define find_spec(), a spec is derived from the loader returned by find_module(). * PathEntryFinder.find_loader() still takes priority over find_module(). * Loader.load_module() is used if exec_module() is not defined. What Will not Change? --------------------- * The syntax and semantics of the import statement. * Existing finders and loaders will continue to work normally. * The import-related module attributes will still be initialized with the same information. * Finders will still create loaders (now storing them in specs). * Loader.load_module(), if a module defines it, will have all the same requirements and may still be called directly. * Loaders will still be responsible for module data APIs. * importlib.reload() will still overwrite the import-related attributes. Responsibilities ---------------- Here's a quick breakdown of where responsibilities lie after this PEP. finders: * create loader * create spec loaders: * create module (optional) * execute module ModuleSpec: * orchestrate module loading * boilerplate for module loading, including managing sys.modules and setting import-related attributes * create module if loader doesn't * call loader.exec_module(), passing in the module in which to exec * contain all the information the loader needs to exec the module * provide the repr for modules What Will Existing Finders and Loaders Have to Do Differently? ============================================================== Immediately? Nothing. The status quo will be deprecated, but will continue working. However, here are the things that the authors of finders and loaders should change relative to this PEP: * Implement find_spec() on finders. * Implement exec_module() on loaders, if possible. The ModuleSpec factory functions in importlib.util are intended to be helpful for converting existing finders. from_loader() and from_file_location() are both straight-forward utilities in this regard. In the case where loaders already expose methods for creating and preparing modules, ModuleSpec.from_module() may be useful to the corresponding finder. For existing loaders, exec_module() should be a relatively direct conversion from the non-boilerplate portion of load_module(). In some uncommon cases the loader should also implement create_module(). ModuleSpec Users ================ ModuleSpec objects have 3 distinct target audiences: Python itself, import hooks, and normal Python users. Python will use specs in the import machinery, in interpreter startup, and in various standard library modules. Some modules are import-oriented, like pkgutil, and others are not, like pickle and pydoc. In all cases, the full ModuleSpec API will get used. Import hooks (finders and loaders) will make use of the spec in specific ways. First of all, finders may use the spec factory functions in importlib.util to create spec objects. They may also directly adjust the spec attributes after the spec is created. Secondly, the finder may bind additional information to the spec (in finder_extras) for the loader to consume during module creation/execution. Finally, loaders will make use of the attributes on a spec when creating and/or executing a module. Python users will be able to inspect a module's ``__spec__`` to get import-related information about the object. Generally, Python applications and interactive users will not be using the ``ModuleSpec`` factory functions nor any the instance methods. How Loading Will Work ===================== This is an outline of what happens in ModuleSpec's loading functionality:: def load(spec): if not hasattr(spec.loader, 'exec_module'): module = spec.loader.load_module(spec.name) spec.init_module_attrs(module) return sys.modules[spec.name] module = None if hasattr(spec.loader, 'create_module'): module = spec.loader.create_module(spec) if module is None: module = ModuleType(spec.name) spec.init_module_attrs(module) sys.modues[spec.name] = module try: spec.loader.exec_module(module) except BaseException: try: del sys.modules[spec.name] except KeyError: pass raise return sys.modules[spec.name] These steps are exactly what Loader.load_module() is already expected to do. Loaders will thus be simplified since they will only need to implement exec_module(). Note that we must return the module from sys.modules. During loading the module may have replaced itself in sys.modules. Since we don't have a post-import hook API to accommodate the use case, we have to deal with it. However, in the replacement case we do not worry about setting the import-related module attributes on the object. The module writer is on their own if they are doing this. ModuleSpec ========== Attributes ---------- Each of the following names is an attribute on ModuleSpec objects. A value of None indicates "not set". This contrasts with module objects where the attribute simply doesn't exist. Most of the attributes correspond to the import-related attributes of modules. Here is the mapping. The reverse of this mapping is used by ModuleSpec.init_module_attrs(). ========================== ============== On ModuleSpec On Modules ========================== ============== name __name__ loader __loader__ package __package__ origin __file__* cached __cached__*,** submodule_search_locations __path__** loader_state \- has_location \- ========================== ============== | \* Set on the module only if spec.has_location is true. | \*\* Set on the module only if the spec attribute is not None. While package and has_location are read-only properties, the remaining attributes can be replaced after the module spec is created and even after import is complete. This allows for unusual cases where directly modifying the spec is the best option. However, typical use should not involve changing the state of a module's spec. **origin** "origin" is a string for the name of the place from which the module originates. See `origin`_ above. Aside from the informational value, it is also used in module_repr(). In the case of a spec where "has_location" is true, ``__file__`` is set to the value of "origin". For built-in modules "origin" would be set to "built-in". **has_location** As explained in the `location`_ section above, many modules are "locatable", meaning there is a corresponding resource from which the module will be loaded and that resource can be described by a string. In contrast, non-locatable modules can't be loaded in this fashion, e.g. builtin modules and modules dynamically created in code. For these, the name is the only way to access them, so they have an "origin" but not a "location". "has_location" is true if the module is locatable. In that case the spec's origin is used as the location and ``__file__`` is set to spec.origin. If additional location information is required (e.g. zipimport), that information may be stored in spec.loader_state. "has_location" may be implied from the existence of a load_data() method on the loader. Incidently, not all locatable modules will be cachable, but most will. **submodule_search_locations** The list of location strings, typically directory paths, in which to search for submodules. If the module is a package this will be set to a list (even an empty one). Otherwise it is None. The name of the corresponding module attribute, ``__path__``, is relatively ambiguous. Instead of mirroring it, we use a more explicit name that makes the purpose clear. **loader_state** A finder may set loader_state to any value to provide additional data for the loader to use during loading. A value of None is the default and indicates that there is no additional data. Otherwise it can be set to any object, such as a dict, list, or types.SimpleNamespace, containing the relevant extra information. For example, zipimporter could use it to pass the zip archive name to the loader directly, rather than needing to derive it from origin or create a custom loader for each find operation. loader_state is meant for use by the finder and corresponding loader. It is not guaranteed to be a stable resource for any other use. Factory Functions ----------------- **spec_from_file_location(name, location, \*, loader=None, submodule_search_locations=None)** Build a spec from file-oriented information and loader APIs. * "origin" will be set to the location. * "has_location" will be set to True. * "cached" will be set to the result of calling cache_from_source(). * "origin" can be deduced from loader.get_filename() (if "location" is not passed in. * "loader" can be deduced from suffix if the location is a filename. * "submodule_search_locations" can be deduced from loader.is_package() and from os.path.dirname(location) if locatin is a filename. **from_loader(name, loader, \*, origin=None, is_package=None)** Build a spec with missing information filled in by using loader APIs. * "has_location" can be deduced from loader.get_data. * "origin" can be deduced from loader.get_filename(). * "submodule_search_locations" can be deduced from loader.is_package() and from os.path.dirname(location) if locatin is a filename. **spec_from_module(module, loader=None)** Build a spec based on the import-related attributes of an existing module. The spec attributes are set to the corresponding import- related module attributes. See the table in `Attributes`_. Omitted Attributes and Methods ------------------------------ There is no "PathModuleSpec" subclass of ModuleSpec that separates out has_location, cached, and submodule_search_locations. While that might make the separation cleaner, module objects don't have that distinction. ModuleSpec will support both cases equally well. While "is_package" would be a simple additional attribute (aliasing self.submodule_search_locations is not None), it perpetuates the artificial (and mostly erroneous) distinction between modules and packages. Conceivably, a ModuleSpec.load() method could optionally take a list of modules with which to interact instead of sys.modules. That capability is left out of this PEP, but may be pursued separately at some other time, including relative to PEP 406 (import engine). Likewise load() could be leveraged to implement multi-version imports. While interesting, doing so is outside the scope of this proposal. Others: * Add ModuleSpec.submodules (RO-property) - returns possible submodules relative to the spec. * Add ModuleSpec.loaded (RO-property) - the module in sys.module, if any. * Add ModuleSpec.data - a descriptor that wraps the data API of the spec's loader. * Also see [cleaner_reload_support]_. Backward Compatibility ---------------------- ModuleSpec doesn't have any. This would be a different story if Finder.find_module() were to return a module spec instead of loader. In that case, specs would have to act like the loader that would have been returned instead. Doing so would be relatively simple, but is an unnecessary complication. It was part of earlier versions of this PEP. Subclassing ----------- Subclasses of ModuleSpec are allowed, but should not be necessary. Simply setting loader_state or adding functionality to a custom finder or loader will likely be a better fit and should be tried first. However, as long as a subclass still fulfills the requirements of the import system, objects of that type are completely fine as the return value of Finder.find_spec(). Existing Types ============== Module Objects -------------- Other than adding ``__spec__``, none of the import-related module attributes will be changed or deprecated, though some of them could be; any such deprecation can wait until Python 4. A module's spec will not be kept in sync with the corresponding import- related attributes. Though they may differ, in practice they will typically be the same. One notable exception is that case where a module is run as a script by using the ``-m`` flag. In that case ``module.__spec__.name`` will reflect the actual module name while ``module.__name__`` will be ``__main__``. A module's spec is not guaranteed to be identical between two modules with the same name. Likewise there is no guarantee that successive calls to importlib.find_spec() will return the same object or even an equivalent object, though at least the latter is likely. Finders ------- Finders are still responsible for creating the loader. That loader will now be stored in the module spec returned by find_spec() rather than returned directly. As is currently the case without the PEP, if a loader would be costly to create, that loader can be designed to defer the cost until later. **MetaPathFinder.find_spec(name, path=None)** **PathEntryFinder.find_spec(name)** Finders will return ModuleSpec objects when find_spec() is called. This new method replaces find_module() and find_loader() (in the PathEntryFinder case). If a loader does not have find_spec(), find_module() and find_loader() are used instead, for backward-compatibility. Adding yet another similar method to loaders is a case of practicality. find_module() could be changed to return specs instead of loaders. This is tempting because the import APIs have suffered enough, especially considering PathEntryFinder.find_loader() was just added in Python 3.3. However, the extra complexity and a less-than- explicit method name aren't worth it. Loaders ------- **Loader.exec_module(module)** Loaders will have a new method, exec_module(). Its only job is to "exec" the module and consequently populate the module's namespace. It is not responsible for creating or preparing the module object, nor for any cleanup afterward. It has no return value. exec_module() will be used during both loading and reloading. exec_module() should properly handle the case where it is called more than once. For some kinds of modules this may mean raising ImportError every time after the first time the method is called. This is particularly relevant for reloading, where some kinds of modules do not support in-place reloading. **Loader.create_module(spec)** Loaders may also implement create_module() that will return a new module to exec. It may return None to indicate that the default module creation code should be used. One use case, though atypical, for create_module() is to provide a module that is a subclass of the builtin module type. Most loaders will not need to implement create_module(), create_module() should properly handle the case where it is called more than once for the same spec/module. This may include returning None or raising ImportError. .. note:: exec_module() and create_module() should not set any import-related module attributes. The fact that load_module() does is a design flaw that this proposal aims to correct. Other changes: PEP 420 introduced the optional module_repr() loader method to limit the amount of special-casing in the module type's ``__repr__()``. Since this method is part of ModuleSpec, it will be deprecated on loaders. However, if it exists on a loader it will be used exclusively. Loader.init_module_attr() method, added prior to Python 3.4's release , will be removed in favor of the same method on ModuleSpec. However, InspectLoader.is_package() will not be deprecated even though the same information is found on ModuleSpec. ModuleSpec can use it to populate its own is_package if that information is not otherwise available. Still, it will be made optional. One consequence of ModuleSpec is that loader ``__init__`` methods will no longer need to accommodate per-module state. The path-based loaders in importlib take arguments in their ``__init__()`` and have corresponding attributes. However, the need for those values is eliminated by module specs. In addition to executing a module during loading, loaders will still be directly responsible for providing APIs concerning module-related data. Other Changes ============= * The various finders and loaders provided by importlib will be updated to comply with this proposal. * The spec for the ``__main__`` module will reflect how the interpreter was started. For instance, with ``-m`` the spec's name will be that of the run module, while ``__main__.__name__`` will still be "__main__". * We add importlib.find_spec() to mirror importlib.find_loader() (which becomes deprecated). * importlib.reload() is changed to use ModuleSpec.load(). * importlib.reload() will now make use of the per-module import lock. Reference Implementation ======================== A reference implementation will be available at http://bugs.python.org/issue18864. Open Issues ============== \* The impact of this change on pkgutil (and setuptools) needs looking into. It has some generic function-based extensions to PEP 302. These may break if importlib starts wrapping loaders without the tools' knowledge. \* Other modules to look at: runpy (and pythonrun.c), pickle, pydoc, inspect. For instance, pickle should be updated in the ``__main__`` case to look at ``module.__spec__.name``. \* Impact on some kinds of lazy loading modules. [lazy_import_concerns]_ References ========== .. [ref_files_pep] http://mail.python.org/pipermail/import-sig/2013-August/000658.html .. [import_system_docs] http://docs.python.org/3/reference/import.html .. [cleaner_reload_support] https://mail.python.org/pipermail/import-sig/2013-September/000735.html .. [lazy_import_concerns] https://mail.python.org/pipermail/python-dev/2013-August/128129.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From larry at hastings.org Sat Oct 5 08:53:12 2013 From: larry at hastings.org (Larry Hastings) Date: Sat, 05 Oct 2013 08:53:12 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: <524DC428.40901@python.org> <524DCBA8.8090305@python.org> <524F4656.20807@hastings.org> Message-ID: <524FB758.5010806@hastings.org> On 10/05/2013 01:14 AM, Victor Stinner wrote: > And how do you retrieve the whole hash value from an HTTP page? You > may retrieve some bits using specific HTTP requests, but not directly > the whole hash value. I don't know any web page displaying directly > the hash value of a string coming from the user request!? Armin Rigo handwaves his way through an approach here: http://bugs.python.org/issue14621#msg173455 You use a "timing attack" to get the algorithm to "leak" a bit at a time. I have no idea how that actually works, I don't have a background in security, nor a sufficiently devious mindset to work it out for myself. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Sat Oct 5 09:25:25 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 05 Oct 2013 09:25:25 +0200 Subject: [Python-Dev] PEP 451: ModuleSpec In-Reply-To: References: Message-ID: Eric Snow, 05.10.2013 07:18: > After a few rounds on import-sig PEP 451 is really for general > consumption. I also have a patch up now. > > HTML: http://www.python.org/dev/peps/pep-0451/ > implementation: http://bugs.python.org/issue18864 Thanks, I'm happy that this is moving forward. > Your comments would be appreciated. As a quick remark, reading the word "name" made me think a couple of times whether it's the simple name of the module or the fully qualified name. I think it's the FQMN in pretty much all cases, but it would be good to make that clear, maybe by stating it somewhere near the beginning of the PEP (and then doing a quick search+reread to make sure it's really the case throughout the document). Stefan From martin at v.loewis.de Sat Oct 5 14:01:16 2013 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 05 Oct 2013 14:01:16 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: <524DC428.40901@python.org> <524DCBA8.8090305@python.org> <20131004171332.Horde.DulYqZVHTxdkY3b8IGNSHg2@webmail.df.eu> Message-ID: <524FFF8C.5080802@v.loewis.de> Am 05.10.13 01:27, schrieb Victor Stinner: > Ok, but why should we invest time to fix this specific DoS wheras > there are other DoS like XML bomb? That is a question about the very mechanics of free software. "We" don't need to invest time into anything (and you may have noticed that I lately actually don't :-) If you think this is a waste of time, just sit back and watch it evolve - it's Christian's time that may get wasted (and the time of anybody who choses to respond). He is writing a PEP, and the same question can be asked about any feature that goes into Python: Why this feature, and not a different one? FWIW, I personally think that a lot of effort was wasted in micro-optimizing the Unicode implementation :-) If you actually think that changing this aspect of Python is a bad idea, then you do need to get involved actively opposing the PEP. I personally think that this "pluggable hash function" stuff is a bad idea. Changing the hash function is ok as long as it doesn't get dramatically slower. > Why not setting a limit on the CPU > time in your favorite web framework instead? Because that is not implementable, in general, and might harm the service. If you disagree about the non-implementability, please propose a specific technology to limit the CPU consumption *per HTTP request*. It might harm the service because /some/ requests might be eligible to high CPU cost. So put in your sandbox technology a mechanism to white-list specific URLs, or to have the CPU limit depend on the URL that is being requested. > Popular DDoS attack are usually the simplest, like flooding the server > with ping requests, flooding the DNS server, flooding with HTTP > requests which take a lot of time ot process, etc. Using a botnet, you > don't care of using an inefficient DoS attack, because your power is > the number of zombi. > > I have no idea of the price of renting a botnet, it's probably > expensive (and illegal as well). Talking about actual attackers, I think the concern here are script kiddies: people who don't want to invest a lot of money into some illegal activity, but who just learned that they can kill service XYZ if they run this-or-that script - and want to try out whether this actually works. I believe that profesional criminals aren't too interested in DDoS; they buy the botnets to distribute spam. Regards, Martin From guido at python.org Sat Oct 5 17:25:01 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 5 Oct 2013 08:25:01 -0700 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: <524DC428.40901@python.org> <524DCBA8.8090305@python.org> Message-ID: On Thu, Oct 3, 2013 at 11:20 PM, Armin Rigo wrote: > Hi Guido, > > On Thu, Oct 3, 2013 at 10:47 PM, Guido van Rossum > wrote: > > Sounds a bit like some security researchers drumming up business. If you > can > > run the binary, presumably you can also recover the seed by looking in > > /proc, right? Or use ctypes or something. This demonstration seems of > > academic interest only. > > I'll not try to defend the opposite point of view very actively, but > let me just say that, in my opinion, your objection is not valid. It > is broken the same way as a different objection, which would claim > that Python can be made sandbox-safe without caring about the numerous > segfault cases. They are all very obscure for sure; I tried at some > point to list them in Lib/test/crashers. I gave up when people > started deleting the files because they no longer crashed on newer > versions, just because details changed --- but not because the general > crash they explained was in any way fixed... Anyway, my point is that > most segfaults can, given enough effort, be transformed into a single, > well-documented tool to conduct a large class of attacks. > > The hash issue is similar. It should be IMHO either ignored (which is > fine for a huge fraction of users), or seriously fixed by people with > the correctly pessimistic approach. The current hash randomization is > simply not preventing anything; someone posted long ago a way to > recover bit-by-bit the hash randomized used by a remote web program in > Python running on a server. The only benefit of this hash > randomization option (-R) was to say to the press that Python fixed > very quickly the problem when it was mediatized :-/ > > This kind of security issues should never be classified as "academic > interest only". Instead they can be classified as "it will take weeks > / months / years before some crazy man manages to put together a > general attack script, but likely, someone will eventually". > > From this point of view I'm saluting Christian's effort, even if I > prefer to stay far away from this kind of issues myself :-) > I don't want to defend my position too actively either: I'm fine with a new hash function as long as it's either faster, or safer and not slower, and I think that being able to change it at compile time is good enough. However, I do want to object to the view (common among security "purists") that any vulnerability, no matter how hard to exploit (or fix) needs to be treated the same. I consider security work as a race of arms, where the effort spent on defense must be commensurate with the risk of attack. Just because your bike lock can still be busted by a professional bike thief (or picked by an ingenuous MIT student) that doesn't mean it's worthless -- especially if the unpickable lock might be worth (or weigh!) more than the bike. A good defense combines multiple techniques -- e.g. common sense about when and where you park your bike together with a mediocre lock might make the theft risk acceptably low. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Oct 5 17:47:15 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 5 Oct 2013 17:47:15 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? References: <524DC428.40901@python.org> <524DCBA8.8090305@python.org> Message-ID: <20131005174715.741f0cf1@fsol> On Sat, 5 Oct 2013 08:25:01 -0700 Guido van Rossum wrote: > > A good defense combines multiple techniques -- e.g. common sense about when > and where you park your bike together with a mediocre lock might make the > theft risk acceptably low. (just remember to park it under a shed with an appropriate colour) Regards Antoine. From storchaka at gmail.com Sat Oct 5 21:42:09 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 05 Oct 2013 22:42:09 +0300 Subject: [Python-Dev] Optimization Message-ID: Please remember me, what was common decision about CPython-only optimizations which change computation complexity? I.g. constant amortization time of += for byte objects and strings, or linear time of sum() for sequences? From g.brandl at gmx.net Sat Oct 5 22:11:43 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 05 Oct 2013 22:11:43 +0200 Subject: [Python-Dev] Optimization In-Reply-To: References: Message-ID: Am 05.10.2013 21:42, schrieb Serhiy Storchaka: > Please remember me, what was common decision about CPython-only > optimizations which change computation complexity? I.g. constant > amortization time of += for byte objects and strings, or linear time of > sum() for sequences? This appears to be about changeset 499a96611baa: Issue #19087: Improve bytearray allocation in order to allow cheap popping of data at the front (slice deletion). I think the best way to describe the CPython strategy is that we don't like to optimize things that both have an idiomatic solution already (see str.join) and can't be replicated easily in other implementations. cheers, Georg From solipsis at pitrou.net Sat Oct 5 22:19:48 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 5 Oct 2013 22:19:48 +0200 Subject: [Python-Dev] Optimization References: Message-ID: <20131005221948.62aa63fd@fsol> On Sat, 05 Oct 2013 22:11:43 +0200 Georg Brandl wrote: > Am 05.10.2013 21:42, schrieb Serhiy Storchaka: > > Please remember me, what was common decision about CPython-only > > optimizations which change computation complexity? I.g. constant > > amortization time of += for byte objects and strings, or linear time of > > sum() for sequences? > > This appears to be about changeset 499a96611baa: > > Issue #19087: Improve bytearray allocation in order to allow cheap popping of > data at the front (slice deletion). > > I think the best way to describe the CPython strategy is that we don't like to > optimize things that both have an idiomatic solution already (see str.join) and > can't be replicated easily in other implementations. Agreed with this. I'll also point out that algorithmic complexity is only an aspect of performance. For example, providing a fast C implementation of decimal is a game changer even though algorithmic complexity may have remained identical. Regards Antoine. From storchaka at gmail.com Sat Oct 5 23:02:14 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 06 Oct 2013 00:02:14 +0300 Subject: [Python-Dev] Optimization In-Reply-To: References: Message-ID: 05.10.13 23:11, Georg Brandl ???????(??): > Am 05.10.2013 21:42, schrieb Serhiy Storchaka: >> Please remember me, what was common decision about CPython-only >> optimizations which change computation complexity? I.g. constant >> amortization time of += for byte objects and strings, or linear time of >> sum() for sequences? > > This appears to be about changeset 499a96611baa: > > Issue #19087: Improve bytearray allocation in order to allow cheap popping of > data at the front (slice deletion). I just need something to which I can refer in similar situations (there was several similar rejected proposition last months). I'm not sure issue #19087 is in same category as constant time += for strings. But the converse was not proven. From victor.stinner at gmail.com Sat Oct 5 23:08:34 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 5 Oct 2013 23:08:34 +0200 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> Message-ID: 2013/10/4 Raymond Hettinger >: > Please do conduct your own API tests and report back. I don't understand why there is a need to evaluate a "new" API: TransformDict has the same API than dict. The only differences are that the constructor takes an extra mandatory parameter and there is a new getitem() method. All other methods are the same: len(dict), key in dict, dict.get(key), dict[key]=value, etc. > This is necessary for a new class like TransformDict > that was constructed from scratch and proposed for > direct admission to the standard library. The class is not "new": it is based on the API of dict, and existing specialized implementations already used in the wild. But I didn't compare TransformDict to existing specialized implementations... Antoine: do you know if your API is different? Maybe in some minor(?) details like storing the first key or last key. > This contrasts with other tools like OrderedDict, ChainMap, > and namedtuple which started their lives outside the standard > library where we we able observe their fitness for real problems > being solved by real users. Why do you say that TransformDict has no real use case, whereas similar containers are already used since many years in the Python standard library? Extract of the PEP: "Several modules in the standard library use identity lookups for object memoization, for example pickle, json, copy, cProfile, doctest and _threading_local." I didn't check this whole list, but it looks like some modules can directly use TransformDict(id), see for example the copy module. And "case insenstive" dictionaries are also used in various projects according to the PEP. Or do you mean that there are too few projects or they are not popular enough? In Python, email.Message uses a case-insensitive container preserving the original case, except that it does also transform the value. email.Message is used in http.client to parse HTTP headers. Twisted has its own InsensitiveDict type for that (which does not transform the value). http://twistedmatrix.com/documents/current/api/twisted.python.util.InsensitiveDict.html > In short, we need to know whether the API will make sense to people, > whether their code will be more readable with a TransformDict, > and whether the zoo of dict variants should continue to grow. Maybe Antoine can provide us a patch on the Python standard library to reuse TransformDict(id) where it is possible. So we can compare the code before/after to see if it's more readable or not. > ?? in order to get things merged you need to solve not only just your own > problem but also realize that the world is bigger than your company and try > to solve things in a way where it makes sense for other people, even if > primarily it is for your own situation.? -- Linus Torvalds > http://www.extremeta.com/2013/09/linus-torvalds-said-linuxcon-kernel-developer-panel/390 Existing implementations are specific (id() or str.lower()), whereas Antoine's container is generic (transform function). Or do you mean that it's not enough generic? It should do something on the value? Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Oct 5 23:18:07 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 5 Oct 2013 23:18:07 +0200 Subject: [Python-Dev] Optimization References: Message-ID: <20131005231807.4dbb16f2@fsol> On Sun, 06 Oct 2013 00:02:14 +0300 Serhiy Storchaka wrote: > 05.10.13 23:11, Georg Brandl ???????(??): > > Am 05.10.2013 21:42, schrieb Serhiy Storchaka: > >> Please remember me, what was common decision about CPython-only > >> optimizations which change computation complexity? I.g. constant > >> amortization time of += for byte objects and strings, or linear time of > >> sum() for sequences? > > > > This appears to be about changeset 499a96611baa: > > > > Issue #19087: Improve bytearray allocation in order to allow cheap popping of > > data at the front (slice deletion). > > I just need something to which I can refer in similar situations (there > was several similar rejected proposition last months). Sometimes there are no hard rules, and several factors can combine into a decision. Complexity of implementation and existing alternatives are useful guidelines, IMHO. Optimized "+=" for strings has been re-enabled after PEP 393, IIRC. Regards Antoine. From storchaka at gmail.com Sat Oct 5 23:22:58 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 06 Oct 2013 00:22:58 +0300 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> Message-ID: 06.10.13 00:08, Victor Stinner ???????(??): > 2013/10/4 Raymond Hettinger >: > > This contrasts with other tools like OrderedDict, ChainMap, > > and namedtuple which started their lives outside the standard > > library where we we able observe their fitness for real problems > > being solved by real users. > > Why do you say that TransformDict has no real use case, whereas similar > containers are already used since many years in the Python standard > library? Extract of the PEP: > > "Several modules in the standard library use identity lookups for object > memoization, for example pickle, json, copy, cProfile, doctest and > _threading_local." > > I didn't check this whole list, but it looks like some modules can > directly use TransformDict(id), see for example the copy module. Unfortunately the pickle and the copy modules can't use TransformDict(id) because they expose their mappings (with integer id keys) in public API. At least until Python 4.0. There are no so much use cases for IdentityDict in stdlib now, and even less for CaseInsensityDict. And a workaround is simple. I don't know about usage of TransformDict with other transfom function. From victor.stinner at gmail.com Sat Oct 5 23:06:00 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 5 Oct 2013 23:06:00 +0200 Subject: [Python-Dev] Optimization In-Reply-To: References: Message-ID: The str type is immutable, bytearray is not. It's easier to justify optimisations on mutable types, like overallocate lists for example. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sat Oct 5 23:58:54 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 06 Oct 2013 00:58:54 +0300 Subject: [Python-Dev] Optimization In-Reply-To: References: Message-ID: 06.10.13 00:06, Victor Stinner ???????(??): > The str type is immutable, bytearray is not. It's easier to justify > optimisations on mutable types, like overallocate lists for example. We can resize str or bytes if its refcount is 1. And resize is cheap in some circumstances. This optimization is CPython-only. From raymond.hettinger at gmail.com Sun Oct 6 00:35:30 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 5 Oct 2013 15:35:30 -0700 Subject: [Python-Dev] Optimization In-Reply-To: References: Message-ID: On Oct 5, 2013, at 12:42 PM, Serhiy Storchaka wrote: > Please remember me, what was common decision about CPython-only optimizations which change computation complexity? IIRC, it is okay optimize C code for just about anything, but we don't want to alter the pure python code after from idiomatic solutions that work on all of the implementations. For example, there is a string concatenation optimization in the C code, but we still use and advocate str.join() and try not to write code that relies on the optimization. That said, it is nice that less informed users are sometimes saved from an accidental performance trap. Making bytearray's efficiently pop from the left side is dubious. This isn't a common idiom, nor should it be. Even if all the other implementations could model this behavior, it wouldn't be a good idea to have bytearrays have different performance characteristics than strings. Right now, it is not too hard to roughly explain the performance characteristics of the various container objects, but that would be imperiled a bit by having bytearrays differing from strings in ways other than their size. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Oct 6 00:58:14 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 6 Oct 2013 00:58:14 +0200 Subject: [Python-Dev] Optimization References: Message-ID: <20131006005814.6d6d7283@fsol> On Sat, 5 Oct 2013 15:35:30 -0700 Raymond Hettinger wrote: > > Making bytearray's efficiently pop from the left side is dubious. > This isn't a common idiom, nor should it be. Even if all the > other implementations could model this behavior, it wouldn't > be a good idea to have bytearrays have different performance > characteristics than strings. Bytearrays are mutable and strings are immutable, so evidently they will have different performance characteristics. Regards Antoine. From rob.cliffe at btinternet.com Sun Oct 6 00:58:58 2013 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sat, 05 Oct 2013 23:58:58 +0100 Subject: [Python-Dev] Optimization In-Reply-To: References: Message-ID: <525099B2.1050905@btinternet.com> Sorry, I may have missed some earlier relevant parts of this discussion. But you appear to be saying that you don't want to optimize something because it would be hard to explain why it performed better. Eh?? Have I misunderstood? Rob Cliffe On 05/10/2013 23:35, Raymond Hettinger wrote: > > On Oct 5, 2013, at 12:42 PM, Serhiy Storchaka > wrote: > >> Please remember me, what was common decision about CPython-only >> optimizations which change computation complexity? > > IIRC, it is okay optimize C code for just about anything, but we don't > want to alter the pure python code after from idiomatic solutions that > work on all of the implementations. > > For example, there is a string concatenation optimization > in the C code, but we still use and advocate str.join() > and try not to write code that relies on the optimization. > That said, it is nice that less informed users are sometimes > saved from an accidental performance trap. > > Making bytearray's efficiently pop from the left side is dubious. > This isn't a common idiom, nor should it be. Even if all the > other implementations could model this behavior, it wouldn't > be a good idea to have bytearrays have different performance > characteristics than strings. Right now, it is not too hard to > roughly explain the performance characteristics of the various > container objects, but that would be imperiled a bit by having > bytearrays differing from strings in ways other than their size. > > > Raymond > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/rob.cliffe%40btinternet.com > > > No virus found in this message. > Checked by AVG - www.avg.com > Version: 2012.0.2242 / Virus Database: 3222/6225 - Release Date: 10/05/13 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Sun Oct 6 01:02:49 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 06 Oct 2013 01:02:49 +0200 Subject: [Python-Dev] project culture: take responsibility for your commits In-Reply-To: <524ED104.2040202@stoneleaf.us> References: <524ED104.2040202@stoneleaf.us> Message-ID: Ethan, I take your three points and apologise for not making it a pleasant experience for any of us. Regarding the rest: Ethan Furman, 04.10.2013 16:30: > starting a trouble ticket with accusations ... and, in fact, I didn't. > that something was snuck in and done behind peoples' backs Sorry for the snotty comment, but at least I explicitly wrapped this in "..." to indicate that I wasn't entirely serious about it. > offensive, as are continual accusations that those you are working with > simply don't understand. > Add to that constant complaints about writing patches yourself... 1) I was far from being the one who *wanted* the feature in question, but 2) I actually contributed 5 patches over the course of the ticket, none of which was really commented on. That's certainly not a very pleasant experience, nor an indication that people *do* understand. At least, that's my side of the story. Stefan From ncoghlan at gmail.com Sun Oct 6 01:32:30 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 6 Oct 2013 09:32:30 +1000 Subject: [Python-Dev] Optimization In-Reply-To: <20131006005814.6d6d7283@fsol> References: <20131006005814.6d6d7283@fsol> Message-ID: On 6 Oct 2013 08:59, "Antoine Pitrou" wrote: > > On Sat, 5 Oct 2013 15:35:30 -0700 > Raymond Hettinger wrote: > > > > Making bytearray's efficiently pop from the left side is dubious. > > This isn't a common idiom, nor should it be. Even if all the > > other implementations could model this behavior, it wouldn't > > be a good idea to have bytearrays have different performance > > characteristics than strings. > > Bytearrays are mutable and strings are immutable, so evidently they > will have different performance characteristics. I suspect "list" may have been the intended comparison there. "array.array" is another appropriate comparison. Having bytearray operations differ in algorithmic complexity from those two types would be very strange and surprising (particularly if it was CPython specific). Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Oct 6 01:37:56 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 6 Oct 2013 01:37:56 +0200 Subject: [Python-Dev] Optimization In-Reply-To: References: <20131006005814.6d6d7283@fsol> Message-ID: <20131006013756.208e5f32@fsol> On Sun, 6 Oct 2013 09:32:30 +1000 Nick Coghlan wrote: > On 6 Oct 2013 08:59, "Antoine Pitrou" wrote: > > > > On Sat, 5 Oct 2013 15:35:30 -0700 > > Raymond Hettinger wrote: > > > > > > Making bytearray's efficiently pop from the left side is dubious. > > > This isn't a common idiom, nor should it be. Even if all the > > > other implementations could model this behavior, it wouldn't > > > be a good idea to have bytearrays have different performance > > > characteristics than strings. > > > > Bytearrays are mutable and strings are immutable, so evidently they > > will have different performance characteristics. > > I suspect "list" may have been the intended comparison there. "array.array" > is another appropriate comparison. > > Having bytearray operations differ in algorithmic complexity from those two > types would be very strange and surprising (particularly if it was CPython > specific). It's only strange because you don't understand the main use case for bytearrays. They may look like arrays of 8-bit integers but they are really used for buffers, so optimizing for stuff like FIFO operation makes sense. Regards Antoine. From g.brandl at gmx.net Sun Oct 6 08:38:12 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 06 Oct 2013 08:38:12 +0200 Subject: [Python-Dev] Optimization In-Reply-To: <20131006013756.208e5f32@fsol> References: <20131006005814.6d6d7283@fsol> <20131006013756.208e5f32@fsol> Message-ID: Am 06.10.2013 01:37, schrieb Antoine Pitrou: > On Sun, 6 Oct 2013 09:32:30 +1000 > Nick Coghlan wrote: >> On 6 Oct 2013 08:59, "Antoine Pitrou" wrote: >> > >> > On Sat, 5 Oct 2013 15:35:30 -0700 >> > Raymond Hettinger wrote: >> > > >> > > Making bytearray's efficiently pop from the left side is dubious. >> > > This isn't a common idiom, nor should it be. Even if all the >> > > other implementations could model this behavior, it wouldn't >> > > be a good idea to have bytearrays have different performance >> > > characteristics than strings. >> > >> > Bytearrays are mutable and strings are immutable, so evidently they >> > will have different performance characteristics. >> >> I suspect "list" may have been the intended comparison there. "array.array" >> is another appropriate comparison. >> >> Having bytearray operations differ in algorithmic complexity from those two >> types would be very strange and surprising (particularly if it was CPython >> specific). > > It's only strange because you don't understand the main use case for > bytearrays. They may look like arrays of 8-bit integers but they are > really used for buffers, so optimizing for stuff like FIFO operation > makes sense. I agree, that is also what came to my mind when I read the commit message. Georg From storchaka at gmail.com Sun Oct 6 08:56:43 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 06 Oct 2013 09:56:43 +0300 Subject: [Python-Dev] Optimization In-Reply-To: <20131006013756.208e5f32@fsol> References: <20131006005814.6d6d7283@fsol> <20131006013756.208e5f32@fsol> Message-ID: 06.10.13 02:37, Antoine Pitrou ???????(??): > It's only strange because you don't understand the main use case for > bytearrays. They may look like arrays of 8-bit integers but they are > really used for buffers, so optimizing for stuff like FIFO operation > makes sense. Could you please provide several examples of "the main use case for bytearrays"? From stephen at xemacs.org Sun Oct 6 12:37:53 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 06 Oct 2013 19:37:53 +0900 Subject: [Python-Dev] Optimization In-Reply-To: References: <20131006005814.6d6d7283@fsol> Message-ID: <87r4bylm66.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > I suspect "list" may have been the intended comparison there. > "array.array" is another appropriate comparison. Having bytearray > operations differ in algorithmic complexity from those two types > would be very strange and surprising I don't see why. If you really wanted bytearrays to be similar to list or array.array in that way, wouldn't you derive them from one of those types[1] rather than provide a primitive? Having a separate primitive makes it easier to implement optimized performance characteristics. Isn't that why we have several different kinds of containers instead of deriving everything from list? For me, the point about string "+=" being efficient (sometimes) isn't that it is surprising compared to similar types, it's that it is surprising for any immutable sequence type. Footnotes: [1] Or all of them from an ABC. From amer7777 at hotmail.com Sun Oct 6 12:47:37 2013 From: amer7777 at hotmail.com (Amer) Date: Sun, 6 Oct 2013 13:47:37 +0300 Subject: [Python-Dev] Wildcard mask and ANY IP Message-ID: Hello, I would like to thank you for your great effort I have this question: I want to do the following IP address style in python ANY.45.10.ANY ANY.x.y.z I tried something like: 10.45.10.1 0.255.255.0, but it is not accepted I got some errors Any help is appreciated Best regards, Amer From victor.stinner at gmail.com Sun Oct 6 17:32:37 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 6 Oct 2013 17:32:37 +0200 Subject: [Python-Dev] Reduce memory footprint of Python Message-ID: Hi, Slowly, I'm trying to see if it would be possible to reduce the memory footprint of Python using the tracemalloc module. First, I noticed that linecache can allocate more than 2 MB. What do you think of adding a registry of "clear cache" functions? For exemple, re.purge() and linecache.clearcache(). gc.collect() clears free lists. I don't know if gc.collect() should be related to this new registy (clear all caches) or not. The dictionary of interned Unicode strings can be large: up to 1.5 MB (with +30,000 strings). Just the dictionary, excluding size of strings. Is the size normal or not? Using tracemalloc, this dictionary is usually to largest memory block. unittest doesn't look to release memory (the TestCase class) after the execution of a test. test_import.test_module_with_large_stack() creates a large Python module and import it, but it does not unload it. Should I open a separated issue for each idea to track them in the bug tracker, or a global issue? Victor From benjamin at python.org Sun Oct 6 17:41:23 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 6 Oct 2013 11:41:23 -0400 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: References: Message-ID: 2013/10/6 Victor Stinner : > Hi, > > Slowly, I'm trying to see if it would be possible to reduce the memory > footprint of Python using the tracemalloc module. > > First, I noticed that linecache can allocate more than 2 MB. What do > you think of adding a registry of "clear cache" functions? For > exemple, re.purge() and linecache.clearcache(). gc.collect() clears > free lists. I don't know if gc.collect() should be related to this new > registy (clear all caches) or not. What is the usecase for minimizing the memory usage of Python in the middle of a program? > > The dictionary of interned Unicode strings can be large: up to 1.5 MB > (with +30,000 strings). Just the dictionary, excluding size of > strings. Is the size normal or not? Using tracemalloc, this dictionary > is usually to largest memory block. 1.5 MB is sort of a pittance. > > unittest doesn't look to release memory (the TestCase class) after the > execution of a test. Is it important to optimize unittests for memory usage? -- Regards, Benjamin From solipsis at pitrou.net Sun Oct 6 17:45:41 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 6 Oct 2013 17:45:41 +0200 Subject: [Python-Dev] Reduce memory footprint of Python References: Message-ID: <20131006174541.31159fb0@fsol> Hi, On Sun, 6 Oct 2013 17:32:37 +0200 Victor Stinner wrote: > > Slowly, I'm trying to see if it would be possible to reduce the memory > footprint of Python using the tracemalloc module. > > First, I noticed that linecache can allocate more than 2 MB. What do > you think of adding a registry of "clear cache" functions? For > exemple, re.purge() and linecache.clearcache(). gc.collect() clears > free lists. I don't know if gc.collect() should be related to this new > registy (clear all caches) or not. Rather than "clear" the cache, perhaps limit its size? AFAICT, linecache is used in the warnings and traceback modules. > The dictionary of interned Unicode strings can be large: up to 1.5 MB > (with +30,000 strings). Just the dictionary, excluding size of > strings. Is the size normal or not? Using tracemalloc, this dictionary > is usually to largest memory block. You don't tell us what the total memory occupation is. A 1.5MB overhead is not the same on a 10MB total or on a 50MB total. (also, how many modules were loaded? large third-party libraries?) The interned strings dictionary could perhaps be replaced with a set? > unittest doesn't look to release memory (the TestCase class) after the > execution of a test. > > test_import.test_module_with_large_stack() creates a large Python > module and import it, but it does not unload it. That's not really relevant to memory use on production systems, though. Regards Antoine. From g.brandl at gmx.net Sun Oct 6 17:49:36 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 06 Oct 2013 17:49:36 +0200 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: References: Message-ID: <52518690.50101@gmx.net> Am 06.10.2013 17:32, schrieb Victor Stinner: > Hi, > > Slowly, I'm trying to see if it would be possible to reduce the memory > footprint of Python using the tracemalloc module. > > First, I noticed that linecache can allocate more than 2 MB. What do > you think of adding a registry of "clear cache" functions? For > exemple, re.purge() and linecache.clearcache(). gc.collect() clears > free lists. I don't know if gc.collect() should be related to this new > registy (clear all caches) or not. > > The dictionary of interned Unicode strings can be large: up to 1.5 MB > (with +30,000 strings). Just the dictionary, excluding size of > strings. Is the size normal or not? Using tracemalloc, this dictionary > is usually to largest memory block. > > unittest doesn't look to release memory (the TestCase class) after the > execution of a test. > > test_import.test_module_with_large_stack() creates a large Python > module and import it, but it does not unload it. That does not seem very important, except if people execute test_import on every interpreter startup :) > Should I open a separated issue for each idea to track them in the bug > tracker, or a global issue? Since the points you mention here are so diverse, it would be good to have separate issues. cheers, Georg From victor.stinner at gmail.com Sun Oct 6 17:51:07 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 6 Oct 2013 17:51:07 +0200 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: References: Message-ID: 2013/10/6 Benjamin Peterson : > 2013/10/6 Victor Stinner : >> Hi, >> >> Slowly, I'm trying to see if it would be possible to reduce the memory >> footprint of Python using the tracemalloc module. >> >> First, I noticed that linecache can allocate more than 2 MB. What do >> you think of adding a registry of "clear cache" functions? For >> exemple, re.purge() and linecache.clearcache(). gc.collect() clears >> free lists. I don't know if gc.collect() should be related to this new >> registy (clear all caches) or not. > > What is the usecase for minimizing the memory usage of Python in the > middle of a program? If you know that your application uses a lot of memory, it is interesting to sometimes (when the application is idle) try to release some bytes to not use all the system memory. On embedded devices, memory is expensive and very limited. Each byte is important :-) >> unittest doesn't look to release memory (the TestCase class) after the >> execution of a test. > > Is it important to optimize unittests for memory usage? I was surprised that running the whole test suite use more and more memory. I expected something more stable with some short peaks. Reducing the memory footprint is useful for performances (of test suite) and be able to run the test suite on system with low memory (embedded devices). Ezio pointed me the following issues: http://bugs.python.org/issue17534 http://bugs.python.org/issue11798 http://bugs.python.org/issue9815 Victor From benjamin at python.org Sun Oct 6 17:55:33 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 6 Oct 2013 11:55:33 -0400 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: References: Message-ID: 2013/10/6 Victor Stinner : > 2013/10/6 Benjamin Peterson : >> 2013/10/6 Victor Stinner : >>> Hi, >>> >>> Slowly, I'm trying to see if it would be possible to reduce the memory >>> footprint of Python using the tracemalloc module. >>> >>> First, I noticed that linecache can allocate more than 2 MB. What do >>> you think of adding a registry of "clear cache" functions? For >>> exemple, re.purge() and linecache.clearcache(). gc.collect() clears >>> free lists. I don't know if gc.collect() should be related to this new >>> registy (clear all caches) or not. >> >> What is the usecase for minimizing the memory usage of Python in the >> middle of a program? > > If you know that your application uses a lot of memory, it is > interesting to sometimes (when the application is idle) try to release > some bytes to not use all the system memory. On embedded devices, > memory is expensive and very limited. Each byte is important :-) How many embedded systems are running Python? -- Regards, Benjamin From stephen at xemacs.org Sun Oct 6 17:58:01 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 07 Oct 2013 00:58:01 +0900 Subject: [Python-Dev] Wildcard mask and ANY IP In-Reply-To: References: Message-ID: <87li26l7cm.fsf@uwakimon.sk.tsukuba.ac.jp> Amer writes: > Hello, > > I would like to thank you for your great effort > I have this question: > > I want to do the following IP address style in python > > ANY.45.10.ANY > ANY.x.y.z This list is for developing the Python language itself (including the CPython interpreter and the standard library of modules). This kind of question is better addressed to python-list at python.org (also accessible from netnews as the comp.lang.python group). Before you post again to an appropriate channel, your question cannot be given a useful answer as it stands. You need to include the code you are using to parse the addresses, and perhaps some idea of what kind of application you are writing. From g.brandl at gmx.net Sun Oct 6 17:49:36 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 06 Oct 2013 17:49:36 +0200 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: References: Message-ID: <52518690.50101@gmx.net> Am 06.10.2013 17:32, schrieb Victor Stinner: > Hi, > > Slowly, I'm trying to see if it would be possible to reduce the memory > footprint of Python using the tracemalloc module. > > First, I noticed that linecache can allocate more than 2 MB. What do > you think of adding a registry of "clear cache" functions? For > exemple, re.purge() and linecache.clearcache(). gc.collect() clears > free lists. I don't know if gc.collect() should be related to this new > registy (clear all caches) or not. > > The dictionary of interned Unicode strings can be large: up to 1.5 MB > (with +30,000 strings). Just the dictionary, excluding size of > strings. Is the size normal or not? Using tracemalloc, this dictionary > is usually to largest memory block. > > unittest doesn't look to release memory (the TestCase class) after the > execution of a test. > > test_import.test_module_with_large_stack() creates a large Python > module and import it, but it does not unload it. That does not seem very important, except if people execute test_import on every interpreter startup :) > Should I open a separated issue for each idea to track them in the bug > tracker, or a global issue? Since the points you mention here are so diverse, it would be good to have separate issues. cheers, Georg From stefan_ml at behnel.de Sun Oct 6 18:11:19 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 06 Oct 2013 18:11:19 +0200 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: References: Message-ID: Benjamin Peterson, 06.10.2013 17:41: > 2013/10/6 Victor Stinner: >> unittest doesn't look to release memory (the TestCase class) after the >> execution of a test. > > Is it important to optimize unittests for memory usage? Maybe "optimise" isn't the best word, but test suites can sometimes be huge and I can't see a reason to keep an old TestClass instance around after all its tests have run. So why not free it? If the tests do any intermediate caching, it's likely to happen at that level (or at the module level, but that's the user's decision). Sure, if memory matters, there's tearDownClass() for manual cleanup, but why require users to care when we otherwise provide automatic memory management everywhere else? It certainly improves the default setup. Stefan From python-dev at masklinn.net Sun Oct 6 18:15:43 2013 From: python-dev at masklinn.net (Xavier Morel) Date: Sun, 6 Oct 2013 18:15:43 +0200 Subject: [Python-Dev] Optimization In-Reply-To: <87r4bylm66.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20131006005814.6d6d7283@fsol> <87r4bylm66.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <9513A63F-8D86-4563-8209-CEB5862FCC2A@masklinn.net> On 2013-10-06, at 12:37 , Stephen J. Turnbull wrote: > > For me, the point about string "+=" being efficient (sometimes) isn't > that it is surprising compared to similar types, it's that it is > surprising for any immutable sequence type. It's clearly nitpicking, but ropes are immutable sequences with O(1) concatenation. Clojure's vectors also have a more-or-less-O(1) append (technically it's O(log32 n)) and one could implement a Python version of them. From victor.stinner at gmail.com Sun Oct 6 22:22:08 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 6 Oct 2013 22:22:08 +0200 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: <52518690.50101@gmx.net> References: <52518690.50101@gmx.net> Message-ID: Benjamin wrote: > Is it important to optimize unittests for memory usage? 2013/10/6 Georg Brandl : > That does not seem very important, except if people execute test_import > on every interpreter startup :) Oh, I just realized that I forgot to explain why I'm starting with unit tests. I ran the Python test suite using -X tracemalloc (tracing all memory allocations) because the test suite is a nice tool to stress a new module (tracemalloc). The test suite creates many threads, subprocesses, send random signals, etc. That's why I started to monitor the test suite. I know that it's not the most critical part of Python :-) Victor From martin at v.loewis.de Sun Oct 6 22:27:52 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Sun, 06 Oct 2013 22:27:52 +0200 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: References: Message-ID: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> Quoting Benjamin Peterson : >> If you know that your application uses a lot of memory, it is >> interesting to sometimes (when the application is idle) try to release >> some bytes to not use all the system memory. On embedded devices, >> memory is expensive and very limited. Each byte is important :-) > > How many embedded systems are running Python? And of those, how many use the linecache module? Regards, Martin From solipsis at pitrou.net Sun Oct 6 22:40:28 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 6 Oct 2013 22:40:28 +0200 Subject: [Python-Dev] Reduce memory footprint of Python References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> Message-ID: <20131006224028.2c0ecfa2@fsol> On Sun, 06 Oct 2013 22:27:52 +0200 martin at v.loewis.de wrote: > > Quoting Benjamin Peterson : > > >> If you know that your application uses a lot of memory, it is > >> interesting to sometimes (when the application is idle) try to release > >> some bytes to not use all the system memory. On embedded devices, > >> memory is expensive and very limited. Each byte is important :-) > > > > How many embedded systems are running Python? > > And of those, how many use the linecache module? The linecache module is used implicitly by the traceback and warnings module, so perhaps quite a bit more than one would imagine :-) I think limiting the linecache cache size would be good enough. Regards Antoine. From martin at v.loewis.de Sun Oct 6 22:45:09 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Sun, 06 Oct 2013 22:45:09 +0200 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: References: Message-ID: <20131006224509.Horde.rUszeFy3dgPit_89gJUZsw2@webmail.df.eu> Quoting Victor Stinner : > Slowly, I'm trying to see if it would be possible to reduce the memory > footprint of Python using the tracemalloc module. [...] > Should I open a separated issue for each idea to track them in the bug > tracker, or a global issue? There is a third alternative which I would recommend: not open tracker issues at all - unless you can also offer a patch. The things you find are not bugs per se, not even "issues". It is fine and applaudable that you look into this, but other people may have other priorities (like reimplementing the hash function of string objects). So if you remember that there is a potential for optimization, that may be enough for the moment. Or share it on python-dev (as you do below); people may be intrigued to look into this further, or ignore it. It's easy to ignore a posting to python-dev, but more difficult to ignore an issue on the tracker (*something* should be done about it, e.g. close with no action). > First, I noticed that linecache can allocate more than 2 MB. What do > you think of adding a registry of "clear cache" functions? For > exemple, re.purge() and linecache.clearcache(). gc.collect() clears > free lists. I don't know if gc.collect() should be related to this new > registy (clear all caches) or not. I'm -1 on this idea. There are some "canonical" events that could trigger clearance of caches, namely - out-of-memory situations - OS signals indicating memory pressure While these sound interesting in theory, they fail in practice. For example, they are very difficult to test. > The dictionary of interned Unicode strings can be large: up to 1.5 MB > (with +30,000 strings). Just the dictionary, excluding size of > strings. Is the size normal or not? Using tracemalloc, this dictionary > is usually to largest memory block. I'd check the contents of the dictionary. How many strings are in there; how many of these are identifiers; how many have more than one outside reference; how many are immortal? If there is a lot of strings that are not identifiers, some code possibly abuses interning, and should use its own dictionary instead. For the refcount-1 mortal identifiers, I'd trace back where they are stored, and check if many of them originate from the same module. Regards, Martin From martin at v.loewis.de Sun Oct 6 22:47:27 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Sun, 06 Oct 2013 22:47:27 +0200 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: <20131006224028.2c0ecfa2@fsol> References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131006224028.2c0ecfa2@fsol> Message-ID: <20131006224727.Horde.7N5zL7Hhr6Is1kDg47JUNg4@webmail.df.eu> Quoting Antoine Pitrou : > The linecache module is used implicitly by the traceback and warnings > module, so perhaps quite a bit more than one would imagine :-) > > I think limiting the linecache cache size would be good enough. So what specific limit would you suggest? Regards, Martin From solipsis at pitrou.net Sun Oct 6 22:59:13 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 6 Oct 2013 22:59:13 +0200 Subject: [Python-Dev] Reduce memory footprint of Python References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131006224028.2c0ecfa2@fsol> <20131006224727.Horde.7N5zL7Hhr6Is1kDg47JUNg4@webmail.df.eu> Message-ID: <20131006225913.126dbc20@fsol> On Sun, 06 Oct 2013 22:47:27 +0200 martin at v.loewis.de wrote: > > Quoting Antoine Pitrou : > > > The linecache module is used implicitly by the traceback and warnings > > module, so perhaps quite a bit more than one would imagine :-) > > > > I think limiting the linecache cache size would be good enough. > > So what specific limit would you suggest? Something like 50 files. Regards Antoine. From victor.stinner at gmail.com Sun Oct 6 23:26:27 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 6 Oct 2013 23:26:27 +0200 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: <20131006224727.Horde.7N5zL7Hhr6Is1kDg47JUNg4@webmail.df.eu> References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131006224028.2c0ecfa2@fsol> <20131006224727.Horde.7N5zL7Hhr6Is1kDg47JUNg4@webmail.df.eu> Message-ID: 2013/10/6 : > Quoting Antoine Pitrou : > >> The linecache module is used implicitly by the traceback and warnings >> module, so perhaps quite a bit more than one would imagine :-) >> >> I think limiting the linecache cache size would be good enough. > > So what specific limit would you suggest? Before using tracemalloc, I didn't even know that lineache has a cache. And Antoine wrote, linecache is used indirectly by many modules. Even if you get a single line, the module does all the whole file content. I was surprised to see that the cache size can be up to 2 MB. (To answer to your question, yes, our application on embedded device does format exceptions and tracebacks, so use linecache.) The re cache is limited. I remember that I saw it once bigger than 700 KB, it was to build a huge range of Unicode characters... but I'm now unable to reproduce the case. I'm quite sure that the cache has an impact on performances. For example, if you format a traceback where most frames come from the same file, you don't want to decode the file from Unicode for each line. I don't want to remove the cache, and I'm unable to configure its size. I suggested a function to clear "all" caches, because *if* you care of memory, you would like to flush all caches (even if you don't know that linecache or re have caches), not anytime, but when your application is idle. I propose a registry because IMO the module should register itself to the registry, the registry should not have to know all modules. (It should not the module to clear its cache ;-)) I worked on a project spending weeks to try to reduce the memory footprint by 1 MB. And it was hard :-) Biggest objects were in the application but it would help to be able to free Python internal caches. Victor From laurentluce49 at yahoo.com Sun Oct 6 23:43:17 2013 From: laurentluce49 at yahoo.com (Laurent Luce) Date: Sun, 6 Oct 2013 14:43:17 -0700 (PDT) Subject: [Python-Dev] intobject.c free_list and int_dealloc() Message-ID: <1381095797.86186.YahooMailNeo@web162802.mail.bf1.yahoo.com> Hello, I am trying to understand how the free list linked-list evolves when an int object is deallocated. At the beginning, free_list is pointing to the last int object (n-1) in the block.? We initialize two int objects, free_list now points to the int object: n-3. free_list -> n-3 n-1 -> n-2 n-2 -> n-3 n-3 -> n-4... We delete the first int object initialized: n-1 -> n-3 free_list -> n-1 It seems to me that two elements are now pointing to the n-3 element: n-1 and n-2.? I am pretty sure I am missing something here.? Can someone let me know? Thanks, Laurent -------------- next part -------------- An HTML attachment was scrubbed... URL: From olemis at gmail.com Sun Oct 6 23:56:57 2013 From: olemis at gmail.com (Olemis Lang) Date: Sun, 6 Oct 2013 16:56:57 -0500 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: References: Message-ID: On 10/6/13, Benjamin Peterson wrote: > 2013/10/6 Victor Stinner : >> Hi, >> :) [...] >> >> unittest doesn't look to release memory (the TestCase class) after the >> execution of a test. > > Is it important to optimize unittests for memory usage? > AFAICT , test results will stored the outcomes of individual test cases , which is O(n) under normal circumstances, but I guess this is not what Victor was talking about (isn't it ?). However , I've been thinking since some time ago that if the only outcome of running the test suite is to dump the output to stdout or any other file-like object then test result lists might be of little value most of the time . Maybe an efficient runner + results implementation focused on streaming output with no caching could help with efficient memory allocations . Regarding the memory allocated in setUp* method(s) then IMO it should be torn down if it will not be used anymore. Such optimizations should be better performed in TCs tearDown* methods themselves rather than e.g. automatically deallocating memory associated to TC instances . Sometimes different tools use such data for certain analysis . BTW , I've detected a few instances where this proves to be useful (to me) ; especially (concurrently) running (matrix jobs of) relatively long test suites on hosted (Jenkins) instances , where quotas apply . -- Regards, Olemis - @olemislc Apache? Bloodhound contributor http://issues.apache.org/bloodhound http://blood-hound.net Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Popularidad de Python, septiembre 2013 - http://goo.gl/fb/tr0XB From olemis at gmail.com Mon Oct 7 00:13:49 2013 From: olemis at gmail.com (Olemis Lang) Date: Sun, 6 Oct 2013 17:13:49 -0500 Subject: [Python-Dev] Fwd: Reduce memory footprint of Python In-Reply-To: References: Message-ID: forwarding to the list , sorry ... ---------- Forwarded message ---------- From: Olemis Lang Date: Sun, 6 Oct 2013 17:09:38 -0500 Subject: Re: [Python-Dev] Reduce memory footprint of Python To: Benjamin Peterson On 10/6/13, Benjamin Peterson wrote: > 2013/10/6 Victor Stinner : >> 2013/10/6 Benjamin Peterson : >>> 2013/10/6 Victor Stinner : >>>> Hi, >>>> >>>> Slowly, I'm trying to see if it would be possible to reduce the memory >>>> footprint of Python using the tracemalloc module. >>>> >>>> First, I noticed that linecache can allocate more than 2 MB. What do >>>> you think of adding a registry of "clear cache" functions? For >>>> exemple, re.purge() and linecache.clearcache(). gc.collect() clears >>>> free lists. I don't know if gc.collect() should be related to this new >>>> registy (clear all caches) or not. >>> >>> What is the usecase for minimizing the memory usage of Python in the >>> middle of a program? >> >> If you know that your application uses a lot of memory, it is >> interesting to sometimes (when the application is idle) try to release >> some bytes to not use all the system memory. On embedded devices, >> memory is expensive and very limited. Each byte is important :-) > > How many embedded systems are running Python? > I know of a few of them, and it seems they will be more popular with the growing interest for devices like Raspberry Pi and technologies like 3D printing e.g. http://raspberry-python.blogspot.com/ -- Regards, Olemis - @olemislc Apache? Bloodhound contributor http://issues.apache.org/bloodhound http://blood-hound.net Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Popularidad de Python, septiembre 2013 - http://goo.gl/fb/tr0XB From fuzzyman at gmail.com Mon Oct 7 01:13:18 2013 From: fuzzyman at gmail.com (Michael Foord) Date: Mon, 7 Oct 2013 00:13:18 +0100 Subject: [Python-Dev] [Python-checkins] cpython (3.3): PythonCAD is now on PyQt, use Wing as a prominent PyGtk example. In-Reply-To: <3ct1gz21sHz7Ljk@mail.python.org> References: <3ct1gz21sHz7Ljk@mail.python.org> Message-ID: On 6 October 2013 11:45, georg.brandl wrote: > http://hg.python.org/cpython/rev/942b9420e7e9 > changeset: 86065:942b9420e7e9 > branch: 3.3 > parent: 86062:6ced4fb4f711 > user: Georg Brandl > date: Sun Oct 06 12:46:13 2013 +0200 > summary: > PythonCAD is now on PyQt, use Wing as a prominent PyGtk example. > > Wing is only a good example of PyGtk until Wing 5 is out of beta. They too have switched to PyQt... Michael > Found by Helge Stenstr?m on docs at . > > files: > Doc/library/othergui.rst | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Doc/library/othergui.rst b/Doc/library/othergui.rst > --- a/Doc/library/othergui.rst > +++ b/Doc/library/othergui.rst > @@ -20,7 +20,7 @@ > of the library, GTK+ 2. It provides an object oriented interface > that > is slightly higher level than the C one. There are also bindings to > `GNOME `_. One well known PyGTK application > is > - `PythonCAD `_. An online `tutorial > + `WingIDE `_. An online `tutorial > `_ is available. > > `PyQt `_ > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Oct 7 01:49:20 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 7 Oct 2013 01:49:20 +0200 Subject: [Python-Dev] PEP 454: Add a new tracemalloc module (final version) In-Reply-To: References: Message-ID: 2013/10/3 Victor Stinner : > I worked on the implementation of the tracemalloc module and its PEP > 454. I consider that this third version of the PEP is ready for a > final review. > > HTML version of the PEP 454: > http://www.python.org/dev/peps/pep-0454/ About the implementation. I'm working in this repository: http://hg.python.org/features/tracemalloc I updated the patch for Python default on the issue: http://bugs.python.org/issue18874 I backported my new version of the module on Python 2.7 and 3.3: https://github.com/haypo/pytracemalloc Victor From ncoghlan at gmail.com Mon Oct 7 03:36:40 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 Oct 2013 11:36:40 +1000 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: Message-ID: On 4 Oct 2013 07:17, "Guido van Rossum" wrote: > > On Thu, Oct 3, 2013 at 2:13 PM, Nick Coghlan wrote: >> >> On 4 Oct 2013 06:08, "Victor Stinner" wrote: >> > >> > 2013/10/3 Christian Heimes : >> > > A hash algorithm can be added and one avaible hash >> > > algorithm can be set before Py_Initialize() is called for the first >> > > time. >> > >> > "Py_Initialize" is not the good guard. Try for example "python3 -X >> > faulthandler": PyObject_Hash() is called before Py_Initialize() to add >> > "faulthandler" key into sys._xoptions dictionary. >> > >> > Today many Python internal functions are used before Python is >> > initialized... See the PEP 432 which proposes to improve the >> > situation: >> > http://www.python.org/dev/peps/pep-0432/ >> >> That problem exists because our main function doesn't follow the C API usage rules, though. We require other embedding applications to be better behaved than that if they want support :) >> >> That said, while I'm mostly in favour of the PEP, I think setting the algorithm should be a private API for 3.4. >> >> I do agree that since the platform support for SipHash is slightly narrower, we need to keep the existing hash algorithm around, make it relatively easy to enable and ensure we continue to test it on the build bots. >> >> I believe that last requirement for buildbot testing is the one that should drive the design of the private configuration API. > > I'll defer to Nick for approval of this PEP. Sure, I'm happy to be BDFL delegate for this one. Christian, go ahead and add the appropriate header to the PEP. Cheers, Nick. > > -- > --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Oct 7 03:31:34 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 Oct 2013 11:31:34 +1000 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131006224028.2c0ecfa2@fsol> <20131006224727.Horde.7N5zL7Hhr6Is1kDg47JUNg4@webmail.df.eu> Message-ID: On 7 Oct 2013 07:28, "Victor Stinner" wrote: > > 2013/10/6 : > > Quoting Antoine Pitrou : > > > >> The linecache module is used implicitly by the traceback and warnings > >> module, so perhaps quite a bit more than one would imagine :-) > >> > >> I think limiting the linecache cache size would be good enough. > > > > So what specific limit would you suggest? > > Before using tracemalloc, I didn't even know that lineache has a > cache. And Antoine wrote, linecache is used indirectly by many > modules. Even if you get a single line, the module does all the whole > file content. I was surprised to see that the cache size can be up to > 2 MB. > > (To answer to your question, yes, our application on embedded device > does format exceptions and tracebacks, so use linecache.) > > The re cache is limited. I remember that I saw it once bigger than 700 > KB, it was to build a huge range of Unicode characters... but I'm now > unable to reproduce the case. > > I'm quite sure that the cache has an impact on performances. For > example, if you format a traceback where most frames come from the > same file, you don't want to decode the file from Unicode for each > line. I don't want to remove the cache, and I'm unable to configure > its size. > > I suggested a function to clear "all" caches, because *if* you care of > memory, you would like to flush all caches (even if you don't know > that linecache or re have caches), not anytime, but when your > application is idle. I propose a registry because IMO the module > should register itself to the registry, the registry should not have > to know all modules. (It should not the module to clear its cache ;-)) We kind of have such a manual registry already: the reference leak hunting in regrtest needs to know about a lot of these caches so it can clear them. Something like a "gc.purge_caches" operation with a "gc.atpurge" registration function sounds like a reasonable operation to me (especially if it offered a way to get a list of which modules had registered cache purging functions). Like any import time side effects, we'd need to look carefully at the impact it has on the lifecycle of the module globals. We'd also need to be sure that any caches we registered really were pure caches. Cheers, Nick. > > I worked on a project spending weeks to try to reduce the memory > footprint by 1 MB. And it was hard :-) Biggest objects were in the > application but it would help to be able to free Python internal > caches. > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From amk at amk.ca Mon Oct 7 15:54:08 2013 From: amk at amk.ca (A.M. Kuchling) Date: Mon, 7 Oct 2013 09:54:08 -0400 Subject: [Python-Dev] [Python-checkins] cpython (3.3): PythonCAD is now on PyQt, use Wing as a prominent PyGtk example. In-Reply-To: References: <3ct1gz21sHz7Ljk@mail.python.org> Message-ID: <20131007135408.GA3134@datlandrewk.home> On Mon, Oct 07, 2013 at 12:13:18AM +0100, Michael Foord wrote: > Wing is only a good example of PyGtk until Wing 5 is out of beta. They too > have switched to PyQt... Perhaps Gramps is a good example: http://www.gramps-project.org/ I just glanced at their code and it's still using PyGtk. --amk From ericsnowcurrently at gmail.com Mon Oct 7 18:55:57 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 7 Oct 2013 10:55:57 -0600 Subject: [Python-Dev] PEP 451: ModuleSpec In-Reply-To: References: Message-ID: On Sat, Oct 5, 2013 at 1:25 AM, Stefan Behnel wrote: > As a quick remark, reading the word "name" made me think a couple of times > whether it's the simple name of the module or the fully qualified name. I > think it's the FQMN in pretty much all cases, but it would be good to make > that clear, maybe by stating it somewhere near the beginning of the PEP > (and then doing a quick search+reread to make sure it's really the case > throughout the document). It's definitely always meant to be FQMN. I'll make this clear. Thanks for mentioning it. -eric From storchaka at gmail.com Mon Oct 7 19:19:40 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 07 Oct 2013 20:19:40 +0300 Subject: [Python-Dev] [Python-checkins] cpython (3.3): PythonCAD is now on PyQt, use Wing as a prominent PyGtk example. In-Reply-To: <20131007135408.GA3134@datlandrewk.home> References: <3ct1gz21sHz7Ljk@mail.python.org> <20131007135408.GA3134@datlandrewk.home> Message-ID: 07.10.13 16:54, A.M. Kuchling ???????(??): > On Mon, Oct 07, 2013 at 12:13:18AM +0100, Michael Foord wrote: >> Wing is only a good example of PyGtk until Wing 5 is out of beta. They too >> have switched to PyQt... > > Perhaps Gramps is a good example: http://www.gramps-project.org/ I > just glanced at their code and it's still using PyGtk. http://en.wikipedia.org/wiki/Pygtk#Notable_applications_that_use_PyGTK However "PyGTK will be phased out with the transition to GTK+ version 3 and be replaced with PyGObject, which uses GObject introspection to generate bindings for Python and other languages on the fly." From rowen at uw.edu Mon Oct 7 22:01:36 2013 From: rowen at uw.edu (Russell E. Owen) Date: Mon, 07 Oct 2013 13:01:36 -0700 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> Message-ID: In article , Raymond Hettinger wrote: > On Sep 22, 2013, at 6:16 PM, Ethan Furman wrote: > > > Are we close to asking for pronouncement? > > When you're ready, let me know. > > In the meantime, I conducting usability tests on students in Python classes > and researching how well it substitutes for existing solutions for > case insensitive dictionaries (the primary use case) and for other > existing cases such as dictionaries with unicode normalized keys. > > If you want to participate in the research, I could also use help looking > at what other languages do. Python is not the first language with > mappings or to encounter use cases for transforming keys prior > to insertion and lookup. I would like to find out what work has > already been done on this problem. > > Another consideration is whether the problem is more general > that just dictionaries. Would you want similar functionality in > all mapping-like objects (i.e. a persistent dictionaries, os.environ, etc)? > Would you want similar functionality for other services > (i.e. case-insensitive filenames or other homomorphisms). > > You can also add to the discussion by trying out your own usability > tests on people who haven't been exposed to this thread or the pep. > > My early results indicate that the API still needs work. > >... > * Another issue is that we're accumulating too many dictionary > variants and that is making it difficult to differentiate and choose > between them. I haven't found anyone (even in advanced classes > with very experienced pythonistas) would knew about > all the variations: dict, defaultdict, Mapping, MutableMapping, > mapping views, OrderedDict, Counter, ChainMap, andTransformDict. I agree. I personally think being able to transform keys would be much more useful as a property of existing dictionaries. I often use case-insensitive keys. But I use them with dict and OrderedDict (and probably ought to use defaultdict, as well). TransformDict is neat, but I'd personally be happier seeing this as a 3rd party library for now. -- Russell From rymg19 at gmail.com Mon Oct 7 23:10:39 2013 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 7 Oct 2013 16:10:39 -0500 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> Message-ID: 100% disagreement. Err, well, 50%. A property of existing dictionaries is useless. A separate object in, say, collections is more organized. 3rd party libraries can be hard to find, even the great ones. On Mon, Oct 7, 2013 at 3:01 PM, Russell E. Owen wrote: > In article , > Raymond Hettinger wrote: > > > On Sep 22, 2013, at 6:16 PM, Ethan Furman wrote: > > > > > Are we close to asking for pronouncement? > > > > When you're ready, let me know. > > > > In the meantime, I conducting usability tests on students in Python > classes > > and researching how well it substitutes for existing solutions for > > case insensitive dictionaries (the primary use case) and for other > > existing cases such as dictionaries with unicode normalized keys. > > > > If you want to participate in the research, I could also use help looking > > at what other languages do. Python is not the first language with > > mappings or to encounter use cases for transforming keys prior > > to insertion and lookup. I would like to find out what work has > > already been done on this problem. > > > > Another consideration is whether the problem is more general > > that just dictionaries. Would you want similar functionality in > > all mapping-like objects (i.e. a persistent dictionaries, os.environ, > etc)? > > Would you want similar functionality for other services > > (i.e. case-insensitive filenames or other homomorphisms). > > > > You can also add to the discussion by trying out your own usability > > tests on people who haven't been exposed to this thread or the pep. > > > > My early results indicate that the API still needs work. > > > >... > > * Another issue is that we're accumulating too many dictionary > > variants and that is making it difficult to differentiate and choose > > between them. I haven't found anyone (even in advanced classes > > with very experienced pythonistas) would knew about > > all the variations: dict, defaultdict, Mapping, MutableMapping, > > mapping views, OrderedDict, Counter, ChainMap, andTransformDict. > > I agree. > > I personally think being able to transform keys would be much more > useful as a property of existing dictionaries. I often use > case-insensitive keys. But I use them with dict and OrderedDict (and > probably ought to use defaultdict, as well). > > TransformDict is neat, but I'd personally be happier seeing this as a > 3rd party library for now. > > -- Russell > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > -- Ryan -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Oct 7 23:24:31 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 8 Oct 2013 08:24:31 +1100 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> Message-ID: <20131007212430.GU7989@ando> On Fri, Oct 04, 2013 at 11:06:15PM +0200, Victor Stinner wrote: > (If we cannot find a better name, we may add more specialized classes: > KeyInsensitiveDict and IdentiyDict. But I like the idea of using my > own "transform" function.) -1 on a plethora of specialised dicts. I do think that a TransformDict seems useful, and might even *be* useful, but would not like to see a whole pile of specialised dicts added to the std lib. I wonder though, are we going about this the wrong way? Since there is apparently disagreement about TransformDict, that suggests that perhaps we need more concrete experience with the basic idea before graduating to a concrete class in the std lib. Perhaps we should follow the example of dict, __missing__ and defaultdict. The dict class could do something like this on key access: if type(self) is not dict: # This only applies to subclasses, not dict itself. try: transform = type(self).__transform__ except AttributeError: pass else: key = transform(key) # now use the key as usual Am I barking up the wrong tree? Would this slow down dict access too much? -- Steven From ncoghlan at gmail.com Tue Oct 8 00:31:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 8 Oct 2013 08:31:46 +1000 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20131007212430.GU7989@ando> References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <20131007212430.GU7989@ando> Message-ID: On 8 Oct 2013 07:26, "Steven D'Aprano" wrote: > > On Fri, Oct 04, 2013 at 11:06:15PM +0200, Victor Stinner wrote: > > > (If we cannot find a better name, we may add more specialized classes: > > KeyInsensitiveDict and IdentiyDict. But I like the idea of using my > > own "transform" function.) > > -1 on a plethora of specialised dicts. > > I do think that a TransformDict seems useful, and might even *be* > useful, but would not like to see a whole pile of specialised dicts > added to the std lib. > > I wonder though, are we going about this the wrong way? Since there is > apparently disagreement about TransformDict, that suggests that perhaps > we need more concrete experience with the basic idea before graduating > to a concrete class in the std lib. Perhaps we should follow the example > of dict, __missing__ and defaultdict. The dict class could do something > like this on key access: > > if type(self) is not dict: > # This only applies to subclasses, not dict itself. > try: > transform = type(self).__transform__ > except AttributeError: > pass > else: > key = transform(key) > # now use the key as usual > > > Am I barking up the wrong tree? Would this slow down dict access too > much? The problem is doing this in a way that keeps a strong reference to the original key (and produces that when iterating) while doing the lookup based on the transformed keys. That said, with the current plan to lower the barrier to entry for PyPI dependencies (I should have the 3.4 only ensurepip proposal written up some time this week), I think it makes sense to let this one bake on PyPI for a while. I think there *is* a potentially worthwhile generalisation here, but I'm far from sure "is-a-dict" is the right data model - for composability reasons, it feels like a "has-a" relationship with an underlying data store may make more sense. (If performance is critical, you're going to write a dedicated type anyway, so composability and simplicity strike me as more important criteria at this point). Cheers, Nick. > > > -- > Steven > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Oct 7 23:55:44 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 07 Oct 2013 14:55:44 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20131007212430.GU7989@ando> References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <20131007212430.GU7989@ando> Message-ID: <52532DE0.6050908@stoneleaf.us> On 10/07/2013 02:24 PM, Steven D'Aprano wrote: > On Fri, Oct 04, 2013 at 11:06:15PM +0200, Victor Stinner wrote: > > if type(self) is not dict: > # This only applies to subclasses, not dict itself. > try: > transform = type(self).__transform__ > except AttributeError: > pass > else: > key = transform(key) > # now use the key as usual > > > Am I barking up the wrong tree? Would this slow down dict access too > much? Considering that __transform__ would usually not exist, and triggered exceptions are costly, I think it would. From the docs[1]: (10) If a subclass of dict defines a method __missing__, if the key k is not present, the a[k] operation calls that method with the key k as argument. The a[k] operation then returns or raises whatever is returned or raised by the __missing__(k) call if the key is not present. No other operations or methods invoke __missing__(). If __missing__ is not defined, KeyError is raised. __missing__ must be a method; it cannot be an instance variable. For an example, see collections.defaultdict. New in version 2.5. So something more like: transform = getattr(self, '__transform__', None) if transform is not None: key = transform(key) ... A key difference (pun unavoidable ;) between __missing__ and __transform__ is that __missing__ is only called when a key is not found, while __transform__ needs to be called /every/ time a key is looked up: d[k] d.get(k) d.has_key(k) d.fromkeys(...) d.setdefault(...) k in d -- ~Ethan~ From steve at pearwood.info Tue Oct 8 02:23:36 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 8 Oct 2013 11:23:36 +1100 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <52532DE0.6050908@stoneleaf.us> References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <20131007212430.GU7989@ando> <52532DE0.6050908@stoneleaf.us> Message-ID: <20131008002336.GV7989@ando> On Mon, Oct 07, 2013 at 02:55:44PM -0700, Ethan Furman wrote: > On 10/07/2013 02:24 PM, Steven D'Aprano wrote: > >On Fri, Oct 04, 2013 at 11:06:15PM +0200, Victor Stinner wrote: > > > >if type(self) is not dict: > > # This only applies to subclasses, not dict itself. > > try: > > transform = type(self).__transform__ > > except AttributeError: > > pass > > else: > > key = transform(key) > ># now use the key as usual > > > > > >Am I barking up the wrong tree? Would this slow down dict access too > >much? > > Considering that __transform__ would usually not exist, and triggered > exceptions are costly, I think it would. I fear you are right, particularly for subclasses. dict itself would only have the cost of testing whether the instance is an actual dict, which I presume is cheap. Still, given enough "cheap" tests, the overall performance hit could be significant. [...] > So something more like: > > transform = getattr(self, '__transform__', None) > if transform is not None: > key = transform(key) > ... One minor point, being a dunder method, it should be grabbed from the class itself, not the instance: getattr(type(self), ...) > A key difference (pun unavoidable ;) between __missing__ and __transform__ > is that __missing__ is only called when a key is not found, while > __transform__ needs to be called /every/ time a key is looked up: Yes. -- Steven From ethan at stoneleaf.us Tue Oct 8 01:45:58 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 07 Oct 2013 16:45:58 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <52532DE0.6050908@stoneleaf.us> References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <20131007212430.GU7989@ando> <52532DE0.6050908@stoneleaf.us> Message-ID: <525347B6.4010906@stoneleaf.us> On 10/07/2013 02:55 PM, Ethan Furman wrote: > A key difference (pun unavoidable ;) between __missing__ and __transform__ is that __missing__ is only called when a key > is not found, while __transform__ needs to be called /every/ time a key is looked up: > > d[k] > d.get(k) > d.has_key(k) > d.fromkeys(...) > d.setdefault(...) > k in d Oh, and d.pop(k) -- ~Ethan~ From dreamingforward at gmail.com Tue Oct 8 03:17:09 2013 From: dreamingforward at gmail.com (Mark Janssen) Date: Mon, 7 Oct 2013 18:17:09 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913204058.518f43d3@fsol> References: <20130913204058.518f43d3@fsol> Message-ID: Sorry I missed the original discussion, but isn't this a simple case of putting a decorator around the getitem method (to transform the input key) and a single line in the body of the setitem method, making this very easy adaptation of the existing dict class? Mark From steve at pearwood.info Tue Oct 8 05:13:30 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 8 Oct 2013 14:13:30 +1100 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: References: <20130913204058.518f43d3@fsol> Message-ID: <20131008031330.GW7989@ando> On Mon, Oct 07, 2013 at 06:17:09PM -0700, Mark Janssen wrote: > Sorry I missed the original discussion, but isn't this a simple case > of putting a decorator around the getitem method (to transform the > input key) and a single line in the body of the setitem method, making > this very easy adaptation of the existing dict class? Not really. We can try what you suggest to implement a case insensitive dict (decorator is not needed): py> class CaseInsensitiveDict(dict): ... def __getitem__(self, key): ... key = key.casefold() # use .lower() before Python 3.3 ... return super().__getitem__(key) ... def __setitem__(self, key, value): ... key = key.casefold() ... super().__setitem__(key, value) ... py> d = CaseInsensitiveDict() py> d['X'] = 42 py> d {'x': 42} Well, that's not exactly what I was hoping for... I was hoping that the dict would preserve case, rather than just convert it. But that's only the start of the problem: py> d['X'] # this works 42 py> d.pop('X') # expecting 42 Traceback (most recent call last): File "", line 1, in KeyError: 'X' So no, it isn't just a matter of a trivial wrapper around __getitem__, __setitem__ and __delitem__. Check the bug tracker for more detail: http://bugs.python.org/issue18986 -- Steven From solipsis at pitrou.net Tue Oct 8 08:21:43 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 8 Oct 2013 08:21:43 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <20131007212430.GU7989@ando> Message-ID: <20131008082143.4ecfef3f@fsol> On Tue, 8 Oct 2013 08:31:46 +1000 Nick Coghlan wrote: > > That said, with the current plan to lower the barrier to entry for PyPI > dependencies (I should have the 3.4 only ensurepip proposal written up some > time this week), I think it makes sense to let this one bake on PyPI for a > while. "the current plan to lower the barrier to entry for PyPI" sounds a lot like the obsession du jour to me :-) It's not like "ensurepip" makes it cheaper / more attractive to add dependencies. It just provides a better experience for those who *want* to use pip (and would otherwise have installed it using an explicit download). > I think there *is* a potentially worthwhile generalisation here, but I'm > far from sure "is-a-dict" is the right data model - for composability > reasons, it feels like a "has-a" relationship with an underlying data store > may make more sense. It doesn't work. Your "underlying mapping" can show too much variation for the wrapper/proxy to have sane semantics. For example, how do you combine with a defaultdict or a WeakKeyDictionary, knowing that the wrapper/proxy has to have its own internal mapping as well? > (If performance is critical, you're going to write a > dedicated type anyway, so composability and simplicity strike me as more > important criteria at this point). A dedicated CaseInsensitiveDict won't be much faster than TransformDict(str.casefold). Regards Antoine. From solipsis at pitrou.net Tue Oct 8 10:34:45 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 8 Oct 2013 10:34:45 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <20131007212430.GU7989@ando> <20131008082143.4ecfef3f@fsol> Message-ID: <20131008103445.0174baa1@pitrou.net> Le Tue, 8 Oct 2013 08:21:43 +0200, Antoine Pitrou a ?crit : > > > I think there *is* a potentially worthwhile generalisation here, > > but I'm far from sure "is-a-dict" is the right data model - for > > composability reasons, it feels like a "has-a" relationship with an > > underlying data store may make more sense. > > It doesn't work. Your "underlying mapping" can show too much variation > for the wrapper/proxy to have sane semantics. For example, how do you > combine with a defaultdict or a WeakKeyDictionary, knowing that the > wrapper/proxy has to have its own internal mapping as well? (for the record, I'm saying that because I did try the composition route.) Regards Antoine. From solipsis at pitrou.net Tue Oct 8 11:23:59 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 8 Oct 2013 11:23:59 +0200 Subject: [Python-Dev] Reduce memory footprint of Python References: Message-ID: <20131008112359.2f95ca1c@pitrou.net> For the record, if you want to reduce memory footprint of Python on a x86-64 bit, you can use the "x32" ABI for which Debian/Ubuntu provide some support. (install the required base packages and use CFLAGS=-mx32 LDFLAGS=-mx32) Regards Antoine. Le Sun, 6 Oct 2013 17:32:37 +0200, Victor Stinner a ?crit : > Hi, > > Slowly, I'm trying to see if it would be possible to reduce the memory > footprint of Python using the tracemalloc module. > > First, I noticed that linecache can allocate more than 2 MB. What do > you think of adding a registry of "clear cache" functions? For > exemple, re.purge() and linecache.clearcache(). gc.collect() clears > free lists. I don't know if gc.collect() should be related to this new > registy (clear all caches) or not. > > The dictionary of interned Unicode strings can be large: up to 1.5 MB > (with +30,000 strings). Just the dictionary, excluding size of > strings. Is the size normal or not? Using tracemalloc, this dictionary > is usually to largest memory block. > > unittest doesn't look to release memory (the TestCase class) after the > execution of a test. > > test_import.test_module_with_large_stack() creates a large Python > module and import it, but it does not unload it. > > Should I open a separated issue for each idea to track them in the bug > tracker, or a global issue? > > Victor From ncoghlan at gmail.com Tue Oct 8 14:12:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 8 Oct 2013 22:12:02 +1000 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20131008103445.0174baa1@pitrou.net> References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <20131007212430.GU7989@ando> <20131008082143.4ecfef3f@fsol> <20131008103445.0174baa1@pitrou.net> Message-ID: On 8 Oct 2013 18:36, "Antoine Pitrou" wrote: > > Le Tue, 8 Oct 2013 08:21:43 +0200, > Antoine Pitrou a ?crit : > > > > > I think there *is* a potentially worthwhile generalisation here, > > > but I'm far from sure "is-a-dict" is the right data model - for > > > composability reasons, it feels like a "has-a" relationship with an > > > underlying data store may make more sense. > > > > It doesn't work. Your "underlying mapping" can show too much variation > > for the wrapper/proxy to have sane semantics. For example, how do you > > combine with a defaultdict or a WeakKeyDictionary, knowing that the > > wrapper/proxy has to have its own internal mapping as well? > > (for the record, I'm saying that because I did try the composition > route.) It's OK if the key transforming API has to constrain the behaviour of the underlying mapping or require an appropriately designed transform function to handle more esoteric containers. Either would still be better than categorically *disallowing* composition to the point where you can't even compose it with OrderedDict. ChainMap doesn't compose sensibly with arbitrary mappings like defaultdict, but composition is still the right design choice because it works well with *most* custom mappings. It's not that I think this is necessarily a *bad* idea (although the composability problem gives me grave doubts), it's that I think it's not an *urgent* idea, so why rush to review and include it in the weeks remaining before the beta, when the option of publishing it as a recipe or a PyPI module remains available? Enums eventually made it in because we wanted to *use* them to dramatically improve error messages from other stdlib modules. What's the comparable end user payoff for including TransformDict in 3.4 rather than 3.5 (or never)? Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Oct 8 14:30:02 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 8 Oct 2013 14:30:02 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <20131007212430.GU7989@ando> <20131008082143.4ecfef3f@fsol> <20131008103445.0174baa1@pitrou.net> Message-ID: <20131008143002.073d80d1@pitrou.net> Le Tue, 8 Oct 2013 22:12:02 +1000, Nick Coghlan a ?crit : > > It's OK if the key transforming API has to constrain the behaviour of > the underlying mapping or require an appropriately designed transform > function to handle more esoteric containers. Either would still be > better than categorically *disallowing* composition to the point > where you can't even compose it with OrderedDict. Well, you could ask the same question about OrderedDict, defaultdict or Weak*Dictionary since neither of them use composition :-) > ChainMap doesn't > compose sensibly with arbitrary mappings like defaultdict, but > composition is still the right design choice because it works well > with *most* custom mappings. ChainMap is easy to compose since it doesn't have to keep any data-driven internal state. > It's not that I think this is necessarily a *bad* idea (although the > composability problem gives me grave doubts), it's that I think it's > not an *urgent* idea, so why rush to review and include it in the > weeks remaining before the beta, when the option of publishing it as > a recipe or a PyPI module remains available? It's just that I disagree we're rushing anything. The feature is fairly simple, many people have already had a need for something like that. (and amongst those people absolutely *zero* have said the design of the feature proposal is inadequate) Regards Antoine. From ncoghlan at gmail.com Tue Oct 8 14:49:28 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 8 Oct 2013 22:49:28 +1000 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20131008143002.073d80d1@pitrou.net> References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <20131007212430.GU7989@ando> <20131008082143.4ecfef3f@fsol> <20131008103445.0174baa1@pitrou.net> <20131008143002.073d80d1@pitrou.net> Message-ID: On 8 Oct 2013 22:31, "Antoine Pitrou" wrote: > > Le Tue, 8 Oct 2013 22:12:02 +1000, > Nick Coghlan a ?crit : > > > > It's OK if the key transforming API has to constrain the behaviour of > > the underlying mapping or require an appropriately designed transform > > function to handle more esoteric containers. Either would still be > > better than categorically *disallowing* composition to the point > > where you can't even compose it with OrderedDict. > > Well, you could ask the same question about OrderedDict, defaultdict or > Weak*Dictionary since neither of them use composition :-) We *did* ask the same question about those (except the Weak* variants, simply due to age). Each time, the point that each new dict variant would be used to justify yet *more* variants was a cause for concern. defaultdict made it through because it's just a convenience API for the underlying "key missing" protocol, while OrderedDict spent time maturing outside the standard library first. > > > ChainMap doesn't > > compose sensibly with arbitrary mappings like defaultdict, but > > composition is still the right design choice because it works well > > with *most* custom mappings. > > ChainMap is easy to compose since it doesn't have to keep any > data-driven internal state. Yet composing ChainMap with defaultdict still breaks, because it's a conceptually incoherent thing to do. "Composition doesn't work with some mappings" isn't an adequate answer to the criticism that a composition based design could work with more mappings than just the builtin dict. > > It's not that I think this is necessarily a *bad* idea (although the > > composability problem gives me grave doubts), it's that I think it's > > not an *urgent* idea, so why rush to review and include it in the > > weeks remaining before the beta, when the option of publishing it as > > a recipe or a PyPI module remains available? > > It's just that I disagree we're rushing anything. The feature is fairly > simple, many people have already had a need for something like that. > (and amongst those people absolutely *zero* have said the design of the > feature proposal is inadequate) Except the one who wanted to combine it with OrderedDict. Users won't be able to combine it with ChainMap either. The concrete container vs container-wrapper design decision is a fundamental one and I don't believe the current PEP adequately makes the case in favour of the former. Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Tue Oct 8 15:41:11 2013 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 8 Oct 2013 06:41:11 -0700 Subject: [Python-Dev] Semi-official read-only Github mirror of the CPython Mercurial repository In-Reply-To: <145A4E43-27F1-46EF-91BE-383C352669FF@stufft.io> References: <145A4E43-27F1-46EF-91BE-383C352669FF@stufft.io> Message-ID: On Mon, Sep 30, 2013 at 6:54 AM, Donald Stufft wrote: > > On Sep 30, 2013, at 9:09 AM, Eli Bendersky wrote: > > Hi all, > > https://github.com/python/cpython is now live as a semi-official, *read > only* Github mirror of the CPython Mercurial repository. Let me know if you > have any problems/concerns. > > I still haven't decided how often to update it (considering either just N > times a day, or maybe use a Hg hook for batching). Suggestions are welcome. > > The methodology I used to create it is via hg-fast-export. I also tried to > pack and gc the git repo as much as possible before the initial Github push > - it went down from almost ~2GB to ~200MB (so this is the size of a fresh > clone right now). > > Eli > > P.S. thanks Jesse for the keys to https://github.com/python > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io > > > Awesome! I find Github way nicer for reading source than hg.python.org's > web interface, any chance I could convince you to do this for the peps repo > too? ;) > Here you go - https://github.com/python/peps Both mirrors are currently updated twice per night (23:00 and 04:30 US Pacific time). I may switch over to a VPS at some point and have more regular updates. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Oct 8 17:05:17 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 8 Oct 2013 17:05:17 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <20131007212430.GU7989@ando> <20131008082143.4ecfef3f@fsol> <20131008103445.0174baa1@pitrou.net> <20131008143002.073d80d1@pitrou.net> Message-ID: <20131008170517.299caf99@pitrou.net> Le Tue, 8 Oct 2013 22:49:28 +1000, Nick Coghlan a ?crit : > > Well, you could ask the same question about OrderedDict, > > defaultdict or Weak*Dictionary since neither of them use > > composition :-) > > We *did* ask the same question about those (except the Weak* variants, > simply due to age). > > Each time, the point that each new dict variant would be used to > justify yet *more* variants was a cause for concern. Could you explain? I don't understand. > "Composition doesn't work with some mappings" isn't an adequate > answer to the criticism that a composition based design could work > with more mappings than just the builtin dict. Fair enough. Regards Antoine. From rdmurray at bitdance.com Tue Oct 8 17:33:53 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 08 Oct 2013 11:33:53 -0400 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> Message-ID: <20131008153353.841BC2507F3@webabinitio.net> On Sun, 06 Oct 2013 22:27:52 +0200, martin at v.loewis.de wrote: > > Quoting Benjamin Peterson : > > >> If you know that your application uses a lot of memory, it is > >> interesting to sometimes (when the application is idle) try to release > >> some bytes to not use all the system memory. On embedded devices, > >> memory is expensive and very limited. Each byte is important :-) > > > > How many embedded systems are running Python? > > And of those, how many use the linecache module? I can give you one data point: a mobile platform that (currently) uses Python3, and does not use linecache because of how much memory it consumes. As far as the other caches go, every MB counts if you don't have swap, even if you have 2GB of RAM. And especially if you don't :) Like Victor, I spent quite a bit of time helping to trim a MB here and a MB there. I would have loved to have had his tools for doing that work! Trimming linecache was one of the early big wins, even though it made some aspects of development harder. We also cleared most of the other caches after startup (of a long running process). --David PS: I have always thought it sad that the ready availability of memory, CPU speed, and disk space tends to result in lazy programs. I understand there is an effort/value tradeoff, and I make those tradeoffs myself all the time...but it still makes me sad. Then, again, in my early programming days I spent a fair amount of time writing and using Forth, and that probably colors my worldview. :) From guido at python.org Tue Oct 8 18:35:19 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 8 Oct 2013 09:35:19 -0700 Subject: [Python-Dev] On the dangers of giving developers the best resources Message-ID: On Tue, Oct 8, 2013 at 8:33 AM, R. David Murray wrote: > PS: I have always thought it sad that the ready availability of memory, > CPU speed, and disk space tends to result in lazy programs. I understand > there is an effort/value tradeoff, and I make those tradeoffs myself > all the time...but it still makes me sad. Then, again, in my early > programming days I spent a fair amount of time writing and using Forth, > and that probably colors my worldview. :) > I never used or cared for Forth, but I have the same worldview. I remember getting it from David Rosenthal, an early Sun reviewer. He stated that engineers should be given the smallest desktop computer available, not the largest, so they would feel their users' pain and optimize appropriately. Sadly software vendors who are also hardware vendors have incentives going in the opposite direction -- they want users to feel the pain so they'll buy a new device. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From yorik.sar at gmail.com Tue Oct 8 20:02:43 2013 From: yorik.sar at gmail.com (Yuriy Taraday) Date: Tue, 8 Oct 2013 22:02:43 +0400 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913204058.518f43d3@fsol> References: <20130913204058.518f43d3@fsol> Message-ID: On Fri, Sep 13, 2013 at 10:40 PM, Antoine Pitrou wrote: > > Hello, > > Following the python-dev discussion, I've written a PEP to recap the > proposal and the various arguments. It's inlined below, and it will > probably appear soon at http://www.python.org/dev/peps/pep-0455/, too. > > Regards > > Antoine. > Hello. Overall I think that's a great idea. Here are some questions on it though. I'm sorry if some of these have already been discussed in some other thread. 1. Thread safety. PEP doesn't mention anything about thread safety while the implementation proposed in the tracker is (very) not thread-safe. I think, PEP should mention that this class have no guarantees. 2. Extra dict. There should be a way to avoid creation of the second dict when there is no need to store original keys. For example, email.message module doesn't store original headers as they are not needed. The same applies to web frameworks parsing HTTP headers or WSGI environment. I'm sure I had another one. I'll send it once I remember. -- Kind regards, Yuriy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Tue Oct 8 20:17:51 2013 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 08 Oct 2013 19:17:51 +0100 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: References: <20130913204058.518f43d3@fsol> Message-ID: <52544C4F.1070002@mrabarnett.plus.com> On 08/10/2013 19:02, Yuriy Taraday wrote: > On Fri, Sep 13, 2013 at 10:40 PM, Antoine Pitrou > wrote: > > > Hello, > > Following the python-dev discussion, I've written a PEP to recap the > proposal and the various arguments. It's inlined below, and it will > probably appear soon at http://www.python.org/dev/peps/pep-0455/, too. > > Regards > > Antoine. > > > Hello. > > Overall I think that's a great idea. > Here are some questions on it though. I'm sorry if some of these have > already been discussed in some other thread. > > 1. Thread safety. > > PEP doesn't mention anything about thread safety while the > implementation proposed in the tracker is (very) not thread-safe. I > think, PEP should mention that this class have no guarantees. > > 2. Extra dict. > > There should be a way to avoid creation of the second dict when there is > no need to store original keys. For example, email.message module > doesn't store original headers as they are not needed. The same applies > to web frameworks parsing HTTP headers or WSGI environment. > If you don't need the original key, then you might as well just use a transform function with a dict. > I'm sure I had another one. I'll send it once I remember. > From benjamin at python.org Tue Oct 8 20:43:03 2013 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 8 Oct 2013 14:43:03 -0400 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: <20131008153353.841BC2507F3@webabinitio.net> References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> Message-ID: 2013/10/8 R. David Murray : > PS: I have always thought it sad that the ready availability of memory, > CPU speed, and disk space tends to result in lazy programs. I understand > there is an effort/value tradeoff, and I make those tradeoffs myself > all the time...but it still makes me sad. Then, again, in my early > programming days I spent a fair amount of time writing and using Forth, > and that probably colors my worldview. :) In defense of my weak-kneed generation, one of major principles of programming as I understand it is to be as lazy as possible. (That is, have the machine do the work. :)) -- Regards, Benjamin From rdmurray at bitdance.com Tue Oct 8 21:19:48 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 08 Oct 2013 15:19:48 -0400 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: References: <20130913204058.518f43d3@fsol> Message-ID: <20131008191948.BC6A8250164@webabinitio.net> On Tue, 08 Oct 2013 22:02:43 +0400, Yuriy Taraday wrote: > There should be a way to avoid creation of the second dict when there is no > need to store original keys. For example, email.message module doesn't > store original headers as they are not needed. The same applies to web > frameworks parsing HTTP headers or WSGI environment. This is not true. email.message *very carefully* preserves the original header name, case and all [*], that's part of its mandate (faithfully reproducing the original parsed message). That said, email.message can't use transformdict, since email.message needs a list-with-case-insensitive-keyed-lookup, not a dict, because it also preserves the original *order* of the headers. And it couldn't use an orderedtransformdict, either, since it also has to preserve *duplicate* headers. --David [*] currently it loses whitespace information in certain cases, but the new header parser/folder is supposed to fix that...or will once I fix a few more corner cases :) From rdmurray at bitdance.com Tue Oct 8 21:39:18 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 08 Oct 2013 15:39:18 -0400 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> Message-ID: <20131008193918.B3F43250164@webabinitio.net> On Tue, 08 Oct 2013 14:43:03 -0400, Benjamin Peterson wrote: > 2013/10/8 R. David Murray : > > PS: I have always thought it sad that the ready availability of memory, > > CPU speed, and disk space tends to result in lazy programs. I understand > > there is an effort/value tradeoff, and I make those tradeoffs myself > > all the time...but it still makes me sad. Then, again, in my early > > programming days I spent a fair amount of time writing and using Forth, > > and that probably colors my worldview. :) > > In defense of my weak-kneed generation, one of major principles of > programming as I understand it is to be as lazy as possible. (That is, > have the machine do the work. :)) Yeah, that's a commonplace...but the obvious interpretation misses the point, really, which I'm guessing is why you included the smiley. It is not so much a matter of being as lazy as possible, as of being as *smart* as practical[*] up front. Frontload all the work (and it can be a *lot* of work, which is why the aphorism is tongue-in-cheek) into the design and algorithms and tools, then be amazingly productive using them. That's really part of Python's fundamental design philosophy, I think. In this context, if we'd been *really* smart-lazy in CPython development, we'd have kept the memory and startup-time and...well, we probably do pretty well on CPU actually...smaller, so that when smartphones came along Python would have been the first high level language used on them, because it fit. Then we'd all be able to be *much* lazier now :) --David [*] The opposite part of that zen[**] being "if you make the code as smart as possible, you won't be smart enough to debug it". [**] Hmm. I wonder if our "zen" is really a dialectic? From yorik.sar at gmail.com Tue Oct 8 21:36:05 2013 From: yorik.sar at gmail.com (Yuriy Taraday) Date: Tue, 8 Oct 2013 23:36:05 +0400 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <52544C4F.1070002@mrabarnett.plus.com> References: <20130913204058.518f43d3@fsol> <52544C4F.1070002@mrabarnett.plus.com> Message-ID: On Tue, Oct 8, 2013 at 10:17 PM, MRAB wrote: > If you don't need the original key, then you might as well just use a > transform function with a dict. As I understood, storing original keys is not the purpose of TransformDict, allowing hashing on something other then provided key itself is. This part presents interest for the cases I mentioned. -- Kind regards, Yuriy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Tue Oct 8 21:43:40 2013 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 8 Oct 2013 15:43:40 -0400 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: <20131008193918.B3F43250164@webabinitio.net> References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> Message-ID: 2013/10/8 R. David Murray : > In this context, if we'd been *really* smart-lazy in CPython development, > we'd have kept the memory and startup-time and...well, we probably do > pretty well on CPU actually...smaller, so that when smartphones came > along Python would have been the first high level language used on them, > because it fit. Then we'd all be able to be *much* lazier now :) Even on desktop, startup time leaves a lot to be desired. -- Regards, Benjamin From solipsis at pitrou.net Tue Oct 8 21:58:32 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 8 Oct 2013 21:58:32 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <52544C4F.1070002@mrabarnett.plus.com> Message-ID: <20131008215832.1f65ff9f@fsol> On Tue, 8 Oct 2013 23:36:05 +0400 Yuriy Taraday wrote: > On Tue, Oct 8, 2013 at 10:17 PM, MRAB wrote: > > > If you don't need the original key, then you might as well just use a > > transform function with a dict. > > As I understood, storing original keys is not the purpose of TransformDict, > allowing hashing on something other then provided key itself is. This part > presents interest for the cases I mentioned. The purpose is really dual. If we didn't want to retain the original keys, a dedicated class would be much less justified. Regards Antoine. From timothy.c.delaney at gmail.com Tue Oct 8 22:30:40 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 9 Oct 2013 07:30:40 +1100 Subject: [Python-Dev] On the dangers of giving developers the best resources In-Reply-To: References: Message-ID: On 9 October 2013 03:35, Guido van Rossum wrote: > On Tue, Oct 8, 2013 at 8:33 AM, R. David Murray wrote: > >> PS: I have always thought it sad that the ready availability of memory, >> CPU speed, and disk space tends to result in lazy programs. I understand >> there is an effort/value tradeoff, and I make those tradeoffs myself >> all the time...but it still makes me sad. Then, again, in my early >> programming days I spent a fair amount of time writing and using Forth, >> and that probably colors my worldview. :) >> > > I never used or cared for Forth, but I have the same worldview. I remember > getting it from David Rosenthal, an early Sun reviewer. He stated that > engineers should be given the smallest desktop computer available, not the > largest, so they would feel their users' pain and optimize appropriately. > Sadly software vendors who are also hardware vendors have incentives going > in the opposite direction -- they want users to feel the pain so they'll > buy a new device. > I look at it a different way. Developers should be given powerful machines to speed up the development cycle (especially important when prototyping and in the code/run unit test cycle), but everything should be tested on the smallest machine available. It's also a good idea for each developer to have a resource-constrained machine for developer testing/profiling/etc. Virtual machines work quite well for this - you can modify the resource constraints (CPU, memory, etc) to simulate different scenarios. I find that this tends to better promote the methodology of "make it right, then make it fast (small, etc)", which I subscribe to. Optimising too early leads to all your code being complicated, rather than just the bits that need it. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Oct 8 22:38:54 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 8 Oct 2013 13:38:54 -0700 Subject: [Python-Dev] On the dangers of giving developers the best resources In-Reply-To: References: Message-ID: Let's agree to disagree then. I see your methodology used all around me with often problematic results. Maybe devs should have two machines -- one monster that is *only* usable to develop fast, one small that where they run their own apps (email, web browser etc.). On Tue, Oct 8, 2013 at 1:30 PM, Tim Delaney wrote: > On 9 October 2013 03:35, Guido van Rossum wrote: > >> On Tue, Oct 8, 2013 at 8:33 AM, R. David Murray wrote: >> >>> PS: I have always thought it sad that the ready availability of memory, >>> CPU speed, and disk space tends to result in lazy programs. I understand >>> there is an effort/value tradeoff, and I make those tradeoffs myself >>> all the time...but it still makes me sad. Then, again, in my early >>> programming days I spent a fair amount of time writing and using Forth, >>> and that probably colors my worldview. :) >>> >> >> I never used or cared for Forth, but I have the same worldview. I >> remember getting it from David Rosenthal, an early Sun reviewer. He stated >> that engineers should be given the smallest desktop computer available, not >> the largest, so they would feel their users' pain and optimize >> appropriately. Sadly software vendors who are also hardware vendors have >> incentives going in the opposite direction -- they want users to feel the >> pain so they'll buy a new device. >> > > I look at it a different way. Developers should be given powerful machines > to speed up the development cycle (especially important when prototyping > and in the code/run unit test cycle), but everything should be tested on > the smallest machine available. > > It's also a good idea for each developer to have a resource-constrained > machine for developer testing/profiling/etc. Virtual machines work quite > well for this - you can modify the resource constraints (CPU, memory, etc) > to simulate different scenarios. > > I find that this tends to better promote the methodology of "make it > right, then make it fast (small, etc)", which I subscribe to. Optimising > too early leads to all your code being complicated, rather than just the > bits that need it. > > Tim Delaney > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Tue Oct 8 23:29:11 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 9 Oct 2013 08:29:11 +1100 Subject: [Python-Dev] On the dangers of giving developers the best resources In-Reply-To: References: Message-ID: On 9 October 2013 07:38, Guido van Rossum wrote: > Let's agree to disagree then. I see your methodology used all around me > with often problematic results. Maybe devs should have two machines -- one > monster that is *only* usable to develop fast, one small that where they > run their own apps (email, web browser etc.). > I've done that before too - it works quite well (especially if you set them up to use a single keyboard/mouse). I suspect the main determination of whether a fast machine as the primary development machine works better depends heavily on the developer and what their background is. I've also worked in resource-constrained environments, so I'm always considering the impact of my choices, even when I go for the less complicated option initially. I've also been fortunate to mainly work in places where software development was considered a craft, with pride in what we produced. However, I think I should probably reconsider my viewpoint in light of my current employment ... I despair at some of the code I see ... Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Oct 8 23:29:02 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 08 Oct 2013 14:29:02 -0700 Subject: [Python-Dev] inspect() and dir() Message-ID: <5254791E.1020803@stoneleaf.us> Greetings, Currently, inspect() is dependent on dir(). Now that we can override what dir() returns on a class by class basis, we are seeing the side-effect of (possibly) incomplete inspect results, with also leads to (possibly) incomplete help(). I would think we want inspect to be more thorough, and return whatever it finds on the object, as well as whatever an abbreviated dir() might return. Thoughts? -- ~Ethan~ From dholth at gmail.com Tue Oct 8 23:57:14 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 8 Oct 2013 17:57:14 -0400 Subject: [Python-Dev] On the dangers of giving developers the best resources In-Reply-To: References: Message-ID: Sounds like you are suggesting we get a raspberry pi. All sorts of dumb waste shows up when you run code on those. El oct 8, 2013 4:46 p.m., "Guido van Rossum" escribi?: > Let's agree to disagree then. I see your methodology used all around me > with often problematic results. Maybe devs should have two machines -- one > monster that is *only* usable to develop fast, one small that where they > run their own apps (email, web browser etc.). > > > On Tue, Oct 8, 2013 at 1:30 PM, Tim Delaney wrote: > >> On 9 October 2013 03:35, Guido van Rossum wrote: >> >>> On Tue, Oct 8, 2013 at 8:33 AM, R. David Murray wrote: >>> >>>> PS: I have always thought it sad that the ready availability of memory, >>>> CPU speed, and disk space tends to result in lazy programs. I >>>> understand >>>> there is an effort/value tradeoff, and I make those tradeoffs myself >>>> all the time...but it still makes me sad. Then, again, in my early >>>> programming days I spent a fair amount of time writing and using Forth, >>>> and that probably colors my worldview. :) >>>> >>> >>> I never used or cared for Forth, but I have the same worldview. I >>> remember getting it from David Rosenthal, an early Sun reviewer. He stated >>> that engineers should be given the smallest desktop computer available, not >>> the largest, so they would feel their users' pain and optimize >>> appropriately. Sadly software vendors who are also hardware vendors have >>> incentives going in the opposite direction -- they want users to feel the >>> pain so they'll buy a new device. >>> >> >> I look at it a different way. Developers should be given powerful >> machines to speed up the development cycle (especially important when >> prototyping and in the code/run unit test cycle), but everything should be >> tested on the smallest machine available. >> >> It's also a good idea for each developer to have a resource-constrained >> machine for developer testing/profiling/etc. Virtual machines work quite >> well for this - you can modify the resource constraints (CPU, memory, etc) >> to simulate different scenarios. >> >> I find that this tends to better promote the methodology of "make it >> right, then make it fast (small, etc)", which I subscribe to. Optimising >> too early leads to all your code being complicated, rather than just the >> bits that need it. >> >> Tim Delaney >> > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Oct 9 00:01:01 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 09 Oct 2013 11:01:01 +1300 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: <20131008153353.841BC2507F3@webabinitio.net> References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> Message-ID: <5254809D.4080605@canterbury.ac.nz> R. David Murray wrote: > I can give you one data point: a mobile platform that (currently) uses > Python3, and does not use linecache because of how much memory it > consumes. Wouldn't a sensible approach be to discard the linecache when you've finished generating a traceback? You're not likely to be generating tracebacks often enough to be worth keeping it from one to the next. -- Greg From guido at python.org Wed Oct 9 00:07:29 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 8 Oct 2013 15:07:29 -0700 Subject: [Python-Dev] inspect() and dir() In-Reply-To: <5254791E.1020803@stoneleaf.us> References: <5254791E.1020803@stoneleaf.us> Message-ID: I'd say that the effect overriding dir() has on help() is intentional. Not sure about inspect -- it has other uses. However, the last time I used it I was definitely hunting for stuff to document. On Tue, Oct 8, 2013 at 2:29 PM, Ethan Furman wrote: > Greetings, > > Currently, inspect() is dependent on dir(). > > Now that we can override what dir() returns on a class by class basis, we > are seeing the side-effect of (possibly) incomplete inspect results, with > also leads to (possibly) incomplete help(). > > I would think we want inspect to be more thorough, and return whatever it > finds on the object, as well as whatever an abbreviated dir() might return. > > Thoughts? > > -- > ~Ethan~ > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/**mailman/options/python-dev/** > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Oct 9 00:10:05 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 8 Oct 2013 15:10:05 -0700 Subject: [Python-Dev] On the dangers of giving developers the best resources In-Reply-To: References: Message-ID: It's not actually so much the extreme waste that I'm looking to expose, but rather the day-to-day annoyances of stuff you use regularly that slows you down by just a second (or ten), or things that gets slower at each release. On Tue, Oct 8, 2013 at 2:57 PM, Daniel Holth wrote: > Sounds like you are suggesting we get a raspberry pi. All sorts of dumb > waste shows up when you run code on those. > El oct 8, 2013 4:46 p.m., "Guido van Rossum" escribi?: > >> Let's agree to disagree then. I see your methodology used all around me >> with often problematic results. Maybe devs should have two machines -- one >> monster that is *only* usable to develop fast, one small that where they >> run their own apps (email, web browser etc.). >> >> >> On Tue, Oct 8, 2013 at 1:30 PM, Tim Delaney wrote: >> >>> On 9 October 2013 03:35, Guido van Rossum wrote: >>> >>>> On Tue, Oct 8, 2013 at 8:33 AM, R. David Murray wrote: >>>> >>>>> PS: I have always thought it sad that the ready availability of memory, >>>>> CPU speed, and disk space tends to result in lazy programs. I >>>>> understand >>>>> there is an effort/value tradeoff, and I make those tradeoffs myself >>>>> all the time...but it still makes me sad. Then, again, in my early >>>>> programming days I spent a fair amount of time writing and using Forth, >>>>> and that probably colors my worldview. :) >>>>> >>>> >>>> I never used or cared for Forth, but I have the same worldview. I >>>> remember getting it from David Rosenthal, an early Sun reviewer. He stated >>>> that engineers should be given the smallest desktop computer available, not >>>> the largest, so they would feel their users' pain and optimize >>>> appropriately. Sadly software vendors who are also hardware vendors have >>>> incentives going in the opposite direction -- they want users to feel the >>>> pain so they'll buy a new device. >>>> >>> >>> I look at it a different way. Developers should be given powerful >>> machines to speed up the development cycle (especially important when >>> prototyping and in the code/run unit test cycle), but everything should be >>> tested on the smallest machine available. >>> >>> It's also a good idea for each developer to have a resource-constrained >>> machine for developer testing/profiling/etc. Virtual machines work quite >>> well for this - you can modify the resource constraints (CPU, memory, etc) >>> to simulate different scenarios. >>> >>> I find that this tends to better promote the methodology of "make it >>> right, then make it fast (small, etc)", which I subscribe to. Optimising >>> too early leads to all your code being complicated, rather than just the >>> bits that need it. >>> >>> Tim Delaney >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com >> >> -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Wed Oct 9 00:21:06 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 9 Oct 2013 09:21:06 +1100 Subject: [Python-Dev] On the dangers of giving developers the best resources In-Reply-To: References: Message-ID: On 9 October 2013 09:10, Guido van Rossum wrote: > It's not actually so much the extreme waste that I'm looking to expose, > but rather the day-to-day annoyances of stuff you use regularly that slows > you down by just a second (or ten), or things that gets slower at each > release. > Veering off-topic (but still related) ... There's a reason I turn off all animations when I set up a machine for someone ... I've found turning off the animations is the quickest way to make a machine feel faster - even better than adding an SSD. The number of times I've fixed a "slow" machine by this one change ... I think everyone even remotely involved in the existence of animations in the OS should be forced to have the slowest animations turned on at all times, no matter the platform (OSX, Windows, Linux ...). Which comes back to the idea of developers having slow machines so they feel the pain ... Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Wed Oct 9 01:33:26 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 01:33:26 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters Message-ID: <52549646.5060909@hastings.org> I've contributed a new PEP to humanity. I include the RST for your reading pleasure below, but you can also read it online here: http://www.python.org/dev/peps/pep-0457/ Discuss, //arry/ ----- PEP: 457 Title: Syntax For Positional-Only Parameters Version: $Revision$ Last-Modified: $Date$ Author: Larry Hastings Discussions-To: Python-Dev Status: Draft Type: Informational Content-Type: text/x-rst Created: 08-Oct-2013 ======== Overview ======== This PEP proposes a syntax for positional-only parameters in Python. Positional-only parameters are parameters without an externally-usable name; when a function accepting positional-only parameters is called, positional arguments are mapped to these parameters based solely on their position. ========= Rationale ========= Python has always supported positional-only parameters. Early versions of Python lacked the concept of specifying parameters by name, so naturally all parameters were positional-only. This changed around Python 1.0, when all parameters suddenly became positional-or-keyword. But, even in current versions of Python, many CPython "builtin" functions still only accept positional-only arguments. Functions implemented in modern Python can accept an arbitrary number of positional-only arguments, via the variadic ``*args`` parameter. However, there is no Python syntax to specify accepting a specific number of positional-only parameters. Put another way, there are many builtin functions whose signatures are simply not expressable with Python syntax. This PEP proposes a backwards-compatible syntax that should permit implementing any builtin in pure Python code. ----------------------------------------------------- Positional-Only Parameter Semantics In Current Python ----------------------------------------------------- There are many, many examples of builtins that only accept positional-only parameters. The resulting semantics are easily experienced by the Python programmer--just try calling one, specifying its arguments by name:: >>> pow(x=5, y=3) Traceback (most recent call last): File "", line 1, in TypeError: pow() takes no keyword arguments In addition, there are some functions with particularly interesting semantics: * ``range()``, which accepts an optional parameter to the *left* of its required parameter. [#RANGE]_ * ``dict()``, whose mapping/iterator parameter is optional and semantically must be positional-only. Any externally visible name for this parameter would occlude that name going into the ``**kwarg`` keyword variadic parameter dict! [#DICT]_ Obviously one can simulate any of these in pure Python code by accepting ``(*args, **kwargs)`` and parsing the arguments by hand. But this results in a disconnect between the Python function's signature and what it actually accepts, not to mention the work of implementing said argument parsing. ========== Motivation ========== This PEP does not propose we implement positional-only parameters in Python. The goal of this PEP is simply to define the syntax, so that: * Documentation can clearly, unambiguously, and consistently express exactly how the arguments for a function will be interpreted. * The syntax is reserved for future use, in case the community decides someday to add positional-only parameters to the language. * Argument Clinic can use a variant of the syntax as part of its input when defining the arguments for built-in functions. ================================================================= The Current State Of Documentation For Positional-Only Parameters ================================================================= The documentation for positional-only parameters is incomplete and inconsistent: * Some functions denote optional groups of positional-only arguments by enclosing them in nested square brackets. [#BORDER]_ * Some functions denote optional groups of positional-only arguments by presenting multiple prototypes with varying numbers of arguments. [#SENDFILE]_ * Some functions use *both* of the above approaches. [#RANGE]_ [#ADDCH]_ One more important idea to consider: currently in the documentation there's no way to tell whether a function takes positional-only parameters. ``open()`` accepts keyword arguments, ``ord()`` does not, but there is no way of telling just by reading the documentation that this is true. ==================== Syntax And Semantics ==================== From the "ten-thousand foot view", and ignoring ``*args`` and ``**kwargs`` for now, the grammar for a function definition currently looks like this:: def name(positional_or_keyword_parameters, *, keyword_only_parameters): Building on that perspective, the new syntax for functions would look like this:: def name(positional_only_parameters, /, positional_or_keyword_parameters, *, keyword_only_parameters): All parameters before the ``/`` are positional-only. If ``/`` is not specified in a function signature, that function does not accept any positional-only parameters. Positional-only parameters can be optional, but the mechanism is significantly different from positional-or-keyword or keyword-only parameters. Positional-only parameters don't accept default values. Instead, positional-only parameters can be specified in optional "groups". Groups of parameters are surrounded by square brackets, like so:: def addch([y, x,] ch, [attr], /): Positional-only parameters that are not in an option group are "required" positional-only parameters. All "required" positional-only parameters must be contiguous. Parameters in an optional group accept arguments in a group; you must provide arguments either for all of the them or for none of them. Using the example of ``addch()`` above, you could not call ``addch()`` in such a way that ``x`` was specified but ``y`` was not (and vice versa). The mapping of positional parameters to optional groups is done based on fitting the number of parameters to groups. Based on the above definition, ``addch()`` would assign arguments to parameters in the following way: +-------------------+------------------------------+ |Number of arguments|Parameter assignment | +-------------------+------------------------------+ |0 |*raises an exception* | +-------------------+------------------------------+ |1 |``ch`` | +-------------------+------------------------------+ |2 |``ch``, ``attr`` | +-------------------+------------------------------+ |3 |``y``, ``x``, ``ch`` | +-------------------+------------------------------+ |4 |``y``, ``x``, ``ch``, ``attr``| +-------------------+------------------------------+ |5 or more |*raises an exception* | +-------------------+------------------------------+ More semantics of positional-only parameters: * Although positional-only parameter technically have names, these names are internal-only; positional-only parameters are *never* externally addressable by name. (Similarly to ``*args`` and ``**kwargs``.) * It's possible to nest option groups. * If there are no required parameters, all option groups behave as if they're to the right of the required parameter group. * For clarity and consistency, the comma for a parameter always comes immediately after the parameter name. It's a syntax error to specify a square bracket between the name of a parameter and the following comma. (This is far more readable than putting the comma outside the square bracket, particularly for nested groups.) * If there are arguments after the ``/``, then you must specify a comma after the ``/``, just as there is a comma after the ``*`` denoting the shift to keyword-only parameters. * This syntax has no effect on ``*args`` or ``**kwargs``. It's possible to specify a function prototype where the mapping of arguments to parameters is ambiguous. Consider:: def range([start,] stop, [range], /): Python disambiguates these situations by preferring optional groups to the *left* of the required group. ====================== Additional Limitations ====================== Argument Clinic uses a form of this syntax for specifying builtins. It imposes further limitations that are theoretically unnecessary but make the implementation easier. Specifically: * A function that has positional-only parameters currently cannot have any other kind of parameter. (This will probably be relaxed slightly in the near future.) * Multiple option groups on either side of the required positional-only parameters must be nested, with the nesting getting deeper the further away the group is from the required positional-parameter group. Put another way: all the left-brackets for option groups to the left of the required group must be specified contiguously, and all the right-brackets for option groups to the right of the required group must be specified contiguously. ============================== Notes For A Future Implementor ============================== If we decide to implement positional-only parameters in a future version of Python, we'd have to do some additional work to preserve their semantics. The problem: how do we inform a parameter that no value was passed in for it when the function was called? The obvious solution: add a new singleton constant to Python that is passed in when a parameter is not mapped to an argument. I propose that the value be called called ``undefined``, and be a singleton of a special class called ``Undefined``. If a positional-only parameter did not receive an argument when called, its value would be set to ``undefined``. But this raises a further problem. How do can we tell the difference between "this positional-only parameter did not receive an argument" and "the caller passed in ``undefined`` for this parameter"? It'd be nice to make it illegal to pass ``undefined`` in as an argument to a function--to, say, raise an exception. But that would slow Python down, and the "consenting adults" rule appears applicable here. So making it illegal should probably be strongly discouraged but not outright prevented. However, it should be allowed (and encouraged) for user functions to specify ``undefined`` as a default value for parameters. ==================== Unresolved Questions ==================== There are three types of parameters in Python: 1. positional-only parameters, 2. positional-or-keyword parameters, and 3. keyword-only parameters. Python allows functions to have both 2 and 3. And some builtins (e.g. range) have both 1 and 3. Does it make sense to have functions that have both 1 and 2? Or all of the above? ====== Thanks ====== Credit for the use of '/' as the separator between positional-only and positional-or-keyword parameters goes to Guido van Rossum, in a proposal from 2012. [#GUIDO]_ Credit for making left option groups higher precedence goes to Nick Coghlan. (Conversation in person at PyCon US 2013.) .. [#DICT] http://docs.python.org/3/library/stdtypes.html#dict .. [#RANGE] http://docs.python.org/3/library/functions.html#func-range .. [#BORDER] http://docs.python.org/3/library/curses.html#curses.window.border .. [#SENDFILE] http://docs.python.org/3/library/os.html#os.sendfile .. [#ADDCH] http://docs.python.org/3/library/curses.html#curses.window.addch .. [#GUIDO] Guido van Rossum, posting to python-ideas, March 2012: http://mail.python.org/pipermail/python-ideas/2012-March/014364.html and http://mail.python.org/pipermail/python-ideas/2012-March/014378.html and http://mail.python.org/pipermail/python-ideas/2012-March/014417.html ========= Copyright ========= This document has been placed in the public domain. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Oct 9 02:15:40 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 9 Oct 2013 11:15:40 +1100 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52549646.5060909@hastings.org> References: <52549646.5060909@hastings.org> Message-ID: <20131009001540.GA7989@ando> On Wed, Oct 09, 2013 at 01:33:26AM +0200, Larry Hastings wrote: > This PEP proposes a syntax for positional-only parameters in Python. > Positional-only parameters are parameters without an externally-usable > name; when a function accepting positional-only parameters is called, > positional arguments are mapped to these parameters based solely on > their position. Nicely done. A few comments follow: > Positional-only parameters can be optional, but the mechanism is > significantly different from positional-or-keyword or keyword-only > parameters. Positional-only parameters don't accept default > values. Instead, positional-only parameters can be specified > in optional "groups". Groups of parameters are surrounded by > square brackets, like so:: > > def addch([y, x,] ch, [attr], /): I think you need to explain the motivation for this, particularly in light of the fact (below) that they will in fact receive a default value, namely the proposed "undefined" singleton. Personally, if I have a function like this: def mypower(x, y, [n,] /): if n is undefined: n = 1 ... I would much prefer to write it like this: def mypower(x, y, [n=1,] /): ... The PEP should explain why I cannot, or accept that I should be able to. > * For clarity and consistency, the comma for a parameter always > comes immediately after the parameter name. It's a syntax error > to specify a square bracket between the name of a parameter and > the following comma. (This is far more readable than putting > the comma outside the square bracket, particularly for nested > groups.) I note that in your example above, you put the comma outside the square bracket: def addch([y, x,] ch, [attr], /): which seems perfectly readable to me. I think that a much better explanation for prohibiting the comma outside the brackets is logical consistency: if you put the comma outside the brackets, and the attr is left out, you get two commas in a row and we don't want that. > It's possible to specify a function prototype where the mapping > of arguments to parameters is ambiguous. Consider:: > > def range([start,] stop, [range], /): And you do it again, putting the comma outside of the square brackets :-) > If we decide to implement positional-only parameters in a future > version of Python, we'd have to do some additional work to preserve > their semantics. The problem: how do we inform a parameter that > no value was passed in for it when the function was called? > > The obvious solution: add a new singleton constant to Python > that is passed in when a parameter is not mapped to an argument. > I propose that the value be called called ``undefined``, > and be a singleton of a special class called ``Undefined``. > If a positional-only parameter did not receive an argument > when called, its value would be set to ``undefined``. I would much prefer Undefined and UndefinedType. That matches other singletons like None, NotImplemented, Ellipsis, even True and False. What (if any) methods and attributes would Undefined have? If it doesn't have any, doesn't that make it functionally equivalent to None? Why not just use None? The PEP needs to explain why it needs to invent yet another singleton that quacks like None. [Bikeshed: perhaps Missing is more appropriate than Undefined? After all, the parameter is defined, only the value is missing.] > But this raises a further problem. How do can we tell the > difference between "this positional-only parameter did not > receive an argument" and "the caller passed in ``undefined`` > for this parameter"? Why treat this as a problem to be solved? If somebody wants to go to the trouble of writing: addchr('c', Undefined) instead of just: addchr('c') why not let them? > It'd be nice to make it illegal to pass ``undefined`` in > as an argument to a function--to, say, raise an exception. > But that would slow Python down, and the "consenting adults" > rule appears applicable here. So making it illegal should > probably be strongly discouraged but not outright prevented. An argument for allowing Undefined (or None, as the case may be): sometimes you have a situation like this: c = get_char() attr = get_attr() if attr is Undefined: # or None addchr(c) else: addchr(c, attr) That sucks. But this isn't much better: c = get_char() attr = get_attr() if attr is Undefined: # or None t = (c,) else: t = (c, attr) addchr(*t) Being able to pass Undefined explicitly avoids this ugliness. > However, it should be allowed (and encouraged) for user > functions to specify ``undefined`` as a default value for > parameters. Because I think this is important, I'm going to raise it again: I think it is important for the PEP to justify why user functions cannot specify arbitrary values as defaults, not just Undefined. Thanks again for tackling this. -- Steven From python at mrabarnett.plus.com Wed Oct 9 02:22:18 2013 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 09 Oct 2013 01:22:18 +0100 Subject: [Python-Dev] On the dangers of giving developers the best resources In-Reply-To: References: Message-ID: <5254A1BA.20107@mrabarnett.plus.com> On 08/10/2013 23:21, Tim Delaney wrote: > On 9 October 2013 09:10, Guido van Rossum > wrote: > > It's not actually so much the extreme waste that I'm looking to > expose, but rather the day-to-day annoyances of stuff you use > regularly that slows you down by just a second (or ten), or things > that gets slower at each release. > > > Veering off-topic (but still related) ... > > There's a reason I turn off all animations when I set up a machine for > someone ... I've found turning off the animations is the quickest way to > make a machine feel faster - even better than adding an SSD. The number > of times I've fixed a "slow" machine by this one change ... > > I think everyone even remotely involved in the existence of animations > in the OS should be forced to have the slowest animations turned on at > all times, no matter the platform (OSX, Windows, Linux ...). Which comes > back to the idea of developers having slow machines so they feel the > pain ... > I remember one time when I was using a Mac. Although it was faster than another machine I was using, the GUI felt sluggish because instead of windows just appearing and disappearing they expanded and contracted, which, of course, took time; not much time, true, but enough to become annoying. From ethan at stoneleaf.us Wed Oct 9 02:16:25 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 08 Oct 2013 17:16:25 -0700 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52549646.5060909@hastings.org> References: <52549646.5060909@hastings.org> Message-ID: <5254A059.9030002@stoneleaf.us> On 10/08/2013 04:33 PM, Larry Hastings wrote: > > I've contributed a new PEP to humanity. I include the RST for your reading pleasure below, but you can also read it > online here: > > http://www.python.org/dev/peps/pep-0457/ I like it! :) Feedback below... > > > ============================== > Notes For A Future Implementor > ============================== > [snip] > > The obvious solution: add a new singleton constant to Python > that is passed in when a parameter is not mapped to an argument. > I propose that the value be called called ``undefined``, ^ called is doubled -------------/ > and be a singleton of a special class called ``Undefined``. > If a positional-only parameter did not receive an argument > when called, its value would be set to ``undefined``. To stick with current practice: --> type(Ellipsis) --> type(None) --> type(NotImplemented) The instance should be Undefined and its class either undefined or UndefinedType. > But this raises a further problem. How do can we tell the > difference between "this positional-only parameter did not > receive an argument" and "the caller passed in ``undefined`` > for this parameter"? > > It'd be nice to make it illegal to pass ``undefined`` in > as an argument to a function--to, say, raise an exception. > But that would slow Python down, and the "consenting adults" > rule appears applicable here. So making it illegal should > probably be strongly discouraged but not outright prevented. Agreed. > However, it should be allowed (and encouraged) for user > functions to specify ``undefined`` as a default value for > parameters. Also agreed. > ==================== > Unresolved Questions > ==================== > > There are three types of parameters in Python: > > 1. positional-only parameters, > 2. positional-or-keyword parameters, and > 3. keyword-only parameters. > > Python allows functions to have both 2 and 3. And some > builtins (e.g. range) have both 1 and 3. Does it make > sense to have functions that have both 1 and 2? Or > all of the above? Rather than try to think up the cases where it makes sense, let's just support all three. Thanks for writing this up, Larry! -- ~Ethan~ From theaney at gmail.com Wed Oct 9 02:49:17 2013 From: theaney at gmail.com (Tim Heaney) Date: Tue, 8 Oct 2013 20:49:17 -0400 Subject: [Python-Dev] lzma and 'x' mode open Message-ID: I love the 'x' mode open in recent versions of Python. I just discovered that lzma.open doesn't support it. It seems there's an elif that explicitly checks the modes allowed. I added "x" and "xb" to the choices and now it seems to work as I would like. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: patch.lzma.py Type: application/octet-stream Size: 464 bytes Desc: not available URL: From eric at trueblade.com Wed Oct 9 03:00:09 2013 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 8 Oct 2013 21:00:09 -0400 Subject: [Python-Dev] lzma and 'x' mode open In-Reply-To: References: Message-ID: <7628BEB0-7FB4-4746-9C38-AF9BDAB9511B@trueblade.com> Please open a bug report on bugs.python.org so this doesn't get lost. -- Eric. > On Oct 8, 2013, at 8:49 PM, Tim Heaney wrote: > > I love the 'x' mode open in recent versions of Python. I just discovered that lzma.open doesn't support it. It seems there's an elif that explicitly checks the modes allowed. I added "x" and "xb" to the choices and now it seems to work as I would like. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com From larry at hastings.org Wed Oct 9 03:05:25 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 03:05:25 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <20131009001540.GA7989@ando> References: <52549646.5060909@hastings.org> <20131009001540.GA7989@ando> Message-ID: <5254ABD5.9080509@hastings.org> A very quick reply, more tomorrow. On 10/09/2013 02:15 AM, Steven D'Aprano wrote: > I note that in your example above, you put the comma outside the square > bracket: > > def addch([y, x,] ch, [attr], /): > > which seems perfectly readable to me. It becomes less readable / more tiresome with nested groups. Which square bracket should the comma come after? Anyway, it seems like you agree with the syntactic requirement for other reasons. (And, I already fixed the two places in the PEP where I had the comma outside the square brackets, thanks for pointing it out.) > I would much prefer Undefined and UndefinedType. That matches other > singletons like None, NotImplemented, Ellipsis, even True and False. You're probably right. > [Bikeshed: perhaps Missing is more appropriate than Undefined? After > all, the parameter is defined, only the value is missing.] Let the bikeshedding begin! > [Argument for allowing explicitly passing "undefined": > this lets the iterable in foo(*iterable) yield "undefined", > which could be convenient] That's a good thought. But I'd be interested in more debate on the subject to see if people have other good reasons for/against encouraging explicitly using "undefined". > Because I think this is important, I'm going to raise it again: I think > it is important for the PEP to justify why user functions cannot specify > arbitrary values as defaults, not just Undefined. Primarily because this codifies existing practice. C bulitins with positional-only decide how to assign their arguments based on how many there are, and when a parameter doesn't receive an argument it almost never gets a default value. I was trying to preserve these exact semantics, which is where the optional groups came from. And since I already had optional groups and "undefined", that seemed sufficient. I'll also admit, I mainly ruled it out back before the prefer-left disambiguation rule, and allowing default values for positional-only led to loads of ambiguity. I hadn't reconsidered the restriction in light of the new rule. But I still think the semantics get weird quickly. Consider: def bar([a=7, b,] c, [d,] /): Default values would have to grow from the outside in, staying away from the required positional-only parameter group. So on the left side they'd be backwards. And if you call bar() with two arguments, you'd get (a b c), not (c d). In fact there'd be no way of specifying d without providing four arguments. By that token there'd be no way of calling bar() and specifying d without stomping on the default value for a. I agree I should do a better job of justifying the design in a future revision. I'd have to think long and hard before allowing default values for positional-only arguments... and that's a restriction I /wouldn't/ relax for Argument Clinic. Anyway it's late, more tomorrow. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Wed Oct 9 03:13:17 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 08 Oct 2013 21:13:17 -0400 Subject: [Python-Dev] lzma and 'x' mode open In-Reply-To: References: Message-ID: <20131009011317.D2D8A25007D@webabinitio.net> On Tue, 08 Oct 2013 20:49:17 -0400, Tim Heaney wrote: > I love the 'x' mode open in recent versions of Python. I just discovered > that lzma.open doesn't support it. It seems there's an elif that explicitly > checks the modes allowed. I added "x" and "xb" to the choices and now it > seems to work as I would like. Please file an enhancement request at bugs.python.org. --David From ethan at stoneleaf.us Wed Oct 9 02:38:05 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 08 Oct 2013 17:38:05 -0700 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <20131009001540.GA7989@ando> References: <52549646.5060909@hastings.org> <20131009001540.GA7989@ando> Message-ID: <5254A56D.7090501@stoneleaf.us> On 10/08/2013 05:15 PM, Steven D'Aprano wrote: > On Wed, Oct 09, 2013 at 01:33:26AM +0200, Larry Hastings wrote: >> >> The obvious solution: add a new singleton constant to Python >> that is passed in when a parameter is not mapped to an argument. >> I propose that the value be called called ``undefined``, >> and be a singleton of a special class called ``Undefined``. >> If a positional-only parameter did not receive an argument >> when called, its value would be set to ``undefined``. > > I would much prefer Undefined and UndefinedType. That matches other > singletons like None, NotImplemented, Ellipsis, even True and False. The type of Ellipsis is ellipsis, and the type of True and False is bool. ;) That aside, I agree that Undefined and UndefinedType would be better. > What (if any) methods and attributes would Undefined have? If it doesn't > have any, doesn't that make it functionally equivalent to None? Why not > just use None? The PEP needs to explain why it needs to invent yet > another singleton that quacks like None. I think the key question here is are there places where None is the passed in value, and its meaning is something besides "no value passed in". I don't know the answer. > [Bikeshed: perhaps Missing is more appropriate than Undefined? After > all, the parameter is defined, only the value is missing.] Or the value is undefined. ;) -- ~Ethan~ From benjamin at python.org Wed Oct 9 03:31:56 2013 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 8 Oct 2013 21:31:56 -0400 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52549646.5060909@hastings.org> References: <52549646.5060909@hastings.org> Message-ID: 2013/10/8 Larry Hastings : > This PEP proposes a backwards-compatible syntax that should > permit implementing any builtin in pure Python code. This is rather too strong. You can certainly implement them; you just have to implement the argument parsing yourself. Python's call/signature syntax is already extremely expressive, and resolving call arguments to formal parameters is already a complicated (and slow) process. Implementing functions with such strange argument semantics is hardly common enough to justify the whole grouping syntax proposed in this PEP. -1 to that. I think I can live with "/", but YANGTNI still. -- Regards, Benjamin From theaney at gmail.com Wed Oct 9 03:41:28 2013 From: theaney at gmail.com (Tim Heaney) Date: Tue, 8 Oct 2013 21:41:28 -0400 Subject: [Python-Dev] lzma and 'x' mode open In-Reply-To: <20131009011317.D2D8A25007D@webabinitio.net> References: <20131009011317.D2D8A25007D@webabinitio.net> Message-ID: Done. http://bugs.python.org/issue19201 I guess I should have known that. Sorry to bother python-dev with this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Wed Oct 9 03:57:51 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Tue, 8 Oct 2013 22:57:51 -0300 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: I am -1 for the optional parameters and grouping stuff - no need to make complicated stuff easier to do just because "range" semantics is strange to start with. (And one can implement a range-like funciton explictly parsing the parameters if needed be.) As for the "/" delimiting positional only parameters - I think it would be nice to have. -but just that - mandatory positional only parameters. Apart from that, I could not figure if the PEP covers a situation where the name of a positional-only parameter is passed in as a kwarg - def a([b],/,**kw): print (b) does calling "a(b=5)" should raise a TypeError, or put "5" in """kw["b"]""" and leave "b" as "undefined" ? js -><- On 8 October 2013 22:31, Benjamin Peterson wrote: > 2013/10/8 Larry Hastings : >> This PEP proposes a backwards-compatible syntax that should >> permit implementing any builtin in pure Python code. > > This is rather too strong. You can certainly implement them; you just > have to implement the argument parsing yourself. Python's > call/signature syntax is already extremely expressive, and resolving > call arguments to formal parameters is already a complicated (and > slow) process. Implementing functions with such strange argument > semantics is hardly common enough to justify the whole grouping syntax > proposed in this PEP. -1 to that. I think I can live with "/", but > YANGTNI still. > > > > -- > Regards, > Benjamin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br From bitsink at gmail.com Wed Oct 9 04:30:30 2013 From: bitsink at gmail.com (Nam Nguyen) Date: Wed, 9 Oct 2013 02:30:30 +0000 Subject: [Python-Dev] On the dangers of giving developers the best resources In-Reply-To: <5254A1BA.20107@mrabarnett.plus.com> References: <5254A1BA.20107@mrabarnett.plus.com> Message-ID: On Wed, Oct 9, 2013 at 12:22 AM, MRAB wrote: > On 08/10/2013 23:21, Tim Delaney wrote: > >> On 9 October 2013 09:10, Guido van Rossum > > wrote: >> >> It's not actually so much the extreme waste that I'm looking to >> expose, but rather the day-to-day annoyances of stuff you use >> regularly that slows you down by just a second (or ten), or things >> that gets slower at each release. >> >> >> Veering off-topic (but still related) ... >> >> There's a reason I turn off all animations when I set up a machine for >> someone ... I've found turning off the animations is the quickest way to >> make a machine feel faster - even better than adding an SSD. The number >> of times I've fixed a "slow" machine by this one change ... >> >> I think everyone even remotely involved in the existence of animations >> in the OS should be forced to have the slowest animations turned on at >> all times, no matter the platform (OSX, Windows, Linux ...). Which comes >> back to the idea of developers having slow machines so they feel the >> pain ... >> >> I remember one time when I was using a Mac. Although it was faster than > another machine I was using, the GUI felt sluggish because instead of > windows just appearing and disappearing they expanded and contracted, > which, of course, took time; not much time, true, but enough to become > annoying. Try holding shift and minimizing/restoring Finder in OS X ;). Nam -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Oct 9 05:02:00 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 08 Oct 2013 23:02:00 -0400 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: On 10/8/2013 9:31 PM, Benjamin Peterson wrote: > 2013/10/8 Larry Hastings : >> This PEP proposes a backwards-compatible syntax that should >> permit implementing any builtin in pure Python code. > > This is rather too strong. You can certainly implement them; you just > have to implement the argument parsing yourself. Python's > call/signature syntax is already extremely expressive, and resolving > call arguments to formal parameters is already a complicated (and > slow) process. Implementing functions with such strange argument > semantics is hardly common enough to justify the whole grouping syntax > proposed in this PEP. -1 to that. I think I can live with "/", but > YANGTNI still. I am for having a way to succintly properly describe the signature of C in the manual and docstrings and help output. As it is now, the only safe thing to do, without trial and exception, is to assume positional only unless one knows otherwise. But the syntax should only be as complicated as needed for 'most' C functions. If there are a few complicated oddballs, perhaps they could be simplified, or given a special-case explanation. -- Terry Jan Reedy From benjamin at python.org Wed Oct 9 05:09:44 2013 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 8 Oct 2013 23:09:44 -0400 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: 2013/10/8 Terry Reedy : > On 10/8/2013 9:31 PM, Benjamin Peterson wrote: >> >> 2013/10/8 Larry Hastings : >>> >>> This PEP proposes a backwards-compatible syntax that should >>> permit implementing any builtin in pure Python code. >> >> >> This is rather too strong. You can certainly implement them; you just >> have to implement the argument parsing yourself. Python's >> call/signature syntax is already extremely expressive, and resolving >> call arguments to formal parameters is already a complicated (and >> slow) process. Implementing functions with such strange argument >> semantics is hardly common enough to justify the whole grouping syntax >> proposed in this PEP. -1 to that. I think I can live with "/", but >> YANGTNI still. > > > I am for having a way to succintly properly describe the signature of C in > the manual and docstrings and help output. As it is now, the only safe thing > to do, without trial and exception, is to assume positional only unless one > knows otherwise. Having a nice syntax for the docs is quite different from implementing it in the language. -- Regards, Benjamin From ethan at stoneleaf.us Wed Oct 9 05:49:57 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 08 Oct 2013 20:49:57 -0700 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: <5254D265.1000509@stoneleaf.us> On 10/08/2013 08:09 PM, Benjamin Peterson wrote: > 2013/10/8 Terry Reedy : >> On 10/8/2013 9:31 PM, Benjamin Peterson wrote: >>> >>> 2013/10/8 Larry Hastings : >>>> >>>> This PEP proposes a backwards-compatible syntax that should >>>> permit implementing any builtin in pure Python code. >>> >>> >>> This is rather too strong. You can certainly implement them; you just >>> have to implement the argument parsing yourself. Python's >>> call/signature syntax is already extremely expressive, and resolving >>> call arguments to formal parameters is already a complicated (and >>> slow) process. Implementing functions with such strange argument >>> semantics is hardly common enough to justify the whole grouping syntax >>> proposed in this PEP. -1 to that. I think I can live with "/", but >>> YANGTNI still. >> >> >> I am for having a way to succintly properly describe the signature of C in >> the manual and docstrings and help output. As it is now, the only safe thing >> to do, without trial and exception, is to assume positional only unless one >> knows otherwise. > > Having a nice syntax for the docs is quite different from implementing > it in the language. It would be nice, however, to have it implemented at some point. -- ~Ethan~ From benjamin at python.org Wed Oct 9 06:55:14 2013 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 9 Oct 2013 00:55:14 -0400 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <5254D265.1000509@stoneleaf.us> References: <52549646.5060909@hastings.org> <5254D265.1000509@stoneleaf.us> Message-ID: 2013/10/8 Ethan Furman : > On 10/08/2013 08:09 PM, Benjamin Peterson wrote: >> >> 2013/10/8 Terry Reedy : >>> >>> On 10/8/2013 9:31 PM, Benjamin Peterson wrote: >>>> >>>> >>>> 2013/10/8 Larry Hastings : >>>>> >>>>> >>>>> This PEP proposes a backwards-compatible syntax that should >>>>> permit implementing any builtin in pure Python code. >>>> >>>> >>>> >>>> This is rather too strong. You can certainly implement them; you just >>>> have to implement the argument parsing yourself. Python's >>>> call/signature syntax is already extremely expressive, and resolving >>>> call arguments to formal parameters is already a complicated (and >>>> slow) process. Implementing functions with such strange argument >>>> semantics is hardly common enough to justify the whole grouping syntax >>>> proposed in this PEP. -1 to that. I think I can live with "/", but >>>> YANGTNI still. >>> >>> >>> >>> I am for having a way to succintly properly describe the signature of C >>> in >>> the manual and docstrings and help output. As it is now, the only safe >>> thing >>> to do, without trial and exception, is to assume positional only unless >>> one >>> knows otherwise. >> >> >> Having a nice syntax for the docs is quite different from implementing >> it in the language. > > > It would be nice, however, to have it implemented at some point. Why? It's quite complex and hardly useful. -- Regards, Benjamin From g.brandl at gmx.net Wed Oct 9 07:15:22 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 09 Oct 2013 07:15:22 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52549646.5060909@hastings.org> References: <52549646.5060909@hastings.org> Message-ID: Am 09.10.2013 01:33, schrieb Larry Hastings: > > I've contributed a new PEP to humanity. I include the RST for your reading > pleasure below, but you can also read it online here: > > http://www.python.org/dev/peps/pep-0457/ ... > * Documentation can clearly, unambiguously, and > consistently express exactly how the arguments > for a function will be interpreted. I don't really like it from the docs perspective. We have quite a large amount of C functions with positional-only parameters. Adding a "/" to each of those is a) a tedious task and b) probably not helpful for a lot of people: For each user who tries to call a C function with keyword args and is confused, there will be ten or twenty who are confused by the strange "/" in the first place. (We get quite a few questions about the *args and **kwargs syntax already that is infrequently used, which is otherwise _part of the language_.) Rather, I would try to make as many C functions as possible "regular", i.e. use PyArgs_ParseTupleAndKeywords or the Argument Clinic equivalent. For the few that should be blazingly fast or have irregular signatures, the documentation can be amended by a single sentence "This function does not support passing keyword arguments." and that's the end of it. > * The syntax is reserved for future use, in case > the community decides someday to add positional-only > parameters to the language. For this purpose I'm also -1. Function definition has become almost too complex already with keyword-only arguments, although they at least have usefulness other than mimicking the deficiencies of an implementation detail... and the "undefined" singleton just smells wrong. Another way to spell "None" is just asking for trouble. > * Argument Clinic can use a variant of the syntax > as part of its input when defining > the arguments for built-in functions. For AC it's good enough I guess :) but that doesn't require a separate PEP. Also, the AC DSL can be changed much, much easier than the language can be if this syntax is accepted. cheers, Georg From ethan at stoneleaf.us Wed Oct 9 06:50:01 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 08 Oct 2013 21:50:01 -0700 Subject: [Python-Dev] inspect() and dir() In-Reply-To: References: <5254791E.1020803@stoneleaf.us> Message-ID: <5254E079.3000809@stoneleaf.us> On 10/08/2013 03:07 PM, Guido van Rossum wrote: > > I'd say that the effect overriding dir() has on help() is intentional. Not sure about inspect -- it has other uses. > However, the last time I used it I was definitely hunting for stuff to document. If want help to be effected that is easy enough -- there is already an inspect.classify_class_attrs wrapper, so if we add some post-processing to remove anything not in dir(obj) we would keep the current behavior in help(). Any other comments/recollections/concerns? -- ~Ethan~ From ethan at stoneleaf.us Wed Oct 9 07:13:13 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 08 Oct 2013 22:13:13 -0700 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> <5254D265.1000509@stoneleaf.us> Message-ID: <5254E5E9.4070403@stoneleaf.us> On 10/08/2013 09:55 PM, Benjamin Peterson wrote: > 2013/10/8 Ethan Furman : >> On 10/08/2013 08:09 PM, Benjamin Peterson wrote: >>> >>> 2013/10/8 Terry Reedy : >>>> >>>> On 10/8/2013 9:31 PM, Benjamin Peterson wrote: >>>>> >>>>> >>>>> 2013/10/8 Larry Hastings : >>>>>> >>>>>> >>>>>> This PEP proposes a backwards-compatible syntax that should >>>>>> permit implementing any builtin in pure Python code. >>>>> >>>>> >>>>> >>>>> This is rather too strong. You can certainly implement them; you just >>>>> have to implement the argument parsing yourself. Python's >>>>> call/signature syntax is already extremely expressive, and resolving >>>>> call arguments to formal parameters is already a complicated (and >>>>> slow) process. Implementing functions with such strange argument >>>>> semantics is hardly common enough to justify the whole grouping syntax >>>>> proposed in this PEP. -1 to that. I think I can live with "/", but >>>>> YANGTNI still. >>>> >>>> >>>> >>>> I am for having a way to succintly properly describe the signature of C >>>> in >>>> the manual and docstrings and help output. As it is now, the only safe >>>> thing >>>> to do, without trial and exception, is to assume positional only unless >>>> one >>>> knows otherwise. >>> >>> >>> Having a nice syntax for the docs is quite different from implementing >>> it in the language. >> >> >> It would be nice, however, to have it implemented at some point. > > Why? It's quite complex and hardly useful. Hmmm...... Let me get back to you on that. ;) -- ~Ethan~ From ericsnowcurrently at gmail.com Wed Oct 9 07:48:18 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 8 Oct 2013 23:48:18 -0600 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52549646.5060909@hastings.org> References: <52549646.5060909@hastings.org> Message-ID: On Tue, Oct 8, 2013 at 5:33 PM, Larry Hastings wrote: > > I've contributed a new PEP to humanity. I include the RST for your reading > pleasure below, but you can also read it online here: > > http://www.python.org/dev/peps/pep-0457/ > > > Discuss, The PEP doesn't mention anything about inspect.Signature, which already supports positional-only parameters. It should. Further comments in-line. -eric > > > /arry > > ----- > > PEP: 457 > Title: Syntax For Positional-Only Parameters > Version: $Revision$ > Last-Modified: $Date$ > Author: Larry Hastings > Discussions-To: Python-Dev > Status: Draft > Type: Informational > Content-Type: text/x-rst > Created: 08-Oct-2013 > > > ======== > Overview > ======== > > This PEP proposes a syntax for positional-only parameters in Python. > Positional-only parameters are parameters without an externally-usable > name; when a function accepting positional-only parameters is called, > positional arguments are mapped to these parameters based solely on > their position. > > ========= > Rationale > ========= > > Python has always supported positional-only parameters. > Early versions of Python lacked the concept of specifying > parameters by name, so naturally all parameters were > positional-only. This changed around Python 1.0, when > all parameters suddenly became positional-or-keyword. > But, even in current versions of Python, many CPython > "builtin" functions still only accept positional-only > arguments. > > Functions implemented in modern Python can accept > an arbitrary number of positional-only arguments, via the > variadic ``*args`` parameter. However, there is no Python > syntax to specify accepting a specific number of > positional-only parameters. Put another way, there are > many builtin functions whose signatures are simply not > expressable with Python syntax. Definitely the key point. This also has an impact on other Python implementations, where maintaining the same positional-only semantics for "builtins" is complicated (at least a little) by the lack of a positional-only syntax in Python. > > This PEP proposes a backwards-compatible syntax that should > permit implementing any builtin in pure Python code. > > ----------------------------------------------------- > Positional-Only Parameter Semantics In Current Python > ----------------------------------------------------- > > There are many, many examples of builtins that only > accept positional-only parameters. The resulting > semantics are easily experienced by the Python > programmer--just try calling one, specifying its > arguments by name:: > > >>> pow(x=5, y=3) > Traceback (most recent call last): > File "", line 1, in > TypeError: pow() takes no keyword arguments > > In addition, there are some functions with particularly > interesting semantics: > > * ``range()``, which accepts an optional parameter > to the *left* of its required parameter. [#RANGE]_ > > * ``dict()``, whose mapping/iterator parameter is optional and > semantically must be positional-only. Any externally > visible name for this parameter would occlude > that name going into the ``**kwarg`` keyword variadic > parameter dict! [#DICT]_ > > Obviously one can simulate any of these in pure Python code > by accepting ``(*args, **kwargs)`` and parsing the arguments > by hand. But this results in a disconnect between the > Python function's signature and what it actually accepts, > not to mention the work of implementing said argument parsing. > > ========== > Motivation > ========== > > This PEP does not propose we implement positional-only > parameters in Python. The goal of this PEP is simply > to define the syntax, This would be worth noting in the "Overview" section. It completely changes the intent of the proposal. > so that: > > * Documentation can clearly, unambiguously, and > consistently express exactly how the arguments > for a function will be interpreted. > > * The syntax is reserved for future use, in case > the community decides someday to add positional-only > parameters to the language. > > * Argument Clinic can use a variant of the syntax > as part of its input when defining > the arguments for built-in functions. > > ================================================================= > The Current State Of Documentation For Positional-Only Parameters > ================================================================= > > The documentation for positional-only parameters is incomplete > and inconsistent: > > * Some functions denote optional groups of positional-only arguments > by enclosing them in nested square brackets. [#BORDER]_ > > * Some functions denote optional groups of positional-only arguments > by presenting multiple prototypes with varying numbers of > arguments. [#SENDFILE]_ > > * Some functions use *both* of the above approaches. [#RANGE]_ [#ADDCH]_ > > One more important idea to consider: currently in the documentation > there's no way to tell whether a function takes positional-only > parameters. ``open()`` accepts keyword arguments, ``ord()`` does > not, but there is no way of telling just by reading the > documentation that this is true. > > ==================== > Syntax And Semantics > ==================== > > From the "ten-thousand foot view", and ignoring ``*args`` and ``**kwargs`` > for now, the grammar for a function definition currently looks like this:: > > def name(positional_or_keyword_parameters, *, keyword_only_parameters): > > Building on that perspective, the new syntax for functions would look > like this:: > > def name(positional_only_parameters, /, > positional_or_keyword_parameters, > *, keyword_only_parameters): > > All parameters before the ``/`` are positional-only. If ``/`` is > not specified in a function signature, that function does not > accept any positional-only parameters. > > Positional-only parameters can be optional, but the mechanism is > significantly different from positional-or-keyword or keyword-only > parameters. Positional-only parameters don't accept default > values. Instead, positional-only parameters can be specified > in optional "groups". Groups of parameters are surrounded by > square brackets, like so:: > > def addch([y, x,] ch, [attr], /): This is for the sake of only a couple unfortunate builtins, right? Parameter grouping will undoubtedly be the most controversial part of this PEP. I'm not convinced this is a capability we need to encourage in any way. Is there some alternate one-off we can use for range, addch, et al.? > > Positional-only parameters that are not in an option group are > "required" positional-only parameters. All "required" positional-only > parameters must be contiguous. > > Parameters in an optional group accept arguments in a group; you > must provide arguments either for all of the them or for none of them. > Using the example of ``addch()`` above, you could not call ``addch()`` > in such a way that ``x`` was specified but ``y`` was not (and vice versa). > The mapping of positional parameters to optional groups is done > based on fitting the number of parameters to groups. Based on the > above definition, ``addch()`` would assign arguments to parameters > in the following way: > > +-------------------+------------------------------+ > |Number of arguments|Parameter assignment | > +-------------------+------------------------------+ > |0 |*raises an exception* | > +-------------------+------------------------------+ > |1 |``ch`` | > +-------------------+------------------------------+ > |2 |``ch``, ``attr`` | > +-------------------+------------------------------+ > |3 |``y``, ``x``, ``ch`` | > +-------------------+------------------------------+ > |4 |``y``, ``x``, ``ch``, ``attr``| > +-------------------+------------------------------+ > |5 or more |*raises an exception* | > +-------------------+------------------------------+ What are the names bound to for unfulfilled optional groups? Not bound (yuck)? What is wrong with default values within optional groups (other than how messy the syntax starts to become)? > > > More semantics of positional-only parameters: > > * Although positional-only parameter technically have names, > these names are internal-only; positional-only parameters > are *never* externally addressable by name. (Similarly > to ``*args`` and ``**kwargs``.) > > * It's possible to nest option groups. > > * If there are no required parameters, all option groups behave > as if they're to the right of the required parameter group. > > * For clarity and consistency, the comma for a parameter always > comes immediately after the parameter name. It's a syntax error > to specify a square bracket between the name of a parameter and > the following comma. (This is far more readable than putting > the comma outside the square bracket, particularly for nested > groups.) Good catch. Presumably parsing this won't be a pain. > > * If there are arguments after the ``/``, then you must specify > a comma after the ``/``, just as there is a comma > after the ``*`` denoting the shift to keyword-only parameters. > > * This syntax has no effect on ``*args`` or ``**kwargs``. > > It's possible to specify a function prototype where the mapping > of arguments to parameters is ambiguous. Consider:: > > def range([start,] stop, [range], /): > > Python disambiguates these situations by preferring optional groups > to the *left* of the required group. What about this: def spam([a,] b, [c,] d, /): Using my intuitive ambiguity-resolution heuristics, I would guess it would resolve spam(1, 2, 3) by binding 3 to d. However, more complicated signatures, particularly involving optional groups, would require even more elaborate ambiguity resolution, wouldn't they? While this proposal isn't about an actual implementation, it does imply that any future implementation would be reasonable. Do you feel that's the case here? > > ====================== > Additional Limitations > ====================== > > Argument Clinic uses a form of this syntax for specifying > builtins. It imposes further limitations that are > theoretically unnecessary but make the implementation > easier. Specifically: > > * A function that has positional-only parameters currently > cannot have any other kind of parameter. (This will > probably be relaxed slightly in the near future.) > > * Multiple option groups on either side of the required > positional-only parameters must be nested, with the > nesting getting deeper the further away the group is > from the required positional-parameter group. Did I miss it or did you already specify that there can only be 1 required positional-only group? It certainly simplifies things a bunch and lessens my concerns about ambiguity-resolution. > > Put another way: > all the left-brackets for option groups to the > left of the required group must be specified contiguously, > and > all the right-brackets for option groups to the > right of the required group must be specified contiguously. > > > ============================== > Notes For A Future Implementor > ============================== > > If we decide to implement positional-only parameters in a future > version of Python, we'd have to do some additional work to preserve > their semantics. The problem: how do we inform a parameter that > no value was passed in for it when the function was called? > > The obvious solution: add a new singleton constant to Python > that is passed in when a parameter is not mapped to an argument. > I propose that the value be called called ``undefined``, I've called this "NOT_SET" when I've had the need. > and be a singleton of a special class called ``Undefined``. > If a positional-only parameter did not receive an argument > when called, its value would be set to ``undefined``. > > But this raises a further problem. How do can we tell the > difference between "this positional-only parameter did not > receive an argument" and "the caller passed in ``undefined`` > for this parameter"? > > It'd be nice to make it illegal to pass ``undefined`` in > as an argument to a function--to, say, raise an exception. > But that would slow Python down, and the "consenting adults" > rule appears applicable here. So making it illegal should > probably be strongly discouraged but not outright prevented. > > However, it should be allowed (and encouraged) for user > functions to specify ``undefined`` as a default value for > parameters. Too bad that None doesn't fit here. This is where allowing defaults in optional groups would be handy. > > ==================== > Unresolved Questions > ==================== > > There are three types of parameters in Python: > > 1. positional-only parameters, > 2. positional-or-keyword parameters, and > 3. keyword-only parameters. > > Python allows functions to have both 2 and 3. And some > builtins (e.g. range) have both 1 and 3. Does it make > sense to have functions that have both 1 and 2? Or > all of the above? I'm pretty sure I've done this before (using *args and **kwargs). However, I don't have any concrete examples. > > > ====== > Thanks > ====== > > Credit for the use of '/' as the separator between positional-only and > positional-or-keyword > parameters goes to Guido van Rossum, in a proposal from 2012. [#GUIDO]_ > > Credit for making left option groups higher precedence goes to > Nick Coghlan. (Conversation in person at PyCon US 2013.) > > .. [#DICT] > http://docs.python.org/3/library/stdtypes.html#dict > > .. [#RANGE] > http://docs.python.org/3/library/functions.html#func-range > > .. [#BORDER] > http://docs.python.org/3/library/curses.html#curses.window.border > > .. [#SENDFILE] > http://docs.python.org/3/library/os.html#os.sendfile > > .. [#ADDCH] > http://docs.python.org/3/library/curses.html#curses.window.addch > > .. [#GUIDO] > Guido van Rossum, posting to python-ideas, March 2012: > http://mail.python.org/pipermail/python-ideas/2012-March/014364.html > and > http://mail.python.org/pipermail/python-ideas/2012-March/014378.html > and > http://mail.python.org/pipermail/python-ideas/2012-March/014417.html > > ========= > Copyright > ========= > > This document has been placed in the public domain. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com > From ericsnowcurrently at gmail.com Wed Oct 9 07:59:53 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 8 Oct 2013 23:59:53 -0600 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: On Tue, Oct 8, 2013 at 9:02 PM, Terry Reedy wrote: > I am for having a way to succintly properly describe the signature of C in > the manual and docstrings and help output. As it is now, the only safe thing > to do, without trial and exception, is to assume positional only unless one > knows otherwise. > > But the syntax should only be as complicated as needed for 'most' C > functions. If there are a few complicated oddballs, perhaps they could be > simplified, or given a special-case explanation. +1 -eric From solipsis at pitrou.net Wed Oct 9 09:04:34 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 9 Oct 2013 09:04:34 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters References: <52549646.5060909@hastings.org> Message-ID: <20131009090434.412da92f@fsol> On Wed, 09 Oct 2013 01:33:26 +0200 Larry Hastings wrote: > > I've contributed a new PEP to humanity. I include the RST for your > reading pleasure below, but you can also read it online here: > > http://www.python.org/dev/peps/pep-0457/ Only skimmed through the PEP, but -1 on the whole idea. I'm also -1 on the "/" separator for docs, it's obscure and brings nothing to the table but pointless complication. Regards Antoine. From larry at hastings.org Wed Oct 9 09:44:53 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 09:44:53 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: <52550975.9070306@hastings.org> On 10/09/2013 03:57 AM, Joao S. O. Bueno wrote: > def a([b],/,**kw): > print (b) > > does calling "a(b=5)" should raise a TypeError, or put "5" in > """kw["b"]""" and leave > "b" as "undefined" ? The latter. This is how they are "similarly to *args and **kwargs": >>> def foo(*a, **kw): ... print(a, kw) ... >>> foo(a=3, kw=5) () {'kw': 5, 'a': 3} >>> //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Oct 9 10:29:30 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 9 Oct 2013 10:29:30 +0200 Subject: [Python-Dev] Reduce memory footprint of Python References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> Message-ID: <20131009102930.4646de4f@pitrou.net> Le Tue, 8 Oct 2013 15:43:40 -0400, Benjamin Peterson a ?crit : > 2013/10/8 R. David Murray : > > In this context, if we'd been *really* smart-lazy in CPython > > development, we'd have kept the memory and startup-time and...well, > > we probably do pretty well on CPU actually...smaller, so that when > > smartphones came along Python would have been the first high level > > language used on them, because it fit. Then we'd all be able to be > > *much* lazier now :) > > Even on desktop, startup time leaves a lot to be desired. That's true. Anyone have any ideas to improve it? From solipsis at pitrou.net Wed Oct 9 10:30:05 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 9 Oct 2013 10:30:05 +0200 Subject: [Python-Dev] Reduce memory footprint of Python References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <5254809D.4080605@canterbury.ac.nz> Message-ID: <20131009103005.36927c72@pitrou.net> Le Wed, 09 Oct 2013 11:01:01 +1300, Greg Ewing a ?crit : > R. David Murray wrote: > > I can give you one data point: a mobile platform that (currently) > > uses Python3, and does not use linecache because of how much memory > > it consumes. > > Wouldn't a sensible approach be to discard the linecache > when you've finished generating a traceback? You're not > likely to be generating tracebacks often enough to be > worth keeping it from one to the next. The linecache is also used by the warnings module. I think the sensiblest approach would be to limit its size (LRU-style). Regards Antoine. From ncoghlan at gmail.com Wed Oct 9 11:46:14 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 9 Oct 2013 19:46:14 +1000 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52549646.5060909@hastings.org> References: <52549646.5060909@hastings.org> Message-ID: The PEP needs to state more clearly up front (preferably in the title) that it's about *reserving* a Python level syntax that matches the syntax we worked out for Argument Clinic at PyCon US. Explicitly stating that the requirements that drive the Argument Clinic design are to support the signature of all current CPython builtins and extension modules would also be helpful. Generally, it needs to be a bit clearer that the intent of the PEP isn't to say "let's do this", it's to be explicit that acceptance of the Argument Clinic PEP severely constrains the design space for possible solutions if we ever *did* implement Python level support for positional only arguments. A few other specific points: - parameters in optional groups should just have an implied "=None" that can be overriden if desired. - parsing positional only args in Python is actually pretty easy in most cases, as you can just use a second private function to do the heavy lifting. - a simpler variant that omits the grouping support and only allows optional parameters on the right should also be reserved - explicitly delegate to the argument clinic PEP for the impact on inspect.Signature Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Oct 9 12:03:32 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Oct 2013 11:03:32 +0100 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: On 9 October 2013 10:46, Nick Coghlan wrote: > The PEP needs to state more clearly up front (preferably in the title) that > it's about *reserving* a Python level syntax that matches the syntax we > worked out for Argument Clinic at PyCon US. Explicitly stating that the > requirements that drive the Argument Clinic design are to support the > signature of all current CPython builtins and extension modules would also > be helpful. > > Generally, it needs to be a bit clearer that the intent of the PEP isn't to > say "let's do this", it's to be explicit that acceptance of the Argument > Clinic PEP severely constrains the design space for possible solutions if we > ever *did* implement Python level support for positional only arguments. I'm confused about the expectations here. To be specific, I hate the syntax, I find the proliferation of punctuation messy and unhelpful. But if it's only for documentation/internal use, I don't really mind - I can see that we need some way to express this stuff, and TBH I don't care enough to make a fuss either way. But if the intention is that any attempt to add the functionality to Python will be constrained to use the syntax here, surely that means we need to have all of the usability arguments and bikeshedding *now*. otherwise the opportunity is lost? Personally, I'm -1 on the idea that this acts as any sort of precedent or design constraint for a Python-level implementation of the functionality, and -0 on it as a syntax for Argument Clinic, and as a documentation convention. And either way, I agree with Nick that the PEP needs to be very clear about its scope and impact. Paul From solipsis at pitrou.net Wed Oct 9 12:20:18 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 9 Oct 2013 12:20:18 +0200 Subject: [Python-Dev] type.__subclasses__() doesn't work Message-ID: <20131009122018.58d169c6@pitrou.net> Hello, Just noticed the following quirk: >>> type.__subclasses__() Traceback (most recent call last): File "", line 1, in TypeError: descriptor '__subclasses__' of 'type' object needs an argument Yet it would be nice to know about the subclasses of type. Regards Antoine. From solipsis at pitrou.net Wed Oct 9 12:29:56 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 9 Oct 2013 12:29:56 +0200 Subject: [Python-Dev] Python startup time References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> Message-ID: <20131009122956.5c1ae34c@pitrou.net> Le Wed, 9 Oct 2013 10:29:30 +0200, Antoine Pitrou a ?crit : > Le Tue, 8 Oct 2013 15:43:40 -0400, > Benjamin Peterson a ?crit : > > > 2013/10/8 R. David Murray : > > > In this context, if we'd been *really* smart-lazy in CPython > > > development, we'd have kept the memory and startup-time > > > and...well, we probably do pretty well on CPU actually...smaller, > > > so that when smartphones came along Python would have been the > > > first high level language used on them, because it fit. Then > > > we'd all be able to be *much* lazier now :) > > > > Even on desktop, startup time leaves a lot to be desired. > > That's true. Anyone have any ideas to improve it? It's difficult to identify significant contributors but some possible factors: - marshal.loads() has become twice slower in 3.x (compared to 2.7) - instantiating a class is slow (type('foo', (), {}) takes around 25ms here) Regards Antoine. From steve at pearwood.info Wed Oct 9 13:34:12 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 9 Oct 2013 22:34:12 +1100 Subject: [Python-Dev] type.__subclasses__() doesn't work In-Reply-To: <20131009122018.58d169c6@pitrou.net> References: <20131009122018.58d169c6@pitrou.net> Message-ID: <20131009113412.GD7989@ando> On Wed, Oct 09, 2013 at 12:20:18PM +0200, Antoine Pitrou wrote: > > Hello, > > Just noticed the following quirk: > > >>> type.__subclasses__() > Traceback (most recent call last): > File "", line 1, in > TypeError: descriptor '__subclasses__' of 'type' object needs an argument > > Yet it would be nice to know about the subclasses of type. py> type.__subclasses__(type) [, ] -- Steven From __peter__ at web.de Wed Oct 9 14:22:29 2013 From: __peter__ at web.de (Peter Otten) Date: Wed, 09 Oct 2013 14:22:29 +0200 Subject: [Python-Dev] type.__subclasses__() doesn't work References: <20131009122018.58d169c6@pitrou.net> <20131009113412.GD7989@ando> Message-ID: Steven D'Aprano wrote: > On Wed, Oct 09, 2013 at 12:20:18PM +0200, Antoine Pitrou wrote: >> >> Hello, >> >> Just noticed the following quirk: >> >> >>> type.__subclasses__() >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: descriptor '__subclasses__' of 'type' object needs an argument >> >> Yet it would be nice to know about the subclasses of type. > > py> type.__subclasses__(type) > [, ] The underlying problem seems to be that there is no helper function to bypass the instance attribute. Compare: >>> class T(type): ... def __len__(self): return 0 ... >>> class A(metaclass=T): ... def __len__(self): return 1 ... >>> A.__len__() Traceback (most recent call last): File "", line 1, in TypeError: __len__() missing 1 required positional argument: 'self' >>> len(A) 0 So should there be a subclasses() function, in the operator module perhaps? From hrvoje.niksic at avl.com Wed Oct 9 14:31:37 2013 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Wed, 9 Oct 2013 14:31:37 +0200 Subject: [Python-Dev] type.__subclasses__() doesn't work In-Reply-To: References: <20131009122018.58d169c6@pitrou.net> <20131009113412.GD7989@ando> Message-ID: <52554CA9.90405@avl.com> On 10/09/2013 02:22 PM, Peter Otten wrote: >> py> type.__subclasses__(type) >> [, ] > > The underlying problem seems to be that there is no helper function to > bypass the instance attribute. Note that the problem is specific to the "type" type, which is its own metatype. With other types that get __subclasses__ from type, is no problem with just calling __subclasses__(): >>> int.__subclasses__() [] From benjamin at python.org Wed Oct 9 14:38:46 2013 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 9 Oct 2013 08:38:46 -0400 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: 2013/10/9 Nick Coghlan : > The PEP needs to state more clearly up front (preferably in the title) that > it's about *reserving* a Python level syntax that matches the syntax we > worked out for Argument Clinic at PyCon US. Explicitly stating that the > requirements that drive the Argument Clinic design are to support the > signature of all current CPython builtins and extension modules would also > be helpful. > > Generally, it needs to be a bit clearer that the intent of the PEP isn't to > say "let's do this", it's to be explicit that acceptance of the Argument > Clinic PEP severely constrains the design space for possible solutions if we > ever *did* implement Python level support for positional only arguments. Why does a syntax need to be reserved? Documentation conventions and the syntax of internal tools like argument clinic may be changed any time we like. -- Regards, Benjamin From benjamin at python.org Wed Oct 9 14:39:40 2013 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 9 Oct 2013 08:39:40 -0400 Subject: [Python-Dev] Python startup time In-Reply-To: <20131009122956.5c1ae34c@pitrou.net> References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> Message-ID: 2013/10/9 Antoine Pitrou : > Le Wed, 9 Oct 2013 10:29:30 +0200, > Antoine Pitrou a ?crit : >> Le Tue, 8 Oct 2013 15:43:40 -0400, >> Benjamin Peterson a ?crit : >> >> > 2013/10/8 R. David Murray : >> > > In this context, if we'd been *really* smart-lazy in CPython >> > > development, we'd have kept the memory and startup-time >> > > and...well, we probably do pretty well on CPU actually...smaller, >> > > so that when smartphones came along Python would have been the >> > > first high level language used on them, because it fit. Then >> > > we'd all be able to be *much* lazier now :) >> > >> > Even on desktop, startup time leaves a lot to be desired. >> >> That's true. Anyone have any ideas to improve it? > > It's difficult to identify significant contributors but some possible > factors: > - marshal.loads() has become twice slower in 3.x (compared to 2.7) > - instantiating a class is slow (type('foo', (), {}) takes around 25ms > here) There's also the increasing number of modules (Python or otherwise) that have to be loaded on startup. -- Regards, Benjamin From solipsis at pitrou.net Wed Oct 9 14:48:05 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 9 Oct 2013 14:48:05 +0200 Subject: [Python-Dev] Python startup time References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> Message-ID: <20131009144805.48ff66ea@pitrou.net> Le Wed, 9 Oct 2013 08:39:40 -0400, Benjamin Peterson a ?crit : > 2013/10/9 Antoine Pitrou : > > Le Wed, 9 Oct 2013 10:29:30 +0200, > > Antoine Pitrou a ?crit : > >> Le Tue, 8 Oct 2013 15:43:40 -0400, > >> Benjamin Peterson a ?crit : > >> > >> > 2013/10/8 R. David Murray : > >> > > In this context, if we'd been *really* smart-lazy in CPython > >> > > development, we'd have kept the memory and startup-time > >> > > and...well, we probably do pretty well on CPU > >> > > actually...smaller, so that when smartphones came along Python > >> > > would have been the first high level language used on them, > >> > > because it fit. Then we'd all be able to be *much* lazier > >> > > now :) > >> > > >> > Even on desktop, startup time leaves a lot to be desired. > >> > >> That's true. Anyone have any ideas to improve it? > > > > It's difficult to identify significant contributors but some > > possible factors: > > - marshal.loads() has become twice slower in 3.x (compared to 2.7) > > - instantiating a class is slow (type('foo', (), {}) takes around > > 25ms here) > > There's also the increasing number of modules (Python or otherwise) > that have to be loaded on startup. See e.g. http://bugs.python.org/issue9548 Regards Antoine. From christian at python.org Wed Oct 9 14:54:24 2013 From: christian at python.org (Christian Heimes) Date: Wed, 09 Oct 2013 14:54:24 +0200 Subject: [Python-Dev] Python startup time In-Reply-To: References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> Message-ID: <52555200.4080301@python.org> Am 09.10.2013 14:39, schrieb Benjamin Peterson: > There's also the increasing number of modules (Python or otherwise) > that have to be loaded on startup. I have gathered some stats: Python 3.4: >>> len(sys.modules) 56 >>> len(list(name for name in sys.modules if name not in sys.builtin_module_names)) 36 >>> sorted(sys.modules) ['__main__', '_codecs', '_collections', '_frozen_importlib', '_functools', '_heapq', '_imp', '_io', '_operator', '_sitebuiltins', '_sre', '_stat', '_sysconfigdata', '_thread', '_warnings', '_weakref', '_weakrefset', 'abc', 'atexit', 'builtins', 'codecs', 'collections', 'collections.abc', 'copyreg', 'encodings', 'encodings.aliases', 'encodings.latin_1', 'encodings.utf_8', 'errno', 'functools', 'genericpath', 'heapq', 'io', 'itertools', 'keyword', 'marshal', 'operator', 'os', 'os.path', 'posix', 'posixpath', 're', 'readline', 'reprlib', 'rlcompleter', 'signal', 'site', 'sre_compile', 'sre_constants', 'sre_parse', 'stat', 'sys', 'sysconfig', 'types', 'weakref', 'zipimport'] Python does quite alot of syscalls, too: $ strace ./python -E -c '' 2>&1 | grep stat | wc -l 236 $ strace ./python -E -c '' 2>&1 | grep open | wc -l 53 Python 2.7 is a bit better: >>> len(sys.modules) 41 >>> len(list(name for name in sys.modules if name not in sys.builtin_module_names)) 29 From larry at hastings.org Wed Oct 9 15:32:10 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 15:32:10 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: <52555ADA.8020107@hastings.org> On 10/09/2013 02:38 PM, Benjamin Peterson wrote: > Why does a syntax need to be reserved? Documentation conventions and > the syntax of internal tools like argument clinic may be changed any > time we like. If Argument Clinic is accepted, and ships with Python, my expectation is that it would cease being an internal tool, and would start being used by external developers. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Oct 9 15:36:48 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 9 Oct 2013 15:36:48 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters References: <52549646.5060909@hastings.org> <52555ADA.8020107@hastings.org> Message-ID: <20131009153648.60354972@pitrou.net> Le Wed, 09 Oct 2013 15:32:10 +0200, Larry Hastings a ?crit : > On 10/09/2013 02:38 PM, Benjamin Peterson wrote: > > Why does a syntax need to be reserved? Documentation conventions and > > the syntax of internal tools like argument clinic may be changed any > > time we like. > > If Argument Clinic is accepted, and ships with Python, my expectation > is that it would cease being an internal tool, and would start being > used by external developers. I think it would be better to exercise it first as an internal tool, eliminate any possible quirks and issues, and *then* perhaps expose it as a public API. By the way, do you think it'll be ready for 3.4? Regards Antoine. From larry at hastings.org Wed Oct 9 15:51:13 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 15:51:13 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: <52555F51.8030307@hastings.org> On 10/09/2013 07:48 AM, Eric Snow wrote: > The PEP doesn't mention anything about inspect.Signature, which > already supports positional-only parameters. It should. Well, inspect.Signature support for positional-only parameters is currently insufficient to express all of Python's builtins. As I mentioned in my long reply to Benjamin just now, I got Brett to agree that I could add a "group" field to inspect.Parameter to indicate a positional parameter's "option group". I guess this could be part of the PEP. > This also has an impact on other Python implementations, where > maintaining the same positional-only semantics for "builtins" is > complicated (at least a little) by the lack of a positional-only > syntax in Python. Again, I don't expect this syntax to be implemented any time soon. But this does raise a mild sore point: Maciej and Armin Rigo tell me that PyPy's implementation of range() looks like this: def range(x, y=None, step=None): The "None" above is not the Python None, it's RPython's internal None, so it's not available to Python programs. But this does mean range() in PyPy accepts keyword parameters, and in fact this works: range(9, step=2) That's way illegal in CPython. > This is for the sake of only a couple unfortunate builtins, right? > Parameter grouping will undoubtedly be the most controversial part of > this PEP. I'm not convinced this is a capability we need to encourage > in any way. Is there some alternate one-off we can use for range, > addch, et al.? I look forward to an alternate suggestion. This is the least-bad thing I could come up with. > What are the names bound to for unfulfilled optional groups? Not > bound (yuck)? What is wrong with default values within optional > groups (other than how messy the syntax starts to become)? Please see my reply to Steven D'Aprano from about twelve hours ago. > What about this: > > def spam([a,] b, [c,] d, /): Quoting the PEP: "All "required" positional-only parameters must be contiguous." b and d are discontinuous. So this is illegal syntax. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Wed Oct 9 15:56:13 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 09 Oct 2013 15:56:13 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52555ADA.8020107@hastings.org> References: <52549646.5060909@hastings.org> <52555ADA.8020107@hastings.org> Message-ID: Am 09.10.2013 15:32, schrieb Larry Hastings: > On 10/09/2013 02:38 PM, Benjamin Peterson wrote: >> Why does a syntax need to be reserved? Documentation conventions and >> the syntax of internal tools like argument clinic may be changed any >> time we like. > > If Argument Clinic is accepted, and ships with Python, my expectation is that it > would cease being an internal tool, and would start being used by external > developers. Definitely it should be treated as internal for at least one minor version, so that we can iron out whatever weaknesses we may find. If others start using it during that time and complain, tough luck. Georg From larry at hastings.org Wed Oct 9 15:26:02 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 15:26:02 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: <5255596A.4050003@hastings.org> On 10/09/2013 03:31 AM, Benjamin Peterson wrote: > 2013/10/8 Larry Hastings : >> This PEP proposes a backwards-compatible syntax that should >> permit implementing any builtin in pure Python code. > This is rather too strong. You can certainly implement them; you just > have to implement the argument parsing yourself. Can you suggest an alternative wording? What I was trying to say was, you cannot express many builtin's signatures in pure Python code. I guess "implement" is the bad word here--you can implement those semantics, but you cannot express it syntactically. So how about This PEP proposes a backwards-compatible syntax that should permit expressing the signature of any builtin as a Python function. > Python's > call/signature syntax is already extremely expressive, and resolving > call arguments to formal parameters is already a complicated (and > slow) process. Well, I did propose that the syntax not actually be implemented. > Implementing functions with such strange argument > semantics is hardly common enough to justify the whole grouping syntax > proposed in this PEP. -1 to that. I think I can live with "/", but > YANGTNI still. On the contrary, I can tell you exactly when I'm going to need it. What happens when you run "help(foo)"? pydoc examines the signature of the function and generates the first line, the function's prototype. For builtins, pydoc currently sees that there is no introspection information and skips the generation, which is why docstrings for builtins by convention have a handwritten prototype as their first line. Let's assume that the Argument Clinic PEP is accepted, and my current implementation goes in, and I add introspection metadata for builtins. Once we have this metadata, pydoc will use it and generate that first line for builtins itself. Now: what happens when you run "help(curses.window.addch)"? * My first problem: Argument Clinic has to allow the core C developer to express the signature of addch. The syntax is Python-like, if you change the commas into newlines and enforce a few extra rules regarding leading whitespace. But Python's existing syntax can't communicate the semantics of addch. Nobody fought me on Clinic's need to express this sort of function, though I had plenty of bikeshedding on the syntax. Then Guido dictated a syntax at PyCon US 2013, and that was that. My second problem: I have to communicate the metadata in Argument Clinic's output to the CPython runtime. Almost nobody cares what this looks like, as it will be an implementation detail. But I suspect there will be one or two places where you can't use Clinic yet, so you'll be writing this metadata by hand. So it needs to be cogent and human readable. Barry Warsaw suggested that the best syntax--not only most readable, but most compact--was to just use Python's own syntax. Again, existing Python syntax can't communicate the semantics of addch. So I expect to use this syntax for the metadata of builtins. My third problem: I have to represent the signature of these functions with inspect.Signature. Parameter objects can be marked as having a "kind" of inspect.Parameter.POSITIONAL_ONLY, so inspect has at least tacit, vestigial support for positional-only parameters. In practice this is insufficient to express the semantics of addch. IIRC Brett said I could add a "group" field to inspect.Parameter indicating the "option group" of a parameter. (There's no way I'm modifying inspect.getargspec() or inspect.getfullargspec() to represent positional-only parameters.) My fourth problem: I have to modify pydoc so it uses inspect.signature() instead of inspect.getfullargspec(), then understands the new semantics of positional-only parameters. And then it needs to generate that first line. But what should that line look like--what should be its syntax? It's written in Python syntax, but Python existing syntax can't communicate the semantics of addch. So I expect to use this syntax for the first line generated by pydoc. And there you have it. I have to represent the semantics of addch in four places, and three of those are textual. Two are implementation details, but in practice they will use Python syntax. The third is documentation but should also be Python syntax. My goal with the PEP is to codify existing practice. I find it a bit perverse that we have functions callable in Python with call signatures one cannot express syntactically in Python, and with undocumented semantics (their parameters are positional-only but this is not mentioned). I understand it's all kind of a historical accident, but it is absolutely part of Python and it's not going away. My proposed syntax is a little complex--but complex is better than complicated and inconsistent and undocumented and inconvenient, which is what we have now. //arr//y/ * The callable is not simply curses.window.addch; window is a class or something. But addch is definitely a callable you can get your hands on in Python. -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Wed Oct 9 15:56:49 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 15:56:49 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: <525560A1.2050704@hastings.org> On 10/09/2013 07:15 AM, Georg Brandl wrote: > We have quite a large > amount of C functions with positional-only parameters. Adding a "/" > to each of those is a) a tedious task and b) probably not helpful for > a lot of people: I volunteer to ensure that the "/"s are added if this PEP is accepted. (I expect I'd do it myself, but who knows, maybe a volunteer would appear out of the woodwork.) > ... and the "undefined" singleton just smells wrong. Another way > to spell "None" is just asking for trouble. It has to be a different value from "None", for the same reasons that all those other different-spellings-for-None (e.g. inspect.Parameter.empty) exist. I realize you are -1 on the proposal in general, but I'd be very interested if you could propose an alternate approach where I didn't need "a new spelling for None" as you put it. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ezio.melotti at gmail.com Wed Oct 9 16:14:12 2013 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Wed, 9 Oct 2013 17:14:12 +0300 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: On Wed, Oct 9, 2013 at 8:15 AM, Georg Brandl wrote: > > [...] > > Rather, I would try to make as many C functions as possible "regular", See http://bugs.python.org/issue8706 and http://bugs.python.org/issue8350 Best Regards, Ezio Melotti From larry at hastings.org Wed Oct 9 16:15:36 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 16:15:36 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: <52556508.9030002@hastings.org> On 10/09/2013 11:46 AM, Nick Coghlan wrote: > Generally, it needs to be a bit clearer that the intent of the PEP > isn't to say "let's do this", it's to be explicit that acceptance of > the Argument Clinic PEP severely constrains the design space for > possible solutions if we ever *did* implement Python level support for > positional only arguments. Can you suggest an edit that would make you happy? > - parameters in optional groups should just have an implied "=None" > that can be overriden if desired. > No no no! You can't have a default value, and you definitely couldn't use None. range() decides what its arguments mean based on how many arguments it receives. If I understand your proposal correctly, you suggest that range(None, 5) would return the same result as range(5) But that's simply not how it works. If you want to propose changing the semantics of range, go ahead, I'll stand back. > - a simpler variant that omits the grouping support and only allows > optional parameters on the right should also be reserved > If that syntax is a subset of this syntax, and this syntax is reserved, then by extension we would automatically reserve that syntax too. In any other circumstance (this PEP is rejected, the simpler variant uses a different syntax) the simpler syntax should get a new PEP. I'll just say that that syntax is insufficient to implement existing functions we all know and love (addch, range). > - explicitly delegate to the argument clinic PEP for the impact on > inspect.Signature > I'd actually rather do it the other way 'round. This PEP is a better place for it. Even though I think the Clinic PEP has a higher chance of being accepted ;-) //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Wed Oct 9 16:24:06 2013 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 9 Oct 2013 10:24:06 -0400 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <5255596A.4050003@hastings.org> References: <52549646.5060909@hastings.org> <5255596A.4050003@hastings.org> Message-ID: 2013/10/9 Larry Hastings : > > On 10/09/2013 03:31 AM, Benjamin Peterson wrote: > > 2013/10/8 Larry Hastings : > > This PEP proposes a backwards-compatible syntax that should > permit implementing any builtin in pure Python code. > > This is rather too strong. You can certainly implement them; you just > have to implement the argument parsing yourself. > > > Can you suggest an alternative wording? What I was trying to say was, you > cannot express many builtin's signatures in pure Python code. I guess > "implement" is the bad word here--you can implement those semantics, but you > cannot express it syntactically. So how about > > This PEP proposes a backwards-compatible syntax that should permit > expressing the signature of any builtin as a Python function. That's fine. > > > Python's > call/signature syntax is already extremely expressive, and resolving > call arguments to formal parameters is already a complicated (and > slow) process. > > > Well, I did propose that the syntax not actually be implemented. Yes, that's good. > > > Implementing functions with such strange argument > semantics is hardly common enough to justify the whole grouping syntax > proposed in this PEP. -1 to that. I think I can live with "/", but > YANGTNI still. > > > On the contrary, I can tell you exactly when I'm going to need it. > > What happens when you run "help(foo)"? pydoc examines the signature of the > function and generates the first line, the function's prototype. For > builtins, pydoc currently sees that there is no introspection information > and skips the generation, which is why docstrings for builtins by convention > have a handwritten prototype as their first line. > > Let's assume that the Argument Clinic PEP is accepted, and my current > implementation goes in, and I add introspection metadata for builtins. Once > we have this metadata, pydoc will use it and generate that first line for > builtins itself. > > Now: what happens when you run "help(curses.window.addch)"? * > > My first problem: Argument Clinic has to allow the core C developer to > express the signature of addch. The syntax is Python-like, if you change > the commas into newlines and enforce a few extra rules regarding leading > whitespace. But Python's existing syntax can't communicate the semantics of > addch. Nobody fought me on Clinic's need to express this sort of function, > though I had plenty of bikeshedding on the syntax. Then Guido dictated a > syntax at PyCon US 2013, and that was that. > > My second problem: I have to communicate the metadata in Argument Clinic's > output to the CPython runtime. Almost nobody cares what this looks like, as > it will be an implementation detail. But I suspect there will be one or two > places where you can't use Clinic yet, so you'll be writing this metadata by > hand. So it needs to be cogent and human readable. Barry Warsaw suggested > that the best syntax--not only most readable, but most compact--was to just > use Python's own syntax. Again, existing Python syntax can't communicate > the semantics of addch. So I expect to use this syntax for the metadata of > builtins. > > My third problem: I have to represent the signature of these functions with > inspect.Signature. Parameter objects can be marked as having a "kind" of > inspect.Parameter.POSITIONAL_ONLY, so inspect has at least tacit, vestigial > support for positional-only parameters. In practice this is insufficient to > express the semantics of addch. IIRC Brett said I could add a "group" field > to inspect.Parameter indicating the "option group" of a parameter. (There's > no way I'm modifying inspect.getargspec() or inspect.getfullargspec() to > represent positional-only parameters.) > > My fourth problem: I have to modify pydoc so it uses inspect.signature() > instead of inspect.getfullargspec(), then understands the new semantics of > positional-only parameters. And then it needs to generate that first line. > But what should that line look like--what should be its syntax? It's > written in Python syntax, but Python existing syntax can't communicate the > semantics of addch. So I expect to use this syntax for the first line > generated by pydoc. > > And there you have it. I have to represent the semantics of addch in four > places, and three of those are textual. Two are implementation details, but > in practice they will use Python syntax. The third is documentation but > should also be Python syntax. > > > My goal with the PEP is to codify existing practice. I find it a bit > perverse that we have functions callable in Python with call signatures one > cannot express syntactically in Python, and with undocumented semantics > (their parameters are positional-only but this is not mentioned). I > understand it's all kind of a historical accident, but it is absolutely part > of Python and it's not going away. You can express any wild argument semantics you want in Python with *args and **kwargs. If the "strange" C functions like range() and addch() get signature objects, I don't think it's the end of world if it looks like a Python *args, **kwargs signature. > > My proposed syntax is a little complex--but complex is better than > complicated and inconsistent and undocumented and inconvenient, which is > what we have now. Certainly the argument conventions of these functions are not undocumented, so wonder what is. Also, inconvenient for what? What inconsistency problem does this PEP solve? -- Regards, Benjamin From g.brandl at gmx.net Wed Oct 9 16:26:21 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 09 Oct 2013 16:26:21 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <525560A1.2050704@hastings.org> References: <52549646.5060909@hastings.org> <525560A1.2050704@hastings.org> Message-ID: Am 09.10.2013 15:56, schrieb Larry Hastings: > On 10/09/2013 07:15 AM, Georg Brandl wrote: >> We have quite a large >> amount of C functions with positional-only parameters. Adding a "/" >> to each of those is a) a tedious task and b) probably not helpful for >> a lot of people: > > I volunteer to ensure that the "/"s are added if this PEP is accepted. (I > expect I'd do it myself, but who knows, maybe a volunteer would appear out of > the woodwork.) Yes, the much bigger concern is the second one. >> ... and the "undefined" singleton just smells wrong. Another way >> to spell "None" is just asking for trouble. > > It has to be a different value from "None", for the same reasons that all those > other different-spellings-for-None (e.g. inspect.Parameter.empty) exist. I know. Still, as a new builtin it will get used for all kinds of other purposes, especially when it has the name that other languages use for our None. Maybe you could get away with NotGiven or NotPassed (in parallel to NotImplemented). > I realize you are -1 on the proposal in general, but I'd be very interested if > you could propose an alternate approach where I didn't need "a new spelling for > None" as you put it. I think I would make Steven's proposed syntax mandatory: let the implementor of the function decide which value stands for "not given" -- just like we do in the C version, BTW. cheers, Georg From christian at python.org Wed Oct 9 16:30:12 2013 From: christian at python.org (Christian Heimes) Date: Wed, 09 Oct 2013 16:30:12 +0200 Subject: [Python-Dev] Python startup time In-Reply-To: <52555200.4080301@python.org> References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> <52555200.4080301@python.org> Message-ID: <52556874.4030707@python.org> Am 09.10.2013 14:54, schrieb Christian Heimes: > Am 09.10.2013 14:39, schrieb Benjamin Peterson: >> There's also the increasing number of modules (Python or otherwise) >> that have to be loaded on startup. We can easily peel off about 11 modules (re, sysconfig and their dependencies) with some changes to site.py: http://bugs.python.org/issue19205 Another low hanging fruit is the copyreg module: http://bugs.python.org/issue19209 The os module imports MutableMapping from collections.abc. That import adds collections, collections.abc and eight more modules. I'm not sure if we can do anything about it, though. Christian From larry at hastings.org Wed Oct 9 16:30:51 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 16:30:51 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <20131009153648.60354972@pitrou.net> References: <52549646.5060909@hastings.org> <52555ADA.8020107@hastings.org> <20131009153648.60354972@pitrou.net> Message-ID: <5255689B.3030804@hastings.org> On 10/09/2013 03:36 PM, Antoine Pitrou wrote: > I think it would be better to exercise [Argument Clinc] first as an internal tool, > eliminate any possible quirks and issues, and *then* perhaps expose it > as a public API. > > By the way, do you think it'll be ready for 3.4? Only if I can stop writing other PEPs and replying to their discussion threads! The implementation of Argument Clinic is done enough to be ready to go in. It doesn't support /everything/ it should. The main thing: it doesn't have enough predefined "converters" and "return converters". But those are pretty quick to add, so that problem wouldn't last long. It also doesn't allow mixing positional-only parameters and any other types of parameters, so for example the dict constructor is currently inexpressable. And (as always) it needs more tests. But I'm finally happy with the internal interfaces, so that's something. The next step: polish the PEP, get final feedback, and submit it for judgment. Then start writing / polishing the documentation. If the PEP is accepted, get the implementation reviewed, and (fingers crossed) checked in. The implementation is available here for your reviewing pleasure: https://bitbucket.org/larry/python-clinic/ It got a /cursory/ high-level code review from Brett; we spent six or eight hours going over it at PyCon CA. He gave it a thumbs up but definitely wanted more coverage in the unit tests. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Wed Oct 9 16:33:39 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 9 Oct 2013 08:33:39 -0600 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: On Oct 9, 2013 4:04 AM, "Paul Moore" wrote: > But if the intention is that any attempt to add the functionality to > Python will be constrained to use the syntax here, surely that means > we need to have all of the usability arguments and bikeshedding *now*. > otherwise the opportunity is lost? I have the same concern. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Oct 9 16:38:15 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Oct 2013 15:38:15 +0100 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: On 9 October 2013 13:38, Benjamin Peterson wrote: >> Generally, it needs to be a bit clearer that the intent of the PEP isn't to >> say "let's do this", it's to be explicit that acceptance of the Argument >> Clinic PEP severely constrains the design space for possible solutions if we >> ever *did* implement Python level support for positional only arguments. > > Why does a syntax need to be reserved? Documentation conventions and > the syntax of internal tools like argument clinic may be changed any > time we like. For that matter, why does the syntax used by Argument Clinic have to be the same as whatever future syntax is used in Python? If indeed, any ever is? What benefit do we get given the cost (rushing in a syntax that nobody is actually convinced we even need in Python yet). In other words, why does PEP this need to be separated out from the Argument Clinic PEP at all? Paul From p.f.moore at gmail.com Wed Oct 9 16:40:08 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Oct 2013 15:40:08 +0100 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <5255689B.3030804@hastings.org> References: <52549646.5060909@hastings.org> <52555ADA.8020107@hastings.org> <20131009153648.60354972@pitrou.net> <5255689B.3030804@hastings.org> Message-ID: On 9 October 2013 15:30, Larry Hastings wrote: > Only if I can stop writing other PEPs and replying to their discussion > threads! So once again, why is this new PEP needed? :-) Paul From ajm at flonidan.dk Wed Oct 9 16:45:44 2013 From: ajm at flonidan.dk (Anders J. Munch) Date: Wed, 9 Oct 2013 14:45:44 +0000 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52555F51.8030307@hastings.org> References: <52549646.5060909@hastings.org> <52555F51.8030307@hastings.org> Message-ID: <81E45292CF3BC5478E88DF55E216AD2F03B8C3C4@FLONIDANPOST.flonidan.net> Larry Hastings wrote: > I look forward to an alternate suggestion. This is the least-bad > thing I could come up with. How about a naming convention instead, where using a leading underscore in a parameter name is a hint that it is positional-only. For example, the docstring of sorted: sorted(iterable, key=None, reverse=False) --> new sorted list would become: sorted(_iterable, key=None, reverse=False) --> new sorted list It seems more intuitive than the slash, and requires no change to the toolchain. Although, it may collide slightly with code that uses a leading underscore to indicate "implementation detail" for a parameter with a default value, but a quick scan of the std. lib. suggests that that's very rare. regards, Anders -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Oct 9 16:49:15 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 9 Oct 2013 16:49:15 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters References: <52549646.5060909@hastings.org> <52555ADA.8020107@hastings.org> <20131009153648.60354972@pitrou.net> <5255689B.3030804@hastings.org> Message-ID: <20131009164915.267c27c2@pitrou.net> Le Wed, 9 Oct 2013 15:40:08 +0100, Paul Moore a ?crit : > On 9 October 2013 15:30, Larry Hastings wrote: > > Only if I can stop writing other PEPs and replying to their > > discussion threads! > > So once again, why is this new PEP needed? :-) Procrastination perhaps ;-) > Paul From larry at hastings.org Wed Oct 9 16:53:15 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 16:53:15 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> <525560A1.2050704@hastings.org> Message-ID: <52556DDB.8040203@hastings.org> On 10/09/2013 04:26 PM, Georg Brandl wrote: >> I realize you are -1 on the proposal in general, but I'd be very interested if >> you could propose an alternate approach where I didn't need "a new spelling for >> None" as you put it. > I think I would make Steven's proposed syntax mandatory: let the implementor > of the function decide which value stands for "not given" -- just like we do > in the C version, BTW. But that's not how addch works. addch counts how many arguments it received; if it is called with one or two, it does one thing, and if it's called with three or four it does something else. You can't duplicate these semantics with Similarly, you can't accurately express the semantics of range's arguments using default values. PyPy's approach is approximately like this: def range(x, y=None, step=None): step = 1 if step is None else step if y is not None: start, stop = x, y else: start, stop = 0, x But now introspection information on range() is inaccurate and unhelpful. (Not to mention, they allow specifying step without specifying y, by using keyword arguments.) My goal in writing the PEP was to codify existing practice, which meant reflecting these (annoying!) corner cases accurately. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Wed Oct 9 16:57:24 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 16:57:24 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> <5255596A.4050003@hastings.org> Message-ID: <52556ED4.5080002@hastings.org> On 10/09/2013 04:24 PM, Benjamin Peterson wrote: >> My proposed syntax is a little complex--but complex is better than >> complicated and inconsistent and undocumented and inconvenient, which is >> what we have now. > Certainly the argument conventions of these functions are not > undocumented, so wonder what is. Also, inconvenient for what? What > inconsistency problem does this PEP solve? Whether or not a particular function accepts keyword arguments is undocumented. I have in the past been inconvenienced by this not being clear. The documentation uses two approaches for documenting option groups, effectively at random, which is inconsistent. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Wed Oct 9 17:07:08 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 17:07:08 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: <5255711C.80208@hastings.org> On 10/09/2013 04:38 PM, Paul Moore wrote: > For that matter, why does the syntax used by Argument Clinic have to > be the same as whatever future syntax is used in Python? If indeed, > any ever is? What benefit do we get given the cost (rushing in a > syntax that nobody is actually convinced we even need in Python yet). > > In other words, why does PEP this need to be separated out from the > Argument Clinic PEP at all? I propose to use this syntax in three places: 1. As input to Argument Clinic. 2. As input to the Python C API, to express the metadata for builtins. 3. As the first line of output of pydoc (aka "help()") when reviewing a function. Of these, 3 is visible to users. Theoretically all of them could use different syntaxes. But consistency is desirable, so it would be better if these syntaxes were as similar as possible. Python syntax works well for all three syntaxes, except it does not support positional-only parameters. It seemed reasonable to codify twenty-year old practice in a syntax extension that I could use in all of the above. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Oct 9 17:07:36 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Oct 2013 16:07:36 +0100 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52556DDB.8040203@hastings.org> References: <52549646.5060909@hastings.org> <525560A1.2050704@hastings.org> <52556DDB.8040203@hastings.org> Message-ID: On 9 October 2013 15:53, Larry Hastings wrote: > My goal in writing the PEP was to codify existing practice, which meant > reflecting these (annoying!) corner cases accurately. That's fair. But I would argue that we very much don't want to encourage anyone ever duplicating that practice with new functions, which is why I don't think that this syntax should ever be implemented in the form described in Python. OTOH, there *may* be reasons in the future for wanting to support positional-only syntax in Python - that design should not be constrained by the need (which is Argument Clinic specific, AIUI) to cater for existing builtins' practice. Paul From g.brandl at gmx.net Wed Oct 9 17:09:47 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 09 Oct 2013 17:09:47 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52556DDB.8040203@hastings.org> References: <52549646.5060909@hastings.org> <525560A1.2050704@hastings.org> <52556DDB.8040203@hastings.org> Message-ID: Am 09.10.2013 16:53, schrieb Larry Hastings: > On 10/09/2013 04:26 PM, Georg Brandl wrote: >>> I realize you are -1 on the proposal in general, but I'd be very interested if >>> you could propose an alternate approach where I didn't need "a new spelling for >>> None" as you put it. >> I think I would make Steven's proposed syntax mandatory: let the implementor >> of the function decide which value stands for "not given" -- just like we do >> in the C version, BTW. > > But that's not how addch works. addch counts how many arguments it received; if > it is called with one or two, it does one thing, and if it's called with three > or four it does something else. You can't duplicate these semantics with Well, why should a function that requires counting arguments even in C code not be implemented using *args? Documentation can cope fine, in the case of range() with two signatures. > Similarly, you can't accurately express the semantics of range's arguments using > default values. PyPy's approach is approximately like this: > > def range(x, y=None, step=None): > step = 1 if step is None else step > if y is not None: > start, stop = x, y > else: > start, stop = 0, x > > But now introspection information on range() is inaccurate and unhelpful. (Not > to mention, they allow specifying step without specifying y, by using keyword > arguments.) > > My goal in writing the PEP was to codify existing practice, which meant > reflecting these (annoying!) corner cases accurately. These are only two, one of them being exceedingly obscure. The hobgoblin comes to mind :) cheers, Georg From larry at hastings.org Wed Oct 9 17:09:26 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 17:09:26 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <81E45292CF3BC5478E88DF55E216AD2F03B8C3C4@FLONIDANPOST.flonidan.net> References: <52549646.5060909@hastings.org> <52555F51.8030307@hastings.org> <81E45292CF3BC5478E88DF55E216AD2F03B8C3C4@FLONIDANPOST.flonidan.net> Message-ID: <525571A6.10404@hastings.org> On 10/09/2013 04:45 PM, Anders J. Munch wrote: > > Larry Hastings wrote: > > > I look forward to an alternate suggestion. This is the least-bad > > > thing I could come up with. > > How about a naming convention instead, where using a leading > > underscore in a parameter name is a hint that it is positional-only. > > For example, the docstring of sorted: > > sorted(iterable, key=None, reverse=False) --> new sorted list > > would become: > > sorted(_iterable, key=None, reverse=False) --> new sorted list > > It seems more intuitive than the slash, and requires no change to the > > toolchain. > And the spelling for range()? curses.window.addch()? I suggest your proposed syntax is insufficient to codify existing practice. In any case, boy am I -1 on this. If you wish to continue this proposal further please write your own PEP. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Wed Oct 9 17:11:25 2013 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 9 Oct 2013 11:11:25 -0400 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52556ED4.5080002@hastings.org> References: <52549646.5060909@hastings.org> <5255596A.4050003@hastings.org> <52556ED4.5080002@hastings.org> Message-ID: 2013/10/9 Larry Hastings : > On 10/09/2013 04:24 PM, Benjamin Peterson wrote: > > My proposed syntax is a little complex--but complex is better than > complicated and inconsistent and undocumented and inconvenient, which is > what we have now. > > Certainly the argument conventions of these functions are not > undocumented, so wonder what is. Also, inconvenient for what? What > inconsistency problem does this PEP solve? > > > Whether or not a particular function accepts keyword arguments is > undocumented. I have in the past been inconvenienced by this not being > clear. The documentation uses two approaches for documenting option groups, > effectively at random, which is inconsistent. I suppose the question is whether people will find it easier to learn your syntax for documenting it, or just relearning it for particular functions whenever they attempt to use a kwarg for a position argument. I don't expect it will show up on many functions. -- Regards, Benjamin From ethan at stoneleaf.us Wed Oct 9 17:29:17 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 09 Oct 2013 08:29:17 -0700 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <525571A6.10404@hastings.org> References: <52549646.5060909@hastings.org> <52555F51.8030307@hastings.org> <81E45292CF3BC5478E88DF55E216AD2F03B8C3C4@FLONIDANPOST.flonidan.net> <525571A6.10404@hastings.org> Message-ID: <5255764D.4010806@stoneleaf.us> On 10/09/2013 08:09 AM, Larry Hastings wrote: > On 10/09/2013 04:45 PM, Anders J. Munch wrote: >> >> How about a naming convention instead, where using a leading >> underscore in a parameter name is a hint that it is positional-only. >> >> For example, the docstring of sorted: >> >> sorted(iterable, key=None, reverse=False) --> new sorted list >> >> would become: >> >> sorted(_iterable, key=None, reverse=False) --> new sorted list > > In any case, boy am I -1 on this. If you wish to continue this proposal further please write your own PEP. A leading underscore indicates a private implementation detail. The proposed signature now looks like we can provide a key and reverse, but only riskily provide the actual thing we want sorted. No warm fuzzy feelings there! ;) -1 -- ~Ethan~ From p.f.moore at gmail.com Wed Oct 9 18:05:12 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Oct 2013 17:05:12 +0100 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <5255711C.80208@hastings.org> References: <52549646.5060909@hastings.org> <5255711C.80208@hastings.org> Message-ID: On 9 October 2013 16:07, Larry Hastings wrote: > On 10/09/2013 04:38 PM, Paul Moore wrote: > > For that matter, why does the syntax used by Argument Clinic have to > be the same as whatever future syntax is used in Python? If indeed, > any ever is? What benefit do we get given the cost (rushing in a > syntax that nobody is actually convinced we even need in Python yet). > > In other words, why does PEP this need to be separated out from the > Argument Clinic PEP at all? > > > I propose to use this syntax in three places: > > As input to Argument Clinic. > As input to the Python C API, to express the metadata for builtins. > As the first line of output of pydoc (aka "help()") when reviewing a > function. > > Of these, 3 is visible to users. Theoretically all of them could use > different syntaxes. But consistency is desirable, so it would be better if > these syntaxes were as similar as possible. OK, all of those are reasonable points. I'm still -0 because I find the proposed syntax ugly, but I can't offer a better syntactic solution. My proposal is to hide the ugly syntax internally, and use prose documentation where users can see. You may disagree, that's fine. > Python syntax works well for > all three syntaxes, except it does not support positional-only parameters. > It seemed reasonable to codify twenty-year old practice in a syntax > extension that I could use in all of the above. I remain -1 on forcing "Python syntax" to support all of these odd corner cases (and positional-only is already a corner case, range/addch are seriously so). I don't see any reason for the proposal to mandate that the arguments made here should apply to any future changes to core Python syntax. Either we have the debate now, or we agree that the proposal doesn't apply to Python. Anyway, thanks for clarifying your position. I'm happy that I understand what you're proposing now, even if I don't agree with you :-) I presume you'll incorporate the various pieces of feedback into a revised version of the PEP that explains the purpose a bit more clearly. Paul From brett at python.org Wed Oct 9 18:40:00 2013 From: brett at python.org (Brett Cannon) Date: Wed, 9 Oct 2013 12:40:00 -0400 Subject: [Python-Dev] inspect() and dir() In-Reply-To: <5254791E.1020803@stoneleaf.us> References: <5254791E.1020803@stoneleaf.us> Message-ID: On Tue, Oct 8, 2013 at 5:29 PM, Ethan Furman wrote: > Greetings, > > Currently, inspect() is dependent on dir(). > What's inspect()? I don't know of any global with that name and you are obviously not talking about the module. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Oct 9 18:48:49 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 09 Oct 2013 12:48:49 -0400 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52555F51.8030307@hastings.org> References: <52549646.5060909@hastings.org> <52555F51.8030307@hastings.org> Message-ID: On 10/9/2013 9:51 AM, Larry Hastings wrote: > Again, I don't expect this syntax to be implemented any time soon. But > this does raise a mild sore point: Maciej and Armin Rigo tell me that > PyPy's implementation of range() looks like this: > > def range(x, y=None, step=None): > > The "None" above is not the Python None, it's RPython's internal None, > so it's not available to Python programs. But this does mean range() in > PyPy accepts keyword parameters, and in fact this works: > > range(9, step=2) > > That's way illegal in CPython. But Georg's point is that it does not have to be illegal in CPython. Range, in particular, does not have to be blazingly fast since it wraps loops rather than being in loops. I quite agree that it would be better to fix most functions. -- Terry Jan Reedy From barry at python.org Wed Oct 9 18:53:39 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 9 Oct 2013 12:53:39 -0400 Subject: [Python-Dev] Reduce memory footprint of Python In-Reply-To: References: Message-ID: <20131009125339.43783bfb@anarchist> On Oct 06, 2013, at 11:41 AM, Benjamin Peterson wrote: >What is the usecase for minimizing the memory usage of Python in the >middle of a program? There are environments where Python is being actively discouraged in part due to its memory footprint. For example, we can't use Python on Ubuntu Touch for long-running processes because the target platforms (phones, tablets) are memory constrained. Python will never be as conservative as C++, but it's also unfortunate that the issue is bad enough for its use to be prohibited. FWIW, Python 3 is acceptable for some short-lived processes on these platforms. -Barry From larry at hastings.org Wed Oct 9 19:05:36 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 19:05:36 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> <52555F51.8030307@hastings.org> Message-ID: <52558CE0.3090700@hastings.org> On 10/09/2013 06:48 PM, Terry Reedy wrote: > On 10/9/2013 9:51 AM, Larry Hastings wrote: > >> range() in >> PyPy accepts keyword parameters, and in fact this works: >> >> range(9, step=2) >> >> That's way illegal in CPython. > > But Georg's point is that it does not have to be illegal in CPython. > Range, in particular, does not have to be blazingly fast since it > wraps loops rather than being in loops. I quite agree that it would be > better to fix most functions. First, if you're proposing to change the signature of range()... well, good luck. Any proposals I made like that got shot down almost immediately. I /think/ Guido said somewhere "range isn't changing. deal with it.", though I admit I don't have a reference for that handy. Also, if you're suggesting "change the semantics of builtins to get rid of positional-only parameters", I've got a different example for you: the dict constructor, which /must/ accept a single optional positional-only parameter. The only way to simulate that in Python code is with *args, but now introspection on that function will be misleading, because the dict constructor doesn't accept more than one positional-only argument. And, as Georg suggests, these niggling inconsistencies are the hobgoblins of my small mind. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From corbellini.andrea at gmail.com Wed Oct 9 19:09:06 2013 From: corbellini.andrea at gmail.com (Andrea Corbellini) Date: Wed, 9 Oct 2013 19:09:06 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52549646.5060909@hastings.org> References: <52549646.5060909@hastings.org> Message-ID: Sorry if I meddle, but wouldn't be better to include a module in the standard library that provides functions and classes to help parsing *args and **kwargs? This way you can avoid defining a new syntax, new builtin objects and so on. From v+python at g.nevcal.com Wed Oct 9 19:25:53 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 09 Oct 2013 10:25:53 -0700 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <81E45292CF3BC5478E88DF55E216AD2F03B8C3C4@FLONIDANPOST.flonidan.net> References: <52549646.5060909@hastings.org> <52555F51.8030307@hastings.org> <81E45292CF3BC5478E88DF55E216AD2F03B8C3C4@FLONIDANPOST.flonidan.net> Message-ID: <525591A1.1040308@g.nevcal.com> On 10/9/2013 7:45 AM, Anders J. Munch wrote: > > Larry Hastings wrote: > > > I look forward to an alternate suggestion. This is the least-bad > > > thing I could come up with. > > How about a naming convention instead, where using a leading > > underscore in a parameter name is a hint that it is positional-only. > > For example, the docstring of sorted: > > sorted(iterable, key=None, reverse=False) --> new sorted list > > would become: > > sorted(_iterable, key=None, reverse=False) --> new sorted list > > It seems more intuitive than the slash, and requires no change to the > > toolchain. > > Although, it may collide slightly with code that uses a leading > > underscore to indicate "implementation detail" for a parameter with a > > default value, but a quick scan of the std. lib. suggests that that's > > very rare. > > regards, Anders > So, to avoid that collision, use two leading underscores... a private name for the parameter which can't be specified in the call, similar to private class members that can't (without extreme difficulty) be specified in a reference outside the class. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Oct 9 18:50:04 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 09 Oct 2013 09:50:04 -0700 Subject: [Python-Dev] inspect() and dir() In-Reply-To: References: <5254791E.1020803@stoneleaf.us> Message-ID: <5255893C.1030506@stoneleaf.us> On 10/09/2013 09:40 AM, Brett Cannon wrote: > On Tue, Oct 8, 2013 at 5:29 PM, Ethan Furman wrote: >> >> Currently, inspect() is dependent on dir(). > > What's inspect()? I don't know of any global with that name and you are obviously not talking about the module. My apologies. I am actually talking about the module. I meant inspect.get_members() and inspect.classify_class_attrs(), which, as near as I can tell, are the only two functions in inspect that attempt to retrieve/look at all of an object's attributes. -- ~Ethan~ From rdmurray at bitdance.com Wed Oct 9 19:38:50 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 09 Oct 2013 13:38:50 -0400 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> <5255711C.80208@hastings.org> Message-ID: <20131009173850.A3A1E2507B6@webabinitio.net> On Wed, 09 Oct 2013 17:05:12 +0100, Paul Moore wrote: > On 9 October 2013 16:07, Larry Hastings wrote: > > On 10/09/2013 04:38 PM, Paul Moore wrote: > > > > For that matter, why does the syntax used by Argument Clinic have to > > be the same as whatever future syntax is used in Python? If indeed, > > any ever is? What benefit do we get given the cost (rushing in a > > syntax that nobody is actually convinced we even need in Python yet). > > > > In other words, why does PEP this need to be separated out from the > > Argument Clinic PEP at all? > > > > > > I propose to use this syntax in three places: > > > > As input to Argument Clinic. > > As input to the Python C API, to express the metadata for builtins. > > As the first line of output of pydoc (aka "help()") when reviewing a > > function. > > > > Of these, 3 is visible to users. Theoretically all of them could use > > different syntaxes. But consistency is desirable, so it would be better if > > these syntaxes were as similar as possible. > > OK, all of those are reasonable points. I'm still -0 because I find > the proposed syntax ugly, but I can't offer a better syntactic > solution. My proposal is to hide the ugly syntax internally, and use > prose documentation where users can see. You may disagree, that's > fine. > > > Python syntax works well for > > all three syntaxes, except it does not support positional-only parameters. > > It seemed reasonable to codify twenty-year old practice in a syntax > > extension that I could use in all of the above. > > I remain -1 on forcing "Python syntax" to support all of these odd > corner cases (and positional-only is already a corner case, > range/addch are seriously so). > > I don't see any reason for the proposal to mandate that the arguments > made here should apply to any future changes to core Python syntax. > Either we have the debate now, or we agree that the proposal doesn't > apply to Python. It seems to me, then, that the solution for the handle-the-ugly-existing-practice-groups issue is to make case (3) (pydoc prototype) match the convention that we have arrived at for the documentation: multiple signature lines to represent what Argument Clinic represents with the group notation. We don't currently have a doc solution for 'positional only' arguments. How about a variant on the '_' markup idea: range() range(, [, ]) This takes advantage of the fact that the <> notation is commonly used in various kinds of programming documentation to indicate "put your specific value here". --David From ethan at stoneleaf.us Wed Oct 9 19:00:27 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 09 Oct 2013 10:00:27 -0700 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> <5255711C.80208@hastings.org> Message-ID: <52558BAB.5040707@stoneleaf.us> On 10/09/2013 09:05 AM, Paul Moore wrote: > > I remain -1 on forcing "Python syntax" to support all of these odd > corner cases (and positional-only is already a corner case, > range/addch are seriously so). Considering the prevalence of positional-only functions and methods, I don't think they can be called a corner-case. (Or maybe I've just had the bad luck to run into all of the few there are.) -- ~Ethan~ From eliben at gmail.com Wed Oct 9 19:50:47 2013 From: eliben at gmail.com (Eli Bendersky) Date: Wed, 9 Oct 2013 10:50:47 -0700 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52549646.5060909@hastings.org> References: <52549646.5060909@hastings.org> Message-ID: On Tue, Oct 8, 2013 at 4:33 PM, Larry Hastings wrote: > > I've contributed a new PEP to humanity. I include the RST for your > reading pleasure below, but you can also read it online here: > > http://www.python.org/dev/peps/pep-0457/ > > > Hi Larry, My comments below. TL;DR I'm -1, but trying to figure out your rationale and motivation in more detail. > Discuss, > > > */arry* > > ----- > > PEP: 457 > Title: Syntax For Positional-Only Parameters > Version: $Revision$ > Last-Modified: $Date$ > Author: Larry Hastings > Discussions-To: Python-Dev > Status: Draft > Type: Informational > Content-Type: text/x-rst > Created: 08-Oct-2013 > > > ======== > Overview > ======== > > This PEP proposes a syntax for positional-only parameters in Python. > Positional-only parameters are parameters without an externally-usable > name; when a function accepting positional-only parameters is called, > positional arguments are mapped to these parameters based solely on > their position. > > ========= > Rationale > ========= > > Python has always supported positional-only parameters. > Early versions of Python lacked the concept of specifying > parameters by name, so naturally all parameters were > positional-only. This changed around Python 1.0, when > all parameters suddenly became positional-or-keyword. > But, even in current versions of Python, many CPython > "builtin" functions still only accept positional-only > arguments. > > Functions implemented in modern Python can accept > an arbitrary number of positional-only arguments, via the > variadic ``*args`` parameter. However, there is no Python > syntax to specify accepting a specific number of > positional-only parameters. Put another way, there are > many builtin functions whose signatures are simply not > expressable with Python syntax. > > This PEP proposes a backwards-compatible syntax that should > permit implementing any builtin in pure Python code. > > ----------------------------------------------------- > Positional-Only Parameter Semantics In Current Python > ----------------------------------------------------- > > There are many, many examples of builtins that only > accept positional-only parameters. The resulting > semantics are easily experienced by the Python > programmer--just try calling one, specifying its > arguments by name:: > > >>> pow(x=5, y=3) > Traceback (most recent call last): > File "", line 1, in > TypeError: pow() takes no keyword arguments > You mean, like: >>> def mypow(*args): return pow(args[0], args[1]) ... >>> mypow(2, 3) 8 >>> mypow(x=2, y=3) Traceback (most recent call last): File "", line 1, in TypeError: mypow() got an unexpected keyword argument 'x' >>> def myfoo(*args): a, b = *args; print a, b File "", line 1 def myfoo(*args): a, b = *args; print a, b ^ ? > > In addition, there are some functions with particularly > interesting semantics: > > * ``range()``, which accepts an optional parameter > to the *left* of its required parameter. [#RANGE]_ > > * ``dict()``, whose mapping/iterator parameter is optional and > semantically must be positional-only. Any externally > visible name for this parameter would occlude > that name going into the ``**kwarg`` keyword variadic > parameter dict! [#DICT]_ > > Obviously one can simulate any of these in pure Python code > by accepting ``(*args, **kwargs)`` and parsing the arguments > by hand. But this results in a disconnect between the > Python function's signature and what it actually accepts, > not to mention the work of implementing said argument parsing. > > I'm not sure what you call "parsing". This? >>> def myfoo(*args): a, b = args; print("%s, then %s" % (a, b)) ? > ========== > Motivation > ========== > > This PEP does not propose we implement positional-only > parameters in Python. The goal of this PEP is simply > to define the syntax, so that: > > * Documentation can clearly, unambiguously, and > consistently express exactly how the arguments > for a function will be interpreted. > Can't we do this now? Is the problem only with those existing built-ins, or for new functions too? > > * The syntax is reserved for future use, in case > the community decides someday to add positional-only > parameters to the language. > This doesn't seem like a valid motivation. YAGNI > > * Argument Clinic can use a variant of the syntax > as part of its input when defining > the arguments for built-in functions. > > Isn't one of the goals of Argument Clinic to provide a DSL anyhow? I'm not sure how this help. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Wed Oct 9 19:59:06 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 09 Oct 2013 19:59:06 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52558BAB.5040707@stoneleaf.us> References: <52549646.5060909@hastings.org> <5255711C.80208@hastings.org> <52558BAB.5040707@stoneleaf.us> Message-ID: Am 09.10.2013 19:00, schrieb Ethan Furman: > On 10/09/2013 09:05 AM, Paul Moore wrote: >> >> I remain -1 on forcing "Python syntax" to support all of these odd corner >> cases (and positional-only is already a corner case, range/addch are >> seriously so). > > Considering the prevalence of positional-only functions and methods, I don't > think they can be called a corner-case. (Or maybe I've just had the bad luck > to run into all of the few there are.) No, but their number can be decreased substantially without much additional work while switching to Argument Clinic. Georg From ericsnowcurrently at gmail.com Wed Oct 9 20:20:28 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 9 Oct 2013 12:20:28 -0600 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52556DDB.8040203@hastings.org> References: <52549646.5060909@hastings.org> <525560A1.2050704@hastings.org> <52556DDB.8040203@hastings.org> Message-ID: On Wed, Oct 9, 2013 at 8:53 AM, Larry Hastings wrote: > But that's not how addch works. addch counts how many arguments it > received; if it is called with one or two, it does one thing, and if it's > called with three or four it does something else. You can't duplicate these > semantics with > > Similarly, you can't accurately express the semantics of range's arguments > using default values. PyPy's approach is approximately like this: > > def range(x, y=None, step=None): > step = 1 if step is None else step > if y is not None: > start, stop = x, y > else: > start, stop = 0, x > > But now introspection information on range() is inaccurate and unhelpful. > (Not to mention, they allow specifying step without specifying y, by using > keyword arguments.) > > My goal in writing the PEP was to codify existing practice, which meant > reflecting these (annoying!) corner cases accurately. The optional group syntax is in the PEP to support a small number of builtin functions, right? Is it just range() and addch()? If so, adding the complexity of optional groups (as syntax) isn't worth it. I'm still +0 on adding positional-only arguments (with '/'). As others have said, you can already achieve all this using *args. For me the allure of positional-only arguments lies in the following: 1. not having to roll my own *args handling; 2. not having to clutter up my code with the *args handling; 3. not having to handle positional-or-keyword params with *args when also using positional-only args; 4. documentation and help() can be more clear/accurate. As far as documentation goes, I'll defer to Georg and other "Doc/library"-savvy folks about the downside of using the syntax in this PEP. However, I agree it would be nice to be able to indicate in the docs that a parameter is positional-only. Perhaps it is as simple as always listing the positional-only parameters right after the signature. For example: type(object) type(name, bases, dict) positional-only: object, name, bases, dict With one argument, return the type of ... range(stop) range(start, stop, step) positional-only: start, stop, step Rather than being a function, ... Regarding help(), getting pydoc to make use of inspect.Signature objects would be great. The idea of supporting optional groups in inspect.Parameter sounds good to me and pydoc could make use of that. -eric From tjreedy at udel.edu Wed Oct 9 20:25:18 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 09 Oct 2013 14:25:18 -0400 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52558CE0.3090700@hastings.org> References: <52549646.5060909@hastings.org> <52555F51.8030307@hastings.org> <52558CE0.3090700@hastings.org> Message-ID: On 10/9/2013 1:05 PM, Larry Hastings wrote: > First, if you're proposing to change the signature of range()... well, > good luck. Any proposals I made like that got shot down almost > immediately. I /think/ Guido said somewhere "range isn't changing. > deal with it.", though I admit I don't have a reference for that handy. I am sure that he has rejected any backward-incompatible change, as would I. Allowing keyword passing *is* compatible, so I would not guess about that. > Also, if you're suggesting "change the semantics of builtins to get rid > of positional-only parameters", Georg, and I agree, said *most*. > I've got a different example for you: the dict constructor, > which /must/ accept a single optional positional-only parameter. An *optional* positional-only parameter can be and sometimes is or has been denoted by [], as with dir([object]). In Python 3, [] is no longer used for normal optional params, which always have defaults. The dict signature could be (and maybe once was) given as dict([mapping-or-iterable], *kwds). There is a definite problem, though with required positional-only params. If these are rarer than at present, then it may matter less to people the indicator is. If they think it obnoxious, they will not have to see it so often, Unlike Georg, I think '/' is plausibly ok. You should, however, explicitly say in the PEP that it was selected not randomly but by rough analogy. Division (/) is the inverse of multiplication (*), and positional-only is sort of the 'inverse' of keyword-only. (If I am wrong, and the above is just an accident, it is at least a way to remember ;-). Another approach would be to make parameter names that are illegal to use in calls illegal as Python names. "form follows function" or in this case, "bad form indicates no function". For instance, always put a *hyphen* in the displayed parameter name, as I intentionally did above. If nothing else, add a '-p' suffix. A hyphen is not legal in a name, so 'object-p', for instance, cannot possibly be used to make a call such as: >>> dir(object-p=int) SyntaxError: keyword can't be an expression Note that I am talking about the doc and help results, not necessarily about the CPython C code most people never see. -- Terry Jan Reedy From ericsnowcurrently at gmail.com Wed Oct 9 20:27:37 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 9 Oct 2013 12:27:37 -0600 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <20131009173850.A3A1E2507B6@webabinitio.net> References: <52549646.5060909@hastings.org> <5255711C.80208@hastings.org> <20131009173850.A3A1E2507B6@webabinitio.net> Message-ID: On Wed, Oct 9, 2013 at 11:38 AM, R. David Murray wrote: > It seems to me, then, that the solution for the > handle-the-ugly-existing-practice-groups issue is to make case (3) > (pydoc prototype) match the convention that we have arrived at for the > documentation: multiple signature lines to represent what Argument Clinic > represents with the group notation. +1 > > We don't currently have a doc solution for 'positional only' arguments. > How about a variant on the '_' markup idea: > > range() > range(, [, ]) > > This takes advantage of the fact that the <> notation is commonly used > in various kinds of programming documentation to indicate "put your > specific value here". +1 I was thinking of exactly this same notation. -eric From p.f.moore at gmail.com Wed Oct 9 20:37:26 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Oct 2013 19:37:26 +0100 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52558BAB.5040707@stoneleaf.us> References: <52549646.5060909@hastings.org> <5255711C.80208@hastings.org> <52558BAB.5040707@stoneleaf.us> Message-ID: On 9 October 2013 18:00, Ethan Furman wrote: > On 10/09/2013 09:05 AM, Paul Moore wrote: >> >> >> I remain -1 on forcing "Python syntax" to support all of these odd >> corner cases (and positional-only is already a corner case, >> range/addch are seriously so). > > > Considering the prevalence of positional-only functions and methods, I don't > think they can be called a corner-case. (Or maybe I've just had the bad luck > to run into all of the few there are.) You're probably right - I was exaggerating, sorry. I do however consider most positional-only arguments in builtins to be an implementation artifact - if they could have been positional-or-keyword when they were introduced, they may well have have been (keyword args in C were a later addition to the C API, I can't recall exactly when but a good chunk of the existing builtins were already there when they arrived). Paul. From g.brandl at gmx.net Wed Oct 9 20:38:55 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 09 Oct 2013 20:38:55 +0200 Subject: [Python-Dev] cpython (3.3): Fix macro expansion of _PyErr_OCCURRED(), and make sure to use it in at least In-Reply-To: <3ctrBM3gCZz7LjT@mail.python.org> References: <3ctrBM3gCZz7LjT@mail.python.org> Message-ID: <5255A2BF.5020006@gmx.net> Am 07.10.2013 20:41, schrieb antoine.pitrou: > http://hg.python.org/cpython/rev/2aee9c41bb79 > changeset: 86140:2aee9c41bb79 > branch: 3.3 > parent: 86136:b4444d16e333 > user: Antoine Pitrou > date: Mon Oct 07 20:38:51 2013 +0200 > summary: > Fix macro expansion of _PyErr_OCCURRED(), and make sure to use it in > at least one place so as to avoid regressions. If it was broken and never used anyway (and is an internal macro), why not remove it completely? Georg From larry at hastings.org Wed Oct 9 21:41:53 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 21:41:53 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> Message-ID: <5255B181.5080604@hastings.org> On 10/09/2013 07:50 PM, Eli Bendersky wrote: > I'm not sure what you call "parsing". This? > > >>> def myfoo(*args): a, b = args; print("%s, then %s" % (a, b)) From the Argument Clinic PEP (436): The primary implementation of Python, "CPython", is written in a mixture of Python and C. One implementation detail of CPython is what are called "built-in" functions -- functions available to Python programs but written in C. When a Python program calls a built-in function and passes in arguments, those arguments must be translated from Python values into C values. This process is called "parsing arguments". I should probably amend that, because deciding how to map arguments to parameters is also part of "parsing arguments". //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Wed Oct 9 22:03:29 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 22:03:29 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> <52555F51.8030307@hastings.org> <52558CE0.3090700@hastings.org> Message-ID: <5255B691.1080001@hastings.org> On 10/09/2013 08:25 PM, Terry Reedy wrote: > I am sure that he has rejected any backward-incompatible change, as > would I. Allowing keyword passing *is* compatible, so I would not > guess about that. I don't know where to look for this discussion, and I read it quite some time ago. So I this is only dimly remembered. If it's crucial to the outcome of this discussion maybe we can ask Guido for a fresh ruling. But. As I recall the discussion, he was specifically against supporting keyword arguments for range, and he was adamant that range argument parsing was not to change in the slightest degree, backwards-incompatible or not. Again, my almost-certainly-misremembered quote: "range isn't changing. deal with it." > An *optional* positional-only parameter can be and sometimes is or has > been denoted by [], as with dir([object]). In Python 3, [] is no > longer used for normal optional params, which always have defaults. > The dict signature could be (and maybe once was) given as > dict([mapping-or-iterable], *kwds). > > There is a definite problem, though with required positional-only > params. If these are rarer than at present, then it may matter less to > people the indicator is. If they think it obnoxious, they will not > have to see it so often, Most of the positional-only parameters in CPython are of the "required" variety. Some examples? Almost all of the "built-in functions": http://docs.python.org/3/library/functions.html only accept positional-only parameters. So, almost anywhere you see a required parameter on that page, it is a required positional-only parameter. Also, many (most?) of the really old C extensions exclusively use PyArg_ParseTuple, so those too. My go-to example is curses simply because that one gave me the most psychological scarring. But there are loads of them. Anything that parses arguments solely with PyArg_ParseTuple only accepts positional-only arguments, and anything that calls it more than once probably has some optional arguments. Run from a fresh Python trunk: % fgrep -w PyArg_ParseTuple */*.c | wc 861 4728 69716 //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at hotpy.org Wed Oct 9 22:46:00 2013 From: mark at hotpy.org (Mark Shannon) Date: Wed, 09 Oct 2013 21:46:00 +0100 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52549646.5060909@hastings.org> References: <52549646.5060909@hastings.org> Message-ID: <5255C088.6020107@hotpy.org> On 09/10/13 00:33, Larry Hastings wrote: > > I've contributed a new PEP to humanity. I include the RST for your > reading pleasure below, but you can also read it online here: > > http://www.python.org/dev/peps/pep-0457/ Overall I'm in favour. As a motivation for positional only parameters, consider: Python 3.2: >>> from decimal import Decimal >>> d = Decimal(4) >>> d.__add__(other=d) Decimal('8') Python 3.3: >>> from decimal import Decimal >>> d = Decimal(4) >>> d.__add__(other=d) Traceback (most recent call last): File "", line 1, in TypeError: wrapper __add__ doesn't take keyword arguments [snip] > The obvious solution: add a new singleton constant to Python > that is passed in when a parameter is not mapped to an argument. > I propose that the value be called called ``undefined``, > and be a singleton of a special class called ``Undefined``. > If a positional-only parameter did not receive an argument > when called, its value would be set to ``undefined``. There is no need to create an "undefined" value. Rather than define a parameter by assigning a fake value, just don't define it. We already do this for non-parameter locals and it could be extended to parameters. 'range' would be defined thus: def range([start,] stop, [step], /): try: start except UnboundLocalError: start = 0 try: step except UnboundLocalError: step = 1 ... Cheers, Mark. From larry at hastings.org Wed Oct 9 22:51:30 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 09 Oct 2013 22:51:30 +0200 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <5255C088.6020107@hotpy.org> References: <52549646.5060909@hastings.org> <5255C088.6020107@hotpy.org> Message-ID: <5255C1D2.7020505@hastings.org> On 10/09/2013 10:46 PM, Mark Shannon wrote: > There is no need to create an "undefined" value. > Rather than define a parameter by assigning a fake value, just don't > define it. We already do this for non-parameter locals and it could be > extended to parameters. > > 'range' would be defined thus: > > def range([start,] stop, [step], /): > try: > start > except UnboundLocalError: > start = 0 > try: > step > except UnboundLocalError: > step = 1 > ... That's a clever, and intuitive, idea. Though the try/excepts look awfully clumsy. Perhaps we could add (egad, no, I can't believe I'm saying this) a new built-in function that tells you whether or not a local variable has been assigned to yet? def range([start,] stop, [step], /): if not bound(start): start = 0 if not bound(step): step = 1 ... Anyway, this section is in the "Notes For Future Implementors" section, so as long as we never actually implement the syntax we don't need to solve the problem. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Oct 9 23:02:45 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 9 Oct 2013 23:02:45 +0200 Subject: [Python-Dev] cpython (3.3): Fix macro expansion of _PyErr_OCCURRED(), and make sure to use it in at least References: <3ctrBM3gCZz7LjT@mail.python.org> <5255A2BF.5020006@gmx.net> Message-ID: <20131009230245.433027b5@fsol> On Wed, 09 Oct 2013 20:38:55 +0200 Georg Brandl wrote: > Am 07.10.2013 20:41, schrieb antoine.pitrou: > > http://hg.python.org/cpython/rev/2aee9c41bb79 > > changeset: 86140:2aee9c41bb79 > > branch: 3.3 > > parent: 86136:b4444d16e333 > > user: Antoine Pitrou > > date: Mon Oct 07 20:38:51 2013 +0200 > > summary: > > Fix macro expansion of _PyErr_OCCURRED(), and make sure to use it in > > at least one place so as to avoid regressions. > > If it was broken and never used anyway (and is an internal macro), why not > remove it completely? Because I used it in a patch :-) Regards Antoine. From ncoghlan at gmail.com Wed Oct 9 23:34:11 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 Oct 2013 07:34:11 +1000 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20131008170517.299caf99@pitrou.net> References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <20131007212430.GU7989@ando> <20131008082143.4ecfef3f@fsol> <20131008103445.0174baa1@pitrou.net> <20131008143002.073d80d1@pitrou.net> <20131008170517.299caf99@pitrou.net> Message-ID: On 9 Oct 2013 01:07, "Antoine Pitrou" wrote: > > Le Tue, 8 Oct 2013 22:49:28 +1000, > Nick Coghlan a ?crit : > > > Well, you could ask the same question about OrderedDict, > > > defaultdict or Weak*Dictionary since neither of them use > > > composition :-) > > > > We *did* ask the same question about those (except the Weak* variants, > > simply due to age). > > > > Each time, the point that each new dict variant would be used to > > justify yet *more* variants was a cause for concern. > > Could you explain? I don't understand. Just that the comment I replied to is getting very close to the argument "we have so many non-composable mapping variants already, what's the harm in adding one more?". I believe potentially enabling that argument in the future was cited as a downside for all of defaultdict, OrderedDict and Counter. > > "Composition doesn't work with some mappings" isn't an adequate > > answer to the criticism that a composition based design could work > > with more mappings than just the builtin dict. > > Fair enough. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Oct 9 23:46:29 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 Oct 2013 07:46:29 +1000 Subject: [Python-Dev] inspect() and dir() In-Reply-To: <5255893C.1030506@stoneleaf.us> References: <5254791E.1020803@stoneleaf.us> <5255893C.1030506@stoneleaf.us> Message-ID: On 10 Oct 2013 03:39, "Ethan Furman" wrote: > > On 10/09/2013 09:40 AM, Brett Cannon wrote: > >> On Tue, Oct 8, 2013 at 5:29 PM, Ethan Furman wrote: >>> >>> >>> Currently, inspect() is dependent on dir(). >> >> >> What's inspect()? I don't know of any global with that name and you are obviously not talking about the module. > > > My apologies. I am actually talking about the module. I meant inspect.get_members() and inspect.classify_class_attrs(), which, as near as I can tell, are the only two functions in inspect that attempt to retrieve/look at all of an object's attributes. Those have to depend on __dir__ so classes (especially proxies) can accurately report *extra* dynamically accessible names. It's also the mechanism by which custom metaclasses, descriptors and __getattribute__ implementations can report the right names to the introspection machinery. However, it may make sense to offer a separate introspection utility to help debug custom __dir__ implementations. Cheers, Nick. > > > -- > ~Ethan~ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Oct 10 00:12:45 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 10 Oct 2013 09:12:45 +1100 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> <52555F51.8030307@hastings.org> <52558CE0.3090700@hastings.org> Message-ID: <20131009221245.GE7989@ando> On Wed, Oct 09, 2013 at 02:25:18PM -0400, Terry Reedy wrote: > Unlike Georg, I think '/' is plausibly ok. You should, however, > explicitly say in the PEP that it was selected not randomly but by rough > analogy. Division (/) is the inverse of multiplication (*), and > positional-only is sort of the 'inverse' of keyword-only. (If I am > wrong, and the above is just an accident, it is at least a way to > remember ;-). I recall that was exactly the analogy Guido used when he first suggested the syntax. -- Steven From steve at pearwood.info Thu Oct 10 00:28:25 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 10 Oct 2013 09:28:25 +1100 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <5255C088.6020107@hotpy.org> References: <52549646.5060909@hastings.org> <5255C088.6020107@hotpy.org> Message-ID: <20131009222825.GF7989@ando> On Wed, Oct 09, 2013 at 09:46:00PM +0100, Mark Shannon wrote: > There is no need to create an "undefined" value. > Rather than define a parameter by assigning a fake value, just don't > define it. We already do this for non-parameter locals and it could be > extended to parameters. > > 'range' would be defined thus: > > def range([start,] stop, [step], /): > try: > start > except UnboundLocalError: > start = 0 > try: > step > except UnboundLocalError: > step = 1 > ... If there is even the tiniest, faintest chance that positional arguments will be available to functions written in Python some day, -1 to this suggestion. There is no way I want to write code like the above instead of: def range([start,] stop, [step,] /): if start is Undefined: start = 0 if step is Undefined: step = 1 But what I really want to write is this: def range(start=0, stop, step=1, /): ... although maybe we don't want to encourage odd calling signatures by making them too easy to write. -- Steven From ethan at stoneleaf.us Thu Oct 10 00:07:17 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 09 Oct 2013 15:07:17 -0700 Subject: [Python-Dev] inspect() and dir() In-Reply-To: References: <5254791E.1020803@stoneleaf.us> <5255893C.1030506@stoneleaf.us> Message-ID: <5255D395.2070405@stoneleaf.us> On 10/09/2013 02:46 PM, Nick Coghlan wrote: > On 10 Oct 2013 03:39, "Ethan Furman" wrote: >> >> My apologies. I am actually talking about the module. I meant >> inspect.get_members() and inspect.classify_class_attrs(), which, >> as near as I can tell, are the only two functions in inspect that >> attempt to retrieve/look at all of an object's attributes. > > Those have to depend on __dir__ so classes (especially proxies) can accurately report *extra* dynamically accessible names. Indeed, my rough idea is to use the results of the new dir() and combining that with the results of the old dir(). The use case being that some classes *ahem* Enum *ahem* may report *less* than is actually available, but introspection should reveal whatever is there even if dir() is not reporting it. -- ~Ethan~ From ncoghlan at gmail.com Thu Oct 10 00:36:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 Oct 2013 08:36:46 +1000 Subject: [Python-Dev] inspect() and dir() In-Reply-To: <5255D395.2070405@stoneleaf.us> References: <5254791E.1020803@stoneleaf.us> <5255893C.1030506@stoneleaf.us> <5255D395.2070405@stoneleaf.us> Message-ID: On 10 Oct 2013 08:07, "Ethan Furman" wrote: > > On 10/09/2013 02:46 PM, Nick Coghlan wrote: > >> On 10 Oct 2013 03:39, "Ethan Furman" wrote: >>> >>> >>> My apologies. I am actually talking about the module. I meant >>> inspect.get_members() and inspect.classify_class_attrs(), which, >>> as near as I can tell, are the only two functions in inspect that >>> attempt to retrieve/look at all of an object's attributes. >> >> >> Those have to depend on __dir__ so classes (especially proxies) can accurately report *extra* dynamically accessible names. > > > Indeed, my rough idea is to use the results of the new dir() and combining that with the results of the old dir(). The use case being that some classes *ahem* Enum *ahem* may report *less* than is actually available, but introspection should reveal whatever is there even if dir() is not reporting it. Not necessarily. For autocompletion, for example, you probably only want the public stuff. That's why I'm inclined to suggest the existing functions should continue to only report advertised attributes, with a separate introspection API that tries harder to find all accessible attributes (potentially including those from the metaclass). That way users can choose the one most appropriate to their use case, as well as being able to use the latter to help test custom dir support. Cheers, Nick. > > -- > ~Ethan~ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Thu Oct 10 00:38:39 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 9 Oct 2013 16:38:39 -0600 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> <525560A1.2050704@hastings.org> <52556DDB.8040203@hastings.org> Message-ID: On Wed, Oct 9, 2013 at 12:20 PM, Eric Snow wrote: > For me the allure of positional-only arguments lies in the following: > > 1. not having to roll my own *args handling; > 2. not having to clutter up my code with the *args handling; > 3. not having to handle positional-or-keyword params with *args when > also using positional-only args; > 4. documentation and help() can be more clear/accurate. > > [snip] > > Regarding help(), getting pydoc to make use of inspect.Signature > objects would be great. The idea of supporting optional groups in > inspect.Parameter sounds good to me and pydoc could make use of that. It may work better to facilitate the syntax in this PEP through inspect.Signature and friends, rather than as a pseudo-addition to the Python syntax. Here is some additional API that could help: * inspect.Signature.from_string(sig) - a factory that parses a signature string (including "/" from this PEP or perhaps ""). Could Argument Clinic make use of this directly? * inspect.Signature.__str__() - updated to also display "/" for positional-only (or the "" syntax that RDM described). * inspect.MultiSignature - a Sequence(?) containing one or more Signature objects (proxies the first one), which facilitates grouping. * inspect.MultiSignature.from_string(sig) - a factory that parses a signature string (including the full syntax from this PEP), adding a Signature for each derivable signature. These, along with a few other additions, would certainly help address the 4 things I listed above. I have a rough implementation of the above APIs (ignore anything else :) [1]. -eric [1] https://bitbucket.org/ericsnowcurrently/odds_and_ends/src/default/signature.py From ncoghlan at gmail.com Thu Oct 10 00:57:44 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 Oct 2013 08:57:44 +1000 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52556508.9030002@hastings.org> References: <52549646.5060909@hastings.org> <52556508.9030002@hastings.org> Message-ID: On 10 Oct 2013 00:17, "Larry Hastings" wrote: > > > On 10/09/2013 11:46 AM, Nick Coghlan wrote: >> >> Generally, it needs to be a bit clearer that the intent of the PEP isn't to say "let's do this", it's to be explicit that acceptance of the Argument Clinic PEP severely constrains the design space for possible solutions if we ever *did* implement Python level support for positional only arguments. > > > Can you suggest an edit that would make you happy? PEP title: Representation of positional-only parameters (Removes the loaded word "Syntax for") The abstract would then draw from your reply to Benjamin: adding introspection support for builtins and extension modules means we need a way to represent positional-only parameters that can't be expressed using the normal function definition syntax. >> - parameters in optional groups should just have an implied "=None" that can be overriden if desired. > > > No no no! You can't have a default value, and you definitely couldn't use None. > > range() decides what its arguments mean based on how many arguments it receives. If I understand your proposal correctly, you suggest that >> >> range(None, 5) > > would return the same result as >> >> range(5) > > But that's simply not how it works. No, you'd use "range([start=0,] stop, [step=1,] /)" to represent that case. Remember that this is about representing how the function effectively works from the user's point of view, not how it actually implements argument unpacking internally. Allowing default values and having "=None" implied for optional groups should cover all the cases we have to handle. If there are exceptions, I'd prefer to fix them over considering the future introduction of a new singleton (even if fixing them means offering better ways to handle None in PyArg_ParseTuple). > If you want to propose changing the semantics of range, go ahead, I'll stand back. Not necessary, it just needs to be possible to express default values. > > >> - a simpler variant that omits the grouping support and only allows optional parameters on the right should also be reserved > > > If that syntax is a subset of this syntax, and this syntax is reserved, then by extension we would automatically reserve that syntax too. In any other circumstance (this PEP is rejected, the simpler variant uses a different syntax) the simpler syntax should get a new PEP. > > I'll just say that that syntax is insufficient to implement existing functions we all know and love (addch, range). Allowing a single optional parameter on the left is enough to handle range, though. >> - explicitly delegate to the argument clinic PEP for the impact on inspect.Signature > > > I'd actually rather do it the other way 'round. This PEP is a better place for it. Even though I think the Clinic PEP has a higher chance of being accepted ;-) I thought the group attribute idea was already in the argument clinic PEP. Since it isn't, documenting it here is fine. Cheers, Nick. > > > /arry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Oct 10 01:02:02 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 10 Oct 2013 10:02:02 +1100 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <52556508.9030002@hastings.org> References: <52549646.5060909@hastings.org> <52556508.9030002@hastings.org> Message-ID: <20131009230202.GG7989@ando> On Wed, Oct 09, 2013 at 04:15:36PM +0200, Larry Hastings wrote: > >- parameters in optional groups should just have an implied "=None" > >that can be overriden if desired. > > > > No no no! You can't have a default value, and you definitely couldn't > use None. > > range() decides what its arguments mean based on how many arguments it > receives. If I understand your proposal correctly, you suggest that > > range(None, 5) > > would return the same result as > > range(5) > > But that's simply not how it works. Hmmm... I'm not entirely sure that I understand, but let me see if I've got it. range is defined something like this, only in C: def range(*args): if len(args) == 1: start, stop, step = 0, args[0], 1 elif len(args) == 2: start, stop, step = args[0], args[1], 1 elif len(args) == 3: start, stop, step = args else: raise TypeError ... So we can't document range as having the signature: range(start=None, stop, range=None) (modulo choice of defaults). Even if some future implementation of positional-only arguments allowed functions to set defaults, you would still have to deal with cases like range that don't. Do I understand you correctly? If so, then I ask that this PEP should not rule out setting defaults in the the future, but make it clear that even if defaults are allowed, there needs to be some way to handle defaultless-but-still-optional arguments, for cases like range. -- Steven From victor.stinner at gmail.com Thu Oct 10 02:01:52 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 10 Oct 2013 02:01:52 +0200 Subject: [Python-Dev] PEP 454: Add a new tracemalloc module (final version) In-Reply-To: References: Message-ID: 2013/10/3 Victor Stinner : > I worked on the implementation of the tracemalloc module and its PEP > 454. I consider that this third version of the PEP is ready for a > final review. > > What should be done to finish the PEP? I splitted the module into two parts. Do you prefer the new API? Module: http://www.haypocalc.com/tmp/tracemalloc.split/library/tracemalloc.html Tool: http://www.haypocalc.com/tmp/tracemalloc.split/library/tracemalloctext.html Full story: Charles-Fran?ois Natali made me some suggestions: * why using a Task class instead of a thread? * the module should be splitted into two parts: module (low-level functions getting data), tool (display/compare data with colors) I modified the implementation in a new "split" branch: * I moved DisplayTop, DisplayTopTask, TakeSnapshotTask and Task classes and the command line interface to a new "tracemalloctext" module * I rewrote Task into Python using a thread The tool would not be integrated in Python, but hosted on PyPI. I just kept it to get its documentation and show clearly how the split was made. tracemalloc API is simpler: http://www.haypocalc.com/tmp/tracemalloc.split/library/tracemalloc.html API of the tool (using a temporary name "tracemalloctext" which may change later): http://www.haypocalc.com/tmp/tracemalloc.split/library/tracemalloctext.html I chose to keep Snapshot.top_by(), GroupedStats and StatsDiff in the (tracemalloc) module. I prefer to "hide" internal structures of tracemalloc (tracemalloc.get_stats(), tracemalloc.get_traces()), and suggest to use higher level structures (GroupedStats, StatsDiff). Internal structures are not hidden, they are even documented. But they might change later, whereas GroupedStats structure should not change. Or do you think that computing "top N allocations" is something specific to a tool? Or each tool may implement this function differently? Using a thread has a drawback. Checking the memory threshold is implemented using a short sleep (100 ms). If the memory peak is shorter than the sleep, it will not be noticed. A workaround is to use a threshold on the maximum traced memory. The advantage of the thread is to have a better resolution than 1 second for the timer, and to not have to get the current time at each memory allocation when a task is scheduled. The current implement (not splitted) uses Py_AddPendingCall(), function already used by the C signal handler (signalmodule.c) to call later the Python signal handler from the bytecode evaluation loop (ceval.c). Victor From ericsnowcurrently at gmail.com Thu Oct 10 02:18:20 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 9 Oct 2013 18:18:20 -0600 Subject: [Python-Dev] Python startup time In-Reply-To: <52556874.4030707@python.org> References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> <52555200.4080301@python.org> <52556874.4030707@python.org> Message-ID: On Wed, Oct 9, 2013 at 8:30 AM, Christian Heimes wrote: > The os module imports MutableMapping from collections.abc. That import > adds collections, collections.abc and eight more modules. I'm not sure > if we can do anything about it, though. Well, that depends on how much we want to eliminate those 10 imports. :) Both environ and environb could be turned into lazy wrappers around an _Environ-created-when-needed. If we used a custom module type for os [1], then adding descriptors for the two attributes is a piece of cake. As it is, with a little metaclass magic (or even with explicit wrapping of the various dunder methods), we could drop those 10 imports from startup. -eric [1] This probably wouldn't be a great idea considering that undoubtedly some code depends on "type(os) is types.ModuleType". From ericsnowcurrently at gmail.com Thu Oct 10 02:53:28 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 9 Oct 2013 18:53:28 -0600 Subject: [Python-Dev] PyCon US 2014 Message-ID: Registration is now open for PyCon US 2014. Are there any plans yet for the language summit? Just the day (e.g. Thursday April 10) will suffice. Then we can make arrangements appropriately. Thanks. -eric From greg.ewing at canterbury.ac.nz Thu Oct 10 04:33:16 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 10 Oct 2013 15:33:16 +1300 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <5255C1D2.7020505@hastings.org> References: <52549646.5060909@hastings.org> <5255C088.6020107@hotpy.org> <5255C1D2.7020505@hastings.org> Message-ID: <525611EC.4050807@canterbury.ac.nz> On 10/10/13 09:51, Larry Hastings wrote: > Perhaps we could add (egad, no, I can't believe I'm saying this) a new > built-in function that tells you whether or not a local variable has been > assigned to yet? > > def range([start,] stop, [step], /): > if not bound(start): > start = 0 > if not bound(step): > step = 1 > ... It couldn't be a function in the usual sense, because attempting to evaluate the unbound parameter would raise an exception before the function was called. It would have to be something magical, probably a new piece of syntax understood by the compiler. -- Greg From greg.ewing at canterbury.ac.nz Thu Oct 10 04:39:23 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 10 Oct 2013 15:39:23 +1300 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: References: <52549646.5060909@hastings.org> <52556508.9030002@hastings.org> Message-ID: <5256135B.9020805@canterbury.ac.nz> On 10/10/13 11:57, Nick Coghlan wrote: > PEP title: Representation of positional-only parameters Something like "Notation for documenting positional-only parameters" would make it even clearer that this is not a proposal for adding to the syntax of the language. -- Greg From ethan at stoneleaf.us Thu Oct 10 06:06:47 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 09 Oct 2013 21:06:47 -0700 Subject: [Python-Dev] inspect() and dir() In-Reply-To: References: <5254791E.1020803@stoneleaf.us> <5255893C.1030506@stoneleaf.us> <5255D395.2070405@stoneleaf.us> Message-ID: <525627D7.2060904@stoneleaf.us> On 10/09/2013 03:36 PM, Nick Coghlan wrote: > On 10 Oct 2013 08:07, "Ethan Furman" wrote: >> On 10/09/2013 02:46 PM, Nick Coghlan wrote: >>> On 10 Oct 2013 03:39, "Ethan Furman" wrote: >>>> >>>> My apologies. I am actually talking about the module. I meant >>>> inspect.get_members() and inspect.classify_class_attrs(), which, >>>> as near as I can tell, are the only two functions in inspect that >>>> attempt to retrieve/look at all of an object's attributes. >>> >>> Those have to depend on __dir__ so classes (especially proxies) >>> can accurately report *extra* dynamically accessible names. >> >> Indeed, my rough idea is to use the results of the new dir() and >> combining that with the results of the old dir(). The use case >> being that some classes *ahem* Enum *ahem* may report *less* than >> is actually available, but introspection should reveal whatever is >> there even if dir() is not reporting it. > > Not necessarily. For autocompletion, for example, you probably > only want the public stuff. That's why I'm inclined to suggest > the existing functions should continue to only report advertised > attributes, with a separate introspection API that tries harder > to find all accessible attributes (potentially including those > from the metaclass). That way users can choose the one most > appropriate to their use case, as well as being able to use the > latter to help test custom dir support. That makes sense. So what should the new functions be called? get_all_members and classify_all_class_attrs ? -- ~Ethan~ From yoavglazner at gmail.com Thu Oct 10 06:41:19 2013 From: yoavglazner at gmail.com (yoav glazner) Date: Thu, 10 Oct 2013 07:41:19 +0300 Subject: [Python-Dev] Python startup time In-Reply-To: References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> <52555200.4080301@python.org> <52556874.4030707@python.org> Message-ID: I'm not sure Droping imports is the best way to go, since every python script/app will import common modules right on the start and it will still seem like the interpeter boot is slow. making modules load faster seems like a better approch On Thu, Oct 10, 2013 at 3:18 AM, Eric Snow wrote: > On Wed, Oct 9, 2013 at 8:30 AM, Christian Heimes > wrote: > > The os module imports MutableMapping from collections.abc. That import > > adds collections, collections.abc and eight more modules. I'm not sure > > if we can do anything about it, though. > > Well, that depends on how much we want to eliminate those 10 imports. > :) Both environ and environb could be turned into lazy wrappers > around an _Environ-created-when-needed. If we used a custom module > type for os [1], then adding descriptors for the two attributes is a > piece of cake. As it is, with a little metaclass magic (or even with > explicit wrapping of the various dunder methods), we could drop those > 10 imports from startup. > > -eric > > [1] This probably wouldn't be a great idea considering that > undoubtedly some code depends on "type(os) is types.ModuleType". > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/yoavglazner%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Oct 10 11:22:25 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 Oct 2013 19:22:25 +1000 Subject: [Python-Dev] PEP 457: Syntax For Positional-Only Parameters In-Reply-To: <5256135B.9020805@canterbury.ac.nz> References: <52549646.5060909@hastings.org> <52556508.9030002@hastings.org> <5256135B.9020805@canterbury.ac.nz> Message-ID: On 10 Oct 2013 12:40, "Greg Ewing" wrote: > > On 10/10/13 11:57, Nick Coghlan wrote: >> >> PEP title: Representation of positional-only parameters > > > Something like "Notation for documenting positional-only parameters" > would make it even clearer that this is not a proposal for adding > to the syntax of the language. It's more than that though, since pydoc and inspect will both emit the chosen format, and it's entirely possible inspect.Signature will grow an alternative constructor that consumes it. Cheers, Nick. > > -- > Greg > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Oct 10 11:25:41 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 Oct 2013 19:25:41 +1000 Subject: [Python-Dev] inspect() and dir() In-Reply-To: <525627D7.2060904@stoneleaf.us> References: <5254791E.1020803@stoneleaf.us> <5255893C.1030506@stoneleaf.us> <5255D395.2070405@stoneleaf.us> <525627D7.2060904@stoneleaf.us> Message-ID: On 10 Oct 2013 14:32, "Ethan Furman" wrote: > > On 10/09/2013 03:36 PM, Nick Coghlan wrote: > >> On 10 Oct 2013 08:07, "Ethan Furman" wrote: >>> >>> On 10/09/2013 02:46 PM, Nick Coghlan wrote: >>>> >>>> On 10 Oct 2013 03:39, "Ethan Furman" wrote: >>>>> >>>>> >>>>> My apologies. I am actually talking about the module. I meant >>>>> inspect.get_members() and inspect.classify_class_attrs(), which, >>>>> as near as I can tell, are the only two functions in inspect that >>>>> attempt to retrieve/look at all of an object's attributes. >>>> >>>> >>>> Those have to depend on __dir__ so classes (especially proxies) >>>> can accurately report *extra* dynamically accessible names. >>> >>> >>> Indeed, my rough idea is to use the results of the new dir() and >>> combining that with the results of the old dir(). The use case >>> being that some classes *ahem* Enum *ahem* may report *less* than >>> is actually available, but introspection should reveal whatever is >>> there even if dir() is not reporting it. >> >> >> Not necessarily. For autocompletion, for example, you probably >> only want the public stuff. That's why I'm inclined to suggest >> the existing functions should continue to only report advertised >> attributes, with a separate introspection API that tries harder >> to find all accessible attributes (potentially including those >> from the metaclass). That way users can choose the one most >> appropriate to their use case, as well as being able to use the >> latter to help test custom dir support. > > > That makes sense. > > So what should the new functions be called? get_all_members and classify_all_class_attrs ? Yeah, those work for me. The only way they should miss anything is if attribute lookup is customised to offer extra dynamic attributes without implementing custom dir support. Cheers, Nick. > > > -- > ~Ethan~ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Thu Oct 10 12:18:14 2013 From: christian at python.org (Christian Heimes) Date: Thu, 10 Oct 2013 12:18:14 +0200 Subject: [Python-Dev] Python startup time In-Reply-To: References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> <52555200.4080301@python.org> <52556874.4030707@python.org> Message-ID: <52567EE6.9060000@python.org> Am 10.10.2013 02:18, schrieb Eric Snow: > On Wed, Oct 9, 2013 at 8:30 AM, Christian Heimes > wrote: >> The os module imports MutableMapping from collections.abc. That >> import adds collections, collections.abc and eight more modules. >> I'm not sure if we can do anything about it, though. > > Well, that depends on how much we want to eliminate those 10 > imports. :) Both environ and environb could be turned into lazy > wrappers around an _Environ-created-when-needed. If we used a > custom module type for os [1], then adding descriptors for the two > attributes is a piece of cake. As it is, with a little metaclass > magic (or even with explicit wrapping of the various dunder > methods), we could drop those 10 imports from startup. We don't have to use a custom module type to get rid of these imports (but I like to get my hands a piece of chocolate cake *g*). We can either implement yet another mutable mapping class for the os module. That would remove the dependency on collections.abc. Or we have to juggle the modules a bit so we can get to MutableMapping without the extra stuff from collections.__init__. The abc and _weakset modules are already loaded by the io module. Only collections.__init__ imports _collections, operator, keyword, heapq, itertools and reprlib. I implemented both as an experiment. A lean and mean MutableMapping works but it involves some code duplication. Next I moved collections.abc to its former place _abcoll and installed a new collections.abc module as facade. $ hg mv Lib/collections/abc.py Lib/_abcoll.py $ echo "from _abcoll import *" > Lib/collections/abc.py $ echo "from _abcoll import __all__" >> Lib/collections/abc.py $ sed -i "s/collections\.abc/_abcoll/" Lib/os.py With three additional patches I'm down 19 modules: $ ./python -c "import sys; print(len(sys.modules))" 34 $ hg revert --all . $ ./python -c "import sys; print(len(sys.modules))" 53 Christian From rdmurray at bitdance.com Thu Oct 10 13:36:23 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 10 Oct 2013 07:36:23 -0400 Subject: [Python-Dev] Python startup time In-Reply-To: References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> <52555200.4080301@python.org> <52556874.4030707@python.org> Message-ID: <20131010113624.08B88250164@webabinitio.net> On Thu, 10 Oct 2013 07:41:19 +0300, yoav glazner wrote: > I'm not sure Droping imports is the best way to go, since every python > script/app will import common modules right on the start and it will still > seem like the interpeter boot is slow. > > making modules load faster seems like a better approch Making any of the infrastructure faster is good. But I certainly have plenty of CLI scripts that import only os and sys, so reducing the number of modules imported will be a win for me. (Now, granted, a lot of those scripts *ought* to import argparse, which imports a bunch of stuff, but they don't ;) --David From christian at python.org Thu Oct 10 14:25:18 2013 From: christian at python.org (Christian Heimes) Date: Thu, 10 Oct 2013 14:25:18 +0200 Subject: [Python-Dev] Python startup time In-Reply-To: References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> <52555200.4080301@python.org> <52556874.4030707@python.org> Message-ID: <52569CAE.50209@python.org> Am 10.10.2013 06:41, schrieb yoav glazner: > I'm not sure Droping imports is the best way to go, since every python > script/app will import common modules right on the start and it will > still seem like the interpeter boot is slow. > > making modules load faster seems like a better approch Not every script uses the re or collections module. Especially short running and simple Python programs suffer from import overuse. Every imported module adds extra syscalls and IO, too. These are costly operations, especially on slow or embedded devices. Benchmark of 1000 times "python -c ''" Python 3.4dev with all my experimental patches: Avg: 0.705161 -> 0.443613: 1.59x faster 2.7 -> 3.4dev: Avg: 0.316177 -> 0.669330: 2.12x slower 2.7 -> 3.4dev with all my patches: Avg: 0.314879 -> 0.449556: 1.43x slower http://pastebin.com/NFrpa7Jh Ain't bad! The benchmarks were conducted on a fast 8 core machine with SSD. Christian From dirkjan at ochtman.nl Thu Oct 10 14:36:26 2013 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Thu, 10 Oct 2013 14:36:26 +0200 Subject: [Python-Dev] Python startup time In-Reply-To: <52569CAE.50209@python.org> References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> <52555200.4080301@python.org> <52556874.4030707@python.org> <52569CAE.50209@python.org> Message-ID: On Thu, Oct 10, 2013 at 2:25 PM, Christian Heimes wrote: > Benchmark of 1000 times "python -c ''" > > Python 3.4dev with all my experimental patches: > > Avg: 0.705161 -> 0.443613: 1.59x faster > > 2.7 -> 3.4dev: > > Avg: 0.316177 -> 0.669330: 2.12x slower > > 2.7 -> 3.4dev with all my patches: > > Avg: 0.314879 -> 0.449556: 1.43x slower > > http://pastebin.com/NFrpa7Jh > > Ain't bad! The benchmarks were conducted on a fast 8 core machine with SSD. This seems promising. What OS are you using? On an older Linux server with old-style HD's, the difference between 2.7 and 3.2 is much larger for me: Avg: 0.0312 -> 0.1422: 4.56x slower (In this case, I think it might be more useful to report as 0.11s faster, though.) Cheers, Dirkjan From solipsis at pitrou.net Thu Oct 10 15:38:15 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 10 Oct 2013 15:38:15 +0200 Subject: [Python-Dev] Python startup time References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> <52555200.4080301@python.org> <52556874.4030707@python.org> <52569CAE.50209@python.org> Message-ID: <20131010153815.2905a0b2@pitrou.net> Le Thu, 10 Oct 2013 14:36:26 +0200, Dirkjan Ochtman a ?crit : > On Thu, Oct 10, 2013 at 2:25 PM, Christian Heimes > wrote: > > Benchmark of 1000 times "python -c ''" > > > > Python 3.4dev with all my experimental patches: > > > > Avg: 0.705161 -> 0.443613: 1.59x faster > > > > 2.7 -> 3.4dev: > > > > Avg: 0.316177 -> 0.669330: 2.12x slower > > > > 2.7 -> 3.4dev with all my patches: > > > > Avg: 0.314879 -> 0.449556: 1.43x slower > > > > http://pastebin.com/NFrpa7Jh > > > > Ain't bad! The benchmarks were conducted on a fast 8 core machine > > with SSD. > > This seems promising. What OS are you using? On an older Linux server > with old-style HD's, the difference between 2.7 and 3.2 is much larger > for me: 3.2 isn't the same as 3.4. Thanks Christian for doing this, this is promising! Regards Antoine. From victor.stinner at gmail.com Thu Oct 10 16:06:31 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 10 Oct 2013 16:06:31 +0200 Subject: [Python-Dev] Python startup time In-Reply-To: <52567EE6.9060000@python.org> References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> <52555200.4080301@python.org> <52556874.4030707@python.org> <52567EE6.9060000@python.org> Message-ID: Hi, In an old issue, I proposed a change to not load the sysconfig module when it's not needed. Nobody reviewed the patch, the issue was closed. http://bugs.python.org/issue14057 When -s or -I option is used, we may skip completly the sysconfig module. (It's already the case when -S is used.) By the way, we should probably remove the site module. It's a pain for the startup time :-) Victor 2013/10/10 Christian Heimes : > Am 10.10.2013 02:18, schrieb Eric Snow: >> On Wed, Oct 9, 2013 at 8:30 AM, Christian Heimes >> wrote: >>> The os module imports MutableMapping from collections.abc. That >>> import adds collections, collections.abc and eight more modules. >>> I'm not sure if we can do anything about it, though. >> >> Well, that depends on how much we want to eliminate those 10 >> imports. :) Both environ and environb could be turned into lazy >> wrappers around an _Environ-created-when-needed. If we used a >> custom module type for os [1], then adding descriptors for the two >> attributes is a piece of cake. As it is, with a little metaclass >> magic (or even with explicit wrapping of the various dunder >> methods), we could drop those 10 imports from startup. > > We don't have to use a custom module type to get rid of these imports > (but I like to get my hands a piece of chocolate cake *g*). We can > either implement yet another mutable mapping class for the os module. > That would remove the dependency on collections.abc. > > Or we have to juggle the modules a bit so we can get to MutableMapping > without the extra stuff from collections.__init__. The abc and > _weakset modules are already loaded by the io module. Only > collections.__init__ imports _collections, operator, keyword, heapq, > itertools and reprlib. > > I implemented both as an experiment. A lean and mean MutableMapping > works but it involves some code duplication. Next I moved > collections.abc to its former place _abcoll and installed a new > collections.abc module as facade. > > $ hg mv Lib/collections/abc.py Lib/_abcoll.py > $ echo "from _abcoll import *" > Lib/collections/abc.py > $ echo "from _abcoll import __all__" >> Lib/collections/abc.py > $ sed -i "s/collections\.abc/_abcoll/" Lib/os.py > > > With three additional patches I'm down 19 modules: > > $ ./python -c "import sys; print(len(sys.modules))" > 34 > $ hg revert --all . > $ ./python -c "import sys; print(len(sys.modules))" > 53 > > Christian > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From benjamin at python.org Thu Oct 10 16:26:25 2013 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 10 Oct 2013 10:26:25 -0400 Subject: [Python-Dev] Python startup time In-Reply-To: <20131009122956.5c1ae34c@pitrou.net> References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> Message-ID: 2013/10/9 Antoine Pitrou : > Le Wed, 9 Oct 2013 10:29:30 +0200, > Antoine Pitrou a ?crit : >> Le Tue, 8 Oct 2013 15:43:40 -0400, >> Benjamin Peterson a ?crit : >> >> > 2013/10/8 R. David Murray : >> > > In this context, if we'd been *really* smart-lazy in CPython >> > > development, we'd have kept the memory and startup-time >> > > and...well, we probably do pretty well on CPU actually...smaller, >> > > so that when smartphones came along Python would have been the >> > > first high level language used on them, because it fit. Then >> > > we'd all be able to be *much* lazier now :) >> > >> > Even on desktop, startup time leaves a lot to be desired. >> >> That's true. Anyone have any ideas to improve it? > > It's difficult to identify significant contributors but some possible > factors: > - marshal.loads() has become twice slower in 3.x (compared to 2.7) > - instantiating a class is slow (type('foo', (), {}) takes around 25ms > here) Do you mean microsecond? $ ./python -m timeit "type('foo', (), {})" 10000 loops, best of 3: 25.9 usec per loop -- Regards, Benjamin From solipsis at pitrou.net Thu Oct 10 16:31:02 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 10 Oct 2013 16:31:02 +0200 Subject: [Python-Dev] Python startup time References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> Message-ID: <20131010163102.3837c390@pitrou.net> Le Thu, 10 Oct 2013 10:26:25 -0400, Benjamin Peterson a ?crit : > 2013/10/9 Antoine Pitrou : > > Le Wed, 9 Oct 2013 10:29:30 +0200, > > Antoine Pitrou a ?crit : > >> Le Tue, 8 Oct 2013 15:43:40 -0400, > >> Benjamin Peterson a ?crit : > >> > >> > 2013/10/8 R. David Murray : > >> > > In this context, if we'd been *really* smart-lazy in CPython > >> > > development, we'd have kept the memory and startup-time > >> > > and...well, we probably do pretty well on CPU > >> > > actually...smaller, so that when smartphones came along Python > >> > > would have been the first high level language used on them, > >> > > because it fit. Then we'd all be able to be *much* lazier > >> > > now :) > >> > > >> > Even on desktop, startup time leaves a lot to be desired. > >> > >> That's true. Anyone have any ideas to improve it? > > > > It's difficult to identify significant contributors but some > > possible factors: > > - marshal.loads() has become twice slower in 3.x (compared to 2.7) > > - instantiating a class is slow (type('foo', (), {}) takes around > > 25ms here) > > Do you mean microsecond? > > $ ./python -m timeit "type('foo', (), {})" > 10000 loops, best of 3: 25.9 usec per loop Yes, I meant that. cheers Antoine. From storchaka at gmail.com Thu Oct 10 17:10:28 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 10 Oct 2013 18:10:28 +0300 Subject: [Python-Dev] Python startup time In-Reply-To: <52569CAE.50209@python.org> References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> <52555200.4080301@python.org> <52556874.4030707@python.org> <52569CAE.50209@python.org> Message-ID: 10.10.13 15:25, Christian Heimes ???????(??): > Benchmark of 1000 times "python -c ''" What about "python -S -c ''"? From montagne29 at wanadoo.fr Thu Oct 10 11:54:06 2013 From: montagne29 at wanadoo.fr (Bastien Montagne) Date: Thu, 10 Oct 2013 11:54:06 +0200 Subject: [Python-Dev] Fix IO Encoding issue for embedded Python under Windows - Issue16129 Message-ID: <5256793E.9060209@wanadoo.fr> Hi Py Devs, As this is my first mail here, first a short presentation: my name is Bastien Montagne (aka mont29), I live in France, and I?ve been using Python for a fair amount of years now in various projects, amongst which Blender 3D software. I?m also a Blender dev (C & Py) since a few years. Now to the topic, to summarize http://bugs.python.org/issue16129 , an application dynamically linking against Python under MS-Windows has no way to control Python's environment, and hence cannot set PYTHONIOENCODING to get another encoding than default Windows' console (which is usually 8bit, like cp437). This is really annoying under Blender, as we use utf-8 strings, users having e.g. "special" chars in their file paths keep getting python errors instead of nice prints? Following Nick Coghlan?s suggestions, I made a patch which solves that issue by adding Py_SetStandardStreamEncoding() to the API, which allows embedding code to directly specify standard IO encodings. So now I?m waiting for reviews & comments from py devs, in the hope that this can make it into python code before 3.4 feature freeze (patch: http://bugs.python.org/file31985/setstdio.diff ). Best regards, Bastien From brett at python.org Thu Oct 10 18:43:49 2013 From: brett at python.org (Brett Cannon) Date: Thu, 10 Oct 2013 12:43:49 -0400 Subject: [Python-Dev] Python startup time In-Reply-To: <20131010153815.2905a0b2@pitrou.net> References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> <52555200.4080301@python.org> <52556874.4030707@python.org> <52569CAE.50209@python.org> <20131010153815.2905a0b2@pitrou.net> Message-ID: On Thu, Oct 10, 2013 at 9:38 AM, Antoine Pitrou wrote: > Le Thu, 10 Oct 2013 14:36:26 +0200, > Dirkjan Ochtman a ?crit : > > > On Thu, Oct 10, 2013 at 2:25 PM, Christian Heimes > > wrote: > > > Benchmark of 1000 times "python -c ''" > > > > > > Python 3.4dev with all my experimental patches: > > > > > > Avg: 0.705161 -> 0.443613: 1.59x faster > > > > > > 2.7 -> 3.4dev: > > > > > > Avg: 0.316177 -> 0.669330: 2.12x slower > > > > > > 2.7 -> 3.4dev with all my patches: > > > > > > Avg: 0.314879 -> 0.449556: 1.43x slower > > > > > > http://pastebin.com/NFrpa7Jh > > > > > > Ain't bad! The benchmarks were conducted on a fast 8 core machine > > > with SSD. > > > > This seems promising. What OS are you using? On an older Linux server > > with old-style HD's, the difference between 2.7 and 3.2 is much larger > > for me: > > 3.2 isn't the same as 3.4. > And I think that is a key point as imports sped up a good deal in Python 3.3 thanks to the stat caching. So if you want to compare 3.4 to 3.3 that makes sense. And if you want to compare 2.7 to 3.4 as a selling point as startup is the worst benchmark performer in that comparison then fine. But otherwise leave 3.0 - 3.2 out of the discussion as they are red herrings. And as to the suggestion of speeding up import itself: good luck with that without changing semantics. =) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Thu Oct 10 20:40:42 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 10 Oct 2013 12:40:42 -0600 Subject: [Python-Dev] Python startup time In-Reply-To: <52567EE6.9060000@python.org> References: <20131006222752.Horde.JYTyB-I3YCW1THAVwguACw1@webmail.df.eu> <20131008153353.841BC2507F3@webabinitio.net> <20131008193918.B3F43250164@webabinitio.net> <20131009102930.4646de4f@pitrou.net> <20131009122956.5c1ae34c@pitrou.net> <52555200.4080301@python.org> <52556874.4030707@python.org> <52567EE6.9060000@python.org> Message-ID: On Thu, Oct 10, 2013 at 4:18 AM, Christian Heimes wrote: > We don't have to use a custom module type to get rid of these imports > (but I like to get my hands a piece of chocolate cake *g*). We can > either implement yet another mutable mapping class for the os module. > That would remove the dependency on collections.abc. > > Or we have to juggle the modules a bit so we can get to MutableMapping > without the extra stuff from collections.__init__. The abc and > _weakset modules are already loaded by the io module. Only > collections.__init__ imports _collections, operator, keyword, heapq, > itertools and reprlib. I've created a ticket for the os/collections issue: http://bugs.python.org/issue19218. I also put a patch on there that takes a metaclass approach. It removes the need to fiddle with the collections package or to implement the other MutableMapping methods (along with the view classes) on _Environ. -eric From tjreedy at udel.edu Thu Oct 10 21:04:30 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 10 Oct 2013 15:04:30 -0400 Subject: [Python-Dev] [Python-checkins] peps: Revise PEP 453 to be docs-only for 2.7 & 3.3 In-Reply-To: <3cwZsf27c8z7LjN@mail.python.org> References: <3cwZsf27c8z7LjN@mail.python.org> Message-ID: <5256FA3E.20907@udel.edu> On 10/10/2013 10:48 AM, nick.coghlan wrote: > http://hg.python.org/peps/rev/405b80d54b7d > changeset: 5188:405b80d54b7d > user: Nick Coghlan > date: Fri Oct 11 00:47:47 2013 +1000 > summary: > Revise PEP 453 to be docs-only for 2.7 & 3.3 > > - all functional changes are now 3.4 only > - still proposes docs changes for 2.7 & 3.3 > - notes current effort to create a Windows installer for pip > - notes possibility of a future PEP to provide a combined > CPython 2.7, pip and Python Launcher installer from python.org > +`Installing Python Modules `__ guide in > +Python 2.7, 3.3 and 3.4 be updated to officially recommend the use of ``pip`` > +as the default installer for Python packages, and that appropriate technical > +changes be made in Python 3.4 to provide ``pip`` by default in support of > +that recommendation. Does one need a separate installation of pip for each Python version or is pip 'separate' enough that one installation works for all Python versions? Is the answer to this the same on all systems? If the answer is 'one' and 'yes', the 2.7 and 3.3 docs should say that installing 3.4 also installs pip and that this is all one needs to do. > +While this PEP only proposes documentation changes for Python 2.7, once > +``pip`` has a Windows installer available, a separate PEP will be created > +and submitted proposing the creation and distribution of aggregate installers > +for future CPython 2.7 maintenance releases that combine the CPython, > +``pip`` and Python Launcher for Windows installers into a single download > +(the separate downloads would still remain available - the aggregate > +installers would be provided as a convenience, and as a clear indication > +of the recommended operating environment for Python in Windows systems). If the combined installer is an optional convenience, I would not think a PEP necessary. I am assuming that the combined installer would not add a module to /lib any more than two separate installers would, and hence would not be adding a feature to 2.7 itself. -- Terry Jan Reedy From ncoghlan at gmail.com Fri Oct 11 00:10:06 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 11 Oct 2013 08:10:06 +1000 Subject: [Python-Dev] Fix IO Encoding issue for embedded Python under Windows - Issue16129 In-Reply-To: <5256793E.9060209@wanadoo.fr> References: <5256793E.9060209@wanadoo.fr> Message-ID: On 11 Oct 2013 01:22, "Bastien Montagne" wrote: > > Hi Py Devs, > > As this is my first mail here, first a short presentation: my name is Bastien Montagne (aka mont29), I live in France, and I?ve been using Python for a fair amount of years now in various projects, amongst which Blender 3D software. I?m also a Blender dev (C & Py) since a few years. > > Now to the topic, to summarize http://bugs.python.org/issue16129 , an application dynamically linking against Python under MS-Windows has no way to control Python's environment, and hence cannot set PYTHONIOENCODING to get another encoding than default Windows' console (which is usually 8bit, like cp437). This is really annoying under Blender, as we use utf-8 strings, users having e.g. "special" chars in their file paths keep getting python errors instead of nice prints? > > Following Nick Coghlan?s suggestions, I made a patch which solves that issue by adding Py_SetStandardStreamEncoding() to the API, which allows embedding code to directly specify standard IO encodings. So now I?m waiting for reviews & comments from py devs, in the hope that this can make it into python code before 3.4 feature freeze (patch: http://bugs.python.org/file31985/setstdio.diff ). Sorry for the radio silence - this is on my list to look at before alpha 4. (The main thing ahead of it on the to do list was the PEP 453 update to limit the functional changes to 3.4 only, which is now with Donald for review before we submit it for pronouncement). However, if someone else wanted to tackle Bastien's patch before I get to it, I certainly wouldn't complain :) Cheers, Nick. > > Best regards, > Bastien > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Oct 11 00:16:59 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 11 Oct 2013 08:16:59 +1000 Subject: [Python-Dev] [Python-checkins] peps: Revise PEP 453 to be docs-only for 2.7 & 3.3 In-Reply-To: <5256FA3E.20907@udel.edu> References: <3cwZsf27c8z7LjN@mail.python.org> <5256FA3E.20907@udel.edu> Message-ID: On 11 Oct 2013 05:04, "Terry Reedy" wrote: > > On 10/10/2013 10:48 AM, nick.coghlan wrote: >> >> http://hg.python.org/peps/rev/405b80d54b7d >> changeset: 5188:405b80d54b7d >> user: Nick Coghlan >> date: Fri Oct 11 00:47:47 2013 +1000 >> summary: >> Revise PEP 453 to be docs-only for 2.7 & 3.3 >> >> - all functional changes are now 3.4 only >> - still proposes docs changes for 2.7 & 3.3 >> - notes current effort to create a Windows installer for pip >> - notes possibility of a future PEP to provide a combined >> CPython 2.7, pip and Python Launcher installer from python.org > > >> +`Installing Python Modules `__ guide in >> +Python 2.7, 3.3 and 3.4 be updated to officially recommend the use of ``pip`` >> +as the default installer for Python packages, and that appropriate technical >> +changes be made in Python 3.4 to provide ``pip`` by default in support of >> +that recommendation. > > > Does one need a separate installation of pip for each Python version or is pip 'separate' enough that one installation works for all Python versions? Is the answer to this the same on all systems? If the answer is 'one' and 'yes', the 2.7 and 3.3 docs should say that installing 3.4 also installs pip and that this is all one needs to do. pip is version specific (it's just a normal Python package). >> +While this PEP only proposes documentation changes for Python 2.7, once >> +``pip`` has a Windows installer available, a separate PEP will be created >> +and submitted proposing the creation and distribution of aggregate installers >> +for future CPython 2.7 maintenance releases that combine the CPython, >> +``pip`` and Python Launcher for Windows installers into a single download >> +(the separate downloads would still remain available - the aggregate >> +installers would be provided as a convenience, and as a clear indication >> +of the recommended operating environment for Python in Windows systems). > > > If the combined installer is an optional convenience, I would not think a PEP necessary. I am assuming that the combined installer would not add a module to /lib any more than two separate installers would, and hence would not be adding a feature to 2.7 itself. That PEP (if created) would be for publishing such an aggregate installer from the 2.7 release page on python.org. Since it would be proposing more work for the creator of the CPython Windows installers, a PEP seems appropriate. Cheers, Nick. > > -- > Terry Jan Reedy > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Oct 11 09:24:46 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 11 Oct 2013 09:24:46 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). References: <3cwyfX6Mdfz7LjX@mail.python.org> Message-ID: <20131011092446.4736e078@fsol> Hello, On Fri, 11 Oct 2013 07:39:48 +0200 (CEST) raymond.hettinger wrote: > http://hg.python.org/cpython/rev/f014b5f0773f > changeset: 86209:f014b5f0773f > user: Raymond Hettinger > date: Thu Oct 10 22:39:39 2013 -0700 > summary: > Rename contextlib.ignored() to contextlib.ignore(). I don't think that this contextlib.ignore() thing has been discussed a lot. Ezio was -1 on the tracker, and Eric Smith was -0. I'd like to add my -1 too. This is a useless addition (the traditional idiom is perfectly obvious) and makes reading foreign code more tedious by adding superfluous API calls. Please consider reverting. There is absolutely zero use case for this that wasn't already solved by the traditional "except ...: pass" idiom. Regards Antoine. From solipsis at pitrou.net Fri Oct 11 11:00:42 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 11 Oct 2013 11:00:42 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> Message-ID: <20131011110042.37365137@pitrou.net> Let me answer here to Nick's argument on the tracker (made last year, before the patch was committed): > As with many context managers, a key benefit here is in the priming > effect for readers. In this code: > > try: > # Whatever > except (A, B, C): > pass > > the reader doesn't know that (A, B, C) exceptions will be ignored > until the end. The with statement form makes it clear before you > start reading the code that certain exceptions won't propagate: > > with ignored(A, B, C): > # Whatever The problem with this argument is that it assumes a very specific idiom: i.e. writing long "try" blocks in the purpose of silencing exceptions. I'd like to make the following points: - when catching an exception, the common (and recommended) behaviour is to do something else - not merely silencing it. Silencing is not very common in my experience, except in badly written code - when catching an exception, it is recommended for the "try" block to be as slim as possible - so that you don't catch other unintended exceptions by mistake. This is a point I already made in PEP 3151. Many exception classes (OSError, KeyError, RuntimeError...) are polysemic. The bottom line is that there shouldn't be any long "try" blocks followed by a single "except FooException: pass" clause in well-written code. The presence of such an idiom is a strong code smell. Therefore contextlib.ignore() seems aimed at making it easier to write bad code, not good code. I don't think it should exist in the stdlib. Regards Antoine. From victor.stinner at gmail.com Fri Oct 11 11:43:31 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 11 Oct 2013 11:43:31 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131011092446.4736e078@fsol> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> Message-ID: [We are discussing issue #15806.] 2013/10/11 Antoine Pitrou : > I don't think that this contextlib.ignore() thing has been discussed a > lot. If we decide to keep the feature, I would prefer a less generic name: contextlib.ignore_excep(), contextlib.ignore_exception() or contextlib.ignore_exceptions(). > Ezio was -1 on the tracker, and Eric Smith was -0. I'd like to add my > -1 too. This is a useless addition (the traditional idiom is perfectly > obvious) and makes reading foreign code more tedious by adding > superfluous API calls. > > Please consider reverting. There is absolutely zero use case for this > that wasn't already solved by the traditional "except ...: pass" idiom. The purpose of the new context manager is to reduce the number of line of code? It goes a little bit against a Zen Principle: There should be one-- and preferably only one --obvious way to do it. Victor From eric at trueblade.com Fri Oct 11 12:33:33 2013 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 11 Oct 2013 06:33:33 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131011110042.37365137@pitrou.net> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011110042.37365137@pitrou.net> Message-ID: <5257D3FD.9070801@trueblade.com> On 10/11/2013 5:00 AM, Antoine Pitrou wrote: > > Let me answer here to Nick's argument on the tracker (made last year, > before the patch was committed): > >> As with many context managers, a key benefit here is in the priming >> effect for readers. In this code: >> >> try: >> # Whatever >> except (A, B, C): >> pass >> >> the reader doesn't know that (A, B, C) exceptions will be ignored >> until the end. The with statement form makes it clear before you >> start reading the code that certain exceptions won't propagate: >> >> with ignored(A, B, C): >> # Whatever > > The problem with this argument is that it assumes a very specific idiom: > i.e. writing long "try" blocks in the purpose of silencing exceptions. > > I'd like to make the following points: > > - when catching an exception, the common (and recommended) behaviour is > to do something else - not merely silencing it. Silencing is not very > common in my experience, except in badly written code > > - when catching an exception, it is recommended for the "try" block to > be as slim as possible - so that you don't catch other unintended > exceptions by mistake. This is a point I already made in PEP 3151. > Many exception classes (OSError, KeyError, RuntimeError...) are > polysemic. > > The bottom line is that there shouldn't be any long "try" blocks > followed by a single "except FooException: pass" clause in well-written > code. The presence of such an idiom is a strong code smell. > > Therefore contextlib.ignore() seems aimed at making it easier to write > bad code, not good code. I don't think it should exist in the stdlib. In case anyone wants to look at the original issue, it's http://bugs.python.org/issue15806 And Antoine has again taught me a new word: polysemic: having more than one meaning; having multiple meanings -- Eric. From ned at nedbatchelder.com Fri Oct 11 13:24:06 2013 From: ned at nedbatchelder.com (Ned Batchelder) Date: Fri, 11 Oct 2013 07:24:06 -0400 Subject: [Python-Dev] PEPs shouldn't be considered docs Message-ID: <5257DFD6.7010701@nedbatchelder.com> I wanted to teach a co-worker about "from __future__ import absolute_import" today, so I thought I'd point them at the docs. The page for "__future__" starts with a bunch of internal details that almost no one needs to know. There's a table at the end that mentions the actual importable names, including "absolute_import", but says nothing about then except to link to a PEP. The absolute imports PEP has no simple description of what it does. Like many PEPs, it's mostly a summary of the debate around the design of the feature. The closest the PEP comes to describing the behavior of "absolute_import" is this parenthetical: For the second problem, it is proposed that all import statements be absolute by default (searching sys.path only) with special syntax (leading dots) for accessing package-relative imports. And notice: that sentence describes it as a "proposal." I'd like to suggest that we not consider PEPs to be documentation. If a PEP has a good succinct description of how something works, then let's copy that text into the documentation at an appropriate place. If a PEP doesn't have such a description, then all the more reason not to send readers there. A link to the PEP is appropriate as a "see also" in the docs, but we shouldn't pretend that a PEP addresses the needs of people new to the feature. --Ned. -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Fri Oct 11 14:04:28 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 11 Oct 2013 15:04:28 +0300 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <5257D3FD.9070801@trueblade.com> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011110042.37365137@pitrou.net> <5257D3FD.9070801@trueblade.com> Message-ID: 11.10.13 13:33, Eric V. Smith ???????(??): > And Antoine has again taught me a new word: > polysemic: having more than one meaning; having multiple meanings There is no such word in my dictionaries. :( Only polysemous and polysemantic. From stefan at bytereef.org Fri Oct 11 14:24:29 2013 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 11 Oct 2013 14:24:29 +0200 Subject: [Python-Dev] C extension import time Message-ID: <20131011122428.GA22353@sleipnir.bytereef.org> Hi, recently there has been some talk about reducing import times. It seems that the current import strategy for C extensions (i.e. importing the extension at the bottom of the .py file) is quite slow: ==================== import sys for i in range(10000): import decimal del sys.modules('decimal') del sys.modules('_decimal') ==================== With the regular decimal.py the script is about 9 times slower than running it with a modified decimal.py that just contains "from _decimal import *". I don't know enough about the import mechanism to say whether this is a particularly good benchmark. If it is, perhaps we should devise another strategy for loading C extensions. Stefan Krah From stefan at bytereef.org Fri Oct 11 14:32:32 2013 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 11 Oct 2013 14:32:32 +0200 Subject: [Python-Dev] C extension import time In-Reply-To: <20131011122428.GA22353@sleipnir.bytereef.org> References: <20131011122428.GA22353@sleipnir.bytereef.org> Message-ID: <20131011123232.GA22449@sleipnir.bytereef.org> Stefan Krah wrote: > import sys > > for i in range(10000): > import decimal > del sys.modules('decimal') > del sys.modules('_decimal') ^^^ This happens when a Linux user is forced to use Putty :( From breamoreboy at yahoo.co.uk Fri Oct 11 14:51:24 2013 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Fri, 11 Oct 2013 13:51:24 +0100 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <5257D3FD.9070801@trueblade.com> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011110042.37365137@pitrou.net> <5257D3FD.9070801@trueblade.com> Message-ID: On 11/10/2013 11:33, Eric V. Smith wrote: > > And Antoine has again taught me a new word: > polysemic: having more than one meaning; having multiple meanings > IMHO a poor word to use. I'm a middle aged Brit who's never heard of it so people who have English as a second language have little or no chance :( -- Roses are red, Violets are blue, Most poems rhyme, But this one doesn't. Mark Lawrence From solipsis at pitrou.net Fri Oct 11 14:51:56 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 11 Oct 2013 14:51:56 +0200 Subject: [Python-Dev] C extension import time References: <20131011122428.GA22353@sleipnir.bytereef.org> Message-ID: <20131011145156.3d441797@pitrou.net> Le Fri, 11 Oct 2013 14:24:29 +0200, Stefan Krah a ?crit : > Hi, > > recently there has been some talk about reducing import times. It > seems that the current import strategy for C extensions (i.e. > importing the extension at the bottom of the .py file) is quite slow: > > > ==================== > > import sys > > for i in range(10000): > import decimal > del sys.modules('decimal') > del sys.modules('_decimal') > > ==================== Try the following: $ ./python -m timeit -s "modname='decimal'; import sys" \ "__import__(modname); del sys.modules[modname]" 1000 loops, best of 3: 2.21 msec per loop $ ./python -m timeit -s "modname='_decimal'; import sys" \ "__import__(modname); del sys.modules[modname]" 10000 loops, best of 3: 112 usec per loop Now compare with the time taken to simply find the module loader: $ ./python -m timeit -s "modname='_decimal'; import importlib" \ "importlib.find_loader(modname)" 10000 loops, best of 3: 87.2 usec per loop So find_loader() is the bigger contributor here. Note that find_loader() can have varying performance: $ touch empty.py $ ./python -m timeit -s "modname='empty'; import importlib" \ "importlib.find_loader(modname)" 10000 loops, best of 3: 24.4 usec per loop When trying to improve import performance, I noticed that merely diagnosing the bottlenecks is not always easy. Regards Antoine. From benjamin at python.org Fri Oct 11 16:07:10 2013 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 11 Oct 2013 10:07:10 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011110042.37365137@pitrou.net> <5257D3FD.9070801@trueblade.com> Message-ID: 2013/10/11 Mark Lawrence : > On 11/10/2013 11:33, Eric V. Smith wrote: >> >> >> And Antoine has again taught me a new word: >> polysemic: having more than one meaning; having multiple meanings >> > > IMHO a poor word to use. I'm a middle aged Brit who's never heard of it so > people who have English as a second language have little or no chance :( OTOH, it's etymology is quite clear, so those with a little in experience in Greek should have no trouble. -- Regards, Benjamin From benjamin at python.org Fri Oct 11 16:10:03 2013 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 11 Oct 2013 10:10:03 -0400 Subject: [Python-Dev] PEPs shouldn't be considered docs In-Reply-To: <5257DFD6.7010701@nedbatchelder.com> References: <5257DFD6.7010701@nedbatchelder.com> Message-ID: 2013/10/11 Ned Batchelder : > I wanted to teach a co-worker about "from __future__ import absolute_import" > today, so I thought I'd point them at the docs. The page for "__future__" > starts with a bunch of internal details that almost no one needs to know. > There's a table at the end that mentions the actual importable names, > including "absolute_import", but says nothing about then except to link to a > PEP. > > The absolute imports PEP has no simple description of what it does. Like > many PEPs, it's mostly a summary of the debate around the design of the > feature. The closest the PEP comes to describing the behavior of > "absolute_import" is this parenthetical: > > For the second problem, it is proposed that all import statements be > absolute by default (searching sys.path only) with special syntax (leading > dots) for accessing package-relative imports. > > And notice: that sentence describes it as a "proposal." > > I'd like to suggest that we not consider PEPs to be documentation. If a PEP > has a good succinct description of how something works, then let's copy that > text into the documentation at an appropriate place. If a PEP doesn't have > such a description, then all the more reason not to send readers there. +1 The writing required to specificy and adovocate a feature are usually quite different from that needed to document it. PEPs also get out of date rather quickly. -- Regards, Benjamin From barry at python.org Fri Oct 11 16:35:56 2013 From: barry at python.org (Barry Warsaw) Date: Fri, 11 Oct 2013 10:35:56 -0400 Subject: [Python-Dev] PEPs shouldn't be considered docs In-Reply-To: <5257DFD6.7010701@nedbatchelder.com> References: <5257DFD6.7010701@nedbatchelder.com> Message-ID: <20131011103556.09fd6606@anarchist> On Oct 11, 2013, at 07:24 AM, Ned Batchelder wrote: >I'd like to suggest that we not consider PEPs to be documentation. Absolutely +1. That was never the intention behind PEPs. -Barry From solipsis at pitrou.net Fri Oct 11 16:38:01 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 11 Oct 2013 16:38:01 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011110042.37365137@pitrou.net> <5257D3FD.9070801@trueblade.com> Message-ID: <20131011163801.4fc361f2@pitrou.net> Le Fri, 11 Oct 2013 13:51:24 +0100, Mark Lawrence a ?crit : > On 11/10/2013 11:33, Eric V. Smith wrote: > > > > And Antoine has again taught me a new word: > > polysemic: having more than one meaning; having multiple meanings > > > > IMHO a poor word to use. I'm a middle aged Brit who's never heard of > it so people who have English as a second language have little or no > chance :( English is my second language. Regards Antoine. From barry at python.org Fri Oct 11 16:47:23 2013 From: barry at python.org (Barry Warsaw) Date: Fri, 11 Oct 2013 10:47:23 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131011092446.4736e078@fsol> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> Message-ID: <20131011104723.05748928@anarchist> On Oct 11, 2013, at 09:24 AM, Antoine Pitrou wrote: >I don't think that this contextlib.ignore() thing has been discussed a >lot. > >Ezio was -1 on the tracker, and Eric Smith was -0. I'd like to add my >-1 too. This is a useless addition (the traditional idiom is perfectly >obvious) and makes reading foreign code more tedious by adding >superfluous API calls. > >Please consider reverting. There is absolutely zero use case for this >that wasn't already solved by the traditional "except ...: pass" idiom. I'm +0; I think it's a nice little addition that could be useful, but I don't care enough to wait for 3.4 to delete similar code in a my own programs. To bikeshed though: why was `ignored` changed to `ignore`? The former reads better to me, and I don't think *that* particular change was discussed at all in the tracker afaict. -Barry From stefan at bytereef.org Fri Oct 11 17:01:35 2013 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 11 Oct 2013 17:01:35 +0200 Subject: [Python-Dev] C extension import time In-Reply-To: <20131011145156.3d441797@pitrou.net> References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> Message-ID: <20131011150135.GA23302@sleipnir.bytereef.org> Antoine Pitrou wrote: > Try the following: > > $ ./python -m timeit -s "modname='decimal'; import sys" \ > "__import__(modname); del sys.modules[modname]" > 1000 loops, best of 3: 2.21 msec per loop > > $ ./python -m timeit -s "modname='_decimal'; import sys" \ > "__import__(modname); del sys.modules[modname]" > 10000 loops, best of 3: 112 usec per loop > > > Now compare with the time taken to simply find the module loader: > > $ ./python -m timeit -s "modname='_decimal'; import importlib" \ > "importlib.find_loader(modname)" > 10000 loops, best of 3: 87.2 usec per loop > > > So find_loader() is the bigger contributor here. I'm getting about the same values as above. I may be misunderstanding something, but I wanted to reduce the difference between the 2.21 msec and the 112 usec. So the first step I tried is something horrible (typing from memory): try: import _decimal except ImportError: else: from _decimal import * That way the 2.21 msec are reduced to 912 usec, but the indentation is an abomination. Another radical way would be to have importlib detect that the C version is present and load it in the first place. For _decimal this should work, since it's self-contained. For other modules probably not, so there would need to be some way of distiguishing between modules that are self-contained and modules that are not. Stefan Krah From solipsis at pitrou.net Fri Oct 11 17:20:34 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 11 Oct 2013 17:20:34 +0200 Subject: [Python-Dev] C extension import time References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> Message-ID: <20131011172034.639648ff@pitrou.net> Le Fri, 11 Oct 2013 17:01:35 +0200, Stefan Krah a ?crit : > > I'm getting about the same values as above. I may be misunderstanding > something, but I wanted to reduce the difference between the 2.21 msec > and the 112 usec. So you aren't complaining about C extension import time, but Python code import time, right? > So the first step I tried is something horrible (typing from memory): > > try: > import _decimal > except ImportError: > > else: > from _decimal import * > > That way the 2.21 msec are reduced to 912 usec, but the indentation is > an abomination. Just create a _pydecimal module (like _pyio). Regards Antoine. From fuzzyman at voidspace.org.uk Fri Oct 11 17:23:16 2013 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 11 Oct 2013 16:23:16 +0100 Subject: [Python-Dev] PyCon US 2014 In-Reply-To: References: Message-ID: <6D03A0D0-BBED-462F-B033-3FB3F492AEE2@voidspace.org.uk> On 10 Oct 2013, at 01:53, Eric Snow wrote: > Registration is now open for PyCon US 2014. Are there any plans yet > for the language summit? Just the day (e.g. Thursday April 10) will > suffice. Then we can make arrangements appropriately. Thanks. Sorry for the late reply. Yes there will be a language summit. I'm pretty sure it will be on Thursday as the last few years. I'm getting definite confirmation of this. Michael > > -eric > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From hao.hu at nsn.com Fri Oct 11 09:37:45 2013 From: hao.hu at nsn.com (Hu, Hao (NSN - CN/Beijing)) Date: Fri, 11 Oct 2013 07:37:45 +0000 Subject: [Python-Dev] Python IDEL issue Message-ID: <495881D629037B40A8C279DB63D98C4D42A06C@SGSIMBX008.nsn-intra.net> Hi, When I open IDLE(Python GUI) from the windows start menu, it always open Python Shell. When I right click on a *.py file, I cannot see the menu item "open with IDLE". Even after I uninstall and reinstall Python several times with different versions, the issue remains. I guess it is because something prerequisite is not installed on my computer. But do you know what is it? -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at bytereef.org Fri Oct 11 17:38:20 2013 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 11 Oct 2013 17:38:20 +0200 Subject: [Python-Dev] C extension [PEP 399] import time In-Reply-To: <20131011172034.639648ff@pitrou.net> References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> Message-ID: <20131011153820.GA23647@sleipnir.bytereef.org> Antoine Pitrou wrote: > > > > I'm getting about the same values as above. I may be misunderstanding > > something, but I wanted to reduce the difference between the 2.21 msec > > and the 112 usec. > > So you aren't complaining about C extension import time, but Python > code import time, right? Right, that wasn't clear from the title. Somehow I thought that PEP 399 explicitly recommended importing the C extension at the end of the file, but in fact it only says "This [importing] is typically done at the end of the module". > Just create a _pydecimal module (like _pyio). Thanks, I'll try that. Stefan Krah From status at bugs.python.org Fri Oct 11 18:07:41 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 11 Oct 2013 18:07:41 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20131011160741.AFEC456A52@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-10-04 - 2013-10-11) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4240 (-16) closed 26757 (+75) total 30997 (+59) Open issues with patches: 1960 Issues opened (37) ================== #18229: attribute headers of http.server.BaseHTTPRequestHandler someti http://bugs.python.org/issue18229 reopened by joru #18716: Deprecate the formatter module http://bugs.python.org/issue18716 reopened by brett.cannon #19167: sqlite3 cursor.description varies across Linux (3.3.1), Win32 http://bugs.python.org/issue19167 opened by mpb #19170: telnetlib: use selectors http://bugs.python.org/issue19170 opened by neologix #19172: selectors: add keys() method http://bugs.python.org/issue19172 opened by neologix #19173: Expose Queue maxsize parameter to multiprocessing.Pool class http://bugs.python.org/issue19173 opened by noxdafox #19175: Erroneous reference to "integer" in format string grammar http://bugs.python.org/issue19175 opened by davidchambers #19176: DeprecationWarning for doctype() method when subclassing _elem http://bugs.python.org/issue19176 opened by vadmium #19179: doc bug: confusing table of values http://bugs.python.org/issue19179 opened by georg.brandl #19180: some RFC references could be updated http://bugs.python.org/issue19180 opened by georg.brandl #19183: PEP 456 Secure and interchangeable hash algorithm http://bugs.python.org/issue19183 opened by christian.heimes #19184: dis module has incorrect docs for RAISE_VARARGS http://bugs.python.org/issue19184 opened by nedbat #19185: Allow multiprocessing Pool initializer to return values http://bugs.python.org/issue19185 opened by noxdafox #19186: expat symbols should be namespaced in pyexpat again http://bugs.python.org/issue19186 opened by athompson #19189: Improve cross-references in pickle documentation. http://bugs.python.org/issue19189 opened by serhiy.storchaka #19191: os.path.splitext in windows , a little question http://bugs.python.org/issue19191 opened by xiaowei.py #19192: Move test_current_time from test_xmlrpc_net to test_xmlrpc http://bugs.python.org/issue19192 opened by vajrasky #19197: Improve cross-references in shlex documentation http://bugs.python.org/issue19197 opened by serhiy.storchaka #19198: Improve cross-references in cgi documentation http://bugs.python.org/issue19198 opened by serhiy.storchaka #19201: lzma and 'x' mode open http://bugs.python.org/issue19201 opened by oylenshpeegul #19202: Additions to docs http://bugs.python.org/issue19202 opened by Esa.Peuha #19203: Improve cross-references in curses howto http://bugs.python.org/issue19203 opened by serhiy.storchaka #19204: Improve cross-references in urllib documentation http://bugs.python.org/issue19204 opened by serhiy.storchaka #19205: Don't import re and sysconfig in site.py http://bugs.python.org/issue19205 opened by christian.heimes #19206: Support disabling file I/O when doing traceback formatting http://bugs.python.org/issue19206 opened by itamarst #19207: Improve cross-references in os documentation http://bugs.python.org/issue19207 opened by serhiy.storchaka #19209: Remove import copyreg from os module http://bugs.python.org/issue19209 opened by christian.heimes #19213: platform.linux_distribution detects Oracle Linux as Red Hat En http://bugs.python.org/issue19213 opened by javawizard #19214: shutil.make_archive should recognize extensions in filenames http://bugs.python.org/issue19214 opened by andreas-h #19216: stat cache for import bootstrap http://bugs.python.org/issue19216 opened by christian.heimes #19217: Calling assertEquals for moderately long list takes too long http://bugs.python.org/issue19217 opened by Jacek.Bzdak #19218: Use of MutableMapping in os module slows down interpreter star http://bugs.python.org/issue19218 opened by eric.snow #19219: speed up marshal.loads() http://bugs.python.org/issue19219 opened by pitrou #19222: gzip and 'x' mode open http://bugs.python.org/issue19222 opened by oylenshpeegul #19223: bz2 and 'x' mode open http://bugs.python.org/issue19223 opened by oylenshpeegul #19224: Make hash(None) consistent among processes http://bugs.python.org/issue19224 opened by hongqn #19225: lack of PyExc_BufferError doc http://bugs.python.org/issue19225 opened by xwhhsprings Most recent 15 issues with no replies (15) ========================================== #19223: bz2 and 'x' mode open http://bugs.python.org/issue19223 #19222: gzip and 'x' mode open http://bugs.python.org/issue19222 #19214: shutil.make_archive should recognize extensions in filenames http://bugs.python.org/issue19214 #19207: Improve cross-references in os documentation http://bugs.python.org/issue19207 #19204: Improve cross-references in urllib documentation http://bugs.python.org/issue19204 #19203: Improve cross-references in curses howto http://bugs.python.org/issue19203 #19198: Improve cross-references in cgi documentation http://bugs.python.org/issue19198 #19197: Improve cross-references in shlex documentation http://bugs.python.org/issue19197 #19189: Improve cross-references in pickle documentation. http://bugs.python.org/issue19189 #19186: expat symbols should be namespaced in pyexpat again http://bugs.python.org/issue19186 #19184: dis module has incorrect docs for RAISE_VARARGS http://bugs.python.org/issue19184 #19180: some RFC references could be updated http://bugs.python.org/issue19180 #19179: doc bug: confusing table of values http://bugs.python.org/issue19179 #19176: DeprecationWarning for doctype() method when subclassing _elem http://bugs.python.org/issue19176 #19157: ipaddress.IPv6Network.hosts function omits network and broadca http://bugs.python.org/issue19157 Most recent 15 issues waiting for review (15) ============================================= #19224: Make hash(None) consistent among processes http://bugs.python.org/issue19224 #19219: speed up marshal.loads() http://bugs.python.org/issue19219 #19218: Use of MutableMapping in os module slows down interpreter star http://bugs.python.org/issue19218 #19216: stat cache for import bootstrap http://bugs.python.org/issue19216 #19209: Remove import copyreg from os module http://bugs.python.org/issue19209 #19207: Improve cross-references in os documentation http://bugs.python.org/issue19207 #19205: Don't import re and sysconfig in site.py http://bugs.python.org/issue19205 #19204: Improve cross-references in urllib documentation http://bugs.python.org/issue19204 #19203: Improve cross-references in curses howto http://bugs.python.org/issue19203 #19202: Additions to docs http://bugs.python.org/issue19202 #19201: lzma and 'x' mode open http://bugs.python.org/issue19201 #19198: Improve cross-references in cgi documentation http://bugs.python.org/issue19198 #19197: Improve cross-references in shlex documentation http://bugs.python.org/issue19197 #19192: Move test_current_time from test_xmlrpc_net to test_xmlrpc http://bugs.python.org/issue19192 #19189: Improve cross-references in pickle documentation. http://bugs.python.org/issue19189 Top 10 most discussed issues (10) ================================= #19219: speed up marshal.loads() http://bugs.python.org/issue19219 31 msgs #3982: support .format for bytes http://bugs.python.org/issue3982 20 msgs #12892: UTF-16 and UTF-32 codecs should reject (lone) surrogates http://bugs.python.org/issue12892 17 msgs #19209: Remove import copyreg from os module http://bugs.python.org/issue19209 14 msgs #19172: selectors: add keys() method http://bugs.python.org/issue19172 12 msgs #19183: PEP 456 Secure and interchangeable hash algorithm http://bugs.python.org/issue19183 11 msgs #19154: AttributeError: 'NoneType' in http/client.py when using select http://bugs.python.org/issue19154 10 msgs #19216: stat cache for import bootstrap http://bugs.python.org/issue19216 10 msgs #19142: Cross-compile fails trying to execute foreign pgen on build ho http://bugs.python.org/issue19142 9 msgs #18758: Fix internal references in the documentation http://bugs.python.org/issue18758 7 msgs Issues closed (71) ================== #1215: documentation doesn't say that you can't handle C segfaults fr http://bugs.python.org/issue1215 closed by python-dev #11585: Documentation 1.8 shows Python 2 example http://bugs.python.org/issue11585 closed by georg.brandl #11789: Extend upon metaclass/type class documentation, here: zope.int http://bugs.python.org/issue11789 closed by georg.brandl #11807: Documentation of add_subparsers lacks information about parame http://bugs.python.org/issue11807 closed by python-dev #12162: Documentation about re \number http://bugs.python.org/issue12162 closed by georg.brandl #12350: Improve stat_result.st_blocks and st_blksize documentation http://bugs.python.org/issue12350 closed by python-dev #13867: misleading comment in weakrefobject.h http://bugs.python.org/issue13867 closed by python-dev #13951: Document that Seg Fault in .so called by ctypes causes the int http://bugs.python.org/issue13951 closed by python-dev #14101: example function in tertools.count docstring is misindented http://bugs.python.org/issue14101 closed by georg.brandl #14224: packaging: path description of resources is mixed up http://bugs.python.org/issue14224 closed by georg.brandl #15172: Document nasm-2.10.01 as required version for openssl http://bugs.python.org/issue15172 closed by georg.brandl #15213: _PyOS_URandom documentation http://bugs.python.org/issue15213 closed by python-dev #15228: os.utime() docs not clear on behavior on nonexistant files http://bugs.python.org/issue15228 closed by python-dev #15264: PyErr_SetFromErrnoWithFilenameObject() undocumented http://bugs.python.org/issue15264 closed by georg.brandl #15329: clarify which deque methods are thread-safe http://bugs.python.org/issue15329 closed by rhettinger #15432: gzip.py: mtime argument only since python 2.7 http://bugs.python.org/issue15432 closed by python-dev #15455: index entries not showing up in glossary http://bugs.python.org/issue15455 closed by georg.brandl #15805: Add stdout redirection tool to contextlib http://bugs.python.org/issue15805 closed by rhettinger #15863: Fine-grained info about Python versions which support changes http://bugs.python.org/issue15863 closed by georg.brandl #15964: SyntaxError in asdl when building 2.7 with system Python 3 http://bugs.python.org/issue15964 closed by eli.bendersky #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which is http://bugs.python.org/issue16742 closed by python-dev #17725: English mistake in Extending and Embedding Python doc page. http://bugs.python.org/issue17725 closed by python-dev #17745: "packaging" no longer planned to be included http://bugs.python.org/issue17745 closed by georg.brandl #18529: Use long dash http://bugs.python.org/issue18529 closed by georg.brandl #18577: lru_cache enhancement: lru_timestamp helper function http://bugs.python.org/issue18577 closed by rhettinger #18646: Improve tutorial entry on 'Lambda Forms'. http://bugs.python.org/issue18646 closed by python-dev #18664: occasional test_threading failure http://bugs.python.org/issue18664 closed by haypo #18764: The pdb print command prints repr instead of str in python3 http://bugs.python.org/issue18764 closed by r.david.murray #18785: Add get_body and iter_attachments to provisional email API http://bugs.python.org/issue18785 closed by r.david.murray #18790: incorrect text in argparse add_help example http://bugs.python.org/issue18790 closed by berker.peksag #18860: Add content manager API to email package http://bugs.python.org/issue18860 closed by r.david.murray #18890: Add a raw_data_manager content manager to the email package. http://bugs.python.org/issue18890 closed by r.david.murray #18927: Lock.acquire() docs incorrect about negative timeout http://bugs.python.org/issue18927 closed by python-dev #18948: deliberately crashing tests should prevent core dumps http://bugs.python.org/issue18948 closed by pitrou #18972: Use argparse in email example scripts http://bugs.python.org/issue18972 closed by serhiy.storchaka #19005: PyIter_Next crashes if passed a non-iterator http://bugs.python.org/issue19005 closed by rhettinger #19067: Built-in range docstrings are not PEP-8 compatible http://bugs.python.org/issue19067 closed by ezio.melotti #19068: Some built-in complex docstrings are not PEP-8 compatible http://bugs.python.org/issue19068 closed by ezio.melotti #19074: Add PySide to GUI FAQ http://bugs.python.org/issue19074 closed by ezio.melotti #19087: bytearray front-slicing not optimized http://bugs.python.org/issue19087 closed by pitrou #19106: Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots http://bugs.python.org/issue19106 closed by eli.bendersky #19129: 6.2.1. Regular Expression Syntax flags http://bugs.python.org/issue19129 closed by georg.brandl #19148: Minor issues with Enum docs http://bugs.python.org/issue19148 closed by ezio.melotti #19156: Enum helper functions test-coverage http://bugs.python.org/issue19156 closed by python-dev #19158: BoundedSemaphore.release() subject to races http://bugs.python.org/issue19158 closed by tim.peters #19161: collections Counter handles nan strangely http://bugs.python.org/issue19161 closed by rhettinger #19166: Unusued variable in test_keys in Lib/test/test_dict.py http://bugs.python.org/issue19166 closed by ezio.melotti #19168: pprint.pprint(..., compact=True) not implemented http://bugs.python.org/issue19168 closed by ezio.melotti #19169: random.py : simple tidying http://bugs.python.org/issue19169 closed by rhettinger #19171: pow() improvement on longs http://bugs.python.org/issue19171 closed by tim.peters #19174: Add range to future_builtins http://bugs.python.org/issue19174 closed by rhettinger #19177: Link to "TLS (Transport Layer Security) and SSL (Secure Socket http://bugs.python.org/issue19177 closed by python-dev #19178: Entries for "module" and "package" in glossary http://bugs.python.org/issue19178 closed by python-dev #19181: ftp.cwi.nl used as example does not exist anymore http://bugs.python.org/issue19181 closed by python-dev #19182: Socket leak in logging.handlers http://bugs.python.org/issue19182 closed by vinay.sajip #19187: Use a set for interned strings http://bugs.python.org/issue19187 closed by rhettinger #19188: Add PySet_CheckExact() http://bugs.python.org/issue19188 closed by rhettinger #19190: Improve cross-references in builtins documentation. http://bugs.python.org/issue19190 closed by serhiy.storchaka #19193: Improve cross-references in tutorial http://bugs.python.org/issue19193 closed by serhiy.storchaka #19194: Improve cross-references in fcntl documentation http://bugs.python.org/issue19194 closed by serhiy.storchaka #19195: Improve cross-references in C API http://bugs.python.org/issue19195 closed by serhiy.storchaka #19196: Improve cross-references in distutils documentation http://bugs.python.org/issue19196 closed by serhiy.storchaka #19199: Remove PyThreadState.tick_counter field http://bugs.python.org/issue19199 closed by python-dev #19200: grammar in forkserver docs http://bugs.python.org/issue19200 closed by python-dev #19208: Bas64.decodestring() returns data instead of throwing exceptio http://bugs.python.org/issue19208 closed by r.david.murray #19210: Unicode Objects in Tuples http://bugs.python.org/issue19210 closed by pitrou #19211: from relative_module import seems to import wrong module http://bugs.python.org/issue19211 closed by r.david.murray #19212: Spam http://bugs.python.org/issue19212 closed by r.david.murray #19215: StringIO.StringIO('foo').readline(0) == 'foo' http://bugs.python.org/issue19215 closed by r.david.murray #19220: Outdated unicodedata.__doc__ http://bugs.python.org/issue19220 closed by python-dev #19221: Upgrade to Unicode 6.3.0 http://bugs.python.org/issue19221 closed by python-dev From g.brandl at gmx.net Fri Oct 11 18:25:23 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 11 Oct 2013 18:25:23 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> Message-ID: Am 11.10.2013 11:43, schrieb Victor Stinner: > [We are discussing issue #15806.] > > 2013/10/11 Antoine Pitrou : >> I don't think that this contextlib.ignore() thing has been discussed a >> lot. > > If we decide to keep the feature, I would prefer a less generic name: > contextlib.ignore_excep() Srsly? :) Georg From g.brandl at gmx.net Fri Oct 11 18:27:06 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 11 Oct 2013 18:27:06 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131011104723.05748928@anarchist> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011104723.05748928@anarchist> Message-ID: Am 11.10.2013 16:47, schrieb Barry Warsaw: > On Oct 11, 2013, at 09:24 AM, Antoine Pitrou wrote: > >>I don't think that this contextlib.ignore() thing has been discussed a >>lot. >> >>Ezio was -1 on the tracker, and Eric Smith was -0. I'd like to add my >>-1 too. This is a useless addition (the traditional idiom is perfectly >>obvious) and makes reading foreign code more tedious by adding >>superfluous API calls. >> >>Please consider reverting. There is absolutely zero use case for this >>that wasn't already solved by the traditional "except ...: pass" idiom. > > I'm +0; I think it's a nice little addition that could be useful, but I don't > care enough to wait for 3.4 to delete similar code in a my own programs. Same here. > To bikeshed though: why was `ignored` changed to `ignore`? The former reads > better to me, and I don't think *that* particular change was discussed at all > in the tracker afaict. Maybe to fit in with other verb-like APIs used as context managers: it's open() not opened(). Georg From barry at python.org Fri Oct 11 18:43:57 2013 From: barry at python.org (Barry Warsaw) Date: Fri, 11 Oct 2013 12:43:57 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011104723.05748928@anarchist> Message-ID: <20131011124357.0dbd8f20@anarchist> On Oct 11, 2013, at 06:27 PM, Georg Brandl wrote: >Maybe to fit in with other verb-like APIs used as context managers: >it's open() not opened(). open() predates context managers, but maybe we need a new convention. with ignore(FileNotFoundError): vs with ignored(FileNotFoundError): To me anyway, the latter sounds better. -Barry From g.rodola at gmail.com Fri Oct 11 19:14:42 2013 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Fri, 11 Oct 2013 19:14:42 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131011110042.37365137@pitrou.net> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011110042.37365137@pitrou.net> Message-ID: On Fri, Oct 11, 2013 at 11:00 AM, Antoine Pitrou wrote: > > Let me answer here to Nick's argument on the tracker (made last year, > before the patch was committed): > > > As with many context managers, a key benefit here is in the priming > > effect for readers. In this code: > > > > try: > > # Whatever > > except (A, B, C): > > pass > > > > the reader doesn't know that (A, B, C) exceptions will be ignored > > until the end. The with statement form makes it clear before you > > start reading the code that certain exceptions won't propagate: > > > > with ignored(A, B, C): > > # Whatever > > The problem with this argument is that it assumes a very specific idiom: > i.e. writing long "try" blocks in the purpose of silencing exceptions. > > I'd like to make the following points: > > - when catching an exception, the common (and recommended) behaviour is > to do something else - not merely silencing it. Silencing is not very > common in my experience, except in badly written code > > - when catching an exception, it is recommended for the "try" block to > be as slim as possible - so that you don't catch other unintended > exceptions by mistake. This is a point I already made in PEP 3151. > Many exception classes (OSError, KeyError, RuntimeError...) are > polysemic. > > The bottom line is that there shouldn't be any long "try" blocks > followed by a single "except FooException: pass" clause in well-written > code. The presence of such an idiom is a strong code smell. > > Therefore contextlib.ignore() seems aimed at making it easier to write > bad code, not good code. I don't think it should exist in the stdlib. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com > I'm with Antoine here. Despite "with ignore(OSError): os.remove" looks appealing to the eye I think many people will end up thinking it's fine to write long "ignore" blocks, because that's perfectly fine when using the "with" statement. Point is this is actually a "try" block disguised as a "with", and "try" blocks should usually be followed by very few indented lines (usually just 1) for a very good reason. -1 --- Giampaolo https://code.google.com/p/pyftpdlib/ https://code.google.com/p/psutil/ https://code.google.com/p/pysendfile/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Fri Oct 11 19:19:18 2013 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 11 Oct 2013 13:19:18 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131011124357.0dbd8f20@anarchist> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011104723.05748928@anarchist> <20131011124357.0dbd8f20@anarchist> Message-ID: <52583316.9010802@trueblade.com> On 10/11/2013 12:43 PM, Barry Warsaw wrote: > On Oct 11, 2013, at 06:27 PM, Georg Brandl wrote: > >> Maybe to fit in with other verb-like APIs used as context managers: >> it's open() not opened(). > > open() predates context managers, but maybe we need a new convention. > > with ignore(FileNotFoundError): > > vs > > with ignored(FileNotFoundError): > > To me anyway, the latter sounds better. I'm still -0, and maybe now -1 on the idea, mostly because it encourages an anti-pattern. But, to continue to paint the shed, shouldn't it be "ignoring", to match "closing"? Eric. From barry at python.org Fri Oct 11 19:24:37 2013 From: barry at python.org (Barry Warsaw) Date: Fri, 11 Oct 2013 13:24:37 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <52583316.9010802@trueblade.com> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011104723.05748928@anarchist> <20131011124357.0dbd8f20@anarchist> <52583316.9010802@trueblade.com> Message-ID: <20131011132437.7d81c40b@anarchist> On Oct 11, 2013, at 01:19 PM, Eric V. Smith wrote: >But, to continue to paint the shed, shouldn't it be "ignoring", to match >"closing"? Maybe so. -Barry From storchaka at gmail.com Fri Oct 11 19:32:45 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 11 Oct 2013 20:32:45 +0300 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131011092446.4736e078@fsol> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> Message-ID: 11.10.13 10:24, Antoine Pitrou ???????(??): > Ezio was -1 on the tracker, and Eric Smith was -0. I'd like to add my > -1 too. This is a useless addition (the traditional idiom is perfectly > obvious) and makes reading foreign code more tedious by adding > superfluous API calls. I am -1 too. But I want to hear more about interact with ExitStack (note that ExitStack itself is not widely used in the stdlib). From rdmurray at bitdance.com Fri Oct 11 19:56:56 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 11 Oct 2013 13:56:56 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> Message-ID: <20131011175656.CB7AC2509A9@webabinitio.net> On Fri, 11 Oct 2013 20:32:45 +0300, Serhiy Storchaka wrote: > 11.10.13 10:24, Antoine Pitrou ??????????????(????): > > Ezio was -1 on the tracker, and Eric Smith was -0. I'd like to add my > > -1 too. This is a useless addition (the traditional idiom is perfectly > > obvious) and makes reading foreign code more tedious by adding > > superfluous API calls. > > I am -1 too. But I want to hear more about interact with ExitStack (note > that ExitStack itself is not widely used in the stdlib). I'm inclined to be -1 as well, because of the anti-pattern encouragement. It seems at first blush like using it with ExitStack would make that tendency worse, so I too would like to hear more about Nick's thoughts on the subject. --David From ethan at stoneleaf.us Fri Oct 11 19:35:46 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 11 Oct 2013 10:35:46 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011110042.37365137@pitrou.net> <5257D3FD.9070801@trueblade.com> Message-ID: <525836F2.80903@stoneleaf.us> On 10/11/2013 05:51 AM, Mark Lawrence wrote: > On 11/10/2013 11:33, Eric V. Smith wrote: >> >> And Antoine has again taught me a new word: >> polysemic: having more than one meaning; having multiple meanings >> > > IMHO a poor word to use. I'm a middle aged Brit who's never heard of it > so people who have English as a second language have little or no chance :( Even us natives can learn! Dictionaries just aren't hard to find any more. ;) -- ~Ethan~ From ethan at stoneleaf.us Fri Oct 11 19:40:21 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 11 Oct 2013 10:40:21 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131011092446.4736e078@fsol> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> Message-ID: <52583805.1060709@stoneleaf.us> -1 to contextlib.ignore(s|d|ing|etc) From barry at python.org Fri Oct 11 20:28:53 2013 From: barry at python.org (Barry Warsaw) Date: Fri, 11 Oct 2013 14:28:53 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> Message-ID: <20131011142853.3d0b972d@anarchist> On Oct 11, 2013, at 08:32 PM, Serhiy Storchaka wrote: >I am -1 too. But I want to hear more about interact with ExitStack (note that >ExitStack itself is not widely used in the stdlib). Yet. :) -Barry From ethan at stoneleaf.us Fri Oct 11 19:39:21 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 11 Oct 2013 10:39:21 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131011124357.0dbd8f20@anarchist> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011104723.05748928@anarchist> <20131011124357.0dbd8f20@anarchist> Message-ID: <525837C9.7030103@stoneleaf.us> On 10/11/2013 09:43 AM, Barry Warsaw wrote: > On Oct 11, 2013, at 06:27 PM, Georg Brandl wrote: > >> Maybe to fit in with other verb-like APIs used as context managers: >> it's open() not opened(). > > open() predates context managers, but maybe we need a new convention. > > with ignore(FileNotFoundError): > > vs > > with ignored(FileNotFoundError): > > To me anyway, the latter sounds better. The practice I follow is to use present tense, singular, unless there's a really good reason not to. Makes remembering a lot easier: was it ignore? ignoring? ignores? ignored? Answer: ignore. -- ~Ethan~ From g.brandl at gmx.net Fri Oct 11 20:37:09 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 11 Oct 2013 20:37:09 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131011132437.7d81c40b@anarchist> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011104723.05748928@anarchist> <20131011124357.0dbd8f20@anarchist> <52583316.9010802@trueblade.com> <20131011132437.7d81c40b@anarchist> Message-ID: Am 11.10.2013 19:24, schrieb Barry Warsaw: > On Oct 11, 2013, at 01:19 PM, Eric V. Smith wrote: > >>But, to continue to paint the shed, shouldn't it be "ignoring", to match >>"closing"? > > Maybe so. Would at least be consistent since both actions (close/ignore) are done at the end of the execution of the block... Georg From v+python at g.nevcal.com Fri Oct 11 20:41:32 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 11 Oct 2013 11:41:32 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <52583316.9010802@trueblade.com> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011104723.05748928@anarchist> <20131011124357.0dbd8f20@anarchist> <52583316.9010802@trueblade.com> Message-ID: <5258465C.1070800@g.nevcal.com> On 10/11/2013 10:19 AM, Eric V. Smith wrote: > On 10/11/2013 12:43 PM, Barry Warsaw wrote: >> On Oct 11, 2013, at 06:27 PM, Georg Brandl wrote: >> >>> Maybe to fit in with other verb-like APIs used as context managers: >>> it's open() not opened(). >> open() predates context managers, but maybe we need a new convention. >> >> with ignore(FileNotFoundError): >> >> vs >> >> with ignored(FileNotFoundError): >> >> To me anyway, the latter sounds better. > I'm still -0, and maybe now -1 on the idea, mostly because it encourages > an anti-pattern. > > But, to continue to paint the shed, shouldn't it be "ignoring", to match > "closing"? Seriously, "with" is the wrong spelling for this using. It should be while ignorning(FileNotFoundError) Insistence on using "with" for the anti-pattern, and proper English, would require: with ignorance_of(FileNotFoundError) :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Fri Oct 11 20:53:42 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 11 Oct 2013 21:53:42 +0300 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <5258465C.1070800@g.nevcal.com> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011104723.05748928@anarchist> <20131011124357.0dbd8f20@anarchist> <52583316.9010802@trueblade.com> <5258465C.1070800@g.nevcal.com> Message-ID: 11.10.13 21:41, Glenn Linderman ???????(??): > Seriously, "with" is the wrong spelling for this using. It should be > > while ignorning(FileNotFoundError) We need extended bool for while condition: http://thedailywtf.com/Articles/What_Is_Truth_0x3f_.aspx From python at mrabarnett.plus.com Fri Oct 11 21:00:47 2013 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 11 Oct 2013 20:00:47 +0100 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <525837C9.7030103@stoneleaf.us> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011104723.05748928@anarchist> <20131011124357.0dbd8f20@anarchist> <525837C9.7030103@stoneleaf.us> Message-ID: <52584ADF.3090802@mrabarnett.plus.com> On 11/10/2013 18:39, Ethan Furman wrote: > On 10/11/2013 09:43 AM, Barry Warsaw wrote: >> On Oct 11, 2013, at 06:27 PM, Georg Brandl wrote: >> >>> Maybe to fit in with other verb-like APIs used as context managers: >>> it's open() not opened(). >> >> open() predates context managers, but maybe we need a new convention. >> >> with ignore(FileNotFoundError): >> >> vs >> >> with ignored(FileNotFoundError): >> >> To me anyway, the latter sounds better. > > The practice I follow is to use present tense, singular, unless there's > a really good reason not to. Makes remembering a lot easier: > > was it ignore? ignoring? ignores? ignored? > > Answer: ignore. > Singular would be "ignores". In this instance, "ignore" is imperative. From python at mrabarnett.plus.com Fri Oct 11 21:01:07 2013 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 11 Oct 2013 20:01:07 +0100 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <5258465C.1070800@g.nevcal.com> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011104723.05748928@anarchist> <20131011124357.0dbd8f20@anarchist> <52583316.9010802@trueblade.com> <5258465C.1070800@g.nevcal.com> Message-ID: <52584AF3.5000906@mrabarnett.plus.com> On 11/10/2013 19:41, Glenn Linderman wrote: > On 10/11/2013 10:19 AM, Eric V. Smith wrote: >> On 10/11/2013 12:43 PM, Barry Warsaw wrote: >>> On Oct 11, 2013, at 06:27 PM, Georg Brandl wrote: >>> >>>> Maybe to fit in with other verb-like APIs used as context managers: >>>> it's open() not opened(). >>> open() predates context managers, but maybe we need a new convention. >>> >>> with ignore(FileNotFoundError): >>> >>> vs >>> >>> with ignored(FileNotFoundError): >>> >>> To me anyway, the latter sounds better. >> I'm still -0, and maybe now -1 on the idea, mostly because it encourages >> an anti-pattern. >> >> But, to continue to paint the shed, shouldn't it be "ignoring", to match >> "closing"? > > Seriously, "with" is the wrong spelling for this using. It should be > > while ignorning(FileNotFoundError) > > Insistence on using "with" for the anti-pattern, and proper English, > would require: > > with ignorance_of(FileNotFoundError) > > :) > "Ignorance" means not knowing, but we _do_ know about FileNotFoundError. What we want is to ignore it. From tjreedy at udel.edu Fri Oct 11 21:13:12 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 11 Oct 2013 15:13:12 -0400 Subject: [Python-Dev] Python IDEL issue In-Reply-To: <495881D629037B40A8C279DB63D98C4D42A06C@SGSIMBX008.nsn-intra.net> References: <495881D629037B40A8C279DB63D98C4D42A06C@SGSIMBX008.nsn-intra.net> Message-ID: On 10/11/2013 3:37 AM, Hu, Hao (NSN - CN/Beijing) wrote: This list is for development *of* Python and CPython. Usage questions should be directed elsewhere, such as python-list. Idle questions can be directed to the idle-sig list. Both can be accessed through news.gmane.org as gmane.comp.python.general and .idle. > When I open IDLE(Python GUI) from the windows start menu, All questions should include the python version. > it always open > Python Shell. When I right click on a *.py file, I cannot see the menu > item ?open with IDLE?. I used to see that, but do not at present. Please continue the discussion on one of the two lists mentioned above. -- Terry Jan Reedy From rdmurray at bitdance.com Fri Oct 11 21:22:34 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 11 Oct 2013 15:22:34 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <52584AF3.5000906@mrabarnett.plus.com> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011104723.05748928@anarchist> <20131011124357.0dbd8f20@anarchist> <52583316.9010802@trueblade.com> <5258465C.1070800@g.nevcal.com> <52584AF3.5000906@mrabarnett.plus.com> Message-ID: <20131011192234.C32E12509A9@webabinitio.net> On Fri, 11 Oct 2013 20:01:07 +0100, MRAB wrote: > On 11/10/2013 19:41, Glenn Linderman wrote: > > On 10/11/2013 10:19 AM, Eric V. Smith wrote: > >> On 10/11/2013 12:43 PM, Barry Warsaw wrote: > >>> On Oct 11, 2013, at 06:27 PM, Georg Brandl wrote: > >>> > >>>> Maybe to fit in with other verb-like APIs used as context managers: > >>>> it's open() not opened(). > >>> open() predates context managers, but maybe we need a new convention. > >>> > >>> with ignore(FileNotFoundError): > >>> > >>> vs > >>> > >>> with ignored(FileNotFoundError): > >>> > >>> To me anyway, the latter sounds better. > >> I'm still -0, and maybe now -1 on the idea, mostly because it encourages > >> an anti-pattern. > >> > >> But, to continue to paint the shed, shouldn't it be "ignoring", to match > >> "closing"? > > > > Seriously, "with" is the wrong spelling for this using. It should be > > > > while ignorning(FileNotFoundError) > > > > Insistence on using "with" for the anti-pattern, and proper English, > > would require: > > > > with ignorance_of(FileNotFoundError) > > > > :) > > > "Ignorance" means not knowing, but we _do_ know about FileNotFoundError. > What we want is to ignore it. Isn't "ignore" in any form really a lie here, though? We are not ignoring it, we are catching it and taking an early exit out of the with block: with ignore(FileNotFoundError): with open('f') as f: print("hello world") print("goodbye world") This does not print anything, whereas for either intuitive meaning I can think of for "ignoring" the error, it ought to print something. You really can't *ignore* errors inside a with block. This incorrect intuition is exactly why this is an anti-pattern. 'ignore' is an attractive nuisance. This having occurred to me, I am now definitely -1. --David From tjreedy at udel.edu Fri Oct 11 21:23:10 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 11 Oct 2013 15:23:10 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011110042.37365137@pitrou.net> <5257D3FD.9070801@trueblade.com> Message-ID: On 10/11/2013 8:04 AM, Serhiy Storchaka wrote: > 11.10.13 13:33, Eric V. Smith ???????(??): >> And Antoine has again taught me a new word: >> polysemic: having more than one meaning; having multiple meanings > > There is no such word in my dictionaries. :( Only polysemous and > polysemantic. http://www.merriam-webster.com/dictionary/polysemic Says polysemic is a variant of polysemous. -- Terry Jan Reedy From ericsnowcurrently at gmail.com Fri Oct 11 21:37:07 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 11 Oct 2013 13:37:07 -0600 Subject: [Python-Dev] PEP 451: ModuleSpec In-Reply-To: References: Message-ID: On Fri, Oct 4, 2013 at 11:18 PM, Eric Snow wrote: > After a few rounds on import-sig PEP 451 is really for general > consumption. I also have a patch up now. > > HTML: http://www.python.org/dev/peps/pep-0451/ > implementation: http://bugs.python.org/issue18864 > > Your comments would be appreciated. > > -eric > > ============================================================= > > PEP: 451 > Title: A ModuleSpec Type for the Import System > Version: $Revision$ > Last-Modified: $Date$ > Author: Eric Snow > Discussions-To: import-sig at python.org > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 8-Aug-2013 > Python-Version: 3.4 > Post-History: 8-Aug-2013, 28-Aug-2013, 18-Sep-2013, 24-Sep-2013 > Resolution: > > > Abstract > ======== > > This PEP proposes to add a new class to importlib.machinery called > "ModuleSpec". It will provide all the import-related information used > to load a module and will be available without needing to load the > module first. Finders will directly provide a module's spec instead of > a loader (which they will continue to provide indirectly). The import > machinery will be adjusted to take advantage of module specs, including > using them to load modules. Any comments? Usually silence implies no disapproval. ;) PEP 451 did go through several rounds of review on import-sig, so I'm not going to stress over low feedback at this point. However, I'd particularly appreciate knowing if there are any objections to ModuleSpec for 3.4. Otherwise I'd like to get pronouncement on the PEP. Thanks! -eric From stefan at bytereef.org Fri Oct 11 21:41:00 2013 From: stefan at bytereef.org (Stefan Krah) Date: Fri, 11 Oct 2013 21:41:00 +0200 Subject: [Python-Dev] C extension import time In-Reply-To: <20131011172034.639648ff@pitrou.net> References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> Message-ID: <20131011194100.GA25838@sleipnir.bytereef.org> Antoine Pitrou wrote: > Just create a _pydecimal module (like _pyio). That's very fast indeed. There's one minor problem: For backwards compatibility and pickling [1] I'd need to add __module__ = 'decimal' to every class of the Python version. Are there any reasons not to do that? Stefan Krah [1] _decimal has always set the module name to 'decimal' because of that. From ericsnowcurrently at gmail.com Fri Oct 11 21:48:02 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 11 Oct 2013 13:48:02 -0600 Subject: [Python-Dev] C extension import time In-Reply-To: <20131011194100.GA25838@sleipnir.bytereef.org> References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> <20131011194100.GA25838@sleipnir.bytereef.org> Message-ID: On Fri, Oct 11, 2013 at 1:41 PM, Stefan Krah wrote: > Antoine Pitrou wrote: >> Just create a _pydecimal module (like _pyio). > > That's very fast indeed. There's one minor problem: For backwards compatibility > and pickling [1] I'd need to add > > __module__ = 'decimal' > > to every class of the Python version. Are there any reasons not to do that? Try just putting "__name__ = 'decimal'" at the top of the source file. -eric From victor.stinner at gmail.com Fri Oct 11 23:08:00 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 11 Oct 2013 23:08:00 +0200 Subject: [Python-Dev] C extension import time In-Reply-To: <20131011172034.639648ff@pitrou.net> References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> Message-ID: 2013/10/11 Antoine Pitrou : >> So the first step I tried is something horrible (typing from memory): >> >> try: >> import _decimal >> except ImportError: >> >> else: >> from _decimal import * >> >> That way the 2.21 msec are reduced to 912 usec, but the indentation is >> an abomination. > > Just create a _pydecimal module (like _pyio). +1 but both modules must be tested (as io and _pyio). Victor From ethan at stoneleaf.us Fri Oct 11 22:55:11 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 11 Oct 2013 13:55:11 -0700 Subject: [Python-Dev] PEPs shouldn't be considered docs In-Reply-To: <5257DFD6.7010701@nedbatchelder.com> References: <5257DFD6.7010701@nedbatchelder.com> Message-ID: <525865AF.6040308@stoneleaf.us> On 10/11/2013 04:24 AM, Ned Batchelder wrote: > > I'd like to suggest that we not consider PEPs to be documentation. +1 The few times I've tried to use the PEPs to understand current Python it was confusing, wrong, and a waste of time. -- ~Ethan~ From ethan at stoneleaf.us Fri Oct 11 22:56:04 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 11 Oct 2013 13:56:04 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <52584ADF.3090802@mrabarnett.plus.com> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011104723.05748928@anarchist> <20131011124357.0dbd8f20@anarchist> <525837C9.7030103@stoneleaf.us> <52584ADF.3090802@mrabarnett.plus.com> Message-ID: <525865E4.1060608@stoneleaf.us> On 10/11/2013 12:00 PM, MRAB wrote: > On 11/10/2013 18:39, Ethan Furman wrote: >> On 10/11/2013 09:43 AM, Barry Warsaw wrote: >>> On Oct 11, 2013, at 06:27 PM, Georg Brandl wrote: >>> >>>> Maybe to fit in with other verb-like APIs used as context managers: >>>> it's open() not opened(). >>> >>> open() predates context managers, but maybe we need a new convention. >>> >>> with ignore(FileNotFoundError): >>> >>> vs >>> >>> with ignored(FileNotFoundError): >>> >>> To me anyway, the latter sounds better. >> >> The practice I follow is to use present tense, singular, unless there's >> a really good reason not to. Makes remembering a lot easier: >> >> was it ignore? ignoring? ignores? ignored? >> >> Answer: ignore. >> > Singular would be "ignores". In this instance, "ignore" is imperative. Then let's go with imperative. ;) -- ~Ethan~ From greg at krypto.org Sat Oct 12 00:05:17 2013 From: greg at krypto.org (Gregory P. Smith) Date: Fri, 11 Oct 2013 15:05:17 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <5258465C.1070800@g.nevcal.com> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011104723.05748928@anarchist> <20131011124357.0dbd8f20@anarchist> <52583316.9010802@trueblade.com> <5258465C.1070800@g.nevcal.com> Message-ID: On Fri, Oct 11, 2013 at 11:41 AM, Glenn Linderman wrote: > On 10/11/2013 10:19 AM, Eric V. Smith wrote: > > On 10/11/2013 12:43 PM, Barry Warsaw wrote: > > On Oct 11, 2013, at 06:27 PM, Georg Brandl wrote: > > > Maybe to fit in with other verb-like APIs used as context managers: > it's open() not opened(). > > open() predates context managers, but maybe we need a new convention. > > with ignore(FileNotFoundError): > > vs > > with ignored(FileNotFoundError): > > To me anyway, the latter sounds better. > > I'm still -0, and maybe now -1 on the idea, mostly because it encourages > an anti-pattern. > > But, to continue to paint the shed, shouldn't it be "ignoring", to match > "closing"? > > > Seriously, "with" is the wrong spelling for this using. It should be > > while ignorning(FileNotFoundError) > > Insistence on using "with" for the anti-pattern, and proper English, would > require: > > with ignorance_of(FileNotFoundError) > I was thinking more along the lines of: with no_chance_in_hell_of_seeing(FileNotFoundError): but seriously, we have plenty of antipattern enablers in the language and standard library itself. contextlib.ignore vs ignored vs all of these others isn't a big deal to me. Just document it as not recommended for most things and let people shoot themselves if they've used "with contextlib.ignored(Documentation):" while writing code. I don't care what it is called and I think it is fine to have in contextlib. I'm unlikely to use it anytime soon as I don't have a 3.4+ only code base. Though suspect I could re-factor code to this in a few places within 3.4's Lib/subprocess.py. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From z at etiol.net Fri Oct 11 21:09:56 2013 From: z at etiol.net (Zero Piraeus) Date: Fri, 11 Oct 2013 16:09:56 -0300 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <52584AF3.5000906@mrabarnett.plus.com> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011104723.05748928@anarchist> <20131011124357.0dbd8f20@anarchist> <52583316.9010802@trueblade.com> <5258465C.1070800@g.nevcal.com> <52584AF3.5000906@mrabarnett.plus.com> Message-ID: <20131011190956.GB19376@piedra> : On Fri, Oct 11, 2013 at 08:01:07PM +0100, MRAB wrote: > On 11/10/2013 19:41, Glenn Linderman wrote: > >Insistence on using "with" for the anti-pattern, and proper English, > >would require: > > > >with ignorance_of(FileNotFoundError) > > > "Ignorance" means not knowing, but we _do_ know about FileNotFoundError. > What we want is to ignore it. Suggestions: with disregard_for(...): with indifference_to(...): with nonchalance_regarding(...): with blithe_disdain_concerning(...): -[]z. -- Zero Piraeus: dictatum erat http://etiol.net/pubkey.asc From rdmurray at bitdance.com Sat Oct 12 00:49:16 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 11 Oct 2013 18:49:16 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131011190956.GB19376@piedra> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011104723.05748928@anarchist> <20131011124357.0dbd8f20@anarchist> <52583316.9010802@trueblade.com> <5258465C.1070800@g.nevcal.com> <52584AF3.5000906@mrabarnett.plus.com> <20131011190956.GB19376@piedra> Message-ID: <20131011224917.0874C250161@webabinitio.net> On Fri, 11 Oct 2013 16:09:56 -0300, Zero Piraeus wrote: > : > > On Fri, Oct 11, 2013 at 08:01:07PM +0100, MRAB wrote: > > On 11/10/2013 19:41, Glenn Linderman wrote: > > >Insistence on using "with" for the anti-pattern, and proper English, > > >would require: > > > > > >with ignorance_of(FileNotFoundError) > > > > > "Ignorance" means not knowing, but we _do_ know about FileNotFoundError. > > What we want is to ignore it. > > Suggestions: > > with disregard_for(...): > > with indifference_to(...): > > with nonchalance_regarding(...): > > with blithe_disdain_concerning(...): with an_immediate_exit_from_the_with_if_we_see(...): --David From breamoreboy at yahoo.co.uk Sat Oct 12 00:45:09 2013 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Fri, 11 Oct 2013 23:45:09 +0100 Subject: [Python-Dev] Thanks everybody (was Summary of Python tracker Issues) In-Reply-To: <20131011160741.AFEC456A52@psf.upfronthosting.co.za> References: <20131011160741.AFEC456A52@psf.upfronthosting.co.za> Message-ID: On 11/10/2013 17:07, Python tracker wrote: > > ACTIVITY SUMMARY (2013-10-04 - 2013-10-11) > Python tracker at http://bugs.python.org/ > > Issues counts and deltas: > open 4240 (-16) > closed 26757 (+75) > total 30997 (+59) > Looking at the figures above I'd just like to say thank you to everybody involved for the work that has been put in. I for one certainly appreciate your efforts. -- Roses are red, Violets are blue, Most poems rhyme, But this one doesn't. Mark Lawrence From ncoghlan at gmail.com Sat Oct 12 01:06:24 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 12 Oct 2013 09:06:24 +1000 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131011175656.CB7AC2509A9@webabinitio.net> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011175656.CB7AC2509A9@webabinitio.net> Message-ID: On 12 Oct 2013 03:58, "R. David Murray" wrote: > > On Fri, 11 Oct 2013 20:32:45 +0300, Serhiy Storchaka wrote: > > 11.10.13 10:24, Antoine Pitrou ???????(??): > > > Ezio was -1 on the tracker, and Eric Smith was -0. I'd like to add my > > > -1 too. This is a useless addition (the traditional idiom is perfectly > > > obvious) and makes reading foreign code more tedious by adding > > > superfluous API calls. > > > > I am -1 too. But I want to hear more about interact with ExitStack (note > > that ExitStack itself is not widely used in the stdlib). > > I'm inclined to be -1 as well, because of the anti-pattern > encouragement. It seems at first blush like using it with ExitStack > would make that tendency worse, so I too would like to hear more about > Nick's thoughts on the subject. In conjunction with ExitStack itself, I see it as a building block for dynamic exception handling, decoupled from the syntactic layout of code. I also see reducing a four line idiom to a two line one that is easier to grep for as a benefit, as well as providing another place in the docs to nudge people towards improved exception handling practices (even if we didn't take advantage of that opportunity in the original patch). For example, a counterpart in logging that *logged* suppressed exceptions rather than ignoring them completely would be a good resource to point to, while being more discoverable than just offering the manager in logging (such a manager would *also* mix nicely with ExitStack). I'm not too fussy about the name (clearly). We originally picked ignored(), Raymond asked if he could change it to ignore() (and I said yes), and there's a valid argument to be made that closing() sets the precedent that it should be ignoring(). (RDM is also right that the exception still has the effect of terminating the block early, but I view names as mnemonics rather than necessarily 100% accurate descriptions of things). Cheers, Nick. > > --David > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Oct 12 01:11:21 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 12 Oct 2013 09:11:21 +1000 Subject: [Python-Dev] C extension import time In-Reply-To: References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> <20131011194100.GA25838@sleipnir.bytereef.org> Message-ID: On 12 Oct 2013 05:49, "Eric Snow" wrote: > > On Fri, Oct 11, 2013 at 1:41 PM, Stefan Krah wrote: > > Antoine Pitrou wrote: > >> Just create a _pydecimal module (like _pyio). > > > > That's very fast indeed. There's one minor problem: For backwards compatibility > > and pickling [1] I'd need to add > > > > __module__ = 'decimal' > > > > to every class of the Python version. Are there any reasons not to do that? > > Try just putting "__name__ = 'decimal'" at the top of the source file. In this case the fixup needs to be conditional on the absence of "_decimal". Aside from that, yes, lying about name is the easiest way to preserve pickle compatibility while still moving code around. Cheers, Nick. > > -eric > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Oct 12 01:13:43 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 12 Oct 2013 01:13:43 +0200 Subject: [Python-Dev] Support keyword in PEP URL? Message-ID: Hi, What do you think of adding an optional identifier to a PEP to get a readable URL? Example: http://www.python.org/dev/peps/qualname/ instead of http://www.python.org/dev/peps/pep-0395/ Other examples: 305: csv 450: statistics 3156: asyncio An identifier must only contain lower case letters (a-z), digits (0-9), dot (.) and underscore (_). It should be short (max. 20 characters)... and unique. Regex: ^[a-z0-9._]{1,20}$. For draft PEP, the identifier may change. If it's an issue (dead URL), we may allow multiple identifiers (ex: add also "tulip" for 3156), or only add an identifier to accepted PEPs. I don't know yet how it can be implemented at server side. Victor From guido at python.org Sat Oct 12 01:28:26 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 11 Oct 2013 16:28:26 -0700 Subject: [Python-Dev] Support keyword in PEP URL? In-Reply-To: References: Message-ID: What's the use case? I just use Google search if I don't recall the PEP number. On Fri, Oct 11, 2013 at 4:13 PM, Victor Stinner wrote: > Hi, > > What do you think of adding an optional identifier to a PEP to get a > readable URL? > > Example: > http://www.python.org/dev/peps/qualname/ > instead of > http://www.python.org/dev/peps/pep-0395/ > > Other examples: > > 305: csv > 450: statistics > 3156: asyncio > > An identifier must only contain lower case letters (a-z), digits > (0-9), dot (.) and underscore (_). It should be short (max. 20 > characters)... and unique. Regex: ^[a-z0-9._]{1,20}$. > > For draft PEP, the identifier may change. If it's an issue (dead URL), > we may allow multiple identifiers (ex: add also "tulip" for 3156), or > only add an identifier to accepted PEPs. > > I don't know yet how it can be implemented at server side. > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Oct 12 01:29:50 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 12 Oct 2013 09:29:50 +1000 Subject: [Python-Dev] PEPs shouldn't be considered docs In-Reply-To: <5257DFD6.7010701@nedbatchelder.com> References: <5257DFD6.7010701@nedbatchelder.com> Message-ID: On 11 Oct 2013 21:25, "Ned Batchelder" wrote: > > I wanted to teach a co-worker about "from __future__ import absolute_import" today, so I thought I'd point them at the docs. The page for "__future__" starts with a bunch of internal details that almost no one needs to know. There's a table at the end that mentions the actual importable names, including "absolute_import", but says nothing about then except to link to a PEP. > > The absolute imports PEP has no simple description of what it does. Like many PEPs, it's mostly a summary of the debate around the design of the feature. The closest the PEP comes to describing the behavior of "absolute_import" is this parenthetical: >> >> For the second problem, it is proposed that all import statements be absolute by default (searching sys.path only) with special syntax (leading dots) for accessing package-relative imports. > > And notice: that sentence describes it as a "proposal." > > I'd like to suggest that we not consider PEPs to be documentation. If a PEP has a good succinct description of how something works, then let's copy that text into the documentation at an appropriate place. If a PEP doesn't have such a description, then all the more reason not to send readers there. > > A link to the PEP is appropriate as a "see also" in the docs, but we shouldn't pretend that a PEP addresses the needs of people new to the feature. Aside from PEPs that describe development practices and versioned interoperability standards, this is already the case. The only major deviation from this practice was the import system, and that was because it was such an incoherent organically grown mess in CPython prior to the importlib migration in Python 3.3 that nobody was prepared to take on the task of writing the reference docs. Providing the missing reference docs for the import system was one of the conditions placed on the introduction of namespace packages in PEP 420. There's no grand policy change or clarification needed here, it's just another consequence of the fact that the import system isn't documented properly in versions prior to 3.3. Cheers, Nick. > --Ned. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Sat Oct 12 01:30:48 2013 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sat, 12 Oct 2013 00:30:48 +0100 Subject: [Python-Dev] Support keyword in PEP URL? In-Reply-To: References: Message-ID: On 12/10/2013 00:13, Victor Stinner wrote: > Hi, > > What do you think of adding an optional identifier to a PEP to get a > readable URL? > > Example: > http://www.python.org/dev/peps/qualname/ > instead of > http://www.python.org/dev/peps/pep-0395/ > > Other examples: > > 305: csv > 450: statistics > 3156: asyncio > > An identifier must only contain lower case letters (a-z), digits > (0-9), dot (.) and underscore (_). It should be short (max. 20 > characters)... and unique. Regex: ^[a-z0-9._]{1,20}$. > > For draft PEP, the identifier may change. If it's an issue (dead URL), > we may allow multiple identifiers (ex: add also "tulip" for 3156), or > only add an identifier to accepted PEPs. > > I don't know yet how it can be implemented at server side. > > Victor > If it's easy to do why not? If it's difficult why bother, I find it easy enough to find stuff by searching directly on the PEP index? -- Roses are red, Violets are blue, Most poems rhyme, But this one doesn't. Mark Lawrence From ncoghlan at gmail.com Sat Oct 12 01:32:50 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 12 Oct 2013 09:32:50 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Issue #19209: Remove import of copyreg from the os module to speed up In-Reply-To: <3cxQLj6bP0z7Lk3@mail.python.org> References: <3cxQLj6bP0z7Lk3@mail.python.org> Message-ID: On 12 Oct 2013 09:27, "christian.heimes" wrote: > > http://hg.python.org/cpython/rev/29c4a6a11e76 > changeset: 86216:29c4a6a11e76 > user: Christian Heimes > date: Sat Oct 12 01:27:08 2013 +0200 > summary: > Issue #19209: Remove import of copyreg from the os module to speed up > interpreter startup. stat_result and statvfs_result are now hard-coded to > reside in the os module. > The patch is based on Victor Stinner's patch. > > files: > Lib/os.py | 27 --------------------------- > Lib/test/test_os.py | 22 ++++++++++++++++++++++ > Lib/test/test_site.py | 5 ++++- > Misc/NEWS | 4 ++++ > Modules/posixmodule.c | 4 ++-- > 5 files changed, 32 insertions(+), 30 deletions(-) > > > diff --git a/Lib/os.py b/Lib/os.py > --- a/Lib/os.py > +++ b/Lib/os.py > @@ -945,33 +945,6 @@ > __all__.extend(["spawnlp", "spawnlpe"]) > > > -import copyreg as _copyreg > - > -def _make_stat_result(tup, dict): > - return stat_result(tup, dict) > - > -def _pickle_stat_result(sr): > - (type, args) = sr.__reduce__() > - return (_make_stat_result, args) > - > -try: > - _copyreg.pickle(stat_result, _pickle_stat_result, _make_stat_result) > -except NameError: # stat_result may not exist > - pass > - > -def _make_statvfs_result(tup, dict): > - return statvfs_result(tup, dict) > - > -def _pickle_statvfs_result(sr): > - (type, args) = sr.__reduce__() > - return (_make_statvfs_result, args) > - > -try: > - _copyreg.pickle(statvfs_result, _pickle_statvfs_result, > - _make_statvfs_result) > -except NameError: # statvfs_result may not exist > - pass > - > # Supply os.popen() > def popen(cmd, mode="r", buffering=-1): > if not isinstance(cmd, str): > diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py > --- a/Lib/test/test_os.py > +++ b/Lib/test/test_os.py > @@ -26,6 +26,7 @@ > import codecs > import decimal > import fractions > +import pickle > try: > import threading > except ImportError: > @@ -264,6 +265,13 @@ > warnings.simplefilter("ignore", DeprecationWarning) > self.check_stat_attributes(fname) > > + def test_stat_result_pickle(self): > + result = os.stat(self.fname) > + p = pickle.dumps(result) > + self.assertIn(b'\x03cos\nstat_result\n', p) > + unpickled = pickle.loads(p) > + self.assertEqual(result, unpickled) > + > def test_statvfs_attributes(self): > if not hasattr(os, "statvfs"): > return > @@ -310,6 +318,20 @@ > except TypeError: > pass > > + @unittest.skipUnless(hasattr(os, 'statvfs'), > + "need os.statvfs()") > + def test_statvfs_result_pickle(self): > + try: > + result = os.statvfs(self.fname) > + except OSError as e: > + # On AtheOS, glibc always returns ENOSYS > + if e.errno == errno.ENOSYS: > + return > + p = pickle.dumps(result) > + self.assertIn(b'\x03cos\nstatvfs_result\n', p) > + unpickled = pickle.loads(p) > + self.assertEqual(result, unpickled) > + > def test_utime_dir(self): > delta = 1000000 > st = os.stat(support.TESTFN) > diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py > --- a/Lib/test/test_site.py > +++ b/Lib/test/test_site.py > @@ -431,10 +431,13 @@ > modules = eval(stdout.decode('utf-8')) > self.assertIn('site', modules) > > + # http://bugs.python.org/issue19205 > re_mods = {'re', '_sre', 'sre_compile', 'sre_constants', 'sre_parse'} > self.assertFalse(modules.intersection(re_mods)) > - > + # http://bugs.python.org/issue9548 > self.assertNotIn('locale', modules) This looks like it snuck in from a separate patch. Cheers, Nick. > + # http://bugs.python.org/issue19209 > + self.assertNotIn('copyreg', modules) > > > if __name__ == "__main__": > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -36,6 +36,10 @@ > Library > ------- > > +- Issue #19209: Remove import of copyreg from the os module to speed up > + interpreter startup. stat_result and statvfs_result are now hard-coded to > + reside in the os module. > + > - Issue #19205: Don't import the 're' module in site and sysconfig module to > to speed up interpreter start. > > diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c > --- a/Modules/posixmodule.c > +++ b/Modules/posixmodule.c > @@ -11974,7 +11974,7 @@ > return NULL; > #endif > > - stat_result_desc.name = MODNAME ".stat_result"; > + stat_result_desc.name = "os.stat_result"; /* see issue #19209 */ > stat_result_desc.fields[7].name = PyStructSequence_UnnamedField; > stat_result_desc.fields[8].name = PyStructSequence_UnnamedField; > stat_result_desc.fields[9].name = PyStructSequence_UnnamedField; > @@ -11983,7 +11983,7 @@ > structseq_new = StatResultType.tp_new; > StatResultType.tp_new = statresult_new; > > - statvfs_result_desc.name = MODNAME ".statvfs_result"; > + statvfs_result_desc.name = "os.statvfs_result"; /* see issue #19209 */ > if (PyStructSequence_InitType2(&StatVFSResultType, > &statvfs_result_desc) < 0) > return NULL; > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Sat Oct 12 01:35:14 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 12 Oct 2013 10:35:14 +1100 Subject: [Python-Dev] Support keyword in PEP URL? References: Message-ID: <7wr4brmle5.fsf@benfinney.id.au> Victor Stinner writes: > For draft PEP, the identifier may change. For an idea implemented in several PEPs, the obvious identifier may be taken first, but the preferred PEP for that identifier may later change. For example, PEP 354 would have the obvious keyword ?enum? when it was being discussed. Then it was rejected. Later, PEP 435 was introduced, and accepted. It is now even *more* deserving of the keyword ?enum? than the earlier PEP. What do you propose in cases like this? Should the keyword always refer to the same PEP it did in the past, even when that PEP is no longer as relevant given later PEPs? Or should the keyword reach a different, newer PEP if that newer PEP becomes a ?more relevant? PEP for the keyword? -- \ ?There are only two ways to live your life. One is as though | `\ nothing is a miracle. The other is as if everything is.? | _o__) ?Albert Einstein | Ben Finney From ncoghlan at gmail.com Sat Oct 12 01:37:40 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 12 Oct 2013 09:37:40 +1000 Subject: [Python-Dev] Support keyword in PEP URL? In-Reply-To: References: Message-ID: On 12 Oct 2013 09:32, "Mark Lawrence" wrote: > > On 12/10/2013 00:13, Victor Stinner wrote: >> >> Hi, >> >> What do you think of adding an optional identifier to a PEP to get a >> readable URL? >> >> Example: >> http://www.python.org/dev/peps/qualname/ >> instead of >> http://www.python.org/dev/peps/pep-0395/ >> >> Other examples: >> >> 305: csv >> 450: statistics >> 3156: asyncio >> >> An identifier must only contain lower case letters (a-z), digits >> (0-9), dot (.) and underscore (_). It should be short (max. 20 >> characters)... and unique. Regex: ^[a-z0-9._]{1,20}$. >> >> For draft PEP, the identifier may change. If it's an issue (dead URL), >> we may allow multiple identifiers (ex: add also "tulip" for 3156), or >> only add an identifier to accepted PEPs. >> >> I don't know yet how it can be implemented at server side. >> >> Victor >> > > If it's easy to do why not? > If it's difficult why bother, I find it easy enough to find stuff by searching directly on the PEP index? Agreed, doesn't seem worth the hassle to me. Cheers, Nick. > > -- > Roses are red, > Violets are blue, > Most poems rhyme, > But this one doesn't. > > Mark Lawrence > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Oct 12 01:53:12 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 12 Oct 2013 09:53:12 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Issue #19209: fix structseq test In-Reply-To: <3cxQgG300bz7Lk2@mail.python.org> References: <3cxQgG300bz7Lk2@mail.python.org> Message-ID: On 12 Oct 2013 09:42, "christian.heimes" wrote: > > http://hg.python.org/cpython/rev/89e405e6a7a9 > changeset: 86218:89e405e6a7a9 > parent: 86216:29c4a6a11e76 > user: Christian Heimes > date: Sat Oct 12 01:38:52 2013 +0200 > summary: > Issue #19209: fix structseq test > > files: > Lib/test/test_structseq.py | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Lib/test/test_structseq.py b/Lib/test/test_structseq.py > --- a/Lib/test/test_structseq.py > +++ b/Lib/test/test_structseq.py > @@ -38,7 +38,7 @@ > # os.stat() gives a complicated struct sequence. > st = os.stat(__file__) > rep = repr(st) > - self.assertTrue(rep.startswith(os.name + ".stat_result")) > + self.assertTrue(rep.startswith("os.stat_result")) If stat results can be pickled, this suggests a possible issue with unpickling pickles generated with older versions of Python. Cheers, Nick. > self.assertIn("st_mode=", rep) > self.assertIn("st_ino=", rep) > self.assertIn("st_dev=", rep) > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Sat Oct 12 02:20:28 2013 From: barry at python.org (Barry Warsaw) Date: Fri, 11 Oct 2013 20:20:28 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011175656.CB7AC2509A9@webabinitio.net> Message-ID: <20131011202028.62175880@anarchist> On Oct 12, 2013, at 09:06 AM, Nick Coghlan wrote: >I'm not too fussy about the name (clearly). We originally picked ignored(), >Raymond asked if he could change it to ignore() (and I said yes), Just as a point of order, it would be good to capture such side-channel discussions in the relevant issue. Mysterious decisions like this are easy to miss in the deluge of checkin messages, and don't help promote a transparent development process. Cheers, -Barry From victor.stinner at gmail.com Sat Oct 12 02:53:01 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 12 Oct 2013 02:53:01 +0200 Subject: [Python-Dev] Support keyword in PEP URL? In-Reply-To: References: Message-ID: 2013/10/12 Guido van Rossum : > What's the use case? I just use Google search if I don't recall the PEP > number. The final goal would be to identify PEPs using a textual identifier instead of a number identifier. We now have 206 PEPs (341 if you count also deferred and rejected PEPs). It's not easy to remember all these PEP numbers. Google should not needed to find a PEP, it's identifier should be enough to write manually the URL. The problem is to remember the numerical identifier. It's easy to mix up PEP numbers. For example, Martin von Loewis wrote two major PEP related to Unicode: 383 and 393. These numbers are close, only one digit is different. It's worse when you discuss recent PEPs: PEP 445 or PEP 454? Oops, no it was the PEP 455. Victor From ethan at stoneleaf.us Sat Oct 12 03:01:08 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 11 Oct 2013 18:01:08 -0700 Subject: [Python-Dev] Support keyword in PEP URL? In-Reply-To: <7wr4brmle5.fsf@benfinney.id.au> References: <7wr4brmle5.fsf@benfinney.id.au> Message-ID: <52589F54.4080702@stoneleaf.us> On 10/11/2013 04:35 PM, Ben Finney wrote: > Victor Stinner writes: > >> For draft PEP, the identifier may change. > > For an idea implemented in several PEPs, the obvious identifier may be > taken first, but the preferred PEP for that identifier may later change. > > For example, PEP 354 would have the obvious keyword ?enum? when it was > being discussed. Then it was rejected. > > Later, PEP 435 was introduced, and accepted. It is now even *more* > deserving of the keyword ?enum? than the earlier PEP. > > What do you propose in cases like this? Should the keyword always refer > to the same PEP it did in the past, even when that PEP is no longer as > relevant given later PEPs? Or should the keyword reach a different, > newer PEP if that newer PEP becomes a ?more relevant? PEP for the > keyword? Heh, that was the same example I was going to use. :) Another question: if the later PEP gets the appropriate name, what name then gets applied to the older one(s)? For than matter, what names would you give to the myriad unicode peps? -- ~Ethan~ From guido at python.org Sat Oct 12 03:01:51 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 11 Oct 2013 18:01:51 -0700 Subject: [Python-Dev] Support keyword in PEP URL? In-Reply-To: References: Message-ID: Hm. I think at that scale giving every PEP a unique name and remembering those names is just as hard. And the issue with different versions or variants of the same idea is real. I think it's not worth the effort. On Fri, Oct 11, 2013 at 5:53 PM, Victor Stinner wrote: > 2013/10/12 Guido van Rossum : > > What's the use case? I just use Google search if I don't recall the PEP > > number. > > The final goal would be to identify PEPs using a textual identifier > instead of a number identifier. > > We now have 206 PEPs (341 if you count also deferred and rejected > PEPs). It's not easy to remember all these PEP numbers. Google should > not needed to find a PEP, it's identifier should be enough to write > manually the URL. The problem is to remember the numerical identifier. > > It's easy to mix up PEP numbers. For example, Martin von Loewis wrote > two major PEP related to Unicode: 383 and 393. These numbers are > close, only one digit is different. It's worse when you discuss recent > PEPs: PEP 445 or PEP 454? Oops, no it was the PEP 455. > > Victor > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Oct 12 03:04:23 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 12 Oct 2013 03:04:23 +0200 Subject: [Python-Dev] Support keyword in PEP URL? In-Reply-To: <7wr4brmle5.fsf@benfinney.id.au> References: <7wr4brmle5.fsf@benfinney.id.au> Message-ID: 2013/10/12 Ben Finney : > Victor Stinner writes: > >> For draft PEP, the identifier may change. > > For an idea implemented in several PEPs, the obvious identifier may be > taken first, but the preferred PEP for that identifier may later change. > > For example, PEP 354 would have the obvious keyword ?enum? when it was > being discussed. Then it was rejected. > > Later, PEP 435 was introduced, and accepted. It is now even *more* > deserving of the keyword ?enum? than the earlier PEP. > > What do you propose in cases like this? Should the keyword always refer > to the same PEP it did in the past, even when that PEP is no longer as > relevant given later PEPs? Or should the keyword reach a different, > newer PEP if that newer PEP becomes a ?more relevant? PEP for the > keyword? I would not be shocked if http://www.python.org/dev/peps/pep-enum/ link is updated from the PEP 354 to the PEP 435, especially if both PEPs are draft. Identifiers should be carefully chosen, "unicode" is maybe not the best identifier for a PEP for example (they are many PEPs related to Unicode). Hum, how are PEP numbers generated? PEP *354* and *435* for enum? These numbers are not very random. We should check our PRNG :-) Victor From victor.stinner at gmail.com Sat Oct 12 03:13:46 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 12 Oct 2013 03:13:46 +0200 Subject: [Python-Dev] Support keyword in PEP URL? In-Reply-To: <52589F54.4080702@stoneleaf.us> References: <7wr4brmle5.fsf@benfinney.id.au> <52589F54.4080702@stoneleaf.us> Message-ID: 2013/10/12 Ethan Furman : >> What do you propose in cases like this? Should the keyword always refer >> to the same PEP it did in the past, even when that PEP is no longer as >> relevant given later PEPs? Or should the keyword reach a different, >> newer PEP if that newer PEP becomes a ?more relevant? PEP for the >> keyword? > > Heh, that was the same example I was going to use. :) > > Another question: if the later PEP gets the appropriate name, what name > then gets applied to the older one(s)? As I wrote in my first message, if changing identifier for draft PEP is an issue, an option is to only add a textual identifier to accepted PEPs. > For than matter, what names would you give to the myriad unicode peps? Let me try to name PEPs related to Unicode: 100: unicode_integration 261: unicode_ucs4 277: windows_unicode_filenames (hum, I proposed a limit of 20 characters, this name is 25 characters long) 383: surrogateescape 393: compact_unicode 414: u_prefix_py33 These names are just propositions. They should be discussed :-) (they are probably much more PEPs related to Unicode :-)) Victor From donald at stufft.io Sat Oct 12 03:20:58 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 11 Oct 2013 21:20:58 -0400 Subject: [Python-Dev] Support keyword in PEP URL? In-Reply-To: References: <7wr4brmle5.fsf@benfinney.id.au> <52589F54.4080702@stoneleaf.us> Message-ID: On Oct 11, 2013, at 9:13 PM, Victor Stinner wrote: > 2013/10/12 Ethan Furman : >>> What do you propose in cases like this? Should the keyword always refer >>> to the same PEP it did in the past, even when that PEP is no longer as >>> relevant given later PEPs? Or should the keyword reach a different, >>> newer PEP if that newer PEP becomes a ?more relevant? PEP for the >>> keyword? >> >> Heh, that was the same example I was going to use. :) >> >> Another question: if the later PEP gets the appropriate name, what name >> then gets applied to the older one(s)? > > As I wrote in my first message, if changing identifier for draft PEP > is an issue, an option is to only add a textual identifier to accepted > PEPs. > >> For than matter, what names would you give to the myriad unicode peps? > > Let me try to name PEPs related to Unicode: > > 100: unicode_integration > 261: unicode_ucs4 > 277: windows_unicode_filenames (hum, I proposed a limit of 20 > characters, this name is 25 characters long) > 383: surrogateescape > 393: compact_unicode > 414: u_prefix_py33 > > These names are just propositions. They should be discussed :-) Perhaps we can have a PEP to decide the name of a PEP! > > (they are probably much more PEPs related to Unicode :-)) > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From victor.stinner at gmail.com Sat Oct 12 03:16:36 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 12 Oct 2013 03:16:36 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Issue #19209: fix structseq test In-Reply-To: References: <3cxQgG300bz7Lk2@mail.python.org> Message-ID: 2013/10/12 Nick Coghlan : >> summary: >> Issue #19209: fix structseq test >> >> diff --git a/Lib/test/test_structseq.py b/Lib/test/test_structseq.py >> --- a/Lib/test/test_structseq.py >> +++ b/Lib/test/test_structseq.py >> @@ -38,7 +38,7 @@ >> # os.stat() gives a complicated struct sequence. >> st = os.stat(__file__) >> rep = repr(st) >> - self.assertTrue(rep.startswith(os.name + ".stat_result")) >> + self.assertTrue(rep.startswith("os.stat_result")) > > If stat results can be pickled, this suggests a possible issue with > unpickling pickles generated with older versions of Python. This change should not break the backward compatibility. The os module was using a copyreg "hack" (is it the good name?) to replace "posix" with "os" when a os.stat or os.statvfs object was serialized with pickle. See the issue for information. Victor From ethan at stoneleaf.us Sat Oct 12 03:30:10 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 11 Oct 2013 18:30:10 -0700 Subject: [Python-Dev] Support keyword in PEP URL? In-Reply-To: References: <7wr4brmle5.fsf@benfinney.id.au> <52589F54.4080702@stoneleaf.us> Message-ID: <5258A622.1030406@stoneleaf.us> On 10/11/2013 06:13 PM, Victor Stinner wrote: > > Let me try to name PEPs related to Unicode: > > 100: unicode_integration > 261: unicode_ucs4 > 277: windows_unicode_filenames (hum, I proposed a limit of 20 > characters, this name is 25 characters long) > 383: surrogateescape > 393: compact_unicode > 414: u_prefix_py33 > > These names are just propositions. They should be discussed :-) Yeah, I will still be using google to get the one I want. ;) -- ~Ethan~ From rdmurray at bitdance.com Sat Oct 12 03:57:33 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 11 Oct 2013 21:57:33 -0400 Subject: [Python-Dev] tracker/irc and development transparency (was: cpython: Rename contextlib.ignored ...) In-Reply-To: <20131011202028.62175880@anarchist> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011175656.CB7AC2509A9@webabinitio.net> <20131011202028.62175880@anarchist> Message-ID: <20131012015734.6652125005F@webabinitio.net> On Fri, 11 Oct 2013 20:20:28 -0400, Barry Warsaw wrote: > On Oct 12, 2013, at 09:06 AM, Nick Coghlan wrote: > > >I'm not too fussy about the name (clearly). We originally picked ignored(), > >Raymond asked if he could change it to ignore() (and I said yes), > > Just as a point of order, it would be good to capture such side-channel > discussions in the relevant issue. Mysterious decisions like this are easy to > miss in the deluge of checkin messages, and don't help promote a transparent > development process. +10. Those of us who regularly hang out and discuss things on irc[*] try to be good about updating the issue with "xxx was discussed on IRC and...". This is critical, because even the sometimes lengthly checkin messages I write carry nowhere near as much information as can be found by visiting the issue number mentioned in the checkin header. (I note that the issue number was also missing from this particular commit message.) As the community referring to this stuff grows (and it has to, given that use of the language is still growing) and the history of the project becomes ever deeper, this kind of documentation becomes more and more important. --David [*] We will often be carrying on a sort of dual conversation: quick sentences on IRC interleaved with longer back-and-forth posts on the tracker, both part of the same ongoing conversation... From stephen at xemacs.org Sat Oct 12 04:47:57 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 12 Oct 2013 11:47:57 +0900 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011175656.CB7AC2509A9@webabinitio.net> Message-ID: <87vc13gq76.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > (RDM is also right that the exception still has the effect of > terminating the block early, but I view names as mnemonics rather > than necessarily 100% accurate descriptions of things). This is just way too ambiguous for my taste. I can't help reading with contextlib.ignore(ExceptionIDontFeelLikeHandlingProperly): stmt1 stmt2 stmt3 as try: stmt1 except ExceptionIDontFeelLikeHandlingProperly: pass try: stmt2 except ExceptionIDontFeelLikeHandlingProperly: pass try: stmt3 except ExceptionIDontFeelLikeHandlingProperly: pass rather than try: stmt1 stmt2 stmt3 except ExceptionIDontFeelLikeHandlingProperly: pass It just feels like the exception should be suppressed at some level "lower" than stmtN, so stmtN fails but the suite continues. How about with contextlib.break_on(ExceptionIDontFeelLikeHandlingProperly): stmt1 stmt2 stmt3 This is not 100% accurate Pythonically (there's no loop to break here), but it does describe what the context manager does more accurately, and it does effectively break out of the 'with' control structure. From stephen at xemacs.org Sat Oct 12 05:28:10 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 12 Oct 2013 12:28:10 +0900 Subject: [Python-Dev] Support keyword in PEP URL? In-Reply-To: References: <7wr4brmle5.fsf@benfinney.id.au> <52589F54.4080702@stoneleaf.us> Message-ID: <87r4brgoc5.fsf@uwakimon.sk.tsukuba.ac.jp> Victor Stinner writes: Quoting someone else: >> For that matter, what names would you give to the myriad unicode >> peps? For what value of "you"? ISTM that's important. > Let me try to name PEPs related to Unicode: Of the ones you suggest, the only one that rings bells for me is "surrogateescape". If it's not mnemonic *to me*, it's not useful *to me*. (I.e., if python-dev can reach a workable consensus on nicknames, it doesn't really matter what I think. But if -- as I suspect -- many people will have different opinions about good nicknames for some PEPs, there won't be such a consensus.) Of the several hundred (and rapidly growing) PEPs, I only ever care about a dozen at a time (and some of those are fixed for all time: PEP 0, PEP 1, PEP 8). I have my own private nicknames for those PEPs. (Oddly enough, PEP 0, PEP 1, and PEP 8 are "pep0", "pep1", and "pep8". ;-) I think that more generally useful would be a PyPI package that grabs the PEP 0 list, parses it into a database, and adds a "nickname" column, plus a UI that allows assigning a personal nick to a PEP, lookup with fuzzy matching on nicks, and optionally sending the URL to the preferred browser. Sharing nicks and keywords (or tags, see below) would be a nice feature, too. N.B. "Keyword" isn't the right term for what you want. I think "nickname" is better. E.g., "unicode" is a fine keyword, but it applies to many PEPs, even if restricted to only those PEPs that directly affect the Unicode implementation. An URL like http://www.python.org/peps/by-keyword/unicode should return a list of all of them IMHO. I think of "keywords" as tags that are formally assigned by the project, "tags" as an informal social networking device. From ncoghlan at gmail.com Sat Oct 12 06:04:48 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 12 Oct 2013 14:04:48 +1000 Subject: [Python-Dev] Support keyword in PEP URL? In-Reply-To: References: <7wr4brmle5.fsf@benfinney.id.au> <52589F54.4080702@stoneleaf.us> Message-ID: On 12 Oct 2013 11:15, "Victor Stinner" wrote: > > 2013/10/12 Ethan Furman : > >> What do you propose in cases like this? Should the keyword always refer > >> to the same PEP it did in the past, even when that PEP is no longer as > >> relevant given later PEPs? Or should the keyword reach a different, > >> newer PEP if that newer PEP becomes a ?more relevant? PEP for the > >> keyword? > > > > Heh, that was the same example I was going to use. :) > > > > Another question: if the later PEP gets the appropriate name, what name > > then gets applied to the older one(s)? > > As I wrote in my first message, if changing identifier for draft PEP > is an issue, an option is to only add a textual identifier to accepted > PEPs. RFCs are referenced by number. Standards are referenced by number. Nicknames are simply too ambiguous to be scalable. If you want a self explanatory reference, use the full PEP title, as that's what it is for (RFCs and standards also have titles for the same reason). Adopting a suitable set of keywords (like unicode, import, builtins, syntax, stdlib, cpython) could be interesting, though. Cheers, Nick. > > > For than matter, what names would you give to the myriad unicode peps? > > Let me try to name PEPs related to Unicode: > > 100: unicode_integration > 261: unicode_ucs4 > 277: windows_unicode_filenames (hum, I proposed a limit of 20 > characters, this name is 25 characters long) > 383: surrogateescape > 393: compact_unicode > 414: u_prefix_py33 > > These names are just propositions. They should be discussed :-) > > (they are probably much more PEPs related to Unicode :-)) > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Sat Oct 12 06:22:30 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 11 Oct 2013 22:22:30 -0600 Subject: [Python-Dev] Support keyword in PEP URL? In-Reply-To: References: <7wr4brmle5.fsf@benfinney.id.au> <52589F54.4080702@stoneleaf.us> Message-ID: On Fri, Oct 11, 2013 at 10:04 PM, Nick Coghlan wrote: > Adopting a suitable set of keywords (like unicode, import, builtins, syntax, > stdlib, cpython) could be interesting, though. A couple years ago I started a page on the wiki for a topical listing of the PEPS [1]. I even brought up having a topical listing in PEP 0 [2]. However, there was some but not enough interest and I didn't have enough time to do much about it. :( -eric [1] https://wiki.python.org/moin/Topically%20Organized%20PEP%20List [2] (private list) https://mail.python.org/mailman/private/core-mentorship/2011-April/000252.html From ethan at stoneleaf.us Sat Oct 12 06:05:20 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 11 Oct 2013 21:05:20 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <87vc13gq76.fsf@uwakimon.sk.tsukuba.ac.jp> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011175656.CB7AC2509A9@webabinitio.net> <87vc13gq76.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <5258CA80.8040506@stoneleaf.us> On 10/11/2013 07:47 PM, Stephen J. Turnbull wrote: > Nick Coghlan writes: > > > (RDM is also right that the exception still has the effect of > > terminating the block early, but I view names as mnemonics rather > > than necessarily 100% accurate descriptions of things). > > This is just way too ambiguous for my taste. I can't help reading [snippers] > How about > > with contextlib.break_on(ExceptionIDontFeelLikeHandlingProperly): > stmt1 > stmt2 > stmt3 > > This is not 100% accurate Pythonically (there's no loop to break > here), but it does describe what the context manager does more > accurately, and it does effectively break out of the 'with' control > structure. +1 (if we have to have it) While I agree that mnemonics are helpful in the right circumstances, these aren't them, not for this name. We aren't /just/ ignoring the exception, the exception is having a very real effect -- it's terminating the with block. -- ~Ethan~ From ben+python at benfinney.id.au Sat Oct 12 08:16:05 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 12 Oct 2013 17:16:05 +1100 Subject: [Python-Dev] Support keyword in PEP URL? References: <7wr4brmle5.fsf@benfinney.id.au> Message-ID: <7wmwmfm2u2.fsf@benfinney.id.au> Victor Stinner writes: > 2013/10/12 Ben Finney : > > Victor Stinner writes: > > > >> For draft PEP, the identifier may change. Note that this isn't a matter of the PEP being draft; once a PEP has a number, it deserves an official URL and (if I understand you correctly) a keyword. Then, *after* those official designations are associated with that PEP, another PEP can come along and be a better candidate for the same keyword. What use is a unique keyword-to-PEP mapping if it's not leading to the PEP most clearly associated with that keyword? > I would not be shocked if http://www.python.org/dev/peps/pep-enum/ > link is updated from the PEP 354 to the PEP 435, especially if both > PEPs are draft. On the other hand, what use is a keyword-to-PEP mapping if it isn't reliably going to persist over time? > Identifiers should be carefully chosen, "unicode" is maybe not the > best identifier for a PEP for example (they are many PEPs related to > Unicode). Right. I don't think it makes sense to have a keyword limited to exactly one PEP. > Hum, how are PEP numbers generated? PEP *354* and *435* for enum? > These numbers are not very random. We should check our PRNG :-) They're not random; they're sequential for the most part, IIUC. -- \ ?The restriction of knowledge to an elite group destroys the | `\ spirit of society and leads to its intellectual | _o__) impoverishment.? ?Albert Einstein | Ben Finney From v+python at g.nevcal.com Sat Oct 12 09:14:01 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 12 Oct 2013 00:14:01 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131011190956.GB19376@piedra> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011104723.05748928@anarchist> <20131011124357.0dbd8f20@anarchist> <52583316.9010802@trueblade.com> <5258465C.1070800@g.nevcal.com> <52584AF3.5000906@mrabarnett.plus.com> <20131011190956.GB19376@piedra> Message-ID: <5258F6B9.4040503@g.nevcal.com> On 10/11/2013 12:09 PM, Zero Piraeus wrote: > with blithe_disdain_concerning(...): +1 :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Oct 12 11:27:03 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 12 Oct 2013 11:27:03 +0200 Subject: [Python-Dev] Support keyword in PEP URL? References: <7wr4brmle5.fsf@benfinney.id.au> <52589F54.4080702@stoneleaf.us> Message-ID: <20131012112703.1eff3aba@fsol> On Sat, 12 Oct 2013 14:04:48 +1000 Nick Coghlan wrote: > On 12 Oct 2013 11:15, "Victor Stinner" wrote: > > > > 2013/10/12 Ethan Furman : > > >> What do you propose in cases like this? Should the keyword always refer > > >> to the same PEP it did in the past, even when that PEP is no longer as > > >> relevant given later PEPs? Or should the keyword reach a different, > > >> newer PEP if that newer PEP becomes a ?more relevant? PEP for the > > >> keyword? > > > > > > Heh, that was the same example I was going to use. :) > > > > > > Another question: if the later PEP gets the appropriate name, what name > > > then gets applied to the older one(s)? > > > > As I wrote in my first message, if changing identifier for draft PEP > > is an issue, an option is to only add a textual identifier to accepted > > PEPs. > > RFCs are referenced by number. Standards are referenced by number. > Nicknames are simply too ambiguous to be scalable. Indeed. One reason is that several RFCs or PEPs may be issued for a single topic. Which RFC gets to be called "http"? Probably the first one, but then when you type "http" you get the HTTP 1.0 RFC. You'd rather have the HTTP 1.1 RFC, I think. Or perhaps a further clarification RFC. Regards Antoine. From p.f.moore at gmail.com Sat Oct 12 11:38:16 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 12 Oct 2013 10:38:16 +0100 Subject: [Python-Dev] PEPs shouldn't be considered docs In-Reply-To: References: <5257DFD6.7010701@nedbatchelder.com> Message-ID: On 12 October 2013 00:29, Nick Coghlan wrote: > There's no grand policy change or clarification needed here, it's just > another consequence of the fact that the import system isn't documented > properly in versions prior to 3.3. And my personal apology for that. I knew when we wrote PEP 302 that it needed proper documentation, but I couldn't find a good way to include it in the docs and I couldn't face rewriting the import docs in a way that would allow me to include it. Many thanks to Brett for ultimately doing it for me. Paul From p.f.moore at gmail.com Sat Oct 12 11:48:41 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 12 Oct 2013 10:48:41 +0100 Subject: [Python-Dev] Support keyword in PEP URL? In-Reply-To: References: Message-ID: On 12 October 2013 01:53, Victor Stinner wrote: > t's easy to mix up PEP numbers. For example, Martin von Loewis wrote > two major PEP related to Unicode: 383 and 393. These numbers are > close, only one digit is different. It's worse when you discuss recent > PEPs: PEP 445 or PEP 454? Oops, no it was the PEP 455. I think that the real issue is that people expect others to remember numbers as easily as they do while writing emails, etc. If in email people refer to PEPs as PEP 302 (Import Hooks) or PEP 3156 (Tulip) or PEP 435 (Enum) etc, there would be a lot less issue. We're all guilty of it, and context is often enough to clarify, but IMO this is more about thinking of your audience when writing. Paul From steve at pearwood.info Sat Oct 12 12:09:34 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 12 Oct 2013 21:09:34 +1100 Subject: [Python-Dev] Support keyword in PEP URL? In-Reply-To: References: Message-ID: <20131012100934.GN7989@ando> On Sat, Oct 12, 2013 at 10:48:41AM +0100, Paul Moore wrote: > On 12 October 2013 01:53, Victor Stinner wrote: > > t's easy to mix up PEP numbers. For example, Martin von Loewis wrote > > two major PEP related to Unicode: 383 and 393. These numbers are > > close, only one digit is different. It's worse when you discuss recent > > PEPs: PEP 445 or PEP 454? Oops, no it was the PEP 455. > > I think that the real issue is that people expect others to remember > numbers as easily as they do while writing emails, etc. If in email > people refer to PEPs as PEP 302 (Import Hooks) or PEP 3156 (Tulip) or > PEP 435 (Enum) etc, there would be a lot less issue. +1 -- Steven From christian at python.org Sat Oct 12 13:55:27 2013 From: christian at python.org (Christian Heimes) Date: Sat, 12 Oct 2013 13:55:27 +0200 Subject: [Python-Dev] cpython: Issue #19209: Remove import of copyreg from the os module to speed up In-Reply-To: References: <3cxQLj6bP0z7Lk3@mail.python.org> Message-ID: Am 12.10.2013 01:32, schrieb Nick Coghlan: >> diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py >> --- a/Lib/test/test_site.py >> +++ b/Lib/test/test_site.py >> @@ -431,10 +431,13 @@ >> modules = eval(stdout.decode('utf-8')) >> self.assertIn('site', modules) >> >> + # http://bugs.python.org/issue19205 >> re_mods = {'re', '_sre', 'sre_compile', 'sre_constants', > 'sre_parse'} >> self.assertFalse(modules.intersection(re_mods)) >> - >> + # http://bugs.python.org/issue9548 >> self.assertNotIn('locale', modules) > > This looks like it snuck in from a separate patch. No harm done. It seems like a good idea to document the reason for the test cases so I added links the links at the same time. From victor.stinner at gmail.com Sat Oct 12 14:40:51 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 12 Oct 2013 14:40:51 +0200 Subject: [Python-Dev] bugs.python.org not reachable in IPv6? Message-ID: Hi, The DNS server of python.org announce the IP address 2a01:4f8:131:2480::3: $ host -t AAAA bugs.python.org bugs.python.org has IPv6 address 2a01:4f8:131:2480::3 The problem is that I cannot connect to this IP address: $ ping6 -c 4 2a01:4f8:131:2480::3 PING 2a01:4f8:131:2480::3(2a01:4f8:131:2480::3) 56 data bytes --- 2a01:4f8:131:2480::3 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 2999ms Do you have a the same issue, or is it just me? Victor From antoine at python.org Sat Oct 12 14:51:34 2013 From: antoine at python.org (Antoine Pitrou) Date: Sat, 12 Oct 2013 14:51:34 +0200 Subject: [Python-Dev] [Infrastructure] bugs.python.org not reachable in IPv6? In-Reply-To: References: Message-ID: <1381582294.2497.0.camel@fsol> Opened issue at http://psf.upfronthosting.co.za/roundup/meta/issue528 Regards Antoine. Le samedi 12 octobre 2013 ? 14:40 +0200, Victor Stinner a ?crit : > Hi, > > The DNS server of python.org announce the IP address 2a01:4f8:131:2480::3: > > $ host -t AAAA bugs.python.org > bugs.python.org has IPv6 address 2a01:4f8:131:2480::3 > > > The problem is that I cannot connect to this IP address: > > $ ping6 -c 4 2a01:4f8:131:2480::3 > PING 2a01:4f8:131:2480::3(2a01:4f8:131:2480::3) 56 data bytes > > --- 2a01:4f8:131:2480::3 ping statistics --- > 4 packets transmitted, 0 received, 100% packet loss, time 2999ms > > > Do you have a the same issue, or is it just me? > > Victor > ________________________________________________ > Infrastructure mailing list > Infrastructure at python.org > https://mail.python.org/mailman/listinfo/infrastructure > Unsubscribe: https://mail.python.org/mailman/options/infrastructure/solipsis%40pitrou.net > From stefan at bytereef.org Sat Oct 12 15:41:46 2013 From: stefan at bytereef.org (Stefan Krah) Date: Sat, 12 Oct 2013 15:41:46 +0200 Subject: [Python-Dev] Change PEP 399 import recommendation (was: C extension import time) In-Reply-To: References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> <20131011194100.GA25838@sleipnir.bytereef.org> Message-ID: <20131012134146.GA19814@sleipnir.bytereef.org> Nick Coghlan wrote: > On 12 Oct 2013 05:49, "Eric Snow" wrote: > > > > On Fri, Oct 11, 2013 at 1:41 PM, Stefan Krah wrote: > > > Antoine Pitrou wrote: > > >> Just create a _pydecimal module (like _pyio). > > > > > > That's very fast indeed. There's one minor problem: For backwards > compatibility > > > and pickling [1] I'd need to add > > > > > > ? ? __module__ = 'decimal' > > > > > > to every class of the Python version. Are there any reasons not to do that? > > > > Try just putting "__name__ = 'decimal'" at the top of the source file. > > In this case the fixup needs to be conditional on the absence of "_decimal". > Aside from that, yes, lying about name is the easiest way to preserve pickle > compatibility while still moving code around. Thanks Eric and Nick. The setup pretty much works (see issue #19232) and the import speedup is quite large. I wonder if Cpython's startup time could be reduced if this strategy was applied to other modules as well (see #19229). There are some concerns whether the change would impact other Python implementations, so I changed the subject (hoping for feedback). Stefan Krah From stefan_ml at behnel.de Sat Oct 12 16:04:23 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 12 Oct 2013 16:04:23 +0200 Subject: [Python-Dev] Change PEP 399 import recommendation In-Reply-To: <20131012134146.GA19814@sleipnir.bytereef.org> References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> <20131011194100.GA25838@sleipnir.bytereef.org> <20131012134146.GA19814@sleipnir.bytereef.org> Message-ID: Stefan Krah, 12.10.2013 15:41: > Nick Coghlan wrote: >> On 12 Oct 2013 05:49, "Eric Snow" wrote: >>> On Fri, Oct 11, 2013 at 1:41 PM, Stefan Krah wrote: >>>> Antoine Pitrou wrote: >>>>> Just create a _pydecimal module (like _pyio). >>>> >>>> That's very fast indeed. There's one minor problem: For backwards >> compatibility >>>> and pickling [1] I'd need to add >>>> >>>> __module__ = 'decimal' >>>> >>>> to every class of the Python version. Are there any reasons not to do that? >>> >>> Try just putting "__name__ = 'decimal'" at the top of the source file. >> >> In this case the fixup needs to be conditional on the absence of "_decimal". >> Aside from that, yes, lying about name is the easiest way to preserve pickle >> compatibility while still moving code around. > > Thanks Eric and Nick. The setup pretty much works (see issue #19232) and the > import speedup is quite large. I wonder if Cpython's startup time could be > reduced if this strategy was applied to other modules as well (see #19229). > > There are some concerns whether the change would impact other Python > implementations, so I changed the subject (hoping for feedback). FWIW, I think this definitely makes sense in cases where the C implementation is essentially a complete replacement of the original module, such as in this case. I even sometimes suggest compiling Python modules with Cython if the import time matters. For "normal" accelerator modules that only replace a part of a Python module with a native implementation, this is less likely to make a large enough difference to make up for the additional complexity due to the code split. Stefan From rdmurray at bitdance.com Sat Oct 12 16:46:02 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 12 Oct 2013 10:46:02 -0400 Subject: [Python-Dev] Change PEP 399 import recommendation In-Reply-To: References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> <20131011194100.GA25838@sleipnir.bytereef.org> <20131012134146.GA19814@sleipnir.bytereef.org> Message-ID: <20131012144602.CDBDE25005F@webabinitio.net> On Sat, 12 Oct 2013 16:04:23 +0200, Stefan Behnel wrote: > Stefan Krah, 12.10.2013 15:41: > > Nick Coghlan wrote: > >> On 12 Oct 2013 05:49, "Eric Snow" wrote: > >>> On Fri, Oct 11, 2013 at 1:41 PM, Stefan Krah wrote: > >>>> Antoine Pitrou wrote: > >>>>> Just create a _pydecimal module (like _pyio). > >>>> > >>>> That's very fast indeed. There's one minor problem: For backwards > >> compatibility > >>>> and pickling [1] I'd need to add > >>>> > >>>> __module__ = 'decimal' > >>>> > >>>> to every class of the Python version. Are there any reasons not to do that? > >>> > >>> Try just putting "__name__ = 'decimal'" at the top of the source file. > >> > >> In this case the fixup needs to be conditional on the absence of "_decimal". > >> Aside from that, yes, lying about name is the easiest way to preserve pickle > >> compatibility while still moving code around. > > > > Thanks Eric and Nick. The setup pretty much works (see issue #19232) and the > > import speedup is quite large. I wonder if Cpython's startup time could be > > reduced if this strategy was applied to other modules as well (see #19229). > > > > There are some concerns whether the change would impact other Python > > implementations, so I changed the subject (hoping for feedback). > > FWIW, I think this definitely makes sense in cases where the C > implementation is essentially a complete replacement of the original > module, such as in this case. I even sometimes suggest compiling Python > modules with Cython if the import time matters. > > For "normal" accelerator modules that only replace a part of a Python > module with a native implementation, this is less likely to make a large > enough difference to make up for the additional complexity due to the code > split. The impact on other implementations is: what if they write an accelerator that only replaces part of the module, whereas CPython's replaces the whole thing? But I think we could just postpone dealing with that until it actually comes up, just as we would if some other implementation writes an accelerator for a module for which CPython doesn't have one. --David From brett at python.org Sat Oct 12 17:19:42 2013 From: brett at python.org (Brett Cannon) Date: Sat, 12 Oct 2013 11:19:42 -0400 Subject: [Python-Dev] PEPs shouldn't be considered docs In-Reply-To: References: <5257DFD6.7010701@nedbatchelder.com> Message-ID: On Sat, Oct 12, 2013 at 5:38 AM, Paul Moore wrote: > On 12 October 2013 00:29, Nick Coghlan wrote: > > There's no grand policy change or clarification needed here, it's just > > another consequence of the fact that the import system isn't documented > > properly in versions prior to 3.3. > > And my personal apology for that. I knew when we wrote PEP 302 that it > needed proper documentation, but I couldn't find a good way to include > it in the docs and I couldn't face rewriting the import docs in a way > that would allow me to include it. > > Many thanks to Brett for ultimately doing it for me. > Actually thanks should go to Barry who rewrote the language ref docs for import. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Oct 12 17:24:18 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 13 Oct 2013 01:24:18 +1000 Subject: [Python-Dev] PEPs shouldn't be considered docs In-Reply-To: References: <5257DFD6.7010701@nedbatchelder.com> Message-ID: On 12 Oct 2013 19:38, "Paul Moore" wrote: > > On 12 October 2013 00:29, Nick Coghlan wrote: > > There's no grand policy change or clarification needed here, it's just > > another consequence of the fact that the import system isn't documented > > properly in versions prior to 3.3. > > And my personal apology for that. I knew when we wrote PEP 302 that it > needed proper documentation, but I couldn't find a good way to include > it in the docs and I couldn't face rewriting the import docs in a way > that would allow me to include it. You were *far* from the only one to contemplate documenting the old import system and then decide we had more edifying things to do with our time ;) > Many thanks to Brett for ultimately doing it for me. Barry actually did the bulk of the initial writing for that, although Brett did the preceding work to make it feasible to document by ensuring the default import system was registered on sys.meta_path. And yes, it was a very happy day when that commit landed :) Cheers, Nick. > Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Sat Oct 12 17:35:37 2013 From: barry at python.org (Barry Warsaw) Date: Sat, 12 Oct 2013 11:35:37 -0400 Subject: [Python-Dev] PEPs shouldn't be considered docs In-Reply-To: References: <5257DFD6.7010701@nedbatchelder.com> Message-ID: <20131012113537.505aaf2b@anarchist> On Oct 12, 2013, at 11:19 AM, Brett Cannon wrote: >Actually thanks should go to Barry who rewrote the language ref docs for >import. I can actually say it was fun due to all the great work on importlib. :) -Barry From ncoghlan at gmail.com Sat Oct 12 17:37:41 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 13 Oct 2013 01:37:41 +1000 Subject: [Python-Dev] Change PEP 399 import recommendation In-Reply-To: <20131012144602.CDBDE25005F@webabinitio.net> References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> <20131011194100.GA25838@sleipnir.bytereef.org> <20131012134146.GA19814@sleipnir.bytereef.org> <20131012144602.CDBDE25005F@webabinitio.net> Message-ID: On 13 Oct 2013 00:47, "R. David Murray" wrote: > > On Sat, 12 Oct 2013 16:04:23 +0200, Stefan Behnel wrote: > > Stefan Krah, 12.10.2013 15:41: > > > Nick Coghlan wrote: > > >> On 12 Oct 2013 05:49, "Eric Snow" wrote: > > >>> On Fri, Oct 11, 2013 at 1:41 PM, Stefan Krah wrote: > > >>>> Antoine Pitrou wrote: > > >>>>> Just create a _pydecimal module (like _pyio). > > >>>> > > >>>> That's very fast indeed. There's one minor problem: For backwards > > >> compatibility > > >>>> and pickling [1] I'd need to add > > >>>> > > >>>> __module__ = 'decimal' > > >>>> > > >>>> to every class of the Python version. Are there any reasons not to do that? > > >>> > > >>> Try just putting "__name__ = 'decimal'" at the top of the source file. > > >> > > >> In this case the fixup needs to be conditional on the absence of "_decimal". > > >> Aside from that, yes, lying about name is the easiest way to preserve pickle > > >> compatibility while still moving code around. > > > > > > Thanks Eric and Nick. The setup pretty much works (see issue #19232) and the > > > import speedup is quite large. I wonder if Cpython's startup time could be > > > reduced if this strategy was applied to other modules as well (see #19229). > > > > > > There are some concerns whether the change would impact other Python > > > implementations, so I changed the subject (hoping for feedback). > > > > FWIW, I think this definitely makes sense in cases where the C > > implementation is essentially a complete replacement of the original > > module, such as in this case. I even sometimes suggest compiling Python > > modules with Cython if the import time matters. > > > > For "normal" accelerator modules that only replace a part of a Python > > module with a native implementation, this is less likely to make a large > > enough difference to make up for the additional complexity due to the code > > split. > > The impact on other implementations is: what if they write an > accelerator that only replaces part of the module, whereas CPython's > replaces the whole thing? > > But I think we could just postpone dealing with that until it actually > comes up, just as we would if some other implementation writes an > accelerator for a module for which CPython doesn't have one. I think the default recommendation in PEP 399 still makes sense - 2 modules are easy to manage than three and the idiom allows for easy partial replacement. If the module is complex enough (and keep in mind that Brett used decimal.py as a data point in import benchmarking due to its sheer size meaning execution time was actually significant, even relative to IO time), or implicitly imported at startup, then the more complex 3 module setup may be worthwhile. If other implementations only provide partial acceleration, they can import the pure Python implementation as part of their accelerator implementation. Cheers, Nick. > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Sat Oct 12 18:55:42 2013 From: christian at python.org (Christian Heimes) Date: Sat, 12 Oct 2013 18:55:42 +0200 Subject: [Python-Dev] Change PEP 399 import recommendation In-Reply-To: References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> <20131011194100.GA25838@sleipnir.bytereef.org> <20131012134146.GA19814@sleipnir.bytereef.org> <20131012144602.CDBDE25005F@webabinitio.net> Message-ID: <52597F0E.5000007@python.org> Am 12.10.2013 17:37, schrieb Nick Coghlan: > I think the default recommendation in PEP 399 still makes sense - 2 > modules are easy to manage than three and the idiom allows for easy > partial replacement. We could ues yet another approach and put the pure Python implementation of modules into another directory (e.g. Lib/pystdlib or Lib/purepython). This directory is appended to sys.path. Alternative implementations of Python or platforms without the necessary bits and pieces would import the Python-only version from that directory. C modules can be renamed so CPython picks them up in favor of the pure Python modules. No hacks, no facade module, no slow down and it's easy to understand, too. The approach can be used for stat, operator, decimal and perhaps other modules, too. Christian From solipsis at pitrou.net Sat Oct 12 19:02:43 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 12 Oct 2013 19:02:43 +0200 Subject: [Python-Dev] Change PEP 399 import recommendation References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> <20131011194100.GA25838@sleipnir.bytereef.org> <20131012134146.GA19814@sleipnir.bytereef.org> <20131012144602.CDBDE25005F@webabinitio.net> <52597F0E.5000007@python.org> Message-ID: <20131012190243.404e477d@fsol> On Sat, 12 Oct 2013 18:55:42 +0200 Christian Heimes wrote: > Am 12.10.2013 17:37, schrieb Nick Coghlan: > > I think the default recommendation in PEP 399 still makes sense - 2 > > modules are easy to manage than three and the idiom allows for easy > > partial replacement. > > We could ues yet another approach and put the pure Python implementation > of modules into another directory (e.g. Lib/pystdlib or Lib/purepython). > This directory is appended to sys.path. Alternative implementations of > Python or platforms without the necessary bits and pieces would import > the Python-only version from that directory. C modules can be renamed so > CPython picks them up in favor of the pure Python modules. No hacks, no > facade module, no slow down and it's easy to understand, too. If you grow sys.path, imports get slower. (IMHO it also makes the source tree more cumbersome to navigate) Regards Antoine. From python at mrabarnett.plus.com Sat Oct 12 19:18:49 2013 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 12 Oct 2013 18:18:49 +0100 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <5258CA80.8040506@stoneleaf.us> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011175656.CB7AC2509A9@webabinitio.net> <87vc13gq76.fsf@uwakimon.sk.tsukuba.ac.jp> <5258CA80.8040506@stoneleaf.us> Message-ID: <52598479.80504@mrabarnett.plus.com> On 12/10/2013 05:05, Ethan Furman wrote: > On 10/11/2013 07:47 PM, Stephen J. Turnbull wrote: >> Nick Coghlan writes: >> >> > (RDM is also right that the exception still has the effect of >> > terminating the block early, but I view names as mnemonics rather >> > than necessarily 100% accurate descriptions of things). >> >> This is just way too ambiguous for my taste. I can't help reading > > [snippers] > >> How about >> >> with contextlib.break_on(ExceptionIDontFeelLikeHandlingProperly): >> stmt1 >> stmt2 >> stmt3 >> >> This is not 100% accurate Pythonically (there's no loop to break >> here), but it does describe what the context manager does more >> accurately, and it does effectively break out of the 'with' control >> structure. > > +1 (if we have to have it) > > While I agree that mnemonics are helpful in the right circumstances, these aren't them, not for this name. We aren't > /just/ ignoring the exception, the exception is having a very real effect -- it's terminating the with block. > Perhaps "leave_on" would be more accurate. From christian at python.org Sat Oct 12 19:19:44 2013 From: christian at python.org (Christian Heimes) Date: Sat, 12 Oct 2013 19:19:44 +0200 Subject: [Python-Dev] Right place for PBKDF2 wrapper Message-ID: Hi, I have written a interface to OpenSSL's PKCS5_PBKDF2_HMAC() function. It implements PKCS#5's password based key derivation function 2 with HMAC as pseudo-random function. It supports any digest that is supported by OpenSSL, e.g. SHA-1, SHA-256 and SHA-512. It's a low level inteface that takes the digest as unicode name, password and salt as bytes/buffer, keylen and rounds as int. I'd like to add the feature to Python 3.4. Now I'm looking for a good place to put it and some high level functions. In the future I like to add scrypt and bcrypt key stretching and key derivation functions, too. What's a good place for them?? * add a new ``kdf`` module (key derivation function) * add PBKDF2 to ``hashlib`` * make ``hashlib`` a package and add PBKDF2 to a new ``hashlib.kdf`` module * make ``hashlib`` a package and add PBKDF2 to a new ``hashlib.pbkdf2`` module * make ``crypt`` work under Windows and add PKBDF2 to it The patch is available at http://bugs.python.org/issue18582 Regards, Christian From solipsis at pitrou.net Sat Oct 12 19:37:54 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 12 Oct 2013 19:37:54 +0200 Subject: [Python-Dev] Right place for PBKDF2 wrapper References: Message-ID: <20131012193754.30af1e9f@fsol> On Sat, 12 Oct 2013 19:19:44 +0200 Christian Heimes wrote: > Hi, > > I have written a interface to OpenSSL's PKCS5_PBKDF2_HMAC() function. It > implements PKCS#5's password based key derivation function 2 with HMAC > as pseudo-random function. It supports any digest that is supported by > OpenSSL, e.g. SHA-1, SHA-256 and SHA-512. It's a low level inteface that > takes the digest as unicode name, password and salt as bytes/buffer, > keylen and rounds as int. > > I'd like to add the feature to Python 3.4. Now I'm looking for a good > place to put it and some high level functions. In the future I like to > add scrypt and bcrypt key stretching and key derivation functions, too. > What's a good place for them?? > > * add a new ``kdf`` module (key derivation function) > * add PBKDF2 to ``hashlib`` > * make ``hashlib`` a package and add PBKDF2 to a new ``hashlib.kdf`` module > * make ``hashlib`` a package and add PBKDF2 to a new ``hashlib.pbkdf2`` > module > * make ``crypt`` work under Windows and add PKBDF2 to it Putting it in "hashlib" sounds fine. There's no reason to create a myriad of small separate modules. Regards Antoine. From stefan at bytereef.org Sat Oct 12 19:39:49 2013 From: stefan at bytereef.org (Stefan Krah) Date: Sat, 12 Oct 2013 19:39:49 +0200 Subject: [Python-Dev] Change PEP 399 import recommendation In-Reply-To: References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> <20131011194100.GA25838@sleipnir.bytereef.org> <20131012134146.GA19814@sleipnir.bytereef.org> Message-ID: <20131012173949.GA31596@sleipnir.bytereef.org> Stefan Behnel wrote: > FWIW, I think this definitely makes sense in cases where the C > implementation is essentially a complete replacement of the original > module, such as in this case. I even sometimes suggest compiling Python > modules with Cython if the import time matters. > > For "normal" accelerator modules that only replace a part of a Python > module with a native implementation, this is less likely to make a large > enough difference to make up for the additional complexity due to the code > split. I wonder how big the cumulative effect is (yes, I should really measure instead of wondering ;)). Anyhow, to measure the effect for decimal in a real world application, these are the startup times of Python if decimal is imported at startup: diff --git a/Lib/site.py b/Lib/site.py --- a/Lib/site.py +++ b/Lib/site.py @@ -72,6 +72,7 @@ import os import builtins import _sitebuiltins +import decimal # Prefixes for site-packages; add additional prefixes like /usr/local here PREFIXES = [sys.prefix, sys.exec_prefix] With the current scheme: ======================== $ time ./python -c "pass" real 0m0.040s user 0m0.036s sys 0m0.004s With the patch for #19232: ========================== $ time ./python -c "pass" real 0m0.023s user 0m0.016s sys 0m0.004s But I agree that decimal.py is a special case, since it's very large *and* imports many other modules. Stefan Krah From g.brandl at gmx.net Sat Oct 12 20:04:59 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 12 Oct 2013 20:04:59 +0200 Subject: [Python-Dev] Support for marking limited API elements in C API docs Message-ID: Hi, in light of the recent thread about PEPs not forming part of the docs, I've just pushed a change that allows to document C API elements not part of the limited API as such. It is done like this: .. c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize) :notlimited: I have not yet begun adding these to the documents; if someone wants to help with this I am glad for volunteers. cheers, Georg From g.brandl at gmx.net Sat Oct 12 20:06:45 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 12 Oct 2013 20:06:45 +0200 Subject: [Python-Dev] Right place for PBKDF2 wrapper In-Reply-To: <20131012193754.30af1e9f@fsol> References: <20131012193754.30af1e9f@fsol> Message-ID: Am 12.10.2013 19:37, schrieb Antoine Pitrou: > On Sat, 12 Oct 2013 19:19:44 +0200 > Christian Heimes wrote: >> Hi, >> >> I have written a interface to OpenSSL's PKCS5_PBKDF2_HMAC() function. It >> implements PKCS#5's password based key derivation function 2 with HMAC >> as pseudo-random function. It supports any digest that is supported by >> OpenSSL, e.g. SHA-1, SHA-256 and SHA-512. It's a low level inteface that >> takes the digest as unicode name, password and salt as bytes/buffer, >> keylen and rounds as int. >> >> I'd like to add the feature to Python 3.4. Now I'm looking for a good >> place to put it and some high level functions. In the future I like to >> add scrypt and bcrypt key stretching and key derivation functions, too. >> What's a good place for them?? >> >> * add a new ``kdf`` module (key derivation function) >> * add PBKDF2 to ``hashlib`` >> * make ``hashlib`` a package and add PBKDF2 to a new ``hashlib.kdf`` module >> * make ``hashlib`` a package and add PBKDF2 to a new ``hashlib.pbkdf2`` >> module >> * make ``crypt`` work under Windows and add PKBDF2 to it > > Putting it in "hashlib" sounds fine. There's no reason to create a > myriad of small separate modules. Maybe it's a good idea to expose HMAC through hashlib as well, and deprecate the standalone module at some point? Georg From storchaka at gmail.com Sat Oct 12 20:20:38 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 12 Oct 2013 21:20:38 +0300 Subject: [Python-Dev] Support for marking limited API elements in C API docs In-Reply-To: References: Message-ID: 12.10.13 21:04, Georg Brandl ???????(??): > in light of the recent thread about PEPs not forming part of the docs, > I've just pushed a change that allows to document C API elements > not part of the limited API as such. It is done like this: > > ... c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize) > :notlimited: > > I have not yet begun adding these to the documents; if someone wants to > help with this I am glad for volunteers. Why this is needed? The limited API is unstable and only developers of CPython can use it (but they can look in headers). From g.brandl at gmx.net Sat Oct 12 21:19:16 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 12 Oct 2013 21:19:16 +0200 Subject: [Python-Dev] Support for marking limited API elements in C API docs In-Reply-To: References: Message-ID: Am 12.10.2013 20:20, schrieb Serhiy Storchaka: > 12.10.13 21:04, Georg Brandl ???????(??): >> in light of the recent thread about PEPs not forming part of the docs, >> I've just pushed a change that allows to document C API elements >> not part of the limited API as such. It is done like this: >> >> ... c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize) >> :notlimited: >> >> I have not yet begun adding these to the documents; if someone wants to >> help with this I am glad for volunteers. > > Why this is needed? The limited API is unstable and only developers of > CPython can use it (but they can look in headers). Well, I may be reading PEP 384 wrongly, but the point is exactly to have a *stable* API for *non-core* developers to rely upon, so that they can build extensions that don't need to be recompiled for every version of Python. The core and standard library extensions do not use the limited API. Georg From solipsis at pitrou.net Sat Oct 12 21:56:21 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 12 Oct 2013 21:56:21 +0200 Subject: [Python-Dev] Support for marking limited API elements in C API docs References: Message-ID: <20131012215621.56b2b98a@fsol> On Sat, 12 Oct 2013 21:19:16 +0200 Georg Brandl wrote: > Am 12.10.2013 20:20, schrieb Serhiy Storchaka: > > 12.10.13 21:04, Georg Brandl ???????(??): > >> in light of the recent thread about PEPs not forming part of the docs, > >> I've just pushed a change that allows to document C API elements > >> not part of the limited API as such. It is done like this: > >> > >> ... c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize) > >> :notlimited: > >> > >> I have not yet begun adding these to the documents; if someone wants to > >> help with this I am glad for volunteers. > > > > Why this is needed? The limited API is unstable and only developers of > > CPython can use it (but they can look in headers). > > Well, I may be reading PEP 384 wrongly, but the point is exactly to have a > *stable* API for *non-core* developers to rely upon, so that they can build > extensions that don't need to be recompiled for every version of Python. This is true. However, I find the proposed markup not very enlightening :-) I would prefer if "limited" APIs where marked with a :stableabi: tag. ("limited API" is really a bad synonym for "stable ABI" IMO) Regards Antoine. From storchaka at gmail.com Sat Oct 12 21:57:29 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 12 Oct 2013 22:57:29 +0300 Subject: [Python-Dev] Support for marking limited API elements in C API docs In-Reply-To: References: Message-ID: 12.10.13 22:19, Georg Brandl ???????(??): > Am 12.10.2013 20:20, schrieb Serhiy Storchaka: >> 12.10.13 21:04, Georg Brandl ???????(??): >>> in light of the recent thread about PEPs not forming part of the docs, >>> I've just pushed a change that allows to document C API elements >>> not part of the limited API as such. It is done like this: >>> >>> ... c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize) >>> :notlimited: >>> >>> I have not yet begun adding these to the documents; if someone wants to >>> help with this I am glad for volunteers. >> >> Why this is needed? The limited API is unstable and only developers of >> CPython can use it (but they can look in headers). > > Well, I may be reading PEP 384 wrongly, but the point is exactly to have a > *stable* API for *non-core* developers to rely upon, so that they can build > extensions that don't need to be recompiled for every version of Python. Sorry, I meant notlimited API. It can vary from version to version and can disappear. For example _PyUnicodeWriter was changed many times (and I suppose it will be changed many times). From g.brandl at gmx.net Sat Oct 12 22:04:57 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 12 Oct 2013 22:04:57 +0200 Subject: [Python-Dev] Support for marking limited API elements in C API docs In-Reply-To: <20131012215621.56b2b98a@fsol> References: <20131012215621.56b2b98a@fsol> Message-ID: Am 12.10.2013 21:56, schrieb Antoine Pitrou: > On Sat, 12 Oct 2013 21:19:16 +0200 > Georg Brandl wrote: >> Am 12.10.2013 20:20, schrieb Serhiy Storchaka: >> > 12.10.13 21:04, Georg Brandl ???????(??): >> >> in light of the recent thread about PEPs not forming part of the docs, >> >> I've just pushed a change that allows to document C API elements >> >> not part of the limited API as such. It is done like this: >> >> >> >> ... c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize) >> >> :notlimited: >> >> >> >> I have not yet begun adding these to the documents; if someone wants to >> >> help with this I am glad for volunteers. >> > >> > Why this is needed? The limited API is unstable and only developers of >> > CPython can use it (but they can look in headers). >> >> Well, I may be reading PEP 384 wrongly, but the point is exactly to have a >> *stable* API for *non-core* developers to rely upon, so that they can build >> extensions that don't need to be recompiled for every version of Python. > > This is true. > > However, I find the proposed markup not very enlightening :-) > I would prefer if "limited" APIs where marked with a :stableabi: tag. The way I did it was based on the expected number of changes, which would be lower with the "not-limited" elements being labeled. But changing it around is trivial. > ("limited API" is really a bad synonym for "stable ABI" IMO) It's not a synonym: to get a stable ABI, you use the limited API. Of course Martin should confirm that this terminology matches his intentions. cheers, Georg From solipsis at pitrou.net Sat Oct 12 22:16:12 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 12 Oct 2013 22:16:12 +0200 Subject: [Python-Dev] Support for marking limited API elements in C API docs References: <20131012215621.56b2b98a@fsol> Message-ID: <20131012221612.42b1e84e@fsol> On Sat, 12 Oct 2013 22:04:57 +0200 Georg Brandl wrote: > Am 12.10.2013 21:56, schrieb Antoine Pitrou: > > On Sat, 12 Oct 2013 21:19:16 +0200 > > Georg Brandl wrote: > >> Am 12.10.2013 20:20, schrieb Serhiy Storchaka: > >> > 12.10.13 21:04, Georg Brandl ???????(??): > >> >> in light of the recent thread about PEPs not forming part of the docs, > >> >> I've just pushed a change that allows to document C API elements > >> >> not part of the limited API as such. It is done like this: > >> >> > >> >> ... c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize) > >> >> :notlimited: > >> >> > >> >> I have not yet begun adding these to the documents; if someone wants to > >> >> help with this I am glad for volunteers. > >> > > >> > Why this is needed? The limited API is unstable and only developers of > >> > CPython can use it (but they can look in headers). > >> > >> Well, I may be reading PEP 384 wrongly, but the point is exactly to have a > >> *stable* API for *non-core* developers to rely upon, so that they can build > >> extensions that don't need to be recompiled for every version of Python. > > > > This is true. > > > > However, I find the proposed markup not very enlightening :-) > > I would prefer if "limited" APIs where marked with a :stableabi: tag. > > The way I did it was based on the expected number of changes, which would > be lower with the "not-limited" elements being labeled. But changing it > around is trivial. Well, tagging things which are "not something" feels weird IMHO. > > ("limited API" is really a bad synonym for "stable ABI" IMO) > > It's not a synonym: to get a stable ABI, you use the limited API. I still don't like that name, because it doesn't convey any interesting information. "Stable ABI" is immediately informative. Regards Antoine. From storchaka at gmail.com Sat Oct 12 22:17:17 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 12 Oct 2013 23:17:17 +0300 Subject: [Python-Dev] Support for marking limited API elements in C API docs In-Reply-To: <20131012215621.56b2b98a@fsol> References: <20131012215621.56b2b98a@fsol> Message-ID: 12.10.13 22:56, Antoine Pitrou ???????(??): > On Sat, 12 Oct 2013 21:19:16 +0200 > Georg Brandl wrote: >> Am 12.10.2013 20:20, schrieb Serhiy Storchaka: >>> 12.10.13 21:04, Georg Brandl ???????(??): >>>> in light of the recent thread about PEPs not forming part of the docs, >>>> I've just pushed a change that allows to document C API elements >>>> not part of the limited API as such. It is done like this: >>>> >>>> ... c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize) >>>> :notlimited: >>>> >>>> I have not yet begun adding these to the documents; if someone wants to >>>> help with this I am glad for volunteers. >>> >>> Why this is needed? The limited API is unstable and only developers of >>> CPython can use it (but they can look in headers). >> >> Well, I may be reading PEP 384 wrongly, but the point is exactly to have a >> *stable* API for *non-core* developers to rely upon, so that they can build >> extensions that don't need to be recompiled for every version of Python. > > This is true. > > However, I find the proposed markup not very enlightening :-) > I would prefer if "limited" APIs where marked with a :stableabi: tag. > > ("limited API" is really a bad synonym for "stable ABI" IMO) Why not limited private API should be documented at all besides sources? From eric at trueblade.com Sat Oct 12 22:09:24 2013 From: eric at trueblade.com (Eric V. Smith) Date: Sat, 12 Oct 2013 16:09:24 -0400 Subject: [Python-Dev] Support for marking limited API elements in C API docs In-Reply-To: References: <20131012215621.56b2b98a@fsol> Message-ID: <5259AC74.9060900@trueblade.com> On 10/12/2013 4:04 PM, Georg Brandl wrote: > Am 12.10.2013 21:56, schrieb Antoine Pitrou: >> On Sat, 12 Oct 2013 21:19:16 +0200 >> Georg Brandl wrote: >>> Am 12.10.2013 20:20, schrieb Serhiy Storchaka: >>>> 12.10.13 21:04, Georg Brandl ???????(??): >>>>> in light of the recent thread about PEPs not forming part of the docs, >>>>> I've just pushed a change that allows to document C API elements >>>>> not part of the limited API as such. It is done like this: >>>>> >>>>> ... c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize) >>>>> :notlimited: >>>>> >>>>> I have not yet begun adding these to the documents; if someone wants to >>>>> help with this I am glad for volunteers. >>>> >>>> Why this is needed? The limited API is unstable and only developers of >>>> CPython can use it (but they can look in headers). >>> >>> Well, I may be reading PEP 384 wrongly, but the point is exactly to have a >>> *stable* API for *non-core* developers to rely upon, so that they can build >>> extensions that don't need to be recompiled for every version of Python. >> >> This is true. >> >> However, I find the proposed markup not very enlightening :-) >> I would prefer if "limited" APIs where marked with a :stableabi: tag. > > The way I did it was based on the expected number of changes, which would > be lower with the "not-limited" elements being labeled. But changing it > around is trivial. I'm not sure I understand what's being labeled "notlimited". But it seems to be a tautology that the stable API/ABI is not changing. So I think it makes more sense to label those as "stableabi" and be finished with this task, rather than trying to remember to add some markup every time we add a new function. -- Eric. From g.brandl at gmx.net Sat Oct 12 22:25:36 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 12 Oct 2013 22:25:36 +0200 Subject: [Python-Dev] Support for marking limited API elements in C API docs In-Reply-To: <20131012221612.42b1e84e@fsol> References: <20131012215621.56b2b98a@fsol> <20131012221612.42b1e84e@fsol> Message-ID: Am 12.10.2013 22:16, schrieb Antoine Pitrou: >> >> Well, I may be reading PEP 384 wrongly, but the point is exactly to have a >> >> *stable* API for *non-core* developers to rely upon, so that they can build >> >> extensions that don't need to be recompiled for every version of Python. >> > >> > This is true. >> > >> > However, I find the proposed markup not very enlightening :-) >> > I would prefer if "limited" APIs where marked with a :stableabi: tag. >> >> The way I did it was based on the expected number of changes, which would >> be lower with the "not-limited" elements being labeled. But changing it >> around is trivial. > > Well, tagging things which are "not something" feels weird IMHO. Sorry, that seems like arbitrary bike-shedding. It's like saying you don't want to document that pickle, shelve and marshal are not safe against untrusted data, but rather document that all other modules are. >> > ("limited API" is really a bad synonym for "stable ABI" IMO) >> >> It's not a synonym: to get a stable ABI, you use the limited API. > > I still don't like that name, because it doesn't convey any interesting > information. "Stable ABI" is immediately informative. It can be changed, but we document the C API, not an ABI. cheers, Georg From solipsis at pitrou.net Sat Oct 12 22:30:39 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 12 Oct 2013 22:30:39 +0200 Subject: [Python-Dev] Support for marking limited API elements in C API docs References: <20131012215621.56b2b98a@fsol> <20131012221612.42b1e84e@fsol> Message-ID: <20131012223039.716be668@fsol> On Sat, 12 Oct 2013 22:25:36 +0200 Georg Brandl wrote: > >> > ("limited API" is really a bad synonym for "stable ABI" IMO) > >> > >> It's not a synonym: to get a stable ABI, you use the limited API. > > > > I still don't like that name, because it doesn't convey any interesting > > information. "Stable ABI" is immediately informative. > > It can be changed, but we document the C API, not an ABI. This is a rather mistaken statement. The only point of using the "limited API" is because it provides the stable ABI. There is *zero* other reason of caring about it. Regards Antoine. From brett at python.org Sat Oct 12 22:31:52 2013 From: brett at python.org (Brett Cannon) Date: Sat, 12 Oct 2013 16:31:52 -0400 Subject: [Python-Dev] Right place for PBKDF2 wrapper In-Reply-To: References: Message-ID: On Sat, Oct 12, 2013 at 1:19 PM, Christian Heimes wrote: > Hi, > > I have written a interface to OpenSSL's PKCS5_PBKDF2_HMAC() function. It > implements PKCS#5's password based key derivation function 2 with HMAC > as pseudo-random function. It supports any digest that is supported by > OpenSSL, e.g. SHA-1, SHA-256 and SHA-512. It's a low level inteface that > takes the digest as unicode name, password and salt as bytes/buffer, > keylen and rounds as int. > > I'd like to add the feature to Python 3.4. Now I'm looking for a good > place to put it and some high level functions. In the future I like to > add scrypt and bcrypt key stretching and key derivation functions, too. > What's a good place for them?? > > [SNIP] > * add PBKDF2 to ``hashlib`` > * make ``hashlib`` a package and add PBKDF2 to a new ``hashlib.kdf`` module > * make ``hashlib`` a package and add PBKDF2 to a new ``hashlib.pbkdf2`` > module > One of these three depending on how bug the new API is (smaller the less need to add new modules). -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Sat Oct 12 22:40:52 2013 From: brian at python.org (Brian Curtin) Date: Sat, 12 Oct 2013 15:40:52 -0500 Subject: [Python-Dev] Support for marking limited API elements in C API docs In-Reply-To: References: <20131012215621.56b2b98a@fsol> Message-ID: On Sat, Oct 12, 2013 at 3:17 PM, Serhiy Storchaka wrote: > 12.10.13 22:56, Antoine Pitrou ???????(??): > >> On Sat, 12 Oct 2013 21:19:16 +0200 >> Georg Brandl wrote: >>> >>> Am 12.10.2013 20:20, schrieb Serhiy Storchaka: >>>> >>>> 12.10.13 21:04, Georg Brandl ???????(??): >>>>> >>>>> in light of the recent thread about PEPs not forming part of the docs, >>>>> I've just pushed a change that allows to document C API elements >>>>> not part of the limited API as such. It is done like this: >>>>> >>>>> ... c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize) >>>>> :notlimited: >>>>> >>>>> I have not yet begun adding these to the documents; if someone wants to >>>>> help with this I am glad for volunteers. >>>> >>>> >>>> Why this is needed? The limited API is unstable and only developers of >>>> CPython can use it (but they can look in headers). >>> >>> >>> Well, I may be reading PEP 384 wrongly, but the point is exactly to have >>> a >>> *stable* API for *non-core* developers to rely upon, so that they can >>> build >>> extensions that don't need to be recompiled for every version of Python. >> >> >> This is true. >> >> However, I find the proposed markup not very enlightening :-) >> I would prefer if "limited" APIs where marked with a :stableabi: tag. >> >> ("limited API" is really a bad synonym for "stable ABI" IMO) > > > Why not limited private API should be documented at all besides sources? Code is not documentation. From g.brandl at gmx.net Sat Oct 12 22:55:29 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 12 Oct 2013 22:55:29 +0200 Subject: [Python-Dev] Support for marking limited API elements in C API docs In-Reply-To: <5259AC74.9060900@trueblade.com> References: <20131012215621.56b2b98a@fsol> <5259AC74.9060900@trueblade.com> Message-ID: Am 12.10.2013 22:09, schrieb Eric V. Smith: > On 10/12/2013 4:04 PM, Georg Brandl wrote: >> Am 12.10.2013 21:56, schrieb Antoine Pitrou: >>> On Sat, 12 Oct 2013 21:19:16 +0200 >>> Georg Brandl wrote: >>>> Am 12.10.2013 20:20, schrieb Serhiy Storchaka: >>>>> 12.10.13 21:04, Georg Brandl ???????(??): >>>>>> in light of the recent thread about PEPs not forming part of the docs, >>>>>> I've just pushed a change that allows to document C API elements >>>>>> not part of the limited API as such. It is done like this: >>>>>> >>>>>> ... c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize) >>>>>> :notlimited: >>>>>> >>>>>> I have not yet begun adding these to the documents; if someone wants to >>>>>> help with this I am glad for volunteers. >>>>> >>>>> Why this is needed? The limited API is unstable and only developers of >>>>> CPython can use it (but they can look in headers). >>>> >>>> Well, I may be reading PEP 384 wrongly, but the point is exactly to have a >>>> *stable* API for *non-core* developers to rely upon, so that they can build >>>> extensions that don't need to be recompiled for every version of Python. >>> >>> This is true. >>> >>> However, I find the proposed markup not very enlightening :-) >>> I would prefer if "limited" APIs where marked with a :stableabi: tag. >> >> The way I did it was based on the expected number of changes, which would >> be lower with the "not-limited" elements being labeled. But changing it >> around is trivial. > > I'm not sure I understand what's being labeled "notlimited". But it > seems to be a tautology that the stable API/ABI is not changing. So I > think it makes more sense to label those as "stableabi" and be finished > with this task, rather than trying to remember to add some markup every > time we add a new function. This is true. I've inverted the logic now. Volunteers still welcome :) Georg From g.brandl at gmx.net Sat Oct 12 23:01:17 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 12 Oct 2013 23:01:17 +0200 Subject: [Python-Dev] Support for marking limited API elements in C API docs In-Reply-To: <20131012223039.716be668@fsol> References: <20131012215621.56b2b98a@fsol> <20131012221612.42b1e84e@fsol> <20131012223039.716be668@fsol> Message-ID: Am 12.10.2013 22:30, schrieb Antoine Pitrou: > On Sat, 12 Oct 2013 22:25:36 +0200 > Georg Brandl wrote: >> >> > ("limited API" is really a bad synonym for "stable ABI" IMO) >> >> >> >> It's not a synonym: to get a stable ABI, you use the limited API. >> > >> > I still don't like that name, because it doesn't convey any interesting >> > information. "Stable ABI" is immediately informative. >> >> It can be changed, but we document the C API, not an ABI. > > This is a rather mistaken statement. The only point of using the > "limited API" is because it provides the stable ABI. There is *zero* > other reason of caring about it. Well, I tried to follow PEP 384 wording. Georg From ethan at stoneleaf.us Sat Oct 12 22:41:40 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 12 Oct 2013 13:41:40 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <5258CA80.8040506@stoneleaf.us> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011175656.CB7AC2509A9@webabinitio.net> <87vc13gq76.fsf@uwakimon.sk.tsukuba.ac.jp> <5258CA80.8040506@stoneleaf.us> Message-ID: <5259B404.1000809@stoneleaf.us> On 10/11/2013 09:05 PM, Ethan Furman wrote: > On 10/11/2013 07:47 PM, Stephen J. Turnbull wrote: >> Nick Coghlan writes: >> >> > (RDM is also right that the exception still has the effect of >> > terminating the block early, but I view names as mnemonics rather >> > than necessarily 100% accurate descriptions of things). > > While I agree that mnemonics are helpful in the right circumstances, these aren't them, not for this name. We aren't > /just/ ignoring the exception, the exception is having a very real effect -- it's terminating the with block. In case it wasn't clear (and it wasn't to me on rereading it ;) this comment was towards Nick, not Stephen. break_on reads just fine to me, although leave_on is probably better. -- ~Ethan~ From greg at krypto.org Sat Oct 12 23:04:41 2013 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 12 Oct 2013 14:04:41 -0700 Subject: [Python-Dev] Right place for PBKDF2 wrapper In-Reply-To: References: <20131012193754.30af1e9f@fsol> Message-ID: On Sat, Oct 12, 2013 at 11:06 AM, Georg Brandl wrote: > Am 12.10.2013 19:37, schrieb Antoine Pitrou: > > On Sat, 12 Oct 2013 19:19:44 +0200 > > Christian Heimes wrote: > ... > >> * add PBKDF2 to ``hashlib`` > >> * make ``hashlib`` a package and add PBKDF2 to a new ``hashlib.kdf`` > module > >> * make ``hashlib`` a package and add PBKDF2 to a new ``hashlib.pbkdf2`` > >> module > agreed with any of these three. > > Putting it in "hashlib" sounds fine. There's no reason to create a > > myriad of small separate modules. > > Maybe it's a good idea to expose HMAC through hashlib as well, and > deprecate > the standalone module at some point? > that also makes sense... I'd leave hmac.py around through at least 3.6 but going ahead and moving the implementation into hashlib in 3.4 makes sense. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Sat Oct 12 23:31:21 2013 From: christian at python.org (Christian Heimes) Date: Sat, 12 Oct 2013 23:31:21 +0200 Subject: [Python-Dev] Right place for PBKDF2 wrapper In-Reply-To: References: <20131012193754.30af1e9f@fsol> Message-ID: <5259BFA9.6000708@python.org> Am 12.10.2013 23:04, schrieb Gregory P. Smith: > agreed with any of these three. I'm going for hashlib.pbkdf2_hmac() for now. Right now it's just one function. We can always add a new module for a high level interface later. > that also makes sense... > > I'd leave hmac.py around through at least 3.6 but going ahead and moving > the implementation into hashlib in 3.4 makes sense. That gives us the opportunity th provide a faster implementation based on OpenSSL's HMAC API http://www.openssl.org/docs/crypto/hmac.html . I'm also tempted to provide a fast one-shot-wonder function that returns the MAC as bytes: hmac(name, key, value) -> result Christian From christian at python.org Sat Oct 12 23:31:21 2013 From: christian at python.org (Christian Heimes) Date: Sat, 12 Oct 2013 23:31:21 +0200 Subject: [Python-Dev] Right place for PBKDF2 wrapper In-Reply-To: References: <20131012193754.30af1e9f@fsol> Message-ID: <5259BFA9.6000708@python.org> Am 12.10.2013 23:04, schrieb Gregory P. Smith: > agreed with any of these three. I'm going for hashlib.pbkdf2_hmac() for now. Right now it's just one function. We can always add a new module for a high level interface later. > that also makes sense... > > I'd leave hmac.py around through at least 3.6 but going ahead and moving > the implementation into hashlib in 3.4 makes sense. That gives us the opportunity th provide a faster implementation based on OpenSSL's HMAC API http://www.openssl.org/docs/crypto/hmac.html . I'm also tempted to provide a fast one-shot-wonder function that returns the MAC as bytes: hmac(name, key, value) -> result Christian From christian at python.org Sat Oct 12 23:35:23 2013 From: christian at python.org (Christian Heimes) Date: Sat, 12 Oct 2013 23:35:23 +0200 Subject: [Python-Dev] Change PEP 399 import recommendation In-Reply-To: <20131012190243.404e477d@fsol> References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> <20131011194100.GA25838@sleipnir.bytereef.org> <20131012134146.GA19814@sleipnir.bytereef.org> <20131012144602.CDBDE25005F@webabinitio.net> <52597F0E.5000007@python.org> <20131012190243.404e477d@fsol> Message-ID: <5259C09B.8050001@python.org> Am 12.10.2013 19:02, schrieb Antoine Pitrou: > If you grow sys.path, imports get slower. > (IMHO it also makes the source tree more cumbersome to navigate) It shouldn't slow down successful imports. The module finder stops at first hit. A directory with pure Python modules is less cumbersome than a couple of facade modules or intended code in try/except ImportError blocks. From solipsis at pitrou.net Sat Oct 12 23:40:04 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 12 Oct 2013 23:40:04 +0200 Subject: [Python-Dev] Change PEP 399 import recommendation In-Reply-To: <5259C09B.8050001@python.org> References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> <20131011194100.GA25838@sleipnir.bytereef.org> <20131012134146.GA19814@sleipnir.bytereef.org> <20131012144602.CDBDE25005F@webabinitio.net> <52597F0E.5000007@python.org> <20131012190243.404e477d@fsol> <5259C09B.8050001@python.org> Message-ID: <20131012234004.55c8ca1c@fsol> On Sat, 12 Oct 2013 23:35:23 +0200 Christian Heimes wrote: > Am 12.10.2013 19:02, schrieb Antoine Pitrou: > > If you grow sys.path, imports get slower. > > (IMHO it also makes the source tree more cumbersome to navigate) > > It shouldn't slow down successful imports. The module finder stops at > first hit. You're forgetting that *other* directories can be appended to sys.path (e.g. the site-packages directories), or any of setuptools' magic things. > A directory with pure Python modules is less cumbersome than a couple > of facade modules or intended code in try/except ImportError blocks. I don't find it cumbersome to be explicit here. It makes it obvious what happens. It also makes it extra easy to import the pure Python if desired (for example for unit tests, or for benchmarking). Regards Antoine. From ethan at stoneleaf.us Sun Oct 13 00:03:49 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 12 Oct 2013 15:03:49 -0700 Subject: [Python-Dev] __objclass__ documentation Message-ID: <5259C745.3000502@stoneleaf.us> It was pointed in Issue16938[1] that __objclass__ is not documented anywhere. Is the following an appropriate description? (in Doc/reference/datamodel.rst in user-defined functions) +-------------------------+-------------------------------+-----------+ | :attr:`__objclass__` | The class this object belongs | | | | to; useful when the object is | | | | a descriptor, or a virtual or | | | | dynamic class attribute, and | | | | it's __class__ attribute does | | | | not match the class it is | | | | associated with, or it is not | | | | in that class' ``__dict__``. | | +-------------------------+-------------------------------+-----------+ [1] http://bugs.python.org/issue16938 -- ~Ethan~ From greg at krypto.org Sun Oct 13 00:27:00 2013 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 12 Oct 2013 15:27:00 -0700 Subject: [Python-Dev] Right place for PBKDF2 wrapper In-Reply-To: <5259BFA9.6000708@python.org> References: <20131012193754.30af1e9f@fsol> <5259BFA9.6000708@python.org> Message-ID: On Sat, Oct 12, 2013 at 2:31 PM, Christian Heimes wrote: > Am 12.10.2013 23:04, schrieb Gregory P. Smith: > > agreed with any of these three. > > I'm going for hashlib.pbkdf2_hmac() for now. Right now it's just one > function. We can always add a new module for a high level interface later. > > > that also makes sense... > > > > I'd leave hmac.py around through at least 3.6 but going ahead and moving > > the implementation into hashlib in 3.4 makes sense. > > That gives us the opportunity th provide a faster implementation based > on OpenSSL's HMAC API http://www.openssl.org/docs/crypto/hmac.html . I'm > also tempted to provide a fast one-shot-wonder function that returns the > MAC as bytes: hmac(name, key, value) -> result > It'd be more consistent with the other hashlib constructors if the one liner was: hashlib.hmac(hash_name_or_func, key, initial_data).digest() (or .hexdigest() for people who want str rather than bytes). -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Sun Oct 13 00:27:33 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 12 Oct 2013 15:27:33 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <5259B404.1000809@stoneleaf.us> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011175656.CB7AC2509A9@webabinitio.net> <87vc13gq76.fsf@uwakimon.sk.tsukuba.ac.jp> <5258CA80.8040506@stoneleaf.us> <5259B404.1000809@stoneleaf.us> Message-ID: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> On Oct 12, 2013, at 1:41 PM, Ethan Furman wrote: > . break_on reads just fine to me, although leave_on is probably better. Sorry, these are both really bad. I've user tested ignore() and had good results. It makes the intention of the code very clear. When you ask someone to describe what "try: f() except SomeException: pass" does, they will say that it ignores the exception. FWIW, I presented this to 2500+ people in the keynote at the 2013 U.S. Pycon and have received favorable feedback. Please don't join Antoine's opposition to my every contribution. The incessant sniping is demoralizing. The module maintainer (Nick) approved the name change from his original preference for ignored(). That should had been the end of it. Now, this thread has 40+ emails and it will probably go on for days. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From janzert at janzert.com Sun Oct 13 01:44:05 2013 From: janzert at janzert.com (Janzert) Date: Sat, 12 Oct 2013 19:44:05 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011175656.CB7AC2509A9@webabinitio.net> <87vc13gq76.fsf@uwakimon.sk.tsukuba.ac.jp> <5258CA80.8040506@stoneleaf.us> <5259B404.1000809@stoneleaf.us> <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> Message-ID: On 10/12/2013 6:27 PM, Raymond Hettinger wrote: > > On Oct 12, 2013, at 1:41 PM, Ethan Furman > wrote: > >> . break_on reads just fine to me, although leave_on is probably better. > > Sorry, these are both really bad. > > I've user tested ignore() and had good results. > It makes the intention of the code very clear. > I'm curious what your users think this will do print("start listing") with ignore(ZeroDivisionError): for i in range(0, 4): inv = 1 / i print("1 over {} is {}".format(i, inv)) print("finished") I think there's a decent chance I'd get it wrong if I was scanning through code with that construct in the midst of it. Janzert From ethan at stoneleaf.us Sun Oct 13 01:30:30 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 12 Oct 2013 16:30:30 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011175656.CB7AC2509A9@webabinitio.net> <87vc13gq76.fsf@uwakimon.sk.tsukuba.ac.jp> <5258CA80.8040506@stoneleaf.us> <5259B404.1000809@stoneleaf.us> <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> Message-ID: <5259DB96.50808@stoneleaf.us> On 10/12/2013 03:27 PM, Raymond Hettinger wrote: > > When you ask someone to describe what > "try: f() except SomeException: pass" does, > they will say that it ignores the exception. And they would be right in that case. > FWIW, I presented this to 2500+ people in the keynote > at the 2013 U.S. Pycon and have received favorable feedback. Were you only displaying the same short form above? Or did you show some with many lines of code inside the with block? If you didn't, is that because it's a bad idea? Compare: try: f() g() h() except SomeException: pass and with contextlib.ignore(SomeException): f() g() h() What's the difference? The difference is that it is far less obvious that g() and h() may never run. And if SomeException is stopping g() and h() from running, it is most certainly *not* being ignored. For the record, I am no longer opposed to this context manager, only to its name. > Please don't join Antoine's opposition to my every contribution. Don't worry, I argue with him, too. ;) And I don't oppose your every contribution (nor his), and I'm pretty sure he doesn't either. > The incessant sniping is demoralizing. You mean like having you state your opposition to a PEP, then having you request to be the PEP delegate, and then having your first few comments show clearly that you have neither followed the thread very closely to that point nor even read the PEP? All that aside, Python is a group project. It's as good as it is because when someone sees something that doesn't make sense, (s)he asks about it. So it seems there are two good courses of action in these situations: 1) realize something is not as clear as it could be, so make a code change and/or a doc change; or 2) take a moment and share your knowledge so others can learn. Complaining is not productive. -- ~Ethan~ From ethan at stoneleaf.us Sun Oct 13 00:46:29 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 12 Oct 2013 15:46:29 -0700 Subject: [Python-Dev] inspect() and dir() In-Reply-To: References: <5254791E.1020803@stoneleaf.us> <5255893C.1030506@stoneleaf.us> <5255D395.2070405@stoneleaf.us> <525627D7.2060904@stoneleaf.us> Message-ID: <5259D145.40406@stoneleaf.us> On 10/10/2013 02:25 AM, Nick Coghlan wrote: > On 10 Oct 2013 14:32, "Ethan Furman" wrote: >> >> That makes sense. >> >> So what should the new functions be called? get_all_members and classify_all_class_attrs ? > > Yeah, those work for me. The only way they should miss anything is if attribute lookup is customised to offer extra > dynamic attributes without implementing custom dir support. http://bugs.python.org/issue19239 From cs at zip.com.au Sun Oct 13 03:15:12 2013 From: cs at zip.com.au (Cameron Simpson) Date: Sun, 13 Oct 2013 12:15:12 +1100 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: Message-ID: <20131013011512.GA61714@cskk.homeip.net> On 11Oct2013 13:51, Mark Lawrence wrote: > On 11/10/2013 11:33, Eric V. Smith wrote: > >And Antoine has again taught me a new word: > >polysemic: having more than one meaning; having multiple meanings > > IMHO a poor word to use. I'm a middle aged Brit who's never heard > of it so people who have English as a second language have little or > no chance :( IMO a fine word. New to me, but perfectly clear and memorable once explained. People with English as a second language might do better than us if they've cottoned onto the patterns arising from the root languages, versus a native speaker basing things entirely on memory. Cheers, -- Cameron Simpson The engine purrs beneath me with a purpose, ready for the pleasure of my hand upon the throttle. - Peter Smith From raymond.hettinger at gmail.com Sun Oct 13 03:50:51 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 12 Oct 2013 18:50:51 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <5259DB96.50808@stoneleaf.us> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011175656.CB7AC2509A9@webabinitio.net> <87vc13gq76.fsf@uwakimon.sk.tsukuba.ac.jp> <5258CA80.8040506@stoneleaf.us> <5259B404.1000809@stoneleaf.us> <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <5259DB96.50808@stoneleaf.us> Message-ID: On Oct 12, 2013, at 4:30 PM, Ethan Furman wrote: >> >> When you ask someone to describe what >> "try: f() except SomeException: pass" does, >> they will say that it ignores the exception. > > And they would be right in that case. > > >> FWIW, I presented this to 2500+ people in the keynote >> at the 2013 U.S. Pycon and have received favorable feedback. > > Were you only displaying the same short form above? Yes. The specific example given was: with ignore(OSError): os.remove(somefile) The intended use for the context manager is for the common cases like these: try: os.mkdir(dirname) except OSError: pass def discard(self, key): """If the keyed message exists, remove it.""" try: self.remove(key) except KeyError: pass Most cases of try/except/pass that I see span only one or two lines in the try-block. > Or did you show some with many lines of code inside the with block? If you didn't, is that because it's a bad idea? > > Compare: > > try: > f() > g() > h() > except SomeException: > pass > Yes, that is usually a bad idea. We don't recommend code like that with try/except. Using a context manager in this case wouldn't make it better. > For the record, I am no longer opposed to this context manager, only to its name. It just like the old days where there were 100+ emails suggesting other names for enumerate() before agreeing that it had been right to begin with. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Oct 13 04:45:06 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 12 Oct 2013 19:45:06 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011175656.CB7AC2509A9@webabinitio.net> <87vc13gq76.fsf@uwakimon.sk.tsukuba.ac.jp> <5258CA80.8040506@stoneleaf.us> <5259B404.1000809@stoneleaf.us> <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <5259DB96.50808@stoneleaf.us> Message-ID: <525A0932.7050405@stoneleaf.us> On 10/12/2013 06:50 PM, Raymond Hettinger wrote: > On Oct 12, 2013, at 4:30 PM, Ethan Furman wrote: >> >> Were you only displaying the same short form above? > > Yes. The specific example given was: > > with ignore(OSError): > os.remove(somefile) That is certainly nicer than the multi-line form. Argh. Okay, okay, I'm +1. If they're not gonna read the docs it doesn't matter what we name it, and in it's proper setting it certainly reads well. Now, if you'll pardon me, I need to go add that to my 2.x collection of goodies. ;) -- ~Ethan~ From vito.detullio at gmail.com Sun Oct 13 06:39:33 2013 From: vito.detullio at gmail.com (Vito De Tullio) Date: Sun, 13 Oct 2013 06:39:33 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). References: <20131013011512.GA61714@cskk.homeip.net> Message-ID: Cameron Simpson wrote: >> >polysemic: having more than one meaning; having multiple meanings >> >> IMHO a poor word to use. I'm a middle aged Brit who's never heard >> of it so people who have English as a second language have little or >> no chance :( > > IMO a fine word. New to me, but perfectly clear and memorable once > explained. > > People with English as a second language might do better than us > if they've cottoned onto the patterns arising from the root languages, > versus a native speaker basing things entirely on memory. just as a data point: polysemic (from my italian native language) it's pretty "self-explanatory": "poly-" it's a very common prefix for "many", and "semic" has the same root of "semantic". Btw the italian version (polisemia) it's pronounced almost the same, so it's "an easy one" to translate. -- By ZeD From cs at zip.com.au Sun Oct 13 06:24:43 2013 From: cs at zip.com.au (Cameron Simpson) Date: Sun, 13 Oct 2013 15:24:43 +1100 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> Message-ID: <20131013042443.GA90160@cskk.homeip.net> On 12Oct2013 15:27, Raymond Hettinger wrote: > On Oct 12, 2013, at 1:41 PM, Ethan Furman wrote: > . break_on reads just fine to me, although leave_on is probably better. > > Sorry, these are both really bad. I agree. > I've user tested ignore() and had good results. > It makes the intention of the code very clear. I think it is good also. I would also be happy with: ignore_exception for greater explicitness or, perhaps better for Ethan's camp: discard_exception But we've got "ignore" in play already. Can't we accept that it is somewhat evocative though clearly not perfect for everyone, and move on? Cheers, -- Cameron Simpson Once a Junior Programmer interrupted a Great Guru of the Sun to ask a Question of no importance. The Great Guru replied in words which the Junior Programmer did not understand. The Junior Programmer sought to rephrase the Question, saying, "Stop me if I appear stupid." The great Guru, without speaking, reached over and pressed L1-A. The Junior Programmer achieved Enlightenment. - Jon Green From cs at zip.com.au Sun Oct 13 08:18:28 2013 From: cs at zip.com.au (Cameron Simpson) Date: Sun, 13 Oct 2013 17:18:28 +1100 Subject: [Python-Dev] __objclass__ documentation In-Reply-To: <5259C745.3000502@stoneleaf.us> References: <5259C745.3000502@stoneleaf.us> Message-ID: <20131013061828.GA65785@cskk.homeip.net> On 12Oct2013 15:03, Ethan Furman wrote: > It was pointed in Issue16938[1] that __objclass__ is not documented anywhere. > > Is the following an appropriate description? (in Doc/reference/datamodel.rst in user-defined functions) > +-------------------------+-------------------------------+-----------+ > | :attr:`__objclass__` | The class this object belongs | | > | | to; useful when the object is | | > | | a descriptor, or a virtual or | | > | | dynamic class attribute, and | | > | | it's __class__ attribute does | | "its" please. Like "his", "her", "their". > | | not match the class it is | | > | | associated with, or it is not | | > | | in that class' ``__dict__``. | | > +-------------------------+-------------------------------+-----------+ > [1] http://bugs.python.org/issue16938 I'd make this two sentences. Just turn the semicolon into a full stop, and commence "This is useful when". I had trouble with the end. How about: or it is not in __class__.__dict__ Do I misunderstand? Cheers, -- Cameron Simpson Trust the computer... the computer is your friend. - Richard Dominelli From ncoghlan at gmail.com Sun Oct 13 08:32:13 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 13 Oct 2013 16:32:13 +1000 Subject: [Python-Dev] Right place for PBKDF2 wrapper In-Reply-To: References: Message-ID: On 13 Oct 2013 06:34, "Brett Cannon" wrote: > > > > > On Sat, Oct 12, 2013 at 1:19 PM, Christian Heimes wrote: >> >> Hi, >> >> I have written a interface to OpenSSL's PKCS5_PBKDF2_HMAC() function. It >> implements PKCS#5's password based key derivation function 2 with HMAC >> as pseudo-random function. It supports any digest that is supported by >> OpenSSL, e.g. SHA-1, SHA-256 and SHA-512. It's a low level inteface that >> takes the digest as unicode name, password and salt as bytes/buffer, >> keylen and rounds as int. >> >> I'd like to add the feature to Python 3.4. Now I'm looking for a good >> place to put it and some high level functions. In the future I like to >> add scrypt and bcrypt key stretching and key derivation functions, too. >> What's a good place for them?? >> >> [SNIP] >> >> * add PBKDF2 to ``hashlib`` >> * make ``hashlib`` a package and add PBKDF2 to a new ``hashlib.kdf`` module >> * make ``hashlib`` a package and add PBKDF2 to a new ``hashlib.pbkdf2`` >> module > > > One of these three depending on how bug the new API is (smaller the less need to add new modules). +1 to hashlib from me (especially since we used that as the best available home for compare_digest). Cheers, Nick. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sun Oct 13 08:36:12 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 13 Oct 2013 15:36:12 +0900 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131013042443.GA90160@cskk.homeip.net> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> Message-ID: <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> Cameron Simpson writes: > But we've got "ignore" in play already. Can't we accept that it is > somewhat evocative though clearly not perfect for everyone, and > move on? No, that is *way* out. We can overrule the objections, recognizing that sometimes compromise is the worst of the four possible actions (this, that, mix, wait). But don't ask me to "accept" what I consider to be an idea that admits a *lot* of improvement.[1] Let time prove me wrong, please. Footnotes: [1] I've said my piece about "with contextlib.ignore()"; this is not a reiteration. From ncoghlan at gmail.com Sun Oct 13 08:57:09 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 13 Oct 2013 16:57:09 +1000 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 13 Oct 2013 16:37, "Stephen J. Turnbull" wrote: > > Cameron Simpson writes: > > > But we've got "ignore" in play already. Can't we accept that it is > > somewhat evocative though clearly not perfect for everyone, and > > move on? > > No, that is *way* out. We can overrule the objections, recognizing > that sometimes compromise is the worst of the four possible actions > (this, that, mix, wait). Right. For the record, this thread did prompt me to consider the new construct anew, but on reflection, I still consider it a reasonable addition to contextlib. It substantially improves the simple cases it is intended to help with, and, if anything, makes overly broad exception suppression *more* obviously dubious (because the name of the construct doesn't match the consequences for multi-line suites). Cheers, Nick. > > But don't ask me to "accept" what I consider to be an idea that admits > a *lot* of improvement.[1] Let time prove me wrong, please. > > > Footnotes: > [1] I've said my piece about "with contextlib.ignore()"; this is not > a reiteration. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Oct 13 09:27:50 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 13 Oct 2013 00:27:50 -0700 Subject: [Python-Dev] __objclass__ documentation In-Reply-To: <20131013061828.GA65785@cskk.homeip.net> References: <5259C745.3000502@stoneleaf.us> <20131013061828.GA65785@cskk.homeip.net> Message-ID: <525A4B76.6060907@stoneleaf.us> On 10/12/2013 11:18 PM, Cameron Simpson wrote: > On 12Oct2013 15:03, Ethan Furman wrote: >> It was pointed in Issue16938[1] that __objclass__ is not documented anywhere. >> >> Is the following an appropriate description? (in Doc/reference/datamodel.rst in user-defined functions) >> +-------------------------+-------------------------------+-----------+ >> | :attr:`__objclass__` | The class this object belongs | | >> | | to; useful when the object is | | >> | | a descriptor, or a virtual or | | >> | | dynamic class attribute, and | | >> | | it's __class__ attribute does | | > > "its" please. Like "his", "her", "their". Right. > >> | | not match the class it is | | >> | | associated with, or it is not | | >> | | in that class' ``__dict__``. | | >> +-------------------------+-------------------------------+-----------+ >> [1] http://bugs.python.org/issue16938 > > I'd make this two sentences. Just turn the semicolon into a full > stop, and commence "This is useful when". > > I had trouble with the end. How about: > > or it is not in __class__.__dict__ > > Do I misunderstand? You have it right. Good change. -- ~Ethan~ From stefan_ml at behnel.de Sun Oct 13 10:18:49 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 13 Oct 2013 10:18:49 +0200 Subject: [Python-Dev] Change PEP 399 import recommendation In-Reply-To: <20131012234004.55c8ca1c@fsol> References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> <20131011194100.GA25838@sleipnir.bytereef.org> <20131012134146.GA19814@sleipnir.bytereef.org> <20131012144602.CDBDE25005F@webabinitio.net> <52597F0E.5000007@python.org> <20131012190243.404e477d@fsol> <5259C09B.8050001@python.org> <20131012234004.55c8ca1c@fsol> Message-ID: Antoine Pitrou, 12.10.2013 23:40: > On Sat, 12 Oct 2013 23:35:23 +0200 > Christian Heimes wrote: >> A directory with pure Python modules is less cumbersome than a couple >> of facade modules or intended code in try/except ImportError blocks. > > I don't find it cumbersome to be explicit here. It makes it obvious > what happens. It also makes it extra easy to import the pure Python if > desired (for example for unit tests, or for benchmarking). I agree. I find it much easier to read a plain and obvious try: from _cmodule import * except ImportError: from _pymodule import * in a facade module than # lots and lots # of unused # Python code # here ... # ... and then this, well hidden at the end, telling # me that I just waded through the entirely wrong code: try: from _cmodule import * except ImportError: pass As I said, that's different for the case where the C module replaces only a somewhat small part of the Python code, but for the case of an (almost) complete replacement, I would like to see the 3-files approach as the One Way To Do It. Stefan From christian at python.org Sun Oct 13 12:39:48 2013 From: christian at python.org (Christian Heimes) Date: Sun, 13 Oct 2013 12:39:48 +0200 Subject: [Python-Dev] Right place for PBKDF2 wrapper In-Reply-To: References: Message-ID: <525A7874.4080408@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Am 13.10.2013 08:32, schrieb Nick Coghlan: > +1 to hashlib from me (especially since we used that as the best > available home for compare_digest). I'm afraid your memory doesn't serve you well. :( compare_digest is implemented in _operator.c. Its official home is hmac.compare_digest. Christian -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCgAGBQJSWnhsAAoJEMeIxMHUVQ1FeCoP/1iuk+IKsDXYnwvFPqD9XAQV 2Bjs8ohY7pP1fSiupgf3Rc88OqYnsbdldo4DJICkCAsrygGEVncQZ2S4+qCkjdIZ xW0IT5NmUR3VapoLOUbom/giODp/cQiEl/aVHhJlp7xbcv2RBoTQoqYprNxqOv3s tGn8hbC3ptdn/g+eF9SqHNEIAC30YqoqHbKNJXfCtZ8rhG3I2ww90IRdmx0d0Iv9 ZfMqSz6gCqTomoxgb/2l/alSyGqoKky9nHvPGdgbJcUugI+RUInQae/ZBzBEvBbq odtu9cfxnRwkX11dioNnVjL8nV99UskA2CRVEQxk+TO3weNsWI6Er0SgFm8Rd6g9 ElW9YnMnenW/SgNLLGj8tklHheAspoI3HlFaEszEdR/6WAAkd9BjD42Ku598W1ZH HVPzlnQPO1/HOP7GHPekVzMco9CBNcSSSg7DrK15WQbJRqid9c7NDM9HouSX6cdo OnMewGg5Q3lq4zWDstSI5VEfjHn/ZEGG1HHXQkP3LEnmqb1gFlL7Y+i5rCr0Kp4q u4xKMlnoq/K5GXd7YmJ6DktLn5bOmfIyDIct6hncr6uZa0pLsFBJ0xOJ6ZkqVvNY SJHzcwjoN+AB2SpALzZjCjhn3OioCTwQ5j/Ah3QqT0+l1Ynkv7LBy6XuSvAa/Nky 2az2mBUKYqagFPMNiRl4 =G4qD -----END PGP SIGNATURE----- From ncoghlan at gmail.com Sun Oct 13 13:44:06 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 13 Oct 2013 21:44:06 +1000 Subject: [Python-Dev] __objclass__ documentation In-Reply-To: <5259C745.3000502@stoneleaf.us> References: <5259C745.3000502@stoneleaf.us> Message-ID: On 13 Oct 2013 08:27, "Ethan Furman" wrote: > > It was pointed in Issue16938[1] that __objclass__ is not documented anywhere. > > Is the following an appropriate description? (in Doc/reference/datamodel.rst in user-defined functions) > > > > +-------------------------+-------------------------------+-----------+ > | :attr:`__objclass__` | The class this object belongs | | > | | to; useful when the object is | | > | | a descriptor, or a virtual or | | > | | dynamic class attribute, and | | > | | it's __class__ attribute does | | > | | not match the class it is | | > | | associated with, or it is not | | > | | in that class' ``__dict__``. | | > +-------------------------+-------------------------------+-----------+ I think this is inaccurate. The meaning of __objclass__ as described in PEP 252 is to denote unbound callable descriptors that only work with a specific type. That's why it's mostly gone in Python 3. It should be documented, but the documentation should say something like: __objclass__: Indicates this callable requires an instance of the given type (or a subclass) as its first positional argument. For example, CPython sets this for unbound methods that are implemented in C rather than Python. The offending block of code in the inspect module (http://hg.python.org/cpython/file/default/Lib/inspect.py#l366) that prompted http://bugs.python.org/issue16938 just looks flat out broken to me. There's *zero* reason to expect that __class__ on the result of retrieving an attribute from an object will appear in the MRO for that object. I just missed that detail when reviewing http://bugs.python.org/issue19030 :) I'll reply on the tracker with some additional details/suggestions. Cheers, Nick. From stefan at bytereef.org Sun Oct 13 13:46:19 2013 From: stefan at bytereef.org (Stefan Krah) Date: Sun, 13 Oct 2013 13:46:19 +0200 Subject: [Python-Dev] Change PEP 399 import recommendation In-Reply-To: References: <20131012134146.GA19814@sleipnir.bytereef.org> <20131012144602.CDBDE25005F@webabinitio.net> <52597F0E.5000007@python.org> <20131012190243.404e477d@fsol> <5259C09B.8050001@python.org> <20131012234004.55c8ca1c@fsol> Message-ID: <20131013114619.GA22426@sleipnir.bytereef.org> Stefan Behnel wrote: > in a facade module than > > # lots and lots > # of unused > # Python code > # here ... > > # ... and then this, well hidden at the end, telling > # me that I just waded through the entirely wrong code: > > try: > from _cmodule import * > except ImportError: > pass I don't care much about the directory versus facade module approach, but the above is really worth noting. For decimal, I find the three-files approach easier to manage: All private symbols of the Python version are automatically no longer imported; they don't have to be deleted when the C version is used. Also, the new scheme uncovered some missing symbols in __all__. The downside for CPython is just a slightly more cluttered directory listing (and the doctests get a bit messier). The other implementations could import their (hypothetical) accelerators in the facade module and override selected symbols there (I think Nick mentioned that already). Stefan Krah From ncoghlan at gmail.com Sun Oct 13 14:00:36 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 13 Oct 2013 22:00:36 +1000 Subject: [Python-Dev] Right place for PBKDF2 wrapper In-Reply-To: <525A7874.4080408@python.org> References: <525A7874.4080408@python.org> Message-ID: On 13 October 2013 20:39, Christian Heimes wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Am 13.10.2013 08:32, schrieb Nick Coghlan: >> +1 to hashlib from me (especially since we used that as the best >> available home for compare_digest). > > I'm afraid your memory doesn't serve you well. :( > compare_digest is implemented in _operator.c. Its official home is > hmac.compare_digest. *sigh* Well, we knew that was an unintuitive home when we picked it :P We'll get the standard library's crypto APIs to a sensible state some day :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sun Oct 13 14:34:27 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 13 Oct 2013 14:34:27 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20131013143427.1e3b4414@fsol> On Sun, 13 Oct 2013 16:57:09 +1000 Nick Coghlan wrote: > > For the record, this thread did prompt me to consider the new construct > anew, but on reflection, I still consider it a reasonable addition to > contextlib. > > It substantially improves the simple cases it is intended to help with, > and, if anything, makes overly broad exception suppression *more* obviously > dubious (because the name of the construct doesn't match the consequences > for multi-line suites). Why? Anyone can still write "try ... except". The only sticking point for this construct is that it allows to save one or two lines in rather uncommon cases (because most of the time you should do something on an exception, not "ignore" it). The saving is not worth it. Basically instead of: try: # something except FooException: pass You write: from contextlib import ignore with ignore(FooException): # something There's actually more typing involved, and one more API to know about... It is just another case of those "one-liners" that we generally refrain from adding to the stdlib. Regards Antoine. From ncoghlan at gmail.com Sun Oct 13 14:50:07 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 13 Oct 2013 22:50:07 +1000 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131013143427.1e3b4414@fsol> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> Message-ID: On 13 October 2013 22:34, Antoine Pitrou wrote: > On Sun, 13 Oct 2013 16:57:09 +1000 > Nick Coghlan wrote: >> >> For the record, this thread did prompt me to consider the new construct >> anew, but on reflection, I still consider it a reasonable addition to >> contextlib. >> >> It substantially improves the simple cases it is intended to help with, >> and, if anything, makes overly broad exception suppression *more* obviously >> dubious (because the name of the construct doesn't match the consequences >> for multi-line suites). > > Why? Anyone can still write "try ... except". > > The only sticking point for this construct is that it allows to save > one or two lines in rather uncommon cases (because most of the time you > should do something on an exception, not "ignore" it). The saving is > not worth it. > > Basically instead of: > > try: > # something > except FooException: > pass > > You write: > > from contextlib import ignore > with ignore(FooException): > # something > > There's actually more typing involved, and one more API to know about... > It is just another case of those "one-liners" that we generally refrain > from adding to the stdlib. It meets my threshold for inclusion (albeit only just). Your objection has been noted. I still do not agree. Now, can you please just let the matter drop? That's the way this works (unless you think this is such a disastrous addition that you want to appeal to Guido to strip me of my responsibilities as contextlib maintainer and go hunting for a new one). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sun Oct 13 16:05:40 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 13 Oct 2013 16:05:40 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> Message-ID: <20131013160540.10452632@fsol> On Sun, 13 Oct 2013 22:50:07 +1000 Nick Coghlan wrote: > That's the way this works (unless you think this is such a disastrous > addition that you want to appeal to Guido to strip me of my > responsibilities as contextlib maintainer and go hunting for a new > one). Wow... I don't think challenging others to "strip you of your responsibilities" is a reasonable way to address criticism. And for the record, it's not *my* objection; several other core developers have said -1 too: Ezio, Serhiy, Giampaolo, etc. Regards Antoine. From solipsis at pitrou.net Sun Oct 13 16:11:12 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 13 Oct 2013 16:11:12 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> <20131013160540.10452632@fsol> Message-ID: <20131013161112.08166875@fsol> On Sun, 13 Oct 2013 16:05:40 +0200 Antoine Pitrou wrote: > On Sun, 13 Oct 2013 22:50:07 +1000 > Nick Coghlan wrote: > > That's the way this works (unless you think this is such a disastrous > > addition that you want to appeal to Guido to strip me of my > > responsibilities as contextlib maintainer and go hunting for a new > > one). > > Wow... I don't think challenging others to "strip you of your > responsibilities" is a reasonable way to address criticism. > > And for the record, it's not *my* objection; several other core > developers have said -1 too: Ezio, Serhiy, Giampaolo, etc. And, by the way, thanks for the doc changes: they make the concerns clear. Regards Antoine. From storchaka at gmail.com Sun Oct 13 16:15:00 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 13 Oct 2013 17:15:00 +0300 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> Message-ID: 13.10.13 15:50, Nick Coghlan ???????(??): > On 13 October 2013 22:34, Antoine Pitrou wrote: >> There's actually more typing involved, and one more API to know about... >> It is just another case of those "one-liners" that we generally refrain >> from adding to the stdlib. > > It meets my threshold for inclusion (albeit only just). Your objection > has been noted. I still do not agree. Now, can you please just let the > matter drop? Not only Antoine's objection. Many other people consider this at least useless. From victor.stinner at gmail.com Sun Oct 13 16:21:42 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 13 Oct 2013 16:21:42 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131013160540.10452632@fsol> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> <20131013160540.10452632@fsol> Message-ID: 2013/10/13 Antoine Pitrou : > On Sun, 13 Oct 2013 22:50:07 +1000 > Nick Coghlan wrote: >> That's the way this works (unless you think this is such a disastrous >> addition that you want to appeal to Guido to strip me of my >> responsibilities as contextlib maintainer and go hunting for a new >> one). > > Wow... I don't think challenging others to "strip you of your > responsibilities" is a reasonable way to address criticism. > > And for the record, it's not *my* objection; several other core > developers have said -1 too: Ezio, Serhiy, Giampaolo, etc. You can also count me as -1: the context manager looks like it ignores the exception, whereas it does not: you exit immediatly the block when the exception occurs. It's maybe an issue with the name. Like Antoine, I'm not sure that we need a new context manage to save 2 lines. Victor From ethan at stoneleaf.us Sun Oct 13 16:15:28 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 13 Oct 2013 07:15:28 -0700 Subject: [Python-Dev] __objclass__ documentation In-Reply-To: References: <5259C745.3000502@stoneleaf.us> Message-ID: <525AAB00.3090404@stoneleaf.us> On 10/13/2013 04:44 AM, Nick Coghlan wrote: > On 13 Oct 2013 08:27, "Ethan Furman" wrote: >> >> It was pointed in Issue16938[1] that __objclass__ is not documented anywhere. >> >> Is the following an appropriate description? (in Doc/reference/datamodel.rst in user-defined functions) >> >> >> >> +-------------------------+-------------------------------+-----------+ >> | :attr:`__objclass__` | The class this object belongs | | >> | | to; useful when the object is | | >> | | a descriptor, or a virtual or | | >> | | dynamic class attribute, and | | >> | | it's __class__ attribute does | | >> | | not match the class it is | | >> | | associated with, or it is not | | >> | | in that class' ``__dict__``. | | >> +-------------------------+-------------------------------+-----------+ > > I think this is inaccurate. The meaning of __objclass__ as described > in PEP 252 is to denote unbound callable descriptors that only work > with a specific type. That's why it's mostly gone in Python 3. PEP 252 has this comment: [Editor's note: the ideas described in this PEP have been incorporated into Python. The PEP no longer accurately describes the implementation.] > It should be documented, but the documentation should say something like: > > __objclass__: Indicates this callable requires an instance of the > given type (or a subclass) as its first positional argument. For > example, CPython sets this for unbound methods that are implemented in > C rather than Python. > The offending block of code in the inspect module > (http://hg.python.org/cpython/file/default/Lib/inspect.py#l366) that > prompted http://bugs.python.org/issue16938 just looks flat out broken > to me. There's *zero* reason to expect that __class__ on the result of > retrieving an attribute from an object will appear in the MRO for that > object. I just missed that detail when reviewing > http://bugs.python.org/issue19030 :) Actually, it's the other way around -- the code currently in inspect.py at that point was prompted by this issue (and tracked in Issue 19030). Interestingly enough, with the current changes in 16938 I can remove the __objclass__ attribute I had added to Enum to get around this. -- ~Ethan~ From ethan at stoneleaf.us Sun Oct 13 16:20:08 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 13 Oct 2013 07:20:08 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131013143427.1e3b4414@fsol> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> Message-ID: <525AAC18.3050009@stoneleaf.us> On 10/13/2013 05:34 AM, Antoine Pitrou wrote: > On Sun, 13 Oct 2013 16:57:09 +1000 > Nick Coghlan wrote: >> >> For the record, this thread did prompt me to consider the new construct >> anew, but on reflection, I still consider it a reasonable addition to >> contextlib. >> >> It substantially improves the simple cases it is intended to help with, >> and, if anything, makes overly broad exception suppression *more* obviously >> dubious (because the name of the construct doesn't match the consequences >> for multi-line suites). > > Why? Anyone can still write "try ... except". True, but Raymond's example of with ignore(OSError): os.remove('somefile') struck a chord -- I've used the try ... except version of that many times, and each time it's been a pain. I now agree this context manager is a win. -- ~Ethan~ From victor.stinner at gmail.com Sun Oct 13 17:05:39 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 13 Oct 2013 17:05:39 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <525AAC18.3050009@stoneleaf.us> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> <525AAC18.3050009@stoneleaf.us> Message-ID: 2013/10/13 Ethan Furman : > True, but Raymond's example of > > with ignore(OSError): > os.remove('somefile') And what about: with ignore(OSError): os.remove('file1') os.remove('file2') ? Victor From ncoghlan at gmail.com Sun Oct 13 17:11:41 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 14 Oct 2013 01:11:41 +1000 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> <525AAC18.3050009@stoneleaf.us> Message-ID: On 14 October 2013 01:05, Victor Stinner wrote: > 2013/10/13 Ethan Furman : >> True, but Raymond's example of >> >> with ignore(OSError): >> os.remove('somefile') > > And what about: > > with ignore(OSError): > os.remove('file1') > os.remove('file2') It's just as broken as the try/except equivalent. I consider that a feature, not a bug. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Oct 13 17:20:30 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 14 Oct 2013 01:20:30 +1000 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131013160540.10452632@fsol> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> <20131013160540.10452632@fsol> Message-ID: On 14 October 2013 00:05, Antoine Pitrou wrote: > On Sun, 13 Oct 2013 22:50:07 +1000 > Nick Coghlan wrote: >> That's the way this works (unless you think this is such a disastrous >> addition that you want to appeal to Guido to strip me of my >> responsibilities as contextlib maintainer and go hunting for a new >> one). > > Wow... I don't think challenging others to "strip you of your > responsibilities" is a reasonable way to address criticism. > > And for the record, it's not *my* objection; several other core > developers have said -1 too: Ezio, Serhiy, Giampaolo, etc. When people keep complaining after I have posted, as the module maintainer, saying: ================== "For the record, this thread did prompt me to consider the new construct anew, but on reflection, I still consider it a reasonable addition to contextlib. It substantially improves the simple cases it is intended to help with, and, if anything, makes overly broad exception suppression *more* obviously dubious (because the name of the construct doesn't match the consequences for multi-line suites)." ================== Then, yes, I think challenging people to try to get me booted as module maintainer for contextlib is a *precisely* proportionate response. Getting to decide what colour the bikeshed is painted (and if we even have a bikeshed at all) is what being a module maintainer *means*. I had previously agreed it was a borderline decision. I stated that I had reviewed that decision. I stated that I hadn't changed my mind. Yet people kept complaining. That's a pretty clear statement that they don't trust my judgement. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sun Oct 13 17:44:15 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 13 Oct 2013 17:44:15 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> <20131013160540.10452632@fsol> Message-ID: <20131013174415.75d82c2c@fsol> On Mon, 14 Oct 2013 01:20:30 +1000 Nick Coghlan wrote: > > Then, yes, I think challenging people to try to get me booted as > module maintainer for contextlib is a *precisely* proportionate > response. Because I don't agree with you on one instance doesn't mean I usually *distrust* your judgement. Reasonable organizations have ways to change decisions that doesn't involve overthrowing people (think about appeal courts, for example). This is much better than expecting some select people to be infallible kings :-) Personally, I generally trust you to 1) make enough research and/or gather enough material 2) take informed decisions that do justice to the community's ethos (after, sometimes, a certain amount of arguing :-)). If my criticism came accross as criticizing your competence, then I'm sorry: that was not my intent. Regards Antoine. From pje at telecommunity.com Sun Oct 13 18:00:25 2013 From: pje at telecommunity.com (PJ Eby) Date: Sun, 13 Oct 2013 12:00:25 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131013160540.10452632@fsol> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> <20131013160540.10452632@fsol> Message-ID: On Sun, Oct 13, 2013 at 10:05 AM, Antoine Pitrou wrote: > And for the record, it's not *my* objection; several other core > developers have said -1 too: Ezio, Serhiy, Giampaolo, etc. FWIW, I'm -1 also; the thread quickly convinced me that this is a horrible idea, at least with the current name. The feature itself I consider +0, maybe +0.5 if a good but short name can be found. I kind of like "abort_on()" as an accurate description of what it actually does, but it most certainly does not *ignore* exceptions, and it's going to create problems as soon as anybody adds more than one statement to the block, and then reads their code back without *really* thinking about it. Not to mention how it's going to bite people who copy and modify code snippets containing it. On Sun, Oct 13, 2013 at 11:11 AM, Nick Coghlan wrote: > It's just as broken as the try/except equivalent. I consider that a > feature, not a bug. (Note: the following rant is about the *name*, not the context manager itself.) Misleadingness and lack of readability is not a feature, it's a bug. For example, even though I've been coding in Python since 1997, and even participated in the original design process for "with", I *still* misread the "with ignore:" block as ignoring the exceptions until I *really* thought about it. Wait, no, I misread it *entirely*, until somebody *else* pointed it out. ;-) And this is *despite* knowing on a gut level that *actually* ignoring all the errors in a block *isn't possible in Python*. I would not give most people much chance of noticing they made this mistake, and even less chance of finding the problem afterwards. This is like the infamous Stroop test, where you have a word like "brown" only it's printed in blue ink and you have to say "blue" to get the answer right. If you've never taken a Stroop test, by the way, it's *really* hard. It almost literally makes your brain *hurt* to disregard the text and say the ink color instead, because your brain automatically reads the word before you can stop it, so you are straining to stop yourself from saying it so you can then try to *think* what color you're supposed to say, and then your brain reads the word *again*, and... well, it's really quite unpleasant is what it is. Anyway, this feature, with its current name, is just the same: you have to override your instinctive response to understand what it *really* does, in any but the one-liner case. And you have to do it *every time you read it in code*. Only, because it'll mostly be used in the one-line case, you'll get used to it being correct, until one day you make a change without thinking, and create a bug that lies dormant for an extended period. Plus, as soon as people see it being used, they'll think, "oh cool", and use it in their code, not even knowing or thinking that it does something they don't want, because they will never read the docs in the first place. (As Guido says, people learn languages by example.) So call it "catching". Call it "catch_and_exit_on". Even "passing" or "skipping" would be better. And maybe "abort_on" or "abort_without_raising" would be better still, as they describe what will *really* happen. But calling it "ignore" isn't "fits your brain", it's "abuses your brain in a cruelly misleading manner". From oscar.j.benjamin at gmail.com Sun Oct 13 18:35:59 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sun, 13 Oct 2013 17:35:59 +0100 Subject: [Python-Dev] Change PEP 399 import recommendation In-Reply-To: <52597F0E.5000007@python.org> References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> <20131011194100.GA25838@sleipnir.bytereef.org> <20131012134146.GA19814@sleipnir.bytereef.org> <20131012144602.CDBDE25005F@webabinitio.net> <52597F0E.5000007@python.org> Message-ID: On 12 October 2013 17:55, Christian Heimes wrote: > Am 12.10.2013 17:37, schrieb Nick Coghlan: >> I think the default recommendation in PEP 399 still makes sense - 2 >> modules are easy to manage than three and the idiom allows for easy >> partial replacement. > > We could ues yet another approach and put the pure Python implementation > of modules into another directory (e.g. Lib/pystdlib or Lib/purepython). > This directory is appended to sys.path. Alternative implementations of > Python or platforms without the necessary bits and pieces would import > the Python-only version from that directory. C modules can be renamed so > CPython picks them up in favor of the pure Python modules. No hacks, no > facade module, no slow down and it's easy to understand, too. > > The approach can be used for stat, operator, decimal and perhaps other > modules, too. CPython preferentially imports extension modules over .py modules with the same name. This is what happens when you use Cython in "pure" mode: You have a pure mod.py file and an "augmenting" mod.pxd that contains the annotations relevant to Cython. The resulting extension module is mod.{pyd,so} and is imported by CPython if present. This way you don't need an additional directory/module. Oscar From rdmurray at bitdance.com Sun Oct 13 19:01:12 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 13 Oct 2013 13:01:12 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> <20131013160540.10452632@fsol> Message-ID: <20131013170113.A349C250942@webabinitio.net> +1 to the below. I don't have a strong opinion about the feature itself. The number of times I can imagine using it in code is pretty close to zero, but might not actually turn out to *be* zero, you never know. But the name hurts my brain for the reasons PJ explains :) --David PS: I think you are overreacting, Nick. After all, you did yourself once argue Guido into submission.... :) On Sun, 13 Oct 2013 12:00:25 -0400, PJ Eby wrote: > On Sun, Oct 13, 2013 at 10:05 AM, Antoine Pitrou wrote: > > And for the record, it's not *my* objection; several other core > > developers have said -1 too: Ezio, Serhiy, Giampaolo, etc. > > FWIW, I'm -1 also; the thread quickly convinced me that this is a > horrible idea, at least with the current name. > > The feature itself I consider +0, maybe +0.5 if a good but short name > can be found. I kind of like "abort_on()" as an accurate description > of what it actually does, but it most certainly does not *ignore* > exceptions, and it's going to create problems as soon as anybody adds > more than one statement to the block, and then reads their code back > without *really* thinking about it. Not to mention how it's going to > bite people who copy and modify code snippets containing it. > > > On Sun, Oct 13, 2013 at 11:11 AM, Nick Coghlan wrote: > > It's just as broken as the try/except equivalent. I consider that a > > feature, not a bug. > > (Note: the following rant is about the *name*, not the context manager itself.) > > Misleadingness and lack of readability is not a feature, it's a bug. > > For example, even though I've been coding in Python since 1997, and > even participated in the original design process for "with", I *still* > misread the "with ignore:" block as ignoring the exceptions until I > *really* thought about it. > > Wait, no, I misread it *entirely*, until somebody *else* pointed it out. ;-) > > And this is *despite* knowing on a gut level that *actually* ignoring > all the errors in a block *isn't possible in Python*. > > I would not give most people much chance of noticing they made this > mistake, and even less chance of finding the problem afterwards. > > This is like the infamous Stroop test, where you have a word like > "brown" only it's printed in blue ink and you have to say "blue" to > get the answer right. > > If you've never taken a Stroop test, by the way, it's *really* hard. > It almost literally makes your brain *hurt* to disregard the text and > say the ink color instead, because your brain automatically reads the > word before you can stop it, so you are straining to stop yourself > from saying it so you can then try to *think* what color you're > supposed to say, and then your brain reads the word *again*, and... > well, it's really quite unpleasant is what it is. > > Anyway, this feature, with its current name, is just the same: you > have to override your instinctive response to understand what it > *really* does, in any but the one-liner case. > > And you have to do it *every time you read it in code*. > > Only, because it'll mostly be used in the one-line case, you'll get > used to it being correct, until one day you make a change without > thinking, and create a bug that lies dormant for an extended period. > > Plus, as soon as people see it being used, they'll think, "oh cool", > and use it in their code, not even knowing or thinking that it does > something they don't want, because they will never read the docs in > the first place. (As Guido says, people learn languages by example.) > > So call it "catching". Call it "catch_and_exit_on". Even "passing" > or "skipping" would be better. And maybe "abort_on" or > "abort_without_raising" would be better still, as they describe what > will *really* happen. > > But calling it "ignore" isn't "fits your brain", it's "abuses your > brain in a cruelly misleading manner". > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com From ncoghlan at gmail.com Sun Oct 13 14:43:57 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 13 Oct 2013 22:43:57 +1000 Subject: [Python-Dev] Submitting PEP 453 (pip bootstrapping) for pronouncement Message-ID: The feedback on the previous version of PEP 453 could be summed up as "yes for 3.4, no for 2.7 and 3.3", and I've now updated the PEP accordingly. HTML: http://www.python.org/dev/peps/pep-0453/ Main diff: http://hg.python.org/peps/rev/405b80d54b7d (there are additional minor cleanup diffs after that one) Updates that directly affect the proposal: - *all* functional changes are now 3.4 only - still proposes docs changes for 2.7 & 3.3 - since ``ensurepip`` now never contacts PyPI, we tweaked the proposal to include updating source installs (via "make install" and "make altinstall") to invoke ensurepip by default, to be consistent with the Windows and Mac OS X binary installers. The proposal is explicit that there will be a mechanism to opt out of that step (we haven't decided on a exact mechanism yet, but a make variable seems likely). We had chosen to make invoking ensurepip a separate step back when the proposal still involved implicitly contacting PyPI, but with that aspect removed, we realised our original rationale for the cross-platform inconsistency was no longer valid. Other changes: - dropped rationale that was no longer relevant with the reduced scope - explicitly noted the rejection of making functional changes in 2.7 and 3.3 - noted current effort to create a Windows installer for pip - noted possibility of a future PEP to provide a combined CPython 2.7, pip and Python Launcher installer from python.org Cheers, Nick. ================================= PEP: 453 Title: Explicit bootstrapping of pip in Python installations Version: $Revision$ Last-Modified: $Date$ Author: Donald Stufft , Nick Coghlan BDFL-Delegate: Martin von L?wis Status: Draft Type: Process Content-Type: text/x-rst Created: 10-Aug-2013 Post-History: 30-Aug-2013, 15-Sep-2013, 18-Sep-2013, 19-Sep-2013, 23-Sep-2013, 29-Sep-2013, 13-Oct-2013 Abstract ======== This PEP proposes that the `Installing Python Modules `__ guide in Python 2.7, 3.3 and 3.4 be updated to officially recommend the use of ``pip`` as the default installer for Python packages, and that appropriate technical changes be made in Python 3.4 to provide ``pip`` by default in support of that recommendation. Rationale ========= There are two related, but distinct rationales for the proposal in this PEP. The first relates to the experience of new users, while the second relates to better enabling the evolution of the broader Python packaging ecosystem. Improving the new user experience --------------------------------- Currently, on systems without a platform package manager and repository, installing a third-party Python package into a freshly installed Python requires first identifying an appropriate package manager and then installing it. Even on systems that *do* have a platform package manager, it is unlikely to include every package that is available on the Python Package Index, and even when a desired third-party package is available, the correct name in the platform package manager may not be clear. This means that, to work effectively with the Python Package Index ecosystem, users must know which package manager to install, where to get it, and how to install it. The effect of this is that third-party Python projects are currently required to choose from a variety of undesirable alternatives: * Assume the user already has a suitable cross-platform package manager installed. * Duplicate the instructions and tell their users how to install the package manager. * Completely forgo the use of dependencies to ease installation concerns for their users. All of these available options have significant drawbacks. If a project simply assumes a user already has the tooling then beginning users may get a confusing error message when the installation command doesn't work. Some operating systems may ease this pain by providing a global hook that looks for commands that don't exist and suggest an OS package they can install to make the command work, but that only works on systems with platform package managers that include a package that provides the relevant cross-platform installer command (such as many major Linux distributions). No such assistance is available for Windows and Mac OS X users, or more conservative Linux distributions. The challenges of dealing with this problem for beginners (who are often also completely new to programming, the use of command line tools and editing system environment variables) are a regular feature of feedback the core Python developers receive from professional educators and others introducing new users to Python. If a project chooses to duplicate the installation instructions and tell their users how to install the package manager before telling them how to install their own project then whenever these instructions need updates they need updating by every project that has duplicated them. This is particular problematic when there are multiple competing installation tools available, and different projects recommend different tools. This specific problem can be partially alleviated by strongly promoting ``pip`` as the default installer and recommending that other projects reference `pip's own bootstrapping instructions `__ rather than duplicating them. However the user experience created by this approach still isn't particularly good (although there is an effort under way to create a combined Windows installer for ``pip`` and its dependencies that should improve matters on that platform, and Mac OS X and \*nix platforms generally have ``wget`` and hence the ability to easily download and run the bootstrap scripts from the command line). The projects that have decided to forgo dependencies altogether are forced to either duplicate the efforts of other projects by inventing their own solutions to problems or are required to simply include the other projects in their own source trees. Both of these options present their own problems either in duplicating maintenance work across the ecosystem or potentially leaving users vulnerable to security issues because the included code or duplicated efforts are not automatically updated when upstream releases a new version. By officially recommending and providing by default a specific cross-platform package manager it will be easier for users trying to install these third-party packages as well as easier for the people distributing them as they should now be able to safely assume that most users will have the appropriate installation tools available (or access to clear instructions on how to obtain them). This is expected to become more important in the future as the Wheel_ package format (deliberately) does not have a built in "installer" in the form of ``setup.py`` so users wishing to install from a wheel file will want an installer even in the simplest cases. Reducing the burden of actually installing a third-party package should also decrease the pressure to add every useful module to the standard library. This will allow additions to the standard library to focus more on why Python should have a particular tool out of the box, and why it is reasonable for that package to adopt the standard library's 18-24 month feature release cycle, instead of using the general difficulty of installing third-party packages as justification for inclusion. Providing a standard installation system also helps with bootstrapping alternate build and installer systems, such as ``setuptools``, ``zc.buildout`` and the ``hashdist``/``conda`` combination that is aimed specifically at the scientific community. So long as ``pip install `` works, then a standard Python-specific installer provides a reasonably secure, cross platform mechanism to get access to these utilities. Enabling the evolution of the broader Python packaging ecosystem ---------------------------------------------------------------- As no new packaging standard can achieve widespread adoption without a transition strategy that covers the versions of Python that are in widespread *current* use (rather than merely future versions, like most language features), the change proposed in this PEP is considered a necessary step in the evolution of the Python packaging ecosystem The broader community has embraced the Python Package Index as a mechanism for distributing and installing Python software, but the different concerns of language evolution and secure software distribution mean that a faster feature release cycle that encompasses older versions is needed to properly support the latter. In addition, the core CPython development team have the luxury of dropping support for earlier Python versions well before the rest of the community, as downstream commercial redistributors pick up the task of providing support for those versions to users that still need it, while many third party libraries maintain compatibility with those versions as long as they remain in widespread use. This means that the current ``setup.py install`` based model for package installation poses serious difficulties for the development and adoption of new packaging standards, as, depending on how a project writes their ``setup.py`` file, the installation command (along with other operations) may end up invoking the standard library's ``distutils`` package. As an indicator of how this may cause problems for the broader ecosystem, consider that the feature set of ``distutils`` in Python 2.6 was frozen in June 2008 (with the release of Python 2.6b1), while the feature set of ``distutils`` in Python 2.7 was frozen in April 2010 (with the release of Python 2.7b1). By contrast, using a separate installer application like ``pip`` (which ensures that even ``setup.py`` files that invoke ``distutils`` directly still support the new packaging standards) makes it possible to support new packaging standards in older versions of Python, just by upgrading ``pip`` (which receives new feature releases roughly every 6 months). The situation on older versions of Python is further improved by making it easier for end users to install and upgrade newer build systems like ``setuptools`` or improved PyPI upload utilities like ``twine``. It is not coincidental that this proposed model of using a separate installer program with more metadata heavy and less active distribution formats matches that used by most operating systems (including Windows since the introduction of the installer service and the MSI file format), as well as many other language specific installers. For Python 2.6, this compatibility issue is largely limited to various enterprise Linux distributions (and their downstream derivatives). These distributions often have even slower update cycles than CPython, so they offer full support for versions of Python that are considered "security fix only" versions upstream (and sometimes may even be to the point where the core development team no longer support them at all - you can still get commercial support for Python 2.3 if you really need it!). In practice, the fact that tools like ``wget`` and ``curl`` are readily available on Linux systems, that most users of Python on Linux are already familiar with the command line, and that most Linux distributions ship with a default configuration that makes running Python scripts easy, means that the existing ``pip`` bootstrapping instructions for any \*nix system are already quite straightforward. Even if ``pip`` isn't provided by the system package manager, then using ``wget`` or ``curl`` to retrieve the bootstrap script from www.pip-installer.org and then running it is just a couple of shell commands that can easily be copied and pasted as necessary. Accordingly, for any version of Python on any \*nix system, the need to bootstrap ``pip`` in older versions isn't considered a major barrier to adoption of new packaging standards, since it's just one more small speedbump encountered by users of these long term stable releases. For \*nix systems, this PEP's formal endorsement of ``pip`` as the preferred default packaging tool is seen as more important than the underlying technical details involved in making ``pip`` available by default, since it shifts the nature of the conversation between the developers of ``pip`` and downstream repackagers of both ``pip`` and CPython. For Python 2.7, on the other hand, the compatibility issue for adopting new metadata standards is far more widespread, as it affects the python.org binary installers for Windows and Mac OS X, as well as even relatively fast moving \*nix platforms. Firstly, and unlike Python 2.6, Python 2.7 is still a fully supported upstream version, and will remain so until the release of Python 2.7.9 (currently scheduled for May 2015), at which time it is expected to enter the usual "security fix only" mode. That means there are at least another 19 months where Python 2.7 is a deployment target for Python applications that enjoys full upstream support. Even after the core development team switches 2.7 to security release only mode in 2015, Python 2.7 will likely remain a commercially supported legacy target out beyond 2020. While Python 3 already presents a compelling alternative over Python 2 for *new* Python applications and deployments without an existing investment in Python 2 and without a dependency on specific Python 2 only third party modules (a set which is getting ever smaller over time), it is going to take longer to create compelling business cases to update existing Python 2.7 based infrastructure to Python 3, especially in situations where the culture of automated testing is weak (or nonexistent), making it difficult to effectively use the available migration utilities. While this PEP only proposes documentation changes for Python 2.7, once ``pip`` has a Windows installer available, a separate PEP will be created and submitted proposing the creation and distribution of aggregate installers for future CPython 2.7 maintenance releases that combine the CPython, ``pip`` and Python Launcher for Windows installers into a single download (the separate downloads would still remain available - the aggregate installers would be provided as a convenience, and as a clear indication of the recommended operating environment for Python in Windows systems). Why pip? -------- ``pip`` has been chosen as the preferred default installer, as it is an already popular tool that addresses several design and user experience issues with its predecessor ``easy_install`` (these issues can't readily be fixed in ``easy_install`` itself due to backwards compatibility concerns). ``pip`` is also well suited to working within the bounds of a single Python runtime installation (including associated virtual environments), which is a desirable feature for a tool bundled with CPython. Other tools like ``zc.buildout`` and ``conda`` are more ambitious in their aims (and hence substantially better than ``pip`` at handling external binary dependencies), so it makes sense for the Python ecosystem to treat them more like platform package managers to interoperate with rather than as the default cross-platform installation tool. This relationship is similar to that between ``pip`` and platform package management systems like ``apt`` and ``yum`` (which are also designed to handle arbitrary binary dependencies). Proposal Overview ================= This PEP proposes that the `Installing Python Modules `__ guide be updated to officially recommend the use of ``pip`` as the default installer for Python packages, rather than the current approach of recommending the direct invocation of the ``setup.py install`` command. However, to avoid recommending a tool that CPython does not provide, it is further proposed that the `pip`_ package manager be made available by default when installing CPython 3.4 or later and when creating virtual environments using the standard library's ``venv`` module via the ``pyvenv`` command line utility. To support that end, this PEP proposes the inclusion of an ``ensurepip`` bootstrapping module in Python 3.4, as well as automatic invocation of that module from ``pyvenv`` and changes to the way Python installed scripts are handled on Windows. Using a bootstrap module rather than providing ``pip`` directly helps to clearly demarcate development responsibilities, and to avoid inadvertently downgrading ``pip`` when updating CPython. To provide clear guidance for new users of Python that may not be starting with the latest release, this PEP also proposes that the "Installing Python Modules" guides in Python 2.7 and 3.3 be updated to recommend installing and using ``pip``, rather than invoking ``distutils`` directly. It does *not* propose backporting any of the code changes that are being proposed for Python 3.4. Finally, the PEP also strongly recommends that CPython redistributors and other Python implementations ensure that ``pip`` is available by default, or at the very least, explicitly document the fact that it is not included. This PEP does *not* propose making pip (or any dependencies) directly available as part of the standard library. Instead, pip will be a bundled application provided along with CPython for the convenience of Python users, but subject to its own development life cycle and able to be upgraded independently of the core interpreter and standard library. Explicit bootstrapping mechanism ================================ An additional module called ``ensurepip`` will be added to the standard library whose purpose is to install pip and any of its dependencies into the appropriate location (most commonly site-packages). It will expose a callable named ``bootstrap()`` as well as offer direct execution via ``python -m ensurepip``. The bootstrap will *not* contact PyPI, but instead rely on a private copy of pip stored inside the standard library. Accordingly, only options related to the installation location will be supported (``--user``, ``--root``, etc). It is considered desirable that users be strongly encouraged to use the latest available version of ``pip``, in order to take advantage of the ongoing efforts to improve the security of the PyPI based ecosystem, as well as benefiting from the efforts to improve the speed, reliability and flexibility of that ecosystem. In order to satisfy this goal of providing the most recent version of ``pip`` by default, the private copy of ``pip`` will be updated in CPython maintenance releases, which should align well with the 6-month cycle used for new ``pip`` releases. Security considerations ----------------------- The design in this PEP has been deliberately chosen to avoid making any significant changes to the trust model of the CPython installers for end users that do not subsequently make use of ``pip``. The installers will contain all the components of a fully functioning version of Python, including the ``pip`` installer. The installation process will *not* require network access, and will *not* rely on trusting the security of the network connection established between ``pip`` and the Python package index. Only users that choose to use ``pip`` directly will need to pay attention to any PyPI related security considerations. Reliability considerations -------------------------- By including the bootstrap as part of the standard library (rather than solely as a feature of the binary installers), the correct operation of the bootstrap command can be easily tested using the existing CPython buildbot infrastructure rather than adding significantly to the testing burden for the installers themselves. Implementation strategy ----------------------- To ensure there is no need for network access when installing Python or creating virtual environments, the ``ensurepip`` module will, as an implementation detail, include a complete private copy of pip and its dependencies which will be used to extract pip and install it into the target environment. It is important to stress that this private copy of pip is *only* an implementation detail and it should *not* be relied on or assumed to exist beyond the public capabilities exposed through the ``ensurepip`` module (and indirectly through ``venv``). There is not yet a reference ``ensurepip`` implementation. The existing ``get-pip.py`` bootstrap script demonstrates an earlier variation of the general concept, but the standard library version would take advantage of the improved distribution capabilities offered by the CPython installers to include private copies of ``pip`` and ``setuptools`` as wheel files (rather than as embedded base64 encoded data), and would not try to contact PyPI (instead installing directly from the private wheel files. Rather than including separate code to handle the bootstrapping, the ``ensurepip`` module will manipulate ``sys.path`` appropriately to allow the wheel files to be used to install themselves, either into the current Python installation or into a virtual environment (as determined by the options passed to the bootstrap command). It is proposed that the implementation be carried out in five separate steps (all steps after the first two are independent of each other and can be carried out in any order): * the first step would update the "Installing Python Modules" documentation to recommend the use of ``pip`` and reference the ``pip`` team's instructions for downloading and installing it. This change would be applied to Python 2.7, 3.3, and 3.4. * the ``ensurepip`` module and the private copies of the most recently released versions of pip and setuptools would be added to Python 3.4 and the 3.4 "Installing Python Modules" documentation updated accordingly. * the CPython Windows installer would be updated to offer the new ``pip`` installation option for Python 3.4. * the CPython Mac OS X installer would be updated to offer the new ``pip`` installation option for Python 3.4. * the ``venv`` module and ``pyvenv`` command would be updated to make use of ``ensurepip`` in Python 3.4 * the PATH handling and ``sysconfig`` directory layout on Windows would be updated for Python 3.4+ Proposed CLI ------------ The proposed CLI is based on a subset of the existing ``pip install`` options:: Usage: python -m ensurepip [options] General Options: -h, --help Show help. -v, --verbose Give more output. Option is additive, and can be used up to 3 times. -V, --version Show the pip version that would be extracted and exit. -q, --quiet Give less output. Installation Options: -U, --upgrade Upgrade pip and dependencies, even if already installed --user Install using the user scheme. --root Install everything relative to this alternate root directory. In most cases, end users won't need to use this CLI directly, as ``pip`` should have been installed automatically when installing Python or when creating a virtual environment. However, it is formally documented as a public interface to support at least these known use cases: * Windows and Mac OS X installations where the "Install pip" option was *not* chosen during installation * any installation where the user previously ran "pip uninstall pip" Users that want to retrieve the latest version from PyPI, or otherwise need more flexibility, can then invoke the extracted ``pip`` appropriately. Proposed module API ------------------- The proposed ``ensurepip`` module API consists of the following two functions:: def version(): """ Returns a string specifying the bundled version of pip. """ def bootstrap(root=None, upgrade=False, user=False, verbosity=0): """ Bootstrap pip into the current Python installation (or the given root directory). """ Invocation from the CPython installers -------------------------------------- The CPython Windows and Mac OS X installers will each gain a new option: * Install pip (the default Python package management utility)? This option will be checked by default. If the option is checked, then the installer will invoke the following command with the just installed Python:: python -m ensurepip --upgrade This ensures that, by default, installing or updating CPython will ensure that the installed version of pip is at least as recent as the one included with that version of CPython. If a newer version of pip has already been installed then ``python -m ensurepip --upgrade`` will simply return without doing anything. Installing from source ---------------------- Just as the prebuilt binary installers will be updated to run ``python -m ensurepip`` by default, a similar change will be made to the ``make install`` and ``make altinstall`` commands of the source distribution. The directory settings in the ``sysconfig`` module should ensure the ``pip`` components are automatically installed to the expected locations. ``ensurepip`` itself (including the private copy of ``pip`` and its dependencies) will always be installed normally (as it is a regular part of the standard library), but an option will be provided to skip the invocation of ``ensurepip``. This means that even installing from source will provide ``pip`` by default, but redistributors provide ``pip`` by other means (or not providing it at all) will still be able to opt out of installing it using ``ensurepip``. Changes to virtual environments ------------------------------- Python 3.3 included a standard library approach to virtual Python environments through the ``venv`` module. Since its release it has become clear that very few users have been willing to use this feature directly, in part due to the lack of an installer present by default inside of the virtual environment. They have instead opted to continue using the ``virtualenv`` package which *does* include pip installed by default. To make the ``venv`` more useful to users it will be modified to issue the pip bootstrap by default inside of the new environment while creating it. This will allow people the same convenience inside of the virtual environment as this PEP provides outside of it as well as bringing the ``venv`` module closer to feature parity with the external ``virtualenv`` package, making it a more suitable replacement. To handle cases where a user does not wish to have pip bootstrapped into their virtual environment a ``--without-pip`` option will be added. The ``venv.EnvBuilder`` and ``venv.create`` APIs will be updated to accept one new parameter: ``with_pip`` (defaulting to ``False``). The new default for the module API is chosen for backwards compatibility with the current behaviour (as it is assumed that most invocation of the ``venv`` module happens through third part tools that likely will not want ``pip`` installed without explicitly requesting it), while the default for the command line interface is chosen to try to ensure ``pip`` is available in most virtual environments without additional action on the part of the end user. As this change will only benefit Python 3.4 and later versions, the third-party ``virtualenv`` project will still be needed to obtain a consistent cross-version experience in Python 3.3 and 2.7. Documentation ------------- The "Installing Python Modules" section of the standard library documentation in Python 2.7, 3.3 and 3.4 will be updated to recommend the use of the ``pip`` installer, either provided by default in Python 3.4 or retrieved and installed by the user in Python 2.7 or 3.3. It will give a brief description of the most common commands and options, but delegate to the externally maintained ``pip`` documentation for the full details. In Python 3.4, the ``pyvenv`` and ``venv`` documentation will also be updated to reference the revised module installation guide. The existing content of the module installation guide will be retained in all versions, but under a new "Invoking distutils directly" subsection. Bundling CA certificates with CPython ------------------------------------- The ``ensurepip`` implementation will include the ``pip`` CA bundle along with the rest of ``pip``. This means CPython effectively includes a CA bundle that is used solely by ``pip`` after it has been extracted. This is considered preferable to relying solely on the system certificate stores, as it ensures that ``pip`` will behave the same across all supported versions of Python, even those prior to Python 3.4 that cannot access the system certificate store on Windows. Automatic installation of setuptools ------------------------------------ ``pip`` currently depends on ``setuptools`` to handle metadata generation during the build process, along with some other features. While work is ongoing to reduce or eliminate this dependency, it is not clear if that work will be complete for pip 1.5 (which is the version likely to be current when Python 3.4.0 is released). This PEP proposes that, if pip still requires it as a dependency, ``ensurepip`` will include a private copy of ``setuptools`` (in addition to the private copy of ``ensurepip``). ``python -m ensurepip`` will then install the private copy in addition to installing ``pip`` itself. However, this behavior is officially considered an implementation detail. Other projects which explicitly require ``setuptools`` must still provide an appropriate dependency declaration, rather than assuming ``setuptools`` will always be installed alongside ``pip``. Once pip is able to run ``pip install --upgrade pip`` without needing ``setuptools`` installed first, then the private copy of ``setuptools`` will be removed from ``ensurepip`` in subsequent CPython releases. As long as setuptools is needed, it will be a completely unmodified copy of the latest upstream setuptools release, including the ``easy_install`` script if the upstream setuptools continues to include it. The installation of ``easy_install`` along with ``pip`` isn't considered desirable, but installing a broken setuptools would be worse. This problem will naturally resolve itself once the ``pip`` developers have managed to eliminate their dependency on ``setuptools`` and the private copy of ``setuptools`` can be removed entirely from CPython. Updating the private copy of pip -------------------------------- In order to keep up with evolutions in packaging as well as providing users with as recent version a possible the ``ensurepip`` module will be regularly updated to the latest versions of everything it bootstraps. After each new ``pip`` release, and again during the preparation for any release of Python (including feature releases), a script, provided as part of the implementation for this PEP, will be run to ensure the private copies stored in the CPython source repository have been updated to the latest versions. Updating the ensurepip module API and CLI ----------------------------------------- Like ``venv`` and ``pyvenv``, the ``ensurepip`` module API and CLI will be governed by the normal rules for the standard library: no new features are permitted in maintenance releases. However, the embedded components may be updated as noted above, so the extracted ``pip`` may offer additional functionality in maintenance releases. Uninstallation ============== No changes are proposed to the CPython uninstallation process by this PEP. The bootstrapped pip will be installed the same way as any other pip installed packages, and will be handled in the same way as any other post-install additions to the Python environment. At least on Windows, that means the bootstrapped files will be left behind after uninstallation, since those files won't be associated with the Python MSI installer. While the case can be made for the CPython installers clearing out these directories automatically, changing that behaviour is considered outside the scope of this PEP. Script Execution on Windows =========================== While the Windows installer was updated in Python 3.3 to optionally make ``python`` available on the PATH, no such change was made to include the Scripts directory. Independently of this PEP, a proposal has also been made to rename the ``Tools\Scripts`` subdirectory to ``bin`` in order to improve consistency with the typical script installation directory names on \*nix systems. Accordingly, in addition to adding the option to extract and install ``pip`` during installation, this PEP proposes that the Windows installer (and ``sysconfig``) in Python 3.4 and later be updated to: - install scripts to PythonXY\bin rather than PythonXY\Tools\Scripts - add PythonXY\bin to the Windows PATH (in addition to PythonXY) when the PATH modification option is enabled during installation Note that these changes will only be available in Python 3.4 and later. This means that, for Python 3.3, the most reliable way to invoke pip globally on Windows (without tinkering manually with PATH) will still remain ``py -m pip`` (or ``py -3 -m pip`` to select the Python 3 version if both Python 2 and 3 are installed) rather than simply calling ``pip``. This works because Python 3.3 provides the Python Launcher for Windows (and the associated ``py`` command) by default. For Python 2.7 and 3.2, the most reliable mechanism will be to install the Python Launcher for Windows using the standalone installer and then use ``py -m pip`` as noted above. Adding the scripts directory to the system PATH will mean that ``pip`` works reliably in the "only one Python installation on the system PATH" case, with ``py -m pip``, ``pipX``, or ``pipX.Y`` needed only to select a non-default version in the parallel installation case (and outside a virtual environment). This change should also make the ``pyvenv`` command substantially easier to invoke on Windows, along with all scripts installed by ``pip``, ``easy_install`` and similar tools. While the script invocations on recent versions of Python will run through the Python launcher for Windows, this shouldn't cause any issues, as long as the Python files in the Scripts directory correctly specify a Python version in their shebang line or have an adjacent Windows executable (as ``easy_install`` and ``pip`` do). Recommendations for Downstream Distributors =========================================== A common source of Python installations are through downstream distributors such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX package managers [#homebrew]_ [#macports]_ [#fink]_, and commercial Python redistributors [#ContinuumIO]_ [#ActiveState]_ [#Enthought]_. In order to provide a consistent, user-friendly experience to all users of Python regardless of how they obtained Python this PEP recommends and asks that downstream distributors: * Ensure that whenever Python is installed ``pip`` is either installed or is otherwise made readily available to end users. * For redistributors using binary installers, this may take the form of optionally executing the ``ensurepip`` bootstrap during installation, similar to the CPython installers. * For redistributors using package management systems, it may take the form of separate packages with dependencies on each other so that installing the Python package installs the pip package and installing the pip package installs the Python package. * Another reasonable way to implement this is to package pip separately but ensure that there is some sort of global hook that will recommend installing the separate pip package when a user executes ``pip`` without it being installed. Systems that choose this option should ensure that the ``ensurepip`` module still installs pip directly when invoked inside a virtual environment, but may modify the module in the system Python installation to redirect to the platform provided mechanism when installing ``pip`` globally. * Even if pip is made available globally by other means, do not remove the ``ensurepip`` module in Python 3.4 or later. * ``ensurepip`` will be required for automatic installation of pip into virtual environments by the ``venv`` module. * This is similar to the existing ``virtualenv`` package for which many downstream distributors have already made exception to the common "debundling" policy. * This does mean that if ``pip`` needs to be updated due to a security issue, so does the private copy in the ``ensurepip`` bootstrap module * However, altering the private copy of pip to remove the embedded CA certificate bundle and rely on the system CA bundle instead is a reasonable change. * Ensure that all features of this PEP continue to work with any modifications made to the redistributed version of Python. * Checking the version of pip that will be bootstrapped using ``python -m ensurepip --version`` or ``ensurepip.version()``. * Installation of pip into a global or virtual python environment using ``python -m ensurepip`` or ``ensurepip.bootstrap()``. * ``pip install --upgrade pip`` in a global installation should not affect any already created virtual environments (but is permitted to affect future virtual environments, even though it will not do so when using the standard implementation of ``ensurepip``). * ``pip install --upgrade pip`` in a virtual environment should not affect the global installation. * Migrate build systems to utilize `pip`_ and `Wheel`_ wherever feasible and avoid directly invoking ``setup.py``. * This will help ensure a smoother and more timely migration to improved metadata formats as the Python packaging ecosystem continues to evolve. In the event that a Python redistributor chooses *not* to follow these recommendations, we request that they explicitly document this fact and provide their users with suitable guidance on translating upstream ``pip`` based installation instructions into something appropriate for the platform. Other Python implementations are also encouraged to follow these guidelines where applicable. Policies & Governance ===================== The maintainers of the bootstrapped software and the CPython core team will work together in order to address the needs of both. The bootstrapped software will still remain external to CPython and this PEP does not include CPython subsuming the development responsibilities or design decisions of the bootstrapped software. This PEP aims to decrease the burden on end users wanting to use third-party packages and the decisions inside it are pragmatic ones that represent the trust that the Python community has already placed in the Python Packaging Authority as the authors and maintainers of ``pip``, ``setuptools``, PyPI, ``virtualenv`` and other related projects. Backwards Compatibility ----------------------- The public API and CLI of the ``ensurepip`` module itself will fall under the typical backwards compatibility policy of Python for its standard library. The externally developed software that this PEP bundles does not. Most importantly, this means that the bootstrapped version of pip may gain new features in CPython maintenance releases, and pip continues to operate on its own 6 month release cycle rather than CPython's 18-24 month cycle. Security Releases ----------------- Any security update that affects the ``ensurepip`` module will be shared prior to release with the Python Security Response Team (security at python.org). The PSRT will then decide if the reported issue warrants a security release of CPython with an updated private copy of ``pip``. Licensing --------- ``pip`` is currently licensed as 1 Clause BSD, and it contains code taken from other projects. Additionally this PEP will include setuptools until such time as pip no longer requires it. The licenses for these appear in the table below. ================= ============ Project License ================= ============ requests Apache 2.0 six 1 Clause BSD html5lib 1 Clause BSD distlib PSF colorama 3 Clause BSD Mozilla CA Bundle LGPL setuptools PSF ================= ============ All of these licenses should be compatible with the PSF license. Additionally it is unclear if a CA Bundle is copyrightable material and thus if it needs or can be licensed at all. Appendix: Rejected Proposals ============================ Including ensurepip in Python 2.7, and 3.3 ------------------------------------------ Earlier versions of this PEP made the case that the challenges of getting ``pip`` bootstrapped for new users posed a significant enough barrier to Python's future growth that it justified adding ``ensurepip`` as a new feature in the upcoming Python 2.7 and 3.3 maintenance releases. While the proposal to provide ``pip`` with Python 3.4 was universally popular, this part of the proposal was highly controversial and ultimately `rejected by MvL as BDFL-Delegate `__. Accordingly, the proposal to backport ``ensurepip`` to Python 2.7 and 3.3 has been removed from this PEP in favour of creating a Windows installer for ``pip`` and a possible future PEP suggesting creation of an aggregate installer for Python 2.7 that combines CPython 2.7, ``pip`` and the Python Launcher for Windows. Automatically contacting PyPI when bootstrapping pip ---------------------------------------------------- Earlier versions of this PEP called the bootstrapping module ``getpip`` and defaulted to downloading and installing ``pip`` from PyPI, with the private copy used only as a fallback option or when explicitly requested. This resulted in several complex edge cases, along with difficulties in defining a clean API and CLI for the bootstrap module. It also significantly altered the default trust model for the binary installers published on python.org, as end users would need to explicitly *opt-out* of trusting the security of the PyPI ecosystem (rather than opting in to it by explicitly invoking ``pip`` following installation). As a result, the PEP was simplified to the current design, where the bootstrapping *always* uses the private copy of ``pip``. Contacting PyPI is now always an explicit separate step, with direct access to the full pip interface. Removing the implicit attempt to access PyPI also made it feasible to invoke ``ensurepip`` by default when installing from a custom source build. Implicit bootstrap ------------------ `PEP439`_, the predecessor for this PEP, proposes its own solution. Its solution involves shipping a fake ``pip`` command that when executed would implicitly bootstrap and install pip if it does not already exist. This has been rejected because it is too "magical". It hides from the end user when exactly the pip command will be installed or that it is being installed at all. It also does not provide any recommendations or considerations towards downstream packagers who wish to manage the globally installed pip through the mechanisms typical for their system. The implicit bootstrap mechanism also ran into possible permissions issues, if a user inadvertently attempted to bootstrap pip without write access to the appropriate installation directories. Including pip directly in the standard library ---------------------------------------------- Similar to this PEP is the proposal of just including pip in the standard library. This would ensure that Python always includes pip and fixes all of the end user facing problems with not having pip present by default. This has been rejected because we've learned, through the inclusion and history of ``distutils`` in the standard library, that losing the ability to update the packaging tools independently can leave the tooling in a state of constant limbo. Making it unable to ever reasonably evolve in a time frame that actually affects users as any new features will not be available to the general population for *years*. Allowing the packaging tools to progress separately from the Python release and adoption schedules allows the improvements to be used by *all* members of the Python community and not just those able to live on the bleeding edge of Python releases. There have also been issues in the past with the "dual maintenance" problem if a project continues to be maintained externally while *also* having a fork maintained in the standard library. Since external maintenance of ``pip`` will always be needed to support earlier Python versions, the proposed bootstrapping mechanism will becoming the explicit responsibility of the CPython core developers (assisted by the pip developers), while pip issues reported to the CPython tracker will be migrated to the pip issue tracker. There will no doubt still be some user confusion over which tracker to use, but hopefully less than has been seen historically when including complete public copies of third-party projects in the standard library. The approach described in this PEP also avoids some technical issues related to handling CPython maintenance updates when pip has been independently updated to a more recent version. The proposed pip-based bootstrapping mechanism handles that automatically, since pip and the system installer never get into a fight about who owns the pip installation (it is always managed through pip, either directly, or indirectly via the ``ensurepip`` bootstrap module). Finally, the separate bootstrapping step means it is also easy to avoid installing ``pip`` at all if end users so desire. This is often the case if integrators are using system packages to handle installation of components written in multiple languages using a common set of tools. Defaulting to --user installation --------------------------------- Some consideration was given to bootstrapping pip into the per-user site-packages directory by default. However, this behavior would be surprising (as it differs from the default behavior of pip itself) and is also not currently considered reliable (there are some edge cases which are not handled correctly when pip is installed into the user site-packages directory rather than the system site-packages). .. _Wheel: http://www.python.org/dev/peps/pep-0427/ .. _pip: http://www.pip-installer.org .. _setuptools: https://pypi.python.org/pypi/setuptools .. _PEP439: http://www.python.org/dev/peps/pep-0439/ References ========== .. [1] Discussion thread 1 (distutils-sig) (https://mail.python.org/pipermail/distutils-sig/2013-August/022529.html) .. [2] Discussion thread 2 (distutils-sig) (https://mail.python.org/pipermail/distutils-sig/2013-September/022702.html) .. [3] Discussion thread 3 (python-dev) (https://mail.python.org/pipermail/python-dev/2013-September/128723.html) .. [4] Discussion thread 4 (python-dev) (https://mail.python.org/pipermail/python-dev/2013-September/128780.html) .. [5] Discussion thread 5 (python-dev) (https://mail.python.org/pipermail/python-dev/2013-September/128894.html) .. [#ubuntu] `Ubuntu ` .. [#debian] `Debian ` .. [#fedora] `Fedora ` .. [#homebrew] `Homebrew ` .. [#macports] `MacPorts ` .. [#fink] `Fink ` .. [#ContinuumIO] `Anaconda ` .. [#ActiveState] `ActivePython ` .. [#Enthought] `Enthought Canopy ` Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From alexander.belopolsky at gmail.com Sun Oct 13 19:58:56 2013 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Sun, 13 Oct 2013 13:58:56 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131013170113.A349C250942@webabinitio.net> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> <20131013160540.10452632@fsol> <20131013170113.A349C250942@webabinitio.net> Message-ID: On Sun, Oct 13, 2013 at 1:01 PM, R. David Murray wrote: > But the name hurts my brain I did not think I would weigh in on this because Ramond's and Nick's position seemed quite rational and the issue minor, but surprisingly this thread does not seem to end. The feature (*and* its name) is a fine solution for the problem that it solves and is an improvement over status quo. I find with ignore(OSError): os.unlink(path) much easier to read than try: os.unlink(path) except OSError: pass Yes, this feature can be abused with multiple statements in the with block, but so can try/except. People who write code using contextlib are expected to know that it is not a good idea to keep resources longer than necessary and multiple unrelated statements within the with block will raise a mental red flag. It is also easy for lint-like tools to warn about abuse of ignore(). I also believe that any of the proposed alternative names will make canonical usage (such as in the unlink example above) harder to understand. Let's not try to improve readability of bad code at the expense of good code. From rdmurray at bitdance.com Sun Oct 13 20:09:16 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 13 Oct 2013 14:09:16 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> <20131013160540.10452632@fsol> <20131013170113.A349C250942@webabinitio.net> Message-ID: <20131013180917.12349250814@webabinitio.net> On Sun, 13 Oct 2013 13:58:56 -0400, Alexander Belopolsky wrote: > The feature (*and* its name) is a fine solution for the problem that > it solves and is an improvement over status quo. > > I find > > with ignore(OSError): > os.unlink(path) > > much easier to read than > > try: > os.unlink(path) > except OSError: > pass > > Yes, this feature can be abused with multiple statements in the with > block, but so can try/except. People who write code using contextlib > are expected to know that it is not a good idea to keep resources > longer than necessary and multiple unrelated statements within the > with block will raise a mental red flag. It is also easy for > lint-like tools to warn about abuse of ignore(). With blocks *very often* include multiple lines of code, because the resource shouldn't be released until all those lines are done. try/except blocks with more than a couple lines of code are a code smell. with replaces try/finally, not try/except. I have no problem with Nick deciding that none of the arguments convince him and leaving it in. We'll see how it turns out in practice :) --David From stefan_ml at behnel.de Sun Oct 13 20:18:14 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 13 Oct 2013 20:18:14 +0200 Subject: [Python-Dev] Change PEP 399 import recommendation In-Reply-To: References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> <20131011194100.GA25838@sleipnir.bytereef.org> <20131012134146.GA19814@sleipnir.bytereef.org> <20131012144602.CDBDE25005F@webabinitio.net> <52597F0E.5000007@python.org> Message-ID: Oscar Benjamin, 13.10.2013 18:35: > CPython preferentially imports extension modules over .py modules with > the same name. This is what happens when you use Cython in "pure" > mode: You have a pure mod.py file and an "augmenting" mod.pxd that > contains the annotations relevant to Cython. The resulting extension > module is mod.{pyd,so} and is imported by CPython if present. This way > you don't need an additional directory/module. While it may work, I wouldn't really recommend that for manually written modules. For Cython, it's a nice feature because you get the same thing, just in native code. It also helps with stack traces because Python can extract the source code lines for exceptions raised in native code from the original .py file. For manually written modules, the source code is naturally unrelated between both implementations. You want to test both separately and may even want to import one or the other specifically for other reasons. Explicitly given them different names is very helpful here. Stefan From alexander.belopolsky at gmail.com Sun Oct 13 20:19:46 2013 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Sun, 13 Oct 2013 14:19:46 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131013180917.12349250814@webabinitio.net> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> <20131013160540.10452632@fsol> <20131013170113.A349C250942@webabinitio.net> <20131013180917.12349250814@webabinitio.net> Message-ID: On Sun, Oct 13, 2013 at 2:09 PM, R. David Murray wrote: > with replaces try/finally, not try/except. with ignore(..) is different. This is one of the things that I like about it - a demonstration of a context manager that replaces try/except. From p.f.moore at gmail.com Sun Oct 13 20:26:23 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 13 Oct 2013 19:26:23 +0100 Subject: [Python-Dev] Submitting PEP 453 (pip bootstrapping) for pronouncement In-Reply-To: References: Message-ID: On 13 October 2013 13:43, Nick Coghlan wrote: > Accordingly, in addition to adding the option to extract and install ``pip`` > during installation, this PEP proposes that the Windows installer (and > ``sysconfig``) in Python 3.4 and later be updated to: > > - install scripts to PythonXY\bin rather than PythonXY\Tools\Scripts > - add PythonXY\bin to the Windows PATH (in addition to PythonXY) when the > PATH modification option is enabled during installation Then implication of this is that venv (and virtualenv, but that's less relevant to python-dev) should be updated to use a "bin" directory on Windows rather than the current "Scripts" directory. This would only be for 3.4, which means that virtualenv will need to make the directory name conditional on version, I guess. Paul. From victor.stinner at gmail.com Sun Oct 13 21:08:13 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 13 Oct 2013 21:08:13 +0200 Subject: [Python-Dev] Change PEP 399 import recommendation In-Reply-To: References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> <20131011194100.GA25838@sleipnir.bytereef.org> <20131012134146.GA19814@sleipnir.bytereef.org> <20131012144602.CDBDE25005F@webabinitio.net> <52597F0E.5000007@python.org> <20131012190243.404e477d@fsol> <5259C09B.8050001@python.org> <20131012234004.55c8ca1c@fsol> Message-ID: Le 13 oct. 2013 10:19, "Stefan Behnel" a ?crit : > I agree. I find it much easier to read a plain and obvious > > try: > from _cmodule import * > except ImportError: > from _pymodule import * > > in a facade module ... I miss maybe something. I don't understand why you would like to use the Python implementation if a C implementation is available and probably (much) faster. Maybe developers of PyPy, IronPython and Jython appreciate our effort to provide a Python implementation which is up to date and well tested. But why should we pay a price (bad performance) if the module is not (never? who uses _pyio?) used in CPython. If another Python uses it, the file can be renamed (ex: _pyio.py becomes simply io.py). IMO if the C module is complete (has no extra functions in the "facade" module), it should get the main name (ex: decimal, not _decimal). It was discussed to put the Python implementation in anothet directory. Good idea, but only put the directory in sys.path for unit tests (prepend it to sys.path to load the Python module). I am only talking of C module which does not need anything of the "facade" module. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Sun Oct 13 21:16:54 2013 From: pje at telecommunity.com (PJ Eby) Date: Sun, 13 Oct 2013 15:16:54 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> <20131013160540.10452632@fsol> <20131013170113.A349C250942@webabinitio.net> Message-ID: On Sun, Oct 13, 2013 at 1:58 PM, Alexander Belopolsky wrote: > People who write code using contextlib > are expected to know People who *read* that code while learning Python cannot be expected to know that it is not really possible to ignore errors in Python. If this feature is used under any name that implies such, it will within a few years become a FAQ and well-known wart, not to mention a meme that "contextlib.ignore() is buggy, it only works if the error is thrown from a single operation performed in C". I say this latter phrasing because now that I've had time to think about it, it is not at all merely a question of whether you wrap a single line or single operation. Quick, is this a safe use, or not: with ignore(OSError): delete_widget_files(spam) It sort of depends on the implementation of delete_widget_files, doesn't it? In contrast: with abort_on(OSError): delete_widget_files(spam) it's immediately clear that the error isn't going to be ignored; the operation will be aborted. Very different concept. > that it is not a good idea to keep resources > multiple unrelated statements within the > with block will raise a mental red flag. How will someone know this when they are reading code they found on the internet? It's one thing to have an operation whose name implies, "you need to do more research to understand this". But it's an entirely different (and dangerous) thing to have an operation whose name implies you already know everything you need to know, no need to think or study further... especially if what you know is actually wrong! > It is also easy for > lint-like tools to warn about abuse of ignore(). Since it's not sufficient to require a single operation, how will a lint-like tool check this? For example: with ignore(AnError, OtherError): ping.pongy(foo, bar.spam(), fizzy()) Is this valid code, or not? If you can't tell, how will a non-human lint tool tell? > Let's not try to improve readability of bad code Actually, I want the good code to be "readable" in the sense of understanding what the operation does, so that people copying it don't end up with a serious misunderstanding of how the context manager actually works. There is no way that naive users aren't going to read it as ignoring errors, and use it with something like: with ignore(OSError): for f in myfiles: os.unlink(f) But this version is obviously *wrong*: with abort_on(OSError): for f in myfiles: os.unlink(f) Upon looking at this code, you will quickly realize that you don't intend to abort the loop, only the unlink, and will therefore rewrite it to put the loop on the outside. So, I am only trying to "improve readability of bad code" in the sense of making it *obvious* that the code is in fact bad. ;-) (To put it another way, "ignore()" improves the readability of bad code in the above example, because it makes the bad code look like it's good.) From ncoghlan at gmail.com Mon Oct 14 00:17:10 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 14 Oct 2013 08:17:10 +1000 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131013174415.75d82c2c@fsol> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <20131013143427.1e3b4414@fsol> <20131013160540.10452632@fsol> <20131013174415.75d82c2c@fsol> Message-ID: On 14 Oct 2013 01:45, "Antoine Pitrou" wrote: > > On Mon, 14 Oct 2013 01:20:30 +1000 > Nick Coghlan wrote: > > > > Then, yes, I think challenging people to try to get me booted as > > module maintainer for contextlib is a *precisely* proportionate > > response. > > Because I don't agree with you on one instance doesn't mean I usually > *distrust* your judgement. Reasonable organizations have ways to change > decisions that doesn't involve overthrowing people (think about appeal > courts, for example). This is much better than expecting some select > people to be infallible kings :-) I think I'm mostly irritated by the fact that this thread (which is almost completely inconsequential for the overall language design regardless of whether the feature stays or goes) attracts dozens of posts, while actually hard 3.x usability problems like getting the codec support back to parity with 2.x are still languishing for lack of interest. These days it seems progress is only made on the latter when I work on it as a means of procrastinating on something else :P > Personally, I generally trust you to 1) make enough research and/or > gather enough material 2) take informed decisions that do justice to the > community's ethos (after, sometimes, a certain amount of arguing :-)). > If my criticism came accross as criticizing your competence, then I'm > sorry: that was not my intent. I suspect lack of sleep was also a factor in my poor reaction ;) PJE's articulation of the potential for misinterpretation as the equivalent of VB's "on error resume next" for users learning by example is an interesting one. I'm actually tempted to move this particular feature out to contextlib2 for the time being (similar to what I did with ExitStack and some other experimental features that never made it back into the stdlib version). A pull request at https://bitbucket.org/ncoghlan/contextlib2 to bring it up to date with the latest contextlib code would likely help push me in that direction (it's not quite a matter of simply copying the current stdlib code, since contextlib2 still supports some of those experimental APIs I didn't end up merging). Cheers, Nick. > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Oct 14 00:21:56 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 14 Oct 2013 08:21:56 +1000 Subject: [Python-Dev] Submitting PEP 453 (pip bootstrapping) for pronouncement In-Reply-To: References: Message-ID: On 14 Oct 2013 04:26, "Paul Moore" wrote: > > On 13 October 2013 13:43, Nick Coghlan wrote: > > Accordingly, in addition to adding the option to extract and install ``pip`` > > during installation, this PEP proposes that the Windows installer (and > > ``sysconfig``) in Python 3.4 and later be updated to: > > > > - install scripts to PythonXY\bin rather than PythonXY\Tools\Scripts > > - add PythonXY\bin to the Windows PATH (in addition to PythonXY) when the > > PATH modification option is enabled during installation > > Then implication of this is that venv (and virtualenv, but that's less > relevant to python-dev) should be updated to use a "bin" directory on > Windows rather than the current "Scripts" directory. This would only > be for 3.4, which means that virtualenv will need to make the > directory name conditional on version, I guess. The sysconfig directory scheme will change accordingly, so it may be possible to leverage that by basing the virtualenv layout on the default scheme. Cheers, Nick. > > Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Mon Oct 14 01:20:13 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sun, 13 Oct 2013 16:20:13 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <525B2AAD.80100@g.nevcal.com> On 10/12/2013 11:57 PM, Nick Coghlan wrote: > > For the record, this thread did prompt me to consider the new > construct anew, but on reflection, I still consider it a reasonable > addition to contextlib. > > It substantially improves the simple cases it is intended to help > with, and, if anything, makes overly broad exception suppression > *more* obviously dubious (because the name of the construct doesn't > match the consequences for multi-line suites). > For the record, the logic in the last paragraph is the most dubious thing I've ever seen you post. -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Mon Oct 14 11:47:03 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 14 Oct 2013 12:47:03 +0300 Subject: [Python-Dev] Right place for PBKDF2 wrapper In-Reply-To: <525A7874.4080408@python.org> References: <525A7874.4080408@python.org> Message-ID: 13.10.13 13:39, Christian Heimes ???????(??): > Am 13.10.2013 08:32, schrieb Nick Coghlan: >> +1 to hashlib from me (especially since we used that as the best >> available home for compare_digest). > > I'm afraid your memory doesn't serve you well. :( > compare_digest is implemented in _operator.c. Its official home is > hmac.compare_digest. Could we move compare_digest() out of _operator.c? It makes Python implementation of the operator module incomplete. From solipsis at pitrou.net Mon Oct 14 12:07:31 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 14 Oct 2013 12:07:31 +0200 Subject: [Python-Dev] Right place for PBKDF2 wrapper References: <525A7874.4080408@python.org> Message-ID: <20131014120731.302c7e98@pitrou.net> Le Mon, 14 Oct 2013 12:47:03 +0300, Serhiy Storchaka a ?crit : > 13.10.13 13:39, Christian Heimes ???????(??): > > Am 13.10.2013 08:32, schrieb Nick Coghlan: > >> +1 to hashlib from me (especially since we used that as the best > >> available home for compare_digest). > > > > I'm afraid your memory doesn't serve you well. :( > > compare_digest is implemented in _operator.c. Its official home is > > hmac.compare_digest. > > Could we move compare_digest() out of _operator.c? It makes Python > implementation of the operator module incomplete. Or we could simply write a reference Python implementation :-) Regards Antoine. From storchaka at gmail.com Mon Oct 14 13:47:08 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 14 Oct 2013 14:47:08 +0300 Subject: [Python-Dev] Right place for PBKDF2 wrapper In-Reply-To: <20131014120731.302c7e98@pitrou.net> References: <525A7874.4080408@python.org> <20131014120731.302c7e98@pitrou.net> Message-ID: 14.10.13 13:07, Antoine Pitrou ???????(??): > Le Mon, 14 Oct 2013 12:47:03 +0300, > Serhiy Storchaka a ?crit : >> 13.10.13 13:39, Christian Heimes ???????(??): >>> Am 13.10.2013 08:32, schrieb Nick Coghlan: >>>> +1 to hashlib from me (especially since we used that as the best >>>> available home for compare_digest). >>> >>> I'm afraid your memory doesn't serve you well. :( >>> compare_digest is implemented in _operator.c. Its official home is >>> hmac.compare_digest. >> >> Could we move compare_digest() out of _operator.c? It makes Python >> implementation of the operator module incomplete. > > Or we could simply write a reference Python implementation :-) It was discussed when compare_digest() was introdused. Looks as pure Python implementation is not possible (without destroying the purpose of compare_digest()). Even with C implementation we should be very careful. From fijall at gmail.com Mon Oct 14 13:49:34 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 14 Oct 2013 13:49:34 +0200 Subject: [Python-Dev] GC pauses in CPython Message-ID: HI I'm working on an incremental GC for PyPy. How do I measure GC pauses in CPython? (that is, the circular reference searching stuff) Cheers, fijal From tds333+pydev at gmail.com Mon Oct 14 13:51:50 2013 From: tds333+pydev at gmail.com (Wolfgang Langner) Date: Mon, 14 Oct 2013 13:51:50 +0200 Subject: [Python-Dev] Make str/bytes hash algorithm pluggable? In-Reply-To: References: Message-ID: <525BDAD6.9070206@gmail.com> Hi, I have found a link to a speed comparison of hash functions: http://code.google.com/p/xxhash/ Regards, Wolfgang From solipsis at pitrou.net Mon Oct 14 13:59:11 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 14 Oct 2013 13:59:11 +0200 Subject: [Python-Dev] GC pauses in CPython References: Message-ID: <20131014135911.10b17ed5@pitrou.net> Le Mon, 14 Oct 2013 13:49:34 +0200, Maciej Fijalkowski a ?crit : > HI > > I'm working on an incremental GC for PyPy. How do I measure GC pauses > in CPython? (that is, the circular reference searching stuff) timeit gc.collect()? From martin at v.loewis.de Mon Oct 14 14:11:07 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 14 Oct 2013 14:11:07 +0200 Subject: [Python-Dev] GC pauses in CPython In-Reply-To: References: Message-ID: <525BDF5B.1030309@v.loewis.de> Am 14.10.13 13:49, schrieb Maciej Fijalkowski: > I'm working on an incremental GC for PyPy. How do I measure GC pauses > in CPython? (that is, the circular reference searching stuff) I would instrument the interpreter. The tricky part may be to subtract the time for any resulting finalization (and whether or not to subtract that at all). Regards, Martin From alex.gaynor at gmail.com Mon Oct 14 14:22:55 2013 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 14 Oct 2013 12:22:55 +0000 (UTC) Subject: [Python-Dev] GC pauses in CPython References: Message-ID: Maciej Fijalkowski gmail.com> writes: > > HI > > I'm working on an incremental GC for PyPy. How do I measure GC pauses > in CPython? (that is, the circular reference searching stuff) > > Cheers, > fijal > For what it's worth I threw together some code that might be helpful: http://bpaste.net/show/140334/ if someone was interested it might be a cool idea to properly organize this up and find a place to expose VM statistics like this. It'd also probably useful to use sane units, and maybe (it's unclear to me) exclude some amount of finalizations (Ideally I think you'd ignore use __del__ functions, but keep the bits of C code that decref other things and actually call free()). Alex From fijall at gmail.com Mon Oct 14 14:25:18 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 14 Oct 2013 14:25:18 +0200 Subject: [Python-Dev] GC pauses in CPython In-Reply-To: <525BDF5B.1030309@v.loewis.de> References: <525BDF5B.1030309@v.loewis.de> Message-ID: On Mon, Oct 14, 2013 at 2:11 PM, "Martin v. L?wis" wrote: > Am 14.10.13 13:49, schrieb Maciej Fijalkowski: >> I'm working on an incremental GC for PyPy. How do I measure GC pauses >> in CPython? (that is, the circular reference searching stuff) > > I would instrument the interpreter. The tricky part may be to subtract > the time for any resulting finalization (and whether or not to subtract > that at all). > > Regards, > Martin > seems that: gc.set_debug(gc.DEBUG_STATS) does the job obviously timing gc.collect is quite the opposite of what I'm trying to achieve (see how cpython deals with when to collect) I think including finalizer time is fine From solipsis at pitrou.net Mon Oct 14 14:34:57 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 14 Oct 2013 14:34:57 +0200 Subject: [Python-Dev] GC pauses in CPython References: <525BDF5B.1030309@v.loewis.de> Message-ID: <20131014143457.3ba44713@pitrou.net> Le Mon, 14 Oct 2013 14:25:18 +0200, Maciej Fijalkowski a ?crit : > On Mon, Oct 14, 2013 at 2:11 PM, "Martin v. L?wis" > wrote: > > Am 14.10.13 13:49, schrieb Maciej Fijalkowski: > >> I'm working on an incremental GC for PyPy. How do I measure GC > >> pauses in CPython? (that is, the circular reference searching > >> stuff) > > > > I would instrument the interpreter. The tricky part may be to > > subtract the time for any resulting finalization (and whether or > > not to subtract that at all). > > > > Regards, > > Martin > > > > seems that: > > gc.set_debug(gc.DEBUG_STATS) does the job > > obviously timing gc.collect is quite the opposite of what I'm trying > to achieve (see how cpython deals with when to collect) Why "the opposite"? The time of a GC pause in CPython is the time executing gc.collect(). And most of the time gc.collect() will spend its time walking live objects, not dead ones, so executing it multiple times should only introduce a minor bias. Regards Antoine. From fijall at gmail.com Mon Oct 14 14:38:44 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 14 Oct 2013 14:38:44 +0200 Subject: [Python-Dev] GC pauses in CPython In-Reply-To: <20131014143457.3ba44713@pitrou.net> References: <525BDF5B.1030309@v.loewis.de> <20131014143457.3ba44713@pitrou.net> Message-ID: On Mon, Oct 14, 2013 at 2:34 PM, Antoine Pitrou wrote: > Le Mon, 14 Oct 2013 14:25:18 +0200, > Maciej Fijalkowski a ?crit : >> On Mon, Oct 14, 2013 at 2:11 PM, "Martin v. L?wis" >> wrote: >> > Am 14.10.13 13:49, schrieb Maciej Fijalkowski: >> >> I'm working on an incremental GC for PyPy. How do I measure GC >> >> pauses in CPython? (that is, the circular reference searching >> >> stuff) >> > >> > I would instrument the interpreter. The tricky part may be to >> > subtract the time for any resulting finalization (and whether or >> > not to subtract that at all). >> > >> > Regards, >> > Martin >> > >> >> seems that: >> >> gc.set_debug(gc.DEBUG_STATS) does the job >> >> obviously timing gc.collect is quite the opposite of what I'm trying >> to achieve (see how cpython deals with when to collect) > > Why "the opposite"? The time of a GC pause in CPython is the time > executing gc.collect(). And most of the time gc.collect() will spend > its time walking live objects, not dead ones, so executing it multiple > times should only introduce a minor bias. well, I'm not trying to measure it at *one point in time* but to provide some statistics. invoking it kills the point quite a bit. From solipsis at pitrou.net Mon Oct 14 14:42:21 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 14 Oct 2013 14:42:21 +0200 Subject: [Python-Dev] GC pauses in CPython References: <525BDF5B.1030309@v.loewis.de> <20131014143457.3ba44713@pitrou.net> Message-ID: <20131014144221.238785fa@pitrou.net> Le Mon, 14 Oct 2013 14:38:44 +0200, Maciej Fijalkowski a ?crit : > On Mon, Oct 14, 2013 at 2:34 PM, Antoine Pitrou > wrote: > > Le Mon, 14 Oct 2013 14:25:18 +0200, > > Maciej Fijalkowski a ?crit : > >> On Mon, Oct 14, 2013 at 2:11 PM, "Martin v. L?wis" > >> wrote: > >> > Am 14.10.13 13:49, schrieb Maciej Fijalkowski: > >> >> I'm working on an incremental GC for PyPy. How do I measure GC > >> >> pauses in CPython? (that is, the circular reference searching > >> >> stuff) > >> > > >> > I would instrument the interpreter. The tricky part may be to > >> > subtract the time for any resulting finalization (and whether or > >> > not to subtract that at all). > >> > > >> > Regards, > >> > Martin > >> > > >> > >> seems that: > >> > >> gc.set_debug(gc.DEBUG_STATS) does the job > >> > >> obviously timing gc.collect is quite the opposite of what I'm > >> trying to achieve (see how cpython deals with when to collect) > > > > Why "the opposite"? The time of a GC pause in CPython is the time > > executing gc.collect(). And most of the time gc.collect() will spend > > its time walking live objects, not dead ones, so executing it > > multiple times should only introduce a minor bias. > > well, I'm not trying to measure it at *one point in time* but to > provide some statistics. invoking it kills the point quite a bit. Ah, then you can plug into the new "callbacks" property: http://docs.python.org/3.3/library/gc.html#gc.callbacks Regards Antoine. From fijall at gmail.com Mon Oct 14 14:44:07 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 14 Oct 2013 14:44:07 +0200 Subject: [Python-Dev] GC pauses in CPython In-Reply-To: <20131014144221.238785fa@pitrou.net> References: <525BDF5B.1030309@v.loewis.de> <20131014143457.3ba44713@pitrou.net> <20131014144221.238785fa@pitrou.net> Message-ID: On Mon, Oct 14, 2013 at 2:42 PM, Antoine Pitrou wrote: > Le Mon, 14 Oct 2013 14:38:44 +0200, > Maciej Fijalkowski a ?crit : > >> On Mon, Oct 14, 2013 at 2:34 PM, Antoine Pitrou >> wrote: >> > Le Mon, 14 Oct 2013 14:25:18 +0200, >> > Maciej Fijalkowski a ?crit : >> >> On Mon, Oct 14, 2013 at 2:11 PM, "Martin v. L?wis" >> >> wrote: >> >> > Am 14.10.13 13:49, schrieb Maciej Fijalkowski: >> >> >> I'm working on an incremental GC for PyPy. How do I measure GC >> >> >> pauses in CPython? (that is, the circular reference searching >> >> >> stuff) >> >> > >> >> > I would instrument the interpreter. The tricky part may be to >> >> > subtract the time for any resulting finalization (and whether or >> >> > not to subtract that at all). >> >> > >> >> > Regards, >> >> > Martin >> >> > >> >> >> >> seems that: >> >> >> >> gc.set_debug(gc.DEBUG_STATS) does the job >> >> >> >> obviously timing gc.collect is quite the opposite of what I'm >> >> trying to achieve (see how cpython deals with when to collect) >> > >> > Why "the opposite"? The time of a GC pause in CPython is the time >> > executing gc.collect(). And most of the time gc.collect() will spend >> > its time walking live objects, not dead ones, so executing it >> > multiple times should only introduce a minor bias. >> >> well, I'm not trying to measure it at *one point in time* but to >> provide some statistics. invoking it kills the point quite a bit. > > Ah, then you can plug into the new "callbacks" property: > http://docs.python.org/3.3/library/gc.html#gc.callbacks > > Regards > > Antoine. New in version 3.3. no, thanks From brett at python.org Mon Oct 14 17:20:56 2013 From: brett at python.org (Brett Cannon) Date: Mon, 14 Oct 2013 11:20:56 -0400 Subject: [Python-Dev] Change PEP 399 import recommendation In-Reply-To: References: <20131011122428.GA22353@sleipnir.bytereef.org> <20131011145156.3d441797@pitrou.net> <20131011150135.GA23302@sleipnir.bytereef.org> <20131011172034.639648ff@pitrou.net> <20131011194100.GA25838@sleipnir.bytereef.org> <20131012134146.GA19814@sleipnir.bytereef.org> <20131012144602.CDBDE25005F@webabinitio.net> <52597F0E.5000007@python.org> <20131012190243.404e477d@fsol> <5259C09B.8050001@python.org> <20131012234004.55c8ca1c@fsol> Message-ID: On Sun, Oct 13, 2013 at 3:08 PM, Victor Stinner wrote: > Le 13 oct. 2013 10:19, "Stefan Behnel" a ?crit : > > > I agree. I find it much easier to read a plain and obvious > > > > try: > > from _cmodule import * > > except ImportError: > > from _pymodule import * > > > > in a facade module ... > > I miss maybe something. I don't understand why you would like to use the > Python implementation if a C implementation is available and probably > (much) faster. > > Maybe developers of PyPy, IronPython and Jython appreciate our effort to > provide a Python implementation which is up to date and well tested. But > why should we pay a price (bad performance) if the module is not (never? > who uses _pyio?) used in CPython. If another Python uses it, the file can > be renamed (ex: _pyio.py becomes simply io.py). > Because doing a rename like that is a deviation between implementations that saves you nothing more than a file with four lines. And "bad performance" in this instance is a single import * which in the grand scheme of things is cheap, especially in this instance where the module is not on the critical path (and even if it was I would argue it isn't worth worrying about). > IMO if the C module is complete (has no extra functions in the "facade" > module), it should get the main name (ex: decimal, not _decimal). > Why? It's not the reference implementation as that's the pure Python version, but your suggested naming scheme purports the opposite. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at fidoman.ru Mon Oct 14 22:55:05 2013 From: python at fidoman.ru (Sergey Dorofeev) Date: Tue, 15 Oct 2013 00:55:05 +0400 Subject: [Python-Dev] non-US zip archives support in zipfile.py Message-ID: Hello, I'd like to submit patch to support zip archives created on systems that use non-US codepage (e.g. russian CP866). Codepage would be specified in additional parameter of ZipFile constructor, named "codepage". If it is not specified, old behavior is preserved (use CP437). --- zipfile.py-orig 2013-09-18 16:45:56.000000000 +0400 +++ zipfile.py 2013-10-15 00:24:06.105157572 +0400 @@ -885,7 +885,7 @@ fp = None # Set here since __del__ checks it _windows_illegal_name_trans_table = None - def __init__(self, file, mode="r", compression=ZIP_STORED, allowZip64=False): + def __init__(self, file, mode="r", compression=ZIP_STORED, allowZip64=False, codepage='cp437'): """Open the ZIP file with mode read "r", write "w" or append "a".""" if mode not in ("r", "w", "a"): raise RuntimeError('ZipFile() requires mode "r", "w", or "a"') @@ -901,6 +901,7 @@ self.mode = key = mode.replace('b', '')[0] self.pwd = None self._comment = b'' + self.codepage = codepage # Check if we were passed a file-like object if isinstance(file, str): @@ -1002,7 +1003,7 @@ filename = filename.decode('utf-8') else: # Historical ZIP filename encoding - filename = filename.decode('cp437') + filename = filename.decode(self.codepage) # Create ZipInfo instance to store file information x = ZipInfo(filename) x.extra = fp.read(centdir[_CD_EXTRA_FIELD_LENGTH]) @@ -1157,7 +1158,7 @@ # UTF-8 filename fname_str = fname.decode("utf-8") else: - fname_str = fname.decode("cp437") + fname_str = fname.decode(self.codepage) if fname_str != zinfo.orig_filename: raise BadZipFile( From solipsis at pitrou.net Mon Oct 14 23:28:03 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 14 Oct 2013 23:28:03 +0200 Subject: [Python-Dev] non-US zip archives support in zipfile.py References: Message-ID: <20131014232803.0c3b7c2a@fsol> Hello, On Tue, 15 Oct 2013 00:55:05 +0400 Sergey Dorofeev wrote: > Hello, > > I'd like to submit patch to support zip archives created on systems that > use non-US codepage (e.g. russian CP866). > Codepage would be specified in additional parameter of ZipFile > constructor, named "codepage". > If it is not specified, old behavior is preserved (use CP437). Thanks for contributing! Patches posted on the mailing-list tend to get lost. You should open an issue on the bug tracker and post your patch there: http://bugs.python.org/ Also, the devguide can help you know how to make your patch more complete (for example by adding unit tests): http://docs.python.org/devguide/ Regards Antoine. From victor.stinner at gmail.com Tue Oct 15 00:13:06 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 15 Oct 2013 00:13:06 +0200 Subject: [Python-Dev] non-US zip archives support in zipfile.py In-Reply-To: References: Message-ID: I opened an issue proposing exactly the same change, but I didn't provide a patch. Thanks for working on the issue. (Sorry I don't know the issue number. Search for "zipfile encoding".) Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Oct 15 00:51:42 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 14 Oct 2013 18:51:42 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011175656.CB7AC2509A9@webabinitio.net> <87vc13gq76.fsf@uwakimon.sk.tsukuba.ac.jp> <5258CA80.8040506@stoneleaf.us> <5259B404.1000809@stoneleaf.us> <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> Message-ID: <20131014185142.709e897d@anarchist> On Oct 12, 2013, at 03:27 PM, Raymond Hettinger wrote: >The module maintainer (Nick) approved the name change from his original >preference for ignored(). That should had been the end of it. As a general plea, please be sure to record the rationale and decisions in the appropriate place, such as the tracker and/or commit message. I think it's fine for such minor decisions to be made in high bandwidth forums such as IRC, it's also incumbent on the participants to be mindful of the larger community of core developers and users. Not doing so makes our process less transparent, puts legitimacy at risk, and alienates users and developers who may not be privileged to those out-of-band discussions. Cheers, -Barry From alexander.belopolsky at gmail.com Tue Oct 15 01:24:53 2013 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 14 Oct 2013 19:24:53 -0400 Subject: [Python-Dev] non-US zip archives support in zipfile.py In-Reply-To: References: Message-ID: On Mon, Oct 14, 2013 at 6:13 PM, Victor Stinner wrote: > I opened an issue proposing exactly the same change, but I didn't provide a > patch. I found this: http://bugs.python.org/issue10614 but it has (seemingly incorrect) patch. Also related: http://bugs.python.org/issue10972 Victor - is any of these what you had in mind? From ncoghlan at gmail.com Tue Oct 15 01:37:57 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Oct 2013 09:37:57 +1000 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131014185142.709e897d@anarchist> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> <20131011175656.CB7AC2509A9@webabinitio.net> <87vc13gq76.fsf@uwakimon.sk.tsukuba.ac.jp> <5258CA80.8040506@stoneleaf.us> <5259B404.1000809@stoneleaf.us> <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131014185142.709e897d@anarchist> Message-ID: On 15 Oct 2013 08:54, "Barry Warsaw" wrote: > > On Oct 12, 2013, at 03:27 PM, Raymond Hettinger wrote: > > >The module maintainer (Nick) approved the name change from his original > >preference for ignored(). That should had been the end of it. > > As a general plea, please be sure to record the rationale and decisions in the > appropriate place, such as the tracker and/or commit message. I think it's > fine for such minor decisions to be made in high bandwidth forums such as IRC, > it's also incumbent on the participants to be mindful of the larger community > of core developers and users. Not doing so makes our process less > transparent, puts legitimacy at risk, and alienates users and developers who > may not be privileged to those out-of-band discussions. Agreed, although in this case there wasn't much additional rationale beyond "Raymond and I decided we preferred the shorter spelling, so we changed it" :) I figured it was a minor enough change that the commit message was adequate documentation, but I acknowledge it would have been better to create a new issue with its own NEWS entry to more explicitly track the change. Cheers, Nick. > > Cheers, > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Oct 15 02:18:56 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Oct 2013 10:18:56 +1000 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <525B2AAD.80100@g.nevcal.com> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> Message-ID: On 14 Oct 2013 09:28, "Glenn Linderman" wrote: > > On 10/12/2013 11:57 PM, Nick Coghlan wrote: >> >> For the record, this thread did prompt me to consider the new construct anew, but on reflection, I still consider it a reasonable addition to contextlib. >> >> It substantially improves the simple cases it is intended to help with, and, if anything, makes overly broad exception suppression *more* obviously dubious (because the name of the construct doesn't match the consequences for multi-line suites). > > For the record, the logic in the last paragraph is the most dubious thing I've ever seen you post. I didn't articulate the point very well. The reason I originally approved the change (and the reason I have kept it despite the objections raised) is because it allows correct-but-ugly code like: try: os.unlink(fname) except FileNotFoundError: pass To be rewritten as the significantly more elegant: with ignore(FileNotFoundError): os.unlink(fname) which is much closer to the version that expects to have complete control over the file's lifecycle: os.unlink(fname) The more explicit statement of intent can also be exploited in static code analysis (such as linting), although I don't expect anyone to actually do that any time soon. This benefit when used correctly then needs to be weighed against the risk of it being used *incorrectly*, especially by people that expect it to work like VB's "on error resume next" rather than by suppressing the exception and resuming execution after the with statement. The problem of overbroad exception handling isn't a new one, though, and I don't believe this feature makes that problem *worse*, and, by making it simpler and cleaner to suppress exceptions from a single statement, may make it better. For example, this code is wrong: try: os.unlink(fname) os.unlink(fname2) except FileNotFoundError: pass You could fix this particular case with a loop (since the two commands differ by a single value), but that option isn't always going to be available. Requiring 3 extra lines per command is a significant factor pushing against correctly suppressing exceptions from cleanup code that may fail for legitimate reasons. The new CM handles these cases a little more neatly, in a way that doesn't require the use of a loop: with ignore(FileNotFoundError): os.unlink(fname) with ignore(FileNotFoundError): os.unlink(fname2) What the new CM *doesn't* handle nicely is multi-statement suites, and I'm OK with that. Handling those is not what *this* CM is for, and I believe trying to generalise it would harm the primary intended use case. I now *do* plan to experiment with a more general with statement termination construct in contextlib2 before Python 3.5, but that's orthogonal to *this* change. Regards, Nick. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Tue Oct 15 02:52:18 2013 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 14 Oct 2013 20:52:18 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> Message-ID: On Mon, Oct 14, 2013 at 8:18 PM, Nick Coghlan wrote: > .. especially by people that expect it to work like VB's "on error resume next" > rather than by suppressing the exception and resuming execution after the > with statement. "It is practically impossible to teach good programming to students that have had a prior exposure to BASIC: as potential programmers they are mentally mutilated beyond hope of regeneration." -- Edsger W.Dijkstra, 18 June 1975, "How do we tell truths that might hurt?" http://www.cs.virginia.edu/~cs655/readings/ewd498.html From z at etiol.net Tue Oct 15 03:04:45 2013 From: z at etiol.net (Zero Piraeus) Date: Mon, 14 Oct 2013 22:04:45 -0300 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> Message-ID: <20131015010445.GA13488@piedra> : On Tue, Oct 15, 2013 at 10:18:56AM +1000, Nick Coghlan wrote: > I didn't articulate the point very well. The reason I originally > approved the change (and the reason I have kept it despite the > objections raised) is because it allows correct-but-ugly code like: > > try: > os.unlink(fname) > except FileNotFoundError: > pass > > To be rewritten as the significantly more elegant: > > with ignore(FileNotFoundError): > os.unlink(fname) I thought that was already clear, to be honest. As an aside, calling the try/except construct above "ugly" seems a stretch to me - it's a little verbose if you're going to be using it a lot, but presumably in that case you'd just wrap it in a safe_unlink() convenience function if it became annoying. > This benefit when used correctly then needs to be weighed against the > risk of it being used *incorrectly*, especially by people that expect > it to work like VB's "on error resume next" rather than by suppressing > the exception and resuming execution after the with statement. > > The problem of overbroad exception handling isn't a new one, though, > and I don't believe this feature makes that problem *worse*, and, by > making it simpler and cleaner to suppress exceptions from a single > statement, may make it better. For example, this code is wrong: > > try: > os.unlink(fname) > os.unlink(fname2) > except FileNotFoundError: > pass Yes, it's bad code, but it's more intuitively obvious what it does (and therefore that it's badly written) than the equivalent with ignore(FileNotFoundError): os.unlink(fname) os.unlink(fname2) ... as others have pointed out. The above suffers from the same (famously solved by Python) problem as C code with inconsistent braces and indentation: there's a disconnect between what it looks like and what it does. I think one problem is that ignore() doesn't actually do what its name suggests - the flow of execution is changed if an exception occurs, which by definition means the exception isn't being ignored. Maybe it would be better to call it silence()? That's still not perfect IMO, but it's closer to an accurate description of what's going on. -[]z. -- Zero Piraeus: obiter dictum http://etiol.net/pubkey.asc From charleshixsn at earthlink.net Tue Oct 15 04:57:24 2013 From: charleshixsn at earthlink.net (Charles Hixson) Date: Mon, 14 Oct 2013 19:57:24 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131015010445.GA13488@piedra> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <20131015010445.GA13488@piedra> Message-ID: <525CAF14.5080502@earthlink.net> On 10/14/2013 06:04 PM, Zero Piraeus wrote: > : > > On Tue, Oct 15, 2013 at 10:18:56AM +1000, Nick Coghlan wrote: >> I didn't articulate the point very well. The reason I originally >> approved the change (and the reason I have kept it despite the >> objections raised) is because it allows correct-but-ugly code like: >> >> try: >> os.unlink(fname) >> except FileNotFoundError: >> pass >> >> To be rewritten as the significantly more elegant: >> >> with ignore(FileNotFoundError): >> os.unlink(fname) > I thought that was already clear, to be honest. As an aside, calling the > try/except construct above "ugly" seems a stretch to me - it's a little > verbose if you're going to be using it a lot, but presumably in that > case you'd just wrap it in a safe_unlink() convenience function if it > became annoying. > >> This benefit when used correctly then needs to be weighed against the >> risk of it being used *incorrectly*, especially by people that expect >> it to work like VB's "on error resume next" rather than by suppressing >> the exception and resuming execution after the with statement. >> >> The problem of overbroad exception handling isn't a new one, though, >> and I don't believe this feature makes that problem *worse*, and, by >> making it simpler and cleaner to suppress exceptions from a single >> statement, may make it better. For example, this code is wrong: >> >> try: >> os.unlink(fname) >> os.unlink(fname2) >> except FileNotFoundError: >> pass > Yes, it's bad code, but it's more intuitively obvious what it does (and > therefore that it's badly written) than the equivalent > > with ignore(FileNotFoundError): > os.unlink(fname) > os.unlink(fname2) > > ... as others have pointed out. The above suffers from the same > (famously solved by Python) problem as C code with inconsistent braces > and indentation: there's a disconnect between what it looks like and > what it does. > > I think one problem is that ignore() doesn't actually do what its name > suggests - the flow of execution is changed if an exception occurs, > which by definition means the exception isn't being ignored. > > Maybe it would be better to call it silence()? That's still not perfect > IMO, but it's closer to an accurate description of what's going on. > > -[]z. > but is: with ignore(FileNotFoundError): os.unlink(fname) os.unlink(fname2) equivalent to: try: os.unlink(fname) os.unlink(fname2) except FileNotFoundError: pass or to: try: os.unlink(fname) except FileNotFoundError: pass try: os.unlink(fname2) except FileNotFoundError: pass If the documentation isn't clear, I'd read it the second way. (And I'd *WANT* it to be the second way.) -- Charles Hixson From v+python at g.nevcal.com Tue Oct 15 05:12:42 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 14 Oct 2013 20:12:42 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> Message-ID: <525CB2AA.3010409@g.nevcal.com> On 10/14/2013 5:18 PM, Nick Coghlan wrote: > What the new CM *doesn't* handle nicely is multi-statement suites, and > I'm OK with that. Yeah, that is the dubious part. You can't even convince python-dev that saving 2 lines to ignore an exception in one line of code is a good idea, at the cost of the potential for misunderstanding and misuse; I can't conceive of how verbose the documentation would have to be to explain it to people that don't know python well... especially to the cut-n-pasters that don't read the documentation anyway (in some sense they don't matter, until they discover the bug, and then try to read the documentation). I suspect that there is no logic that will quiet down this thread with the current implementation and decision, now that it has been exposed for what it is ... maybe the only way would be to appeal to Guido to proclaim that this thread should stop. Of course, if the name were changed to be accurate, or the feature made less prone to misuse, then maybe it would terminate. I've got an extra can of "break_out_if" paint here... -------------- next part -------------- An HTML attachment was scrubbed... URL: From devel at baptiste-carvello.net Tue Oct 15 09:14:43 2013 From: devel at baptiste-carvello.net (Baptiste Carvello) Date: Tue, 15 Oct 2013 09:14:43 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <525CB2AA.3010409@g.nevcal.com> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> Message-ID: Le 15/10/2013 05:12, Glenn Linderman a ?crit : > > I've got an extra can of "break_out_if" paint here... > another shed color: with contextlib.except_pass(FileNotFoundError): os.unlink(fname) explicit and hopefully not too ugly From victor.stinner at gmail.com Tue Oct 15 10:20:03 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 15 Oct 2013 10:20:03 +0200 Subject: [Python-Dev] non-US zip archives support in zipfile.py In-Reply-To: References: Message-ID: Both issues are related and should be fixed in the *same* commit. Python should provide an option to encode explicitly filenames to UTF-8. Or maybe better: encode to UTF-8 by default, with an option to encode to cp437, and another to encode to an arbitrary encoding. Victor 2013/10/15 Alexander Belopolsky : > On Mon, Oct 14, 2013 at 6:13 PM, Victor Stinner > wrote: >> I opened an issue proposing exactly the same change, but I didn't provide a >> patch. > > I found this: > > http://bugs.python.org/issue10614 > > but it has (seemingly incorrect) patch. > > Also related: > > http://bugs.python.org/issue10972 > > Victor - is any of these what you had in mind? From dholth at gmail.com Tue Oct 15 10:53:42 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 15 Oct 2013 10:53:42 +0200 Subject: [Python-Dev] non-US zip archives support in zipfile.py In-Reply-To: References: Message-ID: FYI zipfile does do UTF-8 (http://hg.python.org/cpython/file/d7ebe03fa752/Lib/zipfile.py#l368). Only the non-Unicode encodings might need some help. I like that the patch is only concerned with decoding. Is it necessary to support writing non-UTF8 encodings? On Tue, Oct 15, 2013 at 10:20 AM, Victor Stinner wrote: > Both issues are related and should be fixed in the *same* commit. > Python should provide an option to encode explicitly filenames to > UTF-8. Or maybe better: encode to UTF-8 by default, with an option to > encode to cp437, and another to encode to an arbitrary encoding. > > Victor > > 2013/10/15 Alexander Belopolsky : >> On Mon, Oct 14, 2013 at 6:13 PM, Victor Stinner >> wrote: >>> I opened an issue proposing exactly the same change, but I didn't provide a >>> patch. >> >> I found this: >> >> http://bugs.python.org/issue10614 >> >> but it has (seemingly incorrect) patch. >> >> Also related: >> >> http://bugs.python.org/issue10972 >> >> Victor - is any of these what you had in mind? > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com From martin at v.loewis.de Tue Oct 15 12:22:29 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 15 Oct 2013 12:22:29 +0200 Subject: [Python-Dev] non-US zip archives support in zipfile.py In-Reply-To: References: Message-ID: <525D1765.6040103@v.loewis.de> Am 15.10.13 10:53, schrieb Daniel Holth: > FYI zipfile does do UTF-8 > (http://hg.python.org/cpython/file/d7ebe03fa752/Lib/zipfile.py#l368). > Only the non-Unicode encodings might need some help. I like that the > patch is only concerned with decoding. Is it necessary to support > writing non-UTF8 encodings? For purity, no. For practicality, perhaps. It might be that some tools don't know how to unpack files marked as UTF-8, and always default to the system code page. OTOH, before accepting such a change, I'd like to know what actual existing tool would require such a hack, and a confirmation that this very tool indeed does not support UTF-8. Regards, Martin From steve at pearwood.info Tue Oct 15 14:02:10 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 15 Oct 2013 23:02:10 +1100 Subject: [Python-Dev] non-US zip archives support in zipfile.py In-Reply-To: References: Message-ID: <20131015120209.GA7989@ando> On Tue, Oct 15, 2013 at 10:20:03AM +0200, Victor Stinner wrote: > Both issues are related and should be fixed in the *same* commit. > Python should provide an option to encode explicitly filenames to > UTF-8. Or maybe better: encode to UTF-8 by default, with an option to > encode to cp437, and another to encode to an arbitrary encoding. Why single out cp437 for special treatment? Surely it's enough to use UTF-8 by default, or specify an arbitrary encoding. -- Steven From ncoghlan at gmail.com Tue Oct 15 14:05:41 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Oct 2013 22:05:41 +1000 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <525CB2AA.3010409@g.nevcal.com> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> Message-ID: On 15 October 2013 13:12, Glenn Linderman wrote: > Of course, if the name were changed to be accurate, or the feature made less > prone to misuse, then maybe it would terminate. I've come to the conclusion it makes sense to rename it from ignore to suppress, since that's the term already used for this operation in the rest of the context management docs: http://bugs.python.org/issue19266 Given the number of people vehemently opposed to the old name, I'm sure there will be plenty of volunteers to create a patch while I do some more work on PEP 453 :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Tue Oct 15 14:27:19 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 15 Oct 2013 14:27:19 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> Message-ID: <20131015142719.360e30a8@pitrou.net> Le Tue, 15 Oct 2013 22:05:41 +1000, Nick Coghlan a ?crit : > On 15 October 2013 13:12, Glenn Linderman > wrote: > > Of course, if the name were changed to be accurate, or the feature > > made less prone to misuse, then maybe it would terminate. > > I've come to the conclusion it makes sense to rename it from ignore to > suppress, since that's the term already used for this operation in the > rest of the context management docs: http://bugs.python.org/issue19266 "suppress" and/or "trap" (which I believe is a common name for such operation). Regards Antoine. From dholth at gmail.com Tue Oct 15 14:49:11 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 15 Oct 2013 08:49:11 -0400 Subject: [Python-Dev] non-US zip archives support in zipfile.py In-Reply-To: <20131015120209.GA7989@ando> References: <20131015120209.GA7989@ando> Message-ID: On Tue, Oct 15, 2013 at 8:02 AM, Steven D'Aprano wrote: > On Tue, Oct 15, 2013 at 10:20:03AM +0200, Victor Stinner wrote: >> Both issues are related and should be fixed in the *same* commit. >> Python should provide an option to encode explicitly filenames to >> UTF-8. Or maybe better: encode to UTF-8 by default, with an option to >> encode to cp437, and another to encode to an arbitrary encoding. > > Why single out cp437 for special treatment? Surely it's enough to use > UTF-8 by default, or specify an arbitrary encoding. It is part of the ZIP specification. CP437 or UTF-8 are the two official choices, but other encodings happen on Russian, Japanese systems. From larry at hastings.org Tue Oct 15 14:55:47 2013 From: larry at hastings.org (Larry Hastings) Date: Tue, 15 Oct 2013 13:55:47 +0100 Subject: [Python-Dev] Checking in Argument Clinic early? Message-ID: <525D3B53.6000508@hastings.org> I really want to get Argument Clinic in to Python 3.4. Currently the following is true: * It has an implementation I've been working on for more than a year. * I assert that it's a reasonable approach and the APIs are ready for public consumption. * The syntax for the DSL is the one proposed by Guido at PyCon US 2013. * So far It has little in the way of documentation apart from the PEP. * The PEP is out of date and not ready for pronouncement. * IIUC Guido said he's not delegating this PEP. There's now a discussion afoot about merging "tulip" in, in time for 3.4a4, which I think is going to happen. And as Release Manager for 3.4 I'm happy to see this happen. I'd like to propose much the same thing for Argument Clinic: check it in now, before 3.4a4, and let it bake in trunk a little before feature freeze at beta 1 while I run around and finish the documentation. You can review the existing code here: https://bitbucket.org/larry/python-clinic/ (Yes, in this fork clinic.py is at the root--but then I never intended to merge this fork.) Checking in Clinic would mean: * Adding the tool (one file, "clinic.py") and its test suite into Tools/clinic/ * Adding its documentation somewhere in the Doc/ directory * Merging six or eight places where I've converted random functions to use Clinic Brett Cannon gave the code a cursory review at PyCon CA over about six hours, and iirc he said it was basically reasonable. He wanted more coverage in the tests, and more in-depth review of course, but he had only minor feedback, all of which I've incorporated. (In case you're wondering: last time I tried, coverage from just the unit test suite was about 85%, but if you also ran all my sample usage of it in the CPython code it rose to like 92%.) So, quick poll: do you approve of me checking Argument Clinic in to Python 3.4 trunk in its current state before 3.4a4? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Oct 15 14:57:34 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Oct 2013 22:57:34 +1000 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131015142719.360e30a8@pitrou.net> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> Message-ID: On 15 October 2013 22:27, Antoine Pitrou wrote: > Le Tue, 15 Oct 2013 22:05:41 +1000, > Nick Coghlan a ?crit : > >> On 15 October 2013 13:12, Glenn Linderman >> wrote: >> > Of course, if the name were changed to be accurate, or the feature >> > made less prone to misuse, then maybe it would terminate. >> >> I've come to the conclusion it makes sense to rename it from ignore to >> suppress, since that's the term already used for this operation in the >> rest of the context management docs: http://bugs.python.org/issue19266 > > "suppress" and/or "trap" (which I believe is a common name for such > operation). I chose the new name based on the terminology used in the existing docs (I'll grant that a lot of these docs were written by me at various points in time, but taken together they still create a solid precedent that "suppress" refers to discarding the exception, while "trap" just refers to catching it and then potentially doing something with it, including reraising it). >From http://docs.python.org/dev/library/stdtypes.html#contextmanager.__exit__: "Exit the runtime context and return a Boolean flag indicating if any exception that occurred should be suppressed." "Returning a true value from this method will cause the with statement to suppress the exception and continue execution with the statement immediately following the with statement. " >From http://docs.python.org/dev/reference/datamodel.html#object.__exit__ "If an exception is supplied, and the method wishes to suppress the exception (i.e., prevent it from being propagated), it should return a true value." >From http://docs.python.org/dev/library/contextlib#contextlib.contextmanager "If an exception is trapped merely in order to log it or to perform some action (rather than to suppress it entirely), the generator must reraise that exception." >From http://docs.python.org/dev/library/contextlib#contextlib.ignore (!) "As with any other mechanism that completely suppresses exceptions, it should only be used to cover very specific errors where silently ignoring the exception is known to be the right thing to do." >From http://docs.python.org/dev/library/contextlib#contextlib.ExitStack "...if an inner callback suppresses or replaces an exception, then outer callbacks will be passed arguments based on that updated state." >From http://docs.python.org/dev/library/contextlib#contextlib.ExitStack.enter_context "These context managers may suppress exceptions just as they normally would if used directly as part of a with statement." >From http://docs.python.org/dev/library/contextlib#contextlib.ExitStack.push "By returning true values, these callbacks can suppress exceptions the same way context manager __exit__() methods can." >From http://docs.python.org/dev/library/contextlib#contextlib.ExitStack.callback "Unlike the other methods, callbacks added this way cannot suppress exceptions (as they are never passed the exception details)." So, having been convinced that "ignore" was the wrong choice of name, reviewing the docs made it clear to me what the name *should* be. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tseaver at palladion.com Tue Oct 15 15:07:23 2013 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 15 Oct 2013 09:07:23 -0400 Subject: [Python-Dev] Checking in Argument Clinic early? In-Reply-To: <525D3B53.6000508@hastings.org> References: <525D3B53.6000508@hastings.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/15/2013 08:55 AM, Larry Hastings wrote: > So, quick poll: do you approve of me checking Argument Clinic in to > Python 3.4 trunk in its current state before 3.4a4? +1 Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlJdPgsACgkQ+gerLs4ltQ6CIACbB47gAz5px1MGVhRvqY+S0GP/ SYUAnRSQ2e1xqjySgwrFYz7gCuHbWksJ =+QW/ -----END PGP SIGNATURE----- From ncoghlan at gmail.com Tue Oct 15 15:06:22 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Oct 2013 23:06:22 +1000 Subject: [Python-Dev] Checking in Argument Clinic early? In-Reply-To: <525D3B53.6000508@hastings.org> References: <525D3B53.6000508@hastings.org> Message-ID: On 15 October 2013 22:55, Larry Hastings wrote: > So, quick poll: do you approve of me checking Argument Clinic in to Python > 3.4 trunk in its current state before 3.4a4? I'm inclined to say yes, but ultimately the +1 you need is Guido's :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From larry at hastings.org Tue Oct 15 15:30:38 2013 From: larry at hastings.org (Larry Hastings) Date: Tue, 15 Oct 2013 14:30:38 +0100 Subject: [Python-Dev] Checking in Argument Clinic early? In-Reply-To: References: <525D3B53.6000508@hastings.org> Message-ID: <525D437E.30501@hastings.org> On 10/15/2013 02:06 PM, Nick Coghlan wrote: > On 15 October 2013 22:55, Larry Hastings wrote: >> So, quick poll: do you approve of me checking Argument Clinic in to Python >> 3.4 trunk in its current state before 3.4a4? > I'm inclined to say yes, but ultimately the +1 you need is Guido's :) In the past I've found Guido extremely deferential to the Release Manager. I haven't asked the 3.4 Release Manager his opinion in so many words, but I gather he's quite encouraging. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Tue Oct 15 15:37:43 2013 From: phd at phdru.name (Oleg Broytman) Date: Tue, 15 Oct 2013 17:37:43 +0400 Subject: [Python-Dev] non-US zip archives support in zipfile.py In-Reply-To: <525D1765.6040103@v.loewis.de> References: <525D1765.6040103@v.loewis.de> Message-ID: <20131015133743.GA14173@iskra.aviel.ru> Hi! On Tue, Oct 15, 2013 at 12:22:29PM +0200, "\"Martin v. L?wis\"" wrote: > Am 15.10.13 10:53, schrieb Daniel Holth: > > FYI zipfile does do UTF-8 > > (http://hg.python.org/cpython/file/d7ebe03fa752/Lib/zipfile.py#l368). > > Only the non-Unicode encodings might need some help. I like that the > > patch is only concerned with decoding. Is it necessary to support > > writing non-UTF8 encodings? > > For purity, no. For practicality, perhaps. It might be that some tools > don't know how to unpack files marked as UTF-8, and always default to > the system code page. > > OTOH, before accepting such a change, I'd like to know what actual > existing tool would require such a hack, and a confirmation that this > very tool indeed does not support UTF-8. Just an example how I create and extract zip files with Russian filenames on Linux: http://phdru.name/Software/Python/misc/zip.py.txt http://phdru.name/Software/Python/misc/unzip.py.txt I hardcoded cp866 encoding to handle zip files the same way other tools (like WinZIP/WinRAR) handle Russian filenames. I hardcoded cp866 encoding into the code because it's the only encoding I've ever saw. cp866 is the OEM encoding in Windows; ANSI encoding is cp1251. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From martin at v.loewis.de Tue Oct 15 15:46:15 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 15 Oct 2013 15:46:15 +0200 Subject: [Python-Dev] non-US zip archives support in zipfile.py In-Reply-To: References: <20131015120209.GA7989@ando> Message-ID: <525D4727.6020005@v.loewis.de> Am 15.10.13 14:49, schrieb Daniel Holth: > It is part of the ZIP specification. CP437 or UTF-8 are the two > official choices, but other encodings happen on Russian, Japanese > systems. Indeed. Formally, the other encodings are not supported by the ZIP specification, and are thus formally misuse of the format. I believe (without having proof) that early versions of the specification failed to discuss the file name encoding at all, making people believe that it is unspecified and always the system encoding (which is useless, of course, as you create zip files to move them across systems). Regards, Martin From solipsis at pitrou.net Tue Oct 15 15:59:53 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 15 Oct 2013 15:59:53 +0200 Subject: [Python-Dev] Checking in Argument Clinic early? References: <525D3B53.6000508@hastings.org> Message-ID: <20131015155953.3fc4ac17@pitrou.net> Le Tue, 15 Oct 2013 13:55:47 +0100, Larry Hastings a ?crit : > > You can review the existing code here: > > https://bitbucket.org/larry/python-clinic/ You didn't create it as a (Bitbucket) fork of CPython, and you didn't put your changes in a separate branch, so this repo is a bit useless to review the changes :-/ Is clinic.py automatically invoked by the build chain? (i.e. Makefile) Regards Antoine. From larry at hastings.org Tue Oct 15 16:20:34 2013 From: larry at hastings.org (Larry Hastings) Date: Tue, 15 Oct 2013 15:20:34 +0100 Subject: [Python-Dev] Checking in Argument Clinic early? In-Reply-To: <20131015155953.3fc4ac17@pitrou.net> References: <525D3B53.6000508@hastings.org> <20131015155953.3fc4ac17@pitrou.net> Message-ID: <525D4F32.20005@hastings.org> On 10/15/2013 02:59 PM, Antoine Pitrou wrote: > Le Tue, 15 Oct 2013 13:55:47 +0100, > Larry Hastings a ?crit : >> You can review the existing code here: >> >> https://bitbucket.org/larry/python-clinic/ > You didn't create it as a (Bitbucket) fork of CPython, and you didn't > put your changes in a separate branch, so this repo is a bit useless to > review the changes :-/ Thanks for at least trying! It is a fork of CPython, but yeah I didn't use Bitbucket's UI to create it. If you want to struggle through it by hand, read clinic.py (obviously), and then search for the string "clinic" in these files: Modules/_cursesmodule.c Modules/_datetimemodule.c Modules/_dbmmodule.c Modules/posixmodule.c Modules/unicodedata.c Modules/_weakref.c Modules/zlibmodule.c Objects/dictobject.c Objects/unicodeobject.c Can you suggest how I could make it easier for you to review? I'd be happy to make an issue for it, which I probably should do anyway (if there isn't already one). > Is clinic.py automatically invoked by the build chain? (i.e. Makefile) Not at the moment. I'm a neophyte when it comes to moderne sophisticated Makefile munging. There's also a chicken-and-egg problem there. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Oct 15 16:52:19 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 15 Oct 2013 07:52:19 -0700 Subject: [Python-Dev] Checking in Argument Clinic early? In-Reply-To: References: <525D3B53.6000508@hastings.org> Message-ID: I'm +1 -- I have been ever since Larry & I discussed it at EuroPython 2012. On Tue, Oct 15, 2013 at 6:06 AM, Nick Coghlan wrote: > On 15 October 2013 22:55, Larry Hastings wrote: > > So, quick poll: do you approve of me checking Argument Clinic in to > Python > > 3.4 trunk in its current state before 3.4a4? > > I'm inclined to say yes, but ultimately the +1 you need is Guido's :) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Oct 15 17:52:31 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 15 Oct 2013 11:52:31 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> Message-ID: <20131015155232.8789F25098D@webabinitio.net> On Tue, 15 Oct 2013 22:57:34 +1000, Nick Coghlan wrote: > On 15 October 2013 22:27, Antoine Pitrou wrote: > > Le Tue, 15 Oct 2013 22:05:41 +1000, > > Nick Coghlan a ??crit : > > > >> On 15 October 2013 13:12, Glenn Linderman > >> wrote: > >> > Of course, if the name were changed to be accurate, or the feature > >> > made less prone to misuse, then maybe it would terminate. > >> > >> I've come to the conclusion it makes sense to rename it from ignore to > >> suppress, since that's the term already used for this operation in the > >> rest of the context management docs: http://bugs.python.org/issue19266 > > > > "suppress" and/or "trap" (which I believe is a common name for such > > operation). > > I chose the new name based on the terminology used in the existing > docs (I'll grant that a lot of these docs were written by me at > various points in time, but taken together they still create a solid > precedent that "suppress" refers to discarding the exception, while > "trap" just refers to catching it and then potentially doing something > with it, including reraising it). > > From http://docs.python.org/dev/library/stdtypes.html#contextmanager.__exit__: > > "Exit the runtime context and return a Boolean flag indicating if any > exception that occurred should be suppressed." > > "Returning a true value from this method will cause the with statement > to suppress the exception and continue execution with the statement > immediately following the with statement. " Someone could read this and reason by analogy: "so if the *context manager* is named suppress, then execution will resume following the statement where the exception was suppressed". They'd be wrong, of course, I'm just pointing out some confusion that we might still see. > From http://docs.python.org/dev/reference/datamodel.html#object.__exit__ > > "If an exception is supplied, and the method wishes to suppress the > exception (i.e., prevent it from being propagated), it should return a > true value." > > From http://docs.python.org/dev/library/contextlib#contextlib.contextmanager > > "If an exception is trapped merely in order to log it or to perform > some action (rather than to suppress it entirely), the generator must > reraise that exception." > > From http://docs.python.org/dev/library/contextlib#contextlib.ignore (!) > > "As with any other mechanism that completely suppresses exceptions, it > should only be used to cover very specific errors where silently > ignoring the exception is known to be the right thing to do." This should emphasize that execution resumes with the statement following the with block, and give that as a strong motivation for only using it to cover the line where the exception is to be suppressed (and there are likely to be very few cases where it would reasonable for it to be more than one line, and that is probably worth pointing out). > From http://docs.python.org/dev/library/contextlib#contextlib.ExitStack > > "...if an inner callback suppresses or replaces an exception, then > outer callbacks will be passed arguments based on that updated state." > > From http://docs.python.org/dev/library/contextlib#contextlib.ExitStack.enter_context > > "These context managers may suppress exceptions just as they normally > would if used directly as part of a with statement." > > From http://docs.python.org/dev/library/contextlib#contextlib.ExitStack.push > > "By returning true values, these callbacks can suppress exceptions the > same way context manager __exit__() methods can." > > From http://docs.python.org/dev/library/contextlib#contextlib.ExitStack.callback > > "Unlike the other methods, callbacks added this way cannot suppress > exceptions (as they are never passed the exception details)." > > So, having been convinced that "ignore" was the wrong choice of name, > reviewing the docs made it clear to me what the name *should* be. I think 'trap' would be much clearer. What about making the context manager provide the trapped exception, in a fashion similar to what assertRaises does? Even if that was almost never used in practice, the fact that the CM provides only *one* exception no matter how many statements are in the with block would strongly reinforce the actual semantics of the construct. It would also make it parallel to assertRaises, which seems like a consistency bonus. And I could see it getting used. I think I've had code where the logic was: possibly trap an exception, stuff it in a variable, do some logic, check the variable to see if we had an exception earlier, and if so do something with it or otherwise branch the logic. I won't say this is common, and I won't say there wouldn't often be a better way to write it...but I can think that it might have utility. With that change, I'd be +1. With just suppress, I'm -0. --David From z at etiol.net Tue Oct 15 15:41:10 2013 From: z at etiol.net (Zero Piraeus) Date: Tue, 15 Oct 2013 10:41:10 -0300 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> Message-ID: <20131015134110.GB10360@piedra> : On Tue, Oct 15, 2013 at 10:05:41PM +1000, Nick Coghlan wrote: > I've come to the conclusion it makes sense to rename it from ignore to > suppress, since that's the term already used for this operation in the > rest of the context management docs: http://bugs.python.org/issue19266 > > Given the number of people vehemently opposed to the old name, I'm > sure there will be plenty of volunteers to create a patch while I do > some more work on PEP 453 :) Patch submitted :-) -[]z. -- Zero Piraeus: dictatum erat http://etiol.net/pubkey.asc From barry at python.org Tue Oct 15 17:58:23 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 15 Oct 2013 11:58:23 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131015155232.8789F25098D@webabinitio.net> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> Message-ID: <20131015115823.44401498@limelight.wooz.org> On Oct 15, 2013, at 11:52 AM, R. David Murray wrote: >I think 'trap' would be much clearer. What about making the context >manager provide the trapped exception, in a fashion similar to >what assertRaises does? Even if that was almost never used in practice, >the fact that the CM provides only *one* exception no matter how >many statements are in the with block would strongly reinforce the >actual semantics of the construct. It would also make it parallel to >assertRaises, which seems like a consistency bonus. > >And I could see it getting used. I think I've had code where the logic >was: possibly trap an exception, stuff it in a variable, do some logic, >check the variable to see if we had an exception earlier, and if so do >something with it or otherwise branch the logic. I won't say this is >common, and I won't say there wouldn't often be a better way to write >it...but I can think that it might have utility. > >With that change, I'd be +1. With just suppress, I'm -0. Yeah, I would also be +1 with that. -Barry From raymond.hettinger at gmail.com Tue Oct 15 18:20:24 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Tue, 15 Oct 2013 09:20:24 -0700 Subject: [Python-Dev] Checking in Argument Clinic early? In-Reply-To: <525D3B53.6000508@hastings.org> References: <525D3B53.6000508@hastings.org> Message-ID: On Oct 15, 2013, at 5:55 AM, Larry Hastings wrote: > I'd like to propose much the same thing for Argument Clinic: check it in now, before 3.4a4, and let it bake in trunk a little before feature freeze at beta 1 while I run around and finish the documentation. This is a good plan. "Letting it bake" means we get more of a chance to interact with it and you will give more of a chance to make refinements. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Tue Oct 15 18:22:54 2013 From: larry at hastings.org (Larry Hastings) Date: Tue, 15 Oct 2013 17:22:54 +0100 Subject: [Python-Dev] Checking in Argument Clinic early? In-Reply-To: References: <525D3B53.6000508@hastings.org> Message-ID: <525D6BDE.40408@hastings.org> On 10/15/2013 03:52 PM, Guido van Rossum wrote: > I'm +1 -- I have been ever since Larry & I discussed it at EuroPython > 2012. If this love-in continues I'll prep a release tonight and commit it in the morning... just before my flight home across the Atlantic. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Tue Oct 15 18:28:37 2013 From: skip at pobox.com (Skip Montanaro) Date: Tue, 15 Oct 2013 11:28:37 -0500 Subject: [Python-Dev] Checking in Argument Clinic early? In-Reply-To: <525D6BDE.40408@hastings.org> References: <525D3B53.6000508@hastings.org> <525D6BDE.40408@hastings.org> Message-ID: > If this love-in continues I'll prep a release tonight and commit it in the > morning... just before my flight home across the Atlantic. You've got it backwards. You're supposed to release, then leave town for two weeks, not come home to field complaints. I think it might even be in the Zen of Python somewhere, if you read between the lines... :-) Skip From pje at telecommunity.com Tue Oct 15 18:28:30 2013 From: pje at telecommunity.com (PJ Eby) Date: Tue, 15 Oct 2013 12:28:30 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131015155232.8789F25098D@webabinitio.net> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> Message-ID: On Tue, Oct 15, 2013 at 11:52 AM, R. David Murray wrote: > I think 'trap' would be much clearer. +1. Short and sweet, and just ambiguous enough that you don't leap to the conclusion that the error is ignored. I agree that "suppress" is basically a synonym for "ignore"; trap at least *implies* some kind of control flow change, which is what's needed to prevent misconceptions. Personally, I would rate "catch" higher than "trap" because it further implies that it is catching a thrown exception, but would compromise to "trap" if that'll end the thread sooner. ;-) > What about making the context > manager provide the trapped exception, in a fashion similar to > what assertRaises does? Sadly, that won't work, since context managers provide a value *before* the block is executed, not after. From pje at telecommunity.com Tue Oct 15 18:40:44 2013 From: pje at telecommunity.com (PJ Eby) Date: Tue, 15 Oct 2013 12:40:44 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> Message-ID: On Tue, Oct 15, 2013 at 8:57 AM, Nick Coghlan wrote: > So, having been convinced that "ignore" was the wrong choice of name, > reviewing the docs made it clear to me what the name *should* be. >From the point of view of code *outside* a block, the error is indeed suppressed. But, as one of those examples actually points out, what's happening from the POV *inside* the block is that the exception is "trapped". So using "suppress" creates an ambiguity: are we suppressing these errors *inside* the block, or *outside* the block? The way it actually works is errors are suppressed from the code *surrounding* the block, but the word can equally be interpreted as suppressing errors *inside* the block, in exactly the same way that "ignore" can be misread. So, if we're going with words that have precedent in the doc, the term "trap", as used here: > "If an exception is trapped merely in order to log it or to perform > some action (rather than to suppress it entirely), the generator must > reraise that exception." is the only one used to describe the POV from inside the block, where the error is... well, being trapped. ;-) It is a more apt description of what actually happens, even if it's only usable for the specific use case where an exception is trapped in order to suppress it. From storchaka at gmail.com Tue Oct 15 18:47:58 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 15 Oct 2013 19:47:58 +0300 Subject: [Python-Dev] Checking in Argument Clinic early? In-Reply-To: <525D3B53.6000508@hastings.org> References: <525D3B53.6000508@hastings.org> Message-ID: 15.10.13 15:55, Larry Hastings ???????(??): > So, quick poll: do you approve of me checking Argument Clinic in to > Python 3.4 trunk in its current state before 3.4a4? Could you get us a week for preliminary review before checking? I remember I proposed a little different syntax and I don't know whether it was taken into account and how a syntax evolved. From eliben at gmail.com Tue Oct 15 18:49:17 2013 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 15 Oct 2013 09:49:17 -0700 Subject: [Python-Dev] Checking in Argument Clinic early? In-Reply-To: <525D3B53.6000508@hastings.org> References: <525D3B53.6000508@hastings.org> Message-ID: On Tue, Oct 15, 2013 at 5:55 AM, Larry Hastings wrote: > > > I really want to get Argument Clinic in to Python 3.4. Currently the > following is true: > > - It has an implementation I've been working on for more than a year. > - I assert that it's a reasonable approach and the APIs are ready for > public consumption. > - The syntax for the DSL is the one proposed by Guido at PyCon US 2013. > - So far It has little in the way of documentation apart from the PEP. > - The PEP is out of date and not ready for pronouncement. > - IIUC Guido said he's not delegating this PEP. > > I'm neutral here but just wanted to say that it amazes me that we simultaneously (1) run a centi-thread about some semantic trifles in exception handling, peppered with bikeshedding and emotions and (2) you plan to "just go ahead and commit" a large, significant feature for which: 1. The PEP is out of date and has not even been pronounced 2. The code only got "cursory review" 3. The code is not even in a state suitable for convenient review, as Antoine noted And all of that just because the 3.4 deadling is looming. This makes me somewhat sad about our process :-/ And the comparison to Tulip is completely irrelevant. Every single thing in Tulip is being heatedly discussed among multiple developers on the Tulip list for many months now, and all code gets reviewed. Eli > > - > > > There's now a discussion afoot about merging "tulip" in, in time for > 3.4a4, which I think is going to happen. And as Release Manager for 3.4 > I'm happy to see this happen. > > I'd like to propose much the same thing for Argument Clinic: check it in > now, before 3.4a4, and let it bake in trunk a little before feature freeze > at beta 1 while I run around and finish the documentation. > > > You can review the existing code here: > > https://bitbucket.org/larry/python-clinic/ > > (Yes, in this fork clinic.py is at the root--but then I never intended to > merge this fork.) > > Checking in Clinic would mean: > > - Adding the tool (one file, "clinic.py") and its test suite into > Tools/clinic/ > - Adding its documentation somewhere in the Doc/ directory > - Merging six or eight places where I've converted random functions > to use Clinic > > Brett Cannon gave the code a cursory review at PyCon CA over about six > hours, and iirc he said it was basically reasonable. He wanted more > coverage in the tests, and more in-depth review of course, but he had only > minor feedback, all of which I've incorporated. > > (In case you're wondering: last time I tried, coverage from just the unit > test suite was about 85%, but if you also ran all my sample usage of it in > the CPython code it rose to like 92%.) > > > So, quick poll: do you approve of me checking Argument Clinic in to Python > 3.4 trunk in its current state before 3.4a4? > > > */arry* > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/eliben%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Oct 15 18:55:45 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 15 Oct 2013 09:55:45 -0700 Subject: [Python-Dev] Checking in Argument Clinic early? In-Reply-To: References: <525D3B53.6000508@hastings.org> Message-ID: On Tue, Oct 15, 2013 at 9:49 AM, Eli Bendersky wrote: > And all of that just because the 3.4 deadling is looming. This makes me > somewhat sad about our process :-/ > Well, almost any process occasionally needs some pepper in the butt, and there's nothing like a hard deadline to get people to take some action. So I think this is actually appropriate. > And the comparison to Tulip is completely irrelevant. Every single thing > in Tulip is being heatedly discussed among multiple developers on the Tulip > list for many months now, and all code gets reviewed. > That may be, but it's not completely irrelevant -- the push for getting Tulip checked in was motivated by exactly the same deadline. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Tue Oct 15 19:03:42 2013 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 15 Oct 2013 13:03:42 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131015142719.360e30a8@pitrou.net> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> Message-ID: On Tue, Oct 15, 2013 at 8:27 AM, Antoine Pitrou wrote: >> I've come to the conclusion it makes sense to rename it from ignore to >> suppress, since that's the term already used for this operation in the >> rest of the context management docs: http://bugs.python.org/issue19266 > > "suppress" and/or "trap" (which I believe is a common name for such > operation). +0 for s/ignore/suppress/ -1 for trap/catch or anything similar that would imply that with trap(Error) as e: ... would make any sense. From tseaver at palladion.com Tue Oct 15 19:12:43 2013 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 15 Oct 2013 13:12:43 -0400 Subject: [Python-Dev] Checking in Argument Clinic early? In-Reply-To: References: <525D3B53.6000508@hastings.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/15/2013 12:47 PM, Serhiy Storchaka wrote: > 15.10.13 15:55, Larry Hastings ???????(??): >> So, quick poll: do you approve of me checking Argument Clinic in to >> Python 3.4 trunk in its current state before 3.4a4? > > Could you get us a week for preliminary review before checking? I > remember I proposed a little different syntax and I don't know whether > it was taken into account and how a syntax evolved. The syntax compromise was actually hammered out during / after the Language Summit at PyCon US this year: Guido blessed the resulting spec. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlJdd4sACgkQ+gerLs4ltQ4URgCfRqHMQ4ttE+k1p2w7gMGaCsQI JhoAn2rPnbApwcAbd2TZITzG2UinrK+R =KsLA -----END PGP SIGNATURE----- From solipsis at pitrou.net Tue Oct 15 19:19:52 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 15 Oct 2013 19:19:52 +0200 Subject: [Python-Dev] Checking in Argument Clinic early? References: <525D3B53.6000508@hastings.org> <20131015155953.3fc4ac17@pitrou.net> <525D4F32.20005@hastings.org> Message-ID: <20131015191952.65bc3ca4@fsol> On Tue, 15 Oct 2013 15:20:34 +0100 Larry Hastings wrote: > If you want to struggle through it by hand, read clinic.py (obviously), > and then search for the string "clinic" in these files: > > Modules/_cursesmodule.c > Modules/_datetimemodule.c > Modules/_dbmmodule.c > Modules/posixmodule.c > Modules/unicodedata.c > Modules/_weakref.c > Modules/zlibmodule.c > Objects/dictobject.c > Objects/unicodeobject.c > > > Can you suggest how I could make it easier for you to review? I'd be > happy to make an issue for it, which I probably should do anyway (if > there isn't already one). hg di -r "" sounds like a good approach. > > Is clinic.py automatically invoked by the build chain? (i.e. Makefile) > > Not at the moment. I'm a neophyte when it comes to moderne > sophisticated Makefile munging. There's also a chicken-and-egg problem > there. Well, solving the chicken-and-egg problem is what would make the Makefile integration interesting :-) Regards Antoine. From brett at python.org Tue Oct 15 19:33:03 2013 From: brett at python.org (Brett Cannon) Date: Tue, 15 Oct 2013 13:33:03 -0400 Subject: [Python-Dev] Checking in Argument Clinic early? In-Reply-To: References: <525D3B53.6000508@hastings.org> Message-ID: On Tue, Oct 15, 2013 at 12:49 PM, Eli Bendersky wrote: > On Tue, Oct 15, 2013 at 5:55 AM, Larry Hastings wrote: > >> >> >> I really want to get Argument Clinic in to Python 3.4. Currently the >> following is true: >> >> - It has an implementation I've been working on for more than a year. >> - I assert that it's a reasonable approach and the APIs are ready for >> public consumption. >> - The syntax for the DSL is the one proposed by Guido at PyCon US >> 2013. >> - So far It has little in the way of documentation apart from the >> PEP. >> - The PEP is out of date and not ready for pronouncement. >> - IIUC Guido said he's not delegating this PEP. >> >> > I'm neutral here but just wanted to say that it amazes me that we > simultaneously (1) run a centi-thread about some semantic trifles in > exception handling, peppered with bikeshedding and emotions and (2) you > plan to "just go ahead and commit" a large, significant feature for which: > > 1. The PEP is out of date and has not even been pronounced > Guido has essentially pronounced for a while, Larry just didn't bother to update the PEP based on agreements made in person. > 2. The code only got "cursory review" > Well, I stared at the code over a day and a half with Larry sitting next to me, going over almost all of it line-by-line. So it's cursory by me ignoring PEP 8 style violations, but it wasn't just a quick scroll-through. > 3. The code is not even in a state suitable for convenient review, as > Antoine noted > Beyond the tests and Argument Clinic script, only the output is not self-contained, so reviewing it against a checkout is more about how stuff shifts in the .c files which has already been blessed by Guido, not the main code that does the work. > > And all of that just because the 3.4 deadling is looming. This makes me > somewhat sad about our process :-/ > You also need to remember this is initially a tool for *us* and not the outside world, so brittleness found after release really isn't an issue as long as Python continues to compile. Since we are the users we are the ones using and iterating on it before final release, and so the sooner the better. This isn't a rush where we will have something possible half-baked until it sees public use; it's going to iterate quickly the day it's landed and we all can work towards getting it done for ourselves before 3.4 final is out. -Brett > > And the comparison to Tulip is completely irrelevant. Every single thing > in Tulip is being heatedly discussed among multiple developers on the Tulip > list for many months now, and all code gets reviewed. > > Eli > > > > > > > >> >> - >> >> >> There's now a discussion afoot about merging "tulip" in, in time for >> 3.4a4, which I think is going to happen. And as Release Manager for 3.4 >> I'm happy to see this happen. >> >> I'd like to propose much the same thing for Argument Clinic: check it in >> now, before 3.4a4, and let it bake in trunk a little before feature freeze >> at beta 1 while I run around and finish the documentation. >> >> >> You can review the existing code here: >> >> https://bitbucket.org/larry/python-clinic/ >> >> (Yes, in this fork clinic.py is at the root--but then I never intended to >> merge this fork.) >> >> Checking in Clinic would mean: >> >> - Adding the tool (one file, "clinic.py") and its test suite into >> Tools/clinic/ >> - Adding its documentation somewhere in the Doc/ directory >> - Merging six or eight places where I've converted random functions >> to use Clinic >> >> Brett Cannon gave the code a cursory review at PyCon CA over about six >> hours, and iirc he said it was basically reasonable. He wanted more >> coverage in the tests, and more in-depth review of course, but he had only >> minor feedback, all of which I've incorporated. >> >> (In case you're wondering: last time I tried, coverage from just the unit >> test suite was about 85%, but if you also ran all my sample usage of it in >> the CPython code it rose to like 92%.) >> >> >> So, quick poll: do you approve of me checking Argument Clinic in to >> Python 3.4 trunk in its current state before 3.4a4? >> >> >> */arry* >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/eliben%40gmail.com >> >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Oct 15 18:45:59 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 15 Oct 2013 09:45:59 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> Message-ID: <525D7147.5000505@stoneleaf.us> On 10/15/2013 09:28 AM, PJ Eby wrote: > On Tue, Oct 15, 2013 at 11:52 AM, R. David Murray wrote: >> >> What about making the context manager provide the trapped exception, >> in a fashion similar to what assertRaises does? > > Sadly, that won't work, since context managers provide a value > *before* the block is executed, not after. However, if the CM provides the value before, and the with block accepts it, it can then have the exception added to it: with trap(OSError) as cm: os.unlink('missing.txt') if cm.exc: do_something() -- ~Ethan~ From alexander.belopolsky at gmail.com Tue Oct 15 20:17:15 2013 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 15 Oct 2013 14:17:15 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <525D7147.5000505@stoneleaf.us> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <525D7147.5000505@stoneleaf.us> Message-ID: On Tue, Oct 15, 2013 at 12:45 PM, Ethan Furman wrote: > with trap(OSError) as cm: > os.unlink('missing.txt') > if cm.exc: > do_something() .. and why is this better than try: os.unlink('missing.txt') except OSError as exc: do_something() ? From z at etiol.net Tue Oct 15 20:02:15 2013 From: z at etiol.net (Zero Piraeus) Date: Tue, 15 Oct 2013 15:02:15 -0300 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <525D7147.5000505@stoneleaf.us> References: <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <525D7147.5000505@stoneleaf.us> Message-ID: <20131015180215.GA11026@piedra> : On Tue, Oct 15, 2013 at 09:45:59AM -0700, Ethan Furman wrote: > [...] if the CM provides the value before, and the with block accepts > it, it can then have the exception added to it: > > with trap(OSError) as cm: > os.unlink('missing.txt') > if cm.exc: > do_something() Is there some advantage to this over try: os.unlink('missing.txt') except OSError as exc: do_something() I thought the whole point was to replace code that would otherwise contain 'except: pass' with something slightly more concise. Once you expose the exception, it becomes just an uglier version of a try/except block, as far as I can see. -[]z. -- Zero Piraeus: pons asinorum http://etiol.net/pubkey.asc From v+python at g.nevcal.com Tue Oct 15 21:09:04 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 15 Oct 2013 12:09:04 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> Message-ID: <525D92D0.9040102@g.nevcal.com> On 10/15/2013 9:28 AM, PJ Eby wrote: > On Tue, Oct 15, 2013 at 11:52 AM, R. David Murray wrote: >> >I think 'trap' would be much clearer. > +1. Short and sweet, and just ambiguous enough that you don't leap to > the conclusion that the error is ignored. I agree that "suppress" is > basically a synonym for "ignore"; trap at least*implies* some kind of > control flow change, which is what's needed to prevent misconceptions. Yes, trap or catch imply a control flow change, ignore and suppress do not. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Oct 15 21:58:07 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 15 Oct 2013 15:58:07 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131015180215.GA11026@piedra> References: <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <525D7147.5000505@stoneleaf.us> <20131015180215.GA11026@piedra> Message-ID: <20131015195807.D7AD925091F@webabinitio.net> On Tue, 15 Oct 2013 15:02:15 -0300, Zero Piraeus wrote: > : > > On Tue, Oct 15, 2013 at 09:45:59AM -0700, Ethan Furman wrote: > > [...] if the CM provides the value before, and the with block accepts > > it, it can then have the exception added to it: > > > > with trap(OSError) as cm: > > os.unlink('missing.txt') > > if cm.exc: > > do_something() > > Is there some advantage to this over > > try: > os.unlink('missing.txt') > except OSError as exc: > do_something() > > I thought the whole point was to replace code that would otherwise > contain 'except: pass' with something slightly more concise. Once you > expose the exception, it becomes just an uglier version of a try/except > block, as far as I can see. Obviously not :) You wouldn't use trap/as for that. Instead you use it to replace this: exc = None try: os.unlink('missing.txt') except OSError as exc: pass some other code if exc is not None: do_something() with: with trap(OSError) as cm: os.unlink('missing.txt') some other code if cm.exc is not None: do_something() which saves you *three* lines, not just two :) Of course in real life, in order for 'some other code' really be the best way to organize this snippet, you're going to be doing something more complicated than catching OSError from an unlink call. If this is accepted, I do *not* expect to see this in real code very often at all. The main motivation for it is to make the semantics of the context manager clearer, and as a bonus make it work like assertRaises, which is our existing stdlib context manager that traps exceptions. I'd be happy with 'trap' or 'catch' without 'as' support, but the names make more sense if the exception is actually available after it is trapped/caught. --David From timothy.c.delaney at gmail.com Tue Oct 15 21:59:55 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 16 Oct 2013 06:59:55 +1100 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <525D7147.5000505@stoneleaf.us> Message-ID: On 16 October 2013 05:17, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > On Tue, Oct 15, 2013 at 12:45 PM, Ethan Furman wrote: > > with trap(OSError) as cm: > > os.unlink('missing.txt') > > if cm.exc: > > do_something() > > .. and why is this better than > > try: > os.unlink('missing.txt') > except OSError as exc: > do_something() It would allow you to perform a series of operations then process the any exceptions all together e.g. with trap(OSError) as cm1: os.unlink('missing.txt') with trap(OSError) as cm2: os.unlink('other_missing.txt') with trap(OSError) as cm3: os.unlink('another_missing.txt') for cm in (cm1, cm2, cm3): if cm.exc: do_something(cm.exc) An equivalent implementation would be: exceptions = [] try: os.unlink('missing.txt') except OSError as exc: exceptions.append(exc) try: os.unlink('missing.txt') except OSError as exc: exceptions.append(exc) try: os.unlink('missing.txt') except OSError as exc: exceptions.append(exc) for exc in exceptions: if exc: do_something(exc) Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Oct 15 22:50:52 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 16 Oct 2013 09:50:52 +1300 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131015180215.GA11026@piedra> References: <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <525D7147.5000505@stoneleaf.us> <20131015180215.GA11026@piedra> Message-ID: <525DAAAC.1000600@canterbury.ac.nz> Zero Piraeus wrote: > I thought the whole point was to replace code that would otherwise > contain 'except: pass' with something slightly more concise. Hmmm, that suggests another potential name. with passing_on(OSError): os.remove(somefile) -- Greg From ncoghlan at gmail.com Wed Oct 16 00:51:44 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 16 Oct 2013 08:51:44 +1000 Subject: [Python-Dev] Submitting PEP 453 (pip bootstrapping) for pronouncement In-Reply-To: References: Message-ID: On 14 Oct 2013 08:21, "Nick Coghlan" wrote: > > > On 14 Oct 2013 04:26, "Paul Moore" wrote: > > > > On 13 October 2013 13:43, Nick Coghlan wrote: > > > Accordingly, in addition to adding the option to extract and install ``pip`` > > > during installation, this PEP proposes that the Windows installer (and > > > ``sysconfig``) in Python 3.4 and later be updated to: > > > > > > - install scripts to PythonXY\bin rather than PythonXY\Tools\Scripts > > > - add PythonXY\bin to the Windows PATH (in addition to PythonXY) when the > > > PATH modification option is enabled during installation > > > > Then implication of this is that venv (and virtualenv, but that's less > > relevant to python-dev) should be updated to use a "bin" directory on > > Windows rather than the current "Scripts" directory. This would only > > be for 3.4, which means that virtualenv will need to make the > > directory name conditional on version, I guess. > > The sysconfig directory scheme will change accordingly, so it may be possible to leverage that by basing the virtualenv layout on the default scheme. Paul figured out (over on distutils-sig) that there's almost certainly a compatibility issue with existing Windows installers for Python modules in changing the name of the third party scripts directory. Accordingly, I plan to change that part of the PEP to just refer to the appropriate sysconfig entry, and leave out the idea of changing that subdirectory name from Scripts to bin (while it's likely solvable, it's also completely orthogonal to the main purpose of the PEP). I also still have a couple of tweaks to make to the integration timeline that I added after discussion with the pip devs and the release team. So, once again, not quite ready for pronouncement after all... *sigh* Cheers, Nick. > > Cheers, > Nick. > > > > > Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Oct 16 00:41:33 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 15 Oct 2013 15:41:33 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <525D7147.5000505@stoneleaf.us> Message-ID: <525DC49D.4070802@stoneleaf.us> On 10/15/2013 12:59 PM, Tim Delaney wrote: > On 16 October 2013 05:17, Alexander Belopolsky > > wrote: > > On Tue, Oct 15, 2013 at 12:45 PM, Ethan Furman > wrote: > > with trap(OSError) as cm: > > os.unlink('missing.txt') > > if cm.exc: > > do_something() > > .. and why is this better than > > try: > os.unlink('missing.txt') > except OSError as exc: > do_something() > > > It would allow you to perform a series of operations then process the any exceptions all together e.g. > > with trap(OSError) as cm1: > os.unlink('missing.txt') > > with trap(OSError) as cm2: > os.unlink('other_missing.txt') > > with trap(OSError) as cm3: > os.unlink('another_missing.txt') > > for cm in (cm1, cm2, cm3): > if cm.exc: > do_something(cm.exc) As long as we're having fun ;) trap_os = suppress(OSError) dead_files = ('missing.txt', 'other_missing.txt', 'actually_here.txt') for file in dead_files: with trap_os: os.unlink(missing.txt) for exc in trap_os.suppressed_exceptions: do_something(exc) -- ~Ethan~ From ncoghlan at gmail.com Wed Oct 16 01:24:11 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 16 Oct 2013 09:24:11 +1000 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131015155232.8789F25098D@webabinitio.net> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> Message-ID: On 16 Oct 2013 01:54, "R. David Murray" wrote: > > On Tue, 15 Oct 2013 22:57:34 +1000, Nick Coghlan wrote: > > On 15 October 2013 22:27, Antoine Pitrou wrote: > > > Le Tue, 15 Oct 2013 22:05:41 +1000, > > > Nick Coghlan a ?crit : > > > > > >> On 15 October 2013 13:12, Glenn Linderman > > >> wrote: > > >> > Of course, if the name were changed to be accurate, or the feature > > >> > made less prone to misuse, then maybe it would terminate. > > >> > > >> I've come to the conclusion it makes sense to rename it from ignore to > > >> suppress, since that's the term already used for this operation in the > > >> rest of the context management docs: http://bugs.python.org/issue19266 > > > > > > "suppress" and/or "trap" (which I believe is a common name for such > > > operation). > > > > I chose the new name based on the terminology used in the existing > > docs (I'll grant that a lot of these docs were written by me at > > various points in time, but taken together they still create a solid > > precedent that "suppress" refers to discarding the exception, while > > "trap" just refers to catching it and then potentially doing something > > with it, including reraising it). > > > > From http://docs.python.org/dev/library/stdtypes.html#contextmanager.__exit__: > > > > "Exit the runtime context and return a Boolean flag indicating if any > > exception that occurred should be suppressed." > > > > "Returning a true value from this method will cause the with statement > > to suppress the exception and continue execution with the statement > > immediately following the with statement. " > > Someone could read this and reason by analogy: "so if the *context > manager* is named suppress, then execution will resume following the > statement where the exception was suppressed". They'd be wrong, of > course, I'm just pointing out some confusion that we might still see. > > > From http://docs.python.org/dev/reference/datamodel.html#object.__exit__ > > > > "If an exception is supplied, and the method wishes to suppress the > > exception (i.e., prevent it from being propagated), it should return a > > true value." > > > > From http://docs.python.org/dev/library/contextlib#contextlib.contextmanager > > > > "If an exception is trapped merely in order to log it or to perform > > some action (rather than to suppress it entirely), the generator must > > reraise that exception." > > > > From http://docs.python.org/dev/library/contextlib#contextlib.ignore (!) > > > > "As with any other mechanism that completely suppresses exceptions, it > > should only be used to cover very specific errors where silently > > ignoring the exception is known to be the right thing to do." > > This should emphasize that execution resumes with the statement following > the with block, and give that as a strong motivation for only using it > to cover the line where the exception is to be suppressed (and there are > likely to be very few cases where it would reasonable for it to be more > than one line, and that is probably worth pointing out). > > > From http://docs.python.org/dev/library/contextlib#contextlib.ExitStack > > > > "...if an inner callback suppresses or replaces an exception, then > > outer callbacks will be passed arguments based on that updated state." > > > > From http://docs.python.org/dev/library/contextlib#contextlib.ExitStack.enter_context > > > > "These context managers may suppress exceptions just as they normally > > would if used directly as part of a with statement." > > > > From http://docs.python.org/dev/library/contextlib#contextlib.ExitStack.push > > > > "By returning true values, these callbacks can suppress exceptions the > > same way context manager __exit__() methods can." > > > > From http://docs.python.org/dev/library/contextlib#contextlib.ExitStack.callback > > > > "Unlike the other methods, callbacks added this way cannot suppress > > exceptions (as they are never passed the exception details)." > > > > So, having been convinced that "ignore" was the wrong choice of name, > > reviewing the docs made it clear to me what the name *should* be. > > I think 'trap' would be much clearer. What about making the context > manager provide the trapped exception, in a fashion similar to > what assertRaises does? Even if that was almost never used in practice, > the fact that the CM provides only *one* exception no matter how > many statements are in the with block would strongly reinforce the > actual semantics of the construct. It would also make it parallel to > assertRaises, which seems like a consistency bonus. > > And I could see it getting used. I think I've had code where the logic > was: possibly trap an exception, stuff it in a variable, do some logic, > check the variable to see if we had an exception earlier, and if so do > something with it or otherwise branch the logic. I won't say this is > common, and I won't say there wouldn't often be a better way to write > it...but I can think that it might have utility. Yes, something along those lines is exactly what I plan to explore for 3.5. It's a completely orthogonal question, though - suppress is specifically an up front statement of intent for the case where you genuinely don't care if the exception gets raised or not. With suppress already available, the more general construct can be designed specifically for the case where you *do* care if the exception was raised or not. By contrast, providing only the more general construct would mean that "I don't care if an exception is raised or not" would be implied solely by the absence of an "as" clause, which is a very poor way to communicate intent. > With that change, I'd be +1. With just suppress, I'm -0. Please, please, please just let it drop. Even though it eventually convinced me to change the name, this thread still epitomises everything that sucks about soul destroying, energy draining bikeshed painting that makes me wonder why I ever bother trying to make anything better. Regards, Nick. > > --David > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Oct 16 02:22:31 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 16 Oct 2013 11:22:31 +1100 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> Message-ID: <20131016002231.GC7989@ando> On Wed, Oct 16, 2013 at 09:24:11AM +1000, Nick Coghlan wrote: > Even though it eventually > convinced me to change the name, this thread still epitomises everything > that sucks about soul destroying, energy draining bikeshed painting that > makes me wonder why I ever bother trying to make anything better. I feel your pain, I truly do, but I think the name you ended up with is much better than "ignore(d)". Given that naming things is one of the two hard problems of computer science[1], I think that a bit of pain to get the name just right is worth it. (You did decide on "suppress", right? I must admit I've only skimmed this thread.) In any case, thank you for seeing this through. [1] The other is cache invalidation. -- Steven From rdmurray at bitdance.com Wed Oct 16 03:41:16 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 15 Oct 2013 21:41:16 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> Message-ID: <20131016014116.D13F5250AF0@webabinitio.net> On Wed, 16 Oct 2013 09:24:11 +1000, Nick Coghlan wrote: > On 16 Oct 2013 01:54, "R. David Murray" wrote: > > With that change, I'd be +1. With just suppress, I'm -0. > > Please, please, please just let it drop. Even though it eventually > convinced me to change the name, this thread still epitomises everything > that sucks about soul destroying, energy draining bikeshed painting that > makes me wonder why I ever bother trying to make anything better. I've already said that I was fine with whatever you decide (even if it was to keep ignore). I did think trap/as was a good suggestion, but if you don't like it, you don't like it. --David From ncoghlan at gmail.com Wed Oct 16 08:37:50 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 16 Oct 2013 16:37:50 +1000 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131016014116.D13F5250AF0@webabinitio.net> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <20131016014116.D13F5250AF0@webabinitio.net> Message-ID: On 16 Oct 2013 11:42, "R. David Murray" wrote: > > On Wed, 16 Oct 2013 09:24:11 +1000, Nick Coghlan wrote: > > On 16 Oct 2013 01:54, "R. David Murray" wrote: > > > With that change, I'd be +1. With just suppress, I'm -0. > > > > Please, please, please just let it drop. Even though it eventually > > convinced me to change the name, this thread still epitomises everything > > that sucks about soul destroying, energy draining bikeshed painting that > > makes me wonder why I ever bother trying to make anything better. > > I've already said that I was fine with whatever you decide (even if it > was to keep ignore). I did think trap/as was a good suggestion, but if > you don't like it, you don't like it. Thank you for that :) Cheers, Nick. > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Wed Oct 16 09:07:16 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 16 Oct 2013 00:07:16 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> Message-ID: <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> On Oct 15, 2013, at 4:24 PM, Nick Coghlan wrote: > this thread still epitomises everything that sucks about soul destroying, energy draining bikeshed painting that makes me wonder why I ever bother trying to make anything better. > FWIW, here's a little history: * Last year, one of my clients suggested this feature based on code his team had been using in production. * I thought it might be a good idea, so I scanned the standard library and some third-party modules to see how well it worked with existing examples of try/except/pass. * I posted the proposal to the tracker over a year ago (issue 15806) along with some examples from the standard library. I also posted an egrep search so that others could evaluate it on their own code. * Along the way, I did usability testing on some corporate Python programmers who were attending my intermediate and advance Python classes. The feedback was positive (expressive, readable, and concise). * In February, I presented ignore() in the keynote for the U.S. Pycon. Again, the feedback was positive. * After some refinements, the module maintainer (Nick) blessed it and the code was checked-in to the Python 3.4 codebase in early March. * This month, I made a minor name change from ignored() to ignore(). * The next day, Antoine launched this thread and everything hit the fan. As Nick said, this thread was just awful. I found it painful to read each day. Shoot from the hip comments were given greater weight than months of development. Neither Nick nor I were given an ounce of respect for the thought we put into this or for our long track record of good design. The python-dev mailing list is a complex (and sometimes dysfunctional) environment. I understand that everyone on the soccer field feels a need to step-in and kick the ball to affect the outcome, but this was too much. To each participant, it may not seem like bike-shedding, but there were almost a hundred emails in this thread just to kick around six lines of code that were checked-in seven months ago: @contextmanager def ignored(*exceptions): try: yield except exceptions: pass Raymond P.S. The name suppress() is not as good as ignore(), but I'm so burnt out on this thread that I just don't care anymore. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Oct 16 10:21:18 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 16 Oct 2013 10:21:18 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> Message-ID: <20131016102118.7bd2c0a5@pitrou.net> Le Wed, 16 Oct 2013 00:07:16 -0700, Raymond Hettinger a ?crit : > > As Nick said, this thread was just awful. I found it painful to read > each day. Shoot from the hip comments were given greater weight than > months of development. Neither Nick nor I were given an ounce of > respect for the thought we put into this or for our long track record > of good design. I think it would probably have been much less "awful" if the comments had been addressed from the start. I don't remember anyone being insulting to either you or Nick, just people questioning the decision to include this in the stdlib. You also can't expect people to know about any "user testing" you didn't mention having done. re "User testing": I thought we resisted including "convenient one-liners" in the stdlib - though in this case it's more of a two-liner replacement. I'm not really surprised that some people are spontaneously sympathetic to such additions, but that hasn't really been a criterion in the past, has it? re "Months of development": well... you've said it yourself, it's 6 lines of very simple code. Perhaps you're being *a little* hyperbolic? Both the concept and the implementation are so trivial that I can't imagine more than a couple days being spent on it, total. (real "months of development" get routinely ignored or rejected, for various good or bad reasons... It always hurts when it happens, but I don't think this one is a good instance of the case: for one, your code is still there). > The python-dev mailing list is a complex (and sometimes dysfunctional) > environment. I understand that everyone on the soccer field feels a > need to step-in and kick the ball to affect the outcome, but this was > too much. Well, I don't know about others, but I *personally* find python-dev quite functional. Core development is a team's work, and I think many of us expect it to involve of certain amount of listening to each other. Regards Antoine. From victor.stinner at gmail.com Wed Oct 16 11:40:21 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 16 Oct 2013 11:40:21 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131011092446.4736e078@fsol> References: <3cwyfX6Mdfz7LjX@mail.python.org> <20131011092446.4736e078@fsol> Message-ID: CONGRATULATION! This thread reached 100 mails, it's now time to summarize it into a PEP. Is there a candidate to write it? If no PEP is written, the thread will never die and people will continue to feed it. Victor 2013/10/11 Antoine Pitrou : > > Hello, > > On Fri, 11 Oct 2013 07:39:48 +0200 (CEST) > raymond.hettinger wrote: >> http://hg.python.org/cpython/rev/f014b5f0773f >> changeset: 86209:f014b5f0773f >> user: Raymond Hettinger >> date: Thu Oct 10 22:39:39 2013 -0700 >> summary: >> Rename contextlib.ignored() to contextlib.ignore(). > > I don't think that this contextlib.ignore() thing has been discussed a > lot. > > Ezio was -1 on the tracker, and Eric Smith was -0. I'd like to add my > -1 too. This is a useless addition (the traditional idiom is perfectly > obvious) and makes reading foreign code more tedious by adding > superfluous API calls. > > Please consider reverting. There is absolutely zero use case for this > that wasn't already solved by the traditional "except ...: pass" idiom. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From victor.stinner at gmail.com Wed Oct 16 13:34:08 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 16 Oct 2013 13:34:08 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> Message-ID: 2013/10/16 Raymond Hettinger : > FWIW, here's a little history: Thank you! It helped me to understand the story. > * In February, I presented ignore() in the keynote for the U.S. Pycon. > Again, the feedback was positive. I missed this edition of Pycon US. How did you feedback on the keynote? > To each participant, it may not seem like bike-shedding, but there > were almost a hundred emails in this thread just to kick around six lines > of code that were checked-in seven months ago: Sorry but you cannot summarize this thread to this only point (the name). Many other topics were discussed: * process: most votes were negative on the tracker, Nick didn't mention that he blesssed this function * surprising behaviour when the block contains more than 1 instruction * documentation issue * should we add new functions in Python just to remove 2 lines of code? * possibility of storing the catched exception * etc. If you disagree with the principle of discussing new features, you missed probably an important part of the Python community. The process is not perfect, don't hesisate to suggest enhancement :-) The discussion only started recently (after the function was added) because it started on python-dev which is more visible than the bug tracker. -- Antoine also asked me recently to revert a commit (new malloc API) because it was not discussed enough (so you are not the only one to be harassed by Antoine :-)). I tried to describe my choices on python-dev, but it was hard to follow the flow of emails, so I wrote a PEP (PEP 445). The PEP was quickly accepted, the code is back, BUT the API is now *much* better, and the PEP is a good documentation explaining all technicals choices. (There were many alternatives to the proposed API, the PEP now lists all of them.) I can now say that my first API was no well designed, and the PEP helped me to improve it. It is frustrating to have to revert a commit, it should not happen. My patch was available since many weeks (months), but Antoine was right: it was not discussed enough. I don't know yet how to get more review on the bug tracker, sometimes even emails to python-dev are not enough. It depends on the motivation and availability of each developer :-) Victor From victor.stinner at gmail.com Wed Oct 16 13:42:34 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 16 Oct 2013 13:42:34 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <525D7147.5000505@stoneleaf.us> Message-ID: I might agree with idea of contextlib.ignore() (I'm still opposed to the idea), but I don't understand the purpose of adding a new syntax doing exactly the same than try/except: > with trap(OSError) as cm: > os.unlink('missing.txt') > if cm.exc: > do_something() Nobody noticed that this can be written: try: os.unlink('missing.txt') except OSError as err: # do something with err ?? What happened with the Zen Principle "There should be one-- and preferably only one --obvious way to do it." I don't understand why I would import contextlib and use a new context manager, whereas try/except is already a builtin feature of Python. By the way, what are the performances of contextlib.ignore()? Exceptions can be slow in some cases. Adding something even slower would not be a good idea. Victor 2013/10/15 Tim Delaney : > On 16 October 2013 05:17, Alexander Belopolsky > wrote: >> >> On Tue, Oct 15, 2013 at 12:45 PM, Ethan Furman wrote: >> > with trap(OSError) as cm: >> > os.unlink('missing.txt') >> > if cm.exc: >> > do_something() >> >> .. and why is this better than >> >> try: >> os.unlink('missing.txt') >> except OSError as exc: >> do_something() > > > It would allow you to perform a series of operations then process the any > exceptions all together e.g. > > with trap(OSError) as cm1: > os.unlink('missing.txt') > > with trap(OSError) as cm2: > os.unlink('other_missing.txt') > > with trap(OSError) as cm3: > os.unlink('another_missing.txt') > > for cm in (cm1, cm2, cm3): > if cm.exc: > do_something(cm.exc) > > An equivalent implementation would be: > > exceptions = [] > > try: > os.unlink('missing.txt') > except OSError as exc: > exceptions.append(exc) > > try: > os.unlink('missing.txt') > except OSError as exc: > exceptions.append(exc) > > try: > os.unlink('missing.txt') > except OSError as exc: > exceptions.append(exc) > > for exc in exceptions: > if exc: > do_something(exc) > > Tim Delaney > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > From solipsis at pitrou.net Wed Oct 16 13:50:46 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 16 Oct 2013 13:50:46 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <525D7147.5000505@stoneleaf.us> Message-ID: <20131016135046.260f1aed@pitrou.net> Le Wed, 16 Oct 2013 13:42:34 +0200, Victor Stinner a ?crit : > I might agree with idea of contextlib.ignore() (I'm still opposed to > the idea), but I don't understand the purpose of adding a new syntax > doing exactly the same than try/except: > > > with trap(OSError) as cm: > > os.unlink('missing.txt') > > if cm.exc: > > do_something() > > Nobody noticed that this can be written: > > try: > os.unlink('missing.txt') > except OSError as err: > # do something with err > > ?? What happened with the Zen Principle "There should be one-- and > preferably only one --obvious way to do it." I don't understand why I > would import contextlib and use a new context manager, whereas > try/except is already a builtin feature of Python. > > By the way, what are the performances of contextlib.ignore()? > Exceptions can be slow in some cases. Adding something even slower > would not be a good idea. A "try" block which succeeds is fast. A context manager will always be slower, especially when written in pure Python with a generator wrapped in a contextlib.contextmanager. Regards Antoine. From victor.stinner at gmail.com Wed Oct 16 14:01:37 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 16 Oct 2013 14:01:37 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131016135046.260f1aed@pitrou.net> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <525D7147.5000505@stoneleaf.us> <20131016135046.260f1aed@pitrou.net> Message-ID: 2013/10/16 Antoine Pitrou : >> By the way, what are the performances of contextlib.ignore()? >> Exceptions can be slow in some cases. Adding something even slower >> would not be a good idea. > > A "try" block which succeeds is fast. Ah yes, I never reminder this fact. I try to not care too much of micro-optimizations :-) > A context manager will always be > slower, especially when written in pure Python with a generator wrapped > in a contextlib.contextmanager. ignore() is much slower than try/except even if the block succeeds? Victor From solipsis at pitrou.net Wed Oct 16 14:09:16 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 16 Oct 2013 14:09:16 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <525D7147.5000505@stoneleaf.us> <20131016135046.260f1aed@pitrou.net> Message-ID: <20131016140916.3c73d5d3@pitrou.net> Le Wed, 16 Oct 2013 14:01:37 +0200, Victor Stinner a ?crit : > 2013/10/16 Antoine Pitrou : > >> By the way, what are the performances of contextlib.ignore()? > >> Exceptions can be slow in some cases. Adding something even slower > >> would not be a good idea. > > > > A "try" block which succeeds is fast. > > Ah yes, I never reminder this fact. I try to not care too much of > micro-optimizations :-) It's not so much a micro-optimization than the fact that pushing a block on the stack is very cheap. IIRC, what is expensive is: - creating the exception object with the associated traceback - matching the exception in the "exception SomeException" clause I don't have numbers, though :-) Regards Antoine. From solipsis at pitrou.net Wed Oct 16 14:24:11 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 16 Oct 2013 14:24:11 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <525D7147.5000505@stoneleaf.us> <20131016135046.260f1aed@pitrou.net> <20131016140916.3c73d5d3@pitrou.net> Message-ID: <20131016142411.33190faa@pitrou.net> Le Wed, 16 Oct 2013 14:09:16 +0200, Antoine Pitrou a ?crit : > Le Wed, 16 Oct 2013 14:01:37 +0200, > Victor Stinner a ?crit : > > 2013/10/16 Antoine Pitrou : > > >> By the way, what are the performances of contextlib.ignore()? > > >> Exceptions can be slow in some cases. Adding something even > > >> slower would not be a good idea. > > > > > > A "try" block which succeeds is fast. > > > > Ah yes, I never reminder this fact. I try to not care too much of > > micro-optimizations :-) > > It's not so much a micro-optimization than the fact that pushing a > block on the stack is very cheap. > IIRC, what is expensive is: > - creating the exception object with the associated traceback > - matching the exception in the "exception SomeException" clause By the way, of course this must be kept in proportion. At worse the total cost is a couple of microseconds. Still, a simplistic micro-benchmark: $ ./python -m timeit "try: pass" "except Exception: pass" 100000000 loops, best of 3: 0.0164 usec per loop $ ./python -m timeit -s "from contextlib import ignore" "with ignore(Exception): pass" 100000 loops, best of 3: 2.21 usec per loop Regards Antoine. From ericsnowcurrently at gmail.com Wed Oct 16 16:31:44 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 16 Oct 2013 08:31:44 -0600 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> Message-ID: On Oct 16, 2013 5:35 AM, "Victor Stinner" wrote: > > 2013/10/16 Raymond Hettinger : > > FWIW, here's a little history: > > Thank you! It helped me to understand the story. > > > * In February, I presented ignore() in the keynote for the U.S. Pycon. > > Again, the feedback was positive. > > I missed this edition of Pycon US. How did you feedback on the keynote? > > > To each participant, it may not seem like bike-shedding, but there > > were almost a hundred emails in this thread just to kick around six lines > > of code that were checked-in seven months ago: > > Sorry but you cannot summarize this thread to this only point (the > name). Many other topics were discussed: > > * process: most votes were negative on the tracker, Nick didn't > mention that he blesssed this function > * surprising behaviour when the block contains more than 1 instruction > * documentation issue > * should we add new functions in Python just to remove 2 lines of code? > * possibility of storing the catched exception > * etc. > > If you disagree with the principle of discussing new features, you > missed probably an important part of the Python community. The process > is not perfect, don't hesisate to suggest enhancement :-) > > The discussion only started recently (after the function was added) > because it started on python-dev which is more visible than the bug > tracker. > > -- > > Antoine also asked me recently to revert a commit (new malloc API) > because it was not discussed enough (so you are not the only one to be > harassed by Antoine :-)). I tried to describe my choices on > python-dev, but it was hard to follow the flow of emails, so I wrote a > PEP (PEP 445). The PEP was quickly accepted, the code is back, BUT the > API is now *much* better, and the PEP is a good documentation > explaining all technicals choices. (There were many alternatives to > the proposed API, the PEP now lists all of them.) I can now say that > my first API was no well designed, and the PEP helped me to improve > it. > > It is frustrating to have to revert a commit, it should not happen. My > patch was available since many weeks (months), but Antoine was right: > it was not discussed enough. I don't know yet how to get more review > on the bug tracker, sometimes even emails to python-dev are not > enough. It depends on the motivation and availability of each > developer :-) You make several good points, Victor. However, in this case the change is a new function and a small, innocuous one at that. That is not enough justification alone, as Antoine pointed out, but the module's maintainer made a decision on this. When a module's maintainer makes a decision on a relatively insignificant addition to the module, I'd expect little resistance or even comment (the original commit was months ago). That's why I'm surprised by the reaction to this change. It just seems like the whole thing is being blown way out of proportion to the detriment of other interesting problems. Sure, things could have been done differently. But in this case it's not that big a deal. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Oct 16 16:06:10 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 16 Oct 2013 07:06:10 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> Message-ID: <525E9D52.7020108@stoneleaf.us> On 10/16/2013 12:07 AM, Raymond Hettinger wrote: > On Oct 15, 2013, at 4:24 PM, Nick Coghlan wrote: >> >> this thread still epitomises everything that sucks about soul destroying, energy draining bikeshed painting that makes >> me wonder why I ever bother trying to make anything better. I think each of us has had this experience. It can certainly be draining and disheartening, but keep in mind we all want the same thing: a beautiful language to work with. > FWIW, here's a little history: > * Last year, one of my clients suggested this feature > based on code his team had been using in production. > * I thought it might be a good idea, so I scanned the standard > library and some third-party modules to see how well it worked > with existing examples of try/except/pass. > * I posted the proposal to the tracker over a year ago (issue 15806) > along with some examples from the standard library. I also posted > an egrep search so that others could evaluate it on their own code. These three steps show on the bug tracker > * Along the way, I did usability testing on some corporate Python > programmers who were attending my intermediate and advance > Python classes. The feedback was positive (expressive, readable, > and concise). This does not. > * In February, I presented ignore() in the keynote for the U.S. Pycon. > Again, the feedback was positive. This does not. > * After some refinements, the module maintainer (Nick) blessed it True, but what you are leaving out is that most of the feedback from pydevs in that issue was negative. So to someone reading the issue, as I just did, it looks like: * Here's an idea * Here's examples * Not a good idea feedback * Not a good idea feedback * Here's a better way to write it * Not a good idea feedback * Committed > and the code was checked-in to the Python 3.4 codebase in early March. > * This month, I made a minor name change from ignored() to ignore(). > * The next day, Antoine launched this thread and everything hit the fan. It would have been better if this thread had been launched back in March, but we all have busy lives and sometimes things get missed. > As Nick said, this thread was just awful. I found it painful to read each day. > Shoot from the hip comments were given greater weight than months > of development. Again, items that did not show up in the bug tracker. > Neither Nick nor I were given an ounce of respect > for the thought we put into this or for our long track record of good design. Everybody makes mistakes. > The python-dev mailing list is a complex (and sometimes dysfunctional) > environment. I understand that everyone on the soccer field feels a need > to step-in and kick the ball to affect the outcome, but this was too much. We'll have to agree to disagree on this point. I find pydev an invaluable resource for putting many eyes and many perspectives on code / issues / design questions. > To each participant, it may not seem like bike-shedding, but there > were almost a hundred emails in this thread just to kick around six lines > of code that were checked-in seven months ago: Actually, it was to kick around one line of code, the most import one: def ignored(...): > P.S. The name suppress() is not as good as ignore(), but I'm so burnt out > on this thread that I just don't care anymore. I agree it doesn't read as well, but despite that I think suppress is better. 'ignore' would surely be misinterpreted many more times. -- ~Ethan~ From rdmurray at bitdance.com Wed Oct 16 16:36:04 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 16 Oct 2013 10:36:04 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131016135046.260f1aed@pitrou.net> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <525D7147.5000505@stoneleaf.us> <20131016135046.260f1aed@pitrou.net> Message-ID: <20131016143604.79B8B250AEC@webabinitio.net> Victor Stinner a ??crit : > I might agree with idea of contextlib.ignore() (I'm still opposed to > the idea), but I don't understand the purpose of adding a new syntax > doing exactly the same than try/except: > > > with trap(OSError) as cm: > > os.unlink('missing.txt') > > if cm.exc: > > do_something() > > Nobody noticed that this can be written: > > try: > os.unlink('missing.txt') > except OSError as err: > # do something with err > > ?? What happened with the Zen Principle "There should be one-- and > preferably only one --obvious way to do it." I don't understand why I > would import contextlib and use a new context manager, whereas > try/except is already a builtin feature of Python. For the record, that was *not* my proposal. You'd be silly to replace the try/except with the context manager in the above case. The case where the CM would be useful would be if there were statements in between the context manager and the 'if'. And that *still* wasn't the proposal. My proposal was to provide the exception on the CM so that the name 'trap' made sense, and so that it was clear that only one exception was ever caught, and so that the CM was parallel in functionality to the existing assertRaises CM. But none of this affects the code, because Nick isn't accepting the proposal, and that's fine. --David From victor.stinner at gmail.com Wed Oct 16 17:10:21 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 16 Oct 2013 17:10:21 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131015195807.D7AD925091F@webabinitio.net> References: <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <525D7147.5000505@stoneleaf.us> <20131015180215.GA11026@piedra> <20131015195807.D7AD925091F@webabinitio.net> Message-ID: I would rewrite your examples using try/finally: try: try: os.unlink('missing.txt') finally: some other code except OSError as exc: do_something() It's differently than yours, because it catchs OSError on "some; other; code". Sometimes, I write it differently by copying the cleanup code: try: os.unlink('missing.txt') cleanup() except OSError as exc: cleanup() do_something() (Yeah, it's ugly to duplicate code.) Victor 2013/10/15 R. David Murray : > You wouldn't use trap/as for that. Instead you use it to replace this: > > exc = None > try: > os.unlink('missing.txt') > except OSError as exc: > pass > some > other > code > if exc is not None: > do_something() > > with: > > with trap(OSError) as cm: > os.unlink('missing.txt') > some > other > code > if cm.exc is not None: > do_something() > > which saves you *three* lines, not just two :) > > Of course in real life, in order for 'some other code' really be the > best way to organize this snippet, you're going to be doing something > more complicated than catching OSError from an unlink call. > > If this is accepted, I do *not* expect to see this in real code very > often at all. The main motivation for it is to make the semantics > of the context manager clearer, and as a bonus make it work like > assertRaises, which is our existing stdlib context manager that traps > exceptions. > > I'd be happy with 'trap' or 'catch' without 'as' support, but the > names make more sense if the exception is actually available after > it is trapped/caught. > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From barry at python.org Wed Oct 16 17:24:07 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 16 Oct 2013 11:24:07 -0400 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> Message-ID: <20131016112407.19f8836e@anarchist> On Oct 16, 2013, at 08:31 AM, Eric Snow wrote: >When a module's maintainer makes a decision on a relatively insignificant >addition to the module, I'd expect little resistance or even comment (the >original commit was months ago). That's why I'm surprised by the reaction >to this change. It just seems like the whole thing is being blown way out >of proportion to the detriment of other interesting problems. Sure, things >could have been done differently. But in this case it's not that big a >deal. Any project as big and diverse as Python needs a hierarchical structure of trust and responsibility. I see it *roughly* as core dev < module maintainer < release manager < bdfl delegate < bdfl. However, it's imperative to remain vigilantly transparent so that everyone understands the rationale and motivation behind a change, even if they disagree with it. Trust is extended upwards when this transparency is extended downloads. "'Cause I said so" only works at the top of the chain. ;) I posted my original question because the change seemed so random and arbitrary, and the commit message didn't enlighten me. -Barry From stephen at xemacs.org Wed Oct 16 17:24:23 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 17 Oct 2013 00:24:23 +0900 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <525D7147.5000505@stoneleaf.us> Message-ID: <87ppr5fdco.fsf@uwakimon.sk.tsukuba.ac.jp> Victor Stinner writes: > the idea), but I don't understand the purpose of adding a new syntax > doing exactly the same than try/except: > > > with trap(OSError) as cm: > > os.unlink('missing.txt') > > if cm.exc: > > do_something() > > Nobody noticed that this can be written: > > try: > os.unlink('missing.txt') > except OSError as err: > # do something with err A couple of people did, and it was pointed out that "with trap ... if" allows intervening statements that are *not* in the "with" suite, while all of the statements between "try" and "except" must be part of the "try" suite. The actual equivalent would be try: os.unlink('missing.txt') except OSError as err: pass # unhandled statements may go here if err: # do something with err This is really horrible. > ?? What happened with the Zen Principle "There should be one-- and > preferably only one --obvious way to do it." AIUI, the justification for this feature goes something like this: 1. It is fast and effective to use a try ... except block to catch and ignore an exception, but it's awkward and ugly, which may deter people from catching those exceptions. This is much nicer. 2. The "attractive nuisance" argument against the nicer syntax is probably a fallacy: cargo cult programmers benefit as much as Raymond's students from the correct single-line-suite usage. On the other hand, nicer syntax may make it easier to wean the cargo cult programmers from multi-statement try suites, and there's little evidence that the nicer syntax would actually encourage them to use more multi-statement suites. 3. As for the *several* Zen principles this feature violates, we really should encourage people to "do something appropriate" with exceptions, instead of try: main() except: print("You REALLY SCREWED UP this time, didn't you!", file=sys.stderr) sys.exit(errors.PEBKAC) but encouraging robust programming is hard: exception handling is just plain awkward if you do it in any generality. We hope "with ignore()" can help in one frequently encountered, easily understood, use case. From ethan at stoneleaf.us Wed Oct 16 17:17:43 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 16 Oct 2013 08:17:43 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <525E9D52.7020108@stoneleaf.us> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <525E9D52.7020108@stoneleaf.us> Message-ID: <525EAE17.1050901@stoneleaf.us> On 10/16/2013 07:06 AM, Ethan Furman wrote: > > Actually, it was to kick around one line of code, the most import one: > > def ignored(...): s/import/important/ From solipsis at pitrou.net Wed Oct 16 17:50:14 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 16 Oct 2013 17:50:14 +0200 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> Message-ID: <20131016175014.1624f0a3@pitrou.net> Le Wed, 16 Oct 2013 08:31:44 -0600, Eric Snow a ?crit : > > You make several good points, Victor. However, in this case the > change is a new function and a small, innocuous one at that. That is > not enough justification alone, as Antoine pointed out, but the > module's maintainer made a decision on this. > > When a module's maintainer makes a decision on a relatively > insignificant addition to the module, I'd expect little resistance or > even comment (the original commit was months ago). Just FWIW, I had totally overlooked the original commit, otherwise I would have raised my concerns back then. Regards Antoine. From stephen at xemacs.org Wed Oct 16 17:52:57 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 17 Oct 2013 00:52:57 +0900 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> Message-ID: <87ob6pfc12.fsf@uwakimon.sk.tsukuba.ac.jp> Eric Snow writes: > That's why I'm surprised by the reaction to this change.? It just > seems like the whole thing is being blown way out of proportion to > the detriment of other interesting problems. The feature itself a perfect bikeshedding pitfall. Everybody here understands the Zen, and this addition violates at least a half-dozen Zen principles. That's why the thread had more heads than the Hydra. In cases like this, I would recommend that the responsible maintainer (yeah, I know, sorry for "blaming the victim") just write "Thank you for your comment. I understand that you're thinking about , and I've thought again, but I still think this is good enough. This thread is over" -- and then *refuse to post again*. From facundobatista at gmail.com Wed Oct 16 18:26:11 2013 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 16 Oct 2013 13:26:11 -0300 Subject: [Python-Dev] Making submodules available when importing top one Message-ID: (all this using Python 3.4.0a3+) In the stdlib, I see that (as an example): >>> import os >>> os.path.abspath >>> os.path However, for other (newer) modules: >>> import urllib >>> urllib.requests.urlopen Traceback (most recent call last): File "", line 1, in AttributeError: 'module' object has no attribute 'requests' >>> import concurrent >>> concurrent.futures Traceback (most recent call last): File "", line 1, in AttributeError: 'module' object has no attribute 'futures' So, is there a reason to not expose the submodules in the top module? This should be fixed? Thanks! -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From phd at phdru.name Wed Oct 16 18:41:11 2013 From: phd at phdru.name (Oleg Broytman) Date: Wed, 16 Oct 2013 20:41:11 +0400 Subject: [Python-Dev] Making submodules available when importing top one In-Reply-To: References: Message-ID: <20131016164111.GA15702@iskra.aviel.ru> On Wed, Oct 16, 2013 at 01:26:11PM -0300, Facundo Batista wrote: > (all this using Python 3.4.0a3+) > > In the stdlib, I see that (as an example): > > >>> import os > >>> os.path.abspath > > >>> os.path > > > > However, for other (newer) modules: > > >>> import urllib > >>> urllib.requests.urlopen > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'module' object has no attribute 'requests' > >>> import concurrent > >>> concurrent.futures > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'module' object has no attribute 'futures' > > > So, is there a reason to not expose the submodules in the top module? > This should be fixed? I don't think it should be fixed. To expose submodules automatically top-level modules have to import them. In case one only needs a couple of submodules (and without lazy import) importing all of them would be a waste of time and memory. Module os.path is a special kind of module. On import os imports a platform-dependent module (posixpath, ntpath) and exposes it as os.path. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From guido at python.org Wed Oct 16 18:41:38 2013 From: guido at python.org (Guido van Rossum) Date: Wed, 16 Oct 2013 09:41:38 -0700 Subject: [Python-Dev] Making submodules available when importing top one In-Reply-To: References: Message-ID: On Wed, Oct 16, 2013 at 9:26 AM, Facundo Batista wrote: > (all this using Python 3.4.0a3+) > > In the stdlib, I see that (as an example): > > >>> import os > >>> os.path.abspath > > >>> os.path > > > > However, for other (newer) modules: > > >>> import urllib > >>> urllib.requests.urlopen > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'module' object has no attribute 'requests' > >>> import concurrent > >>> concurrent.futures > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'module' object has no attribute 'futures' > > > So, is there a reason to not expose the submodules in the top module? > This should be fixed? > os.path is special. The behavior of other modules is intentional. In general it is not a good idea to automatically import all submodules when you import a top-level package; imaging what this would do if you have a large top-level package (maybe a namespace package) containing hundreds of modules. For specific top-level packages that want to provide all (or some) submodules upon import of just the top-level package, they should have the sub-imports in the top-level __init__.py. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Wed Oct 16 21:04:51 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 16 Oct 2013 13:04:51 -0600 Subject: [Python-Dev] Making submodules available when importing top one In-Reply-To: References: Message-ID: On Wed, Oct 16, 2013 at 10:26 AM, Facundo Batista wrote: > (all this using Python 3.4.0a3+) > > In the stdlib, I see that (as an example): > >>>> import os >>>> os.path.abspath > >>>> os.path > > > > However, for other (newer) modules: > >>>> import urllib >>>> urllib.requests.urlopen > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'module' object has no attribute 'requests' >>>> import concurrent >>>> concurrent.futures > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'module' object has no attribute 'futures' > > > So, is there a reason to not expose the submodules in the top module? > This should be fixed? -1 When you import a submodule the parent is imported first, the submodule second, and the submodule is bound as an attribute of the parent. Then the parent is bound to the module where the import happened. This is always the case (and not particular to os.path). For example, let's suppose you have these two files (plus a spam package): import spam.eggs import ham import spam When you run "python -c 'import sausage'", here's what happens: 1. ham is loaded: a. spam is loaded and added to sys.modules. b. eggs is loaded (as "spam.eggs") and added to sys.modules (as "spam.eggs"). c. the "eggs" attribute of spam (spam.eggs) is bound to the module at sys.modules["spam.eggs"]. d. the "spam" attribute of the [currently loading] ham module (ham.spam) is bound to the module at sys.modules["spam"]. 2. sausage is loaded: a-c. (already done when ham was imported) d. the "spam" attribute of the [currently loading] sausage module (sausage.spam) is bound to the module at sys.modules["spam"]. So now you could use "spam.eggs." in sausage even though you did not import "spam.eggs" there. It just happens that during interpreter startup some module did "import os.path", so the submodule is still bound to os. If urllib.requests had been imported during startup (or in another module), then your example would have worked. Guido's response covers just about everything else I was going say. If you really need to auto-import submodules of modules you don't control, you can write your own importer to do it. The import system is pretty easily extended. In this case, however, I doubt that's warranted. -eric From ericsnowcurrently at gmail.com Wed Oct 16 21:05:17 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 16 Oct 2013 13:05:17 -0600 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131016112407.19f8836e@anarchist> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> Message-ID: On Wed, Oct 16, 2013 at 9:24 AM, Barry Warsaw wrote: > On Oct 16, 2013, at 08:31 AM, Eric Snow wrote: >>When a module's maintainer makes a decision on a relatively insignificant >>addition to the module, I'd expect little resistance or even comment (the >>original commit was months ago). That's why I'm surprised by the reaction >>to this change. It just seems like the whole thing is being blown way out >>of proportion to the detriment of other interesting problems. Sure, things >>could have been done differently. But in this case it's not that big a >>deal. > > Any project as big and diverse as Python needs a hierarchical structure of > trust and responsibility. I see it *roughly* as core dev < module maintainer > < release manager < bdfl delegate < bdfl. +1 > > However, it's imperative to remain vigilantly transparent so that everyone > understands the rationale and motivation behind a change, even if they > disagree with it. Trust is extended upwards when this transparency is > extended downloads. "'Cause I said so" only works at the top of the chain. ;) Good point. -eric > > I posted my original question because the change seemed so random and > arbitrary, and the commit message didn't enlighten me. > > -Barry From peter.a.portante at gmail.com Wed Oct 16 23:01:18 2013 From: peter.a.portante at gmail.com (Peter Portante) Date: Wed, 16 Oct 2013 17:01:18 -0400 Subject: [Python-Dev] Regarding stdlib socket module, _fileobject.flush() method using ._rbufsize instead of ._wbufsize Message-ID: Hello, Is there a reason why the stdlib socket module _fileobject.flush() method is using ._rbufsize instead of ._wbufsize at line 297 (Python 2.7.3), where it determines the buffer_size value to be used for _sock.sendall()? Does anybody know the history behind this? Based on what I read in the code, there appear to be four code paths possible setting the _rbufsize and _wbufsize when creating an _fileobject: 1. default, or negative, bufsize argument bufsize argument is set to the default_bufsize class attribute _rbufsize = bufsize _wbufsize = bufsize 2. bufsize = 0 _rbufsize = 1 _wbufsize = 0 3. bufsize = 1 _rbufsize = default_bufsize class attribute _wbufsize = 1 4. bufsize > 1 _rbufsize = _wbufsize = bufsize With that, write() and writelines() uses _wbufsize, as expected. Additionally, could flush's buffer_size local variable be determined just the same using: buffer_size = max(self._wbufsize, self.bufsize) Thanks for considering. -peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Oct 17 00:22:08 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 17 Oct 2013 08:22:08 +1000 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> Message-ID: On 16 Oct 2013 21:34, "Victor Stinner" wrote: > > 2013/10/16 Raymond Hettinger : > > FWIW, here's a little history: > > Thank you! It helped me to understand the story. > > > * In February, I presented ignore() in the keynote for the U.S. Pycon. > > Again, the feedback was positive. > > I missed this edition of Pycon US. How did you feedback on the keynote? > > > To each participant, it may not seem like bike-shedding, but there > > were almost a hundred emails in this thread just to kick around six lines > > of code that were checked-in seven months ago: > > Sorry but you cannot summarize this thread to this only point (the > name). Many other topics were discussed: > > * process: most votes were negative on the tracker, Nick didn't > mention that he blesssed this function > * surprising behaviour when the block contains more than 1 instruction > * documentation issue > * should we add new functions in Python just to remove 2 lines of code? > * possibility of storing the catched exception > * etc. > > If you disagree with the principle of discussing new features, you > missed probably an important part of the Python community. The process > is not perfect, don't hesisate to suggest enhancement :-) > > The discussion only started recently (after the function was added) > because it started on python-dev which is more visible than the bug > tracker. This thread was 99% people rehashing the original tracker issue where I approved the addition of contextlib.ignored. *That* was the main problem with this thread: people commenting without even bothering to read up on the original change. There were fortunately a couple of shining moments of signal (thank you PJE) amidst the tidal wave of noise, and that's what really persuaded me to more closely review the name and how it related to the terminology used elsewhere in the docs. Regards, Nick. > > -- > > Antoine also asked me recently to revert a commit (new malloc API) > because it was not discussed enough (so you are not the only one to be > harassed by Antoine :-)). I tried to describe my choices on > python-dev, but it was hard to follow the flow of emails, so I wrote a > PEP (PEP 445). The PEP was quickly accepted, the code is back, BUT the > API is now *much* better, and the PEP is a good documentation > explaining all technicals choices. (There were many alternatives to > the proposed API, the PEP now lists all of them.) I can now say that > my first API was no well designed, and the PEP helped me to improve > it. > > It is frustrating to have to revert a commit, it should not happen. My > patch was available since many weeks (months), but Antoine was right: > it was not discussed enough. I don't know yet how to get more review > on the bug tracker, sometimes even emails to python-dev are not > enough. It depends on the motivation and availability of each > developer :-) > > Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Oct 17 03:30:52 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 16 Oct 2013 21:30:52 -0400 Subject: [Python-Dev] Regarding stdlib socket module, _fileobject.flush() method using ._rbufsize instead of ._wbufsize In-Reply-To: References: Message-ID: On 10/16/2013 5:01 PM, Peter Portante wrote: > Hello, > > Is there a reason why the stdlib socket module _fileobject.flush() > method is using ._rbufsize instead of ._wbufsize at line 297 (Python > 2.7.3), where it determines the buffer_size value to be used for > _sock.sendall()? Does anybody know the history behind this? Annotations and history show that line is from 61875, 2010-5-23, J P Calderone, Merged revisions 74426... I do not understand the message. -- Terry Jan Reedy From solipsis at pitrou.net Thu Oct 17 12:23:31 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 17 Oct 2013 12:23:31 +0200 Subject: [Python-Dev] Regarding stdlib socket module, _fileobject.flush() method using ._rbufsize instead of ._wbufsize References: Message-ID: <20131017122331.0bb85077@fsol> On Wed, 16 Oct 2013 21:30:52 -0400 Terry Reedy wrote: > On 10/16/2013 5:01 PM, Peter Portante wrote: > > Hello, > > > > Is there a reason why the stdlib socket module _fileobject.flush() > > method is using ._rbufsize instead of ._wbufsize at line 297 (Python > > 2.7.3), where it determines the buffer_size value to be used for > > _sock.sendall()? Does anybody know the history behind this? > > Annotations and history show that line is from > 61875, 2010-5-23, J P Calderone, Merged revisions 74426... > I do not understand the message. When mentioning Mercurial changesets, please post the (hexadecimal) changeset id, not the revision number, since revision numbers are not stable accross clones. http://mercurial.selenic.com/wiki/RevisionNumber Regards Antoine. From ethan at stoneleaf.us Thu Oct 17 16:52:30 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 17 Oct 2013 07:52:30 -0700 Subject: [Python-Dev] Bug tracker and docs@python Message-ID: <525FF9AE.2070905@stoneleaf.us> Good day! What is the effect of assigning a bug tracker issue to docs at python? -- ~Ethan~ From ncoghlan at gmail.com Thu Oct 17 17:26:16 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 18 Oct 2013 01:26:16 +1000 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: <20131016112407.19f8836e@anarchist> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> Message-ID: On 17 October 2013 01:24, Barry Warsaw wrote: > On Oct 16, 2013, at 08:31 AM, Eric Snow wrote: > >>When a module's maintainer makes a decision on a relatively insignificant >>addition to the module, I'd expect little resistance or even comment (the >>original commit was months ago). That's why I'm surprised by the reaction >>to this change. It just seems like the whole thing is being blown way out >>of proportion to the detriment of other interesting problems. Sure, things >>could have been done differently. But in this case it's not that big a >>deal. > > Any project as big and diverse as Python needs a hierarchical structure of > trust and responsibility. I see it *roughly* as core dev < module maintainer > < release manager < bdfl delegate < bdfl. > > However, it's imperative to remain vigilantly transparent so that everyone > understands the rationale and motivation behind a change, even if they > disagree with it. Trust is extended upwards when this transparency is > extended downloads. "'Cause I said so" only works at the top of the chain. ;) Right, and that's why I never opted out of the thread entirely, despite being seriously tempted at times :) It's also caused me to reflect on a few things over the past few days, including why the bar for contextlib is (at this point in time) comparatively lower than the one for functools, collections and itertools. I was also asked a very good question off-list as to why TransformDict escalated to needing a PEP, while I still believe a couple of issues is adequate for keeping this minor addition to contextlib (for the record: the patch Zero posted to http://bugs.python.org/issue19266 has been committed, so the pattern is now called "suppress"). An interesting lightning talk at BrisPy last night also included a key point: the speaker spent a lot of time creating custom internal tools for process engineers to use as an alternative to getting expensive Matlab licenses, and his experience was that object-oriented programming was *just* beyond what most process engineers could cope with in terms of writing their own code. If you look at the history of various constructs in Python, it takes years for idioms and patterns to build up around them, even after the core syntax additions is made to the language. Once those idioms are in place, though, you get to a point where you can first teach people how to use them effectively and only later teach them how to build their own. Because the patterns have been enshrined in things with *names*, you also don't need to be able to recognise them on sight - you have something to look up the first you set, rather than having to interpret a particular shape of code as indicating a particular idiom. This pattern recurs again and again: - you learn how to call functions before you learn how to write them - you learn how to instantiate and use classes before you learn how to write new ones - you learn how to import modules before you learn how to write your own - you learn how to iterate before you learn how to write iterators - you learn how to apply decorators before you learn how to build them - you learn how to invoke context managers before you learn how to write your own - you often don't even have to learn how to use descriptors per se, as it's often hidden in learning to use decorators. Learning to create your own then lets you realise how much of what you thought you knew about classes is actually just based on the way a few standard descriptors work. - you also often don't even have to learn how to use metaclasses, as it's often hidden behind class inheritance (e.g. from abc.ABC or enum.Enum, or from a base class in an ORM or web framework). Learning to create your own metaclasses, on the other hand, can be completely mindbending. Over time, this progressively lowers the barrier to entry for Python programming, as the intent is to enable users to *do* more without necessarily needing to learn (much) more, as well as allowing code authors to more clearly communicate their *intent* to code readers by using *named* patterns rather than leaving them implicit in the code structure. Exception handling, however, is a notable exception(!) to that general pattern. Appropriately *handling* exceptions is actually harder than raising them: raise Exception vs: try: except Exception: The introduction of context managers several years ago basically provides an opportunity to change that over time: by standardising various common try/except idioms, it becomes feasible to postpone the explanations of the underlying details. Compared to the extensive toolkits we already have for functional abstractions, iterator based abstractions and data containers, the general purpose context manipulation idioms are still quite weak. contextlib.closing was part of the initial toolkit in Python 2.5, while contextlib.nested was later removed as fundamentally broken in the presence of resources that acquire their resource in __init__ (like files). contextlib.contextmanager is a tool for *writing* context managers not using them, as is contextlib.ContextDecorator. contextlib.ExitStack is similarly on the "power tool" side of the fence, being the context management equivalent of the iter() and next() builtins (letting you cleanly manipulate the individual steps of a with statement, just as those two builtins let you manipulate the individual steps of a for loop). By contrast, suppress() and redirect_stdout() are the *first* general purpose context managers added to contextlib since its incarnation in Python 2.5 (although there have been many various domain specific context manager additions elsewhere in the standard library). It's worth unpacking the "simple" six line definition of contextlib.suppress, to see how many Python concepts it actually requires you to understand: @contextmanager def suppress(*exceptions): try: yield except exceptions: pass By my count, it's at least: 1. defining functions 2. applying decorators 3. accepting arbitrary positional arguments 4. generators 5. implementing context managers with generators 6. exception handling 7. using dynamic tuples for exception handling 8. the interaction between generators, exception handling and context management This function is *short* because of the information density provided by the various constructs it uses, but it's *far* from being simple. We then turn to the motivating example for the addition, taking an eight (or five) line construct and turning it into a four (or even two!) line construct: First, the eight line version: try: os.remove("somefile.tmp") except FileNotFoundError: pass try: os.remove("someotherfile.tmp") except FileNotFoundError: pass And the five line version (which only applies in this particular example because the two commands are almost identical, and carries the cost of conveying less information in the traceback purely through line position): for name in ("somefile.tmp", "someotherfile.tmp"): try: os.remove(name) except FileNotFoundError: pass Now, the four line version: with suppress(FileNotFoundError): os.remove("somefile.tmp") with suppress(FileNotFoundError): os.remove("someotherfile.tmp") And even a two line version: with suppress(FileNotFoundError): os.remove("somefile.tmp") with suppress(FileNotFoundError): os.remove("someotherfile.tmp") With the try/except version, it's much harder to tell *at a glance* what the code is doing, because significant parts of the intent (the specific exception being caught and the "pass" keyword that indicates it is being suppressed) are hidden down and to the right. Now, the other point raised in the thread is that there's a more general idiom that's potentially of interest, which is to keep track of whether or not an exception was thrown. I actually agree with that, and it's something I now plan to explore in contextlib2 and then (if I decide I actually like the construct) a PEP for Python 3.5. Specifically, it would be a replacement for the existing search loop idiom that was based on a context manager in order to generalise to more than one level of loop nesting. However, the potential ramifications of offering such a feature are vastly greater than those of the far more limited contextlib.suppress context manager (which only covers one very specific exception handling pattern), so it would need to go through the PEP process. it would be something along the lines of the following (note: I'm *not* proposing this for 3.4. There's no way I'd propose it or anything like it for the standard library without trialling it in contextlib2 first, and letting it bake there for at least few months. It may even make sense to tie the failed/missing idiom to suppress() as was suggested in this thread rather than to the new construct): # Arbitrarily nested search loop with exit_label() as found: for i in range(x): for j in range(y): if matches(i, j): found.exit((i, j)) if found: print(found.value) # Delayed exception handling with exit_label(OSError) as failed: os.remove("somefile.tmp") # Do other things... if failed: print(failed.exc) with exit_label(ValueError) as missing: location = data.find(substr) if missing: # Implementation sketch def exit_label(exception): return _ExitLabel(exception) class _ExitLabel: def __init__(exception): if not exception: class ExitToLabel(Exception): pass exception = ExitToLabel self._exception = exception self._sentinel = sentinel = object() self._value = sentinel self._exc = sentinel self._entered = False def __enter__(self): if self._entered: raise RuntimeError("Cannot reuse exit label") self._entered = True return self def __exit__(self, exc_type, exc_value, exc_tb): self._exc = exc_value if isinstance(exc_value, self._exceptions): traceback.clear_frames(exc_value.__traceback__) return True # Suppress the exception return False # Propagate the exception def __bool__(self): return self._exc and self._exc is not self._sentinel def exit(self, value=None): self._value = value raise self._exception @property def value(self): if self._value is self._sentinel: raise RuntimeError("Label value not yet set") return self._value @property def exc(self): if self._exc is self._exc: raise RuntimeError("Label exception result not yet set") return self._exc Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From benjamin at python.org Thu Oct 17 17:29:18 2013 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 17 Oct 2013 11:29:18 -0400 Subject: [Python-Dev] Bug tracker and docs@python In-Reply-To: <525FF9AE.2070905@stoneleaf.us> References: <525FF9AE.2070905@stoneleaf.us> Message-ID: 2013/10/17 Ethan Furman : > Good day! > > What is the effect of assigning a bug tracker issue to docs at python? The assignee field is set to docs at python.org. In other words, nothing magical happens. -- Regards, Benjamin From solipsis at pitrou.net Thu Oct 17 17:36:40 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 17 Oct 2013 17:36:40 +0200 Subject: [Python-Dev] cpython: Issue #16129: Add `Py_SetStandardStreamEncoding` References: <3d0rd742bTz7LjR@mail.python.org> Message-ID: <20131017173640.53016af5@fsol> On Thu, 17 Oct 2013 15:22:03 +0200 (CEST) nick.coghlan wrote: > > +.. c:function:: int Py_SetStandardStreamEncoding(char *encoding, char *errors) > + > + .. index:: > + single: Py_Initialize() > + single: main() > + triple: stdin; stdout; sdterr > + > + This function should be called before :c:func:`Py_Initialize`. It > + specifies which encoding and error handling to use with standard io, > + with the same meanings as in :func:`str.encode`. > + > + It overrides :envvar:`PYTHONIOENCODING` values, and allows embedding code > + to control io encoding when the environment variable does not work. > + > + ``encoding`` and/or ``errors`` may be NULL to use > + :envvar:`PYTHONIOENCODING` and/or default values (depending on other > + settings). > + > + Note that :data:`sys.stderr` always uses the "backslashreplace" error > + handler, regardless of this (or any other) setting. > + > + If :c:func:`Py_Finalize` is called, this function will need to be called > + again in order to affect subsequent calls to :c:func:`Py_Initialize`. > + > + Returns 0 if successful. > + > + Needs a versionadded tag. Regards Antoine. From solipsis at pitrou.net Thu Oct 17 17:38:14 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 17 Oct 2013 17:38:14 +0200 Subject: [Python-Dev] cpython: Try doing a raw test of os.fork()/os.kill(). References: <3d0rsp05P1z7Ljs@mail.python.org> Message-ID: <20131017173814.570ab9b2@fsol> On Thu, 17 Oct 2013 15:33:02 +0200 (CEST) richard.oudkerk wrote: > http://hg.python.org/cpython/rev/9558e9360afc > changeset: 86401:9558e9360afc > parent: 86399:9cd88b39ef62 > user: Richard Oudkerk > date: Thu Oct 17 14:24:06 2013 +0100 > summary: > Try doing a raw test of os.fork()/os.kill(). For this kind of ad-hoc testing, you can also use a custom builder to avoid disrupting the main source tree: http://docs.python.org/devguide/buildbots.html#custom-builders Regards Antoine. From g.brandl at gmx.net Thu Oct 17 17:53:19 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 17 Oct 2013 17:53:19 +0200 Subject: [Python-Dev] Bug tracker and docs@python In-Reply-To: References: <525FF9AE.2070905@stoneleaf.us> Message-ID: Am 17.10.2013 17:29, schrieb Benjamin Peterson: > 2013/10/17 Ethan Furman : >> Good day! >> >> What is the effect of assigning a bug tracker issue to docs at python? > > The assignee field is set to docs at python.org. > > In other words, nothing magical happens. Correct so far, but the point of the exercise is that the mailing list docs at python.org is now nosy and will get updates for the issue. Georg From barry at python.org Thu Oct 17 18:06:20 2013 From: barry at python.org (Barry Warsaw) Date: Thu, 17 Oct 2013 12:06:20 -0400 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> Message-ID: <20131017120620.70dfa755@anarchist> On Oct 18, 2013, at 01:26 AM, Nick Coghlan wrote: >By contrast, suppress() and redirect_stdout() are the *first* general >purpose context managers added to contextlib since its incarnation in >Python 2.5 (although there have been many various domain specific >context manager additions elsewhere in the standard library). There's a fundamental conceptual shift here that's worth exploring more, and which I think was first identified by RDM. Until now, context managers were at their heart (at least IMHO) about managing "resources". A general resource might be an open file, or it might be a database transaction, or even the current working directory. Context managers (as expressed elegantly by the `with` statement) are used to ensure that a resource acquired for some limited operation is - to Python's best ability - "released" at the end of that operation, no matter what happens. E.g. the file is closed even if there's a write error, or the current working directory is restored to its original location. We need only look at the typical @contextmanager use to see the idiom they embody. As shown in the docstring: @contextmanager def acquire(): resource = get_some_resource() try: yield # execute the operation finally: # No matter what happened above... resource.free() redirect_stdout() conforms to this fine tradition, with the resource being sys.stdout. suppress() breaks the mold, which I think is what is breaking people's brains. It isn't about guaranteeing that a resource is restored to its original value after some operation. It's about short circuiting that operation. Just look at the implementation to see this shift in perspective. It doesn't use try/finally, it uses try/except. So it's important to acknowledge that suppress() is charting new territory and it will take some exploration and experimentation to get used to, or maybe even to decide whether it's a good idea. It'll be interesting to see whether this fundamental difference is easily explained, understood, and internalized and that will go a long way to figuring out whether this is a good idea to be expanded on in the future. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ncoghlan at gmail.com Thu Oct 17 18:25:26 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 18 Oct 2013 02:25:26 +1000 Subject: [Python-Dev] Help needed: problems getting the expected default encoding for embedded IO config test Message-ID: http://bugs.python.org/issue16129 added a Py_SetStandardStreamEncoding API to allow embedding applications like Blender to explicitly configure the encoding and error handler for the standard IO streams. However, the test case didn't get along with the buildbots, and I'm struggling to figure out why. To deal with the possible variation in the default encoding across systems, I use sys.std*.encoding to work out the expected default encoding for the streams. However, as you can see in an example like http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.x/builds/579/steps/test/logs/stdio, this is failing, with the encoding in the *expected* data being None, while the values from the subprocess actually look right. In the "re-run at the end" case, it all works fine, so it doesn't appear to be anything inherently wrong with the affected systems. The test pass without any problems on my own system (otherwise I wouldn't have checked it in). Running the full test suite the same way the buildbots do also worked fine locally. I've marked the offending test as skipped for now (and reopened the tracker issue), and will look into it more tomorrow. but figured I'd mention it here in case anyone had any good suggestions (the next thing on my to try list is running with the hash seed from one of the failing buildbots, in case it's a cross-test interference problem). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From cf.natali at gmail.com Thu Oct 17 18:35:13 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Thu, 17 Oct 2013 18:35:13 +0200 Subject: [Python-Dev] cpython: Try doing a raw test of os.fork()/os.kill(). In-Reply-To: <20131017173814.570ab9b2@fsol> References: <3d0rsp05P1z7Ljs@mail.python.org> <20131017173814.570ab9b2@fsol> Message-ID: 2013/10/17 Antoine Pitrou : > On Thu, 17 Oct 2013 15:33:02 +0200 (CEST) > richard.oudkerk wrote: >> http://hg.python.org/cpython/rev/9558e9360afc >> changeset: 86401:9558e9360afc >> parent: 86399:9cd88b39ef62 >> user: Richard Oudkerk >> date: Thu Oct 17 14:24:06 2013 +0100 >> summary: >> Try doing a raw test of os.fork()/os.kill(). > > For this kind of ad-hoc testing, you can also use a custom builder to > avoid disrupting the main source tree: AFAICT, the problem he's trying to debug (issue #19227) only occurs on two specific - stable - buildbots. cf From ethan at stoneleaf.us Thu Oct 17 18:05:48 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 17 Oct 2013 09:05:48 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> Message-ID: <52600ADC.9070801@stoneleaf.us> On 10/17/2013 08:26 AM, Nick Coghlan wrote: > > # Arbitrarily nested search loop > with exit_label() as found: > for i in range(x): > for j in range(y): > if matches(i, j): > found.exit((i, j)) > if found: > print(found.value) +1 ! > @property > def exc(self): > if self._exc is self._exc: typo Thanks for this, I like it! :) -- ~Ethan~ From ethan at stoneleaf.us Thu Oct 17 17:52:42 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 17 Oct 2013 08:52:42 -0700 Subject: [Python-Dev] Bug tracker and docs@python In-Reply-To: References: <525FF9AE.2070905@stoneleaf.us> Message-ID: <526007CA.9000104@stoneleaf.us> On 10/17/2013 08:29 AM, Benjamin Peterson wrote: > 2013/10/17 Ethan Furman : >> Good day! >> >> What is the effect of assigning a bug tracker issue to docs at python? > > The assignee field is set to docs at python.org. > > In other words, nothing magical happens. Heh, okay. So commit the doc patch myself, then. :) So when should I assign something to docs at python? -- ~Ethan~ From solipsis at pitrou.net Thu Oct 17 18:46:00 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 17 Oct 2013 18:46:00 +0200 Subject: [Python-Dev] Bug tracker and docs@python References: <525FF9AE.2070905@stoneleaf.us> <526007CA.9000104@stoneleaf.us> Message-ID: <20131017184600.43e3ec52@fsol> On Thu, 17 Oct 2013 08:52:42 -0700 Ethan Furman wrote: > On 10/17/2013 08:29 AM, Benjamin Peterson wrote: > > 2013/10/17 Ethan Furman : > >> Good day! > >> > >> What is the effect of assigning a bug tracker issue to docs at python? > > > > The assignee field is set to docs at python.org. > > > > In other words, nothing magical happens. > > Heh, okay. So commit the doc patch myself, then. :) > > So when should I assign something to docs at python? You don't. You add the issue to the "Documentation" category and it will get assigned automatically. But anybody can of course commit doc patches. (docs at python.org is a mailing-list, AFAIK) Regards Antoine. From shibturn at gmail.com Thu Oct 17 19:21:37 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Thu, 17 Oct 2013 18:21:37 +0100 Subject: [Python-Dev] cpython: Try doing a raw test of os.fork()/os.kill(). In-Reply-To: References: <3d0rsp05P1z7Ljs@mail.python.org> <20131017173814.570ab9b2@fsol> Message-ID: <52601CA1.7010605@gmail.com> On 17/10/2013 5:35pm, Charles-Fran?ois Natali wrote: >> For this kind of ad-hoc testing, you can also use a custom builder to >> avoid disrupting the main source tree: > > AFAICT, the problem he's trying to debug (issue #19227) only occurs on > two specific - stable - buildbots. It looks like there are corresponding custom builders, so I will try using those. -- Richard From shibturn at gmail.com Thu Oct 17 19:21:37 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Thu, 17 Oct 2013 18:21:37 +0100 Subject: [Python-Dev] cpython: Try doing a raw test of os.fork()/os.kill(). In-Reply-To: References: <3d0rsp05P1z7Ljs@mail.python.org> <20131017173814.570ab9b2@fsol> Message-ID: <52601CA1.7010605@gmail.com> On 17/10/2013 5:35pm, Charles-Fran?ois Natali wrote: >> For this kind of ad-hoc testing, you can also use a custom builder to >> avoid disrupting the main source tree: > > AFAICT, the problem he's trying to debug (issue #19227) only occurs on > two specific - stable - buildbots. It looks like there are corresponding custom builders, so I will try using those. -- Richard From victor.stinner at gmail.com Thu Oct 17 19:27:44 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 17 Oct 2013 19:27:44 +0200 Subject: [Python-Dev] Help needed: problems getting the expected default encoding for embedded IO config test In-Reply-To: References: Message-ID: sys.stdout.encoding must never be None, it must be a str. If it is None, it is a regression. I modified Python (maybe in 3.2) to ensure that .encoding is always set. For your failure: what is the locale encoding? What are the values of LC_ALL, LANG, LC_CTYPES and PYTHONIOENCODING env vars? I will try to reproduce the issue. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Oct 17 19:28:00 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 17 Oct 2013 10:28:00 -0700 Subject: [Python-Dev] Regarding stdlib socket module, _fileobject.flush() method using ._rbufsize instead of ._wbufsize In-Reply-To: <20131017122331.0bb85077@fsol> References: <20131017122331.0bb85077@fsol> Message-ID: So that's hex rev dca3cb19dbdd. I'm confident that it is a copy/paste bug. It doesn't have any effect AFAICT though, since given the analysis the OP posted, max(self._wbufsize, self.bufsize) and max(self._rbufsize, self.bufsize) always come out to the same. Additionally, the bug is no longer present in Python 3. On Thu, Oct 17, 2013 at 3:23 AM, Antoine Pitrou wrote: > On Wed, 16 Oct 2013 21:30:52 -0400 > Terry Reedy wrote: > > > On 10/16/2013 5:01 PM, Peter Portante wrote: > > > Hello, > > > > > > Is there a reason why the stdlib socket module _fileobject.flush() > > > method is using ._rbufsize instead of ._wbufsize at line 297 (Python > > > 2.7.3), where it determines the buffer_size value to be used for > > > _sock.sendall()? Does anybody know the history behind this? > > > > Annotations and history show that line is from > > 61875, 2010-5-23, J P Calderone, Merged revisions 74426... > > I do not understand the message. > > When mentioning Mercurial changesets, please post the > (hexadecimal) changeset id, not the revision number, since revision > numbers are not stable accross clones. > http://mercurial.selenic.com/wiki/RevisionNumber > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Thu Oct 17 19:28:11 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 17 Oct 2013 10:28:11 -0700 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: <20131017120620.70dfa755@anarchist> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> Message-ID: <52601E2B.4010906@g.nevcal.com> On 10/17/2013 9:06 AM, Barry Warsaw wrote: > There's a fundamental conceptual shift here that's worth exploring more, and > which I think was first identified by RDM. > > Until now, context managers were at their heart (at least IMHO) about managing > "resources". A general resource might be an open file, or it might be a > database transaction, or even the current working directory. Context managers > (as expressed elegantly by the `with` statement) are used to ensure that a > resource acquired for some limited operation is - to Python's best ability - > "released" at the end of that operation, no matter what happens. E.g. the > file is closed even if there's a write error, or the current working directory > is restored to its original location. The conceptual shift exists, as you two have pointed out. But that is mostly because Structured programming and OO have created sufficient conceptual idioms that people have forgotten that all of these idioms are to avoid bare gotos. The real purpose of all such constructs is to manage the instruction pointer, and context managers, while useful for managing "resources" in the abstract, are truly about managing the control flow -- hopefully in understandable ways. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-dev at masklinn.net Thu Oct 17 20:40:56 2013 From: python-dev at masklinn.net (Xavier Morel) Date: Thu, 17 Oct 2013 20:40:56 +0200 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: <20131017120620.70dfa755@anarchist> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> Message-ID: On 2013-10-17, at 18:06 , Barry Warsaw wrote: > On Oct 18, 2013, at 01:26 AM, Nick Coghlan wrote: >> By contrast, suppress() and redirect_stdout() are the *first* general >> purpose context managers added to contextlib since its incarnation in >> Python 2.5 (although there have been many various domain specific >> context manager additions elsewhere in the standard library). > > There's a fundamental conceptual shift here that's worth exploring more, and > which I think was first identified by RDM. > > Until now, context managers were at their heart (at least IMHO) about managing > "resources". A general resource might be an open file, or it might be a > database transaction, or even the current working directory. Context managers > (as expressed elegantly by the `with` statement) are used to ensure that a > resource acquired for some limited operation is - to Python's best ability - > "released" at the end of that operation, no matter what happens. E.g. the > file is closed even if there's a write error, or the current working directory > is restored to its original location. I think there's already a significant split between context managers which handle the lifecycle of a local resource (file, transaction) and those which purport to locally alter global-ish state (cwd, decimal.localcontext, logging.captureWarnings, redirect_stdout). And the latter worries me (much more than the very localized behavior of suppress) because I don't see any way to implement them safely and correctly when mixing it with coroutines in today's Python (some of them aren't even thread-safe), all of that while I expect coroutines will see significantly more use in the very near future with yield from and tulip's promotion of coroutine-style async. > Just look at the implementation to see this shift in perspective. It doesn't > use try/finally, it uses try/except. Transaction CMs will also use try/except: @contextmanager def transaction(): resource = open_tnx() try: yield resource resource.commit() except: resource.rollback() raise From barry at python.org Thu Oct 17 20:48:59 2013 From: barry at python.org (Barry Warsaw) Date: Thu, 17 Oct 2013 14:48:59 -0400 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> Message-ID: <20131017144859.54fba044@anarchist> On Oct 17, 2013, at 08:40 PM, Xavier Morel wrote: >I think there's already a significant split between context managers >which handle the lifecycle of a local resource (file, transaction) and >those which purport to locally alter global-ish state (cwd, >decimal.localcontext, logging.captureWarnings, redirect_stdout). > >And the latter worries me (much more than the very localized behavior of >suppress) because I don't see any way to implement them safely and >correctly when mixing it with coroutines in today's Python (some of them >aren't even thread-safe), all of that while I expect coroutines will see >significantly more use in the very near future with yield from and >tulip's promotion of coroutine-style async. Although slightly orthogonal to my previous distinction, this is a great point. Anything affecting global state will have this problem of course. It's definitely something to keep thinking about. >> Just look at the implementation to see this shift in perspective. It >> doesn't use try/finally, it uses try/except. > >Transaction CMs will also use try/except: > > @contextmanager > def transaction(): > resource = open_tnx() > try: > yield resource > resource.commit() > except: > resource.rollback() > raise True. I'm sure lots of us have code almost exactly like this. ;) For me, it's closer to the original intent though because the bare-except combined with the re-raising of the exception feels kind of like a finally. In both exit paths, the resource is being released, it's just that you have to know which "release" operation to perform. Cheers, -Barry From oscar.j.benjamin at gmail.com Thu Oct 17 20:55:38 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 17 Oct 2013 19:55:38 +0100 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> Message-ID: On 17 October 2013 19:40, Xavier Morel wrote: > I think there's already a significant split between context managers > which handle the lifecycle of a local resource (file, transaction) and > those which purport to locally alter global-ish state (cwd, > decimal.localcontext, logging.captureWarnings, redirect_stdout). > > And the latter worries me (much more than the very localized behavior of > suppress) because I don't see any way to implement them safely and > correctly when mixing it with coroutines in today's Python (some of them > aren't even thread-safe), all of that while I expect coroutines will see > significantly more use in the very near future with yield from and > tulip's promotion of coroutine-style async. I maybe misunderstanding how the coroutine-style async works but I would have thought that it would be as simple as: don't use global-state-restoring-context-managers around statements that yield control (it would be simpler if there was a good term for describing that kind of CM). That's simpler to implement and computationally cheaper than e.g. the thread-local state used by the decimal module. Oscar From guido at python.org Thu Oct 17 21:01:30 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 17 Oct 2013 12:01:30 -0700 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> Message-ID: On Thu, Oct 17, 2013 at 11:55 AM, Oscar Benjamin wrote: > On 17 October 2013 19:40, Xavier Morel wrote: > > I think there's already a significant split between context managers > > which handle the lifecycle of a local resource (file, transaction) and > > those which purport to locally alter global-ish state (cwd, > > decimal.localcontext, logging.captureWarnings, redirect_stdout). > > > > And the latter worries me (much more than the very localized behavior of > > suppress) because I don't see any way to implement them safely and > > correctly when mixing it with coroutines in today's Python (some of them > > aren't even thread-safe), all of that while I expect coroutines will see > > significantly more use in the very near future with yield from and > > tulip's promotion of coroutine-style async. > > I maybe misunderstanding how the coroutine-style async works but I > would have thought that it would be as simple as: don't use > global-state-restoring-context-managers around statements that yield > control (it would be simpler if there was a good term for describing > that kind of CM). That's simpler to implement and computationally > cheaper than e.g. the thread-local state used by the decimal module. > Context managers that actually save and restore *global* state are already not thread-safe, so concluding they are also not coroutine-safe (or task-safe?) seems a small step. I'd be more worried about context manager that use thread-local state -- there is no similar concept in Tulip. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-dev at masklinn.net Thu Oct 17 21:30:04 2013 From: python-dev at masklinn.net (Xavier Morel) Date: Thu, 17 Oct 2013 21:30:04 +0200 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> Message-ID: <4B0C945B-9E28-406A-ADE1-FF57C1BFBAAE@masklinn.net> On 2013-10-17, at 20:55 , Oscar Benjamin wrote: > On 17 October 2013 19:40, Xavier Morel wrote: >> I think there's already a significant split between context managers >> which handle the lifecycle of a local resource (file, transaction) and >> those which purport to locally alter global-ish state (cwd, >> decimal.localcontext, logging.captureWarnings, redirect_stdout). >> >> And the latter worries me (much more than the very localized behavior of >> suppress) because I don't see any way to implement them safely and >> correctly when mixing it with coroutines in today's Python (some of them >> aren't even thread-safe), all of that while I expect coroutines will see >> significantly more use in the very near future with yield from and >> tulip's promotion of coroutine-style async. > > I maybe misunderstanding how the coroutine-style async works but I > would have thought that it would be as simple as: don't use > global-state-restoring-context-managers around statements that yield > control You have to know which contextmanagers to what and how, and avoid them in these specific situations. I'm really bothered by these being unsafe by default. Technically they're already broken but the chance of them being used in such a context are low, whereas it wouldn't be unexpected for a user to e.g. create a local decimal context in a coroutine and *not* expect that local context to affect other coroutines running concurrently. > Context managers that actually save and restore *global* state are already > not thread-safe Hence my use of "global-ish", a global is process-local after all. A threadlocal is greenlet-global (for the greenlets running in that thread), and a greenlet-local is coroutine-global (although I don't expect multiplexing coroutines inside greenlets to be common, the natural consequence is that threadlocals are also coroutine-global). > , so concluding they are also not coroutine-safe (or > task-safe?) seems a small step. Yes, but not necessarily a step most users will remember to take, and of course the lack of thread-safety must be documented and noticed (as with warnings.catch_warnings[0]). Although I agree that > I'd be more worried about context manager that use thread-local state -- > there is no similar concept in Tulip. indeed. [0] not logging.captureWarnings() which I previously wrote, captureWarnings is not a context manager From tjreedy at udel.edu Thu Oct 17 22:03:31 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 17 Oct 2013 16:03:31 -0400 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: <20131017120620.70dfa755@anarchist> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> Message-ID: On 10/17/2013 12:06 PM, Barry Warsaw wrote: > On Oct 18, 2013, at 01:26 AM, Nick Coghlan wrote: > >> By contrast, suppress() and redirect_stdout() are the *first* general >> purpose context managers added to contextlib since its incarnation in >> Python 2.5 (although there have been many various domain specific >> context manager additions elsewhere in the standard library). > > There's a fundamental conceptual shift here that's worth exploring more, I noticed the same thing upon reading that. With statements are a general special-purpose tool. > and which I think was first identified by RDM. I missed this at the time. > Until now, context managers were at their heart (at least IMHO) about managing > "resources". ... > We need only look at the typical @contextmanager use to see the idiom they > embody. As shown in the docstring: > > @contextmanager > def acquire(): > resource = get_some_resource() > try: > yield # execute the operation > finally: > # No matter what happened above... > resource.free() > > redirect_stdout() conforms to this fine tradition, with the resource being > sys.stdout. From the with statement doc, second sentence: "This allows common try...except...finally usage patterns to be encapsulated for convenient reuse." > suppress() breaks the mold, which I think is what is breaking people's > brains. It isn't about guaranteeing that a resource is restored to its > original value after some operation. It's about short circuiting that > operation. A suppress() is, in a sense, an empty cm as its only effect is to exploit the following: "If the suite was exited due to an exception, and the return value from the __exit__() method was false, the exception is reraised. If the return value was true, the exception is suppressed, and execution continues with the statement following the with statement." This is clear with a direct implementation instead of the magical indirect one that Nick used (with @contextmanager and a generator function). I believe the following is equivalent, and one line shorter. class suppress: def __init__(self, *exceptions): self.exceptions = exceptions def __exit__(self, etype, eval, etrace): return etype in self.exceptions One might consider this an abuse, like comprehensions written purely for their side effect. > Just look at the implementation to see this shift in perspective. It doesn't > use try/finally, it uses try/except. > > So it's important to acknowledge that suppress() is charting new territory and > it will take some exploration and experimentation to get used to, or maybe > even to decide whether it's a good idea. It'll be interesting to see whether > this fundamental difference is easily explained, understood, and internalized > and that will go a long way to figuring out whether this is a good idea to be > expanded on in the future. -- Terry Jan Reedy From stefan at bytereef.org Thu Oct 17 22:05:24 2013 From: stefan at bytereef.org (Stefan Krah) Date: Thu, 17 Oct 2013 22:05:24 +0200 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: <4B0C945B-9E28-406A-ADE1-FF57C1BFBAAE@masklinn.net> References: <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> <4B0C945B-9E28-406A-ADE1-FF57C1BFBAAE@masklinn.net> Message-ID: <20131017200524.GA3360@sleipnir.bytereef.org> Xavier Morel wrote: > On 2013-10-17, at 20:55 , Oscar Benjamin wrote: > > On 17 October 2013 19:40, Xavier Morel wrote: > >> I think there's already a significant split between context managers > >> which handle the lifecycle of a local resource (file, transaction) and > >> those which purport to locally alter global-ish state (cwd, > >> decimal.localcontext, logging.captureWarnings, redirect_stdout). Well, naming is difficult. In the case of decimal it's a temporary global but also thread-local context. > You have to know which contextmanagers to what and how, and avoid them > in these specific situations. I'm really bothered by these being unsafe > by default. Technically they're already broken but the chance of them > being used in such a context are low, whereas it wouldn't be unexpected > for a user to e.g. create a local decimal context in a coroutine and > *not* expect that local context to affect other coroutines running > concurrently. I don't think localcontext() is broken. The fine manual is quite clear that "current contexts" are per thread. The localcontext() docs explicitly say that the "current context" is swapped. Of course there are multiple ways to use individual contexts for each coroutine. Use the context methods or pass the context to the Decimal methods. Stefan Krah From greg at krypto.org Thu Oct 17 22:16:50 2013 From: greg at krypto.org (Gregory P. Smith) Date: Thu, 17 Oct 2013 13:16:50 -0700 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: <20131017120620.70dfa755@anarchist> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> Message-ID: On Thu, Oct 17, 2013 at 9:06 AM, Barry Warsaw wrote: > On Oct 18, 2013, at 01:26 AM, Nick Coghlan wrote: > > >By contrast, suppress() and redirect_stdout() are the *first* general > >purpose context managers added to contextlib since its incarnation in > >Python 2.5 (although there have been many various domain specific > >context manager additions elsewhere in the standard library). > > There's a fundamental conceptual shift here that's worth exploring more, > and > which I think was first identified by RDM. > > Until now, context managers were at their heart (at least IMHO) about > managing > "resources". A general resource might be an open file, or it might be a > database transaction, or even the current working directory. Context > managers > (as expressed elegantly by the `with` statement) are used to ensure that a > resource acquired for some limited operation is - to Python's best ability > - > "released" at the end of that operation, no matter what happens. E.g. the > file is closed even if there's a write error, or the current working > directory > is restored to its original location. > > We need only look at the typical @contextmanager use to see the idiom they > embody. As shown in the docstring: > > @contextmanager > def acquire(): > resource = get_some_resource() > try: > yield # execute the operation > finally: > # No matter what happened above... > resource.free() > > redirect_stdout() conforms to this fine tradition, with the resource being > sys.stdout. > > suppress() breaks the mold, which I think is what is breaking people's > brains. It isn't about guaranteeing that a resource is restored to its > original value after some operation. It's about short circuiting that > operation. > > Just look at the implementation to see this shift in perspective. It > doesn't > use try/finally, it uses try/except. > > So it's important to acknowledge that suppress() is charting new territory > and > it will take some exploration and experimentation to get used to, or maybe > even to decide whether it's a good idea. It'll be interesting to see > whether > this fundamental difference is easily explained, understood, and > internalized > and that will go a long way to figuring out whether this is a good idea to > be > expanded on in the future. > Is this shift to do non-resourcey-things really new? In 2009 when doing *great things* to the unittest module for the upcoming Python 2.7 we (michael foord I believe) turned the previous None return value of assertRaises into a context manager so that it could be used in a non-resource-management manner similar to: with self.assertRaises(SomeError) as context: state = set_something_up() call_offending_thing_on_it(state) self.assertIn('Bloody Murder!', context.exception.msg) I thought this was great. As did a bunch of other people in the room. Thus we have it and happily use it. It certainly is _not at all_ how I had ever been thinking of context managers before this feature was added. I had pigeon holed them into resource management tasks in my mind rather than thinking of them as a replacement for try..except syntax in common cases. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Oct 17 22:11:10 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 17 Oct 2013 13:11:10 -0700 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> Message-ID: <5260445E.6060303@stoneleaf.us> On 10/17/2013 01:03 PM, Terry Reedy wrote: > > class suppress: > def __init__(self, *exceptions): > self.exceptions = exceptions > def __exit__(self, etype, eval, etrace): > return etype in self.exceptions This fails when etype is a subclass of the exceptions, as mentioned in the original issue. -- ~Ethan~ From catch-all at masklinn.net Thu Oct 17 22:57:22 2013 From: catch-all at masklinn.net (Xavier Morel) Date: Thu, 17 Oct 2013 22:57:22 +0200 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: <5260445E.6060303@stoneleaf.us> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> <5260445E.6060303@stoneleaf.us> Message-ID: <875850B8-2182-4885-9782-DCABFC8865D7@masklinn.net> On 2013-10-17, at 22:11 , Ethan Furman wrote: > On 10/17/2013 01:03 PM, Terry Reedy wrote: >> >> class suppress: >> def __init__(self, *exceptions): >> self.exceptions = exceptions >> def __exit__(self, etype, eval, etrace): >> return etype in self.exceptions > > This fails when etype is a subclass of the exceptions, as mentioned in the original issue. That's fixed by using `issubclass` and does not infirm Terry's point does it? From ethan at stoneleaf.us Thu Oct 17 23:24:29 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 17 Oct 2013 14:24:29 -0700 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: <875850B8-2182-4885-9782-DCABFC8865D7@masklinn.net> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> <5260445E.6060303@stoneleaf.us> <875850B8-2182-4885-9782-DCABFC8865D7@masklinn.net> Message-ID: <5260558D.7020109@stoneleaf.us> On 10/17/2013 01:57 PM, Xavier Morel wrote: > On 2013-10-17, at 22:11 , Ethan Furman wrote: > >> On 10/17/2013 01:03 PM, Terry Reedy wrote: >>> >>> class suppress: >>> def __init__(self, *exceptions): >>> self.exceptions = exceptions >>> def __exit__(self, etype, eval, etrace): >>> return etype in self.exceptions >> >> This fails when etype is a subclass of the exceptions, as mentioned in the original issue. > > That's fixed by using `issubclass` and does not infirm Terry's point does it? Nope, and thanks for posting the correct verbage. -- ~Ethan~ From a.badger at gmail.com Thu Oct 17 23:46:26 2013 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 17 Oct 2013 14:46:26 -0700 Subject: [Python-Dev] non-US zip archives support in zipfile.py In-Reply-To: <525D4727.6020005@v.loewis.de> References: <20131015120209.GA7989@ando> <525D4727.6020005@v.loewis.de> Message-ID: <20131017214626.GN27527@unaka.lan> On Tue, Oct 15, 2013 at 03:46:15PM +0200, "Martin v. L?wis" wrote: > Am 15.10.13 14:49, schrieb Daniel Holth: > > It is part of the ZIP specification. CP437 or UTF-8 are the two > > official choices, but other encodings happen on Russian, Japanese > > systems. > > Indeed. Formally, the other encodings are not supported by the > ZIP specification, and are thus formally misuse of the format. > But the tools in the wild misuse the format in this manner. CP437 can encode any byte so zip and unzip on Linux, for instance, take the bytes that represent the filename on the filesystem and use those in the zip file without setting the utf-8 flag. When the files are extracted, the same byte sequence are used as the filename for the new files. > I believe (without having proof) that early versions of the > specification failed to discuss the file name encoding at all, > These might be helpful: No mention of file name encodings in this version of the spec: http://www.pkware.com/documents/APPNOTE/APPNOTE-6.2.2.TXT Appendix D, Language Encoding, shows up here: http://www.pkware.com/documents/APPNOTE/APPNOTE-6.3.0.TXT (Most recent version is 6.3.2) > making people believe that it is unspecified and always the > system encoding (which is useless, of course, as you create > zip files to move them across systems). > Not always. Backups are another use. Also it's not useless. If the files are being moved within an organization (or in some cases geographical regions have standardized on an encoding in practice), the same system encoding could very well be in use on the machines where the files end up. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From oscar.j.benjamin at gmail.com Fri Oct 18 00:51:03 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 17 Oct 2013 23:51:03 +0100 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> Message-ID: On 17 October 2013 20:01, Guido van Rossum wrote: > On Thu, Oct 17, 2013 at 11:55 AM, Oscar Benjamin > wrote: >> >> On 17 October 2013 19:40, Xavier Morel wrote: >> > I think there's already a significant split between context managers >> > which handle the lifecycle of a local resource (file, transaction) and >> > those which purport to locally alter global-ish state (cwd, >> > decimal.localcontext, logging.captureWarnings, redirect_stdout). >> > >> > And the latter worries me (much more than the very localized behavior of >> > suppress) because I don't see any way to implement them safely and >> > correctly when mixing it with coroutines in today's Python (some of them >> > aren't even thread-safe), all of that while I expect coroutines will see >> > significantly more use in the very near future with yield from and >> > tulip's promotion of coroutine-style async. >> >> I maybe misunderstanding how the coroutine-style async works but I >> would have thought that it would be as simple as: don't use >> global-state-restoring-context-managers around statements that yield >> control (it would be simpler if there was a good term for describing >> that kind of CM). That's simpler to implement and computationally >> cheaper than e.g. the thread-local state used by the decimal module. > > Context managers that actually save and restore *global* state are already > not thread-safe, so concluding they are also not coroutine-safe (or > task-safe?) seems a small step. > > I'd be more worried about context manager that use thread-local state -- > there is no similar concept in Tulip. It's unnecessary in Tulip. The need for thread-local state in e.g. decimal contexts is driven by the fact that multi-threaded execution switches in an uncontrollable way. Tulip specifically makes it possible to control the points at which a switch occurs making this safe (even if localcontext() wasn't thread-safe): with decimal.localcontext() as ctx: ctx.prec = 100 c = a + b # more synchronous decimal calculations # State is restored before allowing other code to execute yield from save_in_database(c) So it's fine to use global/thread-local state modifying/restoring context managers in Tulip as long as you don't yield control to other code within the with block. (unless I misunderstand - I lost track of Tulip some time ago). The issue with decimal.localcontext() and yield arises when using generators as much as coroutines e.g.: def exact_sum(nums): start = Decimal(0) with decimal.localcontext() as ctx: ctx.traps[decimal.Inexact] = True for num in nums: try: total += Decimal(num) except decimal.Inexact: ctx.prec *= 2 return total The above is fine for computing the sum of a list of Decimals/ints/floats. However it fails if you pass in a generator that carelessly modifies the arithmetic context around yield calls: def compute(): with decimal.localcontext() as ctx: ctx.prec = 15 ctx.traps[decimal.Inexact] = False yield a + b yield b - c # etc. exact_sum(compute()) There needs to be a convention that either functions like exact_sum() mustn't assume continuity of the context between iterations or a function like compute() must restore before yielding. IMO the only sane approach for async coroutines is to say that if you yield or yield from then it is your responsibility to restore any temporarily altered global/thread-local state first. Oscar From guido at python.org Fri Oct 18 01:02:28 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 17 Oct 2013 16:02:28 -0700 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> Message-ID: On Thu, Oct 17, 2013 at 3:51 PM, Oscar Benjamin wrote: > On 17 October 2013 20:01, Guido van Rossum wrote: > > On Thu, Oct 17, 2013 at 11:55 AM, Oscar Benjamin > > wrote: > >> > >> On 17 October 2013 19:40, Xavier Morel wrote: > >> > I think there's already a significant split between context managers > >> > which handle the lifecycle of a local resource (file, transaction) and > >> > those which purport to locally alter global-ish state (cwd, > >> > decimal.localcontext, logging.captureWarnings, redirect_stdout). > >> > > >> > And the latter worries me (much more than the very localized behavior > of > >> > suppress) because I don't see any way to implement them safely and > >> > correctly when mixing it with coroutines in today's Python (some of > them > >> > aren't even thread-safe), all of that while I expect coroutines will > see > >> > significantly more use in the very near future with yield from and > >> > tulip's promotion of coroutine-style async. > >> > >> I maybe misunderstanding how the coroutine-style async works but I > >> would have thought that it would be as simple as: don't use > >> global-state-restoring-context-managers around statements that yield > >> control (it would be simpler if there was a good term for describing > >> that kind of CM). That's simpler to implement and computationally > >> cheaper than e.g. the thread-local state used by the decimal module. > > > > Context managers that actually save and restore *global* state are > already > > not thread-safe, so concluding they are also not coroutine-safe (or > > task-safe?) seems a small step. > > > > I'd be more worried about context manager that use thread-local state -- > > there is no similar concept in Tulip. > > It's unnecessary in Tulip. The need for thread-local state in e.g. > decimal contexts is driven by the fact that multi-threaded execution > switches in an uncontrollable way. Tulip specifically makes it > possible to control the points at which a switch occurs making this > safe (even if localcontext() wasn't thread-safe): > > with decimal.localcontext() as ctx: > ctx.prec = 100 > c = a + b > # more synchronous decimal calculations > > # State is restored before allowing other code to execute > yield from save_in_database(c) > > So it's fine to use global/thread-local state modifying/restoring > context managers in Tulip as long as you don't yield control to other > code within the with block. (unless I misunderstand - I lost track of > Tulip some time ago). > You've got it exactly right. > The issue with decimal.localcontext() and yield arises when using > generators as much as coroutines e.g.: > > def exact_sum(nums): > start = Decimal(0) > with decimal.localcontext() as ctx: > ctx.traps[decimal.Inexact] = True > for num in nums: > try: > total += Decimal(num) > except decimal.Inexact: > ctx.prec *= 2 > return total > > The above is fine for computing the sum of a list of > Decimals/ints/floats. However it fails if you pass in a generator that > carelessly modifies the arithmetic context around yield calls: > > def compute(): > with decimal.localcontext() as ctx: > ctx.prec = 15 > ctx.traps[decimal.Inexact] = False > yield a + b > yield b - c > # etc. > > exact_sum(compute()) > > There needs to be a convention that either functions like exact_sum() > mustn't assume continuity of the context between iterations or a > function like compute() must restore before yielding. IMO the only > sane approach for async coroutines is to say that if you yield or > yield from then it is your responsibility to restore any temporarily > altered global/thread-local state first. > Right again. The simplest rule to remember seems to be "don't use yield or yield-from inside a with-statement". You can relax it by limiting it to context managers that manage any kind of shared resource, but that is probably already too subtle: e.g. yielding inside "with open(file) as f" seems fine, but yielding inside "with lock" is problematic, since the other side might try to acquire the same lock, and deadlock. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Fri Oct 18 00:40:35 2013 From: larry at hastings.org (Larry Hastings) Date: Thu, 17 Oct 2013 15:40:35 -0700 Subject: [Python-Dev] The Tulip Has Landed Message-ID: <52606763.70006@hastings.org> For those interested parties: Guido just checked "asyncio", aka "Tulip", aka "PEP 3156", in to trunk. I expect it to be part of Python 3.4.0a4, hopefully to be released this weekend. Maybe /that'll/ generate some interest in the alpha ;-) //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Oct 18 01:12:28 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 18 Oct 2013 09:12:28 +1000 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> Message-ID: On 18 Oct 2013 09:05, "Guido van Rossum" wrote: > > On Thu, Oct 17, 2013 at 3:51 PM, Oscar Benjamin < oscar.j.benjamin at gmail.com> wrote: >> >> On 17 October 2013 20:01, Guido van Rossum wrote: >> > On Thu, Oct 17, 2013 at 11:55 AM, Oscar Benjamin >> > wrote: >> >> >> >> On 17 October 2013 19:40, Xavier Morel wrote: >> >> > I think there's already a significant split between context managers >> >> > which handle the lifecycle of a local resource (file, transaction) and >> >> > those which purport to locally alter global-ish state (cwd, >> >> > decimal.localcontext, logging.captureWarnings, redirect_stdout). >> >> > >> >> > And the latter worries me (much more than the very localized behavior of >> >> > suppress) because I don't see any way to implement them safely and >> >> > correctly when mixing it with coroutines in today's Python (some of them >> >> > aren't even thread-safe), all of that while I expect coroutines will see >> >> > significantly more use in the very near future with yield from and >> >> > tulip's promotion of coroutine-style async. >> >> >> >> I maybe misunderstanding how the coroutine-style async works but I >> >> would have thought that it would be as simple as: don't use >> >> global-state-restoring-context-managers around statements that yield >> >> control (it would be simpler if there was a good term for describing >> >> that kind of CM). That's simpler to implement and computationally >> >> cheaper than e.g. the thread-local state used by the decimal module. >> > >> > Context managers that actually save and restore *global* state are already >> > not thread-safe, so concluding they are also not coroutine-safe (or >> > task-safe?) seems a small step. >> > >> > I'd be more worried about context manager that use thread-local state -- >> > there is no similar concept in Tulip. >> >> It's unnecessary in Tulip. The need for thread-local state in e.g. >> decimal contexts is driven by the fact that multi-threaded execution >> switches in an uncontrollable way. Tulip specifically makes it >> possible to control the points at which a switch occurs making this >> safe (even if localcontext() wasn't thread-safe): >> >> with decimal.localcontext() as ctx: >> ctx.prec = 100 >> c = a + b >> # more synchronous decimal calculations >> >> # State is restored before allowing other code to execute >> yield from save_in_database(c) >> >> So it's fine to use global/thread-local state modifying/restoring >> context managers in Tulip as long as you don't yield control to other >> code within the with block. (unless I misunderstand - I lost track of >> Tulip some time ago). > > > You've got it exactly right. > >> >> The issue with decimal.localcontext() and yield arises when using >> generators as much as coroutines e.g.: >> >> def exact_sum(nums): >> start = Decimal(0) >> with decimal.localcontext() as ctx: >> ctx.traps[decimal.Inexact] = True >> for num in nums: >> try: >> total += Decimal(num) >> except decimal.Inexact: >> ctx.prec *= 2 >> return total >> >> The above is fine for computing the sum of a list of >> Decimals/ints/floats. However it fails if you pass in a generator that >> carelessly modifies the arithmetic context around yield calls: >> >> def compute(): >> with decimal.localcontext() as ctx: >> ctx.prec = 15 >> ctx.traps[decimal.Inexact] = False >> yield a + b >> yield b - c >> # etc. >> >> exact_sum(compute()) >> >> There needs to be a convention that either functions like exact_sum() >> mustn't assume continuity of the context between iterations or a >> function like compute() must restore before yielding. IMO the only >> sane approach for async coroutines is to say that if you yield or >> yield from then it is your responsibility to restore any temporarily >> altered global/thread-local state first. > > > Right again. The simplest rule to remember seems to be "don't use yield or yield-from inside a with-statement". You can relax it by limiting it to context managers that manage any kind of shared resource, but that is probably already too subtle: e.g. yielding inside "with open(file) as f" seems fine, but yielding inside "with lock" is problematic, since the other side might try to acquire the same lock, and deadlock. Even the file case can be tricky - if you actually yield *the file* (rather than nothing or its contents), then the recipient may be surprised when it is later closed automatically. It wouldn't surprise me if we see idioms to temporarily revert the effects of context managers start to grow in popularity once asyncio is released as part of 3.4. Cheers, Nick. > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Oct 18 01:35:23 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 18 Oct 2013 09:35:23 +1000 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: <875850B8-2182-4885-9782-DCABFC8865D7@masklinn.net> References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> <5260445E.6060303@stoneleaf.us> <875850B8-2182-4885-9782-DCABFC8865D7@masklinn.net> Message-ID: On 18 Oct 2013 06:59, "Xavier Morel" wrote: > > On 2013-10-17, at 22:11 , Ethan Furman wrote: > > > On 10/17/2013 01:03 PM, Terry Reedy wrote: > >> > >> class suppress: > >> def __init__(self, *exceptions): > >> self.exceptions = exceptions > >> def __exit__(self, etype, eval, etrace): > >> return etype in self.exceptions > > > > This fails when etype is a subclass of the exceptions, as mentioned in the original issue. > > That's fixed by using `issubclass` and does not infirm Terry's point does it? Yeah, it looks like it's worth switching to the class based implementation in this case. I guess I'm too accustomed to that being the more complex alternative, as I hadn't even tried it :) Cheers, Nick. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Fri Oct 18 01:35:27 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 18 Oct 2013 12:35:27 +1300 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> Message-ID: <5260743F.4020803@canterbury.ac.nz> Guido van Rossum wrote: > yielding inside "with open(file) as f" > seems fine, but yielding inside "with lock" is problematic, since the > other side might try to acquire the same lock, and deadlock. So maybe the documentation of a context manager should include whether it's safe to yield inside the block. -- Greg From ncoghlan at gmail.com Fri Oct 18 01:48:34 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 18 Oct 2013 09:48:34 +1000 Subject: [Python-Dev] The Tulip Has Landed In-Reply-To: <52606763.70006@hastings.org> References: <52606763.70006@hastings.org> Message-ID: On 18 Oct 2013 09:07, "Larry Hastings" wrote: > > > For those interested parties: Guido just checked "asyncio", aka "Tulip", aka "PEP 3156", in to trunk. I expect it to be part of Python 3.4.0a4, hopefully to be released this weekend. Huzzah! :) Cheers, Nick. > > Maybe that'll generate some interest in the alpha ;-) > > > /arry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Fri Oct 18 02:23:50 2013 From: greg at krypto.org (Gregory P. Smith) Date: Thu, 17 Oct 2013 17:23:50 -0700 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> Message-ID: > Right again. The simplest rule to remember seems to be "don't use yield or > yield-from inside a with-statement". You can relax it by limiting it to > context managers that manage any kind of shared resource, but that is > probably already too subtle: e.g. yielding inside "with open(file) as f" > seems fine, but yielding inside "with lock" is problematic, since the other > side might try to acquire the same lock, and deadlock. > This seems like the kind of thing that might warrant a pylint warning. Probably the more general case, requiring people to think about it and suppress the warning with an explanatory comment (or write the code in a different manner) when they are doing something known to be safe. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Oct 18 04:33:53 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 17 Oct 2013 22:33:53 -0400 Subject: [Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> <20131017120620.70dfa755@anarchist> <5260445E.6060303@stoneleaf.us> <875850B8-2182-4885-9782-DCABFC8865D7@masklinn.net> Message-ID: On 10/17/2013 7:35 PM, Nick Coghlan wrote: > > On 18 Oct 2013 06:59, "Xavier Morel" > wrote: > > > > On 2013-10-17, at 22:11 , Ethan Furman wrote: > > > > > On 10/17/2013 01:03 PM, Terry Reedy wrote: > > >> > > >> class suppress: > > >> def __init__(self, *exceptions): > > >> self.exceptions = exceptions > > >> def __exit__(self, etype, eval, etrace): > > >> return etype in self.exceptions > > > > > > This fails when etype is a subclass of the exceptions, as mentioned > in the original issue. There are two other failures; see below. I know better than to post code without testing or saying 'untested'; I just wish I would always remember... > > That's fixed by using `issubclass` and does not infirm Terry's point > does it? > > Yeah, it looks like it's worth switching to the class based > implementation in this case. I guess I'm too accustomed to that being > the more complex alternative, as I hadn't even tried it :) With two more changes (add __enter__, account for etype is None), the suppress tests pass. import unittest class suppress: def __init__(self, *exceptions): self.exceptions = exceptions def __enter__(self): pass def __exit__(self, etype, val, tb): return etype is None or issubclass(etype, self.exceptions) class Testsuppress(unittest.TestCase): def test_no_exception(self): with suppress(ValueError): self.assertEqual(pow(2, 5), 32) def test_exact_exception(self): with suppress(TypeError): len(5) def test_multiple_exception_args(self): with suppress(ZeroDivisionError, TypeError): len(5) def test_exception_hierarchy(self): with suppress(LookupError): 'Hello'[50] unittest.main() >>> Ran 4 tests...OK -- Terry Jan Reedy From larry at hastings.org Fri Oct 18 05:16:11 2013 From: larry at hastings.org (Larry Hastings) Date: Thu, 17 Oct 2013 20:16:11 -0700 Subject: [Python-Dev] Checking in Argument Clinic early? In-Reply-To: <525D6BDE.40408@hastings.org> References: <525D3B53.6000508@hastings.org> <525D6BDE.40408@hastings.org> Message-ID: <5260A7FB.4080900@hastings.org> On 10/15/2013 09:22 AM, Larry Hastings wrote: > If this love-in continues I'll prep a release tonight and commit it in > the morning... just before my flight home across the Atlantic. Okay, so, I slowed it down a little--some last-minute things cropped up. I've uploaded a fresh patch to the issue: http://bugs.python.org/issue16612 I plan to check it in about 24 hours from now, so it can be in trunk for most of a day before I tag for 3.4a4. If you'd like feedback at this late stage, please head to the issue above and go through the side-by-side diffs on Rietveld. And thanks for your interest! //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Fri Oct 18 07:19:45 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 18 Oct 2013 07:19:45 +0200 Subject: [Python-Dev] cpython: Issue #16129: Add `Py_SetStandardStreamEncoding` In-Reply-To: <20131017173640.53016af5@fsol> References: <3d0rd742bTz7LjR@mail.python.org> <20131017173640.53016af5@fsol> Message-ID: Am 17.10.2013 17:36, schrieb Antoine Pitrou: > On Thu, 17 Oct 2013 15:22:03 +0200 (CEST) > nick.coghlan wrote: >> >> +.. c:function:: int Py_SetStandardStreamEncoding(char *encoding, char *errors) >> + >> + .. index:: >> + single: Py_Initialize() >> + single: main() >> + triple: stdin; stdout; sdterr >> + >> + This function should be called before :c:func:`Py_Initialize`. It >> + specifies which encoding and error handling to use with standard io, >> + with the same meanings as in :func:`str.encode`. >> + >> + It overrides :envvar:`PYTHONIOENCODING` values, and allows embedding code >> + to control io encoding when the environment variable does not work. >> + >> + ``encoding`` and/or ``errors`` may be NULL to use >> + :envvar:`PYTHONIOENCODING` and/or default values (depending on other >> + settings). >> + >> + Note that :data:`sys.stderr` always uses the "backslashreplace" error >> + handler, regardless of this (or any other) setting. >> + >> + If :c:func:`Py_Finalize` is called, this function will need to be called >> + again in order to affect subsequent calls to :c:func:`Py_Initialize`. >> + >> + Returns 0 if successful. >> + >> + > > Needs a versionadded tag. Also, shouldn't it be excluded from the stable/limited API? Georg From hodgestar+pythondev at gmail.com Fri Oct 18 08:55:20 2013 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Fri, 18 Oct 2013 08:55:20 +0200 Subject: [Python-Dev] The Tulip Has Landed In-Reply-To: <52606763.70006@hastings.org> References: <52606763.70006@hastings.org> Message-ID: Yay! From p.f.moore at gmail.com Fri Oct 18 10:10:58 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 18 Oct 2013 09:10:58 +0100 Subject: [Python-Dev] The Tulip Has Landed In-Reply-To: <52606763.70006@hastings.org> References: <52606763.70006@hastings.org> Message-ID: On 17 October 2013 23:40, Larry Hastings wrote: > For those interested parties: Guido just checked "asyncio", aka "Tulip", aka > "PEP 3156", in to trunk. I expect it to be part of Python 3.4.0a4, > hopefully to be released this weekend. Cool! How often do the online docs get built? There's nothing showing in there yet (http://docs.python.org/3.4/library/index.html)... Paul From tjreedy at udel.edu Fri Oct 18 10:56:29 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 18 Oct 2013 04:56:29 -0400 Subject: [Python-Dev] The Tulip Has Landed In-Reply-To: References: <52606763.70006@hastings.org> Message-ID: On 10/18/2013 4:10 AM, Paul Moore wrote: > On 17 October 2013 23:40, Larry Hastings wrote: >> For those interested parties: Guido just checked "asyncio", aka "Tulip", aka >> "PEP 3156", in to trunk. I expect it to be part of Python 3.4.0a4, >> hopefully to be released this weekend. > > Cool! How often do the online docs get built? There's nothing showing > in there yet (http://docs.python.org/3.4/library/index.html)... I believe once every 24 hours. The current page is dated Oct 17 (at bottom). It is now Oct 18 most everywhere. -- Terry Jan Reedy From p.f.moore at gmail.com Fri Oct 18 11:50:38 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 18 Oct 2013 10:50:38 +0100 Subject: [Python-Dev] The Tulip Has Landed In-Reply-To: References: <52606763.70006@hastings.org> Message-ID: On 18 October 2013 09:56, Terry Reedy wrote: > I believe once every 24 hours. The current page is dated Oct 17 (at bottom). > It is now Oct 18 most everywhere. Thanks, I didn't know there was a generated date at the bottom. Useful to know for the future! I'll wait for the update, cheers. Paul From ncoghlan at gmail.com Fri Oct 18 12:16:03 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 18 Oct 2013 20:16:03 +1000 Subject: [Python-Dev] cpython: Issue #16129: Add `Py_SetStandardStreamEncoding` In-Reply-To: References: <3d0rd742bTz7LjR@mail.python.org> <20131017173640.53016af5@fsol> Message-ID: On 18 Oct 2013 15:20, "Georg Brandl" wrote: > > Am 17.10.2013 17:36, schrieb Antoine Pitrou: > > On Thu, 17 Oct 2013 15:22:03 +0200 (CEST) > > nick.coghlan wrote: > >> > >> +.. c:function:: int Py_SetStandardStreamEncoding(char *encoding, char *errors) > >> + > >> + .. index:: > >> + single: Py_Initialize() > >> + single: main() > >> + triple: stdin; stdout; sdterr > >> + > >> + This function should be called before :c:func:`Py_Initialize`. It > >> + specifies which encoding and error handling to use with standard io, > >> + with the same meanings as in :func:`str.encode`. > >> + > >> + It overrides :envvar:`PYTHONIOENCODING` values, and allows embedding code > >> + to control io encoding when the environment variable does not work. > >> + > >> + ``encoding`` and/or ``errors`` may be NULL to use > >> + :envvar:`PYTHONIOENCODING` and/or default values (depending on other > >> + settings). > >> + > >> + Note that :data:`sys.stderr` always uses the "backslashreplace" error > >> + handler, regardless of this (or any other) setting. > >> + > >> + If :c:func:`Py_Finalize` is called, this function will need to be called > >> + again in order to affect subsequent calls to :c:func:`Py_Initialize`. > >> + > >> + Returns 0 if successful. > >> + > >> + > > > > Needs a versionadded tag. > > Also, shouldn't it be excluded from the stable/limited API? Since it didn't expose any struct layouts, I just followed the precedent set by the other PySet_* APIs on that front. Moving it would be fine, too, though, since "embedding the interpreter" and "using the limited API" don't exactly mix :) Cheers, Nick. > > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Oct 18 12:28:53 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 18 Oct 2013 12:28:53 +0200 Subject: [Python-Dev] cpython: Issue #16129: Add `Py_SetStandardStreamEncoding` References: <3d0rd742bTz7LjR@mail.python.org> <20131017173640.53016af5@fsol> Message-ID: <20131018122853.6843708a@pitrou.net> Le Fri, 18 Oct 2013 20:16:03 +1000, Nick Coghlan a ?crit : > > > > Also, shouldn't it be excluded from the stable/limited API? > > Since it didn't expose any struct layouts, I just followed the > precedent set by the other PySet_* APIs on that front. Moving it > would be fine, too, though, since "embedding the interpreter" and > "using the limited API" don't exactly mix :) As far as I understand, when we *add* something to the stable ABI, it should ideally be versioned. There are a couple examples of this in the include files, e.g.: #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03030000 /* New in 3.3 */ PyAPI_FUNC(int) PyState_AddModule(PyObject*, struct PyModuleDef*); PyAPI_FUNC(int) PyState_RemoveModule(struct PyModuleDef*); #endif (this is so people wanting to conform to e.g. a 3.2-compatible ABI don't start depending on 3.3-and-upwards stuff by mistake) However, it seems we haven't been following that rule, for the most part. Regards Antoine. From victor.stinner at gmail.com Fri Oct 18 14:20:03 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 18 Oct 2013 14:20:03 +0200 Subject: [Python-Dev] PEP 454 (tracemalloc): new minimalist version Message-ID: Hi, I plan to push the new tracemalloc module this week-end, before the next (and last) alpha version, except if someone complains :-) I prefer to have one month before the first beta to have more time to test the module. So if a major issue is raised, we may remove the tracemalloc module before the final version. Charles-Fran?ois Natali reviewed my PEP 454 (tracemalloc). He suggested me to split the module into two parts: keep only essential parts in the tracemalloc Python (added to Python stdlib), and move other tools to an external tool (hosted on PyPI). We may integrate more functions (especially metrics) later (in Python 3.5), after they are well tested externally. I updated the PEP and the implementation. I will update the patch on the bug tracker later. HTML version of the PEP: http://www.python.org/dev/peps/pep-0454/ Since the previous version of the PEP, almost all changes are removed parts: * Remove tasks: the tool now uses classic threads * Remove functions to get metrics: the tool is now responsible to add them to a snapshot * Remove DisplayTop and DisplayTopTask class * Remove TakeSnaphot class PEP: 454 Title: Add a new tracemalloc module to trace Python memory allocations Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3-September-2013 Python-Version: 3.4 Abstract ======== This PEP proposes to add a new ``tracemalloc`` module to trace memory blocks allocated by Python. Rationale ========= Common debug tools tracing memory allocations record the C filename and line number where the allocation occurs. Using such tools to analyze Python memory allocations does not help because most memory blocks are allocated in the same C function, in ``PyMem_Malloc()`` for example. There are debug tools dedicated to the Python language like ``Heapy`` and ``PySizer``. These tools analyze objects type and/or content. They are useful when most memory leaks are instances of the same type and this type is only instantiated in a few functions. Problems arise when the object type is very common like ``str`` or ``tuple``, and it is hard to identify where these objects are instantiated. Finding reference cycles is also a difficult problem. There are different tools to draw a diagram of all references. These tools cannot be used on large applications with thousands of objects because the diagram is too huge to be analyzed manually. Proposal ======== Using the customized allocation API from PEP 445, it becomes easy to set up a hook on Python memory allocators. A hook can inspect Python internals to retrieve Python tracebacks. This PEP proposes to add a new ``tracemalloc`` module, as a debug tool to trace memory blocks allocated by Python. The module provides the following information: * Computed differences between two snapshots to detect memory leaks * Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks * Traceback where a memory block was allocated The API of the tracemalloc module is similar to the API of the faulthandler module: ``enable()``, ``disable()`` and ``is_enabled()`` functions, an environment variable (``PYTHONFAULTHANDLER`` and ``PYTHONTRACEMALLOC``), and a ``-X`` command line option (``-X faulthandler`` and ``-X tracemalloc``). See the `documentation of the faulthandler module `_. The tracemalloc module has been written for CPython. Other implementations of Python may not be able to provide it. API === Main Functions -------------- ``clear_traces()`` function: Clear traces and statistics on Python memory allocations, and reset the ``get_traced_memory()`` counter. ``disable()`` function: Stop tracing Python memory allocations. See also ``enable()`` and ``is_enabled()`` functions. ``enable()`` function: Start tracing Python memory allocations. At fork, the module is automatically disabled in the child process. See also ``disable()`` and ``is_enabled()`` functions. ``get_stats()`` function: Get statistics on traced Python memory blocks as a dictionary ``{filename (str): {line_number (int): stats}}`` where *stats* in a ``(size: int, count: int)`` tuple, *filename* and *line_number* can be ``None``. Return an empty dictionary if the ``tracemalloc`` module is disabled. See also the ``get_traces()`` function. ``get_traced_memory()`` function: Get the current size and maximum size of memory blocks traced by the ``tracemalloc`` module as a tuple: ``(size: int, max_size: int)``. ``get_tracemalloc_memory()`` function: Get the memory usage in bytes of the ``tracemalloc`` module as a tuple: ``(size: int, free: int)``. * *size*: total size of bytes allocated by the module, including *free* bytes * *free*: number of free bytes available to store data ``is_enabled()`` function: ``True`` if the ``tracemalloc`` module is tracing Python memory allocations, ``False`` otherwise. See also ``enable()`` and ``disable()`` functions. Trace Functions --------------- ``get_traceback_limit()`` function: Get the maximum number of frames stored in the traceback of a trace of a memory block. Use the ``set_traceback_limit()`` function to change the limit. ``get_object_address(obj)`` function: Get the address of the main memory block of the specified Python object. A Python object can be composed by multiple memory blocks, the function only returns the address of the main memory block. See also ``get_object_trace()`` and ``gc.get_referrers()`` functions. ``get_object_trace(obj)`` function: Get the trace of a Python object *obj* as a ``(size: int, traceback)`` tuple where *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. The function only returns the trace of the main memory block of the object. The *size* of the trace is smaller than the total size of the object if the object is composed by more than one memory block. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the object. See also ``get_object_address()``, ``get_trace()``, ``get_traces()``, ``gc.get_referrers()`` and ``sys.getsizeof()`` functions. ``get_trace(address)`` function: Get the trace of a memory block as a ``(size: int, traceback)`` tuple where *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the memory block. See also ``get_object_trace()``, ``get_stats()`` and ``get_traces()`` functions. ``get_traces()`` function: Get traces of Python memory allocations as a dictionary ``{address (int): trace}`` where *trace* is a ``(size: int, traceback)`` and *traceback* is a list of ``(filename: str, lineno: int)``. *traceback* can be empty, *filename* and *lineno* can be None. Return an empty dictionary if the ``tracemalloc`` module is disabled. See also ``get_object_trace()``, ``get_stats()`` and ``get_trace()`` functions. ``set_traceback_limit(nframe: int)`` function: Set the maximum number of frames stored in the traceback of a trace of a memory block. Storing the traceback of each memory allocation has an important overhead on the memory usage. Use the ``get_tracemalloc_memory()`` function to measure the overhead and the ``add_filter()`` function to select which memory allocations are traced. Use the ``get_traceback_limit()`` function to get the current limit. Filter Functions ---------------- ``add_filter(filter)`` function: Add a new filter on Python memory allocations, *filter* is a ``Filter`` instance. All inclusive filters are applied at once, a memory allocation is only ignored if no inclusive filters match its trace. A memory allocation is ignored if at least one exclusive filter matchs its trace. The new filter is not applied on already collected traces. Use the ``clear_traces()`` function to ensure that all traces match the new filter. ``add_include_filter(filename: str, lineno: int=None, traceback: bool=False)`` function: Add an inclusive filter: helper for the ``add_filter()`` method creating a ``Filter`` instance with the ``Filter.include`` attribute set to ``True``. Example: ``tracemalloc.add_include_filter(tracemalloc.__file__)`` only includes memory blocks allocated by the ``tracemalloc`` module. ``add_exclude_filter(filename: str, lineno: int=None, traceback: bool=False)`` function: Add an exclusive filter: helper for the ``add_filter()`` method creating a ``Filter`` instance with the ``Filter.include`` attribute set to ``False``. Example: ``tracemalloc.add_exclude_filter(tracemalloc.__file__)`` ignores memory blocks allocated by the ``tracemalloc`` module. ``clear_filters()`` function: Reset the filter list. See also the ``get_filters()`` function. ``get_filters()`` function: Get the filters on Python memory allocations as list of ``Filter`` instances. See also the ``clear_filters()`` function. Filter ------ ``Filter(include: bool, pattern: str, lineno: int=None, traceback: bool=False)`` class: Filter to select which memory allocations are traced. Filters can be used to reduce the memory usage of the ``tracemalloc`` module, which can be read using the ``get_tracemalloc_memory()`` function. ``match(filename: str, lineno: int)`` method: Return ``True`` if the filter matchs the filename and line number, ``False`` otherwise. ``match_filename(filename: str)`` method: Return ``True`` if the filter matchs the filename, ``False`` otherwise. ``match_lineno(lineno: int)`` method: Return ``True`` if the filter matchs the line number, ``False`` otherwise. ``match_traceback(traceback)`` method: Return ``True`` if the filter matchs the *traceback*, ``False`` otherwise. *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples. ``include`` attribute: If *include* is ``True``, only trace memory blocks allocated in a file with a name matching filename ``pattern`` at line number ``lineno``. If *include* is ``False``, ignore memory blocks allocated in a file with a name matching filename ``pattern`` at line number ``lineno``. ``lineno`` attribute: Line number (``int``). If is is ``None`` or less than ``1``, it matches any line number. ``pattern`` attribute: The filename *pattern* can contain one or many ``*`` joker characters which match any substring, including an empty string. The ``.pyc`` and ``.pyo`` file extensions are replaced with ``.py``. On Windows, the comparison is case insensitive and the alternative separator ``/`` is replaced with the standard separator ``\``. ``traceback`` attribute: If *traceback* is ``True``, all frames of the traceback are checked. If *traceback* is ``False``, only the most recent frame is checked. This attribute is ignored if the traceback limit is less than ``2``. See the ``get_traceback_limit()`` function. GroupedStats ------------ ``GroupedStats(timestamp: datetime.datetime, stats: dict, group_by: str, cumulative=False, metrics: dict=None)`` class: Top of allocated memory blocks grouped by *group_by* as a dictionary. The ``Snapshot.top_by()`` method creates a ``GroupedStats`` instance. ``compare_to(old_stats: GroupedStats=None)`` method: Compare to an older ``GroupedStats`` instance. Return a ``StatsDiff`` instance. The ``StatsDiff.differences`` list is not sorted: call the ``StatsDiff.sort()`` method to sort the list. ``None`` values are replaced with an empty string for filenames or zero for line numbers, because ``str`` and ``int`` cannot be compared to ``None``. ``cumulative`` attribute: If ``True``, cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. ``metrics`` attribute: Dictionary storing metrics read when the snapshot was created: ``{name (str): metric}`` where *metric* type is ``Metric``. ``group_by`` attribute: Determine how memory allocations were grouped: see ``Snapshot.top_by()`` for the available values. ``stats`` attribute: Dictionary ``{key: stats}`` where the *key* type depends on the ``group_by`` attribute and *stats* is a ``(size: int, count: int)`` tuple. See the ``Snapshot.top_by()`` method. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. Metric ------ ``Metric(name: str, value: int, format: str)`` class: Value of a metric when a snapshot is created. ``name`` attribute: Name of the metric. ``value`` attribute: Value of the metric. ``format`` attribute: Format of the metric (``str``). Snapshot -------- ``Snapshot(timestamp: datetime.datetime, traces: dict=None, stats: dict=None)`` class: Snapshot of traces and statistics on memory blocks allocated by Python. ``add_metric(name: str, value: int, format: str)`` method: Helper to add a ``Metric`` instance to ``Snapshot.metrics``. Return the newly created ``Metric`` instance. Raise an exception if the name is already present in ``Snapshot.metrics``. ``apply_filters(filters)`` method: Apply filters on the ``traces`` and ``stats`` dictionaries, *filters* is a list of ``Filter`` instances. ``create(traces=False)`` classmethod: Take a snapshot of traces and/or statistics of allocated memory blocks. If *traces* is ``True``, ``get_traces()`` is called and its result is stored in the ``Snapshot.traces`` attribute. This attribute contains more information than ``Snapshot.stats`` and uses more memory and more disk space. If *traces* is ``False``, ``Snapshot.traces`` is set to ``None``. Tracebacks of traces are limited to ``traceback_limit`` frames. Call ``set_traceback_limit()`` before calling ``Snapshot.create()`` to store more frames. The ``tracemalloc`` module must be enabled to take a snapshot. See the the ``enable()`` function. ``get_metric(name, default=None)`` method: Get the value of the metric called *name*. Return *default* if the metric does not exist. ``load(filename, traces=True)`` classmethod: Load a snapshot from a file. If *traces* is ``False``, don't load traces. ``top_by(group_by: str, cumulative: bool=False)`` method: Compute top statistics grouped by *group_by* as a ``GroupedStats`` instance: ===================== ======================== ================================ group_by description key type ===================== ======================== ================================ ``'filename'`` filename ``str`` ``'line'`` filename and line number ``(filename: str, lineno: int)`` ``'address'`` memory block address ``int`` ``'traceback'`` traceback ``(address: int, traceback)`` ===================== ======================== ================================ The ``traceback`` type is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. If *cumulative* is ``True``, cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. The *cumulative* parameter is ignored if *group_by* is ``'address'`` or if the traceback limit is less than ``2``. ``write(filename)`` method: Write the snapshot into a file. ``metrics`` attribute: Dictionary storing metrics read when the snapshot was created: ``{name (str): metric}`` where *metric* type is ``Metric``. ``stats`` attribute: Statistics on traced Python memory, result of the ``get_stats()`` function. ``traceback_limit`` attribute: Maximum number of frames stored in a trace of a memory block allocated by Python. ``traces`` attribute: Traces of Python memory allocations, result of the ``get_traces()`` function, can be ``None``. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. StatsDiff --------- ``StatsDiff(differences, old_stats, new_stats)`` class: Differences between two ``GroupedStats`` instances. The ``GroupedStats.compare_to()`` method creates a ``StatsDiff`` instance. ``sort()`` method: Sort the ``differences`` list from the biggest difference to the smallest difference. Sort by ``abs(size_diff)``, *size*, ``abs(count_diff)``, *count* and then by *key*. ``differences`` attribute: Differences between ``old_stats`` and ``new_stats`` as a list of ``(size_diff, size, count_diff, count, key)`` tuples. *size_diff*, *size*, *count_diff* and *count* are ``int``. The key type depends on the ``GroupedStats.group_by`` attribute of ``new_stats``: see the ``Snapshot.top_by()`` method. ``old_stats`` attribute: Old ``GroupedStats`` instance, can be ``None``. ``new_stats`` attribute: New ``GroupedStats`` instance. Links ===== tracemalloc: * `#18874: Add a new tracemalloc module to trace Python memory allocations `_ * `pytracemalloc on PyPI `_ Similar projects: * `Meliae: Python Memory Usage Analyzer `_ * `Guppy-PE: umbrella package combining Heapy and GSL `_ * `PySizer `_: developed for Python 2.4 * `memory_profiler `_ * `pympler `_ * `Dozer `_: WSGI Middleware version of the CherryPy memory leak debugger * `objgraph `_ * `caulk `_ Copyright ========= This document has been placed in the public domain. Victor From ncoghlan at gmail.com Fri Oct 18 15:05:18 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 18 Oct 2013 23:05:18 +1000 Subject: [Python-Dev] PEP 454 (tracemalloc): new minimalist version In-Reply-To: References: Message-ID: This looks really nice to me, and splitting it so the core functionality is in the standard library and there's a separate higher level tool on PyPI (allowing faster iteration on the analysis features) is a fine idea. Cheers, Nick. From brett at python.org Fri Oct 18 15:33:40 2013 From: brett at python.org (Brett Cannon) Date: Fri, 18 Oct 2013 09:33:40 -0400 Subject: [Python-Dev] The Tulip Has Landed In-Reply-To: References: <52606763.70006@hastings.org> Message-ID: On Fri, Oct 18, 2013 at 4:10 AM, Paul Moore wrote: > On 17 October 2013 23:40, Larry Hastings wrote: > > For those interested parties: Guido just checked "asyncio", aka "Tulip", > aka > > "PEP 3156", in to trunk. I expect it to be part of Python 3.4.0a4, > > hopefully to be released this weekend. > > Cool! How often do the online docs get built? There's nothing showing > in there yet (http://docs.python.org/3.4/library/index.html)... > It doesn't matter as Guido has not checked in any docs yet. -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Oct 18 18:07:38 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 18 Oct 2013 18:07:38 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20131018160738.8E1F156A3A@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-10-11 - 2013-10-18) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4184 (-56) closed 26872 (+115) total 31056 (+59) Open issues with patches: 1948 Issues opened (43) ================== #16203: Proposal: add re.fullmatch() method http://bugs.python.org/issue16203 reopened by georg.brandl #17294: compile-flag for single-execution to return value instead of p http://bugs.python.org/issue17294 reopened by Albert.Zeyer #19205: Don't import re and sysconfig in site.py http://bugs.python.org/issue19205 reopened by pitrou #19227: test_multiprocessing_xxx hangs under Gentoo buildbots http://bugs.python.org/issue19227 opened by pitrou #19229: operator.py: move the Python implementation in the else block http://bugs.python.org/issue19229 opened by haypo #19230: Reimplement the keyword module in C http://bugs.python.org/issue19230 opened by haypo #19232: Speed up _decimal import http://bugs.python.org/issue19232 opened by skrah #19233: test_io.test_interrupted_write_retry_text() hangs on Solaris 1 http://bugs.python.org/issue19233 opened by haypo #19234: socket.fileno() documentation http://bugs.python.org/issue19234 opened by georg.brandl #19235: Add a dedicated subclass for recursion errors http://bugs.python.org/issue19235 opened by elazar #19236: Add Tornado HTTP benchmark http://bugs.python.org/issue19236 opened by pitrou #19237: Proposal : LCM function to complement GCD http://bugs.python.org/issue19237 opened by CliffM #19238: Misleading explanation of fill and align in format_spec http://bugs.python.org/issue19238 opened by davidchambers #19239: add inspect functions to retrieve attributes from both old dir http://bugs.python.org/issue19239 opened by ethan.furman #19240: iglob should try to use `readdir` http://bugs.python.org/issue19240 opened by tebeka #19241: MINGW: install import library http://bugs.python.org/issue19241 opened by rpetrov #19242: MINGW: generalization of posix installation in distutils http://bugs.python.org/issue19242 opened by rpetrov #19243: MINGW: support stdcall without underscore http://bugs.python.org/issue19243 opened by rpetrov #19244: MINGW: use replace instead rename to avoid failure on windows http://bugs.python.org/issue19244 opened by rpetrov #19245: mingw-meta: installation http://bugs.python.org/issue19245 opened by rpetrov #19247: Describe surrogateescape algorithm in the Library Reference http://bugs.python.org/issue19247 opened by ncoghlan #19249: Enumeration.__eq__ http://bugs.python.org/issue19249 opened by CliffM #19251: bitwise ops for bytes of equal length http://bugs.python.org/issue19251 opened by christian.heimes #19253: PyArg_ParseTuple: wrong use of seterror() clips helpful type e http://bugs.python.org/issue19253 opened by georg.brandl #19254: Python implementation of PBKDF2_HMAC http://bugs.python.org/issue19254 opened by christian.heimes #19255: Don't "wipe" builtins at shutdown http://bugs.python.org/issue19255 opened by serhiy.storchaka #19256: Optimize marshal format and add version token. http://bugs.python.org/issue19256 opened by kristjan.jonsson #19257: Sub-optimal error message when importing a non-package http://bugs.python.org/issue19257 opened by alex #19259: Provide Python implementation of operator.compare_digest() http://bugs.python.org/issue19259 opened by christian.heimes #19261: Add support for 24-bit in the sunau module http://bugs.python.org/issue19261 opened by serhiy.storchaka #19262: Add asyncio (tulip, PEP 3156) to stdlib http://bugs.python.org/issue19262 opened by gvanrossum #19263: enum.py : Enum.__new__(). __objclass__ http://bugs.python.org/issue19263 opened by CliffM #19264: subprocess.Popen doesn't support unicode on Windows http://bugs.python.org/issue19264 opened by peter0 #19265: Increased test coverage for datetime pickling http://bugs.python.org/issue19265 opened by Colin.Williams #19270: sched.cancel() breaks events order http://bugs.python.org/issue19270 opened by serhiy.storchaka #19271: Update OrderedDict "see also" link http://bugs.python.org/issue19271 opened by BreamoreBoy #19273: Update PCbuild/readme.txt http://bugs.python.org/issue19273 opened by zach.ware #19274: make zipfile.PyZipFile more usable http://bugs.python.org/issue19274 opened by Christian.Tismer #19279: UTF-7 to UTF-8 decoding crash http://bugs.python.org/issue19279 opened by glebourgeois #19280: Add a datatype to represent mime types to the email module http://bugs.python.org/issue19280 opened by r.david.murray #19281: add __objclass__ to the docs http://bugs.python.org/issue19281 opened by ethan.furman #19282: dbm is not a context manager http://bugs.python.org/issue19282 opened by kousu #19283: Need support to avoid Windows CRT compatibility issue. http://bugs.python.org/issue19283 opened by debarshig Most recent 15 issues with no replies (15) ========================================== #19281: add __objclass__ to the docs http://bugs.python.org/issue19281 #19271: Update OrderedDict "see also" link http://bugs.python.org/issue19271 #19265: Increased test coverage for datetime pickling http://bugs.python.org/issue19265 #19264: subprocess.Popen doesn't support unicode on Windows http://bugs.python.org/issue19264 #19261: Add support for 24-bit in the sunau module http://bugs.python.org/issue19261 #19257: Sub-optimal error message when importing a non-package http://bugs.python.org/issue19257 #19247: Describe surrogateescape algorithm in the Library Reference http://bugs.python.org/issue19247 #19245: mingw-meta: installation http://bugs.python.org/issue19245 #19244: MINGW: use replace instead rename to avoid failure on windows http://bugs.python.org/issue19244 #19243: MINGW: support stdcall without underscore http://bugs.python.org/issue19243 #19242: MINGW: generalization of posix installation in distutils http://bugs.python.org/issue19242 #19241: MINGW: install import library http://bugs.python.org/issue19241 #19238: Misleading explanation of fill and align in format_spec http://bugs.python.org/issue19238 #19222: Add 'x' mode to gzip.open() http://bugs.python.org/issue19222 #19214: shutil.make_archive should recognize extensions in filenames http://bugs.python.org/issue19214 Most recent 15 issues waiting for review (15) ============================================= #19282: dbm is not a context manager http://bugs.python.org/issue19282 #19279: UTF-7 to UTF-8 decoding crash http://bugs.python.org/issue19279 #19274: make zipfile.PyZipFile more usable http://bugs.python.org/issue19274 #19273: Update PCbuild/readme.txt http://bugs.python.org/issue19273 #19270: sched.cancel() breaks events order http://bugs.python.org/issue19270 #19265: Increased test coverage for datetime pickling http://bugs.python.org/issue19265 #19263: enum.py : Enum.__new__(). __objclass__ http://bugs.python.org/issue19263 #19262: Add asyncio (tulip, PEP 3156) to stdlib http://bugs.python.org/issue19262 #19261: Add support for 24-bit in the sunau module http://bugs.python.org/issue19261 #19256: Optimize marshal format and add version token. http://bugs.python.org/issue19256 #19254: Python implementation of PBKDF2_HMAC http://bugs.python.org/issue19254 #19253: PyArg_ParseTuple: wrong use of seterror() clips helpful type e http://bugs.python.org/issue19253 #19244: MINGW: use replace instead rename to avoid failure on windows http://bugs.python.org/issue19244 #19243: MINGW: support stdcall without underscore http://bugs.python.org/issue19243 #19242: MINGW: generalization of posix installation in distutils http://bugs.python.org/issue19242 Top 10 most discussed issues (10) ================================= #19262: Add asyncio (tulip, PEP 3156) to stdlib http://bugs.python.org/issue19262 33 msgs #16129: No good way to set 'PYTHONIOENCODING' when embedding python. http://bugs.python.org/issue16129 22 msgs #19256: Optimize marshal format and add version token. http://bugs.python.org/issue19256 17 msgs #4965: Can doc index of html version be separately scrollable? http://bugs.python.org/issue4965 14 msgs #19227: test_multiprocessing_xxx hangs under Gentoo buildbots http://bugs.python.org/issue19227 14 msgs #19232: Speed up _decimal import http://bugs.python.org/issue19232 14 msgs #19205: Don't import re and sysconfig in site.py http://bugs.python.org/issue19205 10 msgs #19230: Reimplement the keyword module in C http://bugs.python.org/issue19230 9 msgs #19279: UTF-7 to UTF-8 decoding crash http://bugs.python.org/issue19279 9 msgs #13451: sched.py: speedup cancel() method http://bugs.python.org/issue13451 8 msgs Issues closed (85) ================== #2919: Merge profile/cProfile in 3.n+1 http://bugs.python.org/issue2919 closed by alexandre.vassalotti #4555: Smelly exports (global symbols in python not prefixed with Py http://bugs.python.org/issue4555 closed by pitrou #8090: PEP 4 should say something about the standard library http://bugs.python.org/issue8090 closed by python-dev #8507: abc.abstractproperty does not copy docstring http://bugs.python.org/issue8507 closed by georg.brandl #9170: zipfile cannot read AES encrypted files http://bugs.python.org/issue9170 closed by georg.brandl #9548: locale can be imported at startup but relies on too many libra http://bugs.python.org/issue9548 closed by pitrou #11120: threading.Thread.daemon Documentation Incomplete http://bugs.python.org/issue11120 closed by georg.brandl #12348: case sensitivity in packaging.pypi.simple.Crawler http://bugs.python.org/issue12348 closed by georg.brandl #13026: Dis module - documentation of MAKE_FUNCTION http://bugs.python.org/issue13026 closed by python-dev #13203: Doc: say id() is only useful for existing objects http://bugs.python.org/issue13203 closed by python-dev #13740: winsound.SND_NOWAIT ignored on modern Windows platforms http://bugs.python.org/issue13740 closed by python-dev #13744: raw byte strings are described in a confusing way http://bugs.python.org/issue13744 closed by georg.brandl #13833: No documentation for PyStructSequence http://bugs.python.org/issue13833 closed by python-dev #13905: Built-in Types Comparisons should mention rich comparison meth http://bugs.python.org/issue13905 closed by python-dev #13915: Update tutorial/modules for PEP 3147 http://bugs.python.org/issue13915 closed by python-dev #14407: concurrent.futures tests don't adher to test_cases protocol http://bugs.python.org/issue14407 closed by pitrou #14486: Add some versionchanged notes in threading docs http://bugs.python.org/issue14486 closed by python-dev #14999: ctypes ArgumentError lists arguments from 1, not 0 http://bugs.python.org/issue14999 closed by georg.brandl #15648: stderr "refs" output does not respect PYTHONIOENCODING http://bugs.python.org/issue15648 closed by georg.brandl #15829: Threading Lock - Wrong Exception Name http://bugs.python.org/issue15829 closed by python-dev #16253: Docs for PyArg_ParseTupleAndKeywords say it doesn't accept nes http://bugs.python.org/issue16253 closed by georg.brandl #16657: traceback.format_tb incorrect docsting http://bugs.python.org/issue16657 closed by python-dev #16732: setup.py support for xxmodule without tkinker http://bugs.python.org/issue16732 closed by python-dev #16938: pydoc confused by __dir__ http://bugs.python.org/issue16938 closed by ethan.furman #17029: h2py.py: search the multiarch include dir if it does exist http://bugs.python.org/issue17029 closed by ezio.melotti #17084: nntplib.NNTP.xover does not always return results as documente http://bugs.python.org/issue17084 closed by georg.brandl #17154: the 'ignore' pdb command raises IndexError http://bugs.python.org/issue17154 closed by python-dev #17335: FieldStorageClass is messed up http://bugs.python.org/issue17335 closed by python-dev #17375: Add docstrings to methods in the threading module http://bugs.python.org/issue17375 closed by python-dev #17631: inspect getsource does not display full text of lambda http://bugs.python.org/issue17631 closed by georg.brandl #17730: code.interact() doesn't support no banner http://bugs.python.org/issue17730 closed by python-dev #18102: except-clause with own exception class inside generator can le http://bugs.python.org/issue18102 closed by pitrou #18209: Bytearray type not supported as a mutable object in the fcntl. http://bugs.python.org/issue18209 closed by georg.brandl #18254: Accessing attr dict at definition time of a class from a metac http://bugs.python.org/issue18254 closed by georg.brandl #18281: tarfile defines stat constants http://bugs.python.org/issue18281 closed by python-dev #18416: Move to absolute file paths for module.__file__ http://bugs.python.org/issue18416 closed by brett.cannon #18458: interactive interpreter crashes and test_readline fails with n http://bugs.python.org/issue18458 closed by ned.deily #18468: re.group() should never return a bytearray http://bugs.python.org/issue18468 closed by serhiy.storchaka #18521: [cppcheck] Full report http://bugs.python.org/issue18521 closed by georg.brandl #18582: PBKDF2 support http://bugs.python.org/issue18582 closed by christian.heimes #18714: Add tests for pdb.find_function http://bugs.python.org/issue18714 closed by python-dev #18723: shorten function of textwrap module is susceptible to non-norm http://bugs.python.org/issue18723 closed by vajrasky #18725: Multiline shortening http://bugs.python.org/issue18725 closed by serhiy.storchaka #18739: math.log of a long returns a different value of math.log of an http://bugs.python.org/issue18739 closed by mark.dickinson #18754: Run Python child processes in isolated mode in the test suite? http://bugs.python.org/issue18754 closed by python-dev #18758: Fix internal references in the documentation http://bugs.python.org/issue18758 closed by serhiy.storchaka #18776: atexit error display behavior changed in python 3 http://bugs.python.org/issue18776 closed by pitrou #18804: pythorun.c: is_valid_fd() should not duplicate the file descri http://bugs.python.org/issue18804 closed by haypo #18919: Unify audio modules tests http://bugs.python.org/issue18919 closed by serhiy.storchaka #18999: Support different contexts in multiprocessing http://bugs.python.org/issue18999 closed by sbt #19079: chameleon benchmark fails on 3.4 http://bugs.python.org/issue19079 closed by python-dev #19131: Broken support of compressed AIFC files http://bugs.python.org/issue19131 closed by serhiy.storchaka #19179: doc bug: confusing table of values http://bugs.python.org/issue19179 closed by python-dev #19189: Improve cross-references in pickle documentation. http://bugs.python.org/issue19189 closed by serhiy.storchaka #19191: os.path.splitext in windows , a little question http://bugs.python.org/issue19191 closed by terry.reedy #19192: Move test_current_time from test_xmlrpc_net to test_xmlrpc http://bugs.python.org/issue19192 closed by r.david.murray #19197: Improve cross-references in shlex documentation http://bugs.python.org/issue19197 closed by serhiy.storchaka #19198: Improve cross-references in cgi documentation http://bugs.python.org/issue19198 closed by serhiy.storchaka #19202: Additions to function docs: reduce and itertools. http://bugs.python.org/issue19202 closed by rhettinger #19203: Improve cross-references in curses howto http://bugs.python.org/issue19203 closed by serhiy.storchaka #19204: Improve cross-references in urllib documentation http://bugs.python.org/issue19204 closed by serhiy.storchaka #19207: Improve cross-references in os documentation http://bugs.python.org/issue19207 closed by serhiy.storchaka #19209: Remove import copyreg from os module http://bugs.python.org/issue19209 closed by christian.heimes #19218: Use of MutableMapping in os module slows down interpreter star http://bugs.python.org/issue19218 closed by christian.heimes #19219: speed up marshal.loads() http://bugs.python.org/issue19219 closed by pitrou #19224: Make hash(None) consistent among processes http://bugs.python.org/issue19224 closed by rhettinger #19226: distutils/command/upload.py show response gives: TypeError: se http://bugs.python.org/issue19226 closed by ned.deily #19228: type.__qualname__ should not strip the module name http://bugs.python.org/issue19228 closed by pitrou #19231: ctype cant's use printf http://bugs.python.org/issue19231 closed by georg.brandl #19246: high fragmentation of the memory heap on Windows http://bugs.python.org/issue19246 closed by haypo #19248: sphinx is not py3k compatible http://bugs.python.org/issue19248 closed by ezio.melotti #19250: Duplicate import documentation in py3k http://bugs.python.org/issue19250 closed by georg.brandl #19252: Enum.py : Enum.__new__() : Test Coverage http://bugs.python.org/issue19252 closed by python-dev #19258: Got resource warning when running wsgiref demo app http://bugs.python.org/issue19258 closed by python-dev #19260: "int" comment in marshal.c is outdated http://bugs.python.org/issue19260 closed by python-dev #19266: Rename contextlib.ignore to contextlib.suppress http://bugs.python.org/issue19266 closed by python-dev #19267: Logging to file does not accept UTF16 http://bugs.python.org/issue19267 closed by python-dev #19268: Local variable created with reflection cannot be referenced wi http://bugs.python.org/issue19268 closed by r.david.murray #19269: subtraction of pennies incorrect rounding or truncation proble http://bugs.python.org/issue19269 closed by tim.peters #19272: Can't pickle lambda (while named functions are ok) http://bugs.python.org/issue19272 closed by ethan.furman #19275: test_site is failing on AMD64 Snow Leopard http://bugs.python.org/issue19275 closed by ned.deily #19276: test_wave failing on PPC64 Linux http://bugs.python.org/issue19276 closed by serhiy.storchaka #19277: zlib compressobj: missing parameter doc strings http://bugs.python.org/issue19277 closed by python-dev #19278: zlib compressobj: expose missing knobs http://bugs.python.org/issue19278 closed by georg.brandl #19284: subprocess._args_from_interpreter_flags() mishandles -R http://bugs.python.org/issue19284 closed by python-dev From guido at python.org Fri Oct 18 18:31:52 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 18 Oct 2013 09:31:52 -0700 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows Message-ID: I'm working from home today and my Windows laptop is in the office, so I won't be able to test my latest Tulip changes on Windows (I just renamed pause to pause_reading, and hope to commit pause_reading later today). Is anyone luckier? Also, reading through the Windows OpenSSL setup in PCbuild/readme.txt I lost courage, so I've not tested the SSL behavior on Windows at all. Is anyone lucky enough to have a Python 3.4 alpha for Windows setup with OpenSSL already configured? If so, could you run test_asyncio for me and let me know if there are any issues? (Actually, even a working Python 3.3 setup would be useful, although you'd have to check out the Tulip repo and run the tests from there.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Oct 18 18:53:55 2013 From: brett at python.org (Brett Cannon) Date: Fri, 18 Oct 2013 12:53:55 -0400 Subject: [Python-Dev] possibility of shaving a stat call from imports Message-ID: importlib.machinery.FileFinder does a stat call to check if a path is a file if the package check failed. Now I'm willing to bet that the check is rather redundant as the file extension should be a dead give-away that something in a directory is a file and not some non-file type. The import would still fail even if this is the case in the loader when attempting to read from the file, but it happens a little later and it means finders would be more permissive in claiming they found a loader. Does anyone see a good reason not to take the more optimistic route in the finder? As I said, the only thing I see breaking user code is if they have a directory or something named spam.py and so the finder claims it found a module when in fact it didn't and thus stopping the search for the module from continuing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From shibturn at gmail.com Fri Oct 18 19:04:48 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Fri, 18 Oct 2013 18:04:48 +0100 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: References: Message-ID: <52616A30.6000803@gmail.com> On 18/10/2013 5:31pm, Guido van Rossum wrote: > I'm working from home today and my Windows laptop is in the office, so I > won't be able to test my latest Tulip changes on Windows (I just renamed > pause to pause_reading, and hope to commit pause_reading later today). > Is anyone luckier? $ hg id 97f6f35b02e5 (asyncio) tip $ python-release runtests.py Skipping 'test_unix_events': UNIX only ................................................s.................s......ss...ssssssssssss.....s........s....ss..sss.....ss...ssssssssssssss.................................................................................................................................................................................................................................................................................................................................................................................................................... ---------------------------------------------------------------------- Ran 544 tests in 12.790s OK (skipped=39) > Also, reading through the Windows OpenSSL setup in PCbuild/readme.txt I > lost courage, so I've not tested the SSL behavior on Windows at all. Is > anyone lucky enough to have a Python 3.4 alpha for Windows setup with > OpenSSL already configured? If so, could you run test_asyncio for me and > let me know if there are any issues? (Actually, even a working Python > 3.3 setup would be useful, although you'd have to check out the Tulip > repo and run the tests from there.) $ python-release -c 'import sys; print(sys.version)' 3.4.0a3+ (default:7172135d60f6, Oct 18 2013, 17:54:18) [MSC v.1600 32 bit (Intel)] $ python-release -m test test_asyncio [1/1] test_asyncio 1 test OK. BTW, pcbuild.sln was not building _overlapped automatically -- I have pushed a fix. -- Richard From shibturn at gmail.com Fri Oct 18 19:04:48 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Fri, 18 Oct 2013 18:04:48 +0100 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: References: Message-ID: <52616A30.6000803@gmail.com> On 18/10/2013 5:31pm, Guido van Rossum wrote: > I'm working from home today and my Windows laptop is in the office, so I > won't be able to test my latest Tulip changes on Windows (I just renamed > pause to pause_reading, and hope to commit pause_reading later today). > Is anyone luckier? $ hg id 97f6f35b02e5 (asyncio) tip $ python-release runtests.py Skipping 'test_unix_events': UNIX only ................................................s.................s......ss...ssssssssssss.....s........s....ss..sss.....ss...ssssssssssssss.................................................................................................................................................................................................................................................................................................................................................................................................................... ---------------------------------------------------------------------- Ran 544 tests in 12.790s OK (skipped=39) > Also, reading through the Windows OpenSSL setup in PCbuild/readme.txt I > lost courage, so I've not tested the SSL behavior on Windows at all. Is > anyone lucky enough to have a Python 3.4 alpha for Windows setup with > OpenSSL already configured? If so, could you run test_asyncio for me and > let me know if there are any issues? (Actually, even a working Python > 3.3 setup would be useful, although you'd have to check out the Tulip > repo and run the tests from there.) $ python-release -c 'import sys; print(sys.version)' 3.4.0a3+ (default:7172135d60f6, Oct 18 2013, 17:54:18) [MSC v.1600 32 bit (Intel)] $ python-release -m test test_asyncio [1/1] test_asyncio 1 test OK. BTW, pcbuild.sln was not building _overlapped automatically -- I have pushed a fix. -- Richard From guido at python.org Fri Oct 18 19:15:52 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 18 Oct 2013 10:15:52 -0700 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: <52616A30.6000803@gmail.com> References: <52616A30.6000803@gmail.com> Message-ID: Thanks! There are some new changes (I fixed a race with sockets closing) and I hope to land flow control (finally) later today. Do you know what those skips are? I suspect they might be due to ssl not working for you either. :-( On Fri, Oct 18, 2013 at 10:04 AM, Richard Oudkerk wrote: > On 18/10/2013 5:31pm, Guido van Rossum wrote: > >> I'm working from home today and my Windows laptop is in the office, so I >> won't be able to test my latest Tulip changes on Windows (I just renamed >> pause to pause_reading, and hope to commit pause_reading later today). >> Is anyone luckier? >> > > $ hg id > 97f6f35b02e5 (asyncio) tip > > $ python-release runtests.py > Skipping 'test_unix_events': UNIX only > ..............................**..................s...........** > ......s......ss...**ssssssssssss.....s........s...**.ss..sss.....ss...** > ssssssssssssss................**..............................** > ..............................**..............................** > ..............................**..............................** > ..............................**..............................** > ..............................**..............................** > ..............................**..............................** > ..............................**............................ > ------------------------------**------------------------------**---------- > Ran 544 tests in 12.790s > > OK (skipped=39) > > > Also, reading through the Windows OpenSSL setup in PCbuild/readme.txt I >> lost courage, so I've not tested the SSL behavior on Windows at all. Is >> anyone lucky enough to have a Python 3.4 alpha for Windows setup with >> OpenSSL already configured? If so, could you run test_asyncio for me and >> let me know if there are any issues? (Actually, even a working Python >> 3.3 setup would be useful, although you'd have to check out the Tulip >> repo and run the tests from there.) >> > > $ python-release -c 'import sys; print(sys.version)' > 3.4.0a3+ (default:7172135d60f6, Oct 18 2013, 17:54:18) [MSC v.1600 32 bit > (Intel)] > > $ python-release -m test test_asyncio > [1/1] test_asyncio > 1 test OK. > > BTW, pcbuild.sln was not building _overlapped automatically -- I have > pushed a fix. > > -- > Richard > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Oct 18 19:30:02 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 18 Oct 2013 19:30:02 +0200 Subject: [Python-Dev] Daily reference leaks (30f33e6a04c1): sum=975756 In-Reply-To: References: Message-ID: <20131018193002.70dc46e6@fsol> Is anyone looking into those leaks? I suspect they might have to do with Serhiy's latest re changes in add40e9f7cbe. (just a barely educated guess, though) Regards Antoine. On Fri, 18 Oct 2013 09:31:49 +0200 solipsis at pitrou.net wrote: > results for 30f33e6a04c1 on branch "default" > -------------------------------------------- > > test_unittest leaked [317, 317, 317] references, sum=951 > test_unittest leaked [196, 197, 197] memory blocks, sum=590 > test_doctest leaked [2232, 2232, 2232] references, sum=6696 > test_doctest leaked [1667, 1669, 1669] memory blocks, sum=5005 > test_argparse leaked [70935, 70935, 70935] references, sum=212805 > test_argparse leaked [43489, 43491, 43491] memory blocks, sum=130471 > test_asyncio leaked [2577, -2577, 0] references, sum=0 > test_asyncio leaked [786, -1103, 2] memory blocks, sum=-315 > test_cgitb leaked [90, 90, 90] references, sum=270 > test_cgitb leaked [65, 65, 65] memory blocks, sum=195 > test_cmd leaked [41, 41, 41] references, sum=123 > test_cmd leaked [20, 20, 20] memory blocks, sum=60 > test_cmd_line leaked [34, 34, 34] references, sum=102 > test_cmd_line leaked [34, 34, 34] memory blocks, sum=102 > test_cmd_line_script leaked [24, 24, 24] references, sum=72 > test_cmd_line_script leaked [19, 19, 19] memory blocks, sum=57 > test_compileall leaked [6, 6, 6] references, sum=18 > test_compileall leaked [6, 6, 6] memory blocks, sum=18 > test_configparser leaked [34560, 34560, 34560] references, sum=103680 > test_configparser leaked [16778, 16780, 16780] memory blocks, sum=50338 > test_contextlib leaked [10, 10, 10] references, sum=30 > test_contextlib leaked [5, 5, 5] memory blocks, sum=15 > test_ctypes leaked [137, 137, 137] references, sum=411 > test_ctypes leaked [105, 105, 105] memory blocks, sum=315 > test_decimal leaked [10, 10, 10] references, sum=30 > test_decimal leaked [8, 8, 8] memory blocks, sum=24 > test_difflib leaked [401, 401, 401] references, sum=1203 > test_difflib leaked [267, 269, 269] memory blocks, sum=805 > test_distutils leaked [533, 533, 533] references, sum=1599 > test_distutils leaked [389, 391, 391] memory blocks, sum=1171 > test_docxmlrpc leaked [60, 60, 60] references, sum=180 > test_docxmlrpc leaked [60, 60, 60] memory blocks, sum=180 > test_email leaked [4307, 4307, 4307] references, sum=12921 > test_email leaked [2419, 2421, 2421] memory blocks, sum=7261 > test_faulthandler leaked [111, 111, 111] references, sum=333 > test_faulthandler leaked [98, 98, 98] memory blocks, sum=294 > test_gc leaked [4, 4, 4] references, sum=12 > test_gc leaked [4, 4, 4] memory blocks, sum=12 > test_generators leaked [5, 5, 5] references, sum=15 > test_generators leaked [4, 4, 4] memory blocks, sum=12 > test_genexps leaked [5, 5, 5] references, sum=15 > test_genexps leaked [4, 4, 4] memory blocks, sum=12 > test_gettext leaked [134, 134, 134] references, sum=402 > test_gettext leaked [128, 128, 128] memory blocks, sum=384 > test_htmlparser leaked [2177, 2177, 2177] references, sum=6531 > test_htmlparser leaked [40, 41, 41] memory blocks, sum=122 > test_http_cookiejar leaked [2805, 2805, 2805] references, sum=8415 > test_http_cookiejar leaked [1808, 1810, 1810] memory blocks, sum=5428 > test_imp leaked [1, -1, 0] references, sum=0 > test_import leaked [16, 16, 16] references, sum=48 > test_import leaked [10, 10, 10] memory blocks, sum=30 > test_inspect leaked [2, 2, 2] references, sum=6 > test_inspect leaked [2, 2, 2] memory blocks, sum=6 > test_json leaked [2553, 2553, 2553] references, sum=7659 > test_json leaked [1323, 1325, 1325] memory blocks, sum=3973 > test_keyword leaked [39, 39, 39] references, sum=117 > test_keyword leaked [20, 20, 20] memory blocks, sum=60 > test_lib2to3 leaked [57052, 57052, 57052] references, sum=171156 > test_lib2to3 leaked [46298, 46300, 46300] memory blocks, sum=138898 > test_listcomps leaked [10, 10, 10] references, sum=30 > test_listcomps leaked [8, 8, 8] memory blocks, sum=24 > test_locale leaked [43, 43, 43] references, sum=129 > test_locale leaked [19, 19, 19] memory blocks, sum=57 > test_logging leaked [706, 706, 706] references, sum=2118 > test_logging leaked [362, 364, 364] memory blocks, sum=1090 > test_mailbox leaked [46, 46, 46] references, sum=138 > test_mailbox leaked [41, 41, 41] memory blocks, sum=123 > test_multiprocessing_forkserver leaked [2, 2, 2] references, sum=6 > test_multiprocessing_forkserver leaked [2, 2, 2] memory blocks, sum=6 > test_multiprocessing_spawn leaked [2, 2, 2] references, sum=6 > test_multiprocessing_spawn leaked [2, 2, 2] memory blocks, sum=6 > test_netrc leaked [96, 96, 96] references, sum=288 > test_netrc leaked [59, 59, 59] memory blocks, sum=177 > test_nntplib leaked [1774, 1774, 1774] references, sum=5322 > test_nntplib leaked [902, 904, 904] memory blocks, sum=2710 > test_parser leaked [2, 2, 2] references, sum=6 > test_parser leaked [2, 2, 2] memory blocks, sum=6 > test_pep292 leaked [174, 174, 174] references, sum=522 > test_pep292 leaked [107, 107, 107] memory blocks, sum=321 > test_pkg leaked [57, 57, 57] references, sum=171 > test_pkg leaked [31, 31, 31] memory blocks, sum=93 > test_pydoc leaked [1804, 1804, 1804] references, sum=5412 > test_pydoc leaked [684, 686, 686] memory blocks, sum=2056 > test_re leaked [10539, 10539, 10539] references, sum=31617 > test_re leaked [135, 136, 136] memory blocks, sum=407 > test_regrtest leaked [5005, 5005, 5005] references, sum=15015 > test_regrtest leaked [2704, 2706, 2706] memory blocks, sum=8116 > test_robotparser leaked [4, 4, 4] references, sum=12 > test_robotparser leaked [4, 4, 4] memory blocks, sum=12 > test_setcomps leaked [10, 10, 10] references, sum=30 > test_setcomps leaked [8, 8, 8] memory blocks, sum=24 > test_site leaked [0, 2, 2] references, sum=4 > test_site leaked [0, 2, 2] memory blocks, sum=4 > test_smtplib leaked [65, 65, 65] references, sum=195 > test_smtplib leaked [45, 45, 45] memory blocks, sum=135 > test_ssl leaked [15, 15, 15] references, sum=45 > test_ssl leaked [10, 10, 10] memory blocks, sum=30 > test_strptime leaked [627, 627, 627] references, sum=1881 > test_strptime leaked [433, 435, 435] memory blocks, sum=1303 > test_subprocess leaked [138, 138, 138] references, sum=414 > test_subprocess leaked [102, 102, 102] memory blocks, sum=306 > test_sys leaked [22, 22, 22] references, sum=66 > test_sys leaked [12, 12, 12] memory blocks, sum=36 > test_textwrap leaked [192, 192, 192] references, sum=576 > test_textwrap leaked [116, 116, 116] memory blocks, sum=348 > test_threading leaked [2, 2, 2] references, sum=6 > test_threading leaked [2, 2, 2] memory blocks, sum=6 > test_time leaked [100, 100, 100] references, sum=300 > test_time leaked [94, 94, 94] memory blocks, sum=282 > test_timeit leaked [34, 34, 34] references, sum=102 > test_timeit leaked [20, 20, 20] memory blocks, sum=60 > test_tokenize leaked [5, 5, 5] references, sum=15 > test_tokenize leaked [4, 4, 4] memory blocks, sum=12 > test_tools leaked [398, 398, 398] references, sum=1194 > test_tools leaked [214, 215, 215] memory blocks, sum=644 > test_urllib leaked [16, 16, 16] references, sum=48 > test_urllib leaked [11, 11, 11] memory blocks, sum=33 > test_urllib2 leaked [333, 333, 333] references, sum=999 > test_urllib2 leaked [216, 217, 217] memory blocks, sum=650 > test_urllib2_localnet leaked [61, 61, 61] references, sum=183 > test_urllib2_localnet leaked [41, 41, 41] memory blocks, sum=123 > test_urllib2net leaked [38, 38, 38] references, sum=114 > test_urllib2net leaked [29, 29, 29] memory blocks, sum=87 > test_urllibnet leaked [38, 38, 38] references, sum=114 > test_urllibnet leaked [24, 24, 24] memory blocks, sum=72 > test_warnings leaked [4, 4, 4] references, sum=12 > test_warnings leaked [4, 4, 4] memory blocks, sum=12 > test_weakref leaked [2, 2, 2] references, sum=6 > test_weakref leaked [2, 2, 2] memory blocks, sum=6 > test_xmlrpc leaked [2, 2, 2] references, sum=6 > test_zipfile leaked [8, 8, 8] references, sum=24 > test_zipfile leaked [4, 4, 4] memory blocks, sum=12 > test_zipimport_support leaked [1492, 1492, 1492] references, sum=4476 > test_zipimport_support leaked [1146, 1148, 1148] memory blocks, sum=3442 > > > Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog6vlxGJ', '-x'] From zachary.ware+pydev at gmail.com Fri Oct 18 19:34:35 2013 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Fri, 18 Oct 2013 12:34:35 -0500 Subject: [Python-Dev] Daily reference leaks (30f33e6a04c1): sum=975756 In-Reply-To: <20131018193002.70dc46e6@fsol> References: <20131018193002.70dc46e6@fsol> Message-ID: On Fri, Oct 18, 2013 at 12:30 PM, Antoine Pitrou wrote: > > Is anyone looking into those leaks? > I suspect they might have to do with Serhiy's latest re changes in > add40e9f7cbe. > (just a barely educated guess, though) I have confirmed that and was working towards getting my findings posted when this message came in :). Specifically, the leak is somewhere in sub (I believe pattern_subx() in _sre.c), but I currently lack the skill to figure out where. -- Zach From shibturn at gmail.com Fri Oct 18 19:36:28 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Fri, 18 Oct 2013 18:36:28 +0100 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: References: <52616A30.6000803@gmail.com> Message-ID: <5261719C.7000301@gmail.com> On 18/10/2013 6:15pm, Guido van Rossum wrote: > Thanks! There are some new changes (I fixed a race with sockets closing) > and I hope to land flow control (finally) later today. > > Do you know what those skips are? I suspect they might be due to ssl not > working for you either. :-( Lack of support for subprocess, signals, ssl (with iocp), pipes, add_*() (with iocp): $ python-release runtests.py -v 2>&1 | grep skipped test_add_signal_handler (test_events.SelectEventLoopTests) ... skipped 'No SIGKILL' test_read_pipe (test_events.SelectEventLoopTests) ... skipped "Don't support pipes for Windows" test_signal_handling_args (test_events.SelectEventLoopTests) ... skipped 'No SIGALRM' test_signal_handling_while_selecting (test_events.SelectEventLoopTests) ... skipped 'No SIGALRM' test_subprocess_close_after_finish (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_close_client_stream (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_exec (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_exitcode (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_interactive (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_kill (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_send_signal (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_shell (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_stderr (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_stderr_redirect_to_stdout (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_write_pipe (test_events.SelectEventLoopTests) ... skipped "Don't support pipes for Windows" test_write_pipe_disconnect_on_close (test_events.SelectEventLoopTests) ... skipped "Don't support pipes for Windows" test_add_signal_handler (test_events.ProactorEventLoopTests) ... skipped 'No SIGKILL' test_create_datagram_endpoint (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have create_datagram_endpoint()' test_create_server_ssl (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop imcompatible with SSL' test_create_ssl_connection (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop imcompatible with SSL' test_read_pipe (test_events.ProactorEventLoopTests) ... skipped "Don't support pipes for Windows" test_reader_callback (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_reader()' test_reader_callback_cancel (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_reader()' test_signal_handling_args (test_events.ProactorEventLoopTests) ... skipped 'No SIGALRM' test_signal_handling_while_selecting (test_events.ProactorEventLoopTests) ... skipped 'No SIGALRM' test_subprocess_close_after_finish (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_close_client_stream (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_exec (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_exitcode (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_interactive (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_kill (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_send_signal (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_shell (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_stderr (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_stderr_redirect_to_stdout (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_write_pipe (test_events.ProactorEventLoopTests) ... skipped "Don't support pipes for Windows" test_write_pipe_disconnect_on_close (test_events.ProactorEventLoopTests) ... skipped "Don't support pipes for Windows" test_writer_callback (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_writer()' test_writer_callback_cancel (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_writer()' OK (skipped=39) -- Richard From shibturn at gmail.com Fri Oct 18 19:36:28 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Fri, 18 Oct 2013 18:36:28 +0100 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: References: <52616A30.6000803@gmail.com> Message-ID: <5261719C.7000301@gmail.com> On 18/10/2013 6:15pm, Guido van Rossum wrote: > Thanks! There are some new changes (I fixed a race with sockets closing) > and I hope to land flow control (finally) later today. > > Do you know what those skips are? I suspect they might be due to ssl not > working for you either. :-( Lack of support for subprocess, signals, ssl (with iocp), pipes, add_*() (with iocp): $ python-release runtests.py -v 2>&1 | grep skipped test_add_signal_handler (test_events.SelectEventLoopTests) ... skipped 'No SIGKILL' test_read_pipe (test_events.SelectEventLoopTests) ... skipped "Don't support pipes for Windows" test_signal_handling_args (test_events.SelectEventLoopTests) ... skipped 'No SIGALRM' test_signal_handling_while_selecting (test_events.SelectEventLoopTests) ... skipped 'No SIGALRM' test_subprocess_close_after_finish (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_close_client_stream (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_exec (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_exitcode (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_interactive (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_kill (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_send_signal (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_shell (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_stderr (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_stderr_redirect_to_stdout (test_events.SelectEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_write_pipe (test_events.SelectEventLoopTests) ... skipped "Don't support pipes for Windows" test_write_pipe_disconnect_on_close (test_events.SelectEventLoopTests) ... skipped "Don't support pipes for Windows" test_add_signal_handler (test_events.ProactorEventLoopTests) ... skipped 'No SIGKILL' test_create_datagram_endpoint (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have create_datagram_endpoint()' test_create_server_ssl (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop imcompatible with SSL' test_create_ssl_connection (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop imcompatible with SSL' test_read_pipe (test_events.ProactorEventLoopTests) ... skipped "Don't support pipes for Windows" test_reader_callback (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_reader()' test_reader_callback_cancel (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_reader()' test_signal_handling_args (test_events.ProactorEventLoopTests) ... skipped 'No SIGALRM' test_signal_handling_while_selecting (test_events.ProactorEventLoopTests) ... skipped 'No SIGALRM' test_subprocess_close_after_finish (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_close_client_stream (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_exec (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_exitcode (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_interactive (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_kill (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_send_signal (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_shell (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_stderr (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_subprocess_stderr_redirect_to_stdout (test_events.ProactorEventLoopTests) ... skipped "Don't support subprocess for Windows yet" test_write_pipe (test_events.ProactorEventLoopTests) ... skipped "Don't support pipes for Windows" test_write_pipe_disconnect_on_close (test_events.ProactorEventLoopTests) ... skipped "Don't support pipes for Windows" test_writer_callback (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_writer()' test_writer_callback_cancel (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_writer()' OK (skipped=39) -- Richard From brett at python.org Fri Oct 18 19:46:21 2013 From: brett at python.org (Brett Cannon) Date: Fri, 18 Oct 2013 13:46:21 -0400 Subject: [Python-Dev] Current start-up times Message-ID: tl;dr: 2.7 -> 3.3 = 1.92x slower 2.7 -> 3.4 = 1.36x slower 3.3 -> 3.4 = 1.40x faster IOW the work people have been putting in to speed up interpreter startup are definitely paying off. > ./perf.py -b normal_startup,startup_nosite ../cpython/py3.3/python.exe ../cpython/default/python.exe Running normal_startup... INFO:root:Running `../cpython/default/python.exe -c ` 1000 times INFO:root:Running `../cpython/py3.3/python.exe -c ` 1000 times Running startup_nosite... INFO:root:Running `../cpython/default/python.exe -S -c ` 2000 times INFO:root:Running `../cpython/py3.3/python.exe -S -c ` 2000 times Report on Darwin bcannon-macbookpro2.roam.corp.google.com 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64 i386 Total CPU cores: 8 ### normal_startup ### Min: 0.679865 -> 0.484924: 1.40x faster Avg: 0.696644 -> 0.497123: 1.40x faster Significant (t=47.03) Stddev: 0.02796 -> 0.01087: 2.5731x smaller ### startup_nosite ### Min: 0.297685 -> 0.285106: 1.04x faster Avg: 0.319151 -> 0.309291: 1.03x faster Significant (t=5.64) Stddev: 0.01219 -> 0.01252: 1.0272x larger > ./perf.py -b normal_startup,startup_nosite ../cpython/py2.7/python.exe ../cpython/default/python.exe Running normal_startup... INFO:root:Running `../cpython/default/python.exe -c ` 1000 times INFO:root:Running `../cpython/py2.7/python.exe -c ` 1000 times Running startup_nosite... INFO:root:Running `../cpython/default/python.exe -S -c ` 2000 times INFO:root:Running `../cpython/py2.7/python.exe -S -c ` 2000 times Report on Darwin bcannon-macbookpro2.roam.corp.google.com 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64 i386 Total CPU cores: 8 ### normal_startup ### Min: 0.350003 -> 0.484200: 1.38x slower Avg: 0.362935 -> 0.492708: 1.36x slower Significant (t=-44.29) Stddev: 0.02021 -> 0.00459: 4.3999x smaller ### startup_nosite ### Min: 0.165302 -> 0.282765: 1.71x slower Avg: 0.169915 -> 0.293589: 1.73x slower Significant (t=-222.46) Stddev: 0.00273 -> 0.00484: 1.7705x larger > ./perf.py -b normal_startup,startup_nosite ../cpython/py2.7/python.exe ../cpython/py3.3/python.exe Running normal_startup... INFO:root:Running `../cpython/py3.3/python.exe -c ` 1000 times INFO:root:Running `../cpython/py2.7/python.exe -c ` 1000 times Running startup_nosite... INFO:root:Running `../cpython/py3.3/python.exe -S -c ` 2000 times INFO:root:Running `../cpython/py2.7/python.exe -S -c ` 2000 times Report on Darwin bcannon-macbookpro2.roam.corp.google.com 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64 i386 Total CPU cores: 8 ### normal_startup ### Min: 0.350545 -> 0.675245: 1.93x slower Avg: 0.363317 -> 0.696069: 1.92x slower Significant (t=-120.97) Stddev: 0.01682 -> 0.00977: 1.7222x smaller ### startup_nosite ### Min: 0.166232 -> 0.289506: 1.74x slower Avg: 0.172511 -> 0.300979: 1.74x slower Significant (t=-209.92) Stddev: 0.00299 -> 0.00534: 1.7862x larger -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Oct 18 19:51:14 2013 From: brett at python.org (Brett Cannon) Date: Fri, 18 Oct 2013 13:51:14 -0400 Subject: [Python-Dev] Daily reference leaks (30f33e6a04c1): sum=975756 In-Reply-To: References: <20131018193002.70dc46e6@fsol> Message-ID: On Fri, Oct 18, 2013 at 1:34 PM, Zachary Ware wrote: > On Fri, Oct 18, 2013 at 12:30 PM, Antoine Pitrou > wrote: > > > > Is anyone looking into those leaks? > > I suspect they might have to do with Serhiy's latest re changes in > > add40e9f7cbe. > > (just a barely educated guess, though) > > I have confirmed that and was working towards getting my findings > posted when this message came in :). Specifically, the leak is > somewhere in sub (I believe pattern_subx() in _sre.c), but I currently > lack the skill to figure out where. > I think there is a Py_DECREF(list) call missing at line 2434. Running the test suite now to verify. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Oct 18 19:53:21 2013 From: brett at python.org (Brett Cannon) Date: Fri, 18 Oct 2013 13:53:21 -0400 Subject: [Python-Dev] Daily reference leaks (30f33e6a04c1): sum=975756 In-Reply-To: References: <20131018193002.70dc46e6@fsol> Message-ID: On Fri, Oct 18, 2013 at 1:51 PM, Brett Cannon wrote: > > > On Fri, Oct 18, 2013 at 1:34 PM, Zachary Ware < > zachary.ware+pydev at gmail.com> wrote: > >> On Fri, Oct 18, 2013 at 12:30 PM, Antoine Pitrou >> wrote: >> > >> > Is anyone looking into those leaks? >> > I suspect they might have to do with Serhiy's latest re changes in >> > add40e9f7cbe. >> > (just a barely educated guess, though) >> >> I have confirmed that and was working towards getting my findings >> posted when this message came in :). Specifically, the leak is >> somewhere in sub (I believe pattern_subx() in _sre.c), but I currently >> lack the skill to figure out where. >> > > I think there is a Py_DECREF(list) call missing at line 2434. Running the > test suite now to verify. > Make that line 2430. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cf.natali at gmail.com Fri Oct 18 19:56:30 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Fri, 18 Oct 2013 19:56:30 +0200 Subject: [Python-Dev] PEP 454 (tracemalloc): new minimalist version In-Reply-To: References: Message-ID: Hi, I'm happy to see this move forward! > API > === > > Main Functions > -------------- > > ``clear_traces()`` function: > > Clear traces and statistics on Python memory allocations, and reset > the ``get_traced_memory()`` counter. That's nitpicking, but how about just ``reset()`` (I'm probably biased by oprofile's opcontrol --reset)? > ``get_stats()`` function: > > Get statistics on traced Python memory blocks as a dictionary > ``{filename (str): {line_number (int): stats}}`` where *stats* in a > ``(size: int, count: int)`` tuple, *filename* and *line_number* can > be ``None``. It's probably obvious, but you might want to say once what *size* and *count* represent (and the unit for *size*). > ``get_tracemalloc_memory()`` function: > > Get the memory usage in bytes of the ``tracemalloc`` module as a > tuple: ``(size: int, free: int)``. > > * *size*: total size of bytes allocated by the module, > including *free* bytes > * *free*: number of free bytes available to store data What's *free* exactly? I assume it's linked to the internal storage area used by tracemalloc itself, but that's not clear at all. Also, is the tracemalloc overhead included in the above stats (I'm mainly thinking about get_stats() and get_traced_memory()? If yes, I find it somewhat confusing: for example, AFAICT, valgrind's memcheck doesn't report the memory overhead, although it can be quite large, simply because it's not interesting. > Trace Functions > --------------- > > ``get_traceback_limit()`` function: > > Get the maximum number of frames stored in the traceback of a trace > of a memory block. > > Use the ``set_traceback_limit()`` function to change the limit. I didn't see anywhere the default value for this setting: it would be nice to write it somewhere, and also explain the rationale (memory/CPU overhead...). > ``get_object_address(obj)`` function: > > Get the address of the main memory block of the specified Python object. > > A Python object can be composed by multiple memory blocks, the > function only returns the address of the main memory block. IOW, this should return the same as id() on CPython? If yes, it could be an interesting note. > ``get_object_trace(obj)`` function: > > Get the trace of a Python object *obj* as a ``(size: int, > traceback)`` tuple where *traceback* is a tuple of ``(filename: str, > lineno: int)`` tuples, *filename* and *lineno* can be ``None``. I find the "trace" word confusing, so it might be interesting to add a note somewhere explaining what it is ("callstack leading to the object allocation", or whatever). Also, this function leaves me a mixed feeling: it's called get_object_trace(), but you also return the object size - well, a vague estimate thereof. I wonder if the size really belongs here, especially if the information returned isn't really accurate: it will be for an integer, but not for e.g. a list, right? How about just using sys.getsizeof(), which would give a more accurate result? > ``get_trace(address)`` function: > > Get the trace of a memory block as a ``(size: int, traceback)`` > tuple where *traceback* is a tuple of ``(filename: str, lineno: > int)`` tuples, *filename* and *lineno* can be ``None``. > > Return ``None`` if the ``tracemalloc`` module did not trace the > allocation of the memory block. > > See also ``get_object_trace()``, ``get_stats()`` and > ``get_traces()`` functions. Do you have example use cases where you want to work with a raw addresses? > Filter > ------ > > ``Filter(include: bool, pattern: str, lineno: int=None, traceback: > bool=False)`` class: > > Filter to select which memory allocations are traced. Filters can be > used to reduce the memory usage of the ``tracemalloc`` module, which > can be read using the ``get_tracemalloc_memory()`` function. > > ``match(filename: str, lineno: int)`` method: > > Return ``True`` if the filter matchs the filename and line number, > ``False`` otherwise. > > ``match_filename(filename: str)`` method: > > Return ``True`` if the filter matchs the filename, ``False`` otherwise. > > ``match_lineno(lineno: int)`` method: > > Return ``True`` if the filter matchs the line number, ``False`` > otherwise. > > ``match_traceback(traceback)`` method: > > Return ``True`` if the filter matchs the *traceback*, ``False`` > otherwise. > > *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples. Are those ``match`` methods really necessary for the end user, i.e. are they worth being exposed as part of the public API? > StatsDiff > --------- > > ``StatsDiff(differences, old_stats, new_stats)`` class: > > Differences between two ``GroupedStats`` instances. > > The ``GroupedStats.compare_to()`` method creates a ``StatsDiff`` > instance. > > ``sort()`` method: > > Sort the ``differences`` list from the biggest difference to the > smallest difference. Sort by ``abs(size_diff)``, *size*, > ``abs(count_diff)``, *count* and then by *key*. > > ``differences`` attribute: > > Differences between ``old_stats`` and ``new_stats`` as a list of > ``(size_diff, size, count_diff, count, key)`` tuples. *size_diff*, > *size*, *count_diff* and *count* are ``int``. The key type depends > on the ``GroupedStats.group_by`` attribute of ``new_stats``: see the > ``Snapshot.top_by()`` method. > > ``old_stats`` attribute: > > Old ``GroupedStats`` instance, can be ``None``. > > ``new_stats`` attribute: > > New ``GroupedStats`` instance. Why keep references to ``old_stats`` and ``new_stats``? datetime.timedelta doesn't keep references to the date objects it was computed from. Also, if you sort the difference by default (which is a sensible choice), then the StatsDiff becomes pretty much useless, since you would just keep its ``differences`` attribute (sorted). > Snapshot > -------- > > ``Snapshot(timestamp: datetime.datetime, traces: dict=None, stats: > dict=None)`` class: > > Snapshot of traces and statistics on memory blocks allocated by Python. I'm confused. Why are get_trace(), get_object_trace(), get_stats() etc not methods of a Snapshot object? Is it because you don't store all the necessary information in a snapshot, or are they just some sort of shorthands, like: stats = get_stats() vs snapshot = Snapshot.create() stats = snapshot.stats > ``write(filename)`` method: > > Write the snapshot into a file. I assume it's in a serialized form, only readable by Snapshort.load() ? BTW, it's a nitpick and debatable, but write()/read() or load()/dump() would be more consistent (see e.g. pickle's load/dump). > Metric > ------ > > ``Metric(name: str, value: int, format: str)`` class: > > Value of a metric when a snapshot is created. Alright, what's a metric again ;-) ? I don't know if it's customary, but having short examples would IMO be nice. cf From guido at python.org Fri Oct 18 19:57:38 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 18 Oct 2013 10:57:38 -0700 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: <5261719C.7000301@gmail.com> References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> Message-ID: Thanks! Those are all expected (though contributions are always welcome -- not looking at you specifically :-). Does examples/fetch3.py work for you with an https URL? (Try http://dropbox.com, i.e. without 's' -- you get two redirects to https URLs. :-) On Fri, Oct 18, 2013 at 10:36 AM, Richard Oudkerk wrote: > On 18/10/2013 6:15pm, Guido van Rossum wrote: > >> Thanks! There are some new changes (I fixed a race with sockets closing) >> and I hope to land flow control (finally) later today. >> >> Do you know what those skips are? I suspect they might be due to ssl not >> working for you either. :-( >> > > Lack of support for subprocess, signals, ssl (with iocp), pipes, add_*() > (with iocp): > > $ python-release runtests.py -v 2>&1 | grep skipped > test_add_signal_handler (test_events.**SelectEventLoopTests) ... skipped > 'No SIGKILL' > test_read_pipe (test_events.**SelectEventLoopTests) ... skipped "Don't > support pipes for Windows" > test_signal_handling_args (test_events.**SelectEventLoopTests) ... > skipped 'No SIGALRM' > test_signal_handling_while_**selecting (test_events.**SelectEventLoopTests) > ... skipped 'No SIGALRM' > test_subprocess_close_after_**finish (test_events.**SelectEventLoopTests) > ... skipped "Don't support subprocess for Windows yet" > test_subprocess_close_client_**stream (test_events.**SelectEventLoopTests) > ... skipped "Don't support subprocess for Windows yet" > test_subprocess_exec (test_events.**SelectEventLoopTests) ... skipped > "Don't support subprocess for Windows yet" > test_subprocess_exitcode (test_events.**SelectEventLoopTests) ... skipped > "Don't support subprocess for Windows yet" > test_subprocess_interactive (test_events.**SelectEventLoopTests) ... > skipped "Don't support subprocess for Windows yet" > test_subprocess_kill (test_events.**SelectEventLoopTests) ... skipped > "Don't support subprocess for Windows yet" > test_subprocess_send_signal (test_events.**SelectEventLoopTests) ... > skipped "Don't support subprocess for Windows yet" > test_subprocess_shell (test_events.**SelectEventLoopTests) ... skipped > "Don't support subprocess for Windows yet" > test_subprocess_stderr (test_events.**SelectEventLoopTests) ... skipped > "Don't support subprocess for Windows yet" > test_subprocess_stderr_**redirect_to_stdout (test_events.**SelectEventLoopTests) > ... skipped "Don't support subprocess for Windows yet" > test_write_pipe (test_events.**SelectEventLoopTests) ... skipped "Don't > support pipes for Windows" > test_write_pipe_disconnect_on_**close (test_events.**SelectEventLoopTests) > ... skipped "Don't support pipes for Windows" > test_add_signal_handler (test_events.**ProactorEventLoopTests) ... > skipped 'No SIGKILL' > test_create_datagram_endpoint (test_events.**ProactorEventLoopTests) ... > skipped 'IocpEventLoop does not have create_datagram_endpoint()' > test_create_server_ssl (test_events.**ProactorEventLoopTests) ... skipped > 'IocpEventLoop imcompatible with SSL' > test_create_ssl_connection (test_events.**ProactorEventLoopTests) ... > skipped 'IocpEventLoop imcompatible with SSL' > test_read_pipe (test_events.**ProactorEventLoopTests) ... skipped "Don't > support pipes for Windows" > test_reader_callback (test_events.**ProactorEventLoopTests) ... skipped > 'IocpEventLoop does not have add_reader()' > test_reader_callback_cancel (test_events.**ProactorEventLoopTests) ... > skipped 'IocpEventLoop does not have add_reader()' > test_signal_handling_args (test_events.**ProactorEventLoopTests) ... > skipped 'No SIGALRM' > test_signal_handling_while_**selecting (test_events.**ProactorEventLoopTests) > ... skipped 'No SIGALRM' > test_subprocess_close_after_**finish (test_events.**ProactorEventLoopTests) > ... skipped "Don't support subprocess for Windows yet" > test_subprocess_close_client_**stream (test_events.**ProactorEventLoopTests) > ... skipped "Don't support subprocess for Windows yet" > test_subprocess_exec (test_events.**ProactorEventLoopTests) ... skipped > "Don't support subprocess for Windows yet" > test_subprocess_exitcode (test_events.**ProactorEventLoopTests) ... > skipped "Don't support subprocess for Windows yet" > test_subprocess_interactive (test_events.**ProactorEventLoopTests) ... > skipped "Don't support subprocess for Windows yet" > test_subprocess_kill (test_events.**ProactorEventLoopTests) ... skipped > "Don't support subprocess for Windows yet" > test_subprocess_send_signal (test_events.**ProactorEventLoopTests) ... > skipped "Don't support subprocess for Windows yet" > test_subprocess_shell (test_events.**ProactorEventLoopTests) ... skipped > "Don't support subprocess for Windows yet" > test_subprocess_stderr (test_events.**ProactorEventLoopTests) ... skipped > "Don't support subprocess for Windows yet" > test_subprocess_stderr_**redirect_to_stdout (test_events.**ProactorEventLoopTests) > ... skipped "Don't support subprocess for Windows yet" > test_write_pipe (test_events.**ProactorEventLoopTests) ... skipped "Don't > support pipes for Windows" > test_write_pipe_disconnect_on_**close (test_events.**ProactorEventLoopTests) > ... skipped "Don't support pipes for Windows" > test_writer_callback (test_events.**ProactorEventLoopTests) ... skipped > 'IocpEventLoop does not have add_writer()' > test_writer_callback_cancel (test_events.**ProactorEventLoopTests) ... > skipped 'IocpEventLoop does not have add_writer()' > OK (skipped=39) > > -- > Richard > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Oct 18 20:04:09 2013 From: brett at python.org (Brett Cannon) Date: Fri, 18 Oct 2013 14:04:09 -0400 Subject: [Python-Dev] Daily reference leaks (30f33e6a04c1): sum=975756 In-Reply-To: References: <20131018193002.70dc46e6@fsol> Message-ID: On Fri, Oct 18, 2013 at 1:53 PM, Brett Cannon wrote: > > > > On Fri, Oct 18, 2013 at 1:51 PM, Brett Cannon wrote: > >> >> >> On Fri, Oct 18, 2013 at 1:34 PM, Zachary Ware < >> zachary.ware+pydev at gmail.com> wrote: >> >>> On Fri, Oct 18, 2013 at 12:30 PM, Antoine Pitrou >>> wrote: >>> > >>> > Is anyone looking into those leaks? >>> > I suspect they might have to do with Serhiy's latest re changes in >>> > add40e9f7cbe. >>> > (just a barely educated guess, though) >>> >>> I have confirmed that and was working towards getting my findings >>> posted when this message came in :). Specifically, the leak is >>> somewhere in sub (I believe pattern_subx() in _sre.c), but I currently >>> lack the skill to figure out where. >>> >> >> I think there is a Py_DECREF(list) call missing at line 2434. Running the >> test suite now to verify. >> > > Make that line 2430. > That was it, so Antoine and Zach were right about the location. Should be fixed now. -------------- next part -------------- An HTML attachment was scrubbed... URL: From shibturn at gmail.com Fri Oct 18 20:26:14 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Fri, 18 Oct 2013 19:26:14 +0100 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> Message-ID: <52617D46.1000001@gmail.com> On 18/10/2013 6:57pm, Guido van Rossum wrote: > Thanks! Those are all expected (though contributions are always welcome > -- not looking at you specifically :-). > > Does examples/fetch3.py work for you with an https URL? (Try > http://dropbox.com, i.e. without 's' -- you get two redirects to https > URLs. :-) It fails -- not sure why... $ python-release examples/fetch3.py http://dropbox.com redirect to https://dropbox.com/ Traceback (most recent call last): File "examples/fetch3.py", line 211, in main() File "examples/fetch3.py", line 206, in main body = loop.run_until_complete(fetch(sys.argv[1], '-v' in sys.argv)) File "C:\Repos\cpython-dirty\lib\asyncio\base_events.py", line 172, in run_until_complete self.run_forever() File "C:\Repos\cpython-dirty\lib\asyncio\base_events.py", line 153, in run_forever self._run_once() File "C:\Repos\cpython-dirty\lib\asyncio\base_events.py", line 576, in _run_once event_list = self._selector.select(timeout) File "C:\Repos\cpython-dirty\lib\selectors.py", line 219, in select r, w, _ = self._select(self._readers, self._writers, [], timeout) File "C:\Repos\cpython-dirty\lib\selectors.py", line 210, in _select r, w, x = select.select(r, w, w, timeout) OSError: [WinError 10038] An operation was attempted on something that is not a socket -- Richard From shibturn at gmail.com Fri Oct 18 20:26:14 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Fri, 18 Oct 2013 19:26:14 +0100 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> Message-ID: <52617D46.1000001@gmail.com> On 18/10/2013 6:57pm, Guido van Rossum wrote: > Thanks! Those are all expected (though contributions are always welcome > -- not looking at you specifically :-). > > Does examples/fetch3.py work for you with an https URL? (Try > http://dropbox.com, i.e. without 's' -- you get two redirects to https > URLs. :-) It fails -- not sure why... $ python-release examples/fetch3.py http://dropbox.com redirect to https://dropbox.com/ Traceback (most recent call last): File "examples/fetch3.py", line 211, in main() File "examples/fetch3.py", line 206, in main body = loop.run_until_complete(fetch(sys.argv[1], '-v' in sys.argv)) File "C:\Repos\cpython-dirty\lib\asyncio\base_events.py", line 172, in run_until_complete self.run_forever() File "C:\Repos\cpython-dirty\lib\asyncio\base_events.py", line 153, in run_forever self._run_once() File "C:\Repos\cpython-dirty\lib\asyncio\base_events.py", line 576, in _run_once event_list = self._selector.select(timeout) File "C:\Repos\cpython-dirty\lib\selectors.py", line 219, in select r, w, _ = self._select(self._readers, self._writers, [], timeout) File "C:\Repos\cpython-dirty\lib\selectors.py", line 210, in _select r, w, x = select.select(r, w, w, timeout) OSError: [WinError 10038] An operation was attempted on something that is not a socket -- Richard From storchaka at gmail.com Fri Oct 18 20:57:22 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 18 Oct 2013 21:57:22 +0300 Subject: [Python-Dev] Daily reference leaks (30f33e6a04c1): sum=975756 In-Reply-To: References: <20131018193002.70dc46e6@fsol> Message-ID: 18.10.13 21:04, Brett Cannon ???????(??): > That was it, so Antoine and Zach were right about the location. Should > be fixed now. Thank you Brett. From guido at python.org Fri Oct 18 22:19:11 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 18 Oct 2013 13:19:11 -0700 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: <52617D46.1000001@gmail.com> References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> Message-ID: Maybe the dummy socket returned by wrap_socket() is not acceptable for select? --Guido van Rossum (sent from Android phone) On Oct 18, 2013 11:26 AM, "Richard Oudkerk" wrote: > On 18/10/2013 6:57pm, Guido van Rossum wrote: > >> Thanks! Those are all expected (though contributions are always welcome >> -- not looking at you specifically :-). >> >> Does examples/fetch3.py work for you with an https URL? (Try >> http://dropbox.com, i.e. without 's' -- you get two redirects to https >> URLs. :-) >> > > It fails -- not sure why... > > $ python-release examples/fetch3.py http://dropbox.com > redirect to https://dropbox.com/ > Traceback (most recent call last): > File "examples/fetch3.py", line 211, in > main() > File "examples/fetch3.py", line 206, in main > body = loop.run_until_complete(fetch(**sys.argv[1], '-v' in sys.argv)) > File "C:\Repos\cpython-dirty\lib\**asyncio\base_events.py", line 172, > in run_until_complete > self.run_forever() > File "C:\Repos\cpython-dirty\lib\**asyncio\base_events.py", line 153, > in run_forever > self._run_once() > File "C:\Repos\cpython-dirty\lib\**asyncio\base_events.py", line 576, > in _run_once > event_list = self._selector.select(timeout) > File "C:\Repos\cpython-dirty\lib\**selectors.py", line 219, in select > r, w, _ = self._select(self._readers, self._writers, [], timeout) > File "C:\Repos\cpython-dirty\lib\**selectors.py", line 210, in _select > r, w, x = select.select(r, w, w, timeout) > OSError: [WinError 10038] An operation was attempted on something that is > not a socket > > -- > Richard > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Oct 18 22:59:29 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 18 Oct 2013 22:59:29 +0200 Subject: [Python-Dev] possibility of shaving a stat call from imports References: Message-ID: <20131018225929.10aebe3c@fsol> On Fri, 18 Oct 2013 12:53:55 -0400 Brett Cannon wrote: > importlib.machinery.FileFinder does a stat call to check if a path is a > file if the package check failed. Now I'm willing to bet that the check is > rather redundant as the file extension should be a dead give-away that > something in a directory is a file and not some non-file type. The import > would still fail even if this is the case in the loader when attempting to > read from the file, but it happens a little later and it means finders > would be more permissive in claiming they found a loader. Is it one stat() call per successful import? Or one stat() call per sys.path entry? Regards Antoine. From shibturn at gmail.com Fri Oct 18 23:04:44 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Fri, 18 Oct 2013 22:04:44 +0100 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> Message-ID: <5261A26C.7000506@gmail.com> On 18/10/2013 9:19pm, Guido van Rossum wrote: > Maybe the dummy socket returned by wrap_socket() is not acceptable for > select? An error SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:553)') is being raised in _on_handshake(). This seems to result in the socket being closed without being unregistered from the selector. select() fails before the SSLError gets reported, so it does not appear in the traceback. -- Richard From shibturn at gmail.com Fri Oct 18 23:04:44 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Fri, 18 Oct 2013 22:04:44 +0100 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> Message-ID: <5261A26C.7000506@gmail.com> On 18/10/2013 9:19pm, Guido van Rossum wrote: > Maybe the dummy socket returned by wrap_socket() is not acceptable for > select? An error SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:553)') is being raised in _on_handshake(). This seems to result in the socket being closed without being unregistered from the selector. select() fails before the SSLError gets reported, so it does not appear in the traceback. -- Richard From timothy.c.delaney at gmail.com Fri Oct 18 23:12:11 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Sat, 19 Oct 2013 08:12:11 +1100 Subject: [Python-Dev] possibility of shaving a stat call from imports In-Reply-To: References: Message-ID: On 19 October 2013 03:53, Brett Cannon wrote: > importlib.machinery.FileFinder does a stat call to check if a path is a > file if the package check failed. Now I'm willing to bet that the check is > rather redundant as the file extension should be a dead give-away that > something in a directory is a file and not some non-file type. The import > would still fail even if this is the case in the loader when attempting to > read from the file, but it happens a little later and it means finders > would be more permissive in claiming they found a loader. > > Does anyone see a good reason not to take the more optimistic route in the > finder? As I said, the only thing I see breaking user code is if they have > a directory or something named spam.py and so the finder claims it found a > module when in fact it didn't and thus stopping the search for the module > from continuing. > Whilst directories with extensions are unusual on Windows, they're fairly common on UNIX-based systems. For example, blah.rc directories. And I personally often create directories with extensions - usually a timestamp of some kind. If the extension is specifically an extension that Python uses (e.g. .py[cod]) then I think it would be reasonable to make the assumption and let the import fail at the loader instead. Would the extension check be faster or slower than another stat() call? As an alternative, is there another stat call later that could be bypassed if you temporarily cached the result of this stat call? And if so, when should the cached value be cleared? Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Oct 18 23:37:29 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 18 Oct 2013 14:37:29 -0700 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: <5261A26C.7000506@gmail.com> References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> <5261A26C.7000506@gmail.com> Message-ID: Good sleuthing! Does the attached patch fix it? (Off-topic: the code is pretty inconsistent about catching BaseException. Maybe it shouldn't be caught at all?) On Fri, Oct 18, 2013 at 2:04 PM, Richard Oudkerk wrote: > On 18/10/2013 9:19pm, Guido van Rossum wrote: > >> Maybe the dummy socket returned by wrap_socket() is not acceptable for >> select? >> > > An error > > SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify > failed (_ssl.c:553)') > > is being raised in _on_handshake(). This seems to result in the socket > being closed without being unregistered from the selector. > > select() fails before the SSLError gets reported, so it does not appear in > the traceback. > > -- > Richard > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: quickfix.diff Type: application/octet-stream Size: 1498 bytes Desc: not available URL: From shibturn at gmail.com Sat Oct 19 00:42:23 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Fri, 18 Oct 2013 23:42:23 +0100 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> <5261A26C.7000506@gmail.com> Message-ID: <5261B94F.5030006@gmail.com> On 18/10/2013 10:37pm, Guido van Rossum wrote: > Good sleuthing! Does the attached patch fix it? > > (Off-topic: the code is pretty inconsistent about catching > BaseException. Maybe it shouldn't be caught at all?) It fixes it in the sense of printing a sensible traceback;-) $ PYTHONPATH='c:/Repos/tulip' /c/Repos/cpython-33/PCbuild/python fetch3.py http://dropbox.com -v * Connecting to dropbox.com:80 using tcp * dropbox.com resolves to 108.160.165.62, 108.160.166.62, 199.47.216.179, 199.47.217.179 * New connection ('108.160.165.62', 80, False) * Connected to ('108.160.165.62', 80) > GET / HTTP/1.1 > Host: dropbox.com > < HTTP/1.1 301 Moved Permanently < Server: nginx < Date: Fri, 18 Oct 2013 22:40:13 GMT < Content-Type: text/html < Content-Length: 178 < Connection: keep-alive < Location: https://dropbox.com/ < redirect to https://dropbox.com/ * Connecting to dropbox.com:443 using ssl * dropbox.com resolves to 108.160.165.62, 108.160.166.62, 199.47.216.179, 199.47.217.179 Traceback (most recent call last): File "fetch3.py", line 211, in main() File "fetch3.py", line 206, in main body = loop.run_until_complete(fetch(sys.argv[1], '-v' in sys.argv)) File "c:\Repos\tulip\asyncio\base_events.py", line 177, in run_until_complete return future.result() File "c:\Repos\tulip\asyncio\futures.py", line 221, in result raise self._exception File "c:\Repos\tulip\asyncio\tasks.py", line 257, in _step result = coro.throw(exc) File "fetch3.py", line 192, in fetch yield from request.connect(pool) File "fetch3.py", line 80, in connect ssl=self.ssl) File "fetch3.py", line 36, in open_connection reader, writer = yield from open_connection(host, port, ssl=ssl) File "c:\Repos\tulip\asyncio\streams.py", line 41, in open_connection lambda: protocol, host, port, **kwds) File "c:\Repos\tulip\asyncio\base_events.py", line 356, in create_connection yield from waiter File "c:\Repos\tulip\asyncio\futures.py", line 318, in __iter__ yield self # This tells Task to wait for completion. File "c:\Repos\tulip\asyncio\tasks.py", line 308, in _wakeup value = future.result() File "c:\Repos\tulip\asyncio\futures.py", line 221, in result raise self._exception File "c:\Repos\tulip\asyncio\selector_events.py", line 579, in _on_handshake self._sock.do_handshake() File "C:\Repos\cpython-33\lib\ssl.py", line 520, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:553) -- Richard From shibturn at gmail.com Sat Oct 19 00:42:23 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Fri, 18 Oct 2013 23:42:23 +0100 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> <5261A26C.7000506@gmail.com> Message-ID: <5261B94F.5030006@gmail.com> On 18/10/2013 10:37pm, Guido van Rossum wrote: > Good sleuthing! Does the attached patch fix it? > > (Off-topic: the code is pretty inconsistent about catching > BaseException. Maybe it shouldn't be caught at all?) It fixes it in the sense of printing a sensible traceback;-) $ PYTHONPATH='c:/Repos/tulip' /c/Repos/cpython-33/PCbuild/python fetch3.py http://dropbox.com -v * Connecting to dropbox.com:80 using tcp * dropbox.com resolves to 108.160.165.62, 108.160.166.62, 199.47.216.179, 199.47.217.179 * New connection ('108.160.165.62', 80, False) * Connected to ('108.160.165.62', 80) > GET / HTTP/1.1 > Host: dropbox.com > < HTTP/1.1 301 Moved Permanently < Server: nginx < Date: Fri, 18 Oct 2013 22:40:13 GMT < Content-Type: text/html < Content-Length: 178 < Connection: keep-alive < Location: https://dropbox.com/ < redirect to https://dropbox.com/ * Connecting to dropbox.com:443 using ssl * dropbox.com resolves to 108.160.165.62, 108.160.166.62, 199.47.216.179, 199.47.217.179 Traceback (most recent call last): File "fetch3.py", line 211, in main() File "fetch3.py", line 206, in main body = loop.run_until_complete(fetch(sys.argv[1], '-v' in sys.argv)) File "c:\Repos\tulip\asyncio\base_events.py", line 177, in run_until_complete return future.result() File "c:\Repos\tulip\asyncio\futures.py", line 221, in result raise self._exception File "c:\Repos\tulip\asyncio\tasks.py", line 257, in _step result = coro.throw(exc) File "fetch3.py", line 192, in fetch yield from request.connect(pool) File "fetch3.py", line 80, in connect ssl=self.ssl) File "fetch3.py", line 36, in open_connection reader, writer = yield from open_connection(host, port, ssl=ssl) File "c:\Repos\tulip\asyncio\streams.py", line 41, in open_connection lambda: protocol, host, port, **kwds) File "c:\Repos\tulip\asyncio\base_events.py", line 356, in create_connection yield from waiter File "c:\Repos\tulip\asyncio\futures.py", line 318, in __iter__ yield self # This tells Task to wait for completion. File "c:\Repos\tulip\asyncio\tasks.py", line 308, in _wakeup value = future.result() File "c:\Repos\tulip\asyncio\futures.py", line 221, in result raise self._exception File "c:\Repos\tulip\asyncio\selector_events.py", line 579, in _on_handshake self._sock.do_handshake() File "C:\Repos\cpython-33\lib\ssl.py", line 520, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:553) -- Richard From ericsnowcurrently at gmail.com Sat Oct 19 00:52:11 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 18 Oct 2013 16:52:11 -0600 Subject: [Python-Dev] possibility of shaving a stat call from imports In-Reply-To: <20131018225929.10aebe3c@fsol> References: <20131018225929.10aebe3c@fsol> Message-ID: On Fri, Oct 18, 2013 at 2:59 PM, Antoine Pitrou wrote: > Is it one stat() call per successful import? Or one stat() call per > sys.path entry? It's one per finder (i.e. path entry) where a matching name is in the directory (per the finder's cache). So it's a pretty uncommon case with not much overhead when the name is not in the cache. The same thing happens earlier with directories, checking to see if the name was in the cache and *then* checking if it is a directory (_path_isdir). Again, uncommon and otherwise not all that impactful on performance. In the directory case removing the directory check might be inadvisable since it it would result in a _path_isfile call for each registered suffix before moving on (plus it would be incorrectly considered a portion of a potential namespace package). Of course, this is what happens already with directories that have no __init__. file in them, which is also an unlikely case that doesn't make much of a difference on performance normally. In the file case I agree that it's much more unlikely that there will be a directory in just the right place with a .py (etc.) suffix in the name. In either case I don't think it saves us much performance-wise, but it may be nice to make that code a simpler if it won't realistically cause anyone headaches. I'm fine with just leaving it as-is. (Also, FTR Brett's email relates to http://bugs.python.org/issue18810.) -eric p.s. I brought up the same point about dir/file checks in http://bugs.python.org/issue19216. From guido at python.org Sat Oct 19 00:56:42 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 18 Oct 2013 15:56:42 -0700 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: <5261B94F.5030006@gmail.com> References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> <5261A26C.7000506@gmail.com> <5261B94F.5030006@gmail.com> Message-ID: Thanks! That's probably fine for now -- it means the standard library doesn't know where the root certificates are. We had a huge discussion about this over on python-tulip: https://groups.google.com/forum/#!topic/python-tulip/c_lqdFjPEbE TL;DR: The stdlib openssl wrapper ought to know where each platform stores its root certificates and automatically use them, but it currently doesn't always. Users who really don't care but still want to use SSL must create an SSL context with verify_mode set to ssl.CERT_NONE (and live with the risk, obviously). This stuff passes on OS X only because there's a system openssl library that always uses the system root certificates. If anyone can help fixing the ssl.py module (or the _ssl extension) so that sslcontext.set_default_verify_paths() uses the system root certs on Windows that would be a huge help. (I have tried this on an Ubuntu box too, and there it actually works.) On Fri, Oct 18, 2013 at 3:42 PM, Richard Oudkerk wrote: > On 18/10/2013 10:37pm, Guido van Rossum wrote: > >> Good sleuthing! Does the attached patch fix it? >> >> (Off-topic: the code is pretty inconsistent about catching >> BaseException. Maybe it shouldn't be caught at all?) >> > > It fixes it in the sense of printing a sensible traceback;-) > > $ PYTHONPATH='c:/Repos/tulip' /c/Repos/cpython-33/PCbuild/**python > fetch3.py http://dropbox.com -v > * Connecting to dropbox.com:80 using tcp > * dropbox.com resolves to 108.160.165.62, 108.160.166.62, 199.47.216.179, > 199.47.217.179 > * New connection ('108.160.165.62', 80, False) > * Connected to ('108.160.165.62', 80) > > GET / HTTP/1.1 > > Host: dropbox.com > > > < HTTP/1.1 301 Moved Permanently > < Server: nginx > < Date: Fri, 18 Oct 2013 22:40:13 GMT > < Content-Type: text/html > < Content-Length: 178 > < Connection: keep-alive > < Location: https://dropbox.com/ > < > redirect to https://dropbox.com/ > * Connecting to dropbox.com:443 using ssl > * dropbox.com resolves to 108.160.165.62, 108.160.166.62, 199.47.216.179, > 199.47.217.179 > > Traceback (most recent call last): > File "fetch3.py", line 211, in > main() > File "fetch3.py", line 206, in main > > body = loop.run_until_complete(fetch(**sys.argv[1], '-v' in sys.argv)) > File "c:\Repos\tulip\asyncio\base_**events.py", line 177, in > run_until_complete > return future.result() > File "c:\Repos\tulip\asyncio\**futures.py", line 221, in result > raise self._exception > File "c:\Repos\tulip\asyncio\tasks.**py", line 257, in _step > result = coro.throw(exc) > File "fetch3.py", line 192, in fetch > yield from request.connect(pool) > File "fetch3.py", line 80, in connect > ssl=self.ssl) > File "fetch3.py", line 36, in open_connection > reader, writer = yield from open_connection(host, port, ssl=ssl) > File "c:\Repos\tulip\asyncio\**streams.py", line 41, in open_connection > lambda: protocol, host, port, **kwds) > File "c:\Repos\tulip\asyncio\base_**events.py", line 356, in > create_connection > yield from waiter > File "c:\Repos\tulip\asyncio\**futures.py", line 318, in __iter__ > yield self # This tells Task to wait for completion. > File "c:\Repos\tulip\asyncio\tasks.**py", line 308, in _wakeup > value = future.result() > File "c:\Repos\tulip\asyncio\**futures.py", line 221, in result > raise self._exception > File "c:\Repos\tulip\asyncio\**selector_events.py", line 579, in > _on_handshake > self._sock.do_handshake() > File "C:\Repos\cpython-33\lib\ssl.**py", line 520, in do_handshake > self._sslobj.do_handshake() > ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed > (_ssl.c:553) > > -- > Richard > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Oct 19 02:03:23 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 19 Oct 2013 02:03:23 +0200 Subject: [Python-Dev] PEP 454 (tracemalloc): new minimalist version In-Reply-To: References: Message-ID: 2013/10/18 Charles-Fran?ois Natali : > I'm happy to see this move forward! Thanks for your reviews. I had some time in my train travel to improve the implementation. I removed the call to pthread_atfork(): tasks have been removed, it now makes sense to keep tracemalloc enabled in the child process. Call disable() explicitly in the child process to disable tracemalloc. I modified the implementation according to your remarks, here is the updated doc: http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html I will update the PEP if you like the new doc. >> API >> === >> >> Main Functions >> -------------- >> >> ``clear_traces()`` function: >> >> Clear traces and statistics on Python memory allocations, and reset >> the ``get_traced_memory()`` counter. > > That's nitpicking, but how about just ``reset()`` (I'm probably biased > by oprofile's opcontrol --reset)? => done Well, I already hesitated to rename it to reset() because "clear_traces" sounds too specific. The function clears more than traces. >> ``get_stats()`` function: >> >> Get statistics on traced Python memory blocks as a dictionary >> ``{filename (str): {line_number (int): stats}}`` where *stats* in a >> ``(size: int, count: int)`` tuple, *filename* and *line_number* can >> be ``None``. > > It's probably obvious, but you might want to say once what *size* and > *count* represent (and the unit for *size*). => done >> ``get_tracemalloc_memory()`` function: >> >> Get the memory usage in bytes of the ``tracemalloc`` module as a >> tuple: ``(size: int, free: int)``. >> >> * *size*: total size of bytes allocated by the module, >> including *free* bytes >> * *free*: number of free bytes available to store data > > What's *free* exactly? I assume it's linked to the internal storage > area used by tracemalloc itself, but that's not clear at all. > > Also, is the tracemalloc overhead included in the above stats (I'm > mainly thinking about get_stats() and get_traced_memory()? > If yes, I find it somewhat confusing: for example, AFAICT, valgrind's > memcheck doesn't report the memory overhead, although it can be quite > large, simply because it's not interesting. My goal is to able to explain how *every* byte is allocated in Python. If you enable tracemalloc, your RSS memory will double, or something like that. You can use get_tracemalloc_memory() to add metrics to a snapshot. It helps to understand how the RSS memory evolves. Basically, get_tracemalloc_size() is the memory used to store traces. It's something internal to the C module (_tracemalloc). This memory is not traced because it *is* the traces... and so is not counted in get_traced_memory(). The issue is probably the name (or maybe also the doc): would you prefer get_python_memory() / get_traces_memory() names, instead of get_traced_memory() / get_tracemalloc_memory()? FYI Objects allocated in tracemalloc.py (real objects, not traces) are not counted in get_traced_memory() because of a filter set up by default (it was not the case in previous versions of the PEP). You can remove the filter using tracemalloc.clear_filters() to see this memory. There are two exceptions: Python objects created for the result of get_traces() and get_stats() are never traced for efficiency. It *is* possible to trace these objects, but it's really too slow. get_traces() and get_stats() may be called outside tracemalloc.py, so another filter would be needed. Well, it's easier to never trace these objects. Anyway, they are not interesting to understand where your application leaks memory. >> Trace Functions >> --------------- >> >> ``get_traceback_limit()`` function: >> >> Get the maximum number of frames stored in the traceback of a trace >> of a memory block. >> >> Use the ``set_traceback_limit()`` function to change the limit. > > I didn't see anywhere the default value for this setting: it would be > nice to write it somewhere, => done > and also explain the rationale (memory/CPU > overhead...). I already explained this partially in set_traceback_limit() doc. I added something to get_traceback_limit() to explain the default (1 is enough to get statistics). (For information, it's possible to set the limit to 0, but it's not really useful, so it's not mentionned in the doc.) >> ``get_object_address(obj)`` function: >> >> Get the address of the main memory block of the specified Python object. >> >> A Python object can be composed by multiple memory blocks, the >> function only returns the address of the main memory block. > > IOW, this should return the same as id() on CPython? If yes, it could > be an interesting note. => done I modified the doc to mention id(). id() is only the same if the object is not tracked by the garbage collector. Otherwise, there is a difference of sizeof(PyGC_Head) (12 bytes on x86). >> ``get_object_trace(obj)`` function: >> >> Get the trace of a Python object *obj* as a ``(size: int, >> traceback)`` tuple where *traceback* is a tuple of ``(filename: str, >> lineno: int)`` tuples, *filename* and *lineno* can be ``None``. > > I find the "trace" word confusing, so it might be interesting to add a > note somewhere explaining what it is ("callstack leading to the object > allocation", or whatever). => done Ok, I added a note at the beginning of the section. > [get_object_trace] > Also, this function leaves me a mixed feeling: it's called > get_object_trace(), but you also return the object size - well, a > vague estimate thereof. I wonder if the size really belongs here, > especially if the information returned isn't really accurate: it will > be for an integer, but not for e.g. a list, right? How about just > using sys.getsizeof(), which would give a more accurate result? I already modified the doc to add examples: I used set and dict as examples of types using at least 2 memory blocks. get_object_trace(obj) is a shortcut for get_trace(get_object_address(obj)). I agree that the wrong size information can be surprising. I can delete get_object_trace(), or rename the function to get_object_traceback() and modify it to only return the traceback. I prefer to keep the function (modified for get_object_traceback). tracemalloc can be combined with other tools like Melia, Heapy or objgraph to combine information. When you find an interesting object with these tools, you may be interested to know where it was allocated. >> ``get_trace(address)`` function: >> >> Get the trace of a memory block as a ``(size: int, traceback)`` >> tuple where *traceback* is a tuple of ``(filename: str, lineno: >> int)`` tuples, *filename* and *lineno* can be ``None``. >> >> Return ``None`` if the ``tracemalloc`` module did not trace the >> allocation of the memory block. >> >> See also ``get_object_trace()``, ``get_stats()`` and >> ``get_traces()`` functions. > > Do you have example use cases where you want to work with a raw addresses? An address is the unique key to identify a memory block. In Python, you don't manipulate directly memory blocks, that's why you have a get_object_address() function (link objects to traces). I added get_trace() because get_traces() is very slow. It would be stupid to call it if you only need one trace of a memory block. I'm not sure that this function is really useful. I added it to workaround the performance issue, and because I believe that someone will need it later :-) What do you suggest for this function? >> Filter >> ------ >> >> ``Filter(include: bool, pattern: str, lineno: int=None, traceback: >> bool=False)`` class: >> >> Filter to select which memory allocations are traced. Filters can be >> used to reduce the memory usage of the ``tracemalloc`` module, which >> can be read using the ``get_tracemalloc_memory()`` function. >> >> ``match(filename: str, lineno: int)`` method: >> >> Return ``True`` if the filter matchs the filename and line number, >> ``False`` otherwise. >> >> ``match_filename(filename: str)`` method: >> >> Return ``True`` if the filter matchs the filename, ``False`` otherwise. >> >> ``match_lineno(lineno: int)`` method: >> >> Return ``True`` if the filter matchs the line number, ``False`` >> otherwise. >> >> ``match_traceback(traceback)`` method: >> >> Return ``True`` if the filter matchs the *traceback*, ``False`` >> otherwise. >> >> *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples. > > Are those ``match`` methods really necessary for the end user, i.e. > are they worth being exposed as part of the public API? (Oh, I just realized that match_lineno() and may lead to bugs, I removed it.) Initially, I exposed the methods for unit tests. Later, I used them in Snapshot.apply_filters() to factorize the code (before I add 2 implementations to match a filter, one in C, another in Python). I see tracemalloc more as a library, I don't know yet how it will be used by new tools based on it. Snapshot is more an helper (convinient class) than a mandatory API to use tracemalloc. You might want to use directly filters to analyze raw datas. Users are supposed to use tracemalloc.add_filter() and Snapshot.apply_filters(). You prefer to keep them private and not document them? I don't have a strong opinion on this point. >> StatsDiff >> --------- >> >> ``StatsDiff(differences, old_stats, new_stats)`` class: >> >> Differences between two ``GroupedStats`` instances. >> >> The ``GroupedStats.compare_to()`` method creates a ``StatsDiff`` >> instance. >> >> ``sort()`` method: >> >> Sort the ``differences`` list from the biggest difference to the >> smallest difference. Sort by ``abs(size_diff)``, *size*, >> ``abs(count_diff)``, *count* and then by *key*. >> >> ``differences`` attribute: >> >> Differences between ``old_stats`` and ``new_stats`` as a list of >> ``(size_diff, size, count_diff, count, key)`` tuples. *size_diff*, >> *size*, *count_diff* and *count* are ``int``. The key type depends >> on the ``GroupedStats.group_by`` attribute of ``new_stats``: see the >> ``Snapshot.top_by()`` method. >> >> ``old_stats`` attribute: >> >> Old ``GroupedStats`` instance, can be ``None``. >> >> ``new_stats`` attribute: >> >> New ``GroupedStats`` instance. > > Why keep references to ``old_stats`` and ``new_stats``? > > Also, if you sort the difference by default (which is a sensible > choice), then the StatsDiff becomes pretty much useless, since you > would just keep its ``differences`` attribute (sorted). Well, StatsDiff is useless :-) I just removed it. I modified GroupedStats.compare_to() to sort differences by default, but I added a sort parameter to get the list unsorted. sort=False can be used to sort differences differently (sorting the list twice would be inefficient). Another option would be add sort_key and sort_reverse parameters. >> Snapshot >> -------- >> >> ``Snapshot(timestamp: datetime.datetime, traces: dict=None, stats: >> dict=None)`` class: >> >> Snapshot of traces and statistics on memory blocks allocated by Python. > > > I'm confused. > Why are get_trace(), get_object_trace(), get_stats() etc not methods > of a Snapshot object? get_stats() returns the current stats. If you call it twice, you get different results. The principe of a snapshot is to be frozen: stats, traces and metrics are read once when the snapshot was created. To get stats of a snapshot, just read its stats attribute. To get a trace, it's snapshot.traces[address]. > Is it because you don't store all the necessary information in a > snapshot, or are they just some sort of shorthands, like: > stats = get_stats() > vs > snapshot = Snapshot.create() > stats = snapshot.stats I already used other tools like Melia and Heapy, and it's convinient to get access to raw data to compute manually my own view. I don't want to force users to use the high-level API (Snapshot). Is it a problem to have two API (low-level like get_stats() and high-level like Snapshot) for similar use cases? What do you suggest? >> ``write(filename)`` method: >> >> Write the snapshot into a file. > > I assume it's in a serialized form, only readable by Snapshort.load() ? Yes. > BTW, it's a nitpick and debatable, but write()/read() or load()/dump() > would be more consistent (see e.g. pickle's load/dump). => Done, I renamed Snapshot.write() to Snapshot.dump(). By the way, load() and dump() are limited to filenames (string). Should they accept file-like object? isinstance(filename, str) may be used to check if the parameter is a filename or a open file object. >> Metric >> ------ >> >> ``Metric(name: str, value: int, format: str)`` class: >> >> Value of a metric when a snapshot is created. > > Alright, what's a metric again ;-) ? > > I don't know if it's customary, but having short examples would IMO be nice. => done, I improved the doc Victor From ncoghlan at gmail.com Sat Oct 19 02:21:27 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Oct 2013 10:21:27 +1000 Subject: [Python-Dev] possibility of shaving a stat call from imports In-Reply-To: References: Message-ID: On 19 Oct 2013 02:56, "Brett Cannon" wrote: > > importlib.machinery.FileFinder does a stat call to check if a path is a file if the package check failed. Now I'm willing to bet that the check is rather redundant as the file extension should be a dead give-away that something in a directory is a file and not some non-file type. The import would still fail even if this is the case in the loader when attempting to read from the file, but it happens a little later and it means finders would be more permissive in claiming they found a loader. > > Does anyone see a good reason not to take the more optimistic route in the finder? As I said, the only thing I see breaking user code is if they have a directory or something named spam.py and so the finder claims it found a module when in fact it didn't and thus stopping the search for the module from continuing. I suspect anything that makes weird import edge case failures harder to debug than they already are won't be worth it. My opinion would likely differ if the speed up on offer was dramatic, but it sounds like that isn't the case. (Also, customised Python builds may have additional extensions that are at higher risk of being used on directory names) Cheers, Nick. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Oct 19 02:49:49 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Oct 2013 10:49:49 +1000 Subject: [Python-Dev] PEP 454 (tracemalloc): new minimalist version In-Reply-To: References: Message-ID: On 19 Oct 2013 03:57, "Charles-Fran?ois Natali" wrote: > > Hi, > > I'm happy to see this move forward! Speaking of which... Charles-Fran?ois, would you be willing to act as BDFL-Delegate for this PEP? This will be a very useful new analysis tool, and between yourself and Victor it looks like you'll be able to come up with a solid API. I just suggested that approach to Guido and he also liked the idea :) Cheers, Nick. > > > API > > === > > > > Main Functions > > -------------- > > > > ``clear_traces()`` function: > > > > Clear traces and statistics on Python memory allocations, and reset > > the ``get_traced_memory()`` counter. > > That's nitpicking, but how about just ``reset()`` (I'm probably biased > by oprofile's opcontrol --reset)? > > > ``get_stats()`` function: > > > > Get statistics on traced Python memory blocks as a dictionary > > ``{filename (str): {line_number (int): stats}}`` where *stats* in a > > ``(size: int, count: int)`` tuple, *filename* and *line_number* can > > be ``None``. > > It's probably obvious, but you might want to say once what *size* and > *count* represent (and the unit for *size*). > > > ``get_tracemalloc_memory()`` function: > > > > Get the memory usage in bytes of the ``tracemalloc`` module as a > > tuple: ``(size: int, free: int)``. > > > > * *size*: total size of bytes allocated by the module, > > including *free* bytes > > * *free*: number of free bytes available to store data > > What's *free* exactly? I assume it's linked to the internal storage > area used by tracemalloc itself, but that's not clear at all. > > Also, is the tracemalloc overhead included in the above stats (I'm > mainly thinking about get_stats() and get_traced_memory()? > If yes, I find it somewhat confusing: for example, AFAICT, valgrind's > memcheck doesn't report the memory overhead, although it can be quite > large, simply because it's not interesting. > > > Trace Functions > > --------------- > > > > ``get_traceback_limit()`` function: > > > > Get the maximum number of frames stored in the traceback of a trace > > of a memory block. > > > > Use the ``set_traceback_limit()`` function to change the limit. > > I didn't see anywhere the default value for this setting: it would be > nice to write it somewhere, and also explain the rationale (memory/CPU > overhead...). > > > ``get_object_address(obj)`` function: > > > > Get the address of the main memory block of the specified Python object. > > > > A Python object can be composed by multiple memory blocks, the > > function only returns the address of the main memory block. > > IOW, this should return the same as id() on CPython? If yes, it could > be an interesting note. > > > ``get_object_trace(obj)`` function: > > > > Get the trace of a Python object *obj* as a ``(size: int, > > traceback)`` tuple where *traceback* is a tuple of ``(filename: str, > > lineno: int)`` tuples, *filename* and *lineno* can be ``None``. > > I find the "trace" word confusing, so it might be interesting to add a > note somewhere explaining what it is ("callstack leading to the object > allocation", or whatever). > > Also, this function leaves me a mixed feeling: it's called > get_object_trace(), but you also return the object size - well, a > vague estimate thereof. I wonder if the size really belongs here, > especially if the information returned isn't really accurate: it will > be for an integer, but not for e.g. a list, right? How about just > using sys.getsizeof(), which would give a more accurate result? > > > ``get_trace(address)`` function: > > > > Get the trace of a memory block as a ``(size: int, traceback)`` > > tuple where *traceback* is a tuple of ``(filename: str, lineno: > > int)`` tuples, *filename* and *lineno* can be ``None``. > > > > Return ``None`` if the ``tracemalloc`` module did not trace the > > allocation of the memory block. > > > > See also ``get_object_trace()``, ``get_stats()`` and > > ``get_traces()`` functions. > > Do you have example use cases where you want to work with a raw addresses? > > > Filter > > ------ > > > > ``Filter(include: bool, pattern: str, lineno: int=None, traceback: > > bool=False)`` class: > > > > Filter to select which memory allocations are traced. Filters can be > > used to reduce the memory usage of the ``tracemalloc`` module, which > > can be read using the ``get_tracemalloc_memory()`` function. > > > > ``match(filename: str, lineno: int)`` method: > > > > Return ``True`` if the filter matchs the filename and line number, > > ``False`` otherwise. > > > > ``match_filename(filename: str)`` method: > > > > Return ``True`` if the filter matchs the filename, ``False`` otherwise. > > > > ``match_lineno(lineno: int)`` method: > > > > Return ``True`` if the filter matchs the line number, ``False`` > > otherwise. > > > > ``match_traceback(traceback)`` method: > > > > Return ``True`` if the filter matchs the *traceback*, ``False`` > > otherwise. > > > > *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples. > > Are those ``match`` methods really necessary for the end user, i.e. > are they worth being exposed as part of the public API? > > > StatsDiff > > --------- > > > > ``StatsDiff(differences, old_stats, new_stats)`` class: > > > > Differences between two ``GroupedStats`` instances. > > > > The ``GroupedStats.compare_to()`` method creates a ``StatsDiff`` > > instance. > > > > ``sort()`` method: > > > > Sort the ``differences`` list from the biggest difference to the > > smallest difference. Sort by ``abs(size_diff)``, *size*, > > ``abs(count_diff)``, *count* and then by *key*. > > > > ``differences`` attribute: > > > > Differences between ``old_stats`` and ``new_stats`` as a list of > > ``(size_diff, size, count_diff, count, key)`` tuples. *size_diff*, > > *size*, *count_diff* and *count* are ``int``. The key type depends > > on the ``GroupedStats.group_by`` attribute of ``new_stats``: see the > > ``Snapshot.top_by()`` method. > > > > ``old_stats`` attribute: > > > > Old ``GroupedStats`` instance, can be ``None``. > > > > ``new_stats`` attribute: > > > > New ``GroupedStats`` instance. > > Why keep references to ``old_stats`` and ``new_stats``? > datetime.timedelta doesn't keep references to the date objects it was > computed from. > > Also, if you sort the difference by default (which is a sensible > choice), then the StatsDiff becomes pretty much useless, since you > would just keep its ``differences`` attribute (sorted). > > > Snapshot > > -------- > > > > ``Snapshot(timestamp: datetime.datetime, traces: dict=None, stats: > > dict=None)`` class: > > > > Snapshot of traces and statistics on memory blocks allocated by Python. > > > I'm confused. > Why are get_trace(), get_object_trace(), get_stats() etc not methods > of a Snapshot object? > Is it because you don't store all the necessary information in a > snapshot, or are they just some sort of shorthands, like: > stats = get_stats() > vs > snapshot = Snapshot.create() > stats = snapshot.stats > > > ``write(filename)`` method: > > > > Write the snapshot into a file. > > I assume it's in a serialized form, only readable by Snapshort.load() ? > BTW, it's a nitpick and debatable, but write()/read() or load()/dump() > would be more consistent (see e.g. pickle's load/dump). > > > Metric > > ------ > > > > ``Metric(name: str, value: int, format: str)`` class: > > > > Value of a metric when a snapshot is created. > > Alright, what's a metric again ;-) ? > > I don't know if it's customary, but having short examples would IMO be nice. > > cf > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Oct 19 02:55:41 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Oct 2013 10:55:41 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Issue #18416: Fix various os calls in importlib.machinery.FileFinder In-Reply-To: <3d1X6M5RMjz7LjP@mail.python.org> References: <3d1X6M5RMjz7LjP@mail.python.org> Message-ID: On 19 Oct 2013 02:01, "brett.cannon" wrote: > > http://hg.python.org/cpython/rev/33844153cd02 > changeset: 86438:33844153cd02 > user: Brett Cannon > date: Fri Oct 18 12:01:06 2013 -0400 > summary: > Issue #18416: Fix various os calls in importlib.machinery.FileFinder > now that self.path is no longer forced to '.'. Hmm, could this break subclasses in a similar way? It may be safer to keep the empty string -> period conversion, even though PathFinder itself doesn't rely on it any more. Cheers, Nick. > > files: > Lib/importlib/_bootstrap.py | 4 +- > Python/importlib.h | 1554 +++++++++++----------- > 2 files changed, 780 insertions(+), 778 deletions(-) > > > diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py > --- a/Lib/importlib/_bootstrap.py > +++ b/Lib/importlib/_bootstrap.py > @@ -1390,7 +1390,7 @@ > is_namespace = False > tail_module = fullname.rpartition('.')[2] > try: > - mtime = _os.stat(self.path).st_mtime > + mtime = _os.stat(self.path or _os.getcwd()).st_mtime > except OSError: > mtime = -1 > if mtime != self._path_mtime: > @@ -1432,7 +1432,7 @@ > """Fill the cache of potential modules and packages for this directory.""" > path = self.path > try: > - contents = _os.listdir(path) > + contents = _os.listdir(path or _os.getcwd()) > except (FileNotFoundError, PermissionError, NotADirectoryError): > # Directory has either been removed, turned into a file, or made > # unreadable. > diff --git a/Python/importlib.h b/Python/importlib.h > --- a/Python/importlib.h > +++ b/Python/importlib.h > [stripped] > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Oct 19 03:06:19 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Oct 2013 11:06:19 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Issue #18810: Be optimistic with stat calls when seeing if a directory In-Reply-To: <3d1YyH4L2tz7LjP@mail.python.org> References: <3d1YyH4L2tz7LjP@mail.python.org> Message-ID: On 19 Oct 2013 03:24, "brett.cannon" wrote: > > http://hg.python.org/cpython/rev/11f2f4af1979 > changeset: 86444:11f2f4af1979 > user: Brett Cannon > date: Fri Oct 18 13:24:13 2013 -0400 > summary: > Issue #18810: Be optimistic with stat calls when seeing if a directory > exists when checking for a package. > > Before there was an isdir check and then various isfile checks for > possible __init__ files when looking for a package. > This change drops the isdir check by leaning > on the assumption that a directory will not contain something named > after the module being imported which is not a directory. If the module > is a package then it saves a stat call. If there is nothing in the > directory with the potential package name it also saves a stat call. > Only if there is something in the directory named the same thing as > the potential package will the number of stat calls increase > (due to more wasteful __init__ checks). I don't follow this logic. There's now not even an existence check for the base name, so it reads to me like we will look for all the possible init file extensions even if there's *no* directory with an appropriate name. What am I missing? Cheers, Nick. > > Semantically there is no change as the isdir check moved > down so that namespace packages continue to have no chance of > accidentally collecting non-existent directories. > > files: > Lib/importlib/_bootstrap.py | 19 +- > Python/importlib.h | 1537 +++++++++++----------- > 2 files changed, 777 insertions(+), 779 deletions(-) > > > diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py > --- a/Lib/importlib/_bootstrap.py > +++ b/Lib/importlib/_bootstrap.py > @@ -1406,16 +1406,15 @@ > # Check if the module is the name of a directory (and thus a package). > if cache_module in cache: > base_path = _path_join(self.path, tail_module) > - if _path_isdir(base_path): > - for suffix, loader in self._loaders: > - init_filename = '__init__' + suffix > - full_path = _path_join(base_path, init_filename) > - if _path_isfile(full_path): > - return (loader(fullname, full_path), [base_path]) > - else: > - # A namespace package, return the path if we don't also > - # find a module in the next section. > - is_namespace = True > + for suffix, loader in self._loaders: > + init_filename = '__init__' + suffix > + full_path = _path_join(base_path, init_filename) > + if _path_isfile(full_path): > + return (loader(fullname, full_path), [base_path]) > + else: > + # If a namespace package, return the path if we don't > + # find a module in the next section. > + is_namespace = _path_isdir(base_path) > # Check for a file w/ a proper suffix exists. > for suffix, loader in self._loaders: > full_path = _path_join(self.path, tail_module + suffix) > diff --git a/Python/importlib.h b/Python/importlib.h > --- a/Python/importlib.h > +++ b/Python/importlib.h > [stripped] > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Thu Oct 17 19:25:11 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 17 Oct 2013 10:25:11 -0700 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). In-Reply-To: References: <96A679C5-0BAA-402E-A34F-412F02366C45@gmail.com> <20131013042443.GA90160@cskk.homeip.net> <87iox1he3n.fsf@uwakimon.sk.tsukuba.ac.jp> <525B2AAD.80100@g.nevcal.com> <525CB2AA.3010409@g.nevcal.com> <20131015142719.360e30a8@pitrou.net> <20131015155232.8789F25098D@webabinitio.net> <069D29BD-DDAC-4049-B67F-E4DA12240ED2@gmail.com> <20131016112407.19f8836e@anarchist> Message-ID: <52601D77.9010803@g.nevcal.com> First, thanks for the education. What you wrote is extremely edifying about more than just context managers, and I really appreciate the visionary understanding you reported from BrisPy and further elucidated on, regarding the educational pattern of using things before you learn how they work... that applies strongly in arenas other than programming as well: - you learn how to walk before you understand the musculoskeletal physics - you learn how to turn on/off the lights before you understand how electricity works - you learn how to drive before you learn how/why a vehicle works - you learn how to speak before you understand how grammar works - you learn how to locate the constellations before you understand interplanetary gravitational forces - many, many, many, many more things And of course, many people never reach the understanding of how or why for many things they commonly use, do, or observe. That's why some people make things happen, some people watch what happens, and some people wonder "What happened?" What it doesn't do, though is address the dubious part of the whole construct, which is composition. On 10/17/2013 8:26 AM, Nick Coghlan wrote: > And even a two line version: > > with suppress(FileNotFoundError): os.remove("somefile.tmp") > with suppress(FileNotFoundError): os.remove("someotherfile.tmp") The above example, especially if extended beyond two files, begs to used in a loop, like your 5 line version: for name in ("somefile.tmp", "someotherfile.tmp"): with suppress(FileNotFoundError): os.remove(name) which would be fine, of course. But to some with less education about the how and why, it is not clear why it couldn't be written like: with suppress(FileNotFoundError): for name in ("somefile.tmp", "someotherfile.tmp"): os.remove(name) yet to the cognoscenti, it is obvious there are seriously different semantics. In my own code, I have a safe_delete function to bundle the exception handling and the os.remove, and when factored that way, the temptation to nest the loop inside the suppress is gone. With suppress available, though, and if used, the temptation to factor it, either correctly or incorrectly, appears. How many cut-n-paste programmers will get it right and how many will get it wrong, is the serious question here, I think, and while suppress is a slightly better term than ignore, it still hides the implications to the control flow when an exception is actually raised within the block. I'm still dubious that the benefits of this simpler construct, while an interesting composition of powerful underlying constructs, has sufficient benefit to outweigh the na?ve user's potential for misusing it (exacerbated by a name that doesn't imply control flow), or even the extra cost in performance per the microbenchmark someone published. Your more complex examples for future versions may have greater merit because they provide a significantly greater reduction in complexity to offset the significantly greater learning curve required to use and understand them. But even those look like an expensive form of goto (of course, goto is considered harmful, and I generally agree with the reasons why, but have coded them in situations where they are more useful than harmful in languages which support them). I imagine that everyone on python-dev is aware that most of the control flow constructs in structured programming (which is a subset of OO) are to control the context of the CPUs "instruction pointer" without the use of "goto". The real problem with "goto" is not that the instruction pointer is changed non-sequentially, but that arbitrary changes can easily violate poorly documented preconditions of the target location. Hence, structured programming is really an attempt to avoid writing documentation, a laudable goal as the documentation is seldom sufficient at that level of detail... or if sufficient, is repetitive and overwhelming to create, maintain, and comprehend. It achieves that by making control flow constructs that are "higher level" than goto, that have meanings that can be understood and explained in educational texts, which then are implicit documentation for those control flow aspects of a particular program. OO builds on structured programming to make neat packages of state and control flow, to isolate state into understandable chunks so that larger programs can be comprehended, as the BrisPy presenter enlightened us, without understanding all the details of how each object and function within it works. Programmers raised on OO and GUI toolkits are building more and more systems out of more complex parts, which increases productivity, and that is good, although when they fail to fully understand the parts, some "interesting" performance characteristics can result. ignore/suppress seems to me to be a sledge hammer solution for driving a tack. The tack may be driven successfully, but the potential for damage to the surroundings (by misunderstanding the control flow implications) is sufficient to make me dubious regarding its overall value. Adequate documentation may help (if it is both provided and read), but the best constructs are those that are self-documenting, or well documented in existing "programming 101" books. I haven't seen this construct in other languages, nor has such a comparison been made in this thread, so I consider the potential for misuse large. My conclusion: suppress considered harmful, hidden goto within :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Oct 19 08:38:39 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Oct 2013 16:38:39 +1000 Subject: [Python-Dev] Context management patterns Message-ID: On 18 October 2013 03:25, Glenn Linderman wrote: > First, thanks for the education. What you wrote is extremely edifying about > more than just context managers, and I really appreciate the visionary > understanding you reported from BrisPy and further elucidated on, regarding > the educational pattern of using things before you learn how they work... > that applies strongly in arenas other than programming as well: > > - you learn how to walk before you understand the musculoskeletal physics > - you learn how to turn on/off the lights before you understand how > electricity works > - you learn how to drive before you learn how/why a vehicle works > - you learn how to speak before you understand how grammar works > - you learn how to locate the constellations before you understand > interplanetary gravitational forces > - many, many, many, many more things > > And of course, many people never reach the understanding of how or why for > many things they commonly use, do, or observe. That's why some people make > things happen, some people watch what happens, and some people wonder "What > happened?" > > What it doesn't do, though is address the dubious part of the whole > construct, which is composition. However, it's important to be clear as to whether the composition problems are specifically with the context manager form *or* if they also apply to the underlying exception handling pattern. Barry raised a good point the other day about how context managers encapsulate exception handling patterns, and that the level shift happening with contextlib.suppress vs previous standard library usage is that it actually takes advantage of the fact that the "with" statement is a control flow construct that supports suppressing raised exceptions. By contrast, the previously extracted patterns encountered in the standard library are all about correct resource handling and use either the try-finally or the try-except-raise patterns that don't impact control flow. Here are the clearest patterns I've personally noticed in the time since the with statement was added: * deterministic resource management (not a control flow pattern) try: finally: x.close() # Or otherwise clean up - aside from calling close() methods, generally resource specific - closing files, sockets, etc - dropping memoryview buffer references - contextlib.closing * deterministic state management (not a control flow pattern) try: finally: - specific to the state being managed - lock acquisition/release pairs - decimal context management - monkey patching (including standard stream redirection) * transaction management (not a control flow pattern) try: except: raise else: - database sessions - conditional resource cleanup (i.e. only in failure case) - specific to the kind of transaction being managed * logging unhandled exceptions (not a control flow pattern) try: except Exception as exc: # Exception is a good default, but needs to be configurable log.exception(exc) raise # This is important, since *suppressing* the exception is a separate decision - specific to a logging framework - in practice, usually just written out and combined with exception suppression in long running server processes - not commonly seen in scripts, since those rely on the unhandled exception display in the interpreter - stdlib logging module could possibly offer with a "logging.log_unhandled" context manager, permitting things like: with suppress(Exception) with log_unhandled(): * suppressing expected exceptions that aren't errors (control flow pattern!) try: except : pass - general pattern, now explicitly named as contextlib.suppress - indicates exceptions that aren't really "exceptions" in that specific context, but just a different acceptable result - once converted to the class based implementation, will be stateless and reusable - consider if, instead of accepting things like "ignore_errors" flags and/or error callbacks, iteration constructs like shutil.rmtree instead accepted a context manager to wrap around the innermost calls where exceptions are anticipated. There are now two more control flow patterns that I'm considering adding to contextlib. However, I'd experiment with them in contextlib2 before adding them to the standard library's contextlib module: * delayed failure handling try: i = data.index(target) except IndexError: i = None # later if i is None: # Handle the "not found" case else: # Do something with the value - this could be rewritten as (I believe credit is due to RDM for the name): with catch(IndexError) as missing: i = data.index(target) # later if missing: # Handle the "not found" case # The caught exception would be available as missing.exception else: # Do something with the value - unittest.TestCase.assertRaises is an existing construct along these lines - can use a similarly stateless and reusable class-based implementation to that which will be used for suppress - such a "catch" decorator could also take care of saving the exception and calling traceback.clear_frames() on it for easy introspection without inadvertently keeping vast swathes of local objects alive in CPython - unfortunately, warnings.catch_warnings is misnamed - it's really a state management context manager for the warning filter state, with an option to record warnings via monkeypatching. Alas, I understand these concepts far better now than I did back when we extracted that API from the 'check_warnings' helper in the test suite, so I didn't realise the name was wrong until long after it had been published. If we add contextlib.catch in 3.5, it would probably be worth going through the deprecation dance needed to rename catch_warnings to something more sensible like "warning_context". * constrained jumps # Search loop for item in data: if is_desired_result(item): result = item break else: # Handle the "not found" case # Do something with "result" - using a suitable context manager, this can be rewritten as: # Search loop with exit_label() as found: for item in data: if is_desired_result(item): found.exit(item) # Later if found: # Do something with "found.value" else: # Handle the "not found" case - this is the exit_label() idea I posted earlier, but without the ability to specify the exception type (since I realised that's a separate pattern, better handled as the distinct "catch" construct) - rather than replacing exception handling constructs, it replaces break/else search loops with something that's hopefully easier to understand - it's also a generalisation of the SystemExit/GeneratorExit pattern, so the exception types it uses for internal flow control would inherit directly from BaseException - this is the pattern on the list that gets the closest to "goto" like behaviour, since it permits arbitrary "bail out now" behaviour (by passing the exit label to other operations), but the fact every label uses a custom exception type derived directly from BaseException would mean it still ends up being quite heavily constrained (since the *only* thing it would be able to catch is the exception thrown by calling the exit() method on the label). - this pattern would be stateful and explicitly *not* reusable (acting as a further constraint on abuse) > On 10/17/2013 8:26 AM, Nick Coghlan wrote: > > And even a two line version: > > with suppress(FileNotFoundError): os.remove("somefile.tmp") > with suppress(FileNotFoundError): os.remove("someotherfile.tmp") > > > The above example, especially if extended beyond two files, begs to used in > a loop, like your 5 line version: > > > for name in ("somefile.tmp", "someotherfile.tmp"): > with suppress(FileNotFoundError): > os.remove(name) > > which would be fine, of course. > > But to some with less education about the how and why, it is not clear why > it couldn't be written like: > > with suppress(FileNotFoundError): > > for name in ("somefile.tmp", "someotherfile.tmp"): > os.remove(name) > > yet to the cognoscenti, it is obvious there are seriously different > semantics. However, that's a confusion about exception handling in general, not about the suppress context manager in particular. The same potential for conceptual confusion exists between: for name in ("somefile.tmp", "someotherfile.tmp"): try: os.remove(name) except FileNotFoundError: pass and: try: for name in ("somefile.tmp", "someotherfile.tmp"): os.remove(name) except FileNotFoundError: pass At the syntactic level, when composing compound statements, the order of nesting *always* matters. The with/for and for/with constructs are different, just as if/for and for/if are different. If a student makes it through an introductory Python course without learning that much, I'd have grave doubts about that course :) > In my own code, I have a safe_delete function to bundle the exception > handling and the os.remove, and when factored that way, the temptation to > nest the loop inside the suppress is gone. With suppress available, though, > and if used, the temptation to factor it, either correctly or incorrectly, > appears. How many cut-n-paste programmers will get it right and how many > will get it wrong, is the serious question here, I think, and while suppress > is a slightly better term than ignore, it still hides the implications to > the control flow when an exception is actually raised within the block. A *lot* of written out exception handling is better abstracted away into a helper function. However, the body of the try block can't always be factored out without creating swiss-army functions with more knobs and dials than subprocess.Popen, and those are the cases where using a context manager instead really shines. > I'm still dubious that the benefits of this simpler construct, while an > interesting composition of powerful underlying constructs, has sufficient > benefit to outweigh the na?ve user's potential for misusing it (exacerbated > by a name that doesn't imply control flow), or even the extra cost in > performance per the microbenchmark someone published. In the case of contextlib.suppress, my interest is mostly in giving this particular pattern a name, although it also allows for finer granularity in deciding exactly *which* exceptions to ignore. For example, most typical "safe_remove" functions ignore *all* OSErrors, while the contextlib.suppress example in the docs is deliberately constrained to only ignore FileNotFoundError (since "I don't care if it's already gone" is certainly a reasonable thing to say, while "I don't care if I don't have permission to remove it" is more dubious). This inherent ability to suppress exceptions means that "with" statements *are* a control flow construct, and always have been since we added them in PEP 343. However, this also gets back to Barry's point about there being a category shift here: contextlib.suppress is the first *standard library* context manager to actually make use of the fact that with statements are a control flow construct just as much as try/except/else/finally statements are (just a more constrained one at the point of use, which makes them easier to understand). > Your more complex examples for future versions may have greater merit > because they provide a significantly greater reduction in complexity to > offset the significantly greater learning curve required to use and > understand them. But even those look like an expensive form of goto (of > course, goto is considered harmful, and I generally agree with the reasons > why, but have coded them in situations where they are more useful than > harmful in languages which support them). > > I imagine that everyone on python-dev is aware that most of the control flow > constructs in structured programming (which is a subset of OO) are to > control the context of the CPUs "instruction pointer" without the use of > "goto". > > The real problem with "goto" is not that the instruction pointer is changed > non-sequentially, but that arbitrary changes can easily violate poorly > documented preconditions of the target location. Hence, structured > programming is really an attempt to avoid writing documentation, a laudable > goal as the documentation is seldom sufficient at that level of detail... or > if sufficient, is repetitive and overwhelming to create, maintain, and > comprehend. It achieves that by making control flow constructs that are > "higher level" than goto, that have meanings that can be understood and > explained in educational texts, which then are implicit documentation for > those control flow aspects of a particular program. OO builds on structured > programming to make neat packages of state and control flow, to isolate > state into understandable chunks so that larger programs can be > comprehended, as the BrisPy presenter enlightened us, without understanding > all the details of how each object and function within it works. > > Programmers raised on OO and GUI toolkits are building more and more systems > out of more complex parts, which increases productivity, and that is good, > although when they fail to fully understand the parts, some "interesting" > performance characteristics can result. > > ignore/suppress seems to me to be a sledge hammer solution for driving a > tack. The tack may be driven successfully, but the potential for damage to > the surroundings (by misunderstanding the control flow implications) is > sufficient to make me dubious regarding its overall value. Adequate > documentation may help (if it is both provided and read), but the best > constructs are those that are self-documenting, or well documented in > existing "programming 101" books. I haven't seen this construct in other > languages, nor has such a comparison been made in this thread, so I consider > the potential for misuse large. > > My conclusion: suppress considered harmful, hidden goto within :) I believe your underlying concerns are actually with the non-local flow control possibilities that are inherent in a language that offers both exceptions and the ability to suppress them. Since I'm firmly convinced that offering *more* structured exception handling is a vastly better solution to that problem than alternatives like Go's retreat to C-style return codes, I'll be continuing down the path of trying to extract and formalise particular patterns that constitute reasonable and common patterns for try/except/else/finally (or break/else loops!) in Python. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From larry at hastings.org Sat Oct 19 09:13:51 2013 From: larry at hastings.org (Larry Hastings) Date: Sat, 19 Oct 2013 00:13:51 -0700 Subject: [Python-Dev] The Clinic Has Landed Message-ID: <5262312F.80604@hastings.org> "Hello, I'd like to have an argument, please." After more than a year of development, Argument Clinic is now checked in to CPython trunk. Thanks, everyone, for your feedback and encouragement! //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Oct 19 10:00:15 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Oct 2013 18:00:15 +1000 Subject: [Python-Dev] The Clinic Has Landed In-Reply-To: <5262312F.80604@hastings.org> References: <5262312F.80604@hastings.org> Message-ID: On 19 October 2013 17:13, Larry Hastings wrote: > "Hello, I'd like to have an argument, please." > > After more than a year of development, Argument Clinic is now checked in to > CPython trunk. Thanks, everyone, for your feedback and encouragement! Huzzah! Thanks for your efforts pushing through such a huge step towards finally being able to fix this current limitation: >>> inspect.signature(open) Traceback (most recent call last): File "", line 1, in File "/home/ncoghlan/devel/py3k/Lib/inspect.py", line 1533, in signature raise ValueError(msg) ValueError: no signature found for builtin function Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From v+python at g.nevcal.com Sat Oct 19 12:47:45 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 19 Oct 2013 03:47:45 -0700 Subject: [Python-Dev] Context management patterns In-Reply-To: References: Message-ID: <52626351.8030705@g.nevcal.com> On 10/18/2013 11:38 PM, Nick Coghlan wrote: > However, that's a confusion about exception handling in general, not > about the suppress context manager in particular. The same potential > for conceptual confusion exists between: > > for name in ("somefile.tmp", "someotherfile.tmp"): > try: > os.remove(name) > except FileNotFoundError: > pass > > and: > > try: > for name in ("somefile.tmp", "someotherfile.tmp"): > os.remove(name) > except FileNotFoundError: > pass > > At the syntactic level, when composing compound statements, the order > of nesting*always* matters. The with/for and for/with constructs are > different, just as if/for and for/if are different. If a student makes > it through an introductory Python course without learning that much, > I'd have grave doubts about that course Students that actually take courses might be OK... they might even read some of the documentation. Indeed the problem can be produced with try/except syntax or with with/suppress. That has been obvious from the beginning of the discussion, and stated repeatedly. I suppose it is the "context management" connotation that with currently has, that makes me think that with/suppress is reasonably likely to get misused... in context management uses, one puts long blocks of code under with (not always, but often). In this particular case, that is a problem, yet not all the control flow examples would be limited to one line of code in the with block. So this is still a very special case, for a very small benefit, with an unusual limitation. I have a new color of paint: with pass_if( FileNotFoundError ): os_remove("somefile.tmp") One thing about bikeshedding... after trying lots of colors, you might actually find one everyone likes. But if they aren't put out there to see, no one knows if they like it or not. And if there are a significant number of people that don't like the color, it either means the sun is in their eyes, or the color isn't the best, and that is sometimes hard to discern. While I already realized that the with construct is really control flow, even the names that indicated that that have been suggested so far, even by me, didn't really capture the type of control flow. It hit me, reading your last message, that the key part is "pass"... it is an action (control flow), and it is the action that would be taken if written with try/except syntax. It isn't the exception, or the catching, that are the key.... it is the passing... passing over the with block because the exception happened. That even implies you don't want to pass much, certainly not a whole loop full of stuff, or a long string of statements. Sadly, all the documentation that references "suppressing" an exception, is misleading. Misleading enough that you convinced yourself to rename from ignore to suppress after reading it, even though the two are nearly synonymous. The exceptions are not ignored, and the exceptions are not suppressed. They are handled by passing (which avoids the need for surrounding code to handle them, which is sort of like being ignored or suppressed, from the outside looking in... but pass describes the behavior for both inside and outside viewpoints!). > I believe your underlying concerns are actually with the non-local > flow control possibilities that are inherent in a language that offers > both exceptions and the ability to suppress them. Are you referring to a different type of "suppress" than the one you just implemented? Where the language allows you to suppress an exception, and execution proceeds with the next statement? Where it would be more of a declarative style of suppression rather than an active "catch and pass". If so, it is more proof that "suppress" is the wrong term for your new feature. Thanks again for sharing the big picture: you have before, but I see you've evolved it slightly. Delayed error handling I've had occasion to do, but not yet in Python. The constrained jump pattern is interesting, but the example is not complex enough to justify the syntax: the with solution is actually longer... if you initialized result they'd be the same length. The problem with the for/else construct is that the "not found" case falls through to the "found" case, which can be awkward, but that awkwardness isn't really demonstrated in the example. "exit_label" is an awkward name, too, since there are no labels, and no exit(). (yes, I know you meant exit the loop; naming is hard.) I haven't yet wrapped my mind around all the possibilities of with creating an "empty" object, which gets populated by the various execution paths taken, but that combination does seem quite a bit more powerful than the simple with statement alone, which only sits at the front of a block. try/except/finally/else has lots of keywords and lots of places to put code :) Which doesn't always make the code clear, so abstractions using with may indeed be helpful... but care will have to be taken that the requirement for the added object and its manipulation doesn't get more cumbersome than straightforward coding of the problem without the abstraction. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Oct 19 13:21:24 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Oct 2013 21:21:24 +1000 Subject: [Python-Dev] Context management patterns In-Reply-To: <52626351.8030705@g.nevcal.com> References: <52626351.8030705@g.nevcal.com> Message-ID: On 19 October 2013 20:47, Glenn Linderman wrote: > Thanks again for sharing the big picture: you have before, but I see you've > evolved it slightly. Delayed error handling I've had occasion to do, but > not yet in Python. The constrained jump pattern is interesting, but the > example is not complex enough to justify the syntax: the with solution is > actually longer... if you initialized result they'd be the same length. The > problem with the for/else construct is that the "not found" case falls > through to the "found" case, which can be awkward, but that awkwardness > isn't really demonstrated in the example. "exit_label" is an awkward name, > too, since there are no labels, and no exit(). (yes, I know you meant exit > the loop; naming is hard.) You exit with statements - that's why the final method is called __exit__. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Oct 19 13:41:17 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Oct 2013 21:41:17 +1000 Subject: [Python-Dev] An interesting exception handling quirk Message-ID: I've been looking at Terry's suggestion of using a class based implementation for contextlib.suppress, and rediscovered why I suggested Raymond switch the original implementation to using @contextmanager: recreating the *exact* exception subclass check from Python is actually difficult these days. The exception handling only looks for concrete subclasses, ignoring the dynamic subclass hook: >>> import abc >>> class ExceptionABC(abc.ABC, Exception): ... pass ... >>> ExceptionABC.register(ZeroDivisionError) >>> issubclass(ZeroDivisionError, ExceptionABC) True >>> try: ... 1/0 ... except ExceptionABC: ... print("It's a subclass of ExceptionABC!") ... except ZeroDivisionError: ... print("But not according to the exception handling!") ... But not according to the exception handling! I don't actually have a problem with the current behaviour (since raise and except both enforce a requirement for the types involved to be concrete subclasses of BaseException), but as far as I'm aware we don't explicitly state anywhere that if exception handling and an issubclass check give different answers for whether or not exception A is a subclass of exception B, then the fault for any resulting differences in behavious lies with the way A and/or B are defined, not with the code using issubclass to check for exception inheritance. Having noticed the discrepancy, I feel like it should be explicitly recorded somewhere in the language reference, I'm just not sure where. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Oct 19 14:12:59 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Oct 2013 22:12:59 +1000 Subject: [Python-Dev] Context management patterns In-Reply-To: <52626351.8030705@g.nevcal.com> References: <52626351.8030705@g.nevcal.com> Message-ID: On 19 October 2013 20:47, Glenn Linderman wrote: > Sadly, all the documentation that references "suppressing" an exception, is > misleading. Misleading enough that you convinced yourself to rename from > ignore to suppress after reading it, even though the two are nearly > synonymous. You misunderstand the two components that formed my rationale for renaming it. The first part of the rationale is that re-reading all of the current docs reminded me that this pattern *already* had a consistent name in the language and library reference (i.e. suppress), and this name was even mentioned all the way back in PEP 343 (although it wasn't the primary term there - that honour went to "swallow", which understandably didn't make it into the reference docs). To avoid introducing an inconsistency into the documentation for Python 3.4, I had to choose between rewording all the existing context management and with statement docs or just using the established name for the new construct (and I chose the latter path). The second part of my rationale stems from the fact that ignore and suppress aren't quite synonyms, and that suppressing something is *far* more active than ignoring it. If you completely ignore something, it should be as if it never happened - you certainly don't take any kind of action in response to it. When you suppress something, by contrast, there's a clear implication that it still happens, and then you do something later to make it go away. That means the potential for confusion changes: Plausible interpretations of ignore(SomeException): - that kind of exception is genuinely ignored by never raising it in the first place and proceeding with the next statement immediately after the one where it would have been raised - the exception is raised, but mostly ignored by proceeding with the next statement in the with statement body - the exception is raised, but kind of ignored by proceeding with the next statement after the with statement Plausible interpretations of suppress(SomeException): - the exception is raised, but suppressed immediately after the statement that raised it - the exception is raised, but suppressed at the end of the with statement (which is the first change the context manager has to see it) With "suppress", the implication that the "exception is raised" part still happens is *much* stronger, suggesting that the name is not only more consistent with the existing docs, but also more accurate in an absolute sense. Independently of that, one of the things that has become clear to me in these threads is that many experienced Python developers don't realise that with statements actually *are* intended to be usable as control flow constructs (albeit in a quite restrained way, with their influence on the flow of control being limited solely to the ability to suppress exceptions). If coming to grips with the behaviour of contextlib.suppress expands people's horizons beyond merely using them for resource, state and transaction management, then I see that as a good thing. Now, some of the people in that last group are actually *more* likely to be confused than beginners, since they already have the *mistaken* impression that with statements don't affect control flow. And if you have *that* impression, then you're more likely to guess that suppress() has to be a state management context manager that is giving the interpreter the instruction to not raise the named exception types. Discovering that it *doesn't* work that way may then help to enlighten them as to the true nature of the underlying construct. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at python.org Sat Oct 19 14:44:14 2013 From: christian at python.org (Christian Heimes) Date: Sat, 19 Oct 2013 14:44:14 +0200 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> <5261A26C.7000506@gmail.com> <5261B94F.5030006@gmail.com> Message-ID: <52627E9E.3040003@python.org> Am 19.10.2013 00:56, schrieb Guido van Rossum: > Thanks! That's probably fine for now -- it means the standard library > doesn't know where the root certificates are. We had a huge discussion > about this over on python-tulip: > https://groups.google.com/forum/#!topic/python-tulip/c_lqdFjPEbE > > TL;DR: The stdlib openssl wrapper ought to know where each platform > stores its root certificates and automatically use them, but it > currently doesn't always. Users who really don't care but still want to > use SSL must create an SSL context with verify_mode set to ssl.CERT_NONE > (and live with the risk, obviously). This stuff passes on OS X only > because there's a system openssl library that always uses the system > root certificates. > > If anyone can help fixing the ssl.py module (or the _ssl extension) so > that sslcontext.set_default_verify_paths() uses the system root certs on > Windows that would be a huge help. (I have tried this on an Ubuntu box > too, and there it actually works.) I have worked on some patches and even started to write a PEP about it. You can find an old version of my PEP at https://bitbucket.org/tiran/peps/src/tip/pep-9999.txt . The PEP contains a list of possible locations of root CA certs. The root CA certificate situation is troublesome. Several parsers for Mozilla's NSS certdata.txt are plain wrong and don't handle purpose / trust settings correctly. Even Ubuntu is affected by the bug. The /etc/ssl/certs/ directory contains certificates that are NOT suitable for server cert verification. A couple of months I had a long and fruitful discussion with MAL about the issue. Egenix PyOpenSSL installer comes with a root CA bundle. He tried a couple of approaches to handle trust settings with OpenSSL means. Eventually MAL had to split up the bundle into multiple files for each purpuse, see http://www.egenix.com/company/news/eGenix-pyOpenSSL-Distribution-0.13.2.1.0.1.5.html We should *really* write a PEP about it, specify all details and get a proper review from real experts. This stuff is super complex and highly fragile. :( Christian From solipsis at pitrou.net Sat Oct 19 14:53:18 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 19 Oct 2013 14:53:18 +0200 Subject: [Python-Dev] An interesting exception handling quirk References: Message-ID: <20131019145318.547c1953@fsol> On Sat, 19 Oct 2013 21:41:17 +1000 Nick Coghlan wrote: > > Having noticed the discrepancy, I feel like it should be explicitly > recorded somewhere in the language reference, I'm just not sure where. Since it's a quirk, I don't think it should be mentioned in the language reference. Also, see http://bugs.python.org/issue12029 Regards Antoine. From solipsis at pitrou.net Sat Oct 19 14:54:17 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 19 Oct 2013 14:54:17 +0200 Subject: [Python-Dev] cpython: Issue #18582: provide a faster C implementation of pbkdf2_hmac that works with References: <3d23Gw4zz8z7LkS@mail.python.org> Message-ID: <20131019145417.0ec24a27@fsol> On Sat, 19 Oct 2013 14:25:28 +0200 (CEST) christian.heimes wrote: > > - .. note:: A fast implementation of *pbkdf2_hmac* is only available with > - OpenSSL 1.0 and newer. The Python implementation uses an inline > - version of :mod:`hmac` and is about three times slower. Contrary to > - OpenSSL's current code the length of the password has only a minimal > - impact on the runtime of the Python implementation. > + .. note:: A fast implementation of *pbkdf2_hmac* is available with OpenSSL. > + The Python implementation uses an inline version of :mod:`hmac`. It is > + about three times slower and doesn't release the GIL. The documentation should stop talking about the Python implementation if the C implementation is always used by default. Regards Antoine. From ncoghlan at gmail.com Sat Oct 19 15:34:52 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Oct 2013 23:34:52 +1000 Subject: [Python-Dev] An interesting exception handling quirk In-Reply-To: <20131019145318.547c1953@fsol> References: <20131019145318.547c1953@fsol> Message-ID: On 19 October 2013 22:53, Antoine Pitrou wrote: > On Sat, 19 Oct 2013 21:41:17 +1000 > Nick Coghlan wrote: >> >> Having noticed the discrepancy, I feel like it should be explicitly >> recorded somewhere in the language reference, I'm just not sure where. > > Since it's a quirk, I don't think it should be mentioned in the > language reference. Also, see http://bugs.python.org/issue12029 Ah, thanks for the reference. I'll comment further over there :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at python.org Sat Oct 19 15:36:02 2013 From: christian at python.org (Christian Heimes) Date: Sat, 19 Oct 2013 15:36:02 +0200 Subject: [Python-Dev] cpython: Issue #18582: provide a faster C implementation of pbkdf2_hmac that works with In-Reply-To: <20131019145417.0ec24a27@fsol> References: <3d23Gw4zz8z7LkS@mail.python.org> <20131019145417.0ec24a27@fsol> Message-ID: <52628AC2.6040107@python.org> Am 19.10.2013 14:54, schrieb Antoine Pitrou: > On Sat, 19 Oct 2013 14:25:28 +0200 (CEST) > christian.heimes wrote: >> >> - .. note:: A fast implementation of *pbkdf2_hmac* is only available with >> - OpenSSL 1.0 and newer. The Python implementation uses an inline >> - version of :mod:`hmac` and is about three times slower. Contrary to >> - OpenSSL's current code the length of the password has only a minimal >> - impact on the runtime of the Python implementation. >> + .. note:: A fast implementation of *pbkdf2_hmac* is available with OpenSSL. >> + The Python implementation uses an inline version of :mod:`hmac`. It is >> + about three times slower and doesn't release the GIL. > > The documentation should stop talking about the Python implementation > if the C implementation is always used by default. The C implementation is not used if Python is not compiled with OpenSSL support. From python-dev at masklinn.net Sat Oct 19 15:53:49 2013 From: python-dev at masklinn.net (Xavier Morel) Date: Sat, 19 Oct 2013 15:53:49 +0200 Subject: [Python-Dev] Context management patterns In-Reply-To: References: Message-ID: <81C9F969-305B-439A-8892-BCDC026C28C2@masklinn.net> On 2013-10-19, at 08:38 , Nick Coghlan wrote: >> The above example, especially if extended beyond two files, begs to used in >> a loop, like your 5 line version: >> >> >> for name in ("somefile.tmp", "someotherfile.tmp"): >> with suppress(FileNotFoundError): >> os.remove(name) >> >> which would be fine, of course. >> >> But to some with less education about the how and why, it is not clear why >> it couldn't be written like: >> >> with suppress(FileNotFoundError): >> >> for name in ("somefile.tmp", "someotherfile.tmp"): >> os.remove(name) >> >> yet to the cognoscenti, it is obvious there are seriously different >> semantics. > > However, that's a confusion about exception handling in general, not > about the suppress context manager in particular. The same potential > for conceptual confusion exists between: > > for name in ("somefile.tmp", "someotherfile.tmp"): > try: > os.remove(name) > except FileNotFoundError: > pass > > and: > > try: > for name in ("somefile.tmp", "someotherfile.tmp"): > os.remove(name) > except FileNotFoundError: > pass It could work if the exceptions system was extended to a full-blow conditions system though. From brett at python.org Sat Oct 19 15:54:07 2013 From: brett at python.org (Brett Cannon) Date: Sat, 19 Oct 2013 09:54:07 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Issue #18810: Be optimistic with stat calls when seeing if a directory In-Reply-To: References: <3d1YyH4L2tz7LjP@mail.python.org> Message-ID: On Fri, Oct 18, 2013 at 9:06 PM, Nick Coghlan wrote: > > On 19 Oct 2013 03:24, "brett.cannon" wrote: > > > > http://hg.python.org/cpython/rev/11f2f4af1979 > > changeset: 86444:11f2f4af1979 > > user: Brett Cannon > > date: Fri Oct 18 13:24:13 2013 -0400 > > summary: > > Issue #18810: Be optimistic with stat calls when seeing if a directory > > exists when checking for a package. > > > > Before there was an isdir check and then various isfile checks for > > possible __init__ files when looking for a package. > > This change drops the isdir check by leaning > > on the assumption that a directory will not contain something named > > after the module being imported which is not a directory. If the module > > is a package then it saves a stat call. If there is nothing in the > > directory with the potential package name it also saves a stat call. > > Only if there is something in the directory named the same thing as > > the potential package will the number of stat calls increase > > (due to more wasteful __init__ checks). > > I don't follow this logic. There's now not even an existence check for the > base name, so it reads to me like we will look for all the possible init > file extensions even if there's *no* directory with an appropriate name. > > What am I missing? > ``if cache_module in cache:``, the line above the _path_join() call and the guard that blocks the entire package search. -Brett > Cheers, > Nick. > > > > > Semantically there is no change as the isdir check moved > > down so that namespace packages continue to have no chance of > > accidentally collecting non-existent directories. > > > > files: > > Lib/importlib/_bootstrap.py | 19 +- > > Python/importlib.h | 1537 +++++++++++----------- > > 2 files changed, 777 insertions(+), 779 deletions(-) > > > > > > diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py > > --- a/Lib/importlib/_bootstrap.py > > +++ b/Lib/importlib/_bootstrap.py > > @@ -1406,16 +1406,15 @@ > > # Check if the module is the name of a directory (and thus a > package). > > if cache_module in cache: > > base_path = _path_join(self.path, tail_module) > > - if _path_isdir(base_path): > > - for suffix, loader in self._loaders: > > - init_filename = '__init__' + suffix > > - full_path = _path_join(base_path, init_filename) > > - if _path_isfile(full_path): > > - return (loader(fullname, full_path), > [base_path]) > > - else: > > - # A namespace package, return the path if we don't > also > > - # find a module in the next section. > > - is_namespace = True > > + for suffix, loader in self._loaders: > > + init_filename = '__init__' + suffix > > + full_path = _path_join(base_path, init_filename) > > + if _path_isfile(full_path): > > + return (loader(fullname, full_path), [base_path]) > > + else: > > + # If a namespace package, return the path if we don't > > + # find a module in the next section. > > + is_namespace = _path_isdir(base_path) > > # Check for a file w/ a proper suffix exists. > > for suffix, loader in self._loaders: > > full_path = _path_join(self.path, tail_module + suffix) > > diff --git a/Python/importlib.h b/Python/importlib.h > > --- a/Python/importlib.h > > +++ b/Python/importlib.h > > [stripped] > > > > -- > > Repository URL: http://hg.python.org/cpython > > > > _______________________________________________ > > Python-checkins mailing list > > Python-checkins at python.org > > https://mail.python.org/mailman/listinfo/python-checkins > > > > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Oct 19 15:54:52 2013 From: brett at python.org (Brett Cannon) Date: Sat, 19 Oct 2013 09:54:52 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Issue #18416: Fix various os calls in importlib.machinery.FileFinder In-Reply-To: References: <3d1X6M5RMjz7LjP@mail.python.org> Message-ID: On Fri, Oct 18, 2013 at 8:55 PM, Nick Coghlan wrote: > > On 19 Oct 2013 02:01, "brett.cannon" wrote: > > > > http://hg.python.org/cpython/rev/33844153cd02 > > changeset: 86438:33844153cd02 > > user: Brett Cannon > > date: Fri Oct 18 12:01:06 2013 -0400 > > summary: > > Issue #18416: Fix various os calls in importlib.machinery.FileFinder > > now that self.path is no longer forced to '.'. > > Hmm, could this break subclasses in a similar way? It may be safer to keep > the empty string -> period conversion, even though PathFinder itself > doesn't rely on it any more. > Good point. I'll revert it. -Brett > Cheers, > Nick. > > > > > files: > > Lib/importlib/_bootstrap.py | 4 +- > > Python/importlib.h | 1554 +++++++++++----------- > > 2 files changed, 780 insertions(+), 778 deletions(-) > > > > > > diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py > > --- a/Lib/importlib/_bootstrap.py > > +++ b/Lib/importlib/_bootstrap.py > > @@ -1390,7 +1390,7 @@ > > is_namespace = False > > tail_module = fullname.rpartition('.')[2] > > try: > > - mtime = _os.stat(self.path).st_mtime > > + mtime = _os.stat(self.path or _os.getcwd()).st_mtime > > except OSError: > > mtime = -1 > > if mtime != self._path_mtime: > > @@ -1432,7 +1432,7 @@ > > """Fill the cache of potential modules and packages for this > directory.""" > > path = self.path > > try: > > - contents = _os.listdir(path) > > + contents = _os.listdir(path or _os.getcwd()) > > except (FileNotFoundError, PermissionError, NotADirectoryError): > > # Directory has either been removed, turned into a file, or > made > > # unreadable. > > diff --git a/Python/importlib.h b/Python/importlib.h > > --- a/Python/importlib.h > > +++ b/Python/importlib.h > > [stripped] > > > > -- > > Repository URL: http://hg.python.org/cpython > > > > _______________________________________________ > > Python-checkins mailing list > > Python-checkins at python.org > > https://mail.python.org/mailman/listinfo/python-checkins > > > > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cf.natali at gmail.com Sat Oct 19 15:58:32 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Sat, 19 Oct 2013 15:58:32 +0200 Subject: [Python-Dev] PEP 454 (tracemalloc): new minimalist version In-Reply-To: References: Message-ID: >>> ``get_tracemalloc_memory()`` function: >>> >>> Get the memory usage in bytes of the ``tracemalloc`` module as a >>> tuple: ``(size: int, free: int)``. >>> >>> * *size*: total size of bytes allocated by the module, >>> including *free* bytes >>> * *free*: number of free bytes available to store data >> >> What's *free* exactly? I assume it's linked to the internal storage >> area used by tracemalloc itself, but that's not clear at all. >> >> Also, is the tracemalloc overhead included in the above stats (I'm >> mainly thinking about get_stats() and get_traced_memory()? >> If yes, I find it somewhat confusing: for example, AFAICT, valgrind's >> memcheck doesn't report the memory overhead, although it can be quite >> large, simply because it's not interesting. > > My goal is to able to explain how *every* byte is allocated in Python. > If you enable tracemalloc, your RSS memory will double, or something > like that. You can use get_tracemalloc_memory() to add metrics to a > snapshot. It helps to understand how the RSS memory evolves. > > Basically, get_tracemalloc_size() is the memory used to store traces. > It's something internal to the C module (_tracemalloc). This memory is > not traced because it *is* the traces... and so is not counted in > get_traced_memory(). > > The issue is probably the name (or maybe also the doc): would you > prefer get_python_memory() / get_traces_memory() names, instead of > get_traced_memory() / get_tracemalloc_memory()? No, the names are fine as-is. > FYI Objects allocated in tracemalloc.py (real objects, not traces) are > not counted in get_traced_memory() because of a filter set up by > default (it was not the case in previous versions of the PEP). You can > remove the filter using tracemalloc.clear_filters() to see this > memory. There are two exceptions: Python objects created for the > result of get_traces() and get_stats() are never traced for > efficiency. It *is* possible to trace these objects, but it's really > too slow. get_traces() and get_stats() may be called outside > tracemalloc.py, so another filter would be needed. Well, it's easier > to never trace these objects. Anyway, they are not interesting to > understand where your application leaks memory. Perfect, that's all I wanted to know. > get_object_trace(obj) is a shortcut for > get_trace(get_object_address(obj)). I agree that the wrong size > information can be surprising. > > I can delete get_object_trace(), or rename the function to > get_object_traceback() and modify it to only return the traceback. > > I prefer to keep the function (modified for get_object_traceback). > tracemalloc can be combined with other tools like Melia, Heapy or > objgraph to combine information. When you find an interesting object > with these tools, you may be interested to know where it was > allocated. If you mean modify it to return only the trace, then that's fine. As for the name, traceback does indeed sound less confusing than trace, but we should just make sure that the names are consistent across the API (i.e. always use "trace" or "always use "traceback", not both). >>> ``get_trace(address)`` function: >>> >>> Get the trace of a memory block as a ``(size: int, traceback)`` >>> tuple where *traceback* is a tuple of ``(filename: str, lineno: >>> int)`` tuples, *filename* and *lineno* can be ``None``. >>> >>> Return ``None`` if the ``tracemalloc`` module did not trace the >>> allocation of the memory block. >>> >>> See also ``get_object_trace()``, ``get_stats()`` and >>> ``get_traces()`` functions. >> >> Do you have example use cases where you want to work with a raw addresses? > > An address is the unique key to identify a memory block. In Python, > you don't manipulate directly memory blocks, that's why you have a > get_object_address() function (link objects to traces). > > I added get_trace() because get_traces() is very slow. It would be > stupid to call it if you only need one trace of a memory block. > > I'm not sure that this function is really useful. I added it to > workaround the performance issue, and because I believe that someone > will need it later :-) > > What do you suggest for this function? Well, I can certainly find a use-case for get_object_trace(): even if it uses get_trace() internally, I'm not convinced that the later is useful. If we cannot come up with a use case for working with raw addresses, I'm tempted to just keep get_object_trace() public, and make get_object_address() and get_trace() private. In short, don't make any address-manipulating function public. >> Are those ``match`` methods really necessary for the end user, i.e. >> are they worth being exposed as part of the public API? > > (Oh, I just realized that match_lineno() and may lead to bugs, I removed it.) > > Initially, I exposed the methods for unit tests. Later, I used them in > Snapshot.apply_filters() to factorize the code (before I add 2 > implementations to match a filter, one in C, another in Python). > > I see tracemalloc more as a library, I don't know yet how it will be > used by new tools based on it. Snapshot is more an helper (convinient > class) than a mandatory API to use tracemalloc. You might want to use > directly filters to analyze raw datas. > > Users are supposed to use tracemalloc.add_filter() and > Snapshot.apply_filters(). You prefer to keep them private and not > document them? I don't have a strong opinion on this point. IIUC, you only use those match methods for tests and internally for code factorization: IMO, that's a hint they shouldn't be made public. I usually follow a simple rule of thumb for APIs: if you can't come up with a good use case for something, it shouldn't be made public: it leads to simpler, streamlined APIs, much more pleasant to learn and use. Also, one can always add a new method to an API, but it's impossible to remove from it without breaking backward compatibility. So I'd say leave them out for now, and we'll see with time if they're really necessary. >>> StatsDiff >>> --------- > > Well, StatsDiff is useless :-) I just removed it. Great, that's also what I thought. > I modified GroupedStats.compare_to() to sort differences by default, > but I added a sort parameter to get the list unsorted. sort=False can > be used to sort differences differently (sorting the list twice would > be inefficient). OK, that's a good compromise. > Another option would be add sort_key and sort_reverse parameters. IMO that's overkill: the user can use sort() on the list if he wants a special-purpose order. >>> Snapshot >>> -------- >>> >>> ``Snapshot(timestamp: datetime.datetime, traces: dict=None, stats: >>> dict=None)`` class: >>> >>> Snapshot of traces and statistics on memory blocks allocated by Python. >> >> >> I'm confused. >> Why are get_trace(), get_object_trace(), get_stats() etc not methods >> of a Snapshot object? > > get_stats() returns the current stats. If you call it twice, you get > different results. The principe of a snapshot is to be frozen: stats, > traces and metrics are read once when the snapshot was created. > > To get stats of a snapshot, just read its stats attribute. To get a > trace, it's snapshot.traces[address]. > >> Is it because you don't store all the necessary information in a >> snapshot, or are they just some sort of shorthands, like: >> stats = get_stats() >> vs >> snapshot = Snapshot.create() >> stats = snapshot.stats > > I already used other tools like Melia and Heapy, and it's convinient > to get access to raw data to compute manually my own view. I don't > want to force users to use the high-level API (Snapshot). > > Is it a problem to have two API (low-level like get_stats() and > high-level like Snapshot) for similar use cases? What do you suggest? I didn't understand your above explanation: could get_stats() be implemented atop a snapshot, or not? > => Done, I renamed Snapshot.write() to Snapshot.dump(). > > By the way, load() and dump() are limited to filenames (string). > Should they accept file-like object? isinstance(filename, str) may be > used to check if the parameter is a filename or a open file object. IMO, they're fine as-is. >>> Metric >>> ------ >>> >>> ``Metric(name: str, value: int, format: str)`` class: >>> >>> Value of a metric when a snapshot is created. >> >> Alright, what's a metric again ;-) ? >> >> I don't know if it's customary, but having short examples would IMO be nice. > > => done, I improved the doc Humm... I didn't see any metric example at http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html Is it me? cf From cf.natali at gmail.com Sat Oct 19 16:00:26 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Sat, 19 Oct 2013 16:00:26 +0200 Subject: [Python-Dev] PEP 454 (tracemalloc): new minimalist version In-Reply-To: References: Message-ID: 2013/10/19 Nick Coghlan : > > Speaking of which... Charles-Fran?ois, would you be willing to act as > BDFL-Delegate for this PEP? This will be a very useful new analysis tool, > and between yourself and Victor it looks like you'll be able to come up with > a solid API. > > I just suggested that approach to Guido and he also liked the idea :) Well, I'd be happy to help get this merged. There's still the deadline problem: do we have to get this PEP approved and merged within 24 hours? cf From brett at python.org Sat Oct 19 16:03:28 2013 From: brett at python.org (Brett Cannon) Date: Sat, 19 Oct 2013 10:03:28 -0400 Subject: [Python-Dev] PEP 454 (tracemalloc): new minimalist version In-Reply-To: References: Message-ID: On Sat, Oct 19, 2013 at 10:00 AM, Charles-Fran?ois Natali < cf.natali at gmail.com> wrote: > 2013/10/19 Nick Coghlan : > > > > Speaking of which... Charles-Fran?ois, would you be willing to act as > > BDFL-Delegate for this PEP? This will be a very useful new analysis tool, > > and between yourself and Victor it looks like you'll be able to come up > with > > a solid API. > > > > I just suggested that approach to Guido and he also liked the idea :) > > Well, I'd be happy to help get this merged. > > There's still the deadline problem: do we have to get this PEP > approved and merged within 24 hours? No, it just has to land before b1. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Oct 19 16:14:23 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 20 Oct 2013 00:14:23 +1000 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: <52627E9E.3040003@python.org> References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> <5261A26C.7000506@gmail.com> <5261B94F.5030006@gmail.com> <52627E9E.3040003@python.org> Message-ID: On 19 October 2013 22:44, Christian Heimes wrote: > Am 19.10.2013 00:56, schrieb Guido van Rossum: > A couple of months I had a long and fruitful discussion with MAL about > the issue. Egenix PyOpenSSL installer comes with a root CA bundle. He > tried a couple of approaches to handle trust settings with OpenSSL > means. Eventually MAL had to split up the bundle into multiple files for > each purpuse, see > http://www.egenix.com/company/news/eGenix-pyOpenSSL-Distribution-0.13.2.1.0.1.5.html > > We should *really* write a PEP about it, specify all details and get a > proper review from real experts. This stuff is super complex and highly > fragile. :( At the very least, it would be good if you and/or MAL could review the cert verification in pip. PEP 453 makes that kinda important :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Oct 19 16:42:30 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 20 Oct 2013 00:42:30 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Issue #18810: Be optimistic with stat calls when seeing if a directory In-Reply-To: References: <3d1YyH4L2tz7LjP@mail.python.org> Message-ID: On 19 October 2013 23:54, Brett Cannon wrote: >> I don't follow this logic. There's now not even an existence check for the >> base name, so it reads to me like we will look for all the possible init >> file extensions even if there's *no* directory with an appropriate name. >> >> What am I missing? > > > ``if cache_module in cache:``, the line above the _path_join() call and the > guard that blocks the entire package search. Ah, that's what I get for reviewing diffs on my phone and hence not looking up the full context :) Thanks for the clarification. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at python.org Sat Oct 19 16:44:18 2013 From: christian at python.org (Christian Heimes) Date: Sat, 19 Oct 2013 16:44:18 +0200 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> <5261A26C.7000506@gmail.com> <5261B94F.5030006@gmail.com> <52627E9E.3040003@python.org> Message-ID: <52629AC2.6080507@python.org> Am 19.10.2013 16:14, schrieb Nick Coghlan: > At the very least, it would be good if you and/or MAL could review > the cert verification in pip. PEP 453 makes that kinda important > :) Where can I find the code for PEP 453? From solipsis at pitrou.net Sat Oct 19 16:59:47 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 19 Oct 2013 16:59:47 +0200 Subject: [Python-Dev] cpython: Issue #18582: provide a faster C implementation of pbkdf2_hmac that works with In-Reply-To: <52628AC2.6040107@python.org> References: <3d23Gw4zz8z7LkS@mail.python.org> <20131019145417.0ec24a27@fsol> <52628AC2.6040107@python.org> Message-ID: <20131019165947.5450c470@fsol> On Sat, 19 Oct 2013 15:36:02 +0200 Christian Heimes wrote: > Am 19.10.2013 14:54, schrieb Antoine Pitrou: > > On Sat, 19 Oct 2013 14:25:28 +0200 (CEST) > > christian.heimes wrote: > >> > >> - .. note:: A fast implementation of *pbkdf2_hmac* is only available with > >> - OpenSSL 1.0 and newer. The Python implementation uses an inline > >> - version of :mod:`hmac` and is about three times slower. Contrary to > >> - OpenSSL's current code the length of the password has only a minimal > >> - impact on the runtime of the Python implementation. > >> + .. note:: A fast implementation of *pbkdf2_hmac* is available with OpenSSL. > >> + The Python implementation uses an inline version of :mod:`hmac`. It is > >> + about three times slower and doesn't release the GIL. > > > > The documentation should stop talking about the Python implementation > > if the C implementation is always used by default. > > The C implementation is not used if Python is not compiled with OpenSSL > support. But that's a fringe situation. Any normal build of Python should be compiled with OpenSSL support (and any decent binary build is). I think the mention in the docs is distracting and will create pointless uncertainty in the reader. Regards Antoine. From ncoghlan at gmail.com Sat Oct 19 16:59:13 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 20 Oct 2013 00:59:13 +1000 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: <52629AC2.6080507@python.org> References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> <5261A26C.7000506@gmail.com> <5261B94F.5030006@gmail.com> <52627E9E.3040003@python.org> <52629AC2.6080507@python.org> Message-ID: On 20 October 2013 00:44, Christian Heimes wrote: > Am 19.10.2013 16:14, schrieb Nick Coghlan: >> At the very least, it would be good if you and/or MAL could review >> the cert verification in pip. PEP 453 makes that kinda important >> :) > > Where can I find the code for PEP 453? It's the cert verification in pip that's relevant - the PEP was updated so that ensurepip itself never talks to the internet. So I guess that would mean checking the cert verification in pip's vendored copy of requests: https://github.com/pypa/pip/tree/develop/pip/vendor/requests (So I guess if you do find any issues, they would likely be applicable to the upstream requests package as well) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Oct 19 17:15:05 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 20 Oct 2013 01:15:05 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Issue #19254: Provide an optimized Python implementation of PBKDF2_HMAC In-Reply-To: <3d22zd0Rs9z7LjP@mail.python.org> References: <3d22zd0Rs9z7LjP@mail.python.org> Message-ID: On 19 October 2013 22:12, christian.heimes wrote: > http://hg.python.org/cpython/rev/e73627483d2f > changeset: 86467:e73627483d2f > user: Christian Heimes > date: Sat Oct 19 14:12:02 2013 +0200 > summary: > Issue #19254: Provide an optimized Python implementation of PBKDF2_HMAC > > files: > Doc/library/hashlib.rst | 6 ++- > Lib/hashlib.py | 69 +++++++++++++++++++++++++-- > Lib/test/test_hashlib.py | 20 ++++++- > Misc/NEWS | 2 + > 4 files changed, 86 insertions(+), 11 deletions(-) > diff --git a/Lib/test/test_hashlib.py b/Lib/test/test_hashlib.py > --- a/Lib/test/test_hashlib.py > +++ b/Lib/test/test_hashlib.py > @@ -18,11 +18,13 @@ > import unittest > import warnings > from test import support > -from test.support import _4G, bigmemtest > +from test.support import _4G, bigmemtest, import_fresh_module > > # Were we compiled --with-pydebug or with #define Py_DEBUG? > COMPILED_WITH_PYDEBUG = hasattr(sys, 'gettotalrefcount') > > +c_hashlib = import_fresh_module('hashlib', fresh=['_hashlib']) Looks like this import is failing on at least some platforms: http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/6535/steps/test/logs/stdio http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/204/steps/test/logs/stdio Due to the build failing: http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/6535/steps/compile/logs/stdio http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/204/steps/compile/logs/stdio Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at python.org Sat Oct 19 17:52:48 2013 From: christian at python.org (Christian Heimes) Date: Sat, 19 Oct 2013 17:52:48 +0200 Subject: [Python-Dev] pip SSL In-Reply-To: References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> <5261A26C.7000506@gmail.com> <5261B94F.5030006@gmail.com> <52627E9E.3040003@python.org> <52629AC2.6080507@python.org> Message-ID: <5262AAD0.9050900@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Am 19.10.2013 16:59, schrieb Nick Coghlan: > It's the cert verification in pip that's relevant - the PEP was > updated so that ensurepip itself never talks to the internet. So I > guess that would mean checking the cert verification in pip's > vendored copy of requests: > https://github.com/pypa/pip/tree/develop/pip/vendor/requests > > (So I guess if you do find any issues, they would likely be > applicable to the upstream requests package as well) Oh heck, where should I start? The cacert.pem file is outdated. Also it's unclear who has generated the file and how it was generated from certdata.txt. It may very well contain revoked certificates, too. You can find the latest version of the file at http://hg.mozilla.org/mozilla-central/file/tip/security/nss/lib/ckfw/builtins/certdata.txt . A proper tool is required to generate a correct PEM file. It must understand *ALL* fields. I have some code for that but it's not ready yet. pip uses requests and requests rolls its own code for or on top of Python stdlib modules, e.g. urllib3 with ssl_match_hostname. The method has the same security flaw as Python's ssl.match_hostname() function for IDNs. I'm a bit worried that we have to review and validate all 3rd party packages and copies of stdlib modules for issues. The assert_fingerprint() function looks kinda funny. It uses sha1() or md5() on the DER representation of the cert. It's not how you are suppose to take fingerprints for cert pinning. But Python's ssl has no way to get the SPKI from the cert yet. I'm working on that as well. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCgAGBQJSYqrBAAoJEMeIxMHUVQ1FeSAP/3C4g3Sp6Fg976C5eihDpuLo VKB83nf708iwR990lJ6AYAiHDRjwVk6ssgYX4EfA3qQqjiAOykIQKZYcYrBA36lT FDn3gIXkh6x1QnwEOopGqrdbhSbDqPB57zRAZrmzJp8JTvOx4FYVgmx6bi2yumst w2m+ovWjxzUOlr1V7LM2/vzxSJXLyg+Espps3kDgX96qZvHXCfn/M39Y5R39on7v Er3qmD5aHEOnVnA1cH/OC7O8uJm8dPrm7wocztErZWyy006chW2B8edvFpjW8iEn StYxNw7Ko6jr2ncCwAKntVavGRtbHJowaF4l4yTCZ6suCx+LAzy7O+X90Ic1LknN o/RLSfJeyhUOHpADwloKfjRuPk2twq46z96GauoFBThaBCca7mRS29EudWG54Dn1 tT1/7+b3FfiU1GmWqzTpxgrJiRREk+XTEVCmhq2XUdBnGQI7G6RT9BefVfYzep06 Z0hKWdIR2moC21iPcBMIOnXqscaMHjvcVOnScv05UiE5et0fB8lAfoZJ9u1G5UC4 vkifZpfOfCDMh3HXSCiRp2TEUo/GPy35P/8Tk1O602nGj3oRxPJ1fdOlIexu+9bV S/kGwMjhyLQHDp0786AwDnv/NNOK6hJHCiZLLqX6F0+K4RdlRRd/6lVvyxKGT8ca OXxoornL8iyvEnyti7cq =BTNE -----END PGP SIGNATURE----- From christian at python.org Sat Oct 19 18:01:26 2013 From: christian at python.org (Christian Heimes) Date: Sat, 19 Oct 2013 18:01:26 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Issue #19254: Provide an optimized Python implementation of PBKDF2_HMAC In-Reply-To: References: <3d22zd0Rs9z7LjP@mail.python.org> Message-ID: <5262ACD6.2020002@python.org> Am 19.10.2013 17:15, schrieb Nick Coghlan: >> +c_hashlib = import_fresh_module('hashlib', fresh=['_hashlib']) > > Looks like this import is failing on at least some platforms: > > http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/6535/steps/test/logs/stdio > http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/204/steps/test/logs/stdio > > Due to the build failing: > http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/6535/steps/compile/logs/stdio > http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/204/steps/compile/logs/stdio Thanks Nick! Actually it was http://hg.python.org/cpython/rev/0a26ef834a49. The fix in http://hg.python.org/cpython/rev/88fac1574049 should have taken care of the issue. Sorry for the noise, I don't have OpenSSL 0.9.8 on this machine. Christian From christian at python.org Sat Oct 19 18:05:35 2013 From: christian at python.org (Christian Heimes) Date: Sat, 19 Oct 2013 18:05:35 +0200 Subject: [Python-Dev] cpython: Issue #18582: provide a faster C implementation of pbkdf2_hmac that works with In-Reply-To: <20131019165947.5450c470@fsol> References: <3d23Gw4zz8z7LkS@mail.python.org> <20131019145417.0ec24a27@fsol> <52628AC2.6040107@python.org> <20131019165947.5450c470@fsol> Message-ID: <5262ADCF.8080408@python.org> Am 19.10.2013 16:59, schrieb Antoine Pitrou: > But that's a fringe situation. Any normal build of Python should be > compiled with OpenSSL support (and any decent binary build is). I think > the mention in the docs is distracting and will create pointless > uncertainty in the reader. HMAC_CTX_copy() is not available in OpenSSL < 1.0. My optimized code won't run on OpenSSL 0.9.8. IMHO it's important to understand that some code may complete in ~100ms instead of ~30ms. From guido at python.org Sat Oct 19 18:22:36 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 19 Oct 2013 09:22:36 -0700 Subject: [Python-Dev] Looking for volunteers to test Tulip on Windows In-Reply-To: References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> <5261A26C.7000506@gmail.com> <5261B94F.5030006@gmail.com> <52627E9E.3040003@python.org> Message-ID: You should ask Glyph too. He supplied lots of useful info about cert checking on the python-tulip list. On Sat, Oct 19, 2013 at 7:14 AM, Nick Coghlan wrote: > On 19 October 2013 22:44, Christian Heimes wrote: > > Am 19.10.2013 00:56, schrieb Guido van Rossum: > > A couple of months I had a long and fruitful discussion with MAL about > > the issue. Egenix PyOpenSSL installer comes with a root CA bundle. He > > tried a couple of approaches to handle trust settings with OpenSSL > > means. Eventually MAL had to split up the bundle into multiple files for > > each purpuse, see > > > http://www.egenix.com/company/news/eGenix-pyOpenSSL-Distribution-0.13.2.1.0.1.5.html > > > > We should *really* write a PEP about it, specify all details and get a > > proper review from real experts. This stuff is super complex and highly > > fragile. :( > > At the very least, it would be good if you and/or MAL could review the > cert verification in pip. PEP 453 makes that kinda important :) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Oct 19 18:28:10 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 19 Oct 2013 18:28:10 +0200 Subject: [Python-Dev] cpython: Issue #18582: provide a faster C implementation of pbkdf2_hmac that works with In-Reply-To: <5262ADCF.8080408@python.org> References: <3d23Gw4zz8z7LkS@mail.python.org> <20131019145417.0ec24a27@fsol> <52628AC2.6040107@python.org> <20131019165947.5450c470@fsol> <5262ADCF.8080408@python.org> Message-ID: <20131019182810.0653553b@fsol> On Sat, 19 Oct 2013 18:05:35 +0200 Christian Heimes wrote: > Am 19.10.2013 16:59, schrieb Antoine Pitrou: > > But that's a fringe situation. Any normal build of Python should be > > compiled with OpenSSL support (and any decent binary build is). I think > > the mention in the docs is distracting and will create pointless > > uncertainty in the reader. > > HMAC_CTX_copy() is not available in OpenSSL < 1.0. My optimized code > won't run on OpenSSL 0.9.8. IMHO it's important to understand that some > code may complete in ~100ms instead of ~30ms. Well, I disagree about the importance of documenting a 3x difference on a select function (rather than, say, a couple orders of magnitude), but the OpenSSL version issue at least makes the situation more plausible :-) (even though OpenSSL 1.0 is already 3.5 years old) Regards Antoine. From donald at stufft.io Sat Oct 19 19:57:30 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 19 Oct 2013 13:57:30 -0400 Subject: [Python-Dev] pip SSL In-Reply-To: <5262AAD0.9050900@python.org> References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> <5261A26C.7000506@gmail.com> <5261B94F.5030006@gmail.com> <52627E9E.3040003@python.org> <52629AC2.6080507@python.org> <5262AAD0.9050900@python.org> Message-ID: <739A4AB2-5345-4F87-BF59-6F242883C026@stufft.io> One of the reasons we switched to using requests was to help centralize the SSL handling code over to requests. So any issues could be fixed there and we just pull in a newer version of requests. On Oct 19, 2013, at 11:52 AM, Christian Heimes wrote: > Signed PGP part > Am 19.10.2013 16:59, schrieb Nick Coghlan: > > It's the cert verification in pip that's relevant - the PEP was > > updated so that ensurepip itself never talks to the internet. So I > > guess that would mean checking the cert verification in pip's > > vendored copy of requests: > > https://github.com/pypa/pip/tree/develop/pip/vendor/requests > > > > (So I guess if you do find any issues, they would likely be > > applicable to the upstream requests package as well) > > Oh heck, where should I start? > > The cacert.pem file is outdated. Also it's unclear who has generated > the file and how it was generated from certdata.txt. It may very well > contain revoked certificates, too. You can find the latest version of > the file at > > > http://hg.mozilla.org/mozilla-central/file/tip/security/nss/lib/ckfw/builtins/certdata.txt > > . A proper tool is required to generate a correct PEM file. It must > understand *ALL* fields. I have some code for that but it's not ready yet. > > > pip uses requests and requests rolls its own code for or on top of > Python stdlib modules, e.g. urllib3 with ssl_match_hostname. The > method has the same security flaw as Python's ssl.match_hostname() > function for IDNs. I'm a bit worried that we have to review and > validate all 3rd party packages and copies of stdlib modules for issues. > > > The assert_fingerprint() function looks kinda funny. It uses sha1() or > md5() on the DER representation of the cert. It's not how you are > suppose to take fingerprints for cert pinning. But Python's ssl has no > way to get the SPKI from the cert yet. I'm working on that as well. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From victor.stinner at gmail.com Sat Oct 19 20:04:05 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 19 Oct 2013 20:04:05 +0200 Subject: [Python-Dev] cpython: Issue #18582: provide a faster C implementation of pbkdf2_hmac that works with In-Reply-To: <5262ADCF.8080408@python.org> References: <3d23Gw4zz8z7LkS@mail.python.org> <20131019145417.0ec24a27@fsol> <52628AC2.6040107@python.org> <20131019165947.5450c470@fsol> <5262ADCF.8080408@python.org> Message-ID: Does Python officially support opsenssl < 1.0? Which OS uses such old version? On Windows, Python embeds its own copy of openssl for example. Victor Le 19 oct. 2013 18:07, "Christian Heimes" a ?crit : > Am 19.10.2013 16:59, schrieb Antoine Pitrou: > > But that's a fringe situation. Any normal build of Python should be > > compiled with OpenSSL support (and any decent binary build is). I think > > the mention in the docs is distracting and will create pointless > > uncertainty in the reader. > > HMAC_CTX_copy() is not available in OpenSSL < 1.0. My optimized code > won't run on OpenSSL 0.9.8. IMHO it's important to understand that some > code may complete in ~100ms instead of ~30ms. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Sat Oct 19 20:06:11 2013 From: christian at python.org (Christian Heimes) Date: Sat, 19 Oct 2013 20:06:11 +0200 Subject: [Python-Dev] cpython: Issue #18582: provide a faster C implementation of pbkdf2_hmac that works with In-Reply-To: References: <3d23Gw4zz8z7LkS@mail.python.org> <20131019145417.0ec24a27@fsol> <52628AC2.6040107@python.org> <20131019165947.5450c470@fsol> <5262ADCF.8080408@python.org> Message-ID: <5262CA13.1040401@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Am 19.10.2013 20:04, schrieb Victor Stinner: > Does Python officially support opsenssl < 1.0? Which OS uses such > old version? > > On Windows, Python embeds its own copy of openssl for example. Mac OS X has only OpenSSL 0.9.8 and will not receive any updates to 1.0 from Apple. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCgAGBQJSYsoLAAoJEMeIxMHUVQ1FHW4P/RFILORWT9hrPQS5rC1W/ioc JxOhAeuoWTy4cSvwOvvl9B29e7GxUVoGa029Ub6FXsptcs70odrdiJnvYG1J43z/ xZUBgLiog22wmqA35cmM46pChq8X0ERvR6Nfvuc10dQo+ePjujQfa8qiSP35kmvS fI1YxoBxR1diSDDilbS28NrQOLevV7lQR8k6bqqvSb9JPym9OcSrl/UFa57mOIvD kj2ClzylzPGpIWfWiNu/Nte6XNxASe69LfYCHmb12L93Wtf+s/0ZN0uALdBMfSmB RUIAoeP6dTJm0H9AvuEHU4JFURU2Q12x15dmloBw+W3Tj3KpRV7eSCBgQOpL1UMb u2s57JOO1Uv0Nd4OIiOBGCOO5zFbzThKYxiTrXG6HAhd2cxrNNfHKWy9wJdaFfnt /m9rFg2vrCNWu3B0YQIuuBmIVPAy4gw+Vh+pIFLdC4vx14LGNE+E5IG9/48mBQ/7 9XDjHLSpq0EeAv6JYOm257JySQTe/BFxbwXEECM8x8rYk0dapy6DEK6kfPDmOc0M bagHWPuBr2OT+M/MPac+vOs3jzSWaKMynOTEvm5cKLmPI/I/oupLZ8kh48RUFmaH IN0DJXD28p2JG/NykqnQ/z25STAzQOXHcXiFXkaCgnA6LmXwPtkmCAJeqzOc8M/x 4R1akowEUir6Zw4SUjGR =B/+G -----END PGP SIGNATURE----- From solipsis at pitrou.net Sat Oct 19 20:13:37 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 19 Oct 2013 20:13:37 +0200 Subject: [Python-Dev] cpython: Issue #18582: provide a faster C implementation of pbkdf2_hmac that works with In-Reply-To: References: <3d23Gw4zz8z7LkS@mail.python.org> <20131019145417.0ec24a27@fsol> <52628AC2.6040107@python.org> <20131019165947.5450c470@fsol> <5262ADCF.8080408@python.org> Message-ID: <20131019201337.5959ff4e@fsol> On Sat, 19 Oct 2013 20:04:05 +0200 Victor Stinner wrote: > Does Python officially support opsenssl < 1.0? Which OS uses such old > version? We try to support any reasonably modern POSIX-compliant system, e.g. less than 10 years old. Of course, we only "officially support" those platforms which have a stable, green buildbot. Regards Antoine. From larry at hastings.org Sat Oct 19 21:00:39 2013 From: larry at hastings.org (Larry Hastings) Date: Sat, 19 Oct 2013 12:00:39 -0700 Subject: [Python-Dev] I *might* push Python 3.4.0 alpha 4 back a week Message-ID: <5262D6D7.8050908@hastings.org> A lot has landed in trunk in the last day or two: Tulip, Argument Clinic, and statistics just landed too. The buildbots are upset at us humans--there's a lot of red. See for yourself: http://buildbot.python.org/all/waterfall?category=3.x.stable I'd like to tag Alpha 4 late tonight, but if the buildbots are still mostly-red it's a bad idea. If you can help make the buildbots happy, and have some time in the next eight or ten hours, please hop in to #python-dev and ask what you can do to help. SOS -- Save Our Schedule, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From graffatcolmingov at gmail.com Sat Oct 19 21:46:06 2013 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Sat, 19 Oct 2013 14:46:06 -0500 Subject: [Python-Dev] pip SSL In-Reply-To: <739A4AB2-5345-4F87-BF59-6F242883C026@stufft.io> References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> <5261A26C.7000506@gmail.com> <5261B94F.5030006@gmail.com> <52627E9E.3040003@python.org> <52629AC2.6080507@python.org> <5262AAD0.9050900@python.org> <739A4AB2-5345-4F87-BF59-6F242883C026@stufft.io> Message-ID: Also the three of us maintaining requests and the author of urllib3 are all very conscious that the packaged pem file is outdated. We have an open issue about how to rebuild it accurately while taking into consideration (and not including) the ones that have been revoked. Any suggestions you have can be sent to me off list or reported on the issue tracker. On Sat, Oct 19, 2013 at 12:57 PM, Donald Stufft wrote: > One of the reasons we switched to using requests was to help centralize the SSL > handling code over to requests. So any issues could be fixed there and we just > pull in a newer version of requests. > > On Oct 19, 2013, at 11:52 AM, Christian Heimes wrote: > >> Signed PGP part >> Am 19.10.2013 16:59, schrieb Nick Coghlan: >> > It's the cert verification in pip that's relevant - the PEP was >> > updated so that ensurepip itself never talks to the internet. So I >> > guess that would mean checking the cert verification in pip's >> > vendored copy of requests: >> > https://github.com/pypa/pip/tree/develop/pip/vendor/requests >> > >> > (So I guess if you do find any issues, they would likely be >> > applicable to the upstream requests package as well) >> >> Oh heck, where should I start? >> >> The cacert.pem file is outdated. Also it's unclear who has generated >> the file and how it was generated from certdata.txt. It may very well >> contain revoked certificates, too. You can find the latest version of >> the file at >> >> >> http://hg.mozilla.org/mozilla-central/file/tip/security/nss/lib/ckfw/builtins/certdata.txt >> >> . A proper tool is required to generate a correct PEM file. It must >> understand *ALL* fields. I have some code for that but it's not ready yet. >> >> >> pip uses requests and requests rolls its own code for or on top of >> Python stdlib modules, e.g. urllib3 with ssl_match_hostname. The >> method has the same security flaw as Python's ssl.match_hostname() >> function for IDNs. I'm a bit worried that we have to review and >> validate all 3rd party packages and copies of stdlib modules for issues. >> >> >> The assert_fingerprint() function looks kinda funny. It uses sha1() or >> md5() on the DER representation of the cert. It's not how you are >> suppose to take fingerprints for cert pinning. But Python's ssl has no >> way to get the SPKI from the cert yet. I'm working on that as well. >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/graffatcolmingov%40gmail.com > From v+python at g.nevcal.com Sat Oct 19 22:12:12 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 19 Oct 2013 13:12:12 -0700 Subject: [Python-Dev] pip SSL In-Reply-To: References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> <5261A26C.7000506@gmail.com> <5261B94F.5030006@gmail.com> <52627E9E.3040003@python.org> <52629AC2.6080507@python.org> <5262AAD0.9050900@python.org> <739A4AB2-5345-4F87-BF59-6F242883C026@stufft.io> Message-ID: <5262E79C.9000109@g.nevcal.com> On 10/19/2013 12:46 PM, Ian Cordasco wrote: > Also the three of us maintaining requests and the author of urllib3 > are all very conscious that the packaged pem file is outdated. We have > an open issue about how to rebuild it accurately while taking into > consideration (and not including) the ones that have been revoked. Any > suggestions you have can be sent to me off list or reported on the > issue tracker. Is this another issue like the time zone database? Something that needs to be packaged with some versions of Python, but that needs a mechanism to update it later for accuracy (which, in this case, also implies security)? Could a similar mechanism be used for both? -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Oct 19 23:59:29 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 19 Oct 2013 23:59:29 +0200 Subject: [Python-Dev] [python-committers] I *might* push Python 3.4.0 alpha 4 back a week In-Reply-To: <5262D6D7.8050908@hastings.org> References: <5262D6D7.8050908@hastings.org> Message-ID: 2013/10/19 Larry Hastings : > A lot has landed in trunk in the last day or two: Tulip, Argument Clinic, > and statistics just landed too. Wow, Python 3.4 looks great! I was waiting for statistics. Don't forget to update the What's New in Python 3.4 document. Victor From guido at python.org Sun Oct 20 00:44:29 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 19 Oct 2013 15:44:29 -0700 Subject: [Python-Dev] PEP 454 (tracemalloc): new minimalist version In-Reply-To: References: Message-ID: Let it be known to all that Charles-Fran?ois Natali is hereby appointed BDFL for this PEP. On Sat, Oct 19, 2013 at 7:00 AM, Charles-Fran?ois Natali < cf.natali at gmail.com> wrote: > 2013/10/19 Nick Coghlan : > > > > Speaking of which... Charles-Fran?ois, would you be willing to act as > > BDFL-Delegate for this PEP? This will be a very useful new analysis tool, > > and between yourself and Victor it looks like you'll be able to come up > with > > a solid API. > > > > I just suggested that approach to Guido and he also liked the idea :) > > Well, I'd be happy to help get this merged. > > There's still the deadline problem: do we have to get this PEP > approved and merged within 24 hours? > > cf > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Oct 20 03:51:05 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 20 Oct 2013 11:51:05 +1000 Subject: [Python-Dev] pip SSL In-Reply-To: <5262E79C.9000109@g.nevcal.com> References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> <5261A26C.7000506@gmail.com> <5261B94F.5030006@gmail.com> <52627E9E.3040003@python.org> <52629AC2.6080507@python.org> <5262AAD0.9050900@python.org> <739A4AB2-5345-4F87-BF59-6F242883C026@stufft.io> <5262E79C.9000109@g.nevcal.com> Message-ID: On 20 Oct 2013 06:14, "Glenn Linderman" wrote: > > On 10/19/2013 12:46 PM, Ian Cordasco wrote: >> >> Also the three of us maintaining requests and the author of urllib3 >> are all very conscious that the packaged pem file is outdated. We have >> an open issue about how to rebuild it accurately while taking into >> consideration (and not including) the ones that have been revoked. Any >> suggestions you have can be sent to me off list or reported on the >> issue tracker. > > Is this another issue like the time zone database? Something that needs to be packaged with some versions of Python, but that needs a mechanism to update it later for accuracy (which, in this case, also implies security)? > > Could a similar mechanism be used for both? Once pip is installed, then "pip install --upgrade pip" will update it. This request was about getting the *current* state reviewed prior to the pip 1.5 release, since 1.5 is the version likely to be provided by "ensurepip" in CPython 3.4. As Donald noted the fact pip uses requests internally is actually a benefit for the broader Python ecosystem, since it means fixing the cert management and verification for pip (by fixing requests and updating the bundled version) will fix them for a lot of other projects as well. Cheers, Nick. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Oct 20 06:32:33 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 20 Oct 2013 14:32:33 +1000 Subject: [Python-Dev] pip SSL In-Reply-To: References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> <5261A26C.7000506@gmail.com> <5261B94F.5030006@gmail.com> <52627E9E.3040003@python.org> <52629AC2.6080507@python.org> <5262AAD0.9050900@python.org> <739A4AB2-5345-4F87-BF59-6F242883C026@stufft.io> Message-ID: On 20 October 2013 05:46, Ian Cordasco wrote: > Also the three of us maintaining requests and the author of urllib3 > are all very conscious that the packaged pem file is outdated. We have > an open issue about how to rebuild it accurately while taking into > consideration (and not including) the ones that have been revoked. Any > suggestions you have can be sent to me off list or reported on the > issue tracker. The requests issue Ian is referring to: https://github.com/kennethreitz/requests/issues/1659 The next version of PEP 453 will include getting this resolved as part of the integration timeline: ======================== * by December 29th (1 week prior to the scheduled date of 3.4.0 beta 2) ``requests`` certificate management issue resolved ``ensurepip`` updated to the final release of pip 1.5, or a subsequent maintenance release (including a suitably updated vendored copy of ``requests``) ======================== And also mentions it under the "security considerations" section for the bootstrapping mechanism: ======================== Only users that choose to use ``pip`` to communicate with PyPI will need to pay attention to the additional security considerations that come with doing so. However, the core CPython team will also assist with reviewing and resolving the `certificate update management issue `__ currently affecting the ``requests`` project (and hence ``pip``). ======================== Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From arigo at tunes.org Sun Oct 20 08:08:53 2013 From: arigo at tunes.org (Armin Rigo) Date: Sun, 20 Oct 2013 08:08:53 +0200 Subject: [Python-Dev] An interesting exception handling quirk In-Reply-To: References: Message-ID: Hi Nick, On Sat, Oct 19, 2013 at 1:41 PM, Nick Coghlan wrote: > recreating the *exact* exception subclass check from > Python is actually difficult these days. Can't it be done roughly like that? def __exit__(self, typ, val, tb): try: raise typ, val except self.exceptions: return True except: return False A bient?t, Armin. From ncoghlan at gmail.com Sun Oct 20 12:24:25 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 20 Oct 2013 20:24:25 +1000 Subject: [Python-Dev] An interesting exception handling quirk In-Reply-To: References: Message-ID: On 20 October 2013 16:08, Armin Rigo wrote: > Hi Nick, > > On Sat, Oct 19, 2013 at 1:41 PM, Nick Coghlan wrote: >> recreating the *exact* exception subclass check from >> Python is actually difficult these days. > > Can't it be done roughly like that? > > def __exit__(self, typ, val, tb): > try: > raise typ, val > except self.exceptions: > return True > except: > return False In Python 3, you have to use "raise type if val is None else val" instead, and you then have to deal with the fact that raise will overwrite an already set __context__ on the exception value (contextlib.ExitStack actually has to work around that when unwinding the stack by restoring a more appropriate __context__ value). But yes, actually reraising it does let you reproduce the exception matching. That said, it actually occurs to me now that the current behaviour (overwriting an already set __context__) could arguably be considered a bug, since we don't overwrite an already set __traceback__. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Oct 20 14:27:56 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 20 Oct 2013 22:27:56 +1000 Subject: [Python-Dev] PEP 453 (ensurepip) updated Message-ID: I have posted the latest version of PEP 453 to python.org. It is available in the usual place: http://www.python.org/dev/peps/pep-0453/ Three significant changes have been made since the last posted version: * removed the proposal to change the script installation directory on Windows, due to a backwards compatibility issue identified for Windows package installers created with earlier versions of Python (and vice-versa when attempting to use installers created with Python 3.4 on older versions) * noted the current certificate verification concerns for the requests project, and made resolution of that a requirement for inclusion of ensurepip in the final release of Python 3.4 * added an integration timeline, including a December 29th deadline for the inclusion of pip 1.5 (or a subsequent maintenance release) that includes a resolution of the certificate verification issues in requests Rather than posting the whole document again, I have just quoted the most relevant sections for these changes: Explicit bootstrapping mechanism ================================ Security considerations ----------------------- The design in this PEP has been deliberately chosen to avoid making any significant changes to the trust model of CPython for end users that do not subsequently run the command ``pip install --upgrade pip``. The installers will contain all the components of a fully functioning version of Python, including the ``pip`` installer. The installation process will *not* require network access, and will *not* rely on trusting the security of the network connection established between ``pip`` and the Python package index. Only users that choose to use ``pip`` to communicate with PyPI will need to pay attention to the additional security considerations that come with doing so. However, the core CPython team will still assist with reviewing and resolving at least the `certificate update management issue `__ currently affecting the ``requests`` project (and hence ``pip``), and may also be able to offer assistance in resolving other identified security concerns [#cert-verification]_. Integration timeline -------------------- If this PEP is accepted, the proposed time frame for integration of ``pip`` into the CPython release is as follows: * as soon as possible after the release of 3.4.0 alpha 4 * Documentation updated and ``ensurepip`` implemented based on a pre-release version of ``pip`` 1.5. * All other proposed functional changes for Python 3.4 implemented, including the installer updates to invoke ``ensurepip``. * by November 20th (3 days prior to the scheduled date of 3.4.0 beta 1) * ``ensurepip`` updated to use a ``pip`` 1.5 release candidate. * PEP 101 updated to cover ensuring the bundled version of ``pip`` is up to date. * by November 24th (scheduled date of 3.4.0 beta 1) * As with any other new feature, all proposed functional changes for Python 3.4 must be implemented prior to the beta feature freeze. * by December 29th (1 week prior to the scheduled date of 3.4.0 beta 2) * ``requests`` certificate management issue resolved * ``ensurepip`` updated to the final release of ``pip`` 1.5, or a subsequent maintenance release (including a suitably updated vendored copy of ``requests``) (See PEP 429 for the current official scheduled dates of each release. Dates listed above are accurate as of October 20th, 2013.) If there is no final or maintenance release of ``pip`` 1.5 with a suitable updated version of ``requests`` available by one week before the scheduled Python 3.4 beta 2 release, then implementation of this PEP will be deferred to Python 3.5. Note that this scenario is considered unlikely - the tentative date for the ``pip`` 1.5 release is currently December 1st. In future CPython releases, this kind of coordinated scheduling shouldn't be needed: the CPython release manager will be able to just update to the latest released version of ``pip``. However, in this case, some fixes are needed in ``pip`` in order to allow the bundling to work correctly, and the certificate update mechanism for ``requests`` needs to be improved, so the ``pip`` 1.5 release cycle needs to be properly aligned with the CPython 3.4 beta releases. Appendix: Rejected Proposals ============================ Changing the name of the scripts directory on Windows ----------------------------------------------------- Earlier versions of this PEP proposed changing the name of the script installation directory on Windows from "Scripts" to "bin" in order to improve the cross-platform consistency of the virtual environments created by ``pyvenv``. However, Paul Moore determined that this change was likely backwards incompatible with cross-version Windows installers created with previous versions of Python, so the change has been removed from this PEP [#windows-incompatibility]_. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Sun Oct 20 17:03:19 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 20 Oct 2013 17:03:19 +0200 Subject: [Python-Dev] PEP 454 (tracemalloc): new minimalist version In-Reply-To: References: Message-ID: 2013/10/19 Charles-Fran?ois Natali : >> get_object_trace(obj) is a shortcut for >> get_trace(get_object_address(obj)). I agree that the wrong size >> information can be surprising. >> >> I can delete get_object_trace(), or rename the function to >> get_object_traceback() and modify it to only return the traceback. >> >> I prefer to keep the function (modified for get_object_traceback). >> tracemalloc can be combined with other tools like Melia, Heapy or >> objgraph to combine information. When you find an interesting object >> with these tools, you may be interested to know where it was >> allocated. > > If you mean modify it to return only the trace, then that's fine. > As for the name, traceback does indeed sound less confusing than > trace, but we should just make sure that the names are consistent > across the API (i.e. always use "trace" or "always use "traceback", > not both). I'm not sure that you understood my proposition. Extract of the doc: Example of trace: (32, (('x.py', 7), ('x.py', 11))). The memory block has a size of 32 bytes and was allocated at x.py:7, line called from line x.py:11. A trace is (size, traceback). Currently, get_trace(address), get_object_trace(object), get_traces() return traces. You look to be confused by get_object_trace() returning a trace because the size may be smaller than the total size. I proposed to rename get_object_trace() to get_object_traceback() and modify it to only return the traceback: (('x.py', 7), ('x.py', 11)). But get_object_trace/traceback() can also be removed, because it's trivial to reimplement it using get_object_address() and get_trace(). It's maybe less suprising to get a wrong size using get_trace(get_object_address(obj)). > Well, I can certainly find a use-case for get_object_trace(): even if > it uses get_trace() internally, I'm not convinced that the later is > useful. > > If we cannot come up with a use case for working with raw addresses, > I'm tempted to just keep get_object_trace() public, and make > get_object_address() and get_trace() private. > In short, don't make any address-manipulating function public. I would like to keep functions using addresses. Usecase: call get_traces() to get all traces, and then (maybe after the process exited) use it to get the trace of different objects using get_object_address(). > [Filter.match*] > IIUC, you only use those match methods for tests and internally for > code factorization: IMO, that's a hint they shouldn't be made public. Ok, I made the methods private. >> I already used other tools like Melia and Heapy, and it's convinient >> to get access to raw data to compute manually my own view. I don't >> want to force users to use the high-level API (Snapshot). >> >> Is it a problem to have two API (low-level like get_stats() and >> high-level like Snapshot) for similar use cases? What do you suggest? > > I didn't understand your above explanation: could get_stats() be > implemented atop a snapshot, or not? get_stats() cannot be implemented on top of Snapshot. > Humm... > I didn't see any metric example at > http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html > Is it me? http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html#metric "Example of metrics: Resident Set Size (RSS) memory of a process, memory in bytes used by Python, number of Python objects, etc." Did you expect something else for examples? Victor From Mark.Favas at csiro.au Sun Oct 20 09:24:30 2013 From: Mark.Favas at csiro.au (Mark.Favas at csiro.au) Date: Sun, 20 Oct 2013 07:24:30 +0000 Subject: [Python-Dev] Recall: cpython: Rename contextlib.ignored() to contextlib.ignore(). Message-ID: <03E596E5860CB34DA4825777BB5FF6776BB319F4@exmbx03-cdc.nexus.csiro.au> Favas, Mark (CSIRO IM&T, Kensington) would like to recall the message, "cpython: Rename contextlib.ignored() to contextlib.ignore().". From Mark.Favas at csiro.au Sun Oct 20 09:18:36 2013 From: Mark.Favas at csiro.au (Mark.Favas at csiro.au) Date: Sun, 20 Oct 2013 07:18:36 +0000 Subject: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). Message-ID: <03E596E5860CB34DA4825777BB5FF6776BB319E4@exmbx03-cdc.nexus.csiro.au> Or perhaps call it "contextlib.ignore_first(Exception)"? Mark Favas iVEC at CSIRO Director A/Director, Technical Operations, iVEC Advanced Scientific Computing CSIRO IM&T -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Sun Oct 20 17:54:09 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sun, 20 Oct 2013 09:54:09 -0600 Subject: [Python-Dev] PEP 451: ModuleSpec In-Reply-To: References: Message-ID: On Fri, Oct 11, 2013 at 1:37 PM, Eric Snow wrote: > Any comments? Usually silence implies no disapproval. ;) PEP 451 did > go through several rounds of review on import-sig, so I'm not going to > stress over low feedback at this point. However, I'd particularly > appreciate knowing if there are any objections to ModuleSpec for 3.4. > Otherwise I'd like to get pronouncement on the PEP. Thanks! Okay, no objections and the PEP has not changed. I'd like pronouncement on PEP 451. Guido, if you feel so inclined as to delegate and either of them is willing, Brett and Nick are both familiar with the PEP, having been heavily involved in earlier discussions. Thanks! -eric From ericsnowcurrently at gmail.com Sun Oct 20 18:21:08 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sun, 20 Oct 2013 10:21:08 -0600 Subject: [Python-Dev] A C implementation of OrderedDict Message-ID: If anyone is interested in having a (faithful) C implementation of OrderedDict for the 3.4 release, I have a patch up [1]. My interest in having the patch applied is relative to proposals that won't apply to 3.4, so I haven't felt the need to advance the patch. However, in case anyone else would find it useful for 3.4, I figured I would point it out. While the patch isn't all that complicated, it is large and my C/C-API experience isn't proportional to that size. So I don't feel comfortable about moving ahead with the patch without at least one thorough review. Thanks. -eric [1] http://bugs.python.org/issue16991 From guido at python.org Sun Oct 20 18:57:23 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 20 Oct 2013 09:57:23 -0700 Subject: [Python-Dev] PEP 451: ModuleSpec In-Reply-To: References: Message-ID: How about if Brett and Nick both agree that it's good, I approve it? If either one wants more discussion I'll appoint them the PEP-BDFL. On Sun, Oct 20, 2013 at 8:54 AM, Eric Snow wrote: > On Fri, Oct 11, 2013 at 1:37 PM, Eric Snow wrote: >> Any comments? Usually silence implies no disapproval. ;) PEP 451 did >> go through several rounds of review on import-sig, so I'm not going to >> stress over low feedback at this point. However, I'd particularly >> appreciate knowing if there are any objections to ModuleSpec for 3.4. >> Otherwise I'd like to get pronouncement on the PEP. Thanks! > > Okay, no objections and the PEP has not changed. I'd like > pronouncement on PEP 451. > > Guido, if you feel so inclined as to delegate and either of them is > willing, Brett and Nick are both familiar with the PEP, having been > heavily involved in earlier discussions. Thanks! > > -eric -- --Guido van Rossum (python.org/~guido) From larry at hastings.org Mon Oct 21 00:52:57 2013 From: larry at hastings.org (Larry Hastings) Date: Sun, 20 Oct 2013 15:52:57 -0700 Subject: [Python-Dev] Python 3.4.0a4: Coming In For A Bumpy Landing Message-ID: <52645EC9.20906@hastings.org> 3.4.0a4 is tagged and I'm in the process of releasing it. But it's going to be, let's say, more "alpha-quality" than the previous alphas. Known problems: * There's a reference count leak in the compiler. * asyncio test suite sometimes times out, which takes... an hour. * asyncio test suit fails on a couple of platforms. I don't think any of these are bad enough to slip the schedule. However, if you disagree, pipe up, feel free to try and change my mind, it's not too late. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Mon Oct 21 00:57:10 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 20 Oct 2013 18:57:10 -0400 Subject: [Python-Dev] Python 3.4.0a4: Coming In For A Bumpy Landing In-Reply-To: <52645EC9.20906@hastings.org> References: <52645EC9.20906@hastings.org> Message-ID: 2013/10/20 Larry Hastings : > > > 3.4.0a4 is tagged and I'm in the process of releasing it. But it's going to > be, let's say, more "alpha-quality" than the previous alphas. > > Known problems: > > There's a reference count leak in the compiler. This is fixed, so you could just move the tag up a bit. -- Regards, Benjamin From nad at acm.org Mon Oct 21 01:10:14 2013 From: nad at acm.org (Ned Deily) Date: Sun, 20 Oct 2013 16:10:14 -0700 Subject: [Python-Dev] [python-committers] Python 3.4.0a4: Coming In For A Bumpy Landing In-Reply-To: References: <52645EC9.20906@hastings.org> Message-ID: On Oct 20, 2013, at 15:57 , Benjamin Peterson wrote: > 2013/10/20 Larry Hastings : >> >> >> 3.4.0a4 is tagged and I'm in the process of releasing it. But it's going to >> be, let's say, more "alpha-quality" than the previous alphas. >> >> Known problems: >> >> There's a reference count leak in the compiler. > > This is fixed, so you could just move the tag up a bit. Unfortunately, the Windows and OS X installers have already been manufactured so we would either have to redo them or document that there is a discrepancy between the source and binary releases. Probably not a big deal in any case for an alpha release. -- Ned Deily nad at acm.org -- [] From benjamin at python.org Mon Oct 21 01:15:40 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 20 Oct 2013 19:15:40 -0400 Subject: [Python-Dev] [python-committers] Python 3.4.0a4: Coming In For A Bumpy Landing In-Reply-To: References: <52645EC9.20906@hastings.org> Message-ID: 2013/10/20 Ned Deily : > > On Oct 20, 2013, at 15:57 , Benjamin Peterson wrote: > >> 2013/10/20 Larry Hastings : >>> >>> >>> 3.4.0a4 is tagged and I'm in the process of releasing it. But it's going to >>> be, let's say, more "alpha-quality" than the previous alphas. >>> >>> Known problems: >>> >>> There's a reference count leak in the compiler. >> >> This is fixed, so you could just move the tag up a bit. > > > Unfortunately, the Windows and OS X installers have already been manufactured so we would either have to redo them or document that there is a discrepancy between the source and binary releases. Probably not a big deal in any case for an alpha release. Ah, I didn't realize the release process had gotten that far. Carry on. -- Regards, Benjamin From hasan.diwan at gmail.com Mon Oct 21 01:34:30 2013 From: hasan.diwan at gmail.com (Hasan Diwan) Date: Sun, 20 Oct 2013 16:34:30 -0700 Subject: [Python-Dev] Logging Module format Message-ID: I've been using the logging module recently and noticed the default format doesn't timestamp log entries. I've not figured out how to change the format after initialization. This is python 2.7, on Mac OS X. Help, anyone? -- H -- Sent from my mobile device Envoy? de mon portable -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Mon Oct 21 02:04:16 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 20 Oct 2013 20:04:16 -0400 Subject: [Python-Dev] Logging Module format In-Reply-To: References: Message-ID: <20131021000417.621F32501BB@webabinitio.net> On Sun, 20 Oct 2013 16:34:30 -0700, Hasan Diwan wrote: > I've been using the logging module recently and noticed the default format > doesn't timestamp log entries. I've not figured out how to change the > format after initialization. This is python 2.7, on Mac OS X. Help, anyone? This mailing list is for developing Python itself. For help on using python, please post to python-list. There are a lot of smart people hanging out there who are happy to help. --David From larry at hastings.org Mon Oct 21 02:24:57 2013 From: larry at hastings.org (Larry Hastings) Date: Sun, 20 Oct 2013 17:24:57 -0700 Subject: [Python-Dev] [RELEASED] Python 3.4.0a4 Message-ID: <52647459.5000205@hastings.org> On behalf of the Python development team, I'm very pleased to announce the fourth and final alpha release of Python 3.4. This is a preview release, and its use is not recommended for production settings. Python 3.4 includes a range of improvements of the 3.x series, including hundreds of small improvements and bug fixes. Major new features and changes in the 3.4 release series so far include: * PEP 435, a standardized "enum" module * PEP 436, a build enhancement that will help generate introspection information for builtins * PEP 442, improved semantics for object finalization * PEP 443, adding single-dispatch generic functions to the standard library * PEP 445, a new C API for implementing custom memory allocators * PEP 446, changing file descriptors to not be inherited by default in subprocesses * PEP 450, the new "statistics" module * PEP 3156, the new "asyncio" module, a new framework for asynchronous I/O To download Python 3.4.0a4 visit: http://www.python.org/download/releases/3.4.0/ Python 3.4.0a4 has several known issues: * The Python compiler has a small memory leak. * The "asyncio" module test suite fails on some platforms. * I/O conducted by the "asyncio" module may, rarely, erroneously time out. The timeout takes one hour. Please consider trying Python 3.4.0a4 with your code and reporting any new issues you notice to: http://bugs.python.org/ Enjoy! -- Larry Hastings, Release Manager larry at hastings.org (on behalf of the entire python-dev team and 3.4's contributors) From raymond.hettinger at gmail.com Mon Oct 21 04:42:30 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 20 Oct 2013 19:42:30 -0700 Subject: [Python-Dev] Coding practice for context managers Message-ID: Two of the new context managers in contextlib are now wrapped in pass-through factory functions. The intent is to make the help() look cleaner. This practice does have downsides however. The usual way to detect whether something is usable with a with-statement is to check the presence of the __enter__ and __exit__ methods. Wrapping the CM in a pass-through function defeats this and other forms of introspection. Also, the help() output itself is worse-off. When you run help on a CM(), you're trying to find-out what happens on entry and what happens on exit. If those methods had docstrings, the question would be answered directly. The wrapper (intentionally) hides how it works. Since I teach intermediate and advanced python classes to experienced Python users, I've become more sensitive to problems this practice will create. Defeating introspection can make the help look nicer, but it isn't a clean coding practice and is something I hope doesn't catch on. To the extent there is a problem with the output of help(), I think efforts should be directed at making help() better. A lot of work needs to be done on that end -- for example abstract base classes also don't look great in help(). There are a couple of other minor issues as well. One is that the wrapper function hides the class, making it harder to do type checks such as "isinstance(x, suppress)". The other issue is that wrappers cause extra jumping around for people who are tracing code through a debugger or using a visualization tool such as pythontutor. These aren't terribly important issues, but it does support the notion that usually the cleanest code is the best code. In short, I recommend that efforts be directed at improving help() rather than limiting introspection by way of less clean coding practices. Raymond -------- current code for suppress() -------- class _SuppressExceptions: """Helper for suppress.""" def __init__(self, *exceptions): self._exceptions = exceptions def __enter__(self): pass def __exit__(self, exctype, excinst, exctb): return exctype is not None and issubclass(exctype, self._exceptions) # Use a wrapper function since we don't care about supporting inheritance # and a function gives much cleaner output in help() def suppress(*exceptions): """Context manager to suppress specified exceptions After the exception is suppressed, execution proceeds with the next statement following the with statement. with suppress(FileNotFoundError): os.remove(somefile) # Execution still resumes here if the file was already removed """ return _SuppressExceptions(*exceptions) -------- current help() output for suppress() -------- Help on function suppress in module contextlib: suppress(*exceptions) Context manager to suppress specified exceptions After the exception is suppressed, execution proceeds with the next statement following the with statement. with suppress(FileNotFoundError): os.remove(somefile) # Execution still resumes here if the file was already removed -------- current help() output for closing() with does not have a function wrapper -------- Help on class closing in module contextlib: class closing(builtins.object) | Context to automatically close something at the end of a block. | | Code like this: | | with closing(.open()) as f: | | | is equivalent to this: | | f = .open() | try: | | finally: | f.close() | | Methods defined here: | | __enter__(self) | | __exit__(self, *exc_info) | | __init__(self, thing) | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) From ethan at stoneleaf.us Mon Oct 21 04:49:24 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 20 Oct 2013 19:49:24 -0700 Subject: [Python-Dev] Coding practice for context managers In-Reply-To: References: Message-ID: <52649634.7070501@stoneleaf.us> On 10/20/2013 07:42 PM, Raymond Hettinger wrote: > > In short, I recommend that efforts be directed at improving help() rather than limiting introspection by way of less clean coding practices. +1 We also have the option of adding a custom __dir__ to change what help() displays. -- ~Ethan~ From raymond.hettinger at gmail.com Mon Oct 21 05:56:31 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 20 Oct 2013 20:56:31 -0700 Subject: [Python-Dev] A C implementation of OrderedDict In-Reply-To: References: Message-ID: <166A4B96-3490-458C-A86E-3F2F3E0CCBB0@gmail.com> On Oct 20, 2013, at 9:21 AM, Eric Snow wrote: > If anyone is interested in having a (faithful) C implementation of > OrderedDict for the 3.4 release, I have a patch up [1]. My interest > in having the patch applied is relative to proposals that won't apply > to 3.4, so I haven't felt the need to advance the patch. However, in > case anyone else would find it useful for 3.4, I figured I would point > it out. > > While the patch isn't all that complicated, it is large and my C/C-API > experience isn't proportional to that size. So I don't feel > comfortable about moving ahead with the patch without at least one > thorough review. Thanks. I'll look at this in more detail after I've finishing my review of the TransformDict, but my initial impression is that the original show stopper hasn't been overcome: http://bugs.python.org/issue10977 The concrete dict API is very old and is widely used in C extensions outside the standard library. AFAICT, there is no way to prevent that code from bypassing your code and breaking the internal invariants of the ordered dict (that breakage could be silent are muck up the ordering or could fail loudly with a segfault). If we really want a C implementation, I think the choices boil down to: 1) living with the problem and writing defensive code so that the ordered dictionary won't segfault when fed to existing external C code and so that the ordered dict notices whether the parent dictionary has different keys than those in the internal linked list. or 2) biting the bullet and accepting an API change where ordered dicts are no longer a subclass of real dicts. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Mon Oct 21 08:00:43 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 21 Oct 2013 00:00:43 -0600 Subject: [Python-Dev] A C implementation of OrderedDict In-Reply-To: <166A4B96-3490-458C-A86E-3F2F3E0CCBB0@gmail.com> References: <166A4B96-3490-458C-A86E-3F2F3E0CCBB0@gmail.com> Message-ID: On Sun, Oct 20, 2013 at 9:56 PM, Raymond Hettinger wrote: > I'll look at this in more detail after I've finishing my review of the > TransformDict, > but my initial impression is that the original show stopper hasn't been > overcome: > http://bugs.python.org/issue10977 > > The concrete dict API is very old and is widely used in C extensions > outside the standard library. AFAICT, there is no way to prevent that code > from bypassing your code and breaking the internal invariants of the > ordered dict (that breakage could be silent are muck up the ordering > or could fail loudly with a segfault). Just to be clear, the patch I have up is meant to provide a nearly exact duplicate of OrderedDict in C, minus the "private" implementation details. The current OrderedDict already suffers from the same problems you described, since it is also a dict subclass (as you know :-). Those would definitely become show stoppers, as you said, but only once we tried to actually use OrderedDict in the interpreter in places where we currently use dict (which is something we should mostly avoid). Are you otherwise concerned that a C implementation would send the wrong message to extension module authors or cause problems for C extension modules that use OrderedDict, beyond any they face with the pure Python implementation? Regardless, I concur that we would need to find a solution to the existing concrete API situation if OrderedDict were to be used in the intepreter as a drop-in replacement for current dicts. The same would apply for any case of subclassing one of the fundamental types (like str, which came up in a recent issue related to the email module). In the meantime, however, a C implementation of OrderedDict would offer a performance improvement for anyone that uses it. Furthermore, for the couple of proposals I have cooking that make use of OrderedDict, the concrete API problem does not factor in all that much, but having a builtin C implementation does. I will admit that things would be easier for one of ideas if I could just drop OrderedDict in, but that's a rats nest I stared at at one point and then backed away slowly. Ultimately, I don't feel any urgency WRT a C implementation of OrderedDict. I simply wanted to make others aware of the possibility in case it would be useful to them in 3.4. Otherwise I am perfectly comfortable with taking any time you feel is necessary. > > If we really want a C implementation, I think the choices boil down to: > > 1) living with the problem and writing defensive code so that the > ordered dictionary won't segfault when fed to existing external C code > and so that the ordered dict notices whether the parent dictionary has > different keys than those in the internal linked list. no thanks! Dealing with re-entrancy was quite enough. :) > > or 2) biting the bullet and accepting an API change where ordered dicts > are no longer a subclass of real dicts. I know a few people that would argue for this regardless. -eric From ncoghlan at gmail.com Mon Oct 21 12:46:39 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 21 Oct 2013 20:46:39 +1000 Subject: [Python-Dev] Coding practice for context managers In-Reply-To: References: Message-ID: On 21 Oct 2013 12:44, "Raymond Hettinger" wrote: > > Two of the new context managers in contextlib are now wrapped in pass-through factory functions. The intent is to make the help() look cleaner. This practice does have downsides however. > > The usual way to detect whether something is usable with a with-statement is to check the presence of the __enter__ and __exit__ methods. Wrapping the CM in a pass-through function defeats this and other forms of introspection. > > Also, the help() output itself is worse-off. When you run help on a CM(), you're trying to find-out what happens on entry and what happens on exit. If those methods had docstrings, the question would be answered directly. The wrapper (intentionally) hides how it works. > > Since I teach intermediate and advanced python classes to experienced Python users, I've become more sensitive to problems this practice will create. Defeating introspection can make the help look nicer, but it isn't a clean coding practice and is something I hope doesn't catch on. > > To the extent there is a problem with the output of help(), I think efforts should be directed at making help() better. A lot of work needs to be done on that end -- for example abstract base classes also don't look great in help(). > > There are a couple of other minor issues as well. One is that the wrapper function hides the class, making it harder to do type checks such as "isinstance(x, suppress)". The other issue is that wrappers cause extra jumping around for people who are tracing code through a debugger or using a visualization tool such as pythontutor. These aren't terribly important issues, but it does support the notion that usually the cleanest code is the best code. > > In short, I recommend that efforts be directed at improving help() rather than limiting introspection by way of less clean coding practices. That's a fair point, but I *really* dislike exposing implementations that don't match their documentation, and both of these are currently documented as factory functions. There's also the fact that I prefer the current lower case names, but strongly dislike using lower case names for classes (despite the fact closing was included in the original contextlib with a non PEP 8 compliant class name). (And yes, I now realise I completely failed to mention either of those points in the comments giving the rationale for the wrapper functions). Exposing the full object oriented API is certainly a reasonable option, but the docs should be updated accordingly, with suitable public attributes being defined (providing access to the exception tuple for suppress and the target stream for redirect_stdio). Cheers, Nick. > > > Raymond > > > -------- current code for suppress() -------- > > class _SuppressExceptions: > """Helper for suppress.""" > def __init__(self, *exceptions): > self._exceptions = exceptions > > def __enter__(self): > pass > > def __exit__(self, exctype, excinst, exctb): > return exctype is not None and issubclass(exctype, self._exceptions) > > # Use a wrapper function since we don't care about supporting inheritance > # and a function gives much cleaner output in help() > def suppress(*exceptions): > """Context manager to suppress specified exceptions > > After the exception is suppressed, execution proceeds with the next > statement following the with statement. > > with suppress(FileNotFoundError): > os.remove(somefile) > # Execution still resumes here if the file was already removed > """ > return _SuppressExceptions(*exceptions) > > > -------- current help() output for suppress() -------- > > Help on function suppress in module contextlib: > > suppress(*exceptions) > Context manager to suppress specified exceptions > > After the exception is suppressed, execution proceeds with the next > statement following the with statement. > > with suppress(FileNotFoundError): > os.remove(somefile) > # Execution still resumes here if the file was already removed > > -------- current help() output for closing() with does not have a function wrapper -------- > > Help on class closing in module contextlib: > > class closing(builtins.object) > | Context to automatically close something at the end of a block. > | > | Code like this: > | > | with closing(.open()) as f: > | > | > | is equivalent to this: > | > | f = .open() > | try: > | > | finally: > | f.close() > | > | Methods defined here: > | > | __enter__(self) > | > | __exit__(self, *exc_info) > | > | __init__(self, thing) > | > | ---------------------------------------------------------------------- > | Data descriptors defined here: > | > | __dict__ > | dictionary for instance variables (if defined) > | > | __weakref__ > | list of weak references to the object (if defined) > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Mon Oct 21 12:59:27 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 21 Oct 2013 06:59:27 -0400 Subject: [Python-Dev] Coding practice for context managers In-Reply-To: <52649634.7070501@stoneleaf.us> References: <52649634.7070501@stoneleaf.us> Message-ID: <20131021105927.DAC7B250161@webabinitio.net> On Sun, 20 Oct 2013 19:49:24 -0700, Ethan Furman wrote: > On 10/20/2013 07:42 PM, Raymond Hettinger wrote: > > > > In short, I recommend that efforts be directed at improving help() rather than limiting introspection by way of less clean coding practices. > > +1 I'm also +1 on improving help instead of using wrapper classes. > We also have the option of adding a custom __dir__ to change what help() displays. I think you'd want to be careful about __dir__ also interfering with introspection if it is used *just* to make the help cleaner. --David From p.f.moore at gmail.com Mon Oct 21 13:11:57 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 21 Oct 2013 12:11:57 +0100 Subject: [Python-Dev] Coding practice for context managers In-Reply-To: <20131021105927.DAC7B250161@webabinitio.net> References: <52649634.7070501@stoneleaf.us> <20131021105927.DAC7B250161@webabinitio.net> Message-ID: On 21 October 2013 11:59, R. David Murray wrote: > On Sun, 20 Oct 2013 19:49:24 -0700, Ethan Furman wrote: >> On 10/20/2013 07:42 PM, Raymond Hettinger wrote: >> > >> > In short, I recommend that efforts be directed at improving help() rather than limiting introspection by way of less clean coding practices. >> >> +1 > > I'm also +1 on improving help instead of using wrapper classes. Agreed. A wrapper function whose purpose is solely to tidy up help seems like a bad idea in general. I'm somewhat more sympathetic to Nick's point that the name the user types should be all-lowercase and a class would be mixed case, but on that I think it's actually the naming convention that should change (name classes/functions based on usage, not implementation). The rule to me is that changing the underlying implementation shouldn't affect the user interface. Paul From rdmurray at bitdance.com Mon Oct 21 13:21:11 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 21 Oct 2013 07:21:11 -0400 Subject: [Python-Dev] Coding practice for context managers In-Reply-To: References: <52649634.7070501@stoneleaf.us> <20131021105927.DAC7B250161@webabinitio.net> Message-ID: <20131021112112.6087D250814@webabinitio.net> On Mon, 21 Oct 2013 12:11:57 +0100, Paul Moore wrote: > On 21 October 2013 11:59, R. David Murray wrote: > > On Sun, 20 Oct 2013 19:49:24 -0700, Ethan Furman wrote: > >> On 10/20/2013 07:42 PM, Raymond Hettinger wrote: > >> > > >> > In short, I recommend that efforts be directed at improving help() rather than limiting introspection by way of less clean coding practices. > >> > >> +1 > > > > I'm also +1 on improving help instead of using wrapper classes. > > Agreed. A wrapper function whose purpose is solely to tidy up help > seems like a bad idea in general. > > I'm somewhat more sympathetic to Nick's point that the name the user > types should be all-lowercase and a class would be mixed case, but on > that I think it's actually the naming convention that should change > (name classes/functions based on usage, not implementation). The rule > to me is that changing the underlying implementation shouldn't affect > the user interface. +1. I've run into this tension between the naming convention and wanting to change the underlying API before, and I think the naming convention gets in the way. --David From rdmurray at bitdance.com Mon Oct 21 13:26:50 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 21 Oct 2013 07:26:50 -0400 Subject: [Python-Dev] Coding practice for context managers In-Reply-To: <20131021112112.6087D250814@webabinitio.net> References: <52649634.7070501@stoneleaf.us> <20131021105927.DAC7B250161@webabinitio.net> <20131021112112.6087D250814@webabinitio.net> Message-ID: <20131021112651.1BD4C250814@webabinitio.net> On Mon, 21 Oct 2013 07:21:11 -0400, "R. David Murray" wrote: > On Mon, 21 Oct 2013 12:11:57 +0100, Paul Moore wrote: > > On 21 October 2013 11:59, R. David Murray wrote: > > > On Sun, 20 Oct 2013 19:49:24 -0700, Ethan Furman wrote: > > >> On 10/20/2013 07:42 PM, Raymond Hettinger wrote: > > >> > > > >> > In short, I recommend that efforts be directed at improving help() rather than limiting introspection by way of less clean coding practices. > > >> > > >> +1 > > > > > > I'm also +1 on improving help instead of using wrapper classes. > > > > Agreed. A wrapper function whose purpose is solely to tidy up help > > seems like a bad idea in general. > > > > I'm somewhat more sympathetic to Nick's point that the name the user > > types should be all-lowercase and a class would be mixed case, but on > > that I think it's actually the naming convention that should change > > (name classes/functions based on usage, not implementation). The rule > > to me is that changing the underlying implementation shouldn't affect > > the user interface. > > +1. I've run into this tension between the naming convention and > wanting to change the underlying API before, and I think the > naming convention gets in the way. Grr. The underlying *implementation*. This specifically came up in the case of whether a factory function was a function or a class. A class is a perfectly good factory function, but API-wise is often seems more natural to document it and name it as a function. --David From ncoghlan at gmail.com Mon Oct 21 13:34:43 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 21 Oct 2013 21:34:43 +1000 Subject: [Python-Dev] Coding practice for context managers In-Reply-To: <20131021112112.6087D250814@webabinitio.net> References: <52649634.7070501@stoneleaf.us> <20131021105927.DAC7B250161@webabinitio.net> <20131021112112.6087D250814@webabinitio.net> Message-ID: On 21 Oct 2013 21:22, "R. David Murray" wrote: > > On Mon, 21 Oct 2013 12:11:57 +0100, Paul Moore wrote: > > On 21 October 2013 11:59, R. David Murray wrote: > > > On Sun, 20 Oct 2013 19:49:24 -0700, Ethan Furman wrote: > > >> On 10/20/2013 07:42 PM, Raymond Hettinger wrote: > > >> > > > >> > In short, I recommend that efforts be directed at improving help() rather than limiting introspection by way of less clean coding practices. > > >> > > >> +1 > > > > > > I'm also +1 on improving help instead of using wrapper classes. > > > > Agreed. A wrapper function whose purpose is solely to tidy up help > > seems like a bad idea in general. > > > > I'm somewhat more sympathetic to Nick's point that the name the user > > types should be all-lowercase and a class would be mixed case, but on > > that I think it's actually the naming convention that should change > > (name classes/functions based on usage, not implementation). The rule > > to me is that changing the underlying implementation shouldn't affect > > the user interface. > > +1. I've run into this tension between the naming convention and > wanting to change the underlying API before, and I think the > naming convention gets in the way. Right, I think we may want to reconsider that guideline, because I just realised it helped nudge me into introducing an outright bug: using wrapper functions as I did means help() on instances will pick up the private helper class which is definitely *not* what we want. (Perhaps we could relax the rule to require initial capitals for abstract types, but allow it to be skipped for concrete classes? That would better match what we actually do in the builtins and stdlib). Regardless, lesson learned, I'll create an issue and put a patch on the tracker to create a public OO API for these as described in my previous message (but using the current names). Cheers, Nick. > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Oct 21 14:08:39 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 21 Oct 2013 14:08:39 +0200 Subject: [Python-Dev] Coding practice for context managers References: Message-ID: <20131021140839.0456e731@pitrou.net> Le Mon, 21 Oct 2013 20:46:39 +1000, Nick Coghlan a ?crit : > On 21 Oct 2013 12:44, "Raymond Hettinger" > wrote: > > > > Two of the new context managers in contextlib are now wrapped in > pass-through factory functions. The intent is to make the help() look > cleaner. This practice does have downsides however. > > > > The usual way to detect whether something is usable with a > > with-statement > is to check the presence of the __enter__ and __exit__ methods. > Wrapping the CM in a pass-through function defeats this and other > forms of introspection. > > > > Also, the help() output itself is worse-off. When you run help on a > CM(), you're trying to find-out what happens on entry and what > happens on exit. If those methods had docstrings, the question would > be answered directly. The wrapper (intentionally) hides how it > works. > > > > Since I teach intermediate and advanced python classes to > > experienced > Python users, I've become more sensitive to problems this practice > will create. Defeating introspection can make the help look nicer, > but it isn't a clean coding practice and is something I hope doesn't > catch on. > > > > To the extent there is a problem with the output of help(), I think > efforts should be directed at making help() better. A lot of work > needs to be done on that end -- for example abstract base classes > also don't look great in help(). > > > > There are a couple of other minor issues as well. One is that the > wrapper function hides the class, making it harder to do type checks > such as "isinstance(x, suppress)". The other issue is that wrappers > cause extra jumping around for people who are tracing code through a > debugger or using a visualization tool such as pythontutor. These > aren't terribly important issues, but it does support the notion that > usually the cleanest code is the best code. > > > > In short, I recommend that efforts be directed at improving help() > > rather > than limiting introspection by way of less clean coding practices. > > That's a fair point, but I *really* dislike exposing implementations > that don't match their documentation, and both of these are currently > documented as factory functions. Let's call them callables instead? > Exposing the full object oriented API is certainly a reasonable > option, but the docs should be updated accordingly, with suitable > public attributes being defined (providing access to the exception > tuple for suppress and the target stream for redirect_stdio). I don't get why adding public attributes should be related to the proposed change. Regards Antoine. From ncoghlan at gmail.com Mon Oct 21 15:12:40 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 21 Oct 2013 23:12:40 +1000 Subject: [Python-Dev] Coding practice for context managers In-Reply-To: <20131021140839.0456e731@pitrou.net> References: <20131021140839.0456e731@pitrou.net> Message-ID: On 21 Oct 2013 22:10, "Antoine Pitrou" wrote: > > Le Mon, 21 Oct 2013 20:46:39 +1000, > Nick Coghlan a ?crit : > > On 21 Oct 2013 12:44, "Raymond Hettinger" > > wrote: > > > > > > Two of the new context managers in contextlib are now wrapped in > > pass-through factory functions. The intent is to make the help() look > > cleaner. This practice does have downsides however. > > > > > > The usual way to detect whether something is usable with a > > > with-statement > > is to check the presence of the __enter__ and __exit__ methods. > > Wrapping the CM in a pass-through function defeats this and other > > forms of introspection. > > > > > > Also, the help() output itself is worse-off. When you run help on a > > CM(), you're trying to find-out what happens on entry and what > > happens on exit. If those methods had docstrings, the question would > > be answered directly. The wrapper (intentionally) hides how it > > works. > > > > > > Since I teach intermediate and advanced python classes to > > > experienced > > Python users, I've become more sensitive to problems this practice > > will create. Defeating introspection can make the help look nicer, > > but it isn't a clean coding practice and is something I hope doesn't > > catch on. > > > > > > To the extent there is a problem with the output of help(), I think > > efforts should be directed at making help() better. A lot of work > > needs to be done on that end -- for example abstract base classes > > also don't look great in help(). > > > > > > There are a couple of other minor issues as well. One is that the > > wrapper function hides the class, making it harder to do type checks > > such as "isinstance(x, suppress)". The other issue is that wrappers > > cause extra jumping around for people who are tracing code through a > > debugger or using a visualization tool such as pythontutor. These > > aren't terribly important issues, but it does support the notion that > > usually the cleanest code is the best code. > > > > > > In short, I recommend that efforts be directed at improving help() > > > rather > > than limiting introspection by way of less clean coding practices. > > > > That's a fair point, but I *really* dislike exposing implementations > > that don't match their documentation, and both of these are currently > > documented as factory functions. > > Let's call them callables instead? > > > Exposing the full object oriented API is certainly a reasonable > > option, but the docs should be updated accordingly, with suitable > > public attributes being defined (providing access to the exception > > tuple for suppress and the target stream for redirect_stdio). > > I don't get why adding public attributes should be related to the > proposed change. Because redirect_stdout previously had undocumented attributes without an underscore prefix, and it was after I had already renamed those and was deciding whether or not to make suppress.exceptions public (and return self from suppress.__enter__) that I had my ill-advised brainwave about postponing dealing with all those questions by hiding the implementation of both of them behind factory functions. Postponing the questions didn't work, so I may as well address them instead. Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Oct 21 15:52:21 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 21 Oct 2013 15:52:21 +0200 Subject: [Python-Dev] Coding practice for context managers References: <20131021140839.0456e731@pitrou.net> Message-ID: <20131021155221.307558e5@pitrou.net> Le Mon, 21 Oct 2013 23:12:40 +1000, Nick Coghlan a ?crit : > On 21 Oct 2013 22:10, "Antoine Pitrou" wrote: > > > > Le Mon, 21 Oct 2013 20:46:39 +1000, > > Nick Coghlan a ?crit : > > > On 21 Oct 2013 12:44, "Raymond Hettinger" > > > wrote: > > > > > > > > Two of the new context managers in contextlib are now wrapped in > > > pass-through factory functions. The intent is to make the help() > > > look cleaner. This practice does have downsides however. > > > > > > > > The usual way to detect whether something is usable with a > > > > with-statement > > > is to check the presence of the __enter__ and __exit__ methods. > > > Wrapping the CM in a pass-through function defeats this and other > > > forms of introspection. > > > > > > > > Also, the help() output itself is worse-off. When you run help > > > > on a > > > CM(), you're trying to find-out what happens on entry and what > > > happens on exit. If those methods had docstrings, the question > > > would be answered directly. The wrapper (intentionally) hides > > > how it works. > > > > > > > > Since I teach intermediate and advanced python classes to > > > > experienced > > > Python users, I've become more sensitive to problems this practice > > > will create. Defeating introspection can make the help look > > > nicer, but it isn't a clean coding practice and is something I > > > hope doesn't catch on. > > > > > > > > To the extent there is a problem with the output of help(), I > > > > think > > > efforts should be directed at making help() better. A lot of > > > work needs to be done on that end -- for example abstract base > > > classes also don't look great in help(). > > > > > > > > There are a couple of other minor issues as well. One is that > > > > the > > > wrapper function hides the class, making it harder to do type > > > checks such as "isinstance(x, suppress)". The other issue is > > > that wrappers cause extra jumping around for people who are > > > tracing code through a debugger or using a visualization tool > > > such as pythontutor. These aren't terribly important issues, > > > but it does support the notion that usually the cleanest code is > > > the best code. > > > > > > > > In short, I recommend that efforts be directed at improving > > > > help() rather > > > than limiting introspection by way of less clean coding practices. > > > > > > That's a fair point, but I *really* dislike exposing > > > implementations that don't match their documentation, and both of > > > these are currently documented as factory functions. > > > > Let's call them callables instead? > > > > > Exposing the full object oriented API is certainly a reasonable > > > option, but the docs should be updated accordingly, with suitable > > > public attributes being defined (providing access to the exception > > > tuple for suppress and the target stream for redirect_stdio). > > > > I don't get why adding public attributes should be related to the > > proposed change. > > Because redirect_stdout previously had undocumented attributes > without an underscore prefix, and it was after I had already renamed > those and was deciding whether or not to make suppress.exceptions > public (and return self from suppress.__enter__) that I had my > ill-advised brainwave about postponing dealing with all those > questions by hiding the implementation of both of them behind factory > functions. So how about simply adding an underscore to all those attributes? Regards Antoine. From barry at python.org Mon Oct 21 13:14:37 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 21 Oct 2013 07:14:37 -0400 Subject: [Python-Dev] Coding practice for context managers In-Reply-To: References: Message-ID: <20131021071437.155a542e@anarchist> On Oct 21, 2013, at 08:46 PM, Nick Coghlan wrote: >There's also the fact that I prefer the current lower case names, but >strongly dislike using lower case names for classes (despite the fact >closing was included in the original contextlib with a non PEP 8 compliant >class name). Ha! The only thing I don't much like about ExitStack is its capword name. ;) I don't have a problem breaking PEP 8 rules for cases like this. -Barry From barry at python.org Mon Oct 21 13:24:07 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 21 Oct 2013 07:24:07 -0400 Subject: [Python-Dev] Coding practice for context managers In-Reply-To: References: Message-ID: <20131021072407.19f60045@anarchist> On Oct 21, 2013, at 08:46 PM, Nick Coghlan wrote: >There's also the fact that I prefer the current lower case names, but >strongly dislike using lower case names for classes (despite the fact >closing was included in the original contextlib with a non PEP 8 compliant >class name). I think PEP 8 needs to be slightly revised. If you have suggestions for better wording, please comment here: http://bugs.python.org/issue19331 (But *please* let's not bikeshed this on python-dev.) -Barry From barry at python.org Mon Oct 21 13:46:12 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 21 Oct 2013 07:46:12 -0400 Subject: [Python-Dev] Python 2.6.9 final final due out 28 October 2013 Message-ID: <20131021074612.51e95ec1@anarchist> This is a reminder that Python 2.6.9 final - and by that I mean *really* final as it will be the last supported version of Python 2.6 - is scheduled for release one week from today, on October 28, 2013. All known showstopper security bugs have been applied to the branch. If you know of anything that should go into Python 2.6 but has not yet, please contact me asap. Please test rc1 on your platforms and code, and let me know if you have any problems. Barring any showstoppers, I will likely cut the release AM my local time. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ncoghlan at gmail.com Mon Oct 21 16:37:06 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 22 Oct 2013 00:37:06 +1000 Subject: [Python-Dev] Coding practice for context managers In-Reply-To: <20131021155221.307558e5@pitrou.net> References: <20131021140839.0456e731@pitrou.net> <20131021155221.307558e5@pitrou.net> Message-ID: On 21 Oct 2013 23:54, "Antoine Pitrou" wrote: > > Le Mon, 21 Oct 2013 23:12:40 +1000, > Nick Coghlan a ?crit : > > On 21 Oct 2013 22:10, "Antoine Pitrou" wrote: > > > > > > Le Mon, 21 Oct 2013 20:46:39 +1000, > > > Nick Coghlan a ?crit : > > > > On 21 Oct 2013 12:44, "Raymond Hettinger" > > > > wrote: > > > > > > > > > > Two of the new context managers in contextlib are now wrapped in > > > > pass-through factory functions. The intent is to make the help() > > > > look cleaner. This practice does have downsides however. > > > > > > > > > > The usual way to detect whether something is usable with a > > > > > with-statement > > > > is to check the presence of the __enter__ and __exit__ methods. > > > > Wrapping the CM in a pass-through function defeats this and other > > > > forms of introspection. > > > > > > > > > > Also, the help() output itself is worse-off. When you run help > > > > > on a > > > > CM(), you're trying to find-out what happens on entry and what > > > > happens on exit. If those methods had docstrings, the question > > > > would be answered directly. The wrapper (intentionally) hides > > > > how it works. > > > > > > > > > > Since I teach intermediate and advanced python classes to > > > > > experienced > > > > Python users, I've become more sensitive to problems this practice > > > > will create. Defeating introspection can make the help look > > > > nicer, but it isn't a clean coding practice and is something I > > > > hope doesn't catch on. > > > > > > > > > > To the extent there is a problem with the output of help(), I > > > > > think > > > > efforts should be directed at making help() better. A lot of > > > > work needs to be done on that end -- for example abstract base > > > > classes also don't look great in help(). > > > > > > > > > > There are a couple of other minor issues as well. One is that > > > > > the > > > > wrapper function hides the class, making it harder to do type > > > > checks such as "isinstance(x, suppress)". The other issue is > > > > that wrappers cause extra jumping around for people who are > > > > tracing code through a debugger or using a visualization tool > > > > such as pythontutor. These aren't terribly important issues, > > > > but it does support the notion that usually the cleanest code is > > > > the best code. > > > > > > > > > > In short, I recommend that efforts be directed at improving > > > > > help() rather > > > > than limiting introspection by way of less clean coding practices. > > > > > > > > That's a fair point, but I *really* dislike exposing > > > > implementations that don't match their documentation, and both of > > > > these are currently documented as factory functions. > > > > > > Let's call them callables instead? > > > > > > > Exposing the full object oriented API is certainly a reasonable > > > > option, but the docs should be updated accordingly, with suitable > > > > public attributes being defined (providing access to the exception > > > > tuple for suppress and the target stream for redirect_stdio). > > > > > > I don't get why adding public attributes should be related to the > > > proposed change. > > > > Because redirect_stdout previously had undocumented attributes > > without an underscore prefix, and it was after I had already renamed > > those and was deciding whether or not to make suppress.exceptions > > public (and return self from suppress.__enter__) that I had my > > ill-advised brainwave about postponing dealing with all those > > questions by hiding the implementation of both of them behind factory > > functions. > > So how about simply adding an underscore to all those attributes? Certainly an option, but they both have sensible public attributes to offer. Bug is at http://bugs.python.org/issue19330 I'll probably let it sit for a week or so, though, since I'm apparently still too annoyed about the bikeshedding megathread to think entirely clearly about the module API at this point. Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From drsalists at gmail.com Tue Oct 22 03:21:30 2013 From: drsalists at gmail.com (Dan Stromberg) Date: Mon, 21 Oct 2013 18:21:30 -0700 Subject: [Python-Dev] Hashes on same site as download? Message-ID: I may be missing something, but it seems the Python tarballs and hashes are on the same host, and this is not an entirely good thing for security. The way things are now, an attacker breaks into one host, doctors up a tarball, changes the hashes in the same host, and people download without noticing, even if they verify hashes. If you put the hashes on a different host from the tarballs, the attacker has to break into two machines. In this scenario, the hashes add more strength. ISTR I first learned of this issue from an article by Bruce Schneier, though I don't think it was in the context of Python. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Oct 22 03:45:58 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 21 Oct 2013 21:45:58 -0400 Subject: [Python-Dev] Hashes on same site as download? In-Reply-To: References: Message-ID: <20131021214558.10229ac1@anarchist> On Oct 21, 2013, at 06:21 PM, Dan Stromberg wrote: >I may be missing something, but it seems the Python tarballs and hashes are >on the same host, and this is not an entirely good thing for security. All the tarballs are signed with the GPG keys of the release managers. The hashes are just a quick verification that your download succeeded. For extra confidence, check the signatures. Our keys should be independently verifiable. -Barry From timothy.c.delaney at gmail.com Tue Oct 22 03:47:01 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Tue, 22 Oct 2013 12:47:01 +1100 Subject: [Python-Dev] Hashes on same site as download? In-Reply-To: References: Message-ID: On 22 October 2013 12:21, Dan Stromberg wrote: > > I may be missing something, but it seems the Python tarballs and hashes > are on the same host, and this is not an entirely good thing for security. > > The way things are now, an attacker breaks into one host, doctors up a > tarball, changes the hashes in the same host, and people download without > noticing, even if they verify hashes. > > If you put the hashes on a different host from the tarballs, the attacker > has to break into two machines. In this scenario, the hashes add more > strength. > I'm not a security expert, but I can't see how that gives any more security than the current system (I tried to find whatever article you're talking about, but failed). It doesn't matter if you provide downloads in one place and direct people to get the hashes from elsewhere. An attacker has no need to compromise the server where the hashes are stored - they only need to compromise the server that tells you where to get the downloads and hashes. Then the attacker can simply change the download page to direct you to the malicious downloads, hashes and keys (which they can place on the same server, so everything looks legit). Off the top of my head, one way that would give more security would be to store a hash of the download page itself elsewhere (preferably multiple places) and periodically compare that with the live version. Any changes to the live page would be noticed (eventually) unless the attacker also compromised all those other machines. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From drsalists at gmail.com Tue Oct 22 05:25:38 2013 From: drsalists at gmail.com (Dan Stromberg) Date: Mon, 21 Oct 2013 20:25:38 -0700 Subject: [Python-Dev] Hashes on same site as download? In-Reply-To: References: Message-ID: On Mon, Oct 21, 2013 at 6:47 PM, Tim Delaney wrote: > On 22 October 2013 12:21, Dan Stromberg wrote: > >> >> I may be missing something, but it seems the Python tarballs and hashes >> are on the same host, and this is not an entirely good thing for security. >> > I was missing the gpg signing. That's probably more effective than md5 anyway - at least, I hope we're not using gpg with md5 :) Looking at the download pages in rapid-skim-mode, I saw the hashes and ignored the text describing the use of gpg. FWIW, I'm guessing a lot of people do that. The way things are now, an attacker breaks into one host, doctors up a >> tarball, changes the hashes in the same host, and people download without >> noticing, even if they verify hashes. >> >> If you put the hashes on a different host from the tarballs, the attacker >> has to break into two machines. In this scenario, the hashes add more >> strength. >> > > I'm not a security expert, but I can't see how that gives any more > security than the current system (I tried to find whatever article you're > talking about, but failed). It doesn't matter if you provide downloads in > one place and direct people to get the hashes from elsewhere. An attacker > has no need to compromise the server where the hashes are stored - they > only need to compromise the server that tells you where to get the > downloads and hashes. > I don't see the original article anymore, but I believe it was in a Crypto-gram newsletter several years ago. The closest thing I found tonight was: http://en.wikipedia.org/wiki/MD5#Applications -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Oct 22 05:49:55 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 21 Oct 2013 23:49:55 -0400 Subject: [Python-Dev] #19335 is a codeop, not Idle issue Message-ID: I know that many core devs subscribe to the new tracker issues list. If you skipped over http://bugs.python.org/issue19335 because it was (mis) titled as an Idle issue (as I presume most would), you might want to reconsider as it is actually a code and in particular, a codeop issue. Consider the real interpreter. C:\Programs\Python33>python Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:06:53) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> def a(): ... def b(): ... nonlocal c ... _ It waits to see if one enters a blank line (to which it properly responds SyntaxError), or continues with something that makes the statement syntactically valid. ... c = 1 ... >>> The 'close simulation' in the code module acts differently. C:\Programs\Python33>python -m code Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:06:53) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> def a(): ... def b(): ... nonlocal c File "", line None SyntaxError: no binding for nonlocal 'c' found It prematurely acts as if one had entered a fourth, blank line. InteractiveConsole subclasses InteractiveInterpreter (as does Idle, hence the same behavior). II.runsource compiles with codeop.CommandCompile, which wraps codeop._maybe_compile, which returns None (src incomplete), raises (src complete and bad), or returns code (complete and ok) to be executed. For the above code, it re-raises SyntaxError instead of returning None. The issue needs someone who can find and read the C equivalent used by the real interpreter. Perhaps the latter was adjusted when nonlocal was introduced, while codeop was not. -- Terry Jan Reedy From victor.stinner at gmail.com Tue Oct 22 10:54:03 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 22 Oct 2013 10:54:03 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Switch subprocess stdin to a socketpair, attempting to fix issue #19293 (AIX In-Reply-To: <3d3gQD1pkkz7LmM@mail.python.org> References: <3d3gQD1pkkz7LmM@mail.python.org> Message-ID: Hi, Would it be possible to use os.pipe() on all OSes except AIX? Pipes and socket pairs may have minor differences, but some applications may rely on these minor differences. For example, is the buffer size the same? For example, in test.support, we have two constants: PIPE_MAX_SIZE (4 MB) and SOCK_MAX_SIZE (16 MB). Victor 2013/10/22 guido.van.rossum : > http://hg.python.org/cpython/rev/2a0bda8d283d > changeset: 86557:2a0bda8d283d > user: Guido van Rossum > date: Mon Oct 21 20:37:14 2013 -0700 > summary: > Switch subprocess stdin to a socketpair, attempting to fix issue #19293 (AIX hang). > > files: > Lib/asyncio/unix_events.py | 29 +++++++++- > Lib/test/test_asyncio/test_unix_events.py | 7 ++ > 2 files changed, 32 insertions(+), 4 deletions(-) > > > diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py > --- a/Lib/asyncio/unix_events.py > +++ b/Lib/asyncio/unix_events.py > if stdin == subprocess.PIPE: > self._pipes[STDIN] = None > + # Use a socket pair for stdin, since not all platforms > + # support selecting read events on the write end of a > + # socket (which we use in order to detect closing of the > + # other end). Notably this is needed on AIX, and works > + # just fine on other platforms. > + stdin, stdin_w = self._loop._socketpair() > if stdout == subprocess.PIPE: > self._pipes[STDOUT] = None > if stderr == subprocess.PIPE: From martin at v.loewis.de Tue Oct 22 11:33:04 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 22 Oct 2013 11:33:04 +0200 Subject: [Python-Dev] PEP 453 (ensurepip) updated In-Reply-To: References: Message-ID: <52664650.6010907@v.loewis.de> Am 20.10.13 14:27, schrieb Nick Coghlan: > I have posted the latest version of PEP 453 to python.org. This version looks good to me, and I accept it for inclusion in Python 3.4. I'd like to thank Nick for carefully editing this PEP, and I'd like to cite it as an archetype for a well-written PEP. It's very precise, and it elaborates on rejected proposals and the motivation for rejection. I'd also like to thank Donald for pushing this, and for continued work on the implementation of the PEP. I see that this all took longer than expected (also due to my fault in providing timely reviews). I suggest that some of the stuff that needs to be done still is delegated, so that Donald doesn't have to do all of it. Regards, Martin From ncoghlan at gmail.com Tue Oct 22 13:55:50 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 22 Oct 2013 21:55:50 +1000 Subject: [Python-Dev] PEP 453 (ensurepip) updated In-Reply-To: <52664650.6010907@v.loewis.de> References: <52664650.6010907@v.loewis.de> Message-ID: On 22 October 2013 19:33, "Martin v. L?wis" wrote: > Am 20.10.13 14:27, schrieb Nick Coghlan: >> I have posted the latest version of PEP 453 to python.org. > > This version looks good to me, and I accept it for inclusion in Python 3.4. > > I'd like to thank Nick for carefully editing this PEP, and I'd like to > cite it as an archetype for a well-written PEP. It's very precise, and > it elaborates on rejected proposals and the motivation for rejection. Thank you! I have updated the PEP to record the acceptance, and created a tracking issue for the implementation at http://bugs.python.org/issue19347 > I'd also like to thank Donald for pushing this, and for continued work > on the implementation of the PEP. I see that this all took longer than > expected (also due to my fault in providing timely reviews). I suggest > that some of the stuff that needs to be done still is delegated, so that > Donald doesn't have to do all of it. If we could get assistance with the installer and pyvenv updates after the initial implementation of the module itself is checked in, that would be a huge help. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Tue Oct 22 14:09:36 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 22 Oct 2013 08:09:36 -0400 Subject: [Python-Dev] PEP 453 (ensurepip) updated In-Reply-To: <52664650.6010907@v.loewis.de> References: <52664650.6010907@v.loewis.de> Message-ID: <851B458B-E608-4864-ABBC-367251E85FD7@stufft.io> On Oct 22, 2013, at 5:33 AM, Martin v. L?wis wrote: > Am 20.10.13 14:27, schrieb Nick Coghlan: >> I have posted the latest version of PEP 453 to python.org. > > This version looks good to me, and I accept it for inclusion in Python 3.4. > > I'd like to thank Nick for carefully editing this PEP, and I'd like to > cite it as an archetype for a well-written PEP. It's very precise, and > it elaborates on rejected proposals and the motivation for rejection. > > I'd also like to thank Donald for pushing this, and for continued work > on the implementation of the PEP. I see that this all took longer than > expected (also due to my fault in providing timely reviews). I suggest > that some of the stuff that needs to be done still is delegated, so that > Donald doesn't have to do all of it. > > Regards, > Martin > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Let me echo Nick's thank you! Now to get this implemented :D ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From nad at acm.org Tue Oct 22 14:11:31 2013 From: nad at acm.org (Ned Deily) Date: Tue, 22 Oct 2013 05:11:31 -0700 Subject: [Python-Dev] PEP 453 (ensurepip) updated References: <52664650.6010907@v.loewis.de> Message-ID: In article , Nick Coghlan wrote: > On 22 October 2013 19:33, "Martin v. L?wis" wrote: > > Am 20.10.13 14:27, schrieb Nick Coghlan: > >> I have posted the latest version of PEP 453 to python.org. > > > > This version looks good to me, and I accept it for inclusion in Python 3.4. > > > > I'd like to thank Nick for carefully editing this PEP, and I'd like to > > cite it as an archetype for a well-written PEP. It's very precise, and > > it elaborates on rejected proposals and the motivation for rejection. I'd like to second that and thank both Nick and Donald for addressing the concerns I raised. > Thank you! I have updated the PEP to record the acceptance, and > created a tracking issue for the implementation at > http://bugs.python.org/issue19347 > > > I'd also like to thank Donald for pushing this, and for continued work > > on the implementation of the PEP. I see that this all took longer than > > expected (also due to my fault in providing timely reviews). I suggest > > that some of the stuff that needs to be done still is delegated, so that > > Donald doesn't have to do all of it. > > If we could get assistance with the installer and pyvenv updates after > the initial implementation of the module itself is checked in, that > would be a huge help. I'm planning to do the OS X installer support changes. -- Ned Deily, nad at acm.org From bootblack00 at gmail.com Tue Oct 22 17:08:34 2013 From: bootblack00 at gmail.com (o1dnik) Date: Tue, 22 Oct 2013 15:08:34 +0000 (UTC) Subject: [Python-Dev] compile python 3.3 with bz2 support References: <20121221061745.GB12583@iskra.aviel.ru> Message-ID: Oleg Broytman phdru.name> writes: > > Hello. > > We are sorry but we cannot help you. This mailing list is to work on > developing Python (adding new features to Python itself and fixing bugs); > if you're having problems learning, understanding or using Python, please > find another forum. Probably python-list/comp.lang.python mailing list/news > group is the best place; there are Python developers who participate in it; > you may get a faster, and probably more complete, answer there. See > http://www.python.org/community/ for other lists/news groups/fora. Thank > you for understanding. > > On Fri, Dec 21, 2012 at 02:03:40PM +0800, Isml <76069016 qq.com> wrote: > > hi, everyone: > > I want to compile python 3.3 with bz2 support on RedHat 5.5 but fail to do that. Here is how I do it: > > 1??download bzip2 and compile it(make??make -f Makefile_libbz2_so?? make install) > > 2??chang to python 3.3 source directory : ./configure --with- bz2=/usr/local/include > > 3??make > > 4??make install > > > > after installation complete, I test it?? > > [root localhost Python-3.3.0]# python3 -c "import bz2" > > Traceback (most recent call last): > > File "", line 1, in > > File "/usr/local/lib/python3.3/bz2.py", line 21, in > > from _bz2 import BZ2Compressor, BZ2Decompressor > > ImportError: No module named '_bz2' > > You have to install bz2 development files (headers and libraries) > before recompiling python. > > > By the way, RedHat 5.5 has a built-in python 2.4.3. Would it be a problem? > > Depends on what are you going to do. > > Oleg. I've faced the same problem: while installing packages packed in .tar.bz2 using pip it failed with the same error on importing BZDecompress from _bz2 module (it could not find the `_bz` module). So You can solve the problem by adding a prefix with direct location to your compiled python3.3. But first you need to add a custom path to libbz2 in this line[1357] in setup.py: # Gustavo Niemeyer's bz2 module. if (self.compiler.find_library_file('/lib/i386-linux-gnu/libbz2.so.1.0.4/', 'bz2')): # lib_dirs, 'bz2')): then make && make -f Makefile-libbz2_so && make install (and be careful passing correct --prefix to ./configure) From brett at python.org Tue Oct 22 22:17:02 2013 From: brett at python.org (Brett Cannon) Date: Tue, 22 Oct 2013 16:17:02 -0400 Subject: [Python-Dev] PyCon US 2014 In-Reply-To: <6D03A0D0-BBED-462F-B033-3FB3F492AEE2@voidspace.org.uk> References: <6D03A0D0-BBED-462F-B033-3FB3F492AEE2@voidspace.org.uk> Message-ID: On Fri, Oct 11, 2013 at 11:23 AM, Michael Foord wrote: > > On 10 Oct 2013, at 01:53, Eric Snow wrote: > > > Registration is now open for PyCon US 2014. Are there any plans yet > > for the language summit? Just the day (e.g. Thursday April 10) will > > suffice. Then we can make arrangements appropriately. Thanks. > > > Sorry for the late reply. Yes there will be a language summit. I'm pretty > sure it will be on Thursday as the last few years. I'm getting definite > confirmation of this. > I thought it had been on Wednesday typically? I.e. language summit Wednesday and some random other summit Thursday. -Brett > > Michael > > > > > -eric > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > > -- > http://www.voidspace.org.uk/ > > > May you do good and not evil > May you find forgiveness for yourself and forgive others > May you share freely, never taking more than you give. > -- the sqlite blessing > http://www.sqlite.org/different.html > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Oct 23 11:46:44 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 23 Oct 2013 11:46:44 +0200 Subject: [Python-Dev] Usefulness of site-python? Message-ID: <20131023114644.7b214f5b@pitrou.net> Hello, I've just discovered there is a little-known feature in site.py: if a $PREFIX/lib/site-python exists (e.g. /usr/lib/site-python), it is added to sys.path in addition to the versioned site-packages. But only under Unix ("if os.sep == '/'"). Has anyone seen that feature in the real world? Debian doesn't use site-python, but its own /usr/share/pyshared. For the record, it was added in b53347c8260e with the following commit message: user: Guido van Rossum date: Wed Sep 03 21:41:30 1997 +0000 files: Lib/site.py description: Give in to Mike Meyer -- add *both* lib/python1.5/packages and lib/site-python to the path (if they exist). This is a reasonable compromise. Regards Antoine. From solipsis at pitrou.net Wed Oct 23 11:58:49 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 23 Oct 2013 11:58:49 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Switch subprocess stdin to a socketpair, attempting to fix issue #19293 (AIX References: <3d3gQD1pkkz7LmM@mail.python.org> Message-ID: <20131023115849.3e0b446c@pitrou.net> Le Tue, 22 Oct 2013 10:54:03 +0200, Victor Stinner a ?crit : > Hi, > > Would it be possible to use os.pipe() on all OSes except AIX? > > Pipes and socket pairs may have minor differences, but some > applications may rely on these minor differences. For example, is the > buffer size the same? For example, in test.support, we have two > constants: PIPE_MAX_SIZE (4 MB) and SOCK_MAX_SIZE (16 MB). For the record, pipe I/O seems a little faster than socket I/O under Linux: $ ./python -m timeit -s "import os, socket; a,b = socket.socketpair(); r=a.fileno(); w=b.fileno(); x=b'x'*1000" "os.write(w, x); os.read(r, 1000)" 1000000 loops, best of 3: 1.1 usec per loop $ ./python -m timeit -s "import os, socket; a,b = socket.socketpair(); x=b'x'*1000" "a.sendall(x); b.recv(1000)" 1000000 loops, best of 3: 1.02 usec per loop $ ./python -m timeit -s "import os; r, w = os.pipe(); x=b'x'*1000" "os.write(w, x); os.read(r, 1000)" 1000000 loops, best of 3: 0.82 usec per loop Regards Antoine. > > Victor > > 2013/10/22 guido.van.rossum : > > http://hg.python.org/cpython/rev/2a0bda8d283d > > changeset: 86557:2a0bda8d283d > > user: Guido van Rossum > > date: Mon Oct 21 20:37:14 2013 -0700 > > summary: > > Switch subprocess stdin to a socketpair, attempting to fix issue > > #19293 (AIX hang). > > > > files: > > Lib/asyncio/unix_events.py | 29 +++++++++- > > Lib/test/test_asyncio/test_unix_events.py | 7 ++ > > 2 files changed, 32 insertions(+), 4 deletions(-) > > > > > > diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py > > --- a/Lib/asyncio/unix_events.py > > +++ b/Lib/asyncio/unix_events.py > > if stdin == subprocess.PIPE: > > self._pipes[STDIN] = None > > + # Use a socket pair for stdin, since not all platforms > > + # support selecting read events on the write end of a > > + # socket (which we use in order to detect closing of > > the > > + # other end). Notably this is needed on AIX, and works > > + # just fine on other platforms. > > + stdin, stdin_w = self._loop._socketpair() > > if stdout == subprocess.PIPE: > > self._pipes[STDOUT] = None > > if stderr == subprocess.PIPE: From victor.stinner at gmail.com Wed Oct 23 13:53:40 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 23 Oct 2013 13:53:40 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Switch subprocess stdin to a socketpair, attempting to fix issue #19293 (AIX In-Reply-To: <20131023115849.3e0b446c@pitrou.net> References: <3d3gQD1pkkz7LmM@mail.python.org> <20131023115849.3e0b446c@pitrou.net> Message-ID: "For the record, pipe I/O seems a little faster than socket I/O under Linux" In and old (2006) email on LKML (Linux kernel), I read: "as far as I know pipe() is now much faster than socketpair(), because pipe() uses the zero-copy mechanism." https://lkml.org/lkml/2006/9/24/121 On Linux, splice() can also be used with pipes for zero-copy operations. I don't know if splice() works with socketpair(). Well, I don't think that Python uses splice() now, but it may be interesting to use it. Or sendfile() uses it maybe internally? Victor From solipsis at pitrou.net Wed Oct 23 14:08:31 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 23 Oct 2013 14:08:31 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Switch subprocess stdin to a socketpair, attempting to fix issue #19293 (AIX References: <3d3gQD1pkkz7LmM@mail.python.org> <20131023115849.3e0b446c@pitrou.net> Message-ID: <20131023140831.48b484f9@pitrou.net> Le Wed, 23 Oct 2013 13:53:40 +0200, Victor Stinner a ?crit : > "For the record, pipe I/O seems a little faster than socket I/O under > Linux" > > In and old (2006) email on LKML (Linux kernel), I read: > "as far as I know pipe() is now much faster than socketpair(), > because pipe() uses the zero-copy mechanism." > https://lkml.org/lkml/2006/9/24/121 > > On Linux, splice() can also be used with pipes for zero-copy > operations. I don't know if splice() works with socketpair(). splice() only works with pipes. socketpair() returns sockets, which are not pipes :-) > Well, I > don't think that Python uses splice() now, but it may be interesting > to use it. Where do you want to use it? Regards Antoine. From guido at python.org Wed Oct 23 17:14:05 2013 From: guido at python.org (Guido van Rossum) Date: Wed, 23 Oct 2013 08:14:05 -0700 Subject: [Python-Dev] Usefulness of site-python? In-Reply-To: <20131023114644.7b214f5b@pitrou.net> References: <20131023114644.7b214f5b@pitrou.net> Message-ID: On Wed, Oct 23, 2013 at 2:46 AM, Antoine Pitrou wrote: > > Hello, > > I've just discovered there is a little-known feature in site.py: if a > $PREFIX/lib/site-python exists (e.g. /usr/lib/site-python), it is added > to sys.path in addition to the versioned site-packages. But only under > Unix ("if os.sep == '/'"). > > Has anyone seen that feature in the real world? Debian doesn't use > site-python, but its own /usr/share/pyshared. > > For the record, it was added in b53347c8260e with the following commit > message: > > user: Guido van Rossum > date: Wed Sep 03 21:41:30 1997 +0000 > files: Lib/site.py > description: > Give in to Mike Meyer -- add *both* lib/python1.5/packages and > lib/site-python to the path (if they exist). This is a reasonable > compromise. > Wow, a blast from the past. I don't see python-dev archives from those times but I'm sure there must have been discussion about whether site-packages were shareable between Python versions. I think history has shown that it's better to install them per version, so I suggest that we remove that feature. (People who want it can always patch up their own $PYTHONPATH.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From hakril at lse.epita.fr Wed Oct 23 19:29:03 2013 From: hakril at lse.epita.fr (hakril lse) Date: Wed, 23 Oct 2013 19:29:03 +0200 Subject: [Python-Dev] About 'superobject' and descriptors Message-ID: Hi, I have a question about a choice of implementation concerning 'superobject' with the descriptors. When a 'superobject' looks for a given attribute, it runs through the mro of the object. If it finds a descriptor, the 'superobject' calls the __get__ method with 'starttype = su->obj_type' as third argument (in typeobject.c: super_getattro). So, the 'type' argument of __get__ does not give more information about the 'real calling type' in this case. It seems that this is just a redundant information of inst.__class__. For example: # A.descr is a descriptor # B inherit from A # C inherit from B c = C() c.descr super(C, c).descr super(B, c).descr In these 3 cases the __get__ method is called with the same arguments that are : __get__(descr, c, C). If this behavior is really expected: Could you explain why ? because it means that I am missing something obvious. Because, at first sight, the 'type' argument seems to be the perfect place to get the type of the 'real calling class'. Thank you, -- hakril From victor.stinner at gmail.com Wed Oct 23 20:25:11 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 23 Oct 2013 20:25:11 +0200 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! Message-ID: Hi, I was at the restaurant with Charles-Fran?ois and Antoine yesterday to discuss the PEP 454 (tracemalloc). They gave me a lot of advices to improve the PEP. Most remarks were request to remove code :-) I also improved surprising/strange APIs (like the infamous GroupedStats.compate_to(None)). HTML version: http://www.python.org/dev/peps/pep-0454/ See also the documentation of the implementation, especially examples: http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html#examples Major changes: * GroupedStats.compare_to()/statistics() now returns a list of Statistic instances instead of a tuple with 5 items * StatsDiff class has been removed * Metrics have been removed * Remove Filter.match*() methods * Replace get_object_trace() function with get_object_traceback() * More complete list of prior work. There are 11 Python projects to debug memory leaks! I mentioned that PySizer implemented something similar to tracemalloc 8 years ago. I also rewrote the Rationale section * Rename some classes, attributes and functions Mercurial log of the PEP: http://hg.python.org/peps/log/f851d4a1622a/pep-0454.txt PEP: 454 Title: Add a new tracemalloc module to trace Python memory allocations Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3-September-2013 Python-Version: 3.4 Abstract ======== This PEP proposes to add a new ``tracemalloc`` module to trace memory blocks allocated by Python. Rationale ========= Classic generic tools like Valgrind can get the C traceback where a memory block was allocated. Using such tools to analyze Python memory allocations does not help because most memory blocks are allocated in the same C function, in ``PyMem_Malloc()`` for example. Moreover, Python has an allocator for small object called "pymalloc" which keeps free blocks for efficiency. This is not well handled by these tools. There are debug tools dedicated to the Python language like ``Heapy`` ``Pympler`` and ``Meliae`` which lists all live objects using the garbage module (functions like ``gc.get_objects()``, ``gc.get_referrers()`` and ``gc.get_referents()``), compute their size (ex: using ``sys.getsizeof()``) and group objects by type. These tools provide a better estimation of the memory usage of an application. They are useful when most memory leaks are instances of the same type and this type is only instantiated in a few functions. Problems arise when the object type is very common like ``str`` or ``tuple``, and it is hard to identify where these objects are instantiated. Finding reference cycles is also a difficult problem. There are different tools to draw a diagram of all references. These tools cannot be used on large applications with thousands of objects because the diagram is too huge to be analyzed manually. Proposal ======== Using the customized allocation API from PEP 445, it becomes easy to set up a hook on Python memory allocators. A hook can inspect Python internals to retrieve Python tracebacks. The idea of getting the current traceback comes from the faulthandler module. The faulthandler dumps the traceback of all Python threads on a crash, here is the idea is to get the traceback of the current Python thread when a memory block is allocated by Python. This PEP proposes to add a new ``tracemalloc`` module, as a debug tool to trace memory blocks allocated by Python. The module provides the following information: * Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks * Computed differences between two snapshots to detect memory leaks * Traceback where a memory block was allocated The API of the tracemalloc module is similar to the API of the faulthandler module: ``enable()``, ``disable()`` and ``is_enabled()`` functions, an environment variable (``PYTHONFAULTHANDLER`` and ``PYTHONTRACEMALLOC``), and a ``-X`` command line option (``-X faulthandler`` and ``-X tracemalloc``). See the `documentation of the faulthandler module `_. The idea of tracing memory allocations is not new. It was first implemented in the PySizer project in 2005. PySizer was implemented differently: the traceback was stored in frame objects and some Python types were linked the trace with the name of object type. PySizer patch on CPython adds a overhead on performances and memory footprint, even if the PySizer was not used. tracemalloc attachs a traceback to the underlying layer, to memory blocks, and has no overhead when the module is disabled. The tracemalloc module has been written for CPython. Other implementations of Python may not be able to provide it. API === To trace most memory blocks allocated by Python, the module should be enabled as early as possible by setting the ``PYTHONTRACEMALLOC`` environment variable to ``1``, or by using ``-X tracemalloc`` command line option. The ``tracemalloc.enable()`` function can be called at runtime to start tracing Python memory allocations. By default, a trace of an allocated memory block only stores the most recent frame (1 frame). To store 25 frames at startup: set the ``PYTHONTRACEMALLOC`` environment variable to ``25``, or use the ``-X tracemalloc=25`` command line option. The ``set_traceback_limit()`` function can be used at runtime to set the limit. By default, Python memory blocks allocated in the ``tracemalloc`` module are ignored using a filter. Use ``clear_filters()`` to trace also these memory allocations. Main Functions -------------- ``reset()`` function: Clear traces and statistics on Python memory allocations. See also ``disable()``. ``disable()`` function: Stop tracing Python memory allocations and clear traces and statistics. See also ``enable()`` and ``is_enabled()`` functions. ``enable()`` function: Start tracing Python memory allocations. See also ``disable()`` and ``is_enabled()`` functions. ``get_stats()`` function: Get statistics on traced Python memory blocks as a dictionary ``{filename (str): {line_number (int): stats}}`` where *stats* in a ``(size: int, count: int)`` tuple, *filename* and *line_number* can be ``None``. *size* is the total size in bytes of all memory blocks allocated on the line, or *count* is the number of memory blocks allocated on the line. Return an empty dictionary if the ``tracemalloc`` module is disabled. See also the ``get_traces()`` function. ``get_traced_memory()`` function: Get the current size and maximum size of memory blocks traced by the ``tracemalloc`` module as a tuple: ``(size: int, max_size: int)``. ``get_tracemalloc_memory()`` function: Get the memory usage in bytes of the ``tracemalloc`` module used internally to trace memory allocations. Return an ``int``. ``is_enabled()`` function: ``True`` if the ``tracemalloc`` module is tracing Python memory allocations, ``False`` otherwise. See also ``disable()`` and ``enable()`` functions. Trace Functions --------------- When Python allocates a memory block, ``tracemalloc`` attachs a "trace" to it to store information on it: its size in bytes and the traceback where the allocation occured. The following functions give access to these traces. A trace is a ``(size: int, traceback)`` tuple. *size* is the size of the memory block in bytes. *traceback* is a tuple of frames sorted from the most recent to the oldest frame, limited to ``get_traceback_limit()`` frames. A frame is a ``(filename: str, lineno: int)`` tuple where *filename* and *lineno* can be ``None``. Example of trace: ``(32, (('x.py', 7), ('x.py', 11)))``. The memory block has a size of 32 bytes and was allocated at ``x.py:7``, line called from line ``x.py:11``. ``get_object_address(obj)`` function: Get the address of the main memory block of the specified Python object. A Python object can be composed by multiple memory blocks, the function only returns the address of the main memory block. For example, items of ``dict`` and ``set`` containers are stored in a second memory block. See also ``get_object_traceback()`` and ``gc.get_referrers()`` functions. .. note:: The builtin function ``id()`` returns a different address for objects tracked by the garbage collector, because ``id()`` returns the address after the garbage collector header. ``get_object_traceback(obj)`` function: Get the traceback where the Python object *obj* was allocated. Return a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the object. See also ``get_object_address()``, ``gc.get_referrers()`` and ``sys.getsizeof()`` functions. ``get_trace(address)`` function: Get the trace of a memory block allocated by Python. Return a tuple: ``(size: int, traceback)``, *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the memory block. See also ``get_object_traceback()``, ``get_stats()`` and ``get_traces()`` functions. ``get_traceback_limit()`` function: Get the maximum number of frames stored in the traceback of a trace. By default, a trace of an allocated memory block only stores the most recent frame: the limit is ``1``. This limit is enough to get statistics using ``get_stats()``. Use the ``set_traceback_limit()`` function to change the limit. ``get_traces()`` function: Get traces of all memory blocks allocated by Python. Return a dictionary: ``{address (int): trace}``, *trace* is a ``(size: int, traceback)`` tuple, *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be None. Return an empty dictionary if the ``tracemalloc`` module is disabled. See also ``get_object_traceback()``, ``get_stats()`` and ``get_trace()`` functions. ``set_traceback_limit(nframe: int)`` function: Set the maximum number of frames stored in the traceback of a trace. Storing the traceback of each memory allocation has an important overhead on the memory usage. Use the ``get_tracemalloc_memory()`` function to measure the overhead and the ``add_filter()`` function to select which memory allocations are traced. Use the ``get_traceback_limit()`` function to get the current limit. The ``PYTHONTRACEMALLOC`` environment variable and the ``-X`` ``tracemalloc=NFRAME`` command line option can be used to set a limit at startup. Filter Functions ---------------- ``add_filter(filter)`` function: Add a new filter on Python memory allocations, *filter* is a ``Filter`` instance. All inclusive filters are applied at once, a memory allocation is only ignored if no inclusive filters match its trace. A memory allocation is ignored if at least one exclusive filter matchs its trace. The new filter is not applied on already collected traces. Use the ``reset()`` function to ensure that all traces match the new filter. ``add_inclusive_filter(filename_pattern: str, lineno: int=None, traceback: bool=False)`` function: Add an inclusive filter: helper for the ``add_filter()`` function creating a ``Filter`` instance with the ``Filter.include`` attribute set to ``True``. The ``*`` joker character can be used in *filename_pattern* to match any substring, including empty string. Example: ``tracemalloc.add_inclusive_filter(subprocess.__file__)`` only includes memory blocks allocated by the ``subprocess`` module. ``add_exclusive_filter(filename_pattern: str, lineno: int=None, traceback: bool=False)`` function: Add an exclusive filter: helper for the ``add_filter()`` function creating a ``Filter`` instance with the ``Filter.include`` attribute set to ``False``. The ``*`` joker character can be used in *filename_pattern* to match any substring, including empty string. Example: ``tracemalloc.add_exclusive_filter(tracemalloc.__file__)`` ignores memory blocks allocated by the ``tracemalloc`` module. ``clear_filters()`` function: Clear the filter list. See also the ``get_filters()`` function. ``get_filters()`` function: Get the filters on Python memory allocations. Return a list of ``Filter`` instances. By default, there is one exclusive filter to ignore Python memory blocks allocated by the ``tracemalloc`` module. See also the ``clear_filters()`` function. Filter ------ ``Filter(include: bool, filename_pattern: str, lineno: int=None, traceback: bool=False)`` class: Filter to select which memory allocations are traced. Filters can be used to reduce the memory usage of the ``tracemalloc`` module, which can be read using the ``get_tracemalloc_memory()`` function. The ``*`` joker character can be used in *filename_pattern* to match any substring, including empty string. The ``.pyc`` and ``.pyo`` file extensions are replaced with ``.py``. On Windows, the comparison is case insensitive and the alternative separator ``/`` is replaced with the standard separator ``\``. ``include`` attribute: If *include* is ``True``, only trace memory blocks allocated in a file with a name matching ``filename_pattern`` at line number ``lineno``. If *include* is ``False``, ignore memory blocks allocated in a file with a name matching ``filename_pattern`` at line number ``lineno``. ``lineno`` attribute: Line number (``int``) of the filter. If *lineno* is is ``None`` or less than ``1``, the filter matches any line number. ``filename_pattern`` attribute: Filename pattern (``str``) of the filter. ``traceback`` attribute: If *traceback* is ``True``, all frames of the traceback are checked. If *traceback* is ``False``, only the most recent frame is checked. This attribute is ignored if the traceback limit is less than ``2``. See the ``get_traceback_limit()`` function. GroupedStats ------------ ``GroupedStats(timestamp: datetime.datetime, traceback_limit: int, stats: dict, key_type: str, cumulative: bool)`` class: Top of allocated memory blocks grouped by *key_type* as a dictionary. The ``Snapshot.group_by()`` method creates a ``GroupedStats`` instance. ``compare_to(old_stats: GroupedStats, sort=True)`` method: Compare statistics to an older ``GroupedStats`` instance. Return a list of ``Statistic`` instances. The result is sorted in the biggest to the smallest by ``abs(size_diff)``, *size*, ``abs(count_diff)``, *count* and then by *key*. Set the *sort* parameter to ``False`` to get the list unsorted. ``None`` values in keys are replaced with an empty string for filenames or zero for line numbers, because ``str`` and ``int`` cannot be compared to ``None``. See also the ``statistics()`` method. ``statistics(sort=True)`` method: Get statistics as a list of ``Statistic`` instances. ``Statistic.size_diff`` and ``Statistic.count_diff`` attributes are set to zero. The result is sorted in the biggest to the smallest by ``abs(size_diff)``, *size*, ``abs(count_diff)``, *count* and then by *key*. Set the *sort* parameter to ``False`` to get the list unsorted. ``None`` values in keys are replaced with an empty string for filenames or zero for line numbers, because ``str`` and ``int`` cannot be compared to ``None``. See also the ``compare_to()`` method. ``cumulative`` attribute: If ``True``, size and count of memory blocks of all frames of the traceback of a trace were cumulated, not only the most recent frame. ``key_type`` attribute: Determine how memory allocations were grouped: see ``Snapshot.group_by()()`` for the available values. ``stats`` attribute: Dictionary ``{key: (size: int, count: int)}`` where the type of *key* depends on the ``key_type`` attribute. See the ``Snapshot.group_by()`` method. ``traceback_limit`` attribute: Maximum number of frames stored in the traceback of ``traces``, result of the ``get_traceback_limit()`` function. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. Snapshot -------- ``Snapshot(timestamp: datetime.datetime, traceback_limit: int, stats: dict=None, traces: dict=None)`` class: Snapshot of statistics and traces of memory blocks allocated by Python. ``apply_filters(filters)`` method: Apply filters on the ``traces`` and ``stats`` dictionaries, *filters* is a list of ``Filter`` instances. ``create(traces=False)`` classmethod: Take a snapshot of statistics and traces of memory blocks allocated by Python. If *traces* is ``True``, ``get_traces()`` is called and its result is stored in the ``Snapshot.traces`` attribute. This attribute contains more information than ``Snapshot.stats`` and uses more memory and more disk space. If *traces* is ``False``, ``Snapshot.traces`` is set to ``None``. Tracebacks of traces are limited to ``traceback_limit`` frames. Call ``set_traceback_limit()`` before calling ``Snapshot.create()`` to store more frames. The ``tracemalloc`` module must be enabled to take a snapshot, see the the ``enable()`` function. ``dump(filename)`` method: Write the snapshot into a file. Use ``load()`` to reload the snapshot. ``load(filename)`` classmethod: Load a snapshot from a file. See also ``dump()``. ``group_by(key_type: str, cumulative: bool=False)`` method: Group statistics by *key_type* as a ``GroupedStats`` instance: ===================== =================================== ================================ key_type description type ===================== =================================== ================================ ``'filename'`` filename ``str`` ``'line'`` filename and line number ``(filename: str, lineno: int)`` ``'address'`` memory block address ``int`` ``'traceback'`` memory block address with traceback ``(address: int, traceback)`` ===================== =================================== ================================ The ``traceback`` type is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. If *cumulative* is ``True``, cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. The *cumulative* parameter is set to ``False`` if *key_type* is ``'address'``, or if the traceback limit is less than ``2``. ``stats`` attribute: Statistics on traced Python memory, result of the ``get_stats()`` function. ``traceback_limit`` attribute: Maximum number of frames stored in the traceback of ``traces``, result of the ``get_traceback_limit()`` function. ``traces`` attribute: Traces of Python memory allocations, result of the ``get_traces()`` function, can be ``None``. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. Statistic --------- ``Statistic(key, size, size_diff, count, count_diff)`` class: Statistic on memory allocations. ``GroupedStats.compare_to()`` and ``GroupedStats.statistics()`` return a list of ``Statistic`` instances. ``key`` attribute: Key identifying the statistic. The key type depends on ``GroupedStats.key_type``, see the ``Snapshot.group_by()`` method. ``count`` attribute: Number of memory blocks (``int``). ``count_diff`` attribute: Difference of number of memory blocks (``int``). ``size`` attribute: Total size of memory blocks in bytes (``int``). ``size_diff`` attribute: Difference of total size of memory blocks in bytes (``int``). Prior Work ========== * `Python Memory Validator `_ (2005-2013): commercial Python memory validator developed by Software Verification. It uses the Python Reflection API. * `PySizer `_: Google Summer of Code 2005 project by Nick Smallbone. * `Heapy `_ (2006-2013): part of the Guppy-PE project written by Sverker Nilsson. * Draft PEP: `Support Tracking Low-Level Memory Usage in CPython `_ (Brett Canon, 2006) * Muppy: project developed in 2008 by Robert Schuppenies. * `asizeof `_: a pure Python module to estimate the size of objects by Jean Brouwers (2008). * `Heapmonitor `_: It provides facilities to size individual objects and can track all objects of certain classes. It was developed in 2008 by Ludwig Haehne. * `Pympler `_ (2008-2011): project based on asizeof, muppy and HeapMonitor * `objgraph `_ (2008-2012) * `Dozer `_: WSGI Middleware version of the CherryPy memory leak debugger, written by Marius Gedminas (2008-2013) * `Meliae `_: Python Memory Usage Analyzer developed by John A Meinel since 2009 * `caulk `_: written by Ben Timby in 2012 * `memory_profiler `_: written by Fabian Pedregosa (2011-2013) See also `Pympler Related Work `_. Links ===== tracemalloc: * `#18874: Add a new tracemalloc module to trace Python memory allocations `_ * `pytracemalloc on PyPI `_ Copyright ========= This document has been placed in the public domain. From cf.natali at gmail.com Wed Oct 23 20:25:47 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Wed, 23 Oct 2013 20:25:47 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Switch subprocess stdin to a socketpair, attempting to fix issue #19293 (AIX In-Reply-To: <20131023115849.3e0b446c@pitrou.net> References: <3d3gQD1pkkz7LmM@mail.python.org> <20131023115849.3e0b446c@pitrou.net> Message-ID: > For the record, pipe I/O seems a little faster than socket I/O under > Linux: > > $ ./python -m timeit -s "import os, socket; a,b = socket.socketpair(); r=a.fileno(); w=b.fileno(); x=b'x'*1000" "os.write(w, x); os.read(r, 1000)" > 1000000 loops, best of 3: 1.1 usec per loop > > $ ./python -m timeit -s "import os, socket; a,b = socket.socketpair(); x=b'x'*1000" > "a.sendall(x); b.recv(1000)" > 1000000 loops, best of 3: 1.02 usec per loop > > $ ./python -m timeit -s "import os; r, w = os.pipe(); x=b'x'*1000" "os.write(w, x); os.read(r, 1000)" > 1000000 loops, best of 3: 0.82 usec per loop That's a raw write()/read() benchmark, but it's not taking something important into account: pipes/socket are usually used to communicate between concurrently running processes. And in this case, an important factor is the pipe/socket buffer size: the smaller it is, the more context switches (due to blocking writes/reads) you'll get, which greatly decreases throughput. And by default, Unix sockets have large buffers than pipes (between 4K and 64K for pipes depending on the OS): I wrote a quick benchmark forking a child process, with the parent writing data through the pipe, and waiting for the child to read it all. here are the results (on Linux): # time python /tmp/test.py pipe real 0m2.479s user 0m1.344s sys 0m1.860s # time python /tmp/test.py socketpair real 0m1.454s user 0m1.242s sys 0m1.234s So socketpair is actually faster. But as noted by Victor, there a slight differences between pipes and sockets I can think of: - pipes guarantee write atomicity if less than PIPE_BUF is written, which is not the case for sockets - more annoying: in subprocess, the pipes are not set non-blocking: after a select()/poll() returns a FD write-ready, we write less than PIPE_BUF at a time to avoid blocking: this likely wouldn't work with a socketpair But this patch doesn't touch subprocess itself, and the FDs is only used by asyncio, which sets them non-blocking: so this could only be an issue for the spawned process, if it does rely on the two pipe-specific behaviors above. OTOH, having a unique implementation on all platforms makes sense, and I don't know if it'll actually be a problem in practice, we we could ship as-is and wait until someone complains ;-) cf From kristjan at ccpgames.com Wed Oct 23 22:06:42 2013 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 23 Oct 2013 20:06:42 +0000 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: Message-ID: This might be a good place to make some comments. I have discussed some of this in private with Victor, but wanted to make them here, for the record. Mainly, I agree with removing code. I'd like to go further, since in my experience, the less code in C, the better. 1) really, all that is required in terms of data is the traceback.get_traces() function. Further, it _need_ not return addresses since they are not required for analysis. It is sufficient for it to return a list of (traceback, size, count) tuples. I understand that the get_stats function is useful for quick information so it can be kept, although it provides no added information, only convenience 2) get_object_address() and get_trace(address) functions seem redundant. All that is required is get_object_traceback(), I think. 3) set_traceback_limit(). Truncating tracebacks is bad. Particularly if it is truncated at the top end of the callstack, because then information looses cohesion, namely, the common connection point, the root. If traceback limits are required, I suggest being able to specifiy that we truncate the leaf-end of the tracebacks. 4) add_filter(). This is unnecessary. Information can be filtered on the python side. Defining Filter as a C type is not necessary. Similarly, module level filter functions can be dropped. 5) Filter, Snapshot, GroupedStats, Statistics: These classes, if required, can be implemented in a .py module. 6) Snapshot dump/load(): It is unusual to see load and save functions taking filenames in a python module, and a module implementing its own file IO. I have suggested simply to add Pickle support. Alternatively, support file-like objects or bytes (loads/dumps) My experience is that performance and memory use hardly ever matters when you are doing diagnostic analysis of a program. By definition, you are examining your program in a lab and you can afford 2 times, or 10 times, the memory use, and the slowing down of the program by 2 to 10. I think it might be premature to move all of the statistics and analysis into the PEP and into C, because a) it assumes the need to optimize and b) it sets the specification in stone, before the module gets the chance to be honed by actual real-world use cases. I'd also like to point out (just to say "I told you so" :) ) that this module is precisely the reason I suggested we include "const char *file, int lineno" in the API for PEP 445, because that would allow us, in debug builds, to get one extra stack level, namely the position of the actual C allocation in the python source. If the above sounds negative, then that's not the intent. I'm really happy Victor is putting in this effort here and I know this will be an essential tool for the future Python developer. Those that brave the jump to version 3, that is :) Cheers, Kristj?n ________________________________________ Fr?: Python-Dev [python-dev-bounces+kristjan=ccpgames.com at python.org] fyrir hönd Victor Stinner [victor.stinner at gmail.com] Sent: 23. okt?ber 2013 18:25 To: Python Dev Efni: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! Hi, I was at the restaurant with Charles-Fran?ois and Antoine yesterday to discuss the PEP 454 (tracemalloc). They gave me a lot of advices to improve the PEP. Most remarks were request to remove code :-) I also improved surprising/strange APIs (like the infamous GroupedStats.compate_to(None)). HTML version: http://www.python.org/dev/peps/pep-0454/ See also the documentation of the implementation, especially examples: http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html#examples Major changes: * GroupedStats.compare_to()/statistics() now returns a list of Statistic instances instead of a tuple with 5 items * StatsDiff class has been removed * Metrics have been removed * Remove Filter.match*() methods * Replace get_object_trace() function with get_object_traceback() * More complete list of prior work. There are 11 Python projects to debug memory leaks! I mentioned that PySizer implemented something similar to tracemalloc 8 years ago. I also rewrote the Rationale section * Rename some classes, attributes and functions Mercurial log of the PEP: http://hg.python.org/peps/log/f851d4a1622a/pep-0454.txt PEP: 454 Title: Add a new tracemalloc module to trace Python memory allocations Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3-September-2013 Python-Version: 3.4 Abstract ======== This PEP proposes to add a new ``tracemalloc`` module to trace memory blocks allocated by Python. Rationale ========= Classic generic tools like Valgrind can get the C traceback where a memory block was allocated. Using such tools to analyze Python memory allocations does not help because most memory blocks are allocated in the same C function, in ``PyMem_Malloc()`` for example. Moreover, Python has an allocator for small object called "pymalloc" which keeps free blocks for efficiency. This is not well handled by these tools. There are debug tools dedicated to the Python language like ``Heapy`` ``Pympler`` and ``Meliae`` which lists all live objects using the garbage module (functions like ``gc.get_objects()``, ``gc.get_referrers()`` and ``gc.get_referents()``), compute their size (ex: using ``sys.getsizeof()``) and group objects by type. These tools provide a better estimation of the memory usage of an application. They are useful when most memory leaks are instances of the same type and this type is only instantiated in a few functions. Problems arise when the object type is very common like ``str`` or ``tuple``, and it is hard to identify where these objects are instantiated. Finding reference cycles is also a difficult problem. There are different tools to draw a diagram of all references. These tools cannot be used on large applications with thousands of objects because the diagram is too huge to be analyzed manually. Proposal ======== Using the customized allocation API from PEP 445, it becomes easy to set up a hook on Python memory allocators. A hook can inspect Python internals to retrieve Python tracebacks. The idea of getting the current traceback comes from the faulthandler module. The faulthandler dumps the traceback of all Python threads on a crash, here is the idea is to get the traceback of the current Python thread when a memory block is allocated by Python. This PEP proposes to add a new ``tracemalloc`` module, as a debug tool to trace memory blocks allocated by Python. The module provides the following information: * Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks * Computed differences between two snapshots to detect memory leaks * Traceback where a memory block was allocated The API of the tracemalloc module is similar to the API of the faulthandler module: ``enable()``, ``disable()`` and ``is_enabled()`` functions, an environment variable (``PYTHONFAULTHANDLER`` and ``PYTHONTRACEMALLOC``), and a ``-X`` command line option (``-X faulthandler`` and ``-X tracemalloc``). See the `documentation of the faulthandler module `_. The idea of tracing memory allocations is not new. It was first implemented in the PySizer project in 2005. PySizer was implemented differently: the traceback was stored in frame objects and some Python types were linked the trace with the name of object type. PySizer patch on CPython adds a overhead on performances and memory footprint, even if the PySizer was not used. tracemalloc attachs a traceback to the underlying layer, to memory blocks, and has no overhead when the module is disabled. The tracemalloc module has been written for CPython. Other implementations of Python may not be able to provide it. API === To trace most memory blocks allocated by Python, the module should be enabled as early as possible by setting the ``PYTHONTRACEMALLOC`` environment variable to ``1``, or by using ``-X tracemalloc`` command line option. The ``tracemalloc.enable()`` function can be called at runtime to start tracing Python memory allocations. By default, a trace of an allocated memory block only stores the most recent frame (1 frame). To store 25 frames at startup: set the ``PYTHONTRACEMALLOC`` environment variable to ``25``, or use the ``-X tracemalloc=25`` command line option. The ``set_traceback_limit()`` function can be used at runtime to set the limit. By default, Python memory blocks allocated in the ``tracemalloc`` module are ignored using a filter. Use ``clear_filters()`` to trace also these memory allocations. Main Functions -------------- ``reset()`` function: Clear traces and statistics on Python memory allocations. See also ``disable()``. ``disable()`` function: Stop tracing Python memory allocations and clear traces and statistics. See also ``enable()`` and ``is_enabled()`` functions. ``enable()`` function: Start tracing Python memory allocations. See also ``disable()`` and ``is_enabled()`` functions. ``get_stats()`` function: Get statistics on traced Python memory blocks as a dictionary ``{filename (str): {line_number (int): stats}}`` where *stats* in a ``(size: int, count: int)`` tuple, *filename* and *line_number* can be ``None``. *size* is the total size in bytes of all memory blocks allocated on the line, or *count* is the number of memory blocks allocated on the line. Return an empty dictionary if the ``tracemalloc`` module is disabled. See also the ``get_traces()`` function. ``get_traced_memory()`` function: Get the current size and maximum size of memory blocks traced by the ``tracemalloc`` module as a tuple: ``(size: int, max_size: int)``. ``get_tracemalloc_memory()`` function: Get the memory usage in bytes of the ``tracemalloc`` module used internally to trace memory allocations. Return an ``int``. ``is_enabled()`` function: ``True`` if the ``tracemalloc`` module is tracing Python memory allocations, ``False`` otherwise. See also ``disable()`` and ``enable()`` functions. Trace Functions --------------- When Python allocates a memory block, ``tracemalloc`` attachs a "trace" to it to store information on it: its size in bytes and the traceback where the allocation occured. The following functions give access to these traces. A trace is a ``(size: int, traceback)`` tuple. *size* is the size of the memory block in bytes. *traceback* is a tuple of frames sorted from the most recent to the oldest frame, limited to ``get_traceback_limit()`` frames. A frame is a ``(filename: str, lineno: int)`` tuple where *filename* and *lineno* can be ``None``. Example of trace: ``(32, (('x.py', 7), ('x.py', 11)))``. The memory block has a size of 32 bytes and was allocated at ``x.py:7``, line called from line ``x.py:11``. ``get_object_address(obj)`` function: Get the address of the main memory block of the specified Python object. A Python object can be composed by multiple memory blocks, the function only returns the address of the main memory block. For example, items of ``dict`` and ``set`` containers are stored in a second memory block. See also ``get_object_traceback()`` and ``gc.get_referrers()`` functions. .. note:: The builtin function ``id()`` returns a different address for objects tracked by the garbage collector, because ``id()`` returns the address after the garbage collector header. ``get_object_traceback(obj)`` function: Get the traceback where the Python object *obj* was allocated. Return a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the object. See also ``get_object_address()``, ``gc.get_referrers()`` and ``sys.getsizeof()`` functions. ``get_trace(address)`` function: Get the trace of a memory block allocated by Python. Return a tuple: ``(size: int, traceback)``, *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the memory block. See also ``get_object_traceback()``, ``get_stats()`` and ``get_traces()`` functions. ``get_traceback_limit()`` function: Get the maximum number of frames stored in the traceback of a trace. By default, a trace of an allocated memory block only stores the most recent frame: the limit is ``1``. This limit is enough to get statistics using ``get_stats()``. Use the ``set_traceback_limit()`` function to change the limit. ``get_traces()`` function: Get traces of all memory blocks allocated by Python. Return a dictionary: ``{address (int): trace}``, *trace* is a ``(size: int, traceback)`` tuple, *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be None. Return an empty dictionary if the ``tracemalloc`` module is disabled. See also ``get_object_traceback()``, ``get_stats()`` and ``get_trace()`` functions. ``set_traceback_limit(nframe: int)`` function: Set the maximum number of frames stored in the traceback of a trace. Storing the traceback of each memory allocation has an important overhead on the memory usage. Use the ``get_tracemalloc_memory()`` function to measure the overhead and the ``add_filter()`` function to select which memory allocations are traced. Use the ``get_traceback_limit()`` function to get the current limit. The ``PYTHONTRACEMALLOC`` environment variable and the ``-X`` ``tracemalloc=NFRAME`` command line option can be used to set a limit at startup. Filter Functions ---------------- ``add_filter(filter)`` function: Add a new filter on Python memory allocations, *filter* is a ``Filter`` instance. All inclusive filters are applied at once, a memory allocation is only ignored if no inclusive filters match its trace. A memory allocation is ignored if at least one exclusive filter matchs its trace. The new filter is not applied on already collected traces. Use the ``reset()`` function to ensure that all traces match the new filter. ``add_inclusive_filter(filename_pattern: str, lineno: int=None, traceback: bool=False)`` function: Add an inclusive filter: helper for the ``add_filter()`` function creating a ``Filter`` instance with the ``Filter.include`` attribute set to ``True``. The ``*`` joker character can be used in *filename_pattern* to match any substring, including empty string. Example: ``tracemalloc.add_inclusive_filter(subprocess.__file__)`` only includes memory blocks allocated by the ``subprocess`` module. ``add_exclusive_filter(filename_pattern: str, lineno: int=None, traceback: bool=False)`` function: Add an exclusive filter: helper for the ``add_filter()`` function creating a ``Filter`` instance with the ``Filter.include`` attribute set to ``False``. The ``*`` joker character can be used in *filename_pattern* to match any substring, including empty string. Example: ``tracemalloc.add_exclusive_filter(tracemalloc.__file__)`` ignores memory blocks allocated by the ``tracemalloc`` module. ``clear_filters()`` function: Clear the filter list. See also the ``get_filters()`` function. ``get_filters()`` function: Get the filters on Python memory allocations. Return a list of ``Filter`` instances. By default, there is one exclusive filter to ignore Python memory blocks allocated by the ``tracemalloc`` module. See also the ``clear_filters()`` function. Filter ------ ``Filter(include: bool, filename_pattern: str, lineno: int=None, traceback: bool=False)`` class: Filter to select which memory allocations are traced. Filters can be used to reduce the memory usage of the ``tracemalloc`` module, which can be read using the ``get_tracemalloc_memory()`` function. The ``*`` joker character can be used in *filename_pattern* to match any substring, including empty string. The ``.pyc`` and ``.pyo`` file extensions are replaced with ``.py``. On Windows, the comparison is case insensitive and the alternative separator ``/`` is replaced with the standard separator ``\``. ``include`` attribute: If *include* is ``True``, only trace memory blocks allocated in a file with a name matching ``filename_pattern`` at line number ``lineno``. If *include* is ``False``, ignore memory blocks allocated in a file with a name matching ``filename_pattern`` at line number ``lineno``. ``lineno`` attribute: Line number (``int``) of the filter. If *lineno* is is ``None`` or less than ``1``, the filter matches any line number. ``filename_pattern`` attribute: Filename pattern (``str``) of the filter. ``traceback`` attribute: If *traceback* is ``True``, all frames of the traceback are checked. If *traceback* is ``False``, only the most recent frame is checked. This attribute is ignored if the traceback limit is less than ``2``. See the ``get_traceback_limit()`` function. GroupedStats ------------ ``GroupedStats(timestamp: datetime.datetime, traceback_limit: int, stats: dict, key_type: str, cumulative: bool)`` class: Top of allocated memory blocks grouped by *key_type* as a dictionary. The ``Snapshot.group_by()`` method creates a ``GroupedStats`` instance. ``compare_to(old_stats: GroupedStats, sort=True)`` method: Compare statistics to an older ``GroupedStats`` instance. Return a list of ``Statistic`` instances. The result is sorted in the biggest to the smallest by ``abs(size_diff)``, *size*, ``abs(count_diff)``, *count* and then by *key*. Set the *sort* parameter to ``False`` to get the list unsorted. ``None`` values in keys are replaced with an empty string for filenames or zero for line numbers, because ``str`` and ``int`` cannot be compared to ``None``. See also the ``statistics()`` method. ``statistics(sort=True)`` method: Get statistics as a list of ``Statistic`` instances. ``Statistic.size_diff`` and ``Statistic.count_diff`` attributes are set to zero. The result is sorted in the biggest to the smallest by ``abs(size_diff)``, *size*, ``abs(count_diff)``, *count* and then by *key*. Set the *sort* parameter to ``False`` to get the list unsorted. ``None`` values in keys are replaced with an empty string for filenames or zero for line numbers, because ``str`` and ``int`` cannot be compared to ``None``. See also the ``compare_to()`` method. ``cumulative`` attribute: If ``True``, size and count of memory blocks of all frames of the traceback of a trace were cumulated, not only the most recent frame. ``key_type`` attribute: Determine how memory allocations were grouped: see ``Snapshot.group_by()()`` for the available values. ``stats`` attribute: Dictionary ``{key: (size: int, count: int)}`` where the type of *key* depends on the ``key_type`` attribute. See the ``Snapshot.group_by()`` method. ``traceback_limit`` attribute: Maximum number of frames stored in the traceback of ``traces``, result of the ``get_traceback_limit()`` function. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. Snapshot -------- ``Snapshot(timestamp: datetime.datetime, traceback_limit: int, stats: dict=None, traces: dict=None)`` class: Snapshot of statistics and traces of memory blocks allocated by Python. ``apply_filters(filters)`` method: Apply filters on the ``traces`` and ``stats`` dictionaries, *filters* is a list of ``Filter`` instances. ``create(traces=False)`` classmethod: Take a snapshot of statistics and traces of memory blocks allocated by Python. If *traces* is ``True``, ``get_traces()`` is called and its result is stored in the ``Snapshot.traces`` attribute. This attribute contains more information than ``Snapshot.stats`` and uses more memory and more disk space. If *traces* is ``False``, ``Snapshot.traces`` is set to ``None``. Tracebacks of traces are limited to ``traceback_limit`` frames. Call ``set_traceback_limit()`` before calling ``Snapshot.create()`` to store more frames. The ``tracemalloc`` module must be enabled to take a snapshot, see the the ``enable()`` function. ``dump(filename)`` method: Write the snapshot into a file. Use ``load()`` to reload the snapshot. ``load(filename)`` classmethod: Load a snapshot from a file. See also ``dump()``. ``group_by(key_type: str, cumulative: bool=False)`` method: Group statistics by *key_type* as a ``GroupedStats`` instance: ===================== =================================== ================================ key_type description type ===================== =================================== ================================ ``'filename'`` filename ``str`` ``'line'`` filename and line number ``(filename: str, lineno: int)`` ``'address'`` memory block address ``int`` ``'traceback'`` memory block address with traceback ``(address: int, traceback)`` ===================== =================================== ================================ The ``traceback`` type is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. If *cumulative* is ``True``, cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. The *cumulative* parameter is set to ``False`` if *key_type* is ``'address'``, or if the traceback limit is less than ``2``. ``stats`` attribute: Statistics on traced Python memory, result of the ``get_stats()`` function. ``traceback_limit`` attribute: Maximum number of frames stored in the traceback of ``traces``, result of the ``get_traceback_limit()`` function. ``traces`` attribute: Traces of Python memory allocations, result of the ``get_traces()`` function, can be ``None``. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. Statistic --------- ``Statistic(key, size, size_diff, count, count_diff)`` class: Statistic on memory allocations. ``GroupedStats.compare_to()`` and ``GroupedStats.statistics()`` return a list of ``Statistic`` instances. ``key`` attribute: Key identifying the statistic. The key type depends on ``GroupedStats.key_type``, see the ``Snapshot.group_by()`` method. ``count`` attribute: Number of memory blocks (``int``). ``count_diff`` attribute: Difference of number of memory blocks (``int``). ``size`` attribute: Total size of memory blocks in bytes (``int``). ``size_diff`` attribute: Difference of total size of memory blocks in bytes (``int``). Prior Work ========== * `Python Memory Validator `_ (2005-2013): commercial Python memory validator developed by Software Verification. It uses the Python Reflection API. * `PySizer `_: Google Summer of Code 2005 project by Nick Smallbone. * `Heapy `_ (2006-2013): part of the Guppy-PE project written by Sverker Nilsson. * Draft PEP: `Support Tracking Low-Level Memory Usage in CPython `_ (Brett Canon, 2006) * Muppy: project developed in 2008 by Robert Schuppenies. * `asizeof `_: a pure Python module to estimate the size of objects by Jean Brouwers (2008). * `Heapmonitor `_: It provides facilities to size individual objects and can track all objects of certain classes. It was developed in 2008 by Ludwig Haehne. * `Pympler `_ (2008-2011): project based on asizeof, muppy and HeapMonitor * `objgraph `_ (2008-2012) * `Dozer `_: WSGI Middleware version of the CherryPy memory leak debugger, written by Marius Gedminas (2008-2013) * `Meliae `_: Python Memory Usage Analyzer developed by John A Meinel since 2009 * `caulk `_: written by Ben Timby in 2012 * `memory_profiler `_: written by Fabian Pedregosa (2011-2013) See also `Pympler Related Work `_. Links ===== tracemalloc: * `#18874: Add a new tracemalloc module to trace Python memory allocations `_ * `pytracemalloc on PyPI `_ Copyright ========= This document has been placed in the public domain. _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com From cf.natali at gmail.com Wed Oct 23 23:37:45 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Wed, 23 Oct 2013 23:37:45 +0200 Subject: [Python-Dev] pathlib (PEP 428) status Message-ID: Hi, What's the current status of pathlib? Is it targeted for 3.4? It would be a really nice addition, and AFAICT it has already been maturing a while on pypi, and discussed several times here. If I remember correctly, the only remaining issue was stat()'s result caching. cf From christian at python.org Thu Oct 24 00:38:02 2013 From: christian at python.org (Christian Heimes) Date: Thu, 24 Oct 2013 00:38:02 +0200 Subject: [Python-Dev] pathlib (PEP 428) status In-Reply-To: References: Message-ID: <52684FCA.1060904@python.org> Am 23.10.2013 23:37, schrieb Charles-Fran?ois Natali: > Hi, > > What's the current status of pathlib? Is it targeted for 3.4? > > It would be a really nice addition, and AFAICT it has already been > maturing a while on pypi, and discussed several times here. > If I remember correctly, the only remaining issue was stat()'s result caching. Hi, I'd like to see pathlib in 3.4 as well. Last week at PyCon.DE in Cologne several people have asked me about pathlib. We even had a BarCamp session about path libraries for Python. A couple of German Python users have promised to contribute doc improvements soonish. AFAIK stat caching and a os.listdir() generator with stat `recycling` (dirent->d_type) are open issues. I suggest Python 3.4 should ignore these features for now but prepare the API and documentation for future enhancements. +1 for PEP 428 Christian From ncoghlan at gmail.com Thu Oct 24 00:52:23 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 24 Oct 2013 08:52:23 +1000 Subject: [Python-Dev] About 'superobject' and descriptors In-Reply-To: References: Message-ID: On 24 Oct 2013 03:37, "hakril lse" wrote: > > Hi, > > I have a question about a choice of implementation concerning > 'superobject' with the descriptors. > > When a 'superobject' looks for a given attribute, it runs through the > mro of the object. > If it finds a descriptor, the 'superobject' calls the __get__ method > with 'starttype = su->obj_type' as third argument (in typeobject.c: > super_getattro). > > So, the 'type' argument of __get__ does not give more information > about the 'real calling type' in this case. > It seems that this is just a redundant information of inst.__class__. > > For example: > > # A.descr is a descriptor > # B inherit from A > # C inherit from B > > c = C() > c.descr > super(C, c).descr > super(B, c).descr > > In these 3 cases the __get__ method is called with the same arguments > that are : __get__(descr, c, C). > > If this behavior is really expected: Could you explain why ? because > it means that I am missing something obvious. > Because, at first sight, the 'type' argument seems to be the perfect > place to get the type of the 'real calling class'. The third argument is just there to handle the case where the instance is None (i.e. lookup directly on the class rather than an instance). Cheers, Nick. > > Thank you, > > -- > hakril > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Oct 24 01:48:51 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 24 Oct 2013 10:48:51 +1100 Subject: [Python-Dev] About 'superobject' and descriptors In-Reply-To: References: Message-ID: <20131023234850.GR7989@ando> Hi Hakril, I think this question is probably off-topic for this list. This list is for development of the Python compiler, not for development with Python, and questions like this should probably go to python-list at python.org (also mirrored as comp.lang.python if you have Usenet access). But I have a brief observation below: On Wed, Oct 23, 2013 at 07:29:03PM +0200, hakril lse wrote: > For example: > > # A.descr is a descriptor > # B inherit from A > # C inherit from B > > c = C() > c.descr > super(C, c).descr > super(B, c).descr > > In these 3 cases the __get__ method is called with the same arguments > that are : __get__(descr, c, C). > > If this behavior is really expected: Could you explain why ? because > it means that I am missing something obvious. > Because, at first sight, the 'type' argument seems to be the perfect > place to get the type of the 'real calling class'. I'm afraid I don't understand what you mean by "real calling class", if it is not the class of the instance doing the calling. I have a feeling that perhaps you think that calls to super are equivalent to "the parent of the class", and so you expect: c.desc => desc.__get__(c, C) super(C, c).desc => desc.__get__(c, B) super(B, c).desc => desc.__get__(c, A) but that would not be correct. super is equivalent to "the next class in the MRO of the instance doing the calling", and in all cases, the "real-calling instance" is c, and the "real calling class" is the class of c, namely C. Extended discussion of this should go to python-list, but I think I have the basics right. -- Steven From victor.stinner at gmail.com Thu Oct 24 03:03:21 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 24 Oct 2013 03:03:21 +0200 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: Message-ID: Hi, 2013/10/23 Kristj?n Valur J?nsson : > This might be a good place to make some comments. > I have discussed some of this in private with Victor, but wanted to make them here, for the record. Yes, I prefer to discuss the PEP on python-dev. It's nice to get more feedback, I expect to get a better API at the end! Oh, you have a lot of remarks, I will try to reply to all of them. > 1) really, all that is required in terms of data is the traceback.get_traces() function. Further, it _need_ not return addresses since they are not required for analysis. It is sufficient for it to return a list of (traceback, size, count) tuples. I understand that the get_stats function is useful for quick information so it can be kept, although it provides no added information, only convenience > 2) get_object_address() and get_trace(address) functions seem redundant. All that is required is get_object_traceback(), I think. The use case of get_traces() + get_object_trace() is to retrieve the traceback of all alive Python objects for tools like Melia, Pympler or Heapy. The only motivation is performance. I wrote a benchmark using 10^6 objects and... get_traces() x 1 + get_object_address() x N is 40% *slower* than calling get_object_traceback() x N. So get_object_traceback() is faster for this use case, especially if you don't want to the traceback of all objects, but only a few of them. Charles-Francois already asked me to remove everything related to address, so let's remove two more functions: - remove get_object_address() - remove get_trace() - get_traces() returns a list - remove 'address' key type of Snapshot.group_by() - 'traceback' key type of Snapshot.group_by() groups traces by traceback, instead of (address, traceback) => it is closer to what you suggested me privately (generate "top stats" but keep the whole traceback) > 1) really, all that is required in terms of data is the traceback.get_traces() function. Further, it _need_ not return addresses since they are not required for analysis. It is sufficient for it to return a list of (traceback, size, count) tuples. I understand that the get_stats function is useful for quick information so it can be kept, although it provides no added information, only convenience For the get_stats() question, the motivation is also performances. Let's try a benchmark on my laptop. Test 1. With the Python test suite, 467,738 traces limited to 1 frame: * take a snapshot with traces (call get_traces()): 293 ms * write the snapshot on disk: 167 ms * load the snapshot from disk: 184 ms * group by filename *using stats*: 24 ms (754 different filenames) * group by line *using stats*: 28 ms (31827 different lines) * group by traceback using traces: 333 ms (31827 different tracebacks, the traceback is limited to 1 frame) Test 2. With the Python test suite, 495,571 traces limited to 25 frame: * take a snapshot without traces (call get_stats()): 35 ms * take a snapshot with traces (call get_stats() and get_traces()): 532 ms * write the snapshot on disk: 565 ms * load the snapshot from disk: 739 ms * group by filename *using stats*: 25 ms (906 different filenames) * group by line *using stats*: 22 ms (36940 different lines) * group by traceback using traces: 786 ms (66314 different tracebacks) Test 3. tracemalloc modified to not use get_stats() anymore, only use traces. With the Python test suite, 884719 traces limited to 1 frame: * take a snapshot with traces (call get_traces()): 531 ms * write the snapshot on disk: 278 ms * load the snapshot from disk: 298 ms * group by filename *using traces*: 706 ms (1329 different filenames) * group by line *using traces*: 724 ms (55,349 different lines) * group by traceback using traces: 731 ms (55,349 different tracebacks, the traceback is limited to 1 frame) I'm surprised: it's faster than the benchmark I ran some weeks ago. Maybe I optimized something? The most critical operation, taking a snapshot takes half a second, so it's enough efficient. Let's remove even more code: - remove get_stats() - remove Snapshot.stats Snapshot.group_by() can easily recompute statistics by filename and line number from traces. (To be honest, get_stats() and get_traces() used together have an issue: they may be inconsistent if some objects are allocated between. Snapshot.apply_filters() has to apply filters on traces and stats for example. It's simpler to only manipulate traces.) > 3) set_traceback_limit(). Truncating tracebacks is bad. Particularly if it is truncated at the top end of the callstack, because then information looses cohesion, namely, the common connection point, the root. If traceback limits are required, I suggest being able to specifiy that we truncate the leaf-end of the tracebacks. If the traceback is truncated and 90% of all memory is allocated at the same Python line: I prefer to have the get the most recent frame, than the n-th function from main() which may indirectly call 100 different more functions... In this case, how do you guess which function allocated the memory? You get the same issue than Melia/Pympler/Heapy: debug data doesn't help to identify the memory leak. > 4) add_filter(). This is unnecessary. Information can be filtered on the python side. Defining Filter as a C type is not necessary. Similarly, module level filter functions can be dropped. Filters for capture are here for efficiency: attaching a trace to each memory block is expensive. I tried pybench: when using tracemalloc, Python is 2x slower. The memory usage is also doubled. Using filters, the overhead is lower. I don't have numbers for the CPU, but for the memory: ignored traces are not stored, so the memory usage is immediatly reduced. Without filters for capture, I'm not sure that it is even possible to use tracemalloc with 100 frames on a large application. Anyway, you can remove all filters: in this case, the overhead of filters is zero. > 5) Filter, Snapshot, GroupedStats, Statistics: These classes, if required, can be implemented in a .py module. Snapshot, GroupedStats and Statistics are implemented in Python. Filter is implemented in C because I want filters for the capture. > 6) Snapshot dump/load(): It is unusual to see load and save functions taking filenames in a python module, and a module implementing its own file IO. I have suggested simply to add Pickle support. Alternatively, support file-like objects or bytes (loads/dumps) In the latest implementation, load/dump is trivial: def dump(self, filename): with open(filename, "wb") as fp: pickle.dump(self, fp, pickle.HIGHEST_PROTOCOL) @staticmethod def load(filename, traces=True): with open(filename, "rb") as fp: return pickle.load(fp) http://hg.python.org/features/tracemalloc/file/85c0cefb92cb/Lib/tracemalloc.py#l164 So you can easily reimplement your own serialization function (using pickle) with your custom file-like object. I already asked Charles-Francois if he prefers to accept a file-like object as input (in filename, as open() does), but he doesn't feel the need. > I'd also like to point out (just to say "I told you so" :) ) that this module is precisely the reason I suggested we include "const char *file, int lineno" in the API for PEP 445, because that would allow us, in debug builds, to get one extra stack level, namely the position of the actual C allocation in the python source. In my experience, C functions allocating memory are wrapped in Python objects, it's easy to guess the C function from the Python traceback. Victor From ericsnowcurrently at gmail.com Thu Oct 24 08:05:10 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 24 Oct 2013 00:05:10 -0600 Subject: [Python-Dev] PEP 451 update Message-ID: I've had some offline discussion with Brett and Nick about PEP 451 which has led to some meaningful clarifications in the PEP. In the interest of pulling further discussions back onto this (archived/public) list, here's an update of what we'd discussed and where things are at. :) * path entry finders indicate that they found part of a possible namespace package by returning a spec with no loader set (but with submodule_search_locations set). Brett wanted some clarification on this. * The name/path signature and attributes of file-based finders in importlib will no longer be changing. Brett had some suggestions on the proposed change and it became clear that the the change was actually pointless. * I've asserted that there shouldn't be much difficulty in adjusting pkgutil and other modules to work with ModuleSpec. * Brett asked for clarification on whether the "load()" example from the PEP would be realized implicitly by the import machinery or explicitly as a method on ModuleSpec. This has bearing on the ability of finders to return instances of ModuleSpec subclasses or even ModuleSpec-like objects (a la duck typing). The answer is the it will not be a method on ModuleSpec, so it is effectively just part of the general import system implementation. Finders may return any object that provides the attributes of ModuleSpec. I will be updating the PEP to make these points clear. * Nick suggested writing a draft patch for the language reference changes (the import page). Such a patch will be a pretty good indicator of the impact of PEP 451 on the import system and should highlight any design flaws in the API. This is on my to-do list (hopefully by tomorrow). * Nick also suggested moving all ModuleSpec methods to a separate class that will simply make use of a separate, existing ModuleSpec instance. This will help address several issues, particularly by relaxing the constraints on what finders can return, but also by avoiding the unnecessary exposure of the methods via every module.__spec__. I plan on going with this, but currently am trying out the change to see if there are any problems I've missed. Once I feel good about it I'll update the PEP. That about sums up our discussions. I have a couple of outstanding updates to the PEP to make when I get a chance, as well as putting up a language reference patch for review. -eric From solipsis at pitrou.net Thu Oct 24 10:17:03 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 24 Oct 2013 10:17:03 +0200 Subject: [Python-Dev] pathlib (PEP 428) status References: <52684FCA.1060904@python.org> Message-ID: <20131024101703.02d22485@pitrou.net> Le Thu, 24 Oct 2013 00:38:02 +0200, Christian Heimes a ?crit : > Am 23.10.2013 23:37, schrieb Charles-Fran?ois Natali: > > Hi, > > > > What's the current status of pathlib? Is it targeted for 3.4? > > > > It would be a really nice addition, and AFAICT it has already been > > maturing a while on pypi, and discussed several times here. > > If I remember correctly, the only remaining issue was stat()'s > > result caching. > > Hi, > > I'd like to see pathlib in 3.4 as well. Last week at PyCon.DE in > Cologne several people have asked me about pathlib. We even had a > BarCamp session about path libraries for Python. A couple of German > Python users have promised to contribute doc improvements soonish. > > AFAIK stat caching and a os.listdir() generator with stat `recycling` > (dirent->d_type) are open issues. I suggest Python 3.4 should ignore > these features for now but prepare the API and documentation for > future enhancements. Thanks for the push. I have to cleanup / remove stat caching from the PEP (not sure yet which one, Nick expressed interest during the last discussion) and then polish it for pronouncement. Still, I need to find someone to pronounce on it. Regards Antoine. From nad at acm.org Thu Oct 24 11:11:46 2013 From: nad at acm.org (Ned Deily) Date: Thu, 24 Oct 2013 02:11:46 -0700 Subject: [Python-Dev] OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed Message-ID: On Tuesday, Apple released OS X 10.9 (a.k.a. Mavericks). There has already been a lot of interest in it, in part because Apple has made it available for free and has made it easy for users with 10.8, 10.7, and (most) 10.6 systems to upgrade directly to 10.9. Unfortunately, there are issues with our current maintenance releases (3.3.2 and 2.7.5) on OS X 10.9 that call for new maintenance releases as soon as possible. One is critical in that it causes the interpreter to crash when running in interactive mode (http://bugs.python.org/issue18458). The problem was due to a long-standing compatibility issue in libedit's readline compatibility layer that upstream has finally fixed and Apple has now shipped in 10.9. Because the python.org installers dynamically link to libedit, the original workaround in readline.c for the original design flaw in history indexing now causes a segfault on 10.9 when the user types in the second command interactively. Not good. Ronald devised a fix that allows readline.so at runtime to detect and work with either version of libedit so that we continue to have binary compatibility across multiple OS X releases. That fix is already out in the 3.4.0 alphas and backported to the 3.3 and 2.7 branches, awaiting release there. Just in the last 12 hours, there have been at least four duplicates of the issue reported by users. I've updated the original issue to explicitly mention 10.9, now that it is no longer under NDA, and to provide a downloadable script for inexperienced users to workaround the problem by "removing" readline.so. Presumably, as word gets out, there will be fewer duplicate issues opened but the impact will remain. The only other major 10.9-related issue that I'm aware of at the moment is one with the OS X native Tk (of course!) that can affect IDLE and other Tkinter applications. See http://bugs.python.org/issue19373 for details. The workaround for it is to run Tk in 32-bit mode. I've "engaged" the Tk project in the problem. Since it can affect all users of Tk, and not just Python ones, I'm hopeful that there will be a fix soon. Note that, for 3.4.0, we are now packaging a private copy of Tcl/Tk with the 10.6+ OS X installer to avoid the chronic problems caused by Apple not updating the buggy system Tcl/Tk and forcing users to download and install a third-party Tcl/Tk just to be able to use IDLE. With the concurrence of the release managers, I am in the process of backporting that packaging support into 3.3 and 2.7. That means any Tk fix will likely require an update to those private copies. Because of the criticality of the readline issue, we really need to get 2.7.6 and 3.3.3 out as soon as practical. Besides the Tk packaging, there are a few other 10.9 build-related issues that should go into these releases. I expect to complete testing across all releases and have everything checked-in sometime this weekend. I don't think we should wait for the Issue19373 Tk fix if it isn't available when everything else is. If necessary, we can later re-release the OS X installers with just the updated Tcl/Tk shared libraries. I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this point. But I'd like Benjamin and Georg to propose an aggressive schedule so we can get these fixes out there. -- Ned Deily, nad at acm.org From g.brandl at gmx.net Thu Oct 24 11:26:59 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 24 Oct 2013 11:26:59 +0200 Subject: [Python-Dev] OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed In-Reply-To: References: Message-ID: Am 24.10.2013 11:11, schrieb Ned Deily: > I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this > point. But I'd like Benjamin and Georg to propose an aggressive schedule so > we can get these fixes out there. > We've basically agreed to do rc releases this weekend. I don't have time the weekend of Nov 2, so 3.3.3 final would be scheduled for Nov 9. There are a few "unlimited read" issues still open; I will have a look at porting their patches tomorrow. Georg From christian at python.org Thu Oct 24 13:22:09 2013 From: christian at python.org (Christian Heimes) Date: Thu, 24 Oct 2013 13:22:09 +0200 Subject: [Python-Dev] OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed In-Reply-To: References: Message-ID: <526902E1.6000203@python.org> Am 24.10.2013 11:26, schrieb Georg Brandl: > Am 24.10.2013 11:11, schrieb Ned Deily: > >> I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this >> point. But I'd like Benjamin and Georg to propose an aggressive schedule so >> we can get these fixes out there. >> > > We've basically agreed to do rc releases this weekend. I don't have time the > weekend of Nov 2, so 3.3.3 final would be scheduled for Nov 9. There are a > few "unlimited read" issues still open; I will have a look at porting their > patches tomorrow. There seems to be a problem with the security fix "Re-seed OpenSSL's PRNG after fork": http://bugs.python.org/issue18747 http://bugs.python.org/issue19227 Perhaps it's best to disable the pthread_atfork() handler for the upcoming releases and replace it with security note in the ssl module, os.fork() and release notes. If you are using fork() and the ssl module in the same application then you must re-seed the PRNG with ssl.RAND_add() every now and then. Christian From victor.stinner at gmail.com Thu Oct 24 13:36:23 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 24 Oct 2013 13:36:23 +0200 Subject: [Python-Dev] OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed In-Reply-To: <526902E1.6000203@python.org> References: <526902E1.6000203@python.org> Message-ID: 2013/10/24 Christian Heimes : > There seems to be a problem with the security fix "Re-seed OpenSSL's > PRNG after fork": > > http://bugs.python.org/issue18747 > http://bugs.python.org/issue19227 Yes, Charles Francois warned us that adding a pthread_atfork() hook to add entropy may add a deadlock. Such change should *not* be added to stable releases, even if it is supposed to improve the security. > Perhaps it's best to disable the pthread_atfork() handler for the > upcoming releases and replace it with security note in the ssl module, > os.fork() and release notes. IMO the best place to fix the bug is in OpenSSL directly: RAND_bytes() function of OpenSSL can detect a fork using getpid() and add more entropy (in the child or maybe in the parent process). OpenSSL has access to entropy sources and knows all mutexes, whereas Python can only guess the list of mutexes (there are probably many more private mutexes). OpenSSL may use pthread_atfork() internally. http://www.openwall.com/lists/oss-security/2013/04/12/3 "I believe it is wrong to fix this in PostgreSQL. Rather, this is a bug in the OpenSSL fork protection code. It should either install a fork hook, or reseed the PRNG from /dev/urandom if a PID change is detected." I wrote a patch for OpenSSL long time ago (when working on my Hasard project, PRNG library), but I don't remember if I proposed it upstream: https://bitbucket.org/haypo/hasard/src/tip/patches/openssl_rand_fork.patch > > If you are using fork() and the ssl module in the same application then > you must re-seed the PRNG with ssl.RAND_add() every now and then. > You should explain that the issue only occurs if you fork more than 2^16 times and two child processes get the same pid. I wouldn't say that it is unlikely, but just that there is no reason to scare users if they are not vulnerable. The issue can occur if a process spawn a lot of processes. Reminder: you are supposed to only call exec() after fork(), nothing else :-) (Only signal-safe functions are officially supported between exec() and fork()). Victor From ncoghlan at gmail.com Thu Oct 24 14:01:00 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 24 Oct 2013 22:01:00 +1000 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On 24 October 2013 16:05, Eric Snow wrote: > I've had some offline discussion with Brett and Nick about PEP 451 > which has led to some meaningful clarifications in the PEP. In the > interest of pulling further discussions back onto this > (archived/public) list, here's an update of what we'd discussed and > where things are at. :) There's also the fact Guido asked Brett and I to be co-delegates for this PEP, and we accepted (we both already agree with the overall concept, so it's just a matter of hammering out the final API details). > * path entry finders indicate that they found part of a possible > namespace package by returning a spec with no loader set (but with > submodule_search_locations set). Brett wanted some clarification on > this. > * The name/path signature and attributes of file-based finders in > importlib will no longer be changing. Brett had some suggestions on > the proposed change and it became clear that the the change was > actually pointless. > * I've asserted that there shouldn't be much difficulty in adjusting > pkgutil and other modules to work with ModuleSpec. More specifically: importlib finders will still expose the previous import plugin API for backwards compatibility, so the worst case scenario is that we miss something and there's an API somewhere that doesn't accept import plugins that only provide the new API and not the old one. However, the PEP should explicitly state that any such omissions will be treated as bugs in the 3.4.x series (although we'll aim to handle them all in the initial implementation). Thanks for recording the details of the earlier off-list discussion :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Oct 24 14:14:39 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 24 Oct 2013 22:14:39 +1000 Subject: [Python-Dev] pathlib (PEP 428) status In-Reply-To: <20131024101703.02d22485@pitrou.net> References: <52684FCA.1060904@python.org> <20131024101703.02d22485@pitrou.net> Message-ID: On 24 October 2013 18:17, Antoine Pitrou wrote: > Le Thu, 24 Oct 2013 00:38:02 +0200, > Christian Heimes a ?crit : >> Am 23.10.2013 23:37, schrieb Charles-Fran?ois Natali: >> > Hi, >> > >> > What's the current status of pathlib? Is it targeted for 3.4? >> > >> > It would be a really nice addition, and AFAICT it has already been >> > maturing a while on pypi, and discussed several times here. >> > If I remember correctly, the only remaining issue was stat()'s >> > result caching. >> >> Hi, >> >> I'd like to see pathlib in 3.4 as well. Last week at PyCon.DE in >> Cologne several people have asked me about pathlib. We even had a >> BarCamp session about path libraries for Python. A couple of German >> Python users have promised to contribute doc improvements soonish. >> >> AFAIK stat caching and a os.listdir() generator with stat `recycling` >> (dirent->d_type) are open issues. I suggest Python 3.4 should ignore >> these features for now but prepare the API and documentation for >> future enhancements. > > Thanks for the push. I have to cleanup / remove stat caching from the > PEP (not sure yet which one, Nick expressed interest during the last > discussion) and then polish it for pronouncement. A clean stat caching mechanism would potentially help resolve an architectural problem with walkdir (https://pypi.python.org/pypi/walkdir), where offering stat-based filtering as a pipeline of independent iterators would significantly increase the number of stat calls compared to doing such filtering directly outside the iterator. However, since walkdir doesn't currently offer such stat-based filters, and it's unlikely I'll be adding it any time soon, deferring a stat caching design sounds like a reasonable idea to me. > Still, I need to find someone to pronounce on it. I'd offer, but I already have a scary number of things I need to get done by beta 1 :P Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at python.org Thu Oct 24 14:22:38 2013 From: christian at python.org (Christian Heimes) Date: Thu, 24 Oct 2013 14:22:38 +0200 Subject: [Python-Dev] OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed In-Reply-To: References: <526902E1.6000203@python.org> Message-ID: <5269110E.7000906@python.org> Am 24.10.2013 13:36, schrieb Victor Stinner: > IMO the best place to fix the bug is in OpenSSL directly: RAND_bytes() > function of OpenSSL can detect a fork using getpid() and add more > entropy (in the child or maybe in the parent process). OpenSSL has > access to entropy sources and knows all mutexes, whereas Python can > only guess the list of mutexes (there are probably many more private > mutexes). OpenSSL may use pthread_atfork() internally. That's not going to happen anytime soon (if ever). It doesn't solve old installation, too. You are forgetting that ssl.RAND_bytes() and RAND_pseudo_bytes() are our least concern. The problem also affects the TLS/SSL handshake and any internal OpenSSL code that needs a CPRNG. > Reminder: you are supposed to only call exec() after fork(), nothing > else :-) (Only signal-safe functions are officially supported between > exec() and fork()). How are forking web servers suppose to work if you can't use read() or write()? :) From kristjan at ccpgames.com Thu Oct 24 14:34:13 2013 From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=) Date: Thu, 24 Oct 2013 12:34:13 +0000 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: Message-ID: > -----Original Message----- > From: Victor Stinner [mailto:victor.stinner at gmail.com] > Sent: 24. okt?ber 2013 01:03 > To: Kristj?n Valur J?nsson > Cc: Python Dev > Subject: Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! > > > The use case of get_traces() + get_object_trace() is to retrieve the traceback > of all alive Python objects for tools like Melia, Pympler or Heapy. The only > motivation is performance. Well, for me, the use of get_traces() is to get the raw data so that I can perform my own analysis on it. With this data, I foresee people wanting to try to analyse this data in novel ways, as I suggested to you privately. > > I wrote a benchmark using 10^6 objects and... get_traces() x 1 + > get_object_address() x N is 40% *slower* than calling > get_object_traceback() x N. So get_object_traceback() is faster for this use > case, especially if you don't want to the traceback of all objects, but only a > few of them. I understand your desire for things to be fast, but let me just re-iterate my view that for this kind of jobs, performance is completely secondary. Memory debugging and analysis is an off-line, laboratory task. In my opinion, performance should not be driving the design of a module like this. And in particular, it should not be the only reason to write code in C that could just as well be written in .py. This is a lorry. A lorry is for moving refrigerators, on those rare occasions when you need to have refrigerators moved. It doesn't need go-faster-stripes. Well, I think I've made my point on this amply clear now, in this email and the previous, so I won't dwell on it further. > > Charles-Francois already asked me to remove everything related to address, > so let's remove two more functions: Great. > > Test 1. With the Python test suite, 467,738 traces limited to 1 frame: ... > I'm surprised: it's faster than the benchmark I ran some weeks ago. > Maybe I optimized something? The most critical operation, taking a snapshot > takes half a second, so it's enough efficient. Well, to me anything that happens in under a second is fast :) > > Let's remove even more code: > > - remove get_stats() > - remove Snapshot.stats > Removal of code is always nice :) > > > 3) set_traceback_limit(). Truncating tracebacks is bad. Particularly if it is > truncated at the top end of the callstack, because then information looses > cohesion, namely, the common connection point, the root. If traceback > limits are required, I suggest being able to specifiy that we truncate the leaf- > end of the tracebacks. > > If the traceback is truncated and 90% of all memory is allocated at the same > Python line: I prefer to have the get the most recent frame, than the n-th > function from main() which may indirectly call 100 different more functions... > In this case, how do you guess which function allocated the memory? You get > the same issue than > Melia/Pympler/Heapy: debug data doesn't help to identify the memory leak. Debugging memory leaks is not the only use case for your module. Analysing memory usage in a non-leaking application is also very important. In my work, I have been asked to reduce the memory overhead of a python application once it has started up. To do this, you need a top-down view of the application. You need to break it down from the "main" call down towards the leaves. Now, I would personally not truncate the stack, because I can afford the memory, but even if I would, for example, to hide a bunch of detail, I would want to throw away the _lower_ detals of the stack. It is unimportant to me to know if memory was allocated in ...;itertools.py;logging.py;stringutil.py but more important to know that it was allocated in main.py;databaseengine.py;enginesettings.py;... The "main" function here is the one that ties all the different allocations into one tree. If you take a tree, say a nice rowan, and truncate it by leaving only X nodes towards the leaves, you end up with a big heap of small branches. If on the other hand, you trim it so that you leave X nodes beginning at the root, you still have something resembling a tree, albeit a much coarser one. Anyway, this is not so important. I would run this with full traceback myself and truncate the tracebacks during the display stage anyway. > > > > 4) add_filter(). This is unnecessary. Information can be filtered on the > python side. Defining Filter as a C type is not necessary. Similarly, module > level filter functions can be dropped. > > Filters for capture are here for efficiency: attaching a trace to each memory > block is expensive. I tried pybench: when using tracemalloc, Python is 2x > slower. The memory usage is also doubled. Using filters, the overhead is > lower. I don't have numbers for the CPU, but for the > memory: ignored traces are not stored, so the memory usage is immediatly > reduced. Without filters for capture, I'm not sure that it is even possible to > use tracemalloc with 100 frames on a large application. > > Anyway, you can remove all filters: in this case, the overhead of filters is > zero. > > > > > 6) Snapshot dump/load(): It is unusual to see load and save functions > > taking filenames in a python module, and a module implementing its own > > file IO. I have suggested simply to add Pickle support. > > Alternatively, support file-like objects or bytes (loads/dumps) > > In the latest implementation, load/dump is trivial: > > def dump(self, filename): > with open(filename, "wb") as fp: > pickle.dump(self, fp, pickle.HIGHEST_PROTOCOL) > > @staticmethod > def load(filename, traces=True): > with open(filename, "rb") as fp: > return pickle.load(fp) > What does the "traces" argument do in the load() function then? Anyway, in this case, dump and load can be thought of as convenience functions. That's perfectly fine from my viewpoint. > > > I'd also like to point out (just to say "I told you so" :) ) that this module is > precisely the reason I suggested we include "const char *file, int lineno" in > the API for PEP 445, because that would allow us, in debug builds, to get one > extra stack level, namely the position of the actual C allocation in the python > source. > > In my experience, C functions allocating memory are wrapped in Python > objects, it's easy to guess the C function from the Python traceback. Often, yes. But there are big black boxes that remain. The most numerous of those are those big mysterious allocations that can happen as a result of "import mymodule" But apart from that, a lot of code can have unforeseen side effects, like growing some internal list, or other. This sort of information helps with understanding that. Not that we are likely to change PEP 445 at this stage, but this was the use case for my suggestion. Cheers, Kristj?n From ncoghlan at gmail.com Thu Oct 24 14:43:44 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 24 Oct 2013 22:43:44 +1000 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: Message-ID: On 24 October 2013 22:34, Kristj?n Valur J?nsson wrote: > I understand your desire for things to be fast, but let me just re-iterate my view > that for this kind of jobs, performance is completely secondary. Memory > debugging and analysis is an off-line, laboratory task. In my opinion, > performance should not be driving the design of a module like this. And in > particular, it should not be the only reason to write code in C that could > just as well be written in .py. > This is a lorry. A lorry is for moving refrigerators, on those rare occasions when > you need to have refrigerators moved. It doesn't need go-faster-stripes. > > Well, I think I've made my point on this amply clear now, in this email and the > previous, so I won't dwell on it further. Not everything is a PC that you can just add more memory to (or switch to a lab server with the same CPU architecture but more RAM). If Victor were only interested in analysing x86[_64] software, I'd agree with you, but embedded scenarios don't always offer that freedom to do resource consumption analysis on a more powerful system. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Thu Oct 24 14:52:52 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 24 Oct 2013 14:52:52 +0200 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: Message-ID: 2013/10/24 Kristj?n Valur J?nsson : >> Test 1. With the Python test suite, 467,738 traces limited to 1 frame: > ... >> I'm surprised: it's faster than the benchmark I ran some weeks ago. >> Maybe I optimized something? The most critical operation, taking a snapshot >> takes half a second, so it's enough efficient. > > Well, to me anything that happens in under a second is fast :) In a previous version, taking a snapshot and computing the "top 10" took up to 30 seconds. So it was justified to only store "stats" (which is much smaller). > (...) To do this, you need a top-down view of the application. You > need to break it down from the "main" call down towards the leaves. We don't have to agree, I may add an option to decide which frames are stored. I have to check how truncated traceback would fit with the current API, especially with Snapshot.group_by('line'). Which API do you propose to decide which frames are kept (most recent frames or oldest frames)? It may be a new parameter of set_traceback_limit() for example. I have an idea how to implement it in C. But I'm not convinced yet that there is a need for it :-) > Anyway, this is not so important. I would run this with full traceback myself and truncate > the tracebacks during the display stage anyway. FYI tracemalloc has currently has arbitrary limit for 100 frames for a technical reason: traces are stored in the stack in some C functions. It should be possible to drop the limitation (without killing performances). But storing 100 frames is already very slow. Performances depends directly on the number of frames, because the whole traceback is compared at each memory allocation to check if the traceback is new or known in a hash table. >> @staticmethod >> def load(filename, traces=True): >> with open(filename, "rb") as fp: >> return pickle.load(fp) >> > > What does the "traces" argument do in the load() function then? A ghost from the past :-) I will remove it. > Often, yes. But there are big black boxes that remain. The most numerous > of those are those big mysterious allocations that can happen as a > result of > "import mymodule" Yes, tracemalloc is not perfect. You may for example find a huge allocation of 768 KB in a random "import module", it's the dictionary of interned Unicode strings... It's hard to understand why "import module" allocated 768 KB (and depending on the file modification time, the allocation may occur somewhere else...). > Not that we are likely to change PEP 445 at this stage, but this was the use > case for my suggestion. In tracemalloc, a traceback is a tuple of (str, int): you can probably hack the the malloc API and tracemalloc to add C filename and line number. (But I don't want to do it, because the Python traceback is enough in my opinion.) Victor From skip at pobox.com Thu Oct 24 15:38:52 2013 From: skip at pobox.com (Skip Montanaro) Date: Thu, 24 Oct 2013 08:38:52 -0500 Subject: [Python-Dev] OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed In-Reply-To: <5269110E.7000906@python.org> References: <526902E1.6000203@python.org> <5269110E.7000906@python.org> Message-ID: Kind of on a tangent (and I suspect I know what the answer to my question will be), but here at work they have asked people who use Macs not to upgrade to Mavericks until Apple fixes one or more networking problems which make it basically unusable, especially when connecting to our VPN. I realize y'all will be busy trying to fix these various problems as they relate to Python, but might it not be better to simply tell people to hold off on the Python installers until Apple gets their act together? Otherwise, I fear you'll just be trying to hit a moving target. Skip From barry at python.org Thu Oct 24 15:44:36 2013 From: barry at python.org (Barry Warsaw) Date: Thu, 24 Oct 2013 09:44:36 -0400 Subject: [Python-Dev] OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed In-Reply-To: References: Message-ID: <20131024094436.230220bf@anarchist> On Oct 24, 2013, at 02:11 AM, Ned Deily wrote: >I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this >point. But I'd like Benjamin and Georg to propose an aggressive schedule so >we can get these fixes out there. Does this problem affect 2.6? 2.6.9 final is scheduled for Monday, so if there's something we need to get in before then, please let me know asap. Cheers, -Barry From kristjan at ccpgames.com Thu Oct 24 16:40:25 2013 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 24 Oct 2013 14:40:25 +0000 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: Message-ID: > -----Original Message----- > From: Nick Coghlan [mailto:ncoghlan at gmail.com] > Sent: 24. okt?ber 2013 12:44 > To: Kristj?n Valur J?nsson > Cc: Python Dev > Subject: Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! > Not everything is a PC that you can just add more memory to (or switch to a > lab server with the same CPU architecture but more RAM). > > If Victor were only interested in analysing x86[_64] software, I'd agree with > you, but embedded scenarios don't always offer that freedom to do > resource consumption analysis on a more powerful system. > Indeed not. In fact, I was faced with the same problem when developing for the PS3. My solution was to not do it. The memory profiler running on the PS3 performs no analysis whatsoever. For every operation (malloc/realloc/free) it simply records the address and the traceback and sends it along its merry way to a server which is listening on a TCP or UDP port If anyone is interested in adding that functionality to tracemalloc, I can contribute my code as an example. A corresponding server is a pure-python affair. An added benefit of a client-server approach is that there the memory profiling tool is non-intrusive (apart from slowing down the execution either due to cpu or network blockage) and so has to take no special steps to exclude itself from the profiling. K From python at mrabarnett.plus.com Thu Oct 24 17:32:30 2013 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 24 Oct 2013 16:32:30 +0100 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: Message-ID: <52693D8E.4050306@mrabarnett.plus.com> On 24/10/2013 15:40, Kristj?n Valur J?nsson wrote: > > >> -----Original Message----- >> From: Nick Coghlan [mailto:ncoghlan at gmail.com] >> Sent: 24. okt?ber 2013 12:44 >> To: Kristj?n Valur J?nsson >> Cc: Python Dev >> Subject: Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! >> Not everything is a PC that you can just add more memory to (or switch to a >> lab server with the same CPU architecture but more RAM). >> >> If Victor were only interested in analysing x86[_64] software, I'd agree with >> you, but embedded scenarios don't always offer that freedom to do >> resource consumption analysis on a more powerful system. >> > Indeed not. > In fact, I was faced with the same problem when developing for the PS3. > My solution was to not do it. The memory profiler running on the PS3 > performs no analysis whatsoever. For every operation (malloc/realloc/free) it > simply records the address and the traceback and sends it along its merry way > to a server which is listening on a TCP or UDP port > > If anyone is interested in adding that functionality to tracemalloc, I can contribute > my code as an example. > A corresponding server is a pure-python affair. > > An added benefit of a client-server approach is that there the memory profiling > tool is non-intrusive (apart from slowing down the execution either due to > cpu or network blockage) and so has to take no special steps to exclude itself from the profiling. > When I was looking for memory leaks in the regex module I simply wrote all of the allocations, reallocations and deallocations to a log file and then parsed it afterwards using a Python script. Simple, but effective. From victor.stinner at gmail.com Thu Oct 24 18:44:58 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 24 Oct 2013 18:44:58 +0200 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: <52693D8E.4050306@mrabarnett.plus.com> References: <52693D8E.4050306@mrabarnett.plus.com> Message-ID: > When I was looking for memory leaks in the regex module I simply wrote all of the allocations, reallocations and deallocations to a log file and then parsed it afterwards using a Python script. Simple, but effective. He he, it's funny because you described exactly my first implementation of tracemalloc! I wrote output using fprintf() into a text file. The parser was written in Python but too slow to be used it in practice. When running on the slow set top box, it took minutes (5 maybe 10) to analyze the file. Transfering the file to a PC took also minutes: the file was very large (maybe 1 GB, I don't remember) and SimpleHTTPServer too slow for the transfer. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From cf.natali at gmail.com Thu Oct 24 19:09:53 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Thu, 24 Oct 2013 19:09:53 +0200 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: Message-ID: 2013/10/24 Kristj?n Valur J?nsson : > > Now, I would personally not truncate the stack, because I can afford the memory, > but even if I would, for example, to hide a bunch of detail, I would want to throw away > the _lower_ detals of the stack. It is unimportant to me to know if memory was > allocated in > ...;itertools.py;logging.py;stringutil.py > but more important to know that it was allocated in > main.py;databaseengine.py;enginesettings.py;... Well, maybe to you, but if you look at valgrind for example, it keeps the top of the stack: and it makes a lot of sense to me, since otherwise you won't be able to find where the leak occurred. Anyway, since the stack depth is a tunable parameter, this shouldn't be an issue in practice: just save the whole stack. 2013/10/24 MRAB : > > When I was looking for memory leaks in the regex module I simply wrote all > of the allocations, reallocations and deallocations to a log file and then > parsed it afterwards using a Python script. Simple, but effective. We've all done that ;-) > 1) really, all that is required in terms of data is the traceback.get_traces() function. Further, it _need_ not return addresses since they are not required for analysis. It is sufficient for it to return a list of (traceback, size, count) tuples. Sure. Since the beginning, I'm also leaning towards a minimal API, and let third-party tools do the analysis. It makes a lot of sense, since some people will want just basic snapshot information, some others will want to compute various statistics, some others will want to display the result in a GUI... But OTOT, it would be too bad not to ship the stdlib with a basic tool to process data, to as to make it usable out-of-the box. And in this regard, we should probably mimick what's done for CPU profiling: there are both low-level profiling data gathering infrastructure (profile and cProfile), but there's also a pstats.Stats class allowing basic operations/display on this raw data. That's IMO a reasonable balance. cf From benjamin at python.org Thu Oct 24 19:56:36 2013 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 24 Oct 2013 13:56:36 -0400 Subject: [Python-Dev] OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed In-Reply-To: References: Message-ID: 2013/10/24 Georg Brandl : > Am 24.10.2013 11:11, schrieb Ned Deily: > >> I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this >> point. But I'd like Benjamin and Georg to propose an aggressive schedule so >> we can get these fixes out there. >> > > We've basically agreed to do rc releases this weekend. I don't have time the > weekend of Nov 2, so 3.3.3 final would be scheduled for Nov 9. There are a > few "unlimited read" issues still open; I will have a look at porting their > patches tomorrow. What Georg said, but I'll probably be around Nov 2. -- Regards, Benjamin From nad at acm.org Thu Oct 24 22:12:55 2013 From: nad at acm.org (Ned Deily) Date: Thu, 24 Oct 2013 13:12:55 -0700 Subject: [Python-Dev] 2.6.9 readline [Was: OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed] References: <20131024094436.230220bf@anarchist> Message-ID: In article <20131024094436.230220bf at anarchist>, Barry Warsaw wrote: > On Oct 24, 2013, at 02:11 AM, Ned Deily wrote: > >I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this > >point. But I'd like Benjamin and Georg to propose an aggressive schedule so > >we can get these fixes out there. > Does this problem affect 2.6? 2.6.9 final is scheduled for Monday, so if > there's something we need to get in before then, please let me know asap. Yes, this problem also affects 2.6. There are some mitigating factors. The support for libedit on OS X is only enabled when building for an OS X 10.5 or later ABI because in earlier releases, the readline emulation of libedit was judged too buggy. In 2.6 as is also the case in 2.7, when building from source, ./configure defaults to using a 10.4 ABI unless certain universal build options are selected or the user explicitly sets MACOSX_DEPLOYMENT_TARGET=10.5 or higher when running configure. With the default 10.4 setting, readline.so fails to build so there is no crash - no readline features, either. Also, if one supplies a version of GNU readline (which Apple does not ship) as many people do, there is also no crash. The 2.7 change of Issue18458 (http://hg.python.org/cpython/rev/1e03fd72e116) depends on some previous 2.7-only changes in Modules/readline.c so it does not apply cleanly to 2.6. However, there aren't *that* many other earlier changes to 2.7 readline that are not in 2.6 and, for the most part, they have to do with fixing memory leaks, including some introduced by using newer versions of GNU readline, and I didn't see any new features. Copying the current 2.7 tip of readline.c over to the current tip of 2.6 builds cleanly on 10.9. test_readline passes and the simple history scrolling seems to work without crashing. I'd recommend either doing nothing or backporting everything. -- Ned Deily, nad at acm.org From barry at python.org Thu Oct 24 22:40:22 2013 From: barry at python.org (Barry Warsaw) Date: Thu, 24 Oct 2013 16:40:22 -0400 Subject: [Python-Dev] 2.6.9 readline [Was: OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed] In-Reply-To: References: <20131024094436.230220bf@anarchist> Message-ID: <20131024164022.22963098@anarchist> On Oct 24, 2013, at 01:12 PM, Ned Deily wrote: >Yes, this problem also affects 2.6. There are some mitigating factors. The >support for libedit on OS X is only enabled when building for an OS X 10.5 or >later ABI because in earlier releases, the readline emulation of libedit was >judged too buggy. In 2.6 as is also the case in 2.7, when building from >source, ./configure defaults to using a 10.4 ABI unless certain universal >build options are selected or the user explicitly sets >MACOSX_DEPLOYMENT_TARGET=10.5 or higher when running configure. With the >default 10.4 setting, readline.so fails to build so there is no crash - no >readline features, either. Also, if one supplies a version of GNU readline >(which Apple does not ship) as many people do, there is also no crash. > >The 2.7 change of Issue18458 (http://hg.python.org/cpython/rev/1e03fd72e116) >depends on some previous 2.7-only changes in Modules/readline.c so it does not >apply cleanly to 2.6. However, there aren't *that* many other earlier changes >to 2.7 readline that are not in 2.6 and, for the most part, they have to do >with fixing memory leaks, including some introduced by using newer versions of >GNU readline, and I didn't see any new features. Copying the current 2.7 tip >of readline.c over to the current tip of 2.6 builds cleanly on 10.9. >test_readline passes and the simple history scrolling seems to work without >crashing. I'd recommend either doing nothing or backporting everything. Thanks for the background Ned. Given that 2.6.9 will be the last release of the 2.6 series, we won't have much of a chance of going back to fix things if we/I mess it up. Our options then are to do another release candidate, which frankly will probably get about as much testing as the first one (i.e. close to zero, afaict) and push the final release out a few weeks. Or, as you say we can just do nothing. Maybe that means 2.6 will be broken on OS X 10.9 unless someone crafts and applies patches themselves, but I think that's acceptable given how old 2.6 is (EOL) and how new 10.9 is. If 10.9 had come out two weeks from now, "do nothing" would have been the obvious answer. :) So unless I hear strong objections before next Monday, I am not going to fix this in 2.6.9. Cheers, -Barry From brett at python.org Thu Oct 24 22:40:46 2013 From: brett at python.org (Brett Cannon) Date: Thu, 24 Oct 2013 16:40:46 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Close #19379: Lazily import linecache in the warnings module, to make startup In-Reply-To: <3d5Kgk0cVGz7LpV@mail.python.org> References: <3d5Kgk0cVGz7LpV@mail.python.org> Message-ID: The diff doesn't show an addition to BENCH_GROUPS. Maybe apps? On Thu, Oct 24, 2013 at 4:24 PM, antoine.pitrou wrote: > http://hg.python.org/cpython/rev/8939c0196990 > changeset: 86603:8939c0196990 > user: Antoine Pitrou > date: Thu Oct 24 22:23:42 2013 +0200 > summary: > Close #19379: Lazily import linecache in the warnings module, to make > startup with warnings faster until a warning gets printed. > > files: > Lib/warnings.py | 6 ++---- > Misc/NEWS | 3 +++ > 2 files changed, 5 insertions(+), 4 deletions(-) > > > diff --git a/Lib/warnings.py b/Lib/warnings.py > --- a/Lib/warnings.py > +++ b/Lib/warnings.py > @@ -1,9 +1,5 @@ > """Python part of the warnings subsystem.""" > > -# Note: function level imports should *not* be used > -# in this module as it may cause import lock deadlock. > -# See bug 683658. > -import linecache > import sys > > __all__ = ["warn", "showwarning", "formatwarning", "filterwarnings", > @@ -21,6 +17,7 @@ > > def formatwarning(message, category, filename, lineno, line=None): > """Function to format a warning the standard way.""" > + import linecache > s = "%s:%s: %s: %s\n" % (filename, lineno, category.__name__, > message) > line = linecache.getline(filename, lineno) if line is None else line > if line: > @@ -233,6 +230,7 @@ > > # Prime the linecache for formatting, in case the > # "file" is actually in a zipfile or something. > + import linecache > linecache.getlines(filename, module_globals) > > if action == "error": > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -21,6 +21,9 @@ > Library > ------- > > +- Issue #19379: Lazily import linecache in the warnings module, to make > + startup with warnings faster until a warning gets printed. > + > - Issue #19327: Fixed the working of regular expressions with too big > charset. > > - Issue #17400: New 'is_global' attribute for ipaddress to tell if an > address > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Oct 24 23:54:35 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 25 Oct 2013 07:54:35 +1000 Subject: [Python-Dev] 2.6.9 readline [Was: OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed] In-Reply-To: <20131024164022.22963098@anarchist> References: <20131024094436.230220bf@anarchist> <20131024164022.22963098@anarchist> Message-ID: On 25 Oct 2013 06:42, "Barry Warsaw" wrote: > > On Oct 24, 2013, at 01:12 PM, Ned Deily wrote: > > >Yes, this problem also affects 2.6. There are some mitigating factors. The > >support for libedit on OS X is only enabled when building for an OS X 10.5 or > >later ABI because in earlier releases, the readline emulation of libedit was > >judged too buggy. In 2.6 as is also the case in 2.7, when building from > >source, ./configure defaults to using a 10.4 ABI unless certain universal > >build options are selected or the user explicitly sets > >MACOSX_DEPLOYMENT_TARGET=10.5 or higher when running configure. With the > >default 10.4 setting, readline.so fails to build so there is no crash - no > >readline features, either. Also, if one supplies a version of GNU readline > >(which Apple does not ship) as many people do, there is also no crash. > > > >The 2.7 change of Issue18458 ( http://hg.python.org/cpython/rev/1e03fd72e116) > >depends on some previous 2.7-only changes in Modules/readline.c so it does not > >apply cleanly to 2.6. However, there aren't *that* many other earlier changes > >to 2.7 readline that are not in 2.6 and, for the most part, they have to do > >with fixing memory leaks, including some introduced by using newer versions of > >GNU readline, and I didn't see any new features. Copying the current 2.7 tip > >of readline.c over to the current tip of 2.6 builds cleanly on 10.9. > >test_readline passes and the simple history scrolling seems to work without > >crashing. I'd recommend either doing nothing or backporting everything. > > Thanks for the background Ned. > > Given that 2.6.9 will be the last release of the 2.6 series, we won't have > much of a chance of going back to fix things if we/I mess it up. > > Our options then are to do another release candidate, which frankly will > probably get about as much testing as the first one (i.e. close to zero, > afaict) and push the final release out a few weeks. Or, as you say we can > just do nothing. Maybe that means 2.6 will be broken on OS X 10.9 unless > someone crafts and applies patches themselves, but I think that's acceptable > given how old 2.6 is (EOL) and how new 10.9 is. > > If 10.9 had come out two weeks from now, "do nothing" would have been the > obvious answer. :) So unless I hear strong objections before next Monday, I am > not going to fix this in 2.6.9. Since the default build settings work, that sounds reasonable. Perhaps include a note somewhere that targeting a more recent ABI may involve copying the 2.7 readline.c? Cheers, Nick. > > Cheers, > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Oct 25 00:07:04 2013 From: barry at python.org (Barry Warsaw) Date: Thu, 24 Oct 2013 18:07:04 -0400 Subject: [Python-Dev] 2.6.9 readline [Was: OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed] In-Reply-To: References: <20131024094436.230220bf@anarchist> <20131024164022.22963098@anarchist> Message-ID: <20131024180704.568f9be3@anarchist> On Oct 25, 2013, at 07:54 AM, Nick Coghlan wrote: >Since the default build settings work, that sounds reasonable. Perhaps >include a note somewhere that targeting a more recent ABI may involve >copying the 2.7 readline.c? I'll probably add issue 18458 to the announcement and release page. That way we can still use the tracker to manage patches for 2.6, even though they won't be officially applied. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From tjreedy at udel.edu Fri Oct 25 00:57:57 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 24 Oct 2013 18:57:57 -0400 Subject: [Python-Dev] Python 3 to be default in Fedora 22 Message-ID: Progess on the Python3 as 'standard Python' front: Yesterday, FESCO (Fedora Steering Committee?) approved a proposal to make Python 3 (version unspecified, I presume 3.3 or 3.4) the default Python in Fedora 22 (I did not see a date anywhere). http://lwn.net/Articles/571528/ https://fedoraproject.org/wiki/Changes/Python_3_as_Default -- Terry Jan Reedy From ncoghlan at gmail.com Fri Oct 25 05:32:36 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 25 Oct 2013 13:32:36 +1000 Subject: [Python-Dev] Python 3 to be default in Fedora 22 In-Reply-To: References: Message-ID: On 25 Oct 2013 09:02, "Terry Reedy" wrote: > > Progess on the Python3 as 'standard Python' front: Yesterday, FESCO (Fedora Steering Committee?) approved a proposal to make Python 3 (version unspecified, I presume 3.3 or 3.4) the default Python in Fedora 22 (I did not see a date anywhere). The current release cycle is F20, so this is aiming for late 2014 (and hence will be 3.4). > http://lwn.net/Articles/571528/ > https://fedoraproject.org/wiki/Changes/Python_3_as_Default Note that unlike Arch, the Fedora devs currently plan to leave "/usr/bin/python" referring to Python 2 (see the "User Experience" part of the proposal). Cheers, Nick. > > -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Fri Oct 25 08:42:40 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 25 Oct 2013 00:42:40 -0600 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Thu, Oct 24, 2013 at 12:05 AM, Eric Snow wrote: > * Nick suggested writing a draft patch for the language reference > changes (the import page). Such a patch will be a pretty good > indicator of the impact of PEP 451 on the import system and should > highlight any design flaws in the API. This is on my to-do list > (hopefully by tomorrow). Okay, I've posted the patch to http://bugs.python.org/issue18864. I'll get to the other open issues in the next couple days. -eric From ericsnowcurrently at gmail.com Fri Oct 25 08:44:35 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 25 Oct 2013 00:44:35 -0600 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Thu, Oct 24, 2013 at 6:01 AM, Nick Coghlan wrote: > On 24 October 2013 16:05, Eric Snow wrote: >> * I've asserted that there shouldn't be much difficulty in adjusting >> pkgutil and other modules to work with ModuleSpec. > > More specifically: importlib finders will still expose the previous > import plugin API for backwards compatibility, so the worst case > scenario is that we miss something and there's an API somewhere that > doesn't accept import plugins that only provide the new API and not > the old one. > > However, the PEP should explicitly state that any such omissions will > be treated as bugs in the 3.4.x series (although we'll aim to handle > them all in the initial implementation). Good point. I'll add that. > > Thanks for recording the details of the earlier off-list discussion :) :) -eric From brett at python.org Fri Oct 25 17:40:50 2013 From: brett at python.org (Brett Cannon) Date: Fri, 25 Oct 2013 11:40:50 -0400 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Thu, Oct 24, 2013 at 2:05 AM, Eric Snow wrote: > I've had some offline discussion with Brett and Nick about PEP 451 > which has led to some meaningful clarifications in the PEP. In the > interest of pulling further discussions back onto this > (archived/public) list, here's an update of what we'd discussed and > where things are at. :) > > * path entry finders indicate that they found part of a possible > namespace package by returning a spec with no loader set (but with > submodule_search_locations set). Brett wanted some clarification on > this. > * The name/path signature and attributes of file-based finders in > importlib will no longer be changing. Brett had some suggestions on > the proposed change and it became clear that the the change was > actually pointless. > * I've asserted that there shouldn't be much difficulty in adjusting > pkgutil and other modules to work with ModuleSpec. > * Brett asked for clarification on whether the "load()" example from > the PEP would be realized implicitly by the import machinery or > explicitly as a method on ModuleSpec. This has bearing on the ability > of finders to return instances of ModuleSpec subclasses or even > ModuleSpec-like objects (a la duck typing). The answer is the it will > not be a method on ModuleSpec, so it is effectively just part of the > general import system implementation. Finders may return any object > that provides the attributes of ModuleSpec. I will be updating the > PEP to make these points clear. > > * Nick suggested writing a draft patch for the language reference > changes (the import page). Such a patch will be a pretty good > indicator of the impact of PEP 451 on the import system and should > highlight any design flaws in the API. This is on my to-do list > (hopefully by tomorrow). > Eric's got an initial patch for this up on http://bugs.python.org/issue18864. > * Nick also suggested moving all ModuleSpec methods to a separate > class that will simply make use of a separate, existing ModuleSpec > instance. This will help address several issues, particularly by > relaxing the constraints on what finders can return, but also by > avoiding the unnecessary exposure of the methods via every > module.__spec__. I plan on going with this, but currently am trying > out the change to see if there are any problems I've missed. Once I > feel good about it I'll update the PEP. > > That about sums up our discussions. I have a couple of outstanding > updates to the PEP to make when I get a chance, as well as putting up > a language reference patch for review. > After reading Eric's doc patch, I realized there is one change I want to make to the current semantics and that's not to backfill __package__ when set to None. Since import is now going to take over the job of setting __package__ (along with other attributes), this seems like a slight waste of effort. It also kills (or at least complicates) having a loader which does lazy loading since reading the attribute to see if it is None would trigger the load before leaving the import code, thus killing any postponed loading. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Fri Oct 25 17:44:48 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 25 Oct 2013 09:44:48 -0600 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Fri, Oct 25, 2013 at 9:40 AM, Brett Cannon wrote: > After reading Eric's doc patch, I realized there is one change I want to > make to the current semantics and that's not to backfill __package__ when > set to None. Since import is now going to take over the job of setting > __package__ (along with other attributes), this seems like a slight waste of > effort. It also kills (or at least complicates) having a loader which does > lazy loading since reading the attribute to see if it is None would trigger > the load before leaving the import code, thus killing any postponed loading. Fine with me. I may have time today to make the outstanding updates to the PEP. -eric From a.badger at gmail.com Fri Oct 25 17:54:30 2013 From: a.badger at gmail.com (Toshio Kuratomi) Date: Fri, 25 Oct 2013 08:54:30 -0700 Subject: [Python-Dev] Python 3 to be default in Fedora 22 In-Reply-To: References: Message-ID: <20131025155430.GA9506@unaka.lan> On Fri, Oct 25, 2013 at 01:32:36PM +1000, Nick Coghlan wrote: > > On 25 Oct 2013 09:02, "Terry Reedy" wrote: > > > http://lwn.net/Articles/571528/ > > https://fedoraproject.org/wiki/Changes/Python_3_as_Default > > Note that unlike Arch, the Fedora devs currently plan to leave "/usr/bin/ > python" referring to Python 2 (see the "User Experience" part of the proposal). > The tangible changes for this are just that we're hoping to only have python3, not python2 on our default LiveCD and cloud images. This has been a bit hard since many of our core packaging tools (and the large number of release engineering, package-maintainer, distro installer, etc scripts built on top of them) were written in python2. The F22 release is hoping to have a set of C libraries for those tools with both python3 and python2 bindings. That will hopefully allow us to port the user-visible tools (installer and things present on the selected images) to python3 for F22 while leaving the release-engineering and packager-oriented scripts until a later Fedora release. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From status at bugs.python.org Fri Oct 25 18:07:37 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 25 Oct 2013 18:07:37 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20131025160737.52D8656A27@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-10-18 - 2013-10-25) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4197 (+13) closed 26965 (+93) total 31162 (+106) Open issues with patches: 1955 Issues opened (55) ================== #18777: Cannot compile _ssl.c using openssl > 1.0 http://bugs.python.org/issue18777 reopened by christian.heimes #18999: Support different contexts in multiprocessing http://bugs.python.org/issue18999 reopened by sbt #19286: error: can't copy '': doesn't exist or not a regular http://bugs.python.org/issue19286 opened by jason.coombs #19290: Clarify compile and eval interaction. http://bugs.python.org/issue19290 opened by terry.reedy #19291: Add docs for asyncio package (Tulip, PEP 3156) http://bugs.python.org/issue19291 opened by gvanrossum #19292: Make SSLContext.set_default_verify_paths() work on Windows http://bugs.python.org/issue19292 opened by gvanrossum #19296: Compiler warning when compiling dbm module http://bugs.python.org/issue19296 opened by vajrasky #19298: Use __attribute__(cleanup ...) to detect refleaks http://bugs.python.org/issue19298 opened by brett.cannon #19302: Add support for AIX pollset efficient event I/O polling http://bugs.python.org/issue19302 opened by David.Edelsohn #19308: Tools/gdb/libpython.py does not support GDB linked against Pyt http://bugs.python.org/issue19308 opened by dcoles #19316: devguide: compiler - wording http://bugs.python.org/issue19316 opened by numerodix #19317: ctypes.util.find_library should examine binary's RPATH on Sola http://bugs.python.org/issue19317 opened by automatthias #19320: Tkinter tests ran with wantobjects is false http://bugs.python.org/issue19320 opened by serhiy.storchaka #19325: _osx_support imports many modules http://bugs.python.org/issue19325 opened by christian.heimes #19328: Improve PBKDF2 documentation http://bugs.python.org/issue19328 opened by christian.heimes #19329: Faster compiling of charset regexpes http://bugs.python.org/issue19329 opened by serhiy.storchaka #19330: Use public classes for contextlib.suppress and redirect_stdout http://bugs.python.org/issue19330 opened by ncoghlan #19331: Revise PEP 8 recommendation for class names http://bugs.python.org/issue19331 opened by barry #19332: Guard against changing dict during iteration http://bugs.python.org/issue19332 opened by serhiy.storchaka #19333: distutils.util.grok_environment_error loses the error message http://bugs.python.org/issue19333 opened by mgedmin #19334: test_asyncio hanging for 1 hour (non-AIX version) http://bugs.python.org/issue19334 opened by gvanrossum #19335: codeop misclassifies incomplete code with 'nonlocal' http://bugs.python.org/issue19335 opened by Rosuav #19336: No API to get events from epoll without allocating a list http://bugs.python.org/issue19336 opened by alex #19338: multiprocessing: sys.exit() from a child with a non-int exit c http://bugs.python.org/issue19338 opened by brodie #19339: telnetlib: time.monotonic() should be used instead of time.tim http://bugs.python.org/issue19339 opened by haypo #19342: Improve grp module docstrings http://bugs.python.org/issue19342 opened by mgedmin #19343: Expose FreeBSD-specific APIs in resource module http://bugs.python.org/issue19343 opened by koobs #19346: Build fails when there are no shared extensions to be built http://bugs.python.org/issue19346 opened by alanh #19347: PEP 453 implementation tracking issue http://bugs.python.org/issue19347 opened by ncoghlan #19348: Building _testcapimodule as a builtin results in compile error http://bugs.python.org/issue19348 opened by alanh #19349: Not so correct exception message when running venv http://bugs.python.org/issue19349 opened by vajrasky #19351: python msi installers - silent mode http://bugs.python.org/issue19351 opened by Jason.Bray #19353: on RHEL6 simple build fails with test_linux_constants (test.te http://bugs.python.org/issue19353 opened by mcepl #19354: test_format fails on RHEL-6 http://bugs.python.org/issue19354 opened by skrah #19355: Initial modernization of OpenWatcom support http://bugs.python.org/issue19355 opened by Jeffrey.Armstrong #19358: Integrate "Argument Clinic" into CPython build http://bugs.python.org/issue19358 opened by larry #19359: reversed() does not work with weakref.proxy of sequences http://bugs.python.org/issue19359 opened by abingham #19361: Specialize exceptions thrown by JSON parser http://bugs.python.org/issue19361 opened by musically_ut #19362: Documentation for len() fails to mention that it works on sets http://bugs.python.org/issue19362 opened by Gareth.Rees #19363: Python 2.7's future_builtins.map is not compatible with Python http://bugs.python.org/issue19363 opened by Gareth.Rees #19367: IDLE wont open http://bugs.python.org/issue19367 opened by MrE.Merry #19371: timing test too tight http://bugs.python.org/issue19371 opened by pitrou #19372: getaddrinfo() bug http://bugs.python.org/issue19372 opened by Daniel #19373: Tkinter apps including IDLE may not display properly on OS X 1 http://bugs.python.org/issue19373 opened by ned.deily #19375: Deprecate site-python in site.py http://bugs.python.org/issue19375 opened by pitrou #19376: document that strptime() does not support the Feb 29 if the fo http://bugs.python.org/issue19376 opened by Matthew.Earl #19378: Rename "line_offset" parameter in dis.get_instructions to "fir http://bugs.python.org/issue19378 opened by ncoghlan #19380: Optimize parsing of regular expressions http://bugs.python.org/issue19380 opened by serhiy.storchaka #19382: tabnanny unit tests http://bugs.python.org/issue19382 opened by Colin.Williams #19383: nturl2path test coverage http://bugs.python.org/issue19383 opened by Colin.Williams #19385: dbm.dumb should be consistent when the database is closed http://bugs.python.org/issue19385 opened by Claudiu.Popa #19386: selectors test_interrupted_retry is flaky http://bugs.python.org/issue19386 opened by christian.heimes #19387: Explain and test the sre overlap table http://bugs.python.org/issue19387 opened by pitrou #19389: find_file might return a different directory than the tested f http://bugs.python.org/issue19389 opened by Mathieu.Virbel #19390: clinic.py: unhandled syntax error http://bugs.python.org/issue19390 opened by skrah Most recent 15 issues with no replies (15) ========================================== #19390: clinic.py: unhandled syntax error http://bugs.python.org/issue19390 #19389: find_file might return a different directory than the tested f http://bugs.python.org/issue19389 #19387: Explain and test the sre overlap table http://bugs.python.org/issue19387 #19385: dbm.dumb should be consistent when the database is closed http://bugs.python.org/issue19385 #19383: nturl2path test coverage http://bugs.python.org/issue19383 #19378: Rename "line_offset" parameter in dis.get_instructions to "fir http://bugs.python.org/issue19378 #19373: Tkinter apps including IDLE may not display properly on OS X 1 http://bugs.python.org/issue19373 #19372: getaddrinfo() bug http://bugs.python.org/issue19372 #19371: timing test too tight http://bugs.python.org/issue19371 #19363: Python 2.7's future_builtins.map is not compatible with Python http://bugs.python.org/issue19363 #19355: Initial modernization of OpenWatcom support http://bugs.python.org/issue19355 #19354: test_format fails on RHEL-6 http://bugs.python.org/issue19354 #19351: python msi installers - silent mode http://bugs.python.org/issue19351 #19349: Not so correct exception message when running venv http://bugs.python.org/issue19349 #19348: Building _testcapimodule as a builtin results in compile error http://bugs.python.org/issue19348 Most recent 15 issues waiting for review (15) ============================================= #19390: clinic.py: unhandled syntax error http://bugs.python.org/issue19390 #19389: find_file might return a different directory than the tested f http://bugs.python.org/issue19389 #19387: Explain and test the sre overlap table http://bugs.python.org/issue19387 #19385: dbm.dumb should be consistent when the database is closed http://bugs.python.org/issue19385 #19383: nturl2path test coverage http://bugs.python.org/issue19383 #19382: tabnanny unit tests http://bugs.python.org/issue19382 #19380: Optimize parsing of regular expressions http://bugs.python.org/issue19380 #19375: Deprecate site-python in site.py http://bugs.python.org/issue19375 #19362: Documentation for len() fails to mention that it works on sets http://bugs.python.org/issue19362 #19359: reversed() does not work with weakref.proxy of sequences http://bugs.python.org/issue19359 #19355: Initial modernization of OpenWatcom support http://bugs.python.org/issue19355 #19353: on RHEL6 simple build fails with test_linux_constants (test.te http://bugs.python.org/issue19353 #19349: Not so correct exception message when running venv http://bugs.python.org/issue19349 #19342: Improve grp module docstrings http://bugs.python.org/issue19342 #19339: telnetlib: time.monotonic() should be used instead of time.tim http://bugs.python.org/issue19339 Top 10 most discussed issues (10) ================================= #7442: _localemodule.c: str2uni() with different LC_NUMERIC and LC_CT http://bugs.python.org/issue7442 19 msgs #19331: Revise PEP 8 recommendation for class names http://bugs.python.org/issue19331 17 msgs #19292: Make SSLContext.set_default_verify_paths() work on Windows http://bugs.python.org/issue19292 16 msgs #19270: Document that sched.cancel() doesn't distinguish equal events http://bugs.python.org/issue19270 14 msgs #16113: SHA-3 (Keccak) support may need to be removed before 3.4 http://bugs.python.org/issue16113 11 msgs #17400: ipaddress should make it easy to identify rfc6598 addresses http://bugs.python.org/issue17400 11 msgs #19316: devguide: compiler - wording http://bugs.python.org/issue19316 11 msgs #16500: Add an 'atfork' module http://bugs.python.org/issue16500 10 msgs #16595: Add resource.prlimit http://bugs.python.org/issue16595 10 msgs #19302: Add support for AIX pollset efficient event I/O polling http://bugs.python.org/issue19302 10 msgs Issues closed (89) ================== #1540: Refleak tests: test_doctest and test_gc are failing http://bugs.python.org/issue1540 closed by pitrou #4905: Use INVALID_FILE_ATTRIBUTES instead of magic numbers http://bugs.python.org/issue4905 closed by tim.golden #8964: platform._sys_version does not parse correctly IronPython 2.x http://bugs.python.org/issue8964 closed by ezio.melotti #10015: Creating a multiprocess.pool.ThreadPool from a child thread bl http://bugs.python.org/issue10015 closed by terry.reedy #10179: os.stat fails on mapped network drive http://bugs.python.org/issue10179 closed by tim.golden #12866: Add support for 24-bit samples in the audioop module http://bugs.python.org/issue12866 closed by serhiy.storchaka #13515: Consistent documentation practices for security concerns and c http://bugs.python.org/issue13515 closed by ezio.melotti #14027: Add pysetup script for Windows http://bugs.python.org/issue14027 closed by pitrou #15207: mimetypes.read_windows_registry() uses the wrong regkey, creat http://bugs.python.org/issue15207 closed by tim.golden #16175: Add I/O Completion Ports wrapper http://bugs.python.org/issue16175 closed by christian.heimes #16612: Integrate "Argument Clinic" into CPython trunk http://bugs.python.org/issue16612 closed by larry #17087: Improve the repr for regular expression match objects http://bugs.python.org/issue17087 closed by serhiy.storchaka #17759: test_urllibnet.test_bad_address() fails on Ubuntu 13.04 http://bugs.python.org/issue17759 closed by barry #18221: abspath strips trailing spaces on win32 http://bugs.python.org/issue18221 closed by tim.golden #18235: _sysconfigdata.py wrong on AIX installations http://bugs.python.org/issue18235 closed by pitrou #18306: os.stat gives exception for Windows junctions in v3.3 http://bugs.python.org/issue18306 closed by tim.golden #18404: Memory leak in gdbmmodule http://bugs.python.org/issue18404 closed by christian.heimes #18527: Upgrade Modules/zlib to 1.2.8 http://bugs.python.org/issue18527 closed by christian.heimes #18603: PyOS_mystricmp unused and no longer available http://bugs.python.org/issue18603 closed by christian.heimes #18606: Add statistics module to standard library http://bugs.python.org/issue18606 closed by larry #18913: ssl._ssl._test_decode_cert seems to leak memory with certain c http://bugs.python.org/issue18913 closed by pitrou #18958: Exception('No JSON object could be decoded') when parsing a va http://bugs.python.org/issue18958 closed by ezio.melotti #19170: telnetlib: use selectors http://bugs.python.org/issue19170 closed by neologix #19201: Add 'x' mode to lzma.open() http://bugs.python.org/issue19201 closed by nadeem.vawda #19222: Add 'x' mode to gzip.open() http://bugs.python.org/issue19222 closed by nadeem.vawda #19223: Add 'x' mode to bz2.open() http://bugs.python.org/issue19223 closed by nadeem.vawda #19236: Add Tornado HTTP benchmark http://bugs.python.org/issue19236 closed by python-dev #19237: Proposal : LCM function to complement GCD http://bugs.python.org/issue19237 closed by mark.dickinson #19238: Misleading explanation of fill and align in format_spec http://bugs.python.org/issue19238 closed by ezio.melotti #19254: Python implementation of PBKDF2_HMAC http://bugs.python.org/issue19254 closed by christian.heimes #19262: Add asyncio (tulip, PEP 3156) to stdlib http://bugs.python.org/issue19262 closed by gvanrossum #19263: enum.py : Enum.__new__(). __objclass__ http://bugs.python.org/issue19263 closed by python-dev #19271: Update OrderedDict "see also" link http://bugs.python.org/issue19271 closed by rhettinger #19273: Update PCbuild/readme.txt http://bugs.python.org/issue19273 closed by tim.golden #19274: make zipfile.PyZipFile more usable http://bugs.python.org/issue19274 closed by Christian.Tismer #19285: test_asyncio failures on FreeBSD http://bugs.python.org/issue19285 closed by skrah #19287: __contains__() of dbm.ndbm databases fails with str http://bugs.python.org/issue19287 closed by serhiy.storchaka #19288: __contains__() of dbm.gnu databases fails with str http://bugs.python.org/issue19288 closed by serhiy.storchaka #19289: Incorrect documentation for format's fill character. http://bugs.python.org/issue19289 closed by ezio.melotti #19293: test_asyncio hanging for 1 hour (AIX version, hangs in test_su http://bugs.python.org/issue19293 closed by gvanrossum #19294: test_asyncio fails intermittently on OS X 10.4 http://bugs.python.org/issue19294 closed by ned.deily #19295: Make asyncio work without threads http://bugs.python.org/issue19295 closed by gvanrossum #19297: test_asyncio: unclosed sockets http://bugs.python.org/issue19297 closed by python-dev #19299: test_asyncio refleak failures http://bugs.python.org/issue19299 closed by pitrou #19300: Swap keyword arguments in open() to make encoding easier to us http://bugs.python.org/issue19300 closed by ezio.melotti #19301: Globals declared in function scope have wrong __qualname__ http://bugs.python.org/issue19301 closed by python-dev #19303: Typo in devguide - coverage http://bugs.python.org/issue19303 closed by ned.deily #19304: Typo in statistics.rst http://bugs.python.org/issue19304 closed by ned.deily #19305: sporadic test_asyncio failure under FreeBSD http://bugs.python.org/issue19305 closed by pitrou #19306: Warn unsuspecting readers that thread stacks are in reverse or http://bugs.python.org/issue19306 closed by gvanrossum #19307: Improve TypeError message in json.loads() http://bugs.python.org/issue19307 closed by ezio.melotti #19309: asyncio: fix handling of processes in a different process grou http://bugs.python.org/issue19309 closed by neologix #19310: asyncio: fix waitpid() logic http://bugs.python.org/issue19310 closed by gvanrossum #19311: devguide: hg bisect section could be clearer http://bugs.python.org/issue19311 closed by pitrou #19312: Typo in devguide - compiler http://bugs.python.org/issue19312 closed by ezio.melotti #19313: reference leaks http://bugs.python.org/issue19313 closed by python-dev #19314: Typo in devguide - compiler http://bugs.python.org/issue19314 closed by ezio.melotti #19315: devguide: compiler - poor wording http://bugs.python.org/issue19315 closed by ezio.melotti #19318: break more than once http://bugs.python.org/issue19318 closed by pitrou #19319: misleading comment regarding C http://bugs.python.org/issue19319 closed by ezio.melotti #19321: memoryview should be a context manager http://bugs.python.org/issue19321 closed by serhiy.storchaka #19322: Python crashes on re.search in new regex module. http://bugs.python.org/issue19322 closed by ned.deily #19323: typo in statistics documentation http://bugs.python.org/issue19323 closed by python-dev #19324: Expose Linux-specific APIs in resource module http://bugs.python.org/issue19324 closed by christian.heimes #19326: asyncio: child process exit isn't detected if its stdin/stdout http://bugs.python.org/issue19326 closed by neologix #19327: re doesn't work with big charsets http://bugs.python.org/issue19327 closed by serhiy.storchaka #19337: MMAP: Bus error (core dump) under heavy read/write http://bugs.python.org/issue19337 closed by neologix #19340: test_sysconfig.test_srcdir fails when sys.base_prefix="/" http://bugs.python.org/issue19340 closed by pitrou #19341: locale.getdefaultencoding() returns wrong locale and encoding http://bugs.python.org/issue19341 closed by serhiy.storchaka #19344: ReStructuredText error in whatsnew/3.4.rst http://bugs.python.org/issue19344 closed by christian.heimes #19345: Unclear phrasing in whatsnew/3.4.rst http://bugs.python.org/issue19345 closed by python-dev #19350: macurl2path coverage http://bugs.python.org/issue19350 closed by orsenthil #19352: unittest loader barfs on symlinks http://bugs.python.org/issue19352 closed by pitrou #19356: Change argument _self in _io/textio.c http://bugs.python.org/issue19356 closed by pitrou #19357: Module does not get pulled from sys.modules during importlib t http://bugs.python.org/issue19357 closed by eric.snow #19360: test_site fails when installed into $HOME/.local http://bugs.python.org/issue19360 closed by pitrou #19364: Implementing __getattr__ breaks copy and deepcopy http://bugs.python.org/issue19364 closed by eric.snow #19365: Quadratic complexity in the parsing of re replacement string http://bugs.python.org/issue19365 closed by serhiy.storchaka #19366: Segfault in REPL due to escaped tab. http://bugs.python.org/issue19366 closed by ned.deily #19368: Seg Fault after upgrading to OS X Mavericks http://bugs.python.org/issue19368 closed by ned.deily #19369: PyObject_LengthHint is slow http://bugs.python.org/issue19369 closed by serhiy.storchaka #19370: Entering integers in python interpreter causes segfault http://bugs.python.org/issue19370 closed by ned.deily #19374: Segmentation fault: 11 with String.split http://bugs.python.org/issue19374 closed by ned.deily #19377: Backport SVG mime type to Python 2 http://bugs.python.org/issue19377 closed by christian.heimes #19379: Don't import linecache at warnings toplevel http://bugs.python.org/issue19379 closed by python-dev #19381: Seg Fault with complex numbers http://bugs.python.org/issue19381 closed by benjamin.peterson #19384: test_py_compile fails with OSError on FreeBSD for root user http://bugs.python.org/issue19384 closed by christian.heimes #19388: Inflated download counts http://bugs.python.org/issue19388 closed by pitrou #1772673: Replacing char* with const char* http://bugs.python.org/issue1772673 closed by serhiy.storchaka From brett at python.org Fri Oct 25 18:18:10 2013 From: brett at python.org (Brett Cannon) Date: Fri, 25 Oct 2013 12:18:10 -0400 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Fri, Oct 25, 2013 at 11:44 AM, Eric Snow wrote: > On Fri, Oct 25, 2013 at 9:40 AM, Brett Cannon wrote: > > After reading Eric's doc patch, I realized there is one change I want to > > make to the current semantics and that's not to backfill __package__ when > > set to None. Since import is now going to take over the job of setting > > __package__ (along with other attributes), this seems like a slight > waste of > > effort. It also kills (or at least complicates) having a loader which > does > > lazy loading since reading the attribute to see if it is None would > trigger > > the load before leaving the import code, thus killing any postponed > loading. > > Fine with me. I may have time today to make the outstanding updates to > the PEP. > I should also mention this applies to __loader__ as well. Basically everything from http://hg.python.org/cpython/file/eb1edc9e3722/Lib/importlib/_bootstrap.py#l1575on down. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Fri Oct 25 18:24:00 2013 From: pje at telecommunity.com (PJ Eby) Date: Fri, 25 Oct 2013 12:24:00 -0400 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: I've not really had time to review this PEP yet, but from skimming discussion to date, the only thing I'm still worried about is whether this will break lazy import schemes that use a module subclass that hooks __getattribute__ and calls reload() in order to perform what's actually an *initial* load. IOW, does anything in this proposal rely on a module object having *any* attributes besides __name__ set at reload() time? That is, is there an assumption that a module being reloaded has 1. Been loaded, and 2. Is being reloaded via the same location, __loader__, etc. as before? At least through all 2.x, reload() just uses module.__name__ to restart the module find-and-load process, and does not assume that __loader__ is valid in advance. (Also, if this has changed in recent Python versions independent of this PEP, it's a backwards-compatibility break that should be documented somewhere.) On Thu, Oct 24, 2013 at 2:05 AM, Eric Snow wrote: > I've had some offline discussion with Brett and Nick about PEP 451 > which has led to some meaningful clarifications in the PEP. In the > interest of pulling further discussions back onto this > (archived/public) list, here's an update of what we'd discussed and > where things are at. :) > > * path entry finders indicate that they found part of a possible > namespace package by returning a spec with no loader set (but with > submodule_search_locations set). Brett wanted some clarification on > this. > * The name/path signature and attributes of file-based finders in > importlib will no longer be changing. Brett had some suggestions on > the proposed change and it became clear that the the change was > actually pointless. > * I've asserted that there shouldn't be much difficulty in adjusting > pkgutil and other modules to work with ModuleSpec. > * Brett asked for clarification on whether the "load()" example from > the PEP would be realized implicitly by the import machinery or > explicitly as a method on ModuleSpec. This has bearing on the ability > of finders to return instances of ModuleSpec subclasses or even > ModuleSpec-like objects (a la duck typing). The answer is the it will > not be a method on ModuleSpec, so it is effectively just part of the > general import system implementation. Finders may return any object > that provides the attributes of ModuleSpec. I will be updating the > PEP to make these points clear. > > * Nick suggested writing a draft patch for the language reference > changes (the import page). Such a patch will be a pretty good > indicator of the impact of PEP 451 on the import system and should > highlight any design flaws in the API. This is on my to-do list > (hopefully by tomorrow). > * Nick also suggested moving all ModuleSpec methods to a separate > class that will simply make use of a separate, existing ModuleSpec > instance. This will help address several issues, particularly by > relaxing the constraints on what finders can return, but also by > avoiding the unnecessary exposure of the methods via every > module.__spec__. I plan on going with this, but currently am trying > out the change to see if there are any problems I've missed. Once I > feel good about it I'll update the PEP. > > That about sums up our discussions. I have a couple of outstanding > updates to the PEP to make when I get a chance, as well as putting up > a language reference patch for review. > > -eric > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/pje%40telecommunity.com From brett at python.org Fri Oct 25 19:15:03 2013 From: brett at python.org (Brett Cannon) Date: Fri, 25 Oct 2013 13:15:03 -0400 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Fri, Oct 25, 2013 at 12:24 PM, PJ Eby wrote: > I've not really had time to review this PEP yet, but from skimming > discussion to date, the only thing I'm still worried about is whether > this will break lazy import schemes that use a module subclass that > hooks __getattribute__ and calls reload() in order to perform what's > actually an *initial* load. > Depends on how you implement it probably. I have a scheme in my head using __getattribute__ and loaders which will work with this PEP if __package__ and __loader__ are not explicitly checked after execute_module() is called, so lazy imports are not going to be fundamentally broken. Since Python 3.3 lazy loaders relying on __getattribute__ have been broken due to those __package__/__loader__ checks so this is actually going to be an improvement. > > IOW, does anything in this proposal rely on a module object having > *any* attributes besides __name__ set at reload() time? As it stands in Python 3.3 it requires __name__ and __loader__ be defined: http://hg.python.org/cpython/file/e2c3f638c3d0/Lib/importlib/__init__.py#l101 > That is, is > there an assumption that a module being reloaded has > > 1. Been loaded, and > 2. Is being reloaded via the same location, __loader__, etc. as before? > Just 2 in terms of __loader__ since loaders have to work with or without the module already existing. > > At least through all 2.x, reload() just uses module.__name__ to > restart the module find-and-load process, and does not assume that > __loader__ is valid in advance. > That doesn't make much sense in a post-importlib world where import makes sure that __loader__ is set (which it has since Python 3.3). Otherwise you are asking for not just a reload but a re-find as well. As for the PEP, it will probably keep the name/loader requirement but shift it to having to be set on the spec object at __spec__. I guess you could make the argument a reload should include re-running the finder step as well, but I don't think it's worth the code tweaking to make it happen when the finder/loader steps are divided as they are. > > (Also, if this has changed in recent Python versions independent of > this PEP, it's a backwards-compatibility break that should be > documented somewhere.) > http://bugs.python.org/issue19392 -Brett > > > On Thu, Oct 24, 2013 at 2:05 AM, Eric Snow > wrote: > > I've had some offline discussion with Brett and Nick about PEP 451 > > which has led to some meaningful clarifications in the PEP. In the > > interest of pulling further discussions back onto this > > (archived/public) list, here's an update of what we'd discussed and > > where things are at. :) > > > > * path entry finders indicate that they found part of a possible > > namespace package by returning a spec with no loader set (but with > > submodule_search_locations set). Brett wanted some clarification on > > this. > > * The name/path signature and attributes of file-based finders in > > importlib will no longer be changing. Brett had some suggestions on > > the proposed change and it became clear that the the change was > > actually pointless. > > * I've asserted that there shouldn't be much difficulty in adjusting > > pkgutil and other modules to work with ModuleSpec. > > * Brett asked for clarification on whether the "load()" example from > > the PEP would be realized implicitly by the import machinery or > > explicitly as a method on ModuleSpec. This has bearing on the ability > > of finders to return instances of ModuleSpec subclasses or even > > ModuleSpec-like objects (a la duck typing). The answer is the it will > > not be a method on ModuleSpec, so it is effectively just part of the > > general import system implementation. Finders may return any object > > that provides the attributes of ModuleSpec. I will be updating the > > PEP to make these points clear. > > > > * Nick suggested writing a draft patch for the language reference > > changes (the import page). Such a patch will be a pretty good > > indicator of the impact of PEP 451 on the import system and should > > highlight any design flaws in the API. This is on my to-do list > > (hopefully by tomorrow). > > * Nick also suggested moving all ModuleSpec methods to a separate > > class that will simply make use of a separate, existing ModuleSpec > > instance. This will help address several issues, particularly by > > relaxing the constraints on what finders can return, but also by > > avoiding the unnecessary exposure of the methods via every > > module.__spec__. I plan on going with this, but currently am trying > > out the change to see if there are any problems I've missed. Once I > > feel good about it I'll update the PEP. > > > > That about sums up our discussions. I have a couple of outstanding > > updates to the PEP to make when I get a chance, as well as putting up > > a language reference patch for review. > > > > -eric > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/pje%40telecommunity.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Fri Oct 25 19:22:10 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 25 Oct 2013 11:22:10 -0600 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Fri, Oct 25, 2013 at 10:24 AM, PJ Eby wrote: > I've not really had time to review this PEP yet, but from skimming > discussion to date, the only thing I'm still worried about is whether > this will break lazy import schemes that use a module subclass that > hooks __getattribute__ and calls reload() in order to perform what's > actually an *initial* load. > > IOW, does anything in this proposal rely on a module object having > *any* attributes besides __name__ set at reload() time? That is, is > there an assumption that a module being reloaded has > > 1. Been loaded, and > 2. Is being reloaded via the same location, __loader__, etc. as before? > > At least through all 2.x, reload() just uses module.__name__ to > restart the module find-and-load process, and does not assume that > __loader__ is valid in advance. > > (Also, if this has changed in recent Python versions independent of > this PEP, it's a backwards-compatibility break that should be > documented somewhere.) Yeah, this actually changed in 3.3 IIRC, where reload now calls module.__loader__.load_module(module.__name__). I'll double-check. PEP 451 simply changes that to be (basically) module.__loader__.exec_module(module). -eric From pje at telecommunity.com Fri Oct 25 20:10:35 2013 From: pje at telecommunity.com (PJ Eby) Date: Fri, 25 Oct 2013 14:10:35 -0400 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Fri, Oct 25, 2013 at 1:15 PM, Brett Cannon wrote: > > On Fri, Oct 25, 2013 at 12:24 PM, PJ Eby wrote: >> At least through all 2.x, reload() just uses module.__name__ to >> restart the module find-and-load process, and does not assume that >> __loader__ is valid in advance. > > > That doesn't make much sense in a post-importlib world where import makes > sure that __loader__ is set (which it has since Python 3.3). Otherwise you > are asking for not just a reload but a re-find as well. That's a feature, not a bug. A reload() after changing sys.path *should* take into account the change, not to mention any changes to meta_path, path hooks, etc. (And it's how reload() worked before importlib.) I suppose it's not really documented all that well, but way way back in the 2.3 timeframe I asked for a tweak to PEP 302 to make sure that reload() (in the re-find sense) would work properly with PEP 302 loaders like zipimport -- some of the language still in the PEP is there specifically to support this use case. (Specifically, the bit that says loaders *must* use the existing module object in sys.modules if there's one there already, so that reload() will work. It was actually in part to ensure that reload() would work in the case of a re-find.) It appears that since then, the PEP has been changed in a way that invalidates part of the purpose of the prior change; I guess I missed the discussion of that change last year. :-( ISTM there should've been such a discussion, since IIRC importlib wasn't supposed to change any Python semantics, and this is a non-trivial change to the semantics of reload() even in cases that aren't doing lazy imports or other such munging. reload() used to take sys.* and __path__ changes into account, and IMO should continue to do so. If this is an intentional change in reload() semantics, other Python implementations need to know about this too! (That being said, I'm not saying I shouldn't or couldn't have tested this in 3.3 and found out about it that way. And the existence of issue18698 suggests that nobody's relying yet on even the *fundamental* semantics of PEP 302 reload() working properly in 3.3, since that was an even bigger change that nobody spotted till a couple of months ago.) From brett at python.org Fri Oct 25 21:15:55 2013 From: brett at python.org (Brett Cannon) Date: Fri, 25 Oct 2013 15:15:55 -0400 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Fri, Oct 25, 2013 at 2:10 PM, PJ Eby wrote: > On Fri, Oct 25, 2013 at 1:15 PM, Brett Cannon wrote: > > > > On Fri, Oct 25, 2013 at 12:24 PM, PJ Eby wrote: > >> At least through all 2.x, reload() just uses module.__name__ to > >> restart the module find-and-load process, and does not assume that > >> __loader__ is valid in advance. > > > > > > That doesn't make much sense in a post-importlib world where import makes > > sure that __loader__ is set (which it has since Python 3.3). Otherwise > you > > are asking for not just a reload but a re-find as well. > > That's a feature, not a bug. A reload() after changing sys.path > *should* take into account the change, not to mention any changes to > meta_path, path hooks, etc. (And it's how reload() worked before > importlib.) > Fair enough, but in my mind that obviously doesn't really click for what I view as a reload in an importlib world where secret import code no longer exists. When I think re-load I think "load again", not "find the module again and execute a load with a possibly new loader". It's not worth changing in Python 3.3, but if you want to propose to redefine importlib.reload() to use importlib.find_spec() to reset the __spec__ on a module then I'm not going to argue against it (but I'm not going to fight for it either). And in a PEP 451 world it should be dead-simple to make this work the way you want in your own code even if this doesn't go the way you want:: spec = importlib.find_spec(name) module.__spec__ = spec importlib.reload(module) # Which in itself is essentially init_module_attrs(spec, module); spec.loader.exec_module(module) Heck, you can do this in Python 3.3 right now:: loader = importlib.find_loader(name) module = sys.modules[name] module.__loader__ = loader importlib.reload(module) > I suppose it's not really documented all that well, Nope =) > but way way back > in the 2.3 timeframe I asked for a tweak to PEP 302 to make sure that > reload() (in the re-find sense) would work properly with PEP 302 > loaders like zipimport -- some of the language still in the PEP is > there specifically to support this use case. (Specifically, the bit > that says loaders *must* use the existing module object in sys.modules > if there's one there already, so that reload() will work. It was > actually in part to ensure that reload() would work in the case of a > re-find.) > Ah, okay. That is not explicit in the PEP beyond coming off a total nuisance in order to support reloading by the loader, not an explicit finder + loader use-case. > > It appears that since then, the PEP has been changed in a way that > invalidates part of the purpose of the prior change; I guess I missed > the discussion of that change last year. :-( > > ISTM there should've been such a discussion, since IIRC importlib > wasn't supposed to change any Python semantics, and this is a > non-trivial change to the semantics of reload() even in cases that > aren't doing lazy imports or other such munging. Well, not change them to the best of its abilities. I'm sure there are edge cases that I couldn't match properly since I could only go by the test suite and bug reports filed since Python 3.1. > reload() used to > take sys.* and __path__ changes into account, and IMO should continue > to do so. Unfortunately this was never explicitly documented (at least that I noticed) nor was there a test in the stdlib to enforce compliance with it, else this never would have happened. > If this is an intentional change in reload() semantics, > other Python implementations need to know about this too! > Not really. As of Python 3.3 the semantic shift included re-implementing the function in pure Python so they pick up the change for free; remember the function is not a builtin anymore to somewhat de-emphasize its use since it's kind of a hack and really only used typically at the interpreter prompt as an easy part of a load-edit-reload dev cycle, not some fundamental operation. > > (That being said, I'm not saying I shouldn't or couldn't have tested > this in 3.3 and found out about it that way. And the existence of > issue18698 suggests that nobody's relying yet on even the > *fundamental* semantics of PEP 302 reload() working properly in 3.3, > since that was an even bigger change that nobody spotted till a couple > of months ago.) > Yep, never got a bug report on this (although there have been reports of typos in the docs as well as not returning what's in sys.modules so it is being used by people). -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Sat Oct 26 00:51:38 2013 From: pje at telecommunity.com (PJ Eby) Date: Fri, 25 Oct 2013 18:51:38 -0400 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Fri, Oct 25, 2013 at 3:15 PM, Brett Cannon wrote: > On Fri, Oct 25, 2013 at 2:10 PM, PJ Eby wrote: >> On Fri, Oct 25, 2013 at 1:15 PM, Brett Cannon wrote: >> > On Fri, Oct 25, 2013 at 12:24 PM, PJ Eby wrote: >> >> At least through all 2.x, reload() just uses module.__name__ to >> >> restart the module find-and-load process, and does not assume that >> >> __loader__ is valid in advance. >> > >> > >> > That doesn't make much sense in a post-importlib world where import >> > makes >> > sure that __loader__ is set (which it has since Python 3.3). Otherwise >> > you >> > are asking for not just a reload but a re-find as well. >> >> That's a feature, not a bug. A reload() after changing sys.path >> *should* take into account the change, not to mention any changes to >> meta_path, path hooks, etc. (And it's how reload() worked before >> importlib.) > > > Fair enough, but in my mind that obviously doesn't really click for what I > view as a reload in an importlib world where secret import code no longer > exists. When I think re-load I think "load again", not "find the module > again and execute a load with a possibly new loader". Sure, and the reference manual is rather vague on this point. However, I would guess that at least some web frameworks with automatic reload support are going to barf on this change in at least some edge cases. (OTOH, it's unlikely the bugs will ever be reported, because the problem will mysteriously go away once the process is restarted, probably never to occur again.) Mostly, this just seems like an ugly wart -- Python should be dynamic by default, and that includes reloading. While the import machinery has lots of ugly caching under the hood, a user-level function like reload() should not require you to do the equivalent of saying, "no, really... I want you to *really* reload, not just pull in whatever exists where you found it last time, while ignoring whether I switched from module to package or vice versa, or just fixed my sys.path so I can load the right version of the module." It is a really tiny thing in the overall scheme of things, because reload() is not used all that often, but it's still a thing. If this isn't treated as a bug, then the docs for reload() at least need to include a forward-supported workaround so you can say "no, really... *really* reload" in an approved fashion. (ISTM that any production code out there that currently uses reload() would want to perform the "really reload" incantation in order to avoid the edge cases, even if they haven't actually run into any of them yet.) > And in a PEP 451 world it should be dead-simple to make this work the way > you want in your own code even if this doesn't go the way you want:: > > spec = importlib.find_spec(name) > module.__spec__ = spec > importlib.reload(module) # Which in itself is essentially > init_module_attrs(spec, module); spec.loader.exec_module(module) > > Heck, you can do this in Python 3.3 right now:: > > loader = importlib.find_loader(name) > module = sys.modules[name] > module.__loader__ = loader > importlib.reload(module) And will that later version still work correctly in a PEP 451 world, or will you have to detect which world you live in before waving this particular dead chicken? ;-) > Ah, okay. That is not explicit in the PEP beyond coming off a total nuisance > in order to support reloading by the loader, not an explicit finder + loader > use-case. Yeah, it actually was to ensure that you could reload a module using a different loader than the one that originally loaded it, e.g. due to a change in path hooks, etc. From seotaewong40 at gmail.com Sat Oct 26 00:55:31 2013 From: seotaewong40 at gmail.com (Tae Wong) Date: Sat, 26 Oct 2013 07:55:31 +0900 Subject: [Python-Dev] Merge codeaccess.txt from wesnoth-contribcommunity project in Google Code Hosting Message-ID: Here's the codeaccess.txt file. https://wesnoth-contribcommunity.googlecode.com/svn/codeaccess.txt All committer names have (No): Python; check the sixth line. Merging the names of committers in the sixth line to the Misc/ACKS file might be good. From tjreedy at udel.edu Sat Oct 26 02:45:39 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 25 Oct 2013 20:45:39 -0400 Subject: [Python-Dev] Merge codeaccess.txt from wesnoth-contribcommunity project in Google Code Hosting In-Reply-To: References: Message-ID: On 10/25/2013 6:55 PM, Tae Wong wrote: > Here's the codeaccess.txt file. > https://wesnoth-contribcommunity.googlecode.com/svn/codeaccess.txt This seems to be an old copy of Misc/ACKS except that non-ascii chars are not displayed correctly as a result of some encoding snafu. > All committer names have (No): Python; check the sixth line. > > Merging the names of committers in the sixth line to the Misc/ACKS > file might be good. Tae Wong's followup on python-list is even less comprehensible. I think we can ignore the suggestion. -- Terry Jan Reedy From nad at acm.org Sat Oct 26 02:52:08 2013 From: nad at acm.org (Ned Deily) Date: Fri, 25 Oct 2013 17:52:08 -0700 Subject: [Python-Dev] Merge codeaccess.txt from wesnoth-contribcommunity project in Google Code Hosting References: Message-ID: In article , Terry Reedy wrote: > On 10/25/2013 6:55 PM, Tae Wong wrote: > > Here's the codeaccess.txt file. > > https://wesnoth-contribcommunity.googlecode.com/svn/codeaccess.txt > Tae Wong's followup on python-list is even less comprehensible. > I think we can ignore the suggestion. He has a long reputation for spamming mailing lists. Please ignore him. -- Ned Deily, nad at acm.org From seotaewong40 at gmail.com Sat Oct 26 03:15:59 2013 From: seotaewong40 at gmail.com (Tae Wong) Date: Sat, 26 Oct 2013 10:15:59 +0900 Subject: [Python-Dev] Merge codeaccess.txt from wesnoth-contribcommunity project in Google Code Hosting In-Reply-To: References: Message-ID: You wrote: > https://wesnoth-contribcommunity.googlecode.com/svn/codeaccess.txt You can view this file as all non-ASCII characters are rendered correctly if the file is encoded in UTF-8. And you are not subscribed to this mailing list! -- Tae Wong From guido at python.org Sat Oct 26 03:28:10 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 25 Oct 2013 18:28:10 -0700 Subject: [Python-Dev] Merge codeaccess.txt from wesnoth-contribcommunity project in Google Code Hosting In-Reply-To: References: Message-ID: Somebody please block him. --Guido van Rossum (sent from Android phone) On Oct 25, 2013 5:53 PM, "Ned Deily" wrote: > In article , Terry Reedy > wrote: > > On 10/25/2013 6:55 PM, Tae Wong wrote: > > > Here's the codeaccess.txt file. > > > https://wesnoth-contribcommunity.googlecode.com/svn/codeaccess.txt > > Tae Wong's followup on python-list is even less comprehensible. > > I think we can ignore the suggestion. > > He has a long reputation for spamming mailing lists. Please ignore him. > > -- > Ned Deily, > nad at acm.org > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From seotaewong40 at gmail.com Sat Oct 26 04:26:41 2013 From: seotaewong40 at gmail.com (Tae Wong) Date: Sat, 26 Oct 2013 11:26:41 +0900 Subject: [Python-Dev] Merge codeaccess.txt from wesnoth-contribcommunity project in Google Code Hosting In-Reply-To: References: Message-ID: The names that are included in the https://wesnoth-contribcommunity.googlecode.com/svn/codeaccess.txt file but were missing on the Misc/ACKS file are: Jeff Allen Jim Baker John Benediktsson Shashank Bharadwaj Martin Blais Carl Friedrich Bolz George Boutsioukis Josiah Carlson Ron DuPlain Phillip Eby Maciej Fijalkowski Erik Forsberg Alex Gaynor Alex Gronholm Jeff Hardy Doug Hellmann Daniel Holth Oti Humbel Josh Juneau Alan Kennedy Anselm Kruis Patrick Maupin Wayne Meissner Chris Monson Dirkjan Ochtman Travis Oliphant Runar Petursson Rodrigo Bernardo Pimentel Allison Randal Mateusz Rukowicz Robert Schuppenies Stefan Seefeld Jeff Senn Ask Solem Donald Stufft Talin Michael Twomey Jeroen Ruigrok van der Werven Jeffrey Yasskin Some names in Misc/ACKS are wrongly encoded in ASCII. Michal Bozon as Michal Bo?o? Richard Jones as Richard W.M. Jones R. M. Oudkerk as Richard M. Oudkerk Giampaolo Rodola as Giampaolo Rodol? Victor Terr?n as V?ctor Terr?n Artur Zaprzala as Artur Zaprza?a -- Tae Wong From Nikolaus at rath.org Sat Oct 26 05:22:53 2013 From: Nikolaus at rath.org (Nikolaus Rath) Date: Fri, 25 Oct 2013 20:22:53 -0700 Subject: [Python-Dev] OrderedDict.values() behavior for modified instance Message-ID: <87iowklnqq.fsf@vostro.rath.org> Hello, The documentation says the following about modifying a dict while iterating through its view: | Iterating views while adding or deleting entries in the dictionary may | raise a RuntimeError or fail to iterate over all entries. (http://docs.python.org/3/library/stdtypes.html#dict-views) The OrderedDict documentation doesn't have anything to say on the subject. In practice, its implementation actually mostly correponds to naive expectations: >>> from collections import OrderedDict >>> d = OrderedDict() >>> for i in range(5): ... d[i] = i ... >>> i = iter(d.values()) >>> next(i) 0 >>> del d[0] >>> next(i) 1 >>> del d[2] >>> next(i) 3 >>> d.move_to_end(1) >>> next(i) 4 >>> next(i) 1 >>> etc. However, there is one case that results in a rather confusing error: >>> a = OrderedDict() >>> a[0] = 0 >>> a[1] = 1 >>> a[2] = 2 >>> i = iter(a.values()) >>> next(i) 0 >>> del a[0] >>> del a[1] >>> next(i) Traceback (most recent call last): File "", line 1, in File "/usr/lib/python3.3/collections/abc.py", line 495, in __iter__ yield self._mapping[key] KeyError: 1 What happens here is that a[0] is returned from the linked list, but it still contains links to a[1]. When a[1] is deleted, the links of its predecessor and successor are updated, but a[0] still points to a[1]. The OrderedList then hands a non-existing key to the values() implementation in collections.abc. The result is not only mightily confusing, but it is also not covered by the documentation (KeyError isn't a RuntimeError). I would very much like to see this fixed, but I'm not sure if there's a good way to do that. I see the following options: (a) When deleting an element from an OrderedList, update the pointers in the corresponding linked list element to the end of the list. Then removing the currently "active" element during the iteration would automatically end the iteration. For that, we'd just have to add two lines to __delitem__: def __delitem__(self, key, dict_delitem=dict.__delitem__): dict_delitem(self, key) link = self.__map.pop(key) link_prev = link.prev link_next = link.next link_prev.next = link_next link_next.prev = link_prev link.prev = root # new link.next = root # new The new behavior is explicitly covered by the documentation (changing the dict may result in incomplete iteration), but it's a change to what happened before. (b) When iterating through an OrderedDict, check that the next element is actually still in the hash, i.e. change def __iter__(self): root = self.__root curr = root.next while curr is not root: yield curr.key curr = curr.next to def __iter__(self): root = self.__root curr = root.next while curr is not root and curr.key in self: yield curr.key curr = curr.next that would terminate the iteration only in the special case, but incur an extra dict lookup at every iteration. Alternatively, one could try very hard to not stop the iteration: while curr is not root: yield curr.key while curr is not root: curr = curr.next if curr.key in self: break (c) Catch the KeyError in values(), and re-raise the proper exception in class ValuesView: def __iter__(self): for key in self._mapping: try: yield self._mapping[key] except KeyError: raise RuntimeError("underlying Mapping instance modified during iteration") I consider this a bit ugly, because it may mask other problems in a custom Mapping implementation (that may raise a KeyError because of a bug in the Mapping implementation itself). (d) Extend the OrderedDict documentation to explicity document this case. This has the drawback that it would probably be nicer to just have the behavior be consistent and correspond to intuitive expectations. Would any of those fixes be acceptable? Or is there an even better option? Best, Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ?Time flies like an arrow, fruit flies like a Banana.? From kristjan at ccpgames.com Sat Oct 26 12:57:00 2013 From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=) Date: Sat, 26 Oct 2013 10:57:00 +0000 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: <52693D8E.4050306@mrabarnett.plus.com> Message-ID: In that case, how about adding a client/server feature? If you standardize the format, a minimal tracing client could write a log, or send it to a socket, in a way that can be turned into a snapshot by a corresponsing utility reading from a file or listenting to a socket. Just a though. Could be a future addition? From: Python-Dev [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Victor Stinner Sent: 24. okt?ber 2013 16:45 To: python-dev Subject: Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! > When I was looking for memory leaks in the regex module I simply wrote all of the allocations, reallocations and deallocations to a log file and then parsed it afterwards using a Python script. Simple, but effective. He he, it's funny because you described exactly my first implementation of tracemalloc! I wrote output using fprintf() into a text file. The parser was written in Python but too slow to be used it in practice. When running on the slow set top box, it took minutes (5 maybe 10) to analyze the file. Transfering the file to a PC took also minutes: the file was very large (maybe 1 GB, I don't remember) and SimpleHTTPServer too slow for the transfer. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From amk at amk.ca Sat Oct 26 13:29:06 2013 From: amk at amk.ca (A.M. Kuchling) Date: Sat, 26 Oct 2013 07:29:06 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Fiddled Thread.join() to be a little simpler. Kinda ; -) In-Reply-To: <3d678Y6NRJzSs7@mail.python.org> References: <3d678Y6NRJzSs7@mail.python.org> Message-ID: <20131026112906.GA23635@datlandrewk.home> On Sat, Oct 26, 2013 at 05:34:05AM +0200, tim.peters wrote: > Fiddled Thread.join() to be a little simpler. Kinda ;-) > - # else it's a negative timeout - precise behavior isn't documented > - # then, but historically .join() returned in this case > + else: > + # the behavior of a negative timeout isn't documented, but > + # historically .join() has acted as if timeout=0 then I think the final 'then' on the new comment is an editing error. Unless perhaps you meant to write a further sentence of comment? --amk From stefan-usenet at bytereef.org Sat Oct 26 14:50:01 2013 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Sat, 26 Oct 2013 14:50:01 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Issue #19330: Handle the no-docstrings case in tests In-Reply-To: <3d6Lrm0pQcz7LlK@mail.python.org> References: <3d6Lrm0pQcz7LlK@mail.python.org> Message-ID: <20131026125001.GA1257@sleipnir.bytereef.org> nick.coghlan wrote: > http://hg.python.org/cpython/rev/a9bbc2d0c1dc > -HAVE_DOCSTRINGS = (check_impl_detail(cpython=False) or > - sys.platform == 'win32' or > - sysconfig.get_config_var('WITH_DOC_STRINGS')) > +# Rather than trying to enumerate all the cases where docstrings may be > +# disabled, we just check for that directly > + > +def _check_docstrings(): > + """Just used to check if docstrings are enabled""" > + > +HAVE_DOCSTRINGS = (_check_docstrings.__doc__ is not None) > > requires_docstrings = unittest.skipUnless(HAVE_DOCSTRINGS, I think that does not detect --without-doc-strings (i.e. the C docstrings are empty). Stefan Krah From donald at stufft.io Sat Oct 26 15:28:59 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 26 Oct 2013 09:28:59 -0400 Subject: [Python-Dev] pip SSL In-Reply-To: References: <52616A30.6000803@gmail.com> <5261719C.7000301@gmail.com> <52617D46.1000001@gmail.com> <5261A26C.7000506@gmail.com> <5261B94F.5030006@gmail.com> <52627E9E.3040003@python.org> <52629AC2.6080507@python.org> <5262AAD0.9050900@python.org> <739A4AB2-5345-4F87-BF59-6F242883C026@stufft.io> Message-ID: <5E4C3D92-F883-4F8E-9D24-EAD53405691E@stufft.io> This pull request should solve this https://github.com/pypa/pip/pull/1256 On Oct 20, 2013, at 12:32 AM, Nick Coghlan wrote: > On 20 October 2013 05:46, Ian Cordasco wrote: >> Also the three of us maintaining requests and the author of urllib3 >> are all very conscious that the packaged pem file is outdated. We have >> an open issue about how to rebuild it accurately while taking into >> consideration (and not including) the ones that have been revoked. Any >> suggestions you have can be sent to me off list or reported on the >> issue tracker. > > The requests issue Ian is referring to: > https://github.com/kennethreitz/requests/issues/1659 > > The next version of PEP 453 will include getting this resolved as part > of the integration timeline: > > ======================== > * by December 29th (1 week prior to the scheduled date of 3.4.0 beta 2) > > ``requests`` certificate management issue resolved > ``ensurepip`` updated to the final release of pip 1.5, or a subsequent > maintenance release (including a suitably updated vendored copy of > ``requests``) > ======================== > > And also mentions it under the "security considerations" section for > the bootstrapping mechanism: > > ======================== > Only users that choose to use ``pip`` to communicate with PyPI will > need to pay attention to the additional security considerations that come > with doing so. > > However, the core CPython team will also assist with reviewing and > resolving the `certificate update management issue > `__ currently > affecting the ``requests`` project (and hence ``pip``). > ======================== > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From storchaka at gmail.com Sat Oct 26 17:10:33 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 26 Oct 2013 18:10:33 +0300 Subject: [Python-Dev] [Python-checkins] cpython: Issue #19330: Handle the no-docstrings case in tests In-Reply-To: <20131026125001.GA1257@sleipnir.bytereef.org> References: <3d6Lrm0pQcz7LlK@mail.python.org> <20131026125001.GA1257@sleipnir.bytereef.org> Message-ID: 26.10.13 15:50, Stefan Krah ???????(??): > nick.coghlan wrote: >> http://hg.python.org/cpython/rev/a9bbc2d0c1dc >> -HAVE_DOCSTRINGS = (check_impl_detail(cpython=False) or >> - sys.platform == 'win32' or >> - sysconfig.get_config_var('WITH_DOC_STRINGS')) >> +# Rather than trying to enumerate all the cases where docstrings may be >> +# disabled, we just check for that directly >> + >> +def _check_docstrings(): >> + """Just used to check if docstrings are enabled""" >> + >> +HAVE_DOCSTRINGS = (_check_docstrings.__doc__ is not None) >> >> requires_docstrings = unittest.skipUnless(HAVE_DOCSTRINGS, > > I think that does not detect --without-doc-strings (i.e. the C docstrings are > empty). Indeed. HAVE_DOCSTRINGS was introduced to skip tests for the C docstrings. Python docstrings tests are skipped if sys.flags.optimize >= 2. From tim.peters at gmail.com Sat Oct 26 17:42:27 2013 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 26 Oct 2013 10:42:27 -0500 Subject: [Python-Dev] [Python-checkins] cpython: Fiddled Thread.join() to be a little simpler. Kinda ; -) In-Reply-To: <20131026112906.GA23635@datlandrewk.home> References: <3d678Y6NRJzSs7@mail.python.org> <20131026112906.GA23635@datlandrewk.home> Message-ID: On Sat, Oct 26, 2013 at 6:29 AM, A.M. Kuchling wrote: > On Sat, Oct 26, 2013 at 05:34:05AM +0200, tim.peters wrote: >> Fiddled Thread.join() to be a little simpler. Kinda ;-) >> - # else it's a negative timeout - precise behavior isn't documented >> - # then, but historically .join() returned in this case >> + else: >> + # the behavior of a negative timeout isn't documented, but >> + # historically .join() has acted as if timeout=0 then > > I think the final 'then' on the new comment is an editing error. > Unless perhaps you meant to write a further sentence of comment? Reads OK to me. Read "then" as an abbreviation of "in that case"? From v+python at g.nevcal.com Sat Oct 26 18:32:42 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 26 Oct 2013 09:32:42 -0700 Subject: [Python-Dev] [Python-checkins] cpython: Fiddled Thread.join() to be a little simpler. Kinda ; -) In-Reply-To: References: <3d678Y6NRJzSs7@mail.python.org> <20131026112906.GA23635@datlandrewk.home> Message-ID: <526BEEAA.8040509@g.nevcal.com> On 10/26/2013 8:42 AM, Tim Peters wrote: > Read "then" as an abbreviation of "in that case"? In that case, then, "then" is a very poor abbreviation of "in that case", and is unlikely to be interpreted with that expansion by many readers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Oct 26 19:27:57 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 27 Oct 2013 03:27:57 +1000 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On 26 October 2013 08:51, PJ Eby wrote: > Mostly, this just seems like an ugly wart -- Python should be dynamic > by default, and that includes reloading. While the import machinery > has lots of ugly caching under the hood, a user-level function like > reload() should not require you to do the equivalent of saying, "no, > really... I want you to *really* reload, not just pull in whatever > exists where you found it last time, while ignoring whether I switched > from module to package or vice versa, or just fixed my sys.path so I > can load the right version of the module." > > It is a really tiny thing in the overall scheme of things, because > reload() is not used all that often, but it's still a thing. If this > isn't treated as a bug, then the docs for reload() at least need to > include a forward-supported workaround so you can say "no, really... > *really* reload" in an approved fashion. > > (ISTM that any production code out there that currently uses reload() > would want to perform the "really reload" incantation in order to > avoid the edge cases, even if they haven't actually run into any of > them yet.) Having imp.reload() kick off the full reload cycle makes sense to me. The only reason it doesn't currently do that in importlib is because there was no test for it in the regression test suite, and PEP 302 and the library reference are entirely vague on how module reloading works (aside from the requirement that it reuse the same module namespace). >> And in a PEP 451 world it should be dead-simple to make this work the way >> you want in your own code even if this doesn't go the way you want:: >> >> spec = importlib.find_spec(name) >> module.__spec__ = spec >> importlib.reload(module) # Which in itself is essentially >> init_module_attrs(spec, module); spec.loader.exec_module(module) >> >> Heck, you can do this in Python 3.3 right now:: >> >> loader = importlib.find_loader(name) >> module = sys.modules[name] >> module.__loader__ = loader >> importlib.reload(module) > > And will that later version still work correctly in a PEP 451 world, > or will you have to detect which world you live in before waving this > particular dead chicken? ;-) Yep, we already tweaked PEP 451 to say that it has to respect post-import modifications made to the module attributes. >> Ah, okay. That is not explicit in the PEP beyond coming off a total nuisance >> in order to support reloading by the loader, not an explicit finder + loader >> use-case. > > Yeah, it actually was to ensure that you could reload a module using a > different loader than the one that originally loaded it, e.g. due to a > change in path hooks, etc. Yeah, the rationale makes sense, we only missed it due to the lack of a regression test for the behaviour. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Oct 26 19:32:14 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 27 Oct 2013 03:32:14 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Issue #19330: Handle the no-docstrings case in tests In-Reply-To: References: <3d6Lrm0pQcz7LlK@mail.python.org> <20131026125001.GA1257@sleipnir.bytereef.org> Message-ID: On 27 October 2013 01:10, Serhiy Storchaka wrote: > 26.10.13 15:50, Stefan Krah ???????(??): > >> nick.coghlan wrote: >>> >>> http://hg.python.org/cpython/rev/a9bbc2d0c1dc >>> -HAVE_DOCSTRINGS = (check_impl_detail(cpython=False) or >>> - sys.platform == 'win32' or >>> - sysconfig.get_config_var('WITH_DOC_STRINGS')) >>> +# Rather than trying to enumerate all the cases where docstrings may be >>> +# disabled, we just check for that directly >>> + >>> +def _check_docstrings(): >>> + """Just used to check if docstrings are enabled""" >>> + >>> +HAVE_DOCSTRINGS = (_check_docstrings.__doc__ is not None) >>> >>> requires_docstrings = unittest.skipUnless(HAVE_DOCSTRINGS, >> >> >> I think that does not detect --without-doc-strings (i.e. the C docstrings >> are >> empty). > > > Indeed. HAVE_DOCSTRINGS was introduced to skip tests for the C docstrings. > Python docstrings tests are skipped if sys.flags.optimize >= 2. That's *extraordinarily* confusing, especially when Serhiy suggested I use the flag when testing a pure Python module. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Oct 26 19:41:48 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 27 Oct 2013 03:41:48 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Issue #19330: Handle the no-docstrings case in tests In-Reply-To: References: <3d6Lrm0pQcz7LlK@mail.python.org> <20131026125001.GA1257@sleipnir.bytereef.org> Message-ID: On 27 October 2013 03:32, Nick Coghlan wrote: > On 27 October 2013 01:10, Serhiy Storchaka wrote: >> 26.10.13 15:50, Stefan Krah ???????(??): >> >>> nick.coghlan wrote: >>>> >>>> http://hg.python.org/cpython/rev/a9bbc2d0c1dc >>>> -HAVE_DOCSTRINGS = (check_impl_detail(cpython=False) or >>>> - sys.platform == 'win32' or >>>> - sysconfig.get_config_var('WITH_DOC_STRINGS')) >>>> +# Rather than trying to enumerate all the cases where docstrings may be >>>> +# disabled, we just check for that directly >>>> + >>>> +def _check_docstrings(): >>>> + """Just used to check if docstrings are enabled""" >>>> + >>>> +HAVE_DOCSTRINGS = (_check_docstrings.__doc__ is not None) >>>> >>>> requires_docstrings = unittest.skipUnless(HAVE_DOCSTRINGS, >>> >>> >>> I think that does not detect --without-doc-strings (i.e. the C docstrings >>> are >>> empty). >> >> >> Indeed. HAVE_DOCSTRINGS was introduced to skip tests for the C docstrings. >> Python docstrings tests are skipped if sys.flags.optimize >= 2. > > That's *extraordinarily* confusing, especially when Serhiy suggested I > use the flag when testing a pure Python module. http://bugs.python.org/issue19412 Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Sat Oct 26 20:15:07 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 26 Oct 2013 21:15:07 +0300 Subject: [Python-Dev] [Python-checkins] cpython: Issue #19330: Handle the no-docstrings case in tests In-Reply-To: References: <3d6Lrm0pQcz7LlK@mail.python.org> <20131026125001.GA1257@sleipnir.bytereef.org> Message-ID: 26.10.13 20:32, Nick Coghlan ???????(??): > On 27 October 2013 01:10, Serhiy Storchaka wrote: >> 26.10.13 15:50, Stefan Krah ???????(??): >> >>> nick.coghlan wrote: >>>> >>>> http://hg.python.org/cpython/rev/a9bbc2d0c1dc >>>> -HAVE_DOCSTRINGS = (check_impl_detail(cpython=False) or >>>> - sys.platform == 'win32' or >>>> - sysconfig.get_config_var('WITH_DOC_STRINGS')) >>>> +# Rather than trying to enumerate all the cases where docstrings may be >>>> +# disabled, we just check for that directly >>>> + >>>> +def _check_docstrings(): >>>> + """Just used to check if docstrings are enabled""" >>>> + >>>> +HAVE_DOCSTRINGS = (_check_docstrings.__doc__ is not None) >>>> >>>> requires_docstrings = unittest.skipUnless(HAVE_DOCSTRINGS, >>> >>> >>> I think that does not detect --without-doc-strings (i.e. the C docstrings >>> are >>> empty). >> >> >> Indeed. HAVE_DOCSTRINGS was introduced to skip tests for the C docstrings. >> Python docstrings tests are skipped if sys.flags.optimize >= 2. > > That's *extraordinarily* confusing, especially when Serhiy suggested I > use the flag when testing a pure Python module. I'm sorry for misleading you. From christian at python.org Sun Oct 27 01:07:57 2013 From: christian at python.org (Christian Heimes) Date: Sun, 27 Oct 2013 01:07:57 +0200 Subject: [Python-Dev] cpython (2.7): 2.7.6rc1 In-Reply-To: <3d6WgY3D2Tz7Ljk@mail.python.org> References: <3d6WgY3D2Tz7Ljk@mail.python.org> Message-ID: <526C4B4D.5040905@python.org> Am 26.10.2013 20:58, schrieb benjamin.peterson: > http://hg.python.org/cpython/rev/4913d0e9be30 > changeset: 86665:4913d0e9be30 > branch: 2.7 > tag: v2.7.6rc1 > user: Benjamin Peterson > date: Sat Oct 26 14:57:21 2013 -0400 > summary: > 2.7.6rc1 We need to solve http://bugs.python.org/issue19227 before you release 2.7.6. I have suggested two approaches http://bugs.python.org/issue19227#msg201401 From ericsnowcurrently at gmail.com Sun Oct 27 02:38:03 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 26 Oct 2013 18:38:03 -0600 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Fri, Oct 25, 2013 at 9:44 AM, Eric Snow wrote: > On Fri, Oct 25, 2013 at 9:40 AM, Brett Cannon wrote: >> After reading Eric's doc patch, I realized there is one change I want to >> make to the current semantics and that's not to backfill __package__ when >> set to None. Since import is now going to take over the job of setting >> __package__ (along with other attributes), this seems like a slight waste of >> effort. It also kills (or at least complicates) having a loader which does >> lazy loading since reading the attribute to see if it is None would trigger >> the load before leaving the import code, thus killing any postponed loading. > > Fine with me. I may have time today to make the outstanding updates to the PEP. The changes are in now, as is an update to the doc patch on issue18864. If I missed anything, let me know. Thanks for the feedback! -eric From ncoghlan at gmail.com Sun Oct 27 02:34:27 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 27 Oct 2013 11:34:27 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Issue #19330: Handle the no-docstrings case in tests In-Reply-To: References: <3d6Lrm0pQcz7LlK@mail.python.org> <20131026125001.GA1257@sleipnir.bytereef.org> Message-ID: On 27 Oct 2013 04:16, "Serhiy Storchaka" wrote: > > 26.10.13 20:32, Nick Coghlan ???????(??): > >> On 27 October 2013 01:10, Serhiy Storchaka wrote: >>> >>> 26.10.13 15:50, Stefan Krah ???????(??): >>> >>>> nick.coghlan wrote: >>>>> >>>>> >>>>> http://hg.python.org/cpython/rev/a9bbc2d0c1dc >>>>> -HAVE_DOCSTRINGS = (check_impl_detail(cpython=False) or >>>>> - sys.platform == 'win32' or >>>>> - sysconfig.get_config_var('WITH_DOC_STRINGS')) >>>>> +# Rather than trying to enumerate all the cases where docstrings may be >>>>> +# disabled, we just check for that directly >>>>> + >>>>> +def _check_docstrings(): >>>>> + """Just used to check if docstrings are enabled""" >>>>> + >>>>> +HAVE_DOCSTRINGS = (_check_docstrings.__doc__ is not None) >>>>> >>>>> requires_docstrings = unittest.skipUnless(HAVE_DOCSTRINGS, >>>> >>>> >>>> >>>> I think that does not detect --without-doc-strings (i.e. the C docstrings >>>> are >>>> empty). >>> >>> >>> >>> Indeed. HAVE_DOCSTRINGS was introduced to skip tests for the C docstrings. >>> Python docstrings tests are skipped if sys.flags.optimize >= 2. >> >> >> That's *extraordinarily* confusing, especially when Serhiy suggested I >> use the flag when testing a pure Python module. > > > I'm sorry for misleading you. And my apologies for going ahead and assuming I understood what the flag was for rather than seeking clarification. I'll put together a patch to split it into two flags for the different meanings. Cheers, Nick. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Oct 27 02:44:22 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 26 Oct 2013 17:44:22 -0700 Subject: [Python-Dev] OrderedDict.values() behavior for modified instance In-Reply-To: <87iowklnqq.fsf@vostro.rath.org> References: <87iowklnqq.fsf@vostro.rath.org> Message-ID: <526C61E6.2060306@stoneleaf.us> Nikolaus, Good write-up. Please submit it to the bug tracker: http://bugs.python.org -- ~Ethan~ From Nikolaus at rath.org Sun Oct 27 04:32:50 2013 From: Nikolaus at rath.org (Nikolaus Rath) Date: Sat, 26 Oct 2013 20:32:50 -0700 Subject: [Python-Dev] OrderedDict.values() behavior for modified instance In-Reply-To: <526C61E6.2060306@stoneleaf.us> (Ethan Furman's message of "Sat, 26 Oct 2013 17:44:22 -0700") References: <87iowklnqq.fsf@vostro.rath.org> <526C61E6.2060306@stoneleaf.us> Message-ID: <87txg3xual.fsf@vostro.rath.org> Ethan Furman writes: > Nikolaus, > > Good write-up. Please submit it to the bug tracker: > http://bugs.python.org Submitted as http://bugs.python.org/issue19414. If someone gives me the go-ahead for one of the proposed solutions, I'd be happy to create a full patch. Best, Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ?Time flies like an arrow, fruit flies like a Banana.? From ericsnowcurrently at gmail.com Sun Oct 27 04:44:03 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 26 Oct 2013 21:44:03 -0600 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Sat, Oct 26, 2013 at 11:27 AM, Nick Coghlan wrote: > On 26 October 2013 08:51, PJ Eby wrote: >> Yeah, it actually was to ensure that you could reload a module using a >> different loader than the one that originally loaded it, e.g. due to a >> change in path hooks, etc. > > Yeah, the rationale makes sense, we only missed it due to the lack of > a regression test for the behaviour. I put up a patch that should fix this without a lot of work, including a test that would have caught this. http://bugs.python.org/issue19413 -eric From ncoghlan at gmail.com Sun Oct 27 04:44:11 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 27 Oct 2013 13:44:11 +1000 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On 27 October 2013 03:27, Nick Coghlan wrote: > On 26 October 2013 08:51, PJ Eby wrote: >> Mostly, this just seems like an ugly wart -- Python should be dynamic >> by default, and that includes reloading. While the import machinery >> has lots of ugly caching under the hood, a user-level function like >> reload() should not require you to do the equivalent of saying, "no, >> really... I want you to *really* reload, not just pull in whatever >> exists where you found it last time, while ignoring whether I switched >> from module to package or vice versa, or just fixed my sys.path so I >> can load the right version of the module." >> >> It is a really tiny thing in the overall scheme of things, because >> reload() is not used all that often, but it's still a thing. If this >> isn't treated as a bug, then the docs for reload() at least need to >> include a forward-supported workaround so you can say "no, really... >> *really* reload" in an approved fashion. >> >> (ISTM that any production code out there that currently uses reload() >> would want to perform the "really reload" incantation in order to >> avoid the edge cases, even if they haven't actually run into any of >> them yet.) > > Having imp.reload() kick off the full reload cycle makes sense to me. > The only reason it doesn't currently do that in importlib is because > there was no test for it in the regression test suite, and PEP 302 and > the library reference are entirely vague on how module reloading works > (aside from the requirement that it reuse the same module namespace). I've been thinking about this, and I've come to the conclusion that: 1. The old __name__ based behaviour was broken (since __name__ may not be the actual module name, as in "__main__" or when a pseudo-module is lying about its identity) 2. The 3.3 behaviour is still broken for similar reasons *and* added the dependency on __loader__ 3. Even PEP 451 still underspecifies the semantics of reloading (e.g. it's implied create_module isn't called for reload(), but not stated explicitly) Accordingly, I think we should add a "How Reloading Will Work" section, akin to the existing "How Loading Will Work" (http://www.python.org/dev/peps/pep-0451/#how-loading-will-work). We may also want to spit out some import warnings for edge cases that are likely to do the wrong thing. Side note: in "How Loading Will Work", _init_module_attrs needs to be passed both the module *and* the spec (current text only passes it the module). It also needs to be updated to handle the namespace package case (as shown below) And a sketch of some possible fully specified reload semantics: def reload(module): # Get the name to reload from the spec if it is available, otherwise __name__ current_spec = getattr(module, "__spec__") current_name = module.__name__ name_to_reload = current_name if current_spec is None else current_spec.name # Check this module is properly imported before trying to reload it loaded_module = sys.modules.get(name_to_reload) if loaded_module is not module: msg = "module {} not in sys.modules" raise ImportError(msg.format(name_to_reload), name=name_to_reload) parent_name = name_to_reload.rpartition('.')[0] while parent_name: if parent_name not in sys.modules: msg = "parent {!r} not in sys.modules" raise ImportError(msg.format(parent_name), name=parent_name) parent_name = parent_name.rpartition('.')[0] # Check for a modified spec (e.g. if the import hooks have changed) updated_spec = importlib.find_spec(name_to_reload) # The import-related module attributes get updated here # We leave __name__ alone, so reloading __main__ has some hope of working correctly _init_module_attrs(loaded_module, updated_spec, set_name=False) # Now re-execute the module loader = updated_spec.loader if loader is None: # Namespace package, already reinitialised above return loaded_module elif hasattr(loader, 'exec_module'): loader.exec_module(module) else: loader.load_module(name_to_reload) # Reloading modules that replace themselves in sys.modules doesn't work reloaded_module = sys.modules[name_to_reload] if reloaded_module is not loaded_module: # Perhaps this should initially be a DeprecationWarning? # We've never actively *prevented* this case before, even though it is # almost guaranteed to break things if you do it. Alternatively, we could # leave "imp.reload()" as the more permissive version, and include this # check in the new importlib.reload() (with imp.reload calling an internal # importlib._unchecked_reload() instead) msg = "Module {!r} was replaced in sys.modules during reload" raise ImportError(msg.format(name_to_reload), name=name_to_reload) return reloaded_module -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ericsnowcurrently at gmail.com Sun Oct 27 05:29:29 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 26 Oct 2013 22:29:29 -0600 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Sat, Oct 26, 2013 at 9:44 PM, Nick Coghlan wrote: > I've been thinking about this, and I've come to the conclusion that: > > 1. The old __name__ based behaviour was broken (since __name__ may not > be the actual module name, as in "__main__" or when a pseudo-module is > lying about its identity) > 2. The 3.3 behaviour is still broken for similar reasons *and* added > the dependency on __loader__ > 3. Even PEP 451 still underspecifies the semantics of reloading (e.g. > it's implied create_module isn't called for reload(), but not stated > explicitly) > > Accordingly, I think we should add a "How Reloading Will Work" > section, akin to the existing "How Loading Will Work" > (http://www.python.org/dev/peps/pep-0451/#how-loading-will-work). We > may also want to spit out some import warnings for edge cases that are > likely to do the wrong thing. I'm tempted to just say reload should not be a blocker for PEP 451. The premise is that the PEP mostly maintains the status quo, just shifting reload-based-on-loader to reload-based-on-spec (it's still "*load* the same way you did the first time"). I agree that it would be worth getting reload working right, but I'm not convinced it's worth it to delay PEP 451 more for the sake of reload. > > Side note: in "How Loading Will Work", _init_module_attrs needs to be > passed both the module *and* the spec (current text only passes it the > module). Good catch. Fixed! > It also needs to be updated to handle the namespace package > case (as shown below) I knew I missed something. I'll fix this, but it may be a couple days before I get a minute. :( -eric From ncoghlan at gmail.com Sun Oct 27 06:03:33 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 27 Oct 2013 15:03:33 +1000 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On 27 October 2013 14:29, Eric Snow wrote: > On Sat, Oct 26, 2013 at 9:44 PM, Nick Coghlan wrote: >> Accordingly, I think we should add a "How Reloading Will Work" >> section, akin to the existing "How Loading Will Work" >> (http://www.python.org/dev/peps/pep-0451/#how-loading-will-work). We >> may also want to spit out some import warnings for edge cases that are >> likely to do the wrong thing. > > I'm tempted to just say reload should not be a blocker for PEP 451. > The premise is that the PEP mostly maintains the status quo, just > shifting reload-based-on-loader to reload-based-on-spec (it's still > "*load* the same way you did the first time"). > > I agree that it would be worth getting reload working right, but I'm > not convinced it's worth it to delay PEP 451 more for the sake of > reload. I don't think we can postpone it to later, as we need to be clear on: 1. Does reload use __name__ or __spec__.name when both are available? 2. Does __name__ get restored to its original value if reloading via __spec__.name? 3. Do other import related attributes get restored to their original values? 4. Does create_module get called if the loader has an exec_module method? 5. Does the loader have access to the previous spec when reloading a module? My answers to these questions (note that my answer to 2 differs from what I had in my initial sketch): 1. __spec__.name 2. Yes, __name__ is updated to match __spec__.name, *expect* if it is currently "__main__" 3. Yes, other import related attributes are restored to their baseline values 4. No, create_module is never called when reloading 5. Currently no, but I'm thinking that may be worth changing (more on that below) The reload() implementation in my message is actually based on the current importlib.reload implementation. The only PEP 451 related changes were: - using __spec__.name (if available) instead of __name__ - checking all parent modules exist rather than just the immediate parent module - calling import.find_spec() rather than using the __loader__ attribute directly - being explicit that __name__ is left at the value it had prior to the reload - handling the namespace package, exec_module and no exec_module cases I also added an explicit check that the module was re-used properly, but I agree *that* change is out of scope for the PEP and should be considered as a separate question. Now, regarding the signature of exec_module(): I'm back to believing that loaders should receive a clear indication that a reload is taking place. Legacy loaders have to figure that out for themselves (by seeing that the module already exists in sys.modules), but we can do better for the new API by making the exec_module signature look like: def exec_module(self, module, previous_spec=None): # module is as per the current PEP 451 text # previous_spec would be set *only* in the reload() case # loaders that don't care still need to accept it, but can just ignore it ... So, with those revisions, the reload() semantics would be defined as: def reload(module): # Get the name to reload from the spec if it is available, # otherwise use __name__ directly current_spec = getattr(module, "__spec__") current_name = module.__name__ name_to_reload = current_name if current_spec is None else current_spec.name # Check this module is properly imported before trying to reload it loaded_module = sys.modules.get(name_to_reload) if loaded_module is not module: msg = "module {} not in sys.modules" raise ImportError(msg.format(name_to_reload), name=name_to_reload) parent_name = name_to_reload.rpartition('.')[0] while parent_name: if parent_name not in sys.modules: msg = "parent {!r} not in sys.modules" raise ImportError(msg.format(parent_name), name=parent_name) parent_name = parent_name.rpartition('.')[0] # Check for a modified spec (e.g. if the import hooks have changed) updated_spec = importlib.find_spec(name_to_reload) # The import-related module attributes get updated here _init_module_attrs(loaded_module, updated_spec) if current_name == "__main__": loaded_module.__name__ = "__main__" # Now re-execute the module loader = updated_spec.loader if loader is None: # Namespace package, already reinitialised above return loaded_module elif hasattr(loader, 'exec_module'): loader.exec_module(module, current_spec) else: loader.load_module(name_to_reload) # Allow for the module to be replaced in sys.modules # (even though that likely won't work very well) return sys.modules[name_to_reload] Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Sun Oct 27 17:26:17 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 27 Oct 2013 17:26:17 +0100 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: Message-ID: I updated the PEP according to different comments: * replace Snapshot.create() class method with take_snapshot() function * get_traces() now returns a list instead of a dict (remove addresses) * remove get_stats() * unknown frames are now stored as ("", 0) instead of (None, None) * remove get_object_address() * remove get_trace() * remove add_inclusive_filter() and add_exclusive_filter() functions * remove "address" key type from Snapshot.group_by() http://www.python.org/dev/peps/pep-0454/ (not updated yet) Remaining questions: * do you have a better suggestion for GroupedStats.stats attribute or GroupedStats.statistics() method: "stats" and "statistics" are too close * should GroupedStats.statistics() and GroupedStats.compare_to() always sort the result to drop the sort=True parameter? * do you prefer Snapshot.create() class method or take_snapshot() function? There is also a Snapload.load() class method Victor From pje at telecommunity.com Sun Oct 27 17:37:00 2013 From: pje at telecommunity.com (PJ Eby) Date: Sun, 27 Oct 2013 12:37:00 -0400 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Sun, Oct 27, 2013 at 1:03 AM, Nick Coghlan wrote: > Now, regarding the signature of exec_module(): I'm back to believing > that loaders should receive a clear indication that a reload is taking > place. Legacy loaders have to figure that out for themselves (by > seeing that the module already exists in sys.modules), but we can do > better for the new API by making the exec_module signature look like: > > def exec_module(self, module, previous_spec=None): > # module is as per the current PEP 451 text > # previous_spec would be set *only* in the reload() case > # loaders that don't care still need to accept it, but can > just ignore it Just to be clear, this means that a lazy import implementation that creates a module object without a __spec__ in the first place will look like an initial import? Or will that crash importlib because of a missing __spec__ attribute? That is, is reload()'s contract adding a new prerequisite for the object passed to it? (The specific use case is creating a ModuleType subclass instance for lazy importing upon attribute access. Pre-importlib, all that was needed was a working __name__ attribute on the module.) From ethan at stoneleaf.us Sun Oct 27 17:56:09 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 27 Oct 2013 09:56:09 -0700 Subject: [Python-Dev] OrderedDict.values() behavior for modified instance In-Reply-To: <87txg3xual.fsf@vostro.rath.org> References: <87iowklnqq.fsf@vostro.rath.org> <526C61E6.2060306@stoneleaf.us> <87txg3xual.fsf@vostro.rath.org> Message-ID: <526D45A9.4030205@stoneleaf.us> On 10/26/2013 08:32 PM, Nikolaus Rath wrote: > Ethan Furman writes: >> >> Good write-up. Please submit it to the bug tracker: >> http://bugs.python.org > > Submitted as http://bugs.python.org/issue19414. > > If someone gives me the go-ahead for one of the proposed solutions, I'd > be happy to create a full patch. That would be Raymond's call. Give him a few days to respond. -- ~Ethan~ From victor.stinner at gmail.com Sun Oct 27 19:02:40 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 27 Oct 2013 19:02:40 +0100 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: Message-ID: Hum, timestamp and traceback_limit attributed of GroupedStats can also be removed, they are just of the same attribute of the Snapshot class and GroupedStats is created from Snapshot.group_by(). Le 27 oct. 2013 17:26, "Victor Stinner" a ?crit : > * do you have a better suggestion for GroupedStats.stats attribute or > GroupedStats.statistics() method: "stats" and "statistics" are too > close An option is to modify GroupedStats to inherit from dict, so GroupedStats.stats would be GroupedStats directly. > * should GroupedStats.statistics() and GroupedStats.compare_to() > always sort the result to drop the sort=True parameter? The result is a list which can be bigger than 100,000 items. The question is if it would inefficient to sort the list twice. Another option is to replace sort parameter with a new sort_key parameter (callable), passed to list.sort as the ke parameter. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sun Oct 27 20:51:13 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 27 Oct 2013 15:51:13 -0400 Subject: [Python-Dev] [RELEASE] Python 2.7.6 release candidate 1 Message-ID: I'm happy to announce the availability of Python 2.7.6 release candidate 1. Most importantly, this release candidate resolves crashes of the interactive interpreter on OS X 10.9. It also includes the usual collection of bugfixes over 2.7.5. These are described in excruciating detail in the Misc/NEWS file of the source tarball. You can view it online at http://hg.python.org/cpython/raw-file/9750acbf7c40/Misc/NEWS Downloads are at http://python.org/download/releases/2.7.6/ As always, please test the release and report bugs to http://bugs.python.org/ With any luck, the final release will follow in a week. Enjoy, Benjamin Peterson 2.7 Release Manager From georg at python.org Sun Oct 27 21:47:05 2013 From: georg at python.org (Georg Brandl) Date: Sun, 27 Oct 2013 21:47:05 +0100 Subject: [Python-Dev] [RELEASED] Python 3.3.3 release candidate 1 Message-ID: <526D7BC9.5060609@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team, I'm quite happy to announce the Python 3.3.3 release candidate 1. Python 3.3.3 includes several security fixes and over 150 bug fixes compared to the Python 3.3.2 release. This release fully supports OS X 10.9 Mavericks. In particular, this release fixes an issue that could cause previous versions of Python to crash when typing in interactive mode on OS X 10.9. Python 3.3.3 also contains a new batteries-included feature for OS X users of IDLE and other Tkinter-based programs. The python.org Mac OS X 64-bit/32-bit x86-64/i386 Installer for OS X 10.6+ now includes its own builtin version of Tcl/Tk 8.5. It is no longer necessary to install a third-party version of Tcl/Tk 8.5 to workaround the problematic system versions. See http://www.python.org/download/mac/tcltk/ for more information. Python 3.3 includes a range of improvements of the 3.x series, as well as easier porting between 2.x and 3.x. In total, almost 500 API items are new or improved in Python 3.3. For a more extensive list of changes in the 3.3 series, see http://docs.python.org/3.3/whatsnew/3.3.html and for the detailed changelog of 3.3.3, see http://docs.python.org/3.3/whatsnew/changelog.html To download Python 3.3.3 rc1 visit: http://www.python.org/download/releases/3.3.3/ This is a preview release, please report any bugs to http://bugs.python.org/ The final version is scheduled to be released in two weeks' time, on or about the 10th of November. Enjoy! - -- Georg Brandl, Release Manager georg at python.org (on behalf of the entire python-dev team and 3.3's contributors) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlJte8kACgkQN9GcIYhpnLDx8wCgqtabbC8DaoW+Vy03AYGWyLhw sWcAoIK5jQeXDAxHayG+VWH/rRF1+qHC =yOed -----END PGP SIGNATURE----- From ncoghlan at gmail.com Sun Oct 27 21:59:24 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 28 Oct 2013 06:59:24 +1000 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On 28 Oct 2013 02:37, "PJ Eby" wrote: > > On Sun, Oct 27, 2013 at 1:03 AM, Nick Coghlan wrote: > > Now, regarding the signature of exec_module(): I'm back to believing > > that loaders should receive a clear indication that a reload is taking > > place. Legacy loaders have to figure that out for themselves (by > > seeing that the module already exists in sys.modules), but we can do > > better for the new API by making the exec_module signature look like: > > > > def exec_module(self, module, previous_spec=None): > > # module is as per the current PEP 451 text > > # previous_spec would be set *only* in the reload() case > > # loaders that don't care still need to accept it, but can > > just ignore it > > Just to be clear, this means that a lazy import implementation that > creates a module object without a __spec__ in the first place will > look like an initial import? Or will that crash importlib because of > a missing __spec__ attribute? > > That is, is reload()'s contract adding a new prerequisite for the > object passed to it? > > (The specific use case is creating a ModuleType subclass instance for > lazy importing upon attribute access. Pre-importlib, all that was > needed was a working __name__ attribute on the module.) For custom loaders, that's part of the contract for create_module() (since you'll get an ordinary module otherwise), and so long as *setting* the special module attributes doesn't cause the module to be imported during the initial load operation, attribute access based lazy loading will work fine (and you don't even have to set __name__, since the import machinery will take care of that). For module level lazy loading that injects a partially initialised module object into sys.modules rather than using a custom loader or setting a __spec__ attribute, yes, the exec_module invocation on reloading would always look like a fresh load operation (aside from the fact that the custom instance would already be in sys.modules from the first load operation). It *will* still work, though (at least, it won't break any worse than such code does today, since injecting a replacement into sys.modules really isn't reload friendly in the first place). Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliver at bestwalter.de Sun Oct 27 21:31:19 2013 From: oliver at bestwalter.de (Oliver Bestwalter) Date: Sun, 27 Oct 2013 20:31:19 +0000 (UTC) Subject: [Python-Dev] pathlib (PEP 428) status References: <52684FCA.1060904@python.org> Message-ID: Hi, Christian Heimes python.org> writes: > [...] Last week at PyCon.DE in Cologne > several people have asked me about pathlib. We even had a BarCamp > session about path libraries for Python. A couple of German Python users > have promised to contribute doc improvements soonish. Yes I am one of those germans :) I gave a talk there, where I asked the naive question about the 'missing' path class in Python and it resonated with quite a few of the participants, so I guess this would be appreciated by a lot of Python users. Anyway - I am willing to give Antoine a hand in getting pathlib into Python 3.4. I will see what I can do. So +1 from me :) Cheers Oliver From pje at telecommunity.com Sun Oct 27 23:41:25 2013 From: pje at telecommunity.com (PJ Eby) Date: Sun, 27 Oct 2013 18:41:25 -0400 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Sun, Oct 27, 2013 at 4:59 PM, Nick Coghlan wrote: > > On 28 Oct 2013 02:37, "PJ Eby" wrote: >> >> On Sun, Oct 27, 2013 at 1:03 AM, Nick Coghlan wrote: >> > Now, regarding the signature of exec_module(): I'm back to believing >> > that loaders should receive a clear indication that a reload is taking >> > place. Legacy loaders have to figure that out for themselves (by >> > seeing that the module already exists in sys.modules), but we can do >> > better for the new API by making the exec_module signature look like: >> > >> > def exec_module(self, module, previous_spec=None): >> > # module is as per the current PEP 451 text >> > # previous_spec would be set *only* in the reload() case >> > # loaders that don't care still need to accept it, but can >> > just ignore it >> >> Just to be clear, this means that a lazy import implementation that >> creates a module object without a __spec__ in the first place will >> look like an initial import? Or will that crash importlib because of >> a missing __spec__ attribute? >> >> That is, is reload()'s contract adding a new prerequisite for the >> object passed to it? >> >> (The specific use case is creating a ModuleType subclass instance for >> lazy importing upon attribute access. Pre-importlib, all that was >> needed was a working __name__ attribute on the module.) > > For custom loaders, that's part of the contract for create_module() (since > you'll get an ordinary module otherwise), Huh? I don't understand where custom loaders come into it. For that matter, I don't understand what "get an ordinary module object" means here, either. I'm talking about userspace code that implements lazy importing features, like the lazyModule() function in this module: http://svn.eby-sarna.com/Importing/peak/util/imports.py?view=markup Specifically, I'm trying to get an idea of how much that code will need to change under the PEP (and apparently under importlib in general). > and so long as *setting* the > special module attributes doesn't cause the module to be imported during the > initial load operation, attribute access based lazy loading will work fine > (and you don't even have to set __name__, since the import machinery will > take care of that). There's no "initial load operation", just creation of a dummy module and stuffing it into sys.modules. The way it works is that in, say, foo/__init__.py, one uses: bar = lazyModule('foo.bar') baz = lazyModule('foo.baz') Then anybody importing 'foo.bar' or 'foo.baz' (or using "from foo import bar", etc.) ends up with the lazy module. That is, it's for lazily exposing APIs, not something used as an import hook. > For module level lazy loading that injects a partially initialised module > object into sys.modules rather than using a custom loader or setting a > __spec__ attribute, yes, the exec_module invocation on reloading would > always look like a fresh load operation (aside from the fact that the custom > instance would already be in sys.modules from the first load operation). Right. > It *will* still work, though (at least, it won't break any worse than such code > does today, since injecting a replacement into sys.modules really isn't > reload friendly in the first place). Wait, what? Who's injecting a replacement into sys.modules? A replacement of what? Or do you mean that loaders aren't supposed to create new modules, but use the one in sys.modules? Honestly, I'm finding all this stuff *really* confusing, which is kind of worrying. I mean, I gather I'm one of the handful of people who really understood how importing *used to work*, and I'm having a lot of trouble wrapping my brain around the new world. (Granted, I think that may be because I understand how a lot of old corner cases work, but what's bugging me is that I no longer understand how those old corners work under the new regime, nor do I feel I understand what the *new* corners will be. This may also just be communication problems, and the fact that it's been months since I really walked through importlib line by line, and have never really walked through it (or PEP 451) quite as thoroughly as I have import.c. I also seem to be having trouble grokking why the motivating use cases for PEP 451 can't be solved by just providing people with good base classes to use for writing loaders -- i.e., I don't get why the core protocol has to change to address the use case of writing loaders more easily. The new protocol seems way more complex than PEP 302, and ISTM the complexity could just be pushed off to the loader side of the protocol without creating more interdependency between importlib and the loaders.) From ncoghlan at gmail.com Mon Oct 28 04:53:11 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 28 Oct 2013 13:53:11 +1000 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On 28 Oct 2013 08:41, "PJ Eby" wrote: > > On Sun, Oct 27, 2013 at 4:59 PM, Nick Coghlan wrote: > > > > On 28 Oct 2013 02:37, "PJ Eby" wrote: > >> > >> On Sun, Oct 27, 2013 at 1:03 AM, Nick Coghlan wrote: > >> > Now, regarding the signature of exec_module(): I'm back to believing > >> > that loaders should receive a clear indication that a reload is taking > >> > place. Legacy loaders have to figure that out for themselves (by > >> > seeing that the module already exists in sys.modules), but we can do > >> > better for the new API by making the exec_module signature look like: > >> > > >> > def exec_module(self, module, previous_spec=None): > >> > # module is as per the current PEP 451 text > >> > # previous_spec would be set *only* in the reload() case > >> > # loaders that don't care still need to accept it, but can > >> > just ignore it > >> > >> Just to be clear, this means that a lazy import implementation that > >> creates a module object without a __spec__ in the first place will > >> look like an initial import? Or will that crash importlib because of > >> a missing __spec__ attribute? > >> > >> That is, is reload()'s contract adding a new prerequisite for the > >> object passed to it? > >> > >> (The specific use case is creating a ModuleType subclass instance for > >> lazy importing upon attribute access. Pre-importlib, all that was > >> needed was a working __name__ attribute on the module.) > > > > For custom loaders, that's part of the contract for create_module() (since > > you'll get an ordinary module otherwise), > > Huh? I don't understand where custom loaders come into it. For that > matter, I don't understand what "get an ordinary module object" means > here, either. > > I'm talking about userspace code that implements lazy importing > features, like the lazyModule() function in this module: > > http://svn.eby-sarna.com/Importing/peak/util/imports.py?view=markup > > Specifically, I'm trying to get an idea of how much that code will > need to change under the PEP (and apparently under importlib in > general). If the lazy import is injected by a *different* module, then nothing changes. > > and so long as *setting* the > > special module attributes doesn't cause the module to be imported during the > > initial load operation, attribute access based lazy loading will work fine > > (and you don't even have to set __name__, since the import machinery will > > take care of that). > > There's no "initial load operation", just creation of a dummy module > and stuffing it into sys.modules. The way it works is that in, say, > foo/__init__.py, one uses: > > bar = lazyModule('foo.bar') > baz = lazyModule('foo.baz') > > Then anybody importing 'foo.bar' or 'foo.baz' (or using "from foo > import bar", etc.) ends up with the lazy module. That is, it's for > lazily exposing APIs, not something used as an import hook. I was thinking of the more complex case where a module causes *itself* to be loaded lazily. Nothing changes for the simpler case where the injection occurs in a different module. > > For module level lazy loading that injects a partially initialised module > > object into sys.modules rather than using a custom loader or setting a > > __spec__ attribute, yes, the exec_module invocation on reloading would > > always look like a fresh load operation (aside from the fact that the custom > > instance would already be in sys.modules from the first load operation). > > Right. > > > > It *will* still work, though (at least, it won't break any worse than such code > > does today, since injecting a replacement into sys.modules really isn't > > reload friendly in the first place). > > Wait, what? Who's injecting a replacement into sys.modules? A > replacement of what? Or do you mean that loaders aren't supposed to > create new modules, but use the one in sys.modules? > > Honestly, I'm finding all this stuff *really* confusing, which is kind > of worrying. I mean, I gather I'm one of the handful of people who > really understood how importing *used to work*, and I'm having a lot > of trouble wrapping my brain around the new world. Provide test cases that exercise the situations you're concerned about supporting and then you don't need to worry about them breaking. > > (Granted, I think that may be because I understand how a lot of old > corner cases work, but what's bugging me is that I no longer > understand how those old corners work under the new regime, nor do I > feel I understand what the *new* corners will be. This may also just > be communication problems, and the fact that it's been months since I > really walked through importlib line by line, and have never really > walked through it (or PEP 451) quite as thoroughly as I have import.c. > I also seem to be having trouble grokking why the motivating use > cases for PEP 451 can't be solved by just providing people with good > base classes to use for writing loaders -- i.e., I don't get why the > core protocol has to change to address the use case of writing loaders > more easily. The new protocol seems way more complex than PEP 302, > and ISTM the complexity could just be pushed off to the loader side of > the protocol without creating more interdependency between importlib > and the loaders.) Bad loader implementations have too much power to break the import system, and loaders don't currently expose enough information about modules that *could* be imported. We also take "inherit from this class to implement the protocol correctly, import will break if you don't" as a sign that the *protocol is broken* by pushing too much of the work onto the plugins. New style loaders can be much simpler, because the import system takes care of the complexity, without relying on users inheriting from a particular base class. The complexity comes from the fact that we're now breaking down what the *real* expectations on a loader actually are, and making those part of the import system itself. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From seotaewong40 at gmail.com Mon Oct 28 11:13:49 2013 From: seotaewong40 at gmail.com (Tae Wong) Date: Mon, 28 Oct 2013 19:13:49 +0900 Subject: [Python-Dev] Merge codeaccess.txt from wesnoth-contribcommunity project in Google Code Hosting In-Reply-To: References: Message-ID: Your suggestion needs to be accepted in the Misc/ACKS file. -- Tae Wong From victor.stinner at gmail.com Mon Oct 28 19:45:33 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 28 Oct 2013 19:45:33 +0100 Subject: [Python-Dev] Block module import during Python finalization? Message-ID: Hi, While working on a fix, I got an assertion error during Python finalization because Python tried to import the "io" module whereas the module was just unloaded. Python tried to import the io module to display a warning and display the Python line where the warning was emitted. See the following issue for detail: http://bugs.python.org/issue19421 I propose to block importing new modules (or import again unloaded modules) during Python finalization to avoid such tricky bugs. What do you think? Does anyone rely on the ability of importing Python modules at exit? :-) At least, the whole test suite pass with my import_finalization.patch which block import during Python finalization. Victor From victor.stinner at gmail.com Mon Oct 28 20:16:45 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 28 Oct 2013 20:16:45 +0100 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> Message-ID: 2013/10/4 Raymond Hettinger : >> On Sep 22, 2013, at 6:16 PM, Ethan Furman wrote: >> > Are we close to asking for pronouncement? > > When you're ready, let me know. The deadline for new features is in less than 1 month, so what is the status of the PEP 455 (TransformDict)? Victor From benjamin at python.org Mon Oct 28 21:13:36 2013 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 28 Oct 2013 16:13:36 -0400 Subject: [Python-Dev] Block module import during Python finalization? In-Reply-To: References: Message-ID: 2013/10/28 Victor Stinner : > Hi, > > While working on a fix, I got an assertion error during Python > finalization because Python tried to import the "io" module whereas > the module was just unloaded. Python tried to import the io module to > display a warning and display the Python line where the warning was > emitted. See the following issue for detail: > http://bugs.python.org/issue19421 > > I propose to block importing new modules (or import again unloaded > modules) during Python finalization to avoid such tricky bugs. > > What do you think? Though the practice of importing during finalization seems dubious, baning it may be a compatibility problem. -- Regards, Benjamin From solipsis at pitrou.net Mon Oct 28 21:18:44 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 28 Oct 2013 21:18:44 +0100 Subject: [Python-Dev] Block module import during Python finalization? References: Message-ID: <20131028211844.7a31b031@fsol> On Mon, 28 Oct 2013 16:13:36 -0400 Benjamin Peterson wrote: > 2013/10/28 Victor Stinner : > > Hi, > > > > While working on a fix, I got an assertion error during Python > > finalization because Python tried to import the "io" module whereas > > the module was just unloaded. Python tried to import the io module to > > display a warning and display the Python line where the warning was > > emitted. See the following issue for detail: > > http://bugs.python.org/issue19421 > > > > I propose to block importing new modules (or import again unloaded > > modules) during Python finalization to avoid such tricky bugs. > > > > What do you think? > > Though the practice of importing during finalization seems dubious, > baning it may be a compatibility problem. It may just be because some finalizer is calling an API that uses lazy imports. Honestly, I don't event think this issue is a bug. Importing modules at shutdown isn't worse than creating functions or classes at shutdown. Regards Antoine. From ericsnowcurrently at gmail.com Tue Oct 29 00:38:17 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 28 Oct 2013 17:38:17 -0600 Subject: [Python-Dev] Block module import during Python finalization? In-Reply-To: References: Message-ID: On Mon, Oct 28, 2013 at 12:45 PM, Victor Stinner wrote: > I propose to block importing new modules (or import again unloaded > modules) during Python finalization to avoid such tricky bugs. Alternatively, we could set sys.modules[name] to None for each builtin/frozen module when they get finalized. This will prevent importing those modules afterward. -eric From jimjjewett at gmail.com Tue Oct 29 03:00:34 2013 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 28 Oct 2013 22:00:34 -0400 Subject: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear? Message-ID: reset() function: Clear traces of memory blocks allocated by Python. Does this do anything besides clear? If not, why not just re-use the 'clear' name from dicts? disable() function: Stop tracing Python memory allocations and clear traces of memory blocks allocated by Python. I would disable to stop tracing, but I would not expect it to clear out the traces it had already captured. If it has to do that, please put in some sample code showing how to save the current traces before disabling. -jJ From ericsnowcurrently at gmail.com Tue Oct 29 05:45:17 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 28 Oct 2013 22:45:17 -0600 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Sat, Oct 26, 2013 at 11:03 PM, Nick Coghlan wrote: > I don't think we can postpone it to later, as we need to be clear on: > > 1. Does reload use __name__ or __spec__.name when both are available? > 2. Does __name__ get restored to its original value if reloading via > __spec__.name? > 3. Do other import related attributes get restored to their original values? > 4. Does create_module get called if the loader has an exec_module method? > 5. Does the loader have access to the previous spec when reloading a module? > > My answers to these questions (note that my answer to 2 differs from > what I had in my initial sketch): > > 1. __spec__.name > 2. Yes, __name__ is updated to match __spec__.name, *expect* if it is > currently "__main__" > 3. Yes, other import related attributes are restored to their baseline values > 4. No, create_module is never called when reloading I agree on all of these. I'm adding a "How reloading will work" section to the PEP. > 5. Currently no, but I'm thinking that may be worth changing (more on > that below) > > The reload() implementation in my message is actually based on the > current importlib.reload implementation. The only PEP 451 related > changes were: > > - using __spec__.name (if available) instead of __name__ > - checking all parent modules exist rather than just the immediate parent module > - calling import.find_spec() rather than using the __loader__ attribute directly > - being explicit that __name__ is left at the value it had prior to the reload > - handling the namespace package, exec_module and no exec_module cases > > I also added an explicit check that the module was re-used properly, > but I agree *that* change is out of scope for the PEP and should be > considered as a separate question. > > Now, regarding the signature of exec_module(): I'm back to believing > that loaders should receive a clear indication that a reload is taking > place. Legacy loaders have to figure that out for themselves (by > seeing that the module already exists in sys.modules), but we can do > better for the new API by making the exec_module signature look like: > > def exec_module(self, module, previous_spec=None): > # module is as per the current PEP 451 text > # previous_spec would be set *only* in the reload() case > # loaders that don't care still need to accept it, but can > just ignore it > ... I'd rather give loaders another optional method: supports_reload(name). Complicating the spec methods signatures (to support passing the old spec through) just for the sake of reload seems less than optimal. I don't see any benefit to exposing the old spec to loader.exec_module(). > > So, with those revisions, the reload() semantics would be defined as: > > def reload(module): > # Get the name to reload from the spec if it is available, > # otherwise use __name__ directly > current_spec = getattr(module, "__spec__") > current_name = module.__name__ > name_to_reload = current_name if current_spec is None else > current_spec.name > > # Check this module is properly imported before trying to reload it > loaded_module = sys.modules.get(name_to_reload) > if loaded_module is not module: > msg = "module {} not in sys.modules" > raise ImportError(msg.format(name_to_reload), name=name_to_reload) > parent_name = name_to_reload.rpartition('.')[0] > while parent_name: > if parent_name not in sys.modules: > msg = "parent {!r} not in sys.modules" > raise ImportError(msg.format(parent_name), name=parent_name) > parent_name = parent_name.rpartition('.')[0] > > # Check for a modified spec (e.g. if the import hooks have changed) > updated_spec = importlib.find_spec(name_to_reload) > > # The import-related module attributes get updated here > _init_module_attrs(loaded_module, updated_spec) > if current_name == "__main__": > loaded_module.__name__ = "__main__" > > # Now re-execute the module > loader = updated_spec.loader > if loader is None: > # Namespace package, already reinitialised above > return loaded_module > elif hasattr(loader, 'exec_module'): > loader.exec_module(module, current_spec) > else: > loader.load_module(name_to_reload) > > # Allow for the module to be replaced in sys.modules > # (even though that likely won't work very well) > return sys.modules[name_to_reload] This is pretty much the same thing I've implemented. :) -eric From ericsnowcurrently at gmail.com Tue Oct 29 05:56:37 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 28 Oct 2013 22:56:37 -0600 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Sun, Oct 27, 2013 at 4:41 PM, PJ Eby wrote: > I'm talking about userspace code that implements lazy importing > features, like the lazyModule() function in this module: > > http://svn.eby-sarna.com/Importing/peak/util/imports.py?view=markup > > Specifically, I'm trying to get an idea of how much that code will > need to change under the PEP (and apparently under importlib in > general). Depending on the outcome of issue19413, you shouldn't have to change anything. PEP 451 is aiming for strict backward compatibility. > Honestly, I'm finding all this stuff *really* confusing, which is kind > of worrying. I mean, I gather I'm one of the handful of people who > really understood how importing *used to work*, and I'm having a lot > of trouble wrapping my brain around the new world. > > (Granted, I think that may be because I understand how a lot of old > corner cases work, Like Nick, I would love more tests that cover these corner cases. It would have saved us (Brett especially) a lot of headache. At the least any description you can offer would be great. > but what's bugging me is that I no longer > understand how those old corners work under the new regime, nor do I > feel I understand what the *new* corners will be. This may also just > be communication problems, and the fact that it's been months since I > really walked through importlib line by line, and have never really > walked through it (or PEP 451) quite as thoroughly as I have import.c. > I also seem to be having trouble grokking why the motivating use > cases for PEP 451 can't be solved by just providing people with good > base classes to use for writing loaders -- i.e., I don't get why the > core protocol has to change to address the use case of writing loaders > more easily. The new protocol seems way more complex than PEP 302, What new protocol specifically? Finder.find_module() now returns a module spec instead of a loader. Loader.exec_module() gets used now rather than load_module(). Loaders don't have to worry about all the boilerplate stuff anymore (managing sys.modules and import-related module attributes). From my perspective PEP 451 is simplifying protocols. -eric > and ISTM the complexity could just be pushed off to the loader side of > the protocol without creating more interdependency between importlib > and the loaders.) From g.brandl at gmx.net Tue Oct 29 07:52:59 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 29 Oct 2013 07:52:59 +0100 Subject: [Python-Dev] cpython: Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle In-Reply-To: <3d7tjB3VP0zM9M@mail.python.org> References: <3d7tjB3VP0zM9M@mail.python.org> Message-ID: Am 29.10.2013 01:19, schrieb victor.stinner: > http://hg.python.org/cpython/rev/4ef4578db38a > changeset: 86715:4ef4578db38a > user: Victor Stinner > date: Tue Oct 29 01:19:37 2013 +0100 > summary: > Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle > exceptions when merging fast locals into f_locals of a frame. > PyEval_GetLocals() now raises an exception and return NULL on failure. You'll have to either make this private or document it. Georg From g.brandl at gmx.net Tue Oct 29 07:54:32 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 29 Oct 2013 07:54:32 +0100 Subject: [Python-Dev] cpython: Issue #18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on In-Reply-To: <3d7vJ308CDz7LjM@mail.python.org> References: <3d7vJ308CDz7LjM@mail.python.org> Message-ID: Am 29.10.2013 01:46, schrieb victor.stinner: > http://hg.python.org/cpython/rev/e1d51c42e5a1 > changeset: 86716:e1d51c42e5a1 > user: Victor Stinner > date: Tue Oct 29 01:28:23 2013 +0100 > summary: > Issue #18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on > memory allocation failure > > files: > Objects/unicodeobject.c | 1 + > 1 files changed, 1 insertions(+), 0 deletions(-) > > > diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c > --- a/Objects/unicodeobject.c > +++ b/Objects/unicodeobject.c > @@ -3766,6 +3766,7 @@ > return NULL; > _PyUnicode_UTF8(unicode) = PyObject_MALLOC(PyBytes_GET_SIZE(bytes) + 1); > if (_PyUnicode_UTF8(unicode) == NULL) { > + PyErr_NoMemory(); > Py_DECREF(bytes); > return NULL; > } Shouldn't this (and related commits from #18408) have been committed to the 3.3 branch? Georg From ncoghlan at gmail.com Tue Oct 29 10:32:31 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 29 Oct 2013 19:32:31 +1000 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On 29 Oct 2013 14:45, "Eric Snow" wrote: > > On Sat, Oct 26, 2013 at 11:03 PM, Nick Coghlan wrote: > > I don't think we can postpone it to later, as we need to be clear on: > > > > 1. Does reload use __name__ or __spec__.name when both are available? > > 2. Does __name__ get restored to its original value if reloading via > > __spec__.name? > > 3. Do other import related attributes get restored to their original values? > > 4. Does create_module get called if the loader has an exec_module method? > > 5. Does the loader have access to the previous spec when reloading a module? > > > > My answers to these questions (note that my answer to 2 differs from > > what I had in my initial sketch): > > > > 1. __spec__.name > > 2. Yes, __name__ is updated to match __spec__.name, *expect* if it is > > currently "__main__" > > 3. Yes, other import related attributes are restored to their baseline values > > 4. No, create_module is never called when reloading > > I agree on all of these. I'm adding a "How reloading will work" > section to the PEP. > > > 5. Currently no, but I'm thinking that may be worth changing (more on > > that below) > > > > The reload() implementation in my message is actually based on the > > current importlib.reload implementation. The only PEP 451 related > > changes were: > > > > - using __spec__.name (if available) instead of __name__ > > - checking all parent modules exist rather than just the immediate parent module > > - calling import.find_spec() rather than using the __loader__ attribute directly > > - being explicit that __name__ is left at the value it had prior to the reload > > - handling the namespace package, exec_module and no exec_module cases > > > > I also added an explicit check that the module was re-used properly, > > but I agree *that* change is out of scope for the PEP and should be > > considered as a separate question. > > > > Now, regarding the signature of exec_module(): I'm back to believing > > that loaders should receive a clear indication that a reload is taking > > place. Legacy loaders have to figure that out for themselves (by > > seeing that the module already exists in sys.modules), but we can do > > better for the new API by making the exec_module signature look like: > > > > def exec_module(self, module, previous_spec=None): > > # module is as per the current PEP 451 text > > # previous_spec would be set *only* in the reload() case > > # loaders that don't care still need to accept it, but can > > just ignore it > > ... > > I'd rather give loaders another optional method: > supports_reload(name). Complicating the spec methods signatures (to > support passing the old spec through) just for the sake of reload > seems less than optimal. I don't see any benefit to exposing the old > spec to loader.exec_module(). I do: it lets the loader tell if anything changed from the previous load operation, allowing it to make an informed decision on whether or not to permit the reload operation or throw an exception. However, I accept such a check would be better implemented as a separate yes/no method, so it makes sense to postpone this aspect to 3.5 at the earliest. Cheers, Nick. > > > > > So, with those revisions, the reload() semantics would be defined as: > > > > def reload(module): > > # Get the name to reload from the spec if it is available, > > # otherwise use __name__ directly > > current_spec = getattr(module, "__spec__") > > current_name = module.__name__ > > name_to_reload = current_name if current_spec is None else > > current_spec.name > > > > # Check this module is properly imported before trying to reload it > > loaded_module = sys.modules.get(name_to_reload) > > if loaded_module is not module: > > msg = "module {} not in sys.modules" > > raise ImportError(msg.format(name_to_reload), name=name_to_reload) > > parent_name = name_to_reload.rpartition('.')[0] > > while parent_name: > > if parent_name not in sys.modules: > > msg = "parent {!r} not in sys.modules" > > raise ImportError(msg.format(parent_name), name=parent_name) > > parent_name = parent_name.rpartition('.')[0] > > > > # Check for a modified spec (e.g. if the import hooks have changed) > > updated_spec = importlib.find_spec(name_to_reload) > > > > # The import-related module attributes get updated here > > _init_module_attrs(loaded_module, updated_spec) > > if current_name == "__main__": > > loaded_module.__name__ = "__main__" > > > > # Now re-execute the module > > loader = updated_spec.loader > > if loader is None: > > # Namespace package, already reinitialised above > > return loaded_module > > elif hasattr(loader, 'exec_module'): > > loader.exec_module(module, current_spec) > > else: > > loader.load_module(name_to_reload) > > > > # Allow for the module to be replaced in sys.modules > > # (even though that likely won't work very well) > > return sys.modules[name_to_reload] > > This is pretty much the same thing I've implemented. :) > > -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Oct 29 10:38:23 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 29 Oct 2013 10:38:23 +0100 Subject: [Python-Dev] cpython: Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle In-Reply-To: References: <3d7tjB3VP0zM9M@mail.python.org> Message-ID: 2013/10/29 Georg Brandl : > Am 29.10.2013 01:19, schrieb victor.stinner: >> http://hg.python.org/cpython/rev/4ef4578db38a >> changeset: 86715:4ef4578db38a >> user: Victor Stinner >> date: Tue Oct 29 01:19:37 2013 +0100 >> summary: >> Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle >> exceptions when merging fast locals into f_locals of a frame. >> PyEval_GetLocals() now raises an exception and return NULL on failure. > > You'll have to either make this private or document it. > > Georg I didn't find the documentation of PyFrame_FastToLocals(). Please open an issue if you consider that these functions must be documented. IMO these functions should be private because very low-level, but it's too late: PyFrame_FastToLocals() is part of the stable ABI, even if it's not documented. PyFrame_FastToLocalsWithError() is the same than PyFrame_FastToLocals(), except that it has a return value and keep the exception. Victor From solipsis at pitrou.net Tue Oct 29 10:47:16 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 29 Oct 2013 10:47:16 +0100 Subject: [Python-Dev] cpython: Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle References: <3d7tjB3VP0zM9M@mail.python.org> Message-ID: <20131029104716.35b862c3@pitrou.net> Le Tue, 29 Oct 2013 10:38:23 +0100, Victor Stinner a ?crit : > 2013/10/29 Georg Brandl : > > Am 29.10.2013 01:19, schrieb victor.stinner: > >> http://hg.python.org/cpython/rev/4ef4578db38a > >> changeset: 86715:4ef4578db38a > >> user: Victor Stinner > >> date: Tue Oct 29 01:19:37 2013 +0100 > >> summary: > >> Issue #18408: Add a new PyFrame_FastToLocalsWithError() function > >> to handle exceptions when merging fast locals into f_locals of a > >> frame. PyEval_GetLocals() now raises an exception and return NULL > >> on failure. > > > > You'll have to either make this private or document it. > > > > Georg > > I didn't find the documentation of PyFrame_FastToLocals(). Please open > an issue if you consider that these functions must be documented. Just because something else isn't documented doesn't mean new APIs shouldn't be documented. Please let's not enter such a vicious circle. > IMO these functions should be private because very low-level, but it's > too late: PyFrame_FastToLocals() is part of the stable ABI, even if > it's not documented. It's not too late for PyFrame_FastToLocalsWithError() to be private, though. I agree with Georg: you should either make it private or document it. Regards Antoine. From kristjan at ccpgames.com Tue Oct 29 11:00:28 2013 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Tue, 29 Oct 2013 10:00:28 +0000 Subject: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear? In-Reply-To: References: Message-ID: A > > > disable() function: > > Stop tracing Python memory allocations and clear traces of > memory blocks allocated by Python. > > I would disable to stop tracing, but I would not expect it to clear out the > traces it had already captured. If it has to do that, please put in some sample > code showing how to save the current traces before disabling. I was thinking something similar. It would be useful to be able to "pause" and "resume" if one is doing any analysis work in the live environment. This would reduce the need to have "Filter" objects. K From victor.stinner at gmail.com Tue Oct 29 10:54:08 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 29 Oct 2013 10:54:08 +0100 Subject: [Python-Dev] cpython: Issue #18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on In-Reply-To: References: <3d7vJ308CDz7LjM@mail.python.org> Message-ID: 2013/10/29 Georg Brandl : >> diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c >> --- a/Objects/unicodeobject.c >> +++ b/Objects/unicodeobject.c >> @@ -3766,6 +3766,7 @@ >> return NULL; >> _PyUnicode_UTF8(unicode) = PyObject_MALLOC(PyBytes_GET_SIZE(bytes) + 1); >> if (_PyUnicode_UTF8(unicode) == NULL) { >> + PyErr_NoMemory(); >> Py_DECREF(bytes); >> return NULL; >> } > > Shouldn't this (and related commits from #18408) have been committed to the 3.3 > branch? All changes of #18408 "should" be backported, but I don't plan to backport them. It is not trivial to backport them. Nobody complained before (memory allocation failure are usually bad handled anyway, it will crash later if it does not crash here). And I'm not 100% confident that these changes would not break anything. Examples of possible regression: - PyEval_GetLocals() now raises an exception in case of an error. This change "should" not break anything, because callers were already raising an exception when PyEval_GetLocals() returns NULL. The builtin locals() function was not raising an exception in case of error, but it's probably because it is very unlikely that the function is called without any frame (PyEval_GetFrame(); returns NULL). - many functions now fail with an assertion error when they are called with an exception set (assert(!PyErr_Occurred());) because they may replace the exception without noticing the caller - I tried to check reference counters, but I may have introduce a regression leak in the error handling code If there is really a regression, I prefer to limit it to the new version, not to a stable version. Note: I'm not saying that I'm 0% confident in my changes :-) Victor From victor.stinner at gmail.com Tue Oct 29 12:37:52 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 29 Oct 2013 12:37:52 +0100 Subject: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear? In-Reply-To: References: Message-ID: 2013/10/29 Jim Jewett : > reset() function: > > Clear traces of memory blocks allocated by Python. > > Does this do anything besides clear? If not, why not just re-use the > 'clear' name from dicts? (I like the reset() name. Charles-Fran?ois suggested this name inspired by OProfile API.) > disable() function: > > Stop tracing Python memory allocations and clear traces of > memory blocks allocated by Python. > > I would disable to stop tracing, but I would not expect it to clear > out the traces it had already captured. If it has to do that, please > put in some sample code showing how to save the current traces before > disabling. For consistency, you cannot keep traces when tracing is disabled. The free() must be enabled to remove allocated memory blocks, or next malloc() may get the same address which would raise an assertion error (you cannot have two memory blocks at the same address). Just call get_traces() to get traces before clearing them. I can explain it in the doc. 2013/10/29 Kristj?n Valur J?nsson : > I was thinking something similar. It would be useful to be able to "pause" and "resume" > if one is doing any analysis work in the live environment. This would reduce the > need to have "Filter" objects. For the reason explained above, it's not possible to disable the whole module temporarly. Internally, tracemalloc uses a thread-local variable (called the "reentrant" flag) to disable temporarly tracing allocations in the current thread. It only disables tracing new allocations, deallocations are still proceed. Victor From victor.stinner at gmail.com Tue Oct 29 13:07:18 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 29 Oct 2013 13:07:18 +0100 Subject: [Python-Dev] tracemalloc.get_object_traceback() and "unclosed" ResourceWarning Message-ID: Hi, Since Python 3.3, when a file or socket object is destroyed before being closed, a ResourceWarning is emitted. The warning is emitted where the object is destroyed: it's common to see "gc.collect()" location in the Python test suite for example. Tarek Ziad? asked on Twitter if there is a tool to track the origin of the object: "I wonder is there's a tool to trace back objects declarations that raise ResourceWarning in Py" https://twitter.com/tarek_ziade/status/394006754779877377 Nick Coghlan ?replied that tracemalloc may help: "@tarek_ziade tracemalloc (http://www.python.org/dev/peps/pep-0454/ ? - not accepted yet) will likely make it possible in 3.4, dunno about earlier versions." https://twitter.com/ncoghlan_dev/status/394010756242350080 I checked the code and it's not so simple. FileIO destructor emits the warning (fileio_dealloc_warn() in Modules/_io/fileio.c), but it's not possible to "hook" this function. I tried something with weakref, but I failed to set my hook. Does anyone see how to reuse tracemalloc (which is optional and disabled by default) in FileIO destructor? The file/object destructor may retrieve somehow the most recent frame of the object and call PyErr_WarnExplicitObject() instead of PyErr_WarnFormat() to pass the filename and lineno of the object. But how would Python decide to use tracemalloc or not? Add a new genric "get object traceback" callback which would be set by tracemalloc when the module is enabled? A frame is maybe not enough, it would be better to get the whole traceback! Do you know other features where it would be nice to know the origin of a Python object? Victor From ncoghlan at gmail.com Tue Oct 29 14:37:37 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 29 Oct 2013 23:37:37 +1000 Subject: [Python-Dev] cpython: Issue #18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on In-Reply-To: References: <3d7vJ308CDz7LjM@mail.python.org> Message-ID: On 29 October 2013 19:54, Victor Stinner wrote: > > If there is really a regression, I prefer to limit it to the new > version, not to a stable version. > > Note: I'm not saying that I'm 0% confident in my changes :-) Sounds like a solid rationale for limiting the changes to 3.4 to me :) /me should take another run at the known crasher tests one of these days. Maybe during the beta cycle... Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Oct 29 14:54:49 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 29 Oct 2013 23:54:49 +1000 Subject: [Python-Dev] tracemalloc.get_object_traceback() and "unclosed" ResourceWarning In-Reply-To: References: Message-ID: On 29 October 2013 22:07, Victor Stinner wrote: > Hi, > > Since Python 3.3, when a file or socket object is destroyed before > being closed, a ResourceWarning is emitted. The warning is emitted > where the object is destroyed: it's common to see "gc.collect()" > location in the Python test suite for example. > > Tarek Ziad? asked on Twitter if there is a tool to track the origin of > the object: > "I wonder is there's a tool to trace back objects declarations that > raise ResourceWarning in Py" > https://twitter.com/tarek_ziade/status/394006754779877377 > > Nick Coghlan replied that tracemalloc may help: > "@tarek_ziade tracemalloc (http://www.python.org/dev/peps/pep-0454/ ? > - not accepted yet) will likely make it possible in 3.4, dunno about > earlier versions." > https://twitter.com/ncoghlan_dev/status/394010756242350080 > > I checked the code and it's not so simple. FileIO destructor emits the > warning (fileio_dealloc_warn() in Modules/_io/fileio.c), but it's not > possible to "hook" this function. I tried something with weakref, but > I failed to set my hook. Ah, I wasn't clear about what I meant (yay, 140 characters!), but I was also wrong. I was thinking you could turn on warnings->errors and then catch the exception, and use that to get hold of the original object that triggered the resource warning, and then fed *that* into tracemalloc.get_object_traceback(). But, alas, even if we tweaked ResourceWarning to make sure that was possible, exceptions from __del__ methods are squashed by the interpreter, so you're only left with the text output. Didn't an earlier version of the PEP have the ability to find a traceback based on an object's ID rather than the object itself? If that capability existed (including for snapshots), then an object id printed in an error message could be matched up with the tracemalloc output. (ResourceWarning currently only prints the id if the object repr includes the id, but we could change that to ensure the object id is always shown). Anyway, I don't think this feature needs to be included in the PEP, but it's something we may decide we want to add later. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Tue Oct 29 15:05:32 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 29 Oct 2013 15:05:32 +0100 Subject: [Python-Dev] tracemalloc.get_object_traceback() and "unclosed" ResourceWarning In-Reply-To: References: Message-ID: 2013/10/29 Nick Coghlan : > I was thinking you could turn on warnings->errors and then catch the > exception, and use that to get hold of the original object that > triggered the resource warning, and then fed *that* into > tracemalloc.get_object_traceback(). But, alas, even if we tweaked > ResourceWarning to make sure that was possible, exceptions from > __del__ methods are squashed by the interpreter, so you're only left > with the text output. > > Didn't an earlier version of the PEP have the ability to find a > traceback based on an object's ID rather than the object itself? If > that capability existed (including for snapshots), then an object id > printed in an error message could be matched up with the tracemalloc > output. (ResourceWarning currently only prints the id if the object > repr includes the id, but we could change that to ensure the object id > is always shown). An address is only valid while the object is alive. When an object is destroyed, its trace is immediatly removed. So it's safer to get the traceback when the warning is emitted, not before, not after. > Anyway, I don't think this feature needs to be included in the PEP, > but it's something we may decide we want to add later. I don't want to add the ResourceWarning hack to the PEP :-) I'm just asking for help to find the best solution. Victor From g.brandl at gmx.net Tue Oct 29 13:45:25 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 29 Oct 2013 13:45:25 +0100 (CET) Subject: [Python-Dev] cpython: Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle In-Reply-To: References: <3d7tjB3VP0zM9M@mail.python.org> , Message-ID: > Gesendet: Dienstag, 29. Oktober 2013 um 10:38 Uhr > Von: "Victor Stinner" > An: "Georg Brandl" > Cc: "Python Dev" > Betreff: Re: [Python-Dev] cpython: Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle > > 2013/10/29 Georg Brandl : > > Am 29.10.2013 01:19, schrieb victor.stinner: > >> http://hg.python.org/cpython/rev/4ef4578db38a > >> changeset: 86715:4ef4578db38a > >> user: Victor Stinner > >> date: Tue Oct 29 01:19:37 2013 +0100 > >> summary: > >> Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle > >> exceptions when merging fast locals into f_locals of a frame. > >> PyEval_GetLocals() now raises an exception and return NULL on failure. > > > > You'll have to either make this private or document it. > > > > Georg > > I didn't find the documentation of PyFrame_FastToLocals(). Please open > an issue if you consider that these functions must be documented. I see you opened one already; thanks for that. In any case, our policy is that all public API function (those without _Py prefixes) *should* be documented. > IMO these functions should be private because very low-level, but it's > too late: PyFrame_FastToLocals() is part of the stable ABI, even if > it's not documented. Thankfully not; all of frameobject.h is not part of the stable ABI. cheers, Georg From g.brandl at gmx.net Tue Oct 29 13:47:47 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 29 Oct 2013 13:47:47 +0100 (CET) Subject: [Python-Dev] cpython: Issue #18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on In-Reply-To: References: <3d7vJ308CDz7LjM@mail.python.org> , Message-ID: > Gesendet: Dienstag, 29. Oktober 2013 um 10:54 Uhr > Von: "Victor Stinner" > An: "Georg Brandl" > Cc: "Python Dev" > Betreff: Re: [Python-Dev] cpython: Issue #18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on > > 2013/10/29 Georg Brandl : > >> diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c > >> --- a/Objects/unicodeobject.c > >> +++ b/Objects/unicodeobject.c > >> @@ -3766,6 +3766,7 @@ > >> return NULL; > >> _PyUnicode_UTF8(unicode) = PyObject_MALLOC(PyBytes_GET_SIZE(bytes) + 1); > >> if (_PyUnicode_UTF8(unicode) == NULL) { > >> + PyErr_NoMemory(); > >> Py_DECREF(bytes); > >> return NULL; > >> } > > > > Shouldn't this (and related commits from #18408) have been committed to the 3.3 > > branch? > > All changes of #18408 "should" be backported, but I don't plan to > backport them. It is not trivial to backport them. Nobody complained > before (memory allocation failure are usually bad handled anyway, it > will crash later if it does not crash here). And I'm not 100% > confident that these changes would not break anything. OK, that's a good enough reason, if it's "only" no-memory-related errors. > Examples of possible regression: > > - PyEval_GetLocals() now raises an exception in case of an error. This > change "should" not break anything, because callers were already > raising an exception when PyEval_GetLocals() returns NULL. The builtin > locals() function was not raising an exception in case of error, but > it's probably because it is very unlikely that the function is called > without any frame (PyEval_GetFrame(); returns NULL). > > - many functions now fail with an assertion error when they are called > with an exception set (assert(!PyErr_Occurred());) because they may > replace the exception without noticing the caller > > - I tried to check reference counters, but I may have introduce a > regression leak in the error handling code > > If there is really a regression, I prefer to limit it to the new > version, not to a stable version. > > Note: I'm not saying that I'm 0% confident in my changes :-) And I certainly think they have a better than 0% chance of being correct ;) Georg From barry at python.org Tue Oct 29 18:48:58 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 29 Oct 2013 13:48:58 -0400 Subject: [Python-Dev] RELEASED: Python 2.6.9 final Message-ID: <20131029134858.7fc75346@anarchist> Hello Pythoneers and Pythonistas, Five years ago this month, we released Python 2.6. It was an important version in Python's evolution, as it was developed in tandem with Python 3.0 and had the express intent of starting the transition toward Python 3. Amazingly, here we are five years later, and I am happy (and a little sad) to announce the Python 2.6.9 final release. Python 2.6.9 is a security-only source-only release, and with this, I officially retire the Python 2.6 branch. All maintenance of Python 2.6, including for security issues, has now ended. So too has my latest stint as Python Release Manager. Over the 19 years I have been involved with Python, I've been honored, challenged, stressed, and immeasurably rewarded by managing several Python releases. I wish I could thank each of you individually for all the support you've generously given me in this role. You deserve most of the credit for all these great releases; I'm just a monkey pushing buttons. :) Here then are some release notes about 2.6.9. For ongoing maintenance, please see Python 2.7. Future issues with Python 2.6 may still be tracked in the Python bug tracker, and you may wish to visit specific issues to see if unofficial patches are available for 2.6. http://bugs.python.org The source can be downloaded from: http://www.python.org/download/releases/2.6.9/ You can also see what's changed since Python 2.6.8: http://www.python.org/download/releases/2.6.9/NEWS.txt Users on OS X 10.9 (Mavericks) please note that issue 18458, which can crash the interactive interpreter, is *not* fixed in 2.6.9. If this issue affects you, please review the tracker for possible options: http://bugs.python.org/issue18458 Enjoy, -Barry (on behalf of the Python development community) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From guido at python.org Tue Oct 29 18:58:28 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 29 Oct 2013 10:58:28 -0700 Subject: [Python-Dev] RELEASED: Python 2.6.9 final In-Reply-To: <20131029134858.7fc75346@anarchist> References: <20131029134858.7fc75346@anarchist> Message-ID: Thanks Barry! That was a great announcement note, right from the heart. Python 2.6 may retire, but Python is far from retired, and I am looking forward to many more great interactions with you here on python-dev. Cheers! --Guido On Tue, Oct 29, 2013 at 10:48 AM, Barry Warsaw wrote: > Hello Pythoneers and Pythonistas, > > Five years ago this month, we released Python 2.6. It was an important > version in Python's evolution, as it was developed in tandem with Python > 3.0 > and had the express intent of starting the transition toward Python 3. > > Amazingly, here we are five years later, and I am happy (and a little sad) > to > announce the Python 2.6.9 final release. > > Python 2.6.9 is a security-only source-only release, and with this, I > officially retire the Python 2.6 branch. All maintenance of Python 2.6, > including for security issues, has now ended. > > So too has my latest stint as Python Release Manager. Over the 19 years I > have been involved with Python, I've been honored, challenged, stressed, > and > immeasurably rewarded by managing several Python releases. I wish I could > thank each of you individually for all the support you've generously given > me > in this role. You deserve most of the credit for all these great releases; > I'm just a monkey pushing buttons. :) > > Here then are some release notes about 2.6.9. > > For ongoing maintenance, please see Python 2.7. > > Future issues with Python 2.6 may still be tracked in the Python bug > tracker, > and you may wish to visit specific issues to see if unofficial patches are > available for 2.6. > > http://bugs.python.org > > The source can be downloaded from: > > http://www.python.org/download/releases/2.6.9/ > > You can also see what's changed since Python 2.6.8: > > http://www.python.org/download/releases/2.6.9/NEWS.txt > > Users on OS X 10.9 (Mavericks) please note that issue 18458, which can > crash > the interactive interpreter, is *not* fixed in 2.6.9. If this issue > affects > you, please review the tracker for possible options: > > http://bugs.python.org/issue18458 > > Enjoy, > -Barry > (on behalf of the Python development community) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Oct 29 22:29:57 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 29 Oct 2013 22:29:57 +0100 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: <52693D8E.4050306@mrabarnett.plus.com> Message-ID: 2013/10/26 Kristj?n Valur J?nsson : > In that case, how about adding a client/server feature? > > If you standardize the format, a minimal tracing client could write a log, > or send it to a socket, in a way that can be turned into a snapshot by a > corresponsing utility reading from a file or listenting to a socket. > > Just a though. Could be a future addition? tracemalloc maintains a dictionary of all allocated memory blocks, which is slow and eats a lot of memory. You don't need tracemalloc to log calls to malloc/realloc/free. You can write your own hook using the PEP 445 (malloc API). A code just writing to stderr should not be longer than 100 linues (tracemalloc is closer to 2500 lines). As I wrote, I began to log all calls into a file, but it was too slow. The memory leak occurred after between 2 hours and 1 day. To know the current state (ex: total memory currently allocated), you have to parse the whole log file, which can be really huge. Writing the log file, transfering the log file and analyzing the log file are too slow were too slow in my use cases. Victor From ericsnowcurrently at gmail.com Wed Oct 30 00:02:49 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 29 Oct 2013 17:02:49 -0600 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Tue, Oct 29, 2013 at 3:32 AM, Nick Coghlan wrote: > On 29 Oct 2013 14:45, "Eric Snow" wrote: >> I'd rather give loaders another optional method: >> supports_reload(name). Complicating the spec methods signatures (to >> support passing the old spec through) just for the sake of reload >> seems less than optimal. I don't see any benefit to exposing the old >> spec to loader.exec_module(). > > I do: it lets the loader tell if anything changed from the previous load > operation, allowing it to make an informed decision on whether or not to > permit the reload operation or throw an exception. > > However, I accept such a check would be better implemented as a separate > yes/no method, so it makes sense to postpone this aspect to 3.5 at the > earliest. With the PEP, loaders are no longer in charge of the module-related boilerplate. This means that by the time exec_module() gets called, they won't be able to determine for themselves if it's a reload by checking sys.modules. That's the point of adding loader.supports_reload(). (I haven't added that rationale to the PEP yet). If it makes a difference to loaders to also look at the previous spec during supports_reload(), then I think we should add that parameter now. Otherwise if we add the parameter later it makes it messier. The signature would now be: Loader.supports_reload(name, old_spec) -eric From ncoghlan at gmail.com Wed Oct 30 02:29:58 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 30 Oct 2013 11:29:58 +1000 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On 30 October 2013 09:02, Eric Snow wrote: > On Tue, Oct 29, 2013 at 3:32 AM, Nick Coghlan wrote: >> On 29 Oct 2013 14:45, "Eric Snow" wrote: >>> I'd rather give loaders another optional method: >>> supports_reload(name). Complicating the spec methods signatures (to >>> support passing the old spec through) just for the sake of reload >>> seems less than optimal. I don't see any benefit to exposing the old >>> spec to loader.exec_module(). >> >> I do: it lets the loader tell if anything changed from the previous load >> operation, allowing it to make an informed decision on whether or not to >> permit the reload operation or throw an exception. >> >> However, I accept such a check would be better implemented as a separate >> yes/no method, so it makes sense to postpone this aspect to 3.5 at the >> earliest. > > With the PEP, loaders are no longer in charge of the module-related > boilerplate. This means that by the time exec_module() gets called, > they won't be able to determine for themselves if it's a reload by > checking sys.modules. That's the point of adding > loader.supports_reload(). (I haven't added that rationale to the PEP > yet). Ah, good point. > If it makes a difference to loaders to also look at the previous spec > during supports_reload(), then I think we should add that parameter > now. Otherwise if we add the parameter later it makes it messier. > The signature would now be: > > Loader.supports_reload(name, old_spec) OK, time for me to stop trying to remember the details of the problem I'm trying to solve and go look them up in the source code :) One of my goals here is to be able to migrate extension loading from the old API to the new plugin API. That means being able to break up the existing load_module implementation: http://hg.python.org/cpython/file/1787277915e9/Python/importdl.c#l23 For loading, that's a fairly straightforward create_module/exec_module split, but reloading gets a bit more interesting. Specifically, I'd like to be able to implement the relevant parts of _PyImport_FindExtensionObject as a precheck for reloading: http://hg.python.org/cpython/file/1787277915e9/Python/import.c#l533 That means just having access to the module name isn't enough: the extensions dictionary is keyed by a (name, filename) 2-tuple rather than just by the module name. Using the find_spec API, that filename would be stored in the loader state on the spec object rather than being looked up anew during the load operation. However, rereading this method also indicates that I really want to know *in exec_module* whether this is a reload or not, since extension loading needs to handle reloads differently from initial execution. So I'm back to my original preference: I'd like the previous spec to be passed to exec_module in the reloading case. If reloading is not supported at all, it's up to the loader to throw an appropriate exception when the the previous spec is not None. If loading and reloading doesn't make any difference, then they can just ignore it. But when both are supported, but handled differently (as for extension modules), then that can be detected, and the information from the old spec (including the original loader and loader state) is available if needed. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jimjjewett at gmail.com Wed Oct 30 04:45:40 2013 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Tue, 29 Oct 2013 20:45:40 -0700 (PDT) Subject: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear? In-Reply-To: Message-ID: <527080e4.651e310a.0765.25fa@mx.google.com> (Tue Oct 29 12:37:52 CET 2013) Victor Stinner wrote: > For consistency, you cannot keep traces when tracing is disabled. > The free() must be enabled to remove allocated memory blocks, or > next malloc() may get the same address which would raise an assertion > error (you cannot have two memory blocks at the same address). That seems like an a quirk of the implementation, particularly since the actual address is not returned to the user. Nor do I see any way of knowing when that allocation is freed. Well, unless I missed it... I don't see how to get anything beyond the return value of get_traces, which is a (time-ordered?) list of allocation size with then-current call stack. It doesn't mention any attribute for indicating that some entries are de-allocations, let alone the actual address of each allocation. > For the reason explained above, it's not possible to disable the whole > module temporarly. > Internally, tracemalloc uses a thread-local variable (called the > "reentrant" flag) to disable temporarly tracing allocations in the > current thread. It only disables tracing new allocations, > deallocations are still proceed. Even assuming the restriction is needed, this just seems to mean that disabling (or filtering) should not affect de-allocation events, for fear of corrupting tracemalloc's internal structures. In that case, I would expect disabling (and filtering) to stop capturing new allocation events for me, but I would still expect tracemalloc to do proper internal maintenance. It would at least explain why you need both disable *and* reset; reset would empty those internal structures, so that tracemalloc could shortcut that maintenance. I would NOT assume that I needed to call reset when changing the filters, nor would I assume that changing them threw out existing traces. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From stephen at xemacs.org Wed Oct 30 05:16:08 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 30 Oct 2013 13:16:08 +0900 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: <87habztmuv.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > OK, time for me to stop trying to remember the details of the problem > I'm trying to solve and go look them up in the source code :) OMG! Even Nick does that. I feel better now! ;) From stephen at xemacs.org Wed Oct 30 05:09:48 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 30 Oct 2013 13:09:48 +0900 Subject: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear? In-Reply-To: References: Message-ID: <87iowftn5f.fsf@uwakimon.sk.tsukuba.ac.jp> Victor Stinner writes: > 2013/10/29 Jim Jewett : > > reset() function: > > > > Clear traces of memory blocks allocated by Python. > > > > Does this do anything besides clear? If not, why not just re-use the > > 'clear' name from dicts? > > (I like the reset() name. Charles-Fran?ois suggested this name > inspired by OProfile API.) Just "reset" implies to me that you're ready to start over. Not just traced memory blocks but accumulated statistics and any configuration (such as Filters) would also be reset. Also tracing would be disabled until started explicitly. If you want it to apply just to the traces, reset_traces() would be more appropriate. > > disable() function: > > > > Stop tracing Python memory allocations and clear traces of > > memory blocks allocated by Python. > > > > I would disable to stop tracing, but I would not expect it to clear > > out the traces it had already captured. If it has to do that, please > > put in some sample code showing how to save the current traces before > > disabling. > > For consistency, you cannot keep traces when tracing is disabled. The > free() must be enabled to remove allocated memory blocks, or next > malloc() may get the same address which would raise an assertion error > (you cannot have two memory blocks at the same address). Then I would not call this "disable". disable() should not "destroy" data. > Just call get_traces() to get traces before clearing them. I can > explain it in the doc. Shouldn't disable() do this automatically, perhaps with an optional discard_traces flag (which would be False by default)? But I definitely agree with Jim: You *must* provide an example here showing how to save the traces (even though it's trivial to do so), because that will make clear that disable() is a destructive operation. (It is not destructive in any other debugging tool that I've used.) Even with documentation, be prepared for user complaints. From raymond.hettinger at gmail.com Wed Oct 30 08:12:03 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 30 Oct 2013 01:12:03 -0600 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> Message-ID: On Oct 28, 2013, at 1:16 PM, Victor Stinner wrote: > so what is the > status of the PEP 455 (TransformDict)? I'm giving a thorough evaluation of the proposal and am devoting chunks of time each weekend to reviewing the email threads, the links provided in the PEPs, looking at how well the TD fits in existing code. I'm giving this work priority over my own list of things to add to 3.4 (most of which will now have to wait until 3.5). This week, I'm teaching a five-day intermediate python class to highly experienced network engineers in Denver. We'll do some exercises using the TD and evaluate the results against alternative approaches. Here are some preliminary notes (in no particular order): * A first reading of the python-dev threads suggests that the two-dict TD implementation seems better suited to implementing a case-folding-case-preserving dictionary and is far less well suited for a case-folding dict or an identity dict. * There are interesting differences between the proposed TD and the CaseInsensitiveDict implemented in Kenneth Reitz's HTTP requests library. The latter keeps the last key added rather than the first. It also has a cleaner implementation and the API is a bit nicer (no getitem() method). * The "originals" dict maps a transformed key back to its first saved original value. An alternative would be to map back to a set of original values or a list of original values. * A possible use case is for a unicode normalizing dictionary where 'L' + chr(111) + chr(776) + 'wis' would match 'L' + chr(246) + 'wis'. * The implementation looks rough at this point, but that is easily fixed-up. I'll leave some specific suggestions on the tracker (getting it to accept a list of tuples in the constructor, a recursive repr, renaming the getitem() method, deprivatizing the attributes, getting it to work with __missing__, etc). * Having two-mappings-in-one seems to be the most awkward part of the design and wouldn't be needed for the two strongest use cases, a case-insensitive-but-not-case-preserving dict and an identity dict. * In http://stackoverflow.com/questions/13230414, the OP wants a CI dict but doesn't need the case preserving aspect. The OP is provided with a regular dictionary containing mixed case keys and needs to compare a list of potential matches of unknown case. Copying the dict to a case-folding TD wouldn't provide any incremental advantage over building a regular dict with lower case keys. * In http://www.gossamer-threads.com/lists/python/python/209527, the OP wanted an ID comparison for a symbolic calculation. The objects already had an eq function and he wanted to temporarily bypass that in a symbol lookup. Essentially he needed a dictionary that would allow the use of an alternative equality function. A TD would work here but there would be no need for the key preserving feature. There doesn't seem to be any advantage over using a regular dict that directly stores the id() as the key. * In https://mail.python.org/pipermail/python-ideas/2010-May/007235.html, the OP wants a highly optimized identity dictionary that doesn't call an expensive id() function. The proposed TD doesn't fulfill this request -- it would be far from being optimized and would call id() on every store and every lookup. * In http://msdn.microsoft.com/en-us/library/xfhwa508.aspx, the API describes a dict with an alternative equality comparison. This is a different design than the TD and is for a world that is somewhat different from Python. In that dict, eq and hash are specified at the dict level rather than object level (in Python, the key objects define their own __eq__ and __hash__ rather than having the user attach the functions directly to the dictionary). * In http://docs.oracle.com/javase/6/docs/api/java/util/IdentityHashMap.html, the IdentityHashMap is described as being for rare use cases such as topology-preserving object graph transformations likes serialization or deep-copying. Looking at Python's own code for copy.deepcopy(), the TD would not be a good replacement for the existing code (more memory intensive, slower, no use for the originals dict, and no use for most of the TD functionality). It appears that it is simpler and faster to store and lookup d[id(obj)] than to use a TD. * If this were being modeled in a database, we would have one table with a many-to-one mapping of original keys to transformed keys and another table with a transformed key as the primary key in a table of key-value pairs. This suggests two lookup patterns original->tranformed->value and transformed->all_originals. * The Apache case insensitive dict documentation includes these thoughts: "This map will violate the detail of various Map and map view contracts. As a general rule, don't compare this map to other maps. In particular, you can't use decorators like ListOrderedMap on it, which silently assume that these contracts are fulfilled. --- Note that CaseInsensitiveMap is not synchronized and is not thread-safetiveMap.html --- This map will violate the detail of various Map and map view contracts. As a general rule, don't compare this map to other maps. In particular, you can't use decorators like ListOrderedMap on it, which silently assume that these contracts are fulfilled." * Using ACK to search through Django, it looks like there over a half-dozen case-insensitive lookups with are implemented using d[k.lower()] and wouldn't be better-off using the proposed TD. The notes above are rough and I still have more work to do: * close reading of the remaining links in the PEP * another round of user testing this week (this time with far more experienced engineers) * review the long python-dev thread in more detail * put detailed code suggestions on the tracker All of this will take some time. I understand that the 3.4 feature deadline is looming, but I would like to take the time to thoroughly think this through and make a good decision. If I had to choose right now, a safe choice would be to focus on the primary use case and implement a clean CaseInsensitiveDict without the double-dict first-saved case-preserving feature. That said, I find the TD to be fascinating and there's more work to do before making a decision. Hopefully, this post will make the thought process more transparent. Cheers, Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Oct 30 11:02:59 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 30 Oct 2013 11:02:59 +0100 Subject: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear? In-Reply-To: <527080e4.651e310a.0765.25fa@mx.google.com> References: <527080e4.651e310a.0765.25fa@mx.google.com> Message-ID: Hi, 2013/10/30 Jim J. Jewett : > Well, unless I missed it... I don't see how to get anything beyond > the return value of get_traces, which is a (time-ordered?) list > of allocation size with then-current call stack. It doesn't mention > any attribute for indicating that some entries are de-allocations, > let alone the actual address of each allocation. get_traces() does return the traces of the currently allocated memory blocks. It's not a log of alloc/dealloc calls. The list is not sorted. If you want a sorted list, use take_snapshot.statistics('lineno') for example. > In that case, I would expect disabling (and filtering) to stop > capturing new allocation events for me, but I would still expect > tracemalloc to do proper internal maintenance. tracemalloc has an important overhead in term of performances and memory. The purpose of disable() is to... disable the module, to remove complelty the overhead. In practice, enable() installs on memory allocators, disable() uninstalls these hooks. I don't understand why you are so concerned by disable(). Why would you like to keep traces and disable the module? I never called disable() in my own tests, the module is automatically disabled at exit. Victor From victor.stinner at gmail.com Wed Oct 30 11:09:30 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 30 Oct 2013 11:09:30 +0100 Subject: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear? In-Reply-To: <87iowftn5f.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87iowftn5f.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: 2013/10/30 Stephen J. Turnbull : > Just "reset" implies to me that you're ready to start over. Not just > traced memory blocks but accumulated statistics and any configuration > (such as Filters) would also be reset. Also tracing would be disabled > until started explicitly. If the name is really the problem, I propose the restore the previous name: clear_traces(). It's symmetric with get_traces(), like add_filter()/get_filters()/clear_filters(). > Shouldn't disable() do this automatically, perhaps with an optional > discard_traces flag (which would be False by default)? The pattern is something like that: enable() snapshot1 = take_snapshot() ... snapshot2 = take_snapshot() disable() I don't see why disable() would return data. > But I definitely agree with Jim: You *must* provide an example here > showing how to save the traces (even though it's trivial to do so), > because that will make clear that disable() is a destructive > operation. (It is not destructive in any other debugging tool that > I've used.) Even with documentation, be prepared for user complaints. I added "Call get_traces() or take_snapshot() function to get traces before clearing them." to the doc: http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html#tracemalloc.disable Victor From kristjan at ccpgames.com Wed Oct 30 13:37:48 2013 From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=) Date: Wed, 30 Oct 2013 12:37:48 +0000 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: <52693D8E.4050306@mrabarnett.plus.com> Message-ID: > -----Original Message----- > From: Victor Stinner [mailto:victor.stinner at gmail.com] > Sent: 29. okt?ber 2013 21:30 > To: Kristj?n Valur J?nsson > Cc: python-dev > Subject: Re: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! > tracemalloc maintains a dictionary of all allocated memory blocks, which is > slow and eats a lot of memory. You don't need tracemalloc to log calls to > malloc/realloc/free. You can write your own hook using the PEP 445 (malloc > API). A code just writing to stderr should not be longer than 100 linues > (tracemalloc is closer to 2500 lines). > The point of a PEP is getting something into standard python. The command line flag is also part of this. Piggybacking a lightweight client/server data-gathering version of this on top of the PEP could be beneficial in that respect. Unless I am mistaken, the Pep 445 hooks must be setup before calling Py_Initialize() and so using them is not trivial. Anyway, just a suggestion, for the record. K From victor.stinner at gmail.com Wed Oct 30 14:17:14 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 30 Oct 2013 14:17:14 +0100 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: <52693D8E.4050306@mrabarnett.plus.com> Message-ID: 2013/10/30 Kristj?n Valur J?nsson : > The point of a PEP is getting something into standard python. The command line flag is also part of this. > Piggybacking a lightweight client/server data-gathering version of this on top of the PEP > could be beneficial in that respect. In my opinion, your use case (log malloc/free and sent it to the network) is completly different to what tracemalloc does. Reusing tracemalloc for you would be inefficient (slow, use too much memory). You can use tracemalloc if you want to send a snapshot of traces every N minutes. It should not be hard (less than 100 lines of Python) to implement that using a thread, pickle and a socket. But I prefer to not include it to the PEP, Charles Fran?ois wants a minimal module and prefers to develop tools on top of the module. I *now* agree with him (first I wanted to pack everything into the stdlib! even escape sequences to write text with colors!). For example, the old code using "tasks" to take automatically a snapshot every N minutes or display the top 10 allocations every N minutes into the terminal with colors has been moved to a new project: https://github.com/haypo/pytracemalloctext/blob/master/doc/index.rst (the project is not usable yet, I will finish it after the PEP 454, and after updating the pytracemalloc module on PyPI) > Unless I am mistaken, the Pep 445 hooks must be setup before calling Py_Initialize() and so using > them is not trivial. It depends on how you use the API. If you want to replace the memory allocators (use your own "malloc"), you have to call PyMem_SetAllocator() *before the first memory allocation* ! In Python, the first memory allocation occurs much earlier than Py_Initialize(), PyMem_RawMalloc() is the *first* instruction (!) executed by Python in its main() function... (see Modules/python.c). If you want to install an hook calling the previous allocator, you can call PyMem_SetAllocator() anytime: that's why it's possible to call tracemalloc.enable() anytime. Victor From rdmurray at bitdance.com Wed Oct 30 14:30:10 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 30 Oct 2013 09:30:10 -0400 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> Message-ID: <20131030133010.9A002250164@webabinitio.net> On Wed, 30 Oct 2013 01:12:03 -0600, Raymond Hettinger wrote: > If I had to choose right now, a safe choice would be to focus on > the primary use case and implement a clean CaseInsensitiveDict > without the double-dict first-saved case-preserving feature. > That said, I find the TD to be fascinating and there's more work > to do before making a decision. Please be aware that the PEP author's motivation in submitting the PEP was to have a case insensitive, case *preserving* dict. The generalization serves to make the new datatype more useful, but if the end result doesn't satisfy the original use case of the author, I won't be surprised if he has no motivation to work on it further :). --David From rdmurray at bitdance.com Wed Oct 30 14:32:08 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 30 Oct 2013 09:32:08 -0400 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> Message-ID: <20131030133208.ADB89250164@webabinitio.net> On Wed, 30 Oct 2013 01:12:03 -0600, Raymond Hettinger wrote: > I'm giving a thorough evaluation of the proposal > and am devoting chunks of time each weekend > to reviewing the email threads, the links provided > in the PEPs, looking at how well the TD fits in existing code. > > I'm giving this work priority over my own list of things > to add to 3.4 (most of which will now have to wait until 3.5). And thanks for doing all this work, Raymond. I forgot to say that in my previous post. --David From ethan at stoneleaf.us Wed Oct 30 15:18:21 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 30 Oct 2013 07:18:21 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> Message-ID: <5271152D.9010908@stoneleaf.us> On 10/30/2013 12:12 AM, Raymond Hettinger wrote: > > Hopefully, this post will make the thought process more transparent. Thanks, Raymond. Your time is appreciated. -- ~Ethan~ From nigel at nigelsmall.com Wed Oct 30 17:34:33 2013 From: nigel at nigelsmall.com (Nigel Small) Date: Wed, 30 Oct 2013 16:34:33 +0000 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <5271152D.9010908@stoneleaf.us> References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <5271152D.9010908@stoneleaf.us> Message-ID: It strikes me that there could be an alternative approach to some of the use cases discussed here. Instead of a new type of dictionary, the case-insensitivity problem could be solved with something akin to a * CaseInsensitiveString* class used for keys within a standard dictionary. This would be very similar to a normal string except with comparison and hashing. It would mean that CaseInsensitiveString("Foo") is considered equal to CaseInsensitiveString("foo") and allow code such as the following: >>> headers = {} >>> headers[CaseInsensitiveString("content-type")] = "text/plain" >>> headers[CaseInsensitiveString("Content-Type")] "text/plain" This would obviously also be usable in other places where case-insensitive strings are required. Just my two pence/cents/other minor currency units. Nigel On 30 October 2013 14:18, Ethan Furman wrote: > On 10/30/2013 12:12 AM, Raymond Hettinger wrote: > >> >> Hopefully, this post will make the thought process more transparent. >> > > Thanks, Raymond. Your time is appreciated. > > -- > ~Ethan~ > > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/**mailman/options/python-dev/** > nigel%40nigelsmall.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Oct 30 18:16:45 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 30 Oct 2013 18:16:45 +0100 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: Message-ID: > Snapshot > -------- > > ``Snapshot(timestamp: datetime.datetime, traceback_limit: int, stats: > dict=None, traces: dict=None)`` class: > > Snapshot of statistics and traces of memory blocks allocated by > Python. > > ``apply_filters(filters)`` method: > > Apply filters on the ``traces`` and ``stats`` dictionaries, > *filters* is a list of ``Filter`` instances. Snapshot.apply_filters() currently works in-place. This is not convinient. It should create a new Snapshot instance. For example, I have a huge snapshot with +800K traces. I would like to ignore and filenames: I apply a first filter to exclude <*>. Then I only want to see allocations related to the regular expressions: I apply a second pair of filters to only include */sre*.py and */re.py. Ok, now I want to see other files. Uh oh, I loose all others traces, I have to reload the huge snapshot. And again, exclude <*>. I would prefer something like: full_snapshot = Snapshot.load("huge.pickle") clean = full_snapshot.apply_filters([Filter(False, "<*>")]) # delete maybe full_snapshot here regex = clean.apply_filters([Filter(True, "*/re.py"), Filter(True, "*/sre*.py")]) other = clean.apply_filters([Filter(False, "*/re.py"), Filter(False, "*/sre*.py")]) ... > ``Filter(include: bool, filename_pattern: str, lineno: int=None, > traceback: bool=False)`` class: > ... > ``traceback`` attribute: > > If *traceback* is ``True``, all frames of the traceback are checked. > If *traceback* is ``False``, only the most recent frame is checked. > > This attribute is ignored if the traceback limit is less than ``2``. > See the ``get_traceback_limit()`` function. Hum, I don't really like the traceback name. traceback=False is confusing because the traceback is used by the filter even if traceback=False. Other names: all_frames, any_frame, most_recent_frame_only, ...? Example: f1 = Filter("*/linecache.py", all_frames=True) f2 = Filter("*/linecache.py") # all_frames is False by default Victor From ethan at stoneleaf.us Wed Oct 30 18:04:29 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 30 Oct 2013 10:04:29 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <5271152D.9010908@stoneleaf.us> Message-ID: <52713C1D.7060304@stoneleaf.us> On 10/30/2013 09:34 AM, Nigel Small wrote: > > It strikes me that there could be an alternative approach to some of the use cases discussed here. Instead of a new type > of dictionary, the case-insensitivity problem could be solved with something akin to a *CaseInsensitiveString* class [...] The nice thing about the TransformDict is it is usable for much more than simple case-insensitivity. -- ~Ethan~ From ericsnowcurrently at gmail.com Wed Oct 30 18:40:17 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 30 Oct 2013 11:40:17 -0600 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Tue, Oct 29, 2013 at 7:29 PM, Nick Coghlan wrote: > OK, time for me to stop trying to remember the details of the problem > I'm trying to solve and go look them up in the source code :) > > One of my goals here is to be able to migrate extension loading from > the old API to the new plugin API. That means being able to break up > the existing load_module implementation: > > http://hg.python.org/cpython/file/1787277915e9/Python/importdl.c#l23 > > For loading, that's a fairly straightforward create_module/exec_module > split, but reloading gets a bit more interesting. > > Specifically, I'd like to be able to implement the relevant parts of > _PyImport_FindExtensionObject as a precheck for reloading: > > http://hg.python.org/cpython/file/1787277915e9/Python/import.c#l533 > > That means just having access to the module name isn't enough: the > extensions dictionary is keyed by a (name, filename) 2-tuple rather > than just by the module name. Using the find_spec API, that filename > would be stored in the loader state on the spec object rather than > being looked up anew during the load operation. > > However, rereading this method also indicates that I really want to > know *in exec_module* whether this is a reload or not, since extension > loading needs to handle reloads differently from initial execution. > > So I'm back to my original preference: I'd like the previous spec to > be passed to exec_module in the reloading case. If reloading is not > supported at all, it's up to the loader to throw an appropriate > exception when the the previous spec is not None. If loading and > reloading doesn't make any difference, then they can just ignore it. > But when both are supported, but handled differently (as for extension > modules), then that can be detected, and the information from the old > spec (including the original loader and loader state) is available if > needed. Our recent discovery about reloading should probably be reflected in the signature of finder.find_spec(): MetaPathFinder.find_spec(name, path=None, existing=None) PathEntryFinder.find_spec(name, existing=None) This way the finder has an opportunity to incorporate information from an existing spec into the spec it returns. reload() would make use of this by passing module.__spec__ (or None if the module has no __spec__) to _bootstrap._find_spec(). This approach should also address what you are looking for. I'd prefer it over passing the existing spec to exec_module(). The module (and its __spec__) should have everything exec_module() needs to do its job. We would still need to use loader.supports_reload() in reload(). However, it may make more sense to pass in the module-to-be-reloaded (after updating its __spec__ to the one found by _bootstrap._find_spec()). -eric From nigel at nigelsmall.com Wed Oct 30 19:32:53 2013 From: nigel at nigelsmall.com (Nigel Small) Date: Wed, 30 Oct 2013 18:32:53 +0000 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <52713C1D.7060304@stoneleaf.us> References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <5271152D.9010908@stoneleaf.us> <52713C1D.7060304@stoneleaf.us> Message-ID: True, but I could similarly argue that the nice thing about CaseInsensitiveString is it is usable for much more than dictionary keys - it just depends on your point of view. There would be nothing stopping other types of dictionary key transformation being covered by other key data types in a similar way, I'm simply trying to raise the question of where the genericity could sit: in the dictionary or in the key. Nigel On 30 October 2013 17:04, Ethan Furman wrote: > On 10/30/2013 09:34 AM, Nigel Small wrote: > >> >> It strikes me that there could be an alternative approach to some of the >> use cases discussed here. Instead of a new type >> of dictionary, the case-insensitivity problem could be solved with >> something akin to a *CaseInsensitiveString* class [...] >> > > The nice thing about the TransformDict is it is usable for much more than > simple case-insensitivity. > > > -- > ~Ethan~ > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/**mailman/options/python-dev/** > nigel%40nigelsmall.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Oct 30 20:09:59 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 30 Oct 2013 20:09:59 +0100 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <5271152D.9010908@stoneleaf.us> Message-ID: <20131030200959.44389929@fsol> On Wed, 30 Oct 2013 16:34:33 +0000 Nigel Small wrote: > It strikes me that there could be an alternative approach to some of the > use cases discussed here. Instead of a new type of dictionary, the > case-insensitivity problem could be solved with something akin to a * > CaseInsensitiveString* class used for keys within a standard dictionary. > This would be very similar to a normal string except with comparison and > hashing. It would mean that CaseInsensitiveString("Foo") is considered > equal to CaseInsensitiveString("foo") and allow code such as the following: And how does a case-insensitive string compare with a normal (case-sensitive) string? This is a can of worms. (if you answer, please don't answer in this thread but open a separate one for case-insensitive strings, thanks) Regards Antoine. From skip at pobox.com Wed Oct 30 20:21:16 2013 From: skip at pobox.com (Skip Montanaro) Date: Wed, 30 Oct 2013 14:21:16 -0500 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20131030200959.44389929@fsol> References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> <5271152D.9010908@stoneleaf.us> <20131030200959.44389929@fsol> Message-ID: > And how does a case-insensitive string compare with a normal > (case-sensitive) string? This is a can of worms. I was wondering this myself. I suspect it would depend which string is on the left hand side of the comparison operator, yes? Can of worms, indeed. implicit-insensitve-i-n-ly, y'rs, Skip From solipsis at pitrou.net Wed Oct 30 20:30:42 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 30 Oct 2013 20:30:42 +0100 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <523F968A.6090907@stoneleaf.us> Message-ID: <20131030203042.3db522fe@fsol> Hi Raymond, On Wed, 30 Oct 2013 01:12:03 -0600 Raymond Hettinger wrote: > > On Oct 28, 2013, at 1:16 PM, Victor Stinner wrote: > > > so what is the > > status of the PEP 455 (TransformDict)? > > > I'm giving a thorough evaluation of the proposal > and am devoting chunks of time each weekend > to reviewing the email threads, the links provided > in the PEPs, looking at how well the TD fits in existing code. Thanks for the thorough status report. > * There are interesting differences between the proposed TD > and the CaseInsensitiveDict implemented in Kenneth Reitz's > HTTP requests library. The latter keeps the last key added > rather than the first. It also has a cleaner implementation > and the API is a bit nicer (no getitem() method). First-vs-last has already been discussed in the previous thread. My initial hunch was to keep the last key, but other people made the point that first was both more compliant (with current dict behaviour) and more useful (since you can override it by deleting and then reinserting the entry). Regards Antoine. From jimjjewett at gmail.com Wed Oct 30 20:58:20 2013 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 30 Oct 2013 15:58:20 -0400 Subject: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear? In-Reply-To: References: <527080e4.651e310a.0765.25fa@mx.google.com> Message-ID: On Wed, Oct 30, 2013 at 6:02 AM, Victor Stinner wrote: > 2013/10/30 Jim J. Jewett : >> Well, unless I missed it... I don't see how to get anything beyond >> the return value of get_traces, which is a (time-ordered?) list >> of allocation size with then-current call stack. It doesn't mention >> any attribute for indicating that some entries are de-allocations, >> let alone the actual address of each allocation. > get_traces() does return the traces of the currently allocated memory > blocks. It's not a log of alloc/dealloc calls. The list is not sorted. > If you want a sorted list, use take_snapshot.statistics('lineno') for > example. Any list is sorted somehow; I had assumed that it was defaulting to order-of-creation, though if you use a dict internally, that might not be the case. If you return it as a list instead of a dict, but that list is NOT in time-order, that is worth documenting Also, am I misreading the documentation of get_traces() function? Get traces of memory blocks allocated by Python. Return a list of (size: int, traceback: tuple) tuples. traceback is a tuple of (filename: str, lineno: int) tuples. So it now sounds like you don't bother to emit de-allocation events because you just remove the allocation from your internal data structure. In other words, you provide a snapshot, but not a history -- except that the snapshot isn't complete either, because it only shows things that appeared after a certain event (the most recent enablement). I still don't see anything here(*) that requires even saving the address, let alone preventing re-use. (*) get_object_traceback(obj) might require a stored address for efficiency, but the base functionality of getting traces doesn't. I still wouldn't worry about address re-use though, because the address should not be re-used until the object has been deleted -- and is no longer available to be passed to get_object_traceback. So the worst that can happen is that an object which was not traced might return a bogus answer instead of failing. >> In that case, I would expect disabling (and filtering) to stop >> capturing new allocation events for me, but I would still expect >> tracemalloc to do proper internal maintenance. > tracemalloc has an important overhead in term of performances and > memory. The purpose of disable() is to... disable the module, to > remove completely the overhead. > ... Why would you like to keep traces and disable the module? Because of that very overhead. I think my use typical use case would be similar to Kristj?n Valur's, but I'll try to spell it out in more detail here. (1) Whoa -- memory hog! How can I fix this? (2) I know -- track all allocations, with a traceback showing why they were made. (At a minimum, I would like to be able to subclass your tool to do this -- preferably without also keeping the full history in memory.) (3) Oh, maybe I should skip the ones that really are temporary and get cleaned up. (You make this easy by handling the de-allocs, though I'm not sure those events get exposed to anyone working at the python level, as opposed to modifying and re-compiling.) (4) hmm... still too big ... I should use filters. (But will changing those filters while tracing is enabled mess up your current implementation?) (5) Argh. What I really want is to know what gets allocated at times like XXX. I can do that if times-like-XXX only ever occur once per process. I *might* be able to do it with filters. But I would rather do it by saying "trace on" and "trace off". Maybe even with a context manager around the suspicious places. (6) Then, at the end of the run, I would say "give me the info about how much was allocated when tracing was on." Some of that might be going away again when tracing is off, but at least I know what is making the allocations in the first place. And I know that they're sticking around "long enough". Under your current proposal, step (5) turns into set filters trace on ... get_traces serialize to some other storage trace off and step (6) turns into read in from that other storage I just made up on the fly, and do my own summarizing, because my format is almost by definition non-standard. This complication isn't intolerable, but neither is it what I expect from python. And it certainly isn't what I expect from a binary toggle like enable/disable. (So yes, changing the name to clear_traces would help, because I would still be disappointed, but at least I wouldn't be surprised.) Also, if you do stick with the current limitations, then why even have get_traces, as opposed to just take_snapshot? Is there some difference between them, except that a snapshot has some convenience methods and some simple metadata? Later, he wrote: > I don't see why disable() would return data. disable is indeed a bad name for something that returns data. The only reason to return data from "disable" is that (currently) you're throwing the data away, so either you want the data now or you should have turned it off earlier. -jJ From victor.stinner at gmail.com Wed Oct 30 21:40:39 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 30 Oct 2013 21:40:39 +0100 Subject: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear? In-Reply-To: References: <527080e4.651e310a.0765.25fa@mx.google.com> Message-ID: Le 30 oct. 2013 20:58, "Jim Jewett" a ?crit : > hough if you use a dict internally, that might not > be the case. Tracemalloc uses a {address: trace} duct internally. > If you return it as a list instead of a dict, but that list is > NOT in time-order, that is worth documenting Ok i will document it. > Also, am I misreading the documentation of get_traces() function? > > Get traces of memory blocks allocated by Python. > Return a list of (size: int, traceback: tuple) tuples. > traceback is a tuple of (filename: str, lineno: int) tuples. > > > So it now sounds like you don't bother to emit de-allocation > events because you just remove the allocation from your > internal data structure. I don't understand your question. Tracemalloc does not store events but traces. When a memory block is deallocated, it us removed from the internal dict (and so from get_traces() list). > I still don't see anything here(*) that requires even saving > the address, let alone preventing re-use. The address must be stored internally to maintain the internal dict. See the C code. > (1) Whoa -- memory hog! How can I fix this? > > (2) I know -- track allocallocations, with a traceback showing why they > were made. (At a minimum, I would like to be able to subclass your > tool to do this -- preferably without also keeping the full history in > memory.) What do you mean by "full history" and "subclass your tool"? > (3) Oh, maybe I should skip the ones that really are temporary and > get cleaned up. (You make this easy by handling the de-allocs, > though I'm not sure those events get exposed to anyone working at > the python level, as opposed to modifying and re-compiling.) If your temporary objects are destroyed before you call get_traces(), you will not see them in get_traces(). I don't understand. > (4) hmm... still too big ... I should use filters. (But will changing those > filters while tracing is enabled mess up your current implementation?) If you call add_filter(), new traces() will be filtered. Not the old ones, as explained in the doc. What do you mean by "mess up"? > (5) Argh. What I really want is to know what gets allocated at times > like XXX. > I can do that if times-like-XXX only ever occur once per process. I *might* be > able to do it with filters. But I would rather do it by saying "trace on" and > "trace off". Maybe even with a context manager around the suspicious > places. I don't understand "times like XXX", what is it? To see what happened between two lines of code, you can compare two snapshots. No need to disable tracing. > (6) Then, at the end of the run, I would say "give me the info about how much > was allocated when tracing was on." Some of that might be going away > again when tracing is off, but at least I know what is making the allocations > in the first place. And I know that they're sticking around "long enough". I think you musunderstood how tracemalloc works. You should compile it and play with it. In my opinion, you already have everything in tracemalloc for you scenario. > Under your current proposal, step (5) turns into > > set filters > trace on > ... > get_traces > serialize to some other storage > trace off s1=take_snapshot() ... s2=take_snapshot() ... diff=s2.statistics("lines", compare_to=s1) > why even have > get_traces, > as opposed to just take_snapshot? Is there some difference between them, > except that a snapshot has some convenience methods and some simple > metadata? See the doc: Snapshot.traces is the result of get_traces(). get_traces() is here is you want to write your own tool without Snapshot. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Oct 30 23:09:44 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 31 Oct 2013 08:09:44 +1000 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On 31 Oct 2013 03:41, "Eric Snow" wrote: > > On Tue, Oct 29, 2013 at 7:29 PM, Nick Coghlan wrote: > > OK, time for me to stop trying to remember the details of the problem > > I'm trying to solve and go look them up in the source code :) > > > > One of my goals here is to be able to migrate extension loading from > > the old API to the new plugin API. That means being able to break up > > the existing load_module implementation: > > > > http://hg.python.org/cpython/file/1787277915e9/Python/importdl.c#l23 > > > > For loading, that's a fairly straightforward create_module/exec_module > > split, but reloading gets a bit more interesting. > > > > Specifically, I'd like to be able to implement the relevant parts of > > _PyImport_FindExtensionObject as a precheck for reloading: > > > > http://hg.python.org/cpython/file/1787277915e9/Python/import.c#l533 > > > > That means just having access to the module name isn't enough: the > > extensions dictionary is keyed by a (name, filename) 2-tuple rather > > than just by the module name. Using the find_spec API, that filename > > would be stored in the loader state on the spec object rather than > > being looked up anew during the load operation. > > > > However, rereading this method also indicates that I really want to > > know *in exec_module* whether this is a reload or not, since extension > > loading needs to handle reloads differently from initial execution. > > > > So I'm back to my original preference: I'd like the previous spec to > > be passed to exec_module in the reloading case. If reloading is not > > supported at all, it's up to the loader to throw an appropriate > > exception when the the previous spec is not None. If loading and > > reloading doesn't make any difference, then they can just ignore it. > > But when both are supported, but handled differently (as for extension > > modules), then that can be detected, and the information from the old > > spec (including the original loader and loader state) is available if > > needed. > > Our recent discovery about reloading should probably be reflected in > the signature of finder.find_spec(): > > MetaPathFinder.find_spec(name, path=None, existing=None) > PathEntryFinder.find_spec(name, existing=None) > > This way the finder has an opportunity to incorporate information from > an existing spec into the spec it returns. reload() would make use of > this by passing module.__spec__ (or None if the module has no > __spec__) to _bootstrap._find_spec(). > > This approach should also address what you are looking for. I'd > prefer it over passing the existing spec to exec_module(). The module > (and its __spec__) should have everything exec_module() needs to do > its job. Yes, that should work. > We would still need to use loader.supports_reload() in reload(). Why? If the reload isn't supported, exec_module can just throw an exception based on the loader state in the spec. >From the import system's point of view "reload not permitted" is no different from any other exec time failure. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Wed Oct 30 23:54:13 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 30 Oct 2013 16:54:13 -0600 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Wed, Oct 30, 2013 at 4:09 PM, Nick Coghlan wrote: > On 31 Oct 2013 03:41, "Eric Snow" wrote: >> Our recent discovery about reloading should probably be reflected in >> the signature of finder.find_spec(): >> >> MetaPathFinder.find_spec(name, path=None, existing=None) >> PathEntryFinder.find_spec(name, existing=None) >> >> This way the finder has an opportunity to incorporate information from >> an existing spec into the spec it returns. reload() would make use of >> this by passing module.__spec__ (or None if the module has no >> __spec__) to _bootstrap._find_spec(). >> >> This approach should also address what you are looking for. I'd >> prefer it over passing the existing spec to exec_module(). The module >> (and its __spec__) should have everything exec_module() needs to do >> its job. > > Yes, that should work. Cool. I'll update the PEP. > >> We would still need to use loader.supports_reload() in reload(). > > Why? If the reload isn't supported, exec_module can just throw an exception > based on the loader state in the spec. At the point that exec_module() gets called, the loader can't check sys.modules to see if it's a reload or not. As a workaround, the finder could set up some loader state to indicate to the loader that it's a reload and then the loader, during exec_module(), would check that and act accordingly. However, that's the sort of boilerplate that PEP 451 is trying to offload onto the import machinery. With Loader.supports_reload() it's a lot cleaner. -eric > > From the import system's point of view "reload not permitted" is no > different from any other exec time failure. > > Cheers, > Nick. From victor.stinner at gmail.com Thu Oct 31 02:35:25 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 31 Oct 2013 02:35:25 +0100 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: Message-ID: New update of the PEP combining various remarks: * Remove GroupedStats class and Snapshot.group_by(): replaced with a new Snapshot.statistics() method which combines all features * Rename reset() to clear_traces() and explain how to get traces before clearing traces * Snapshot.apply_filters() now returns a new Snapshot instance * Rename Filter.include to Filter.inclusive * Rename Filter.traceback to Filter.all_frames * Add a section "Log calls to the memory allocator" Thanks Jim, Charles-Fran?ois and Kristjan for your feedback! Here is the new section, tell me if it sounds good. I didn't implement logging just to compare performances. "slower than" is just a previous experience with a very basic logging code. Tell me if you disagree. @Kristjan: I understood that you implemented a tool to log calls on a Playstation3 and send them over the network. How do you process so much data (I computed 29 GB/hour)? Do you log all calls, or only a few of them? Rejected Alternatives ===================== Log calls to the memory allocator --------------------------------- A different approach is to log calls to ``malloc()``, ``realloc()`` and ``free()`` functions. Calls can be logged into a file or send to another computer through the network. Example of a log entry: name of the function, size of the memory block, address of the memory block, Python traceback where the allocation occurred, timestamp. Logs cannot be used directly, getting the current status of the memory requires to parse previous logs. For example, it is not possible to get directly the traceback of a Python object, like ``get_object_traceback(obj)`` does with traces. Python uses objects with a very short lifetime and so makes an extensive use of memory allocators. It has an allocator optimized for small objects (less than 512 bytes) with a short lifetime. For example, the Python test suites calls ``malloc()``, ``realloc()`` or ``free()`` 270,000 times per second in average. If the size of log entry is 32 bytes, logging produces 8.2 MB per second or 29.0 GB per hour. The alternative was rejected because it is less efficient and has less features. Parsing logs in a different process or a different computer is slower than maintaining traces on allocated memory blocks in the same process. Victor From victor.stinner at gmail.com Thu Oct 31 02:42:53 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 31 Oct 2013 02:42:53 +0100 Subject: [Python-Dev] Updated PEP 454 (tracemalloc): no more metrics! In-Reply-To: References: Message-ID: 2013/10/31 Victor Stinner : > Log calls to the memory allocator > --------------------------------- > > A different approach is to log calls to ``malloc()``, ``realloc()`` and > ``free()`` functions. Calls can be logged into a file or send to another > computer through the network. Example of a log entry: name of the > function, size of the memory block, address of the memory block, Python > traceback where the allocation occurred, timestamp. > > Logs cannot be used directly, getting the current status of the memory > requires to parse previous logs. For example, it is not possible to get > directly the traceback of a Python object, like > ``get_object_traceback(obj)`` does with traces. > > Python uses objects with a very short lifetime and so makes an extensive > use of memory allocators. It has an allocator optimized for small > objects (less than 512 bytes) with a short lifetime. For example, the > Python test suites calls ``malloc()``, ``realloc()`` or ``free()`` > 270,000 times per second in average. If the size of log entry is 32 > bytes, logging produces 8.2 MB per second or 29.0 GB per hour. > > The alternative was rejected because it is less efficient and has less > features. Parsing logs in a different process or a different computer is > slower than maintaining traces on allocated memory blocks in the same > process. "less features": get_object_traceback(obj), get_traces() and Snapshot.statistics() can be computed from the log, but you have to process a lot of data. How much time does it take to compute statistics on 1 hour of logs? And for 1 week of logs? With tracemalloc you get these information in a few seconds (immediatly for get_object_traceback(). It should be possible to compute statistics every N minutes and store the result to not have to parse the whole log file at once. Victor From ncoghlan at gmail.com Thu Oct 31 05:24:17 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 31 Oct 2013 14:24:17 +1000 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On 31 Oct 2013 08:54, "Eric Snow" wrote: > > On Wed, Oct 30, 2013 at 4:09 PM, Nick Coghlan wrote: > > On 31 Oct 2013 03:41, "Eric Snow" wrote: > >> Our recent discovery about reloading should probably be reflected in > >> the signature of finder.find_spec(): > >> > >> MetaPathFinder.find_spec(name, path=None, existing=None) > >> PathEntryFinder.find_spec(name, existing=None) > >> > >> This way the finder has an opportunity to incorporate information from > >> an existing spec into the spec it returns. reload() would make use of > >> this by passing module.__spec__ (or None if the module has no > >> __spec__) to _bootstrap._find_spec(). > >> > >> This approach should also address what you are looking for. I'd > >> prefer it over passing the existing spec to exec_module(). The module > >> (and its __spec__) should have everything exec_module() needs to do > >> its job. > > > > Yes, that should work. > > Cool. I'll update the PEP. > > > > >> We would still need to use loader.supports_reload() in reload(). > > > > Why? If the reload isn't supported, exec_module can just throw an exception > > based on the loader state in the spec. > > At the point that exec_module() gets called, the loader can't check > sys.modules to see if it's a reload or not. As a workaround, the > finder could set up some loader state to indicate to the loader that > it's a reload and then the loader, during exec_module(), would check > that and act accordingly. However, that's the sort of boilerplate > that PEP 451 is trying to offload onto the import machinery. With > Loader.supports_reload() it's a lot cleaner. There's also the option of implementing the constraint directly in the finder, which *does* have the necessary info (with the change to pass the previous spec to find_spec). I still think it makes more sense to leave this out for the moment - it's not at all clear we need the extra method, and adding it later would be a straightforward protocol update. Cheers, Nick. > > -eric > > > > > From the import system's point of view "reload not permitted" is no > > different from any other exec time failure. > > > > Cheers, > > Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull at sk.tsukuba.ac.jp Thu Oct 31 06:08:42 2013 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 31 Oct 2013 14:08:42 +0900 Subject: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear? In-Reply-To: References: <527080e4.651e310a.0765.25fa@mx.google.com> Message-ID: <87sivirpr9.fsf@uwakimon.sk.tsukuba.ac.jp> Jim Jewett writes: > Later, he wrote: > > I don't see why disable() would return data. > > disable is indeed a bad name for something that returns data. Note that I never proposed that disable() *return* anything, only that it *get* the trace. It could store it in some specified object, or a file, rather than return it, for example. I deliberately left what it does with the retrieved data unspecified. The important thing to me is that it not be dropped on the floor by something named "disable". From ericsnowcurrently at gmail.com Thu Oct 31 09:52:11 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 31 Oct 2013 02:52:11 -0600 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Wed, Oct 30, 2013 at 10:24 PM, Nick Coghlan wrote: > There's also the option of implementing the constraint directly in the > finder, which *does* have the necessary info (with the change to pass the > previous spec to find_spec). Yeah, I thought of that. I just prefer the more explicit supports_reload(). That said... > > I still think it makes more sense to leave this out for the moment - it's > not at all clear we need the extra method, and adding it later would be a > straightforward protocol update. ...I agree that makes the most sense for now. :) BTW, thanks for pushing these issues. I think the API has gotten pretty solid. I just need to make sure the PEP covers the cases and conclusions we're discussing. -eric From ncoghlan at gmail.com Thu Oct 31 10:52:54 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 31 Oct 2013 19:52:54 +1000 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On 31 Oct 2013 18:52, "Eric Snow" wrote: > > On Wed, Oct 30, 2013 at 10:24 PM, Nick Coghlan wrote: > > There's also the option of implementing the constraint directly in the > > finder, which *does* have the necessary info (with the change to pass the > > previous spec to find_spec). > > Yeah, I thought of that. I just prefer the more explicit > supports_reload(). That said... > > > > > I still think it makes more sense to leave this out for the moment - it's > > not at all clear we need the extra method, and adding it later would be a > > straightforward protocol update. > > ...I agree that makes the most sense for now. :) > > BTW, thanks for pushing these issues. I think the API has gotten > pretty solid. I just need to make sure the PEP covers the cases and > conclusions we're discussing. Thanks are also due to PJE for making me realise we were handwaving too much when it came to the expected reload semantics :) Cheers, Nick. > > -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Oct 31 11:41:20 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 31 Oct 2013 11:41:20 +0100 Subject: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear? In-Reply-To: References: Message-ID: 2013/10/29 Victor Stinner : > 2013/10/29 Kristj?n Valur J?nsson : >> I was thinking something similar. It would be useful to be able to "pause" and "resume" >> if one is doing any analysis work in the live environment. This would reduce the >> need to have "Filter" objects. > > Internally, tracemalloc uses a thread-local variable (called the > "reentrant" flag) to disable temporarly tracing allocations in the > current thread. It only disables tracing new allocations, > deallocations are still proceed. If I give access to this flag, it would be possible to disable temporarily tracing in the current thread, but tracing would still be enabled in other threads. Would it fit your requirement? Example: --------------- tracemalloc.enable() # start your application ... # spawn many threads ... # oh no, I don't want to trace this ugly function tracemalloc.disable_local() ugly_function() tracemalloc.enable_local() ... snapshot = take_snapshot() --------------- You can imagine a context manager based on these two functions: --------------- with disable_tracing_temporarily_in_current_thread(): ugly_function() --------------- I still don't understand why you would need to stop tracing temporarily. When I use tracemalloc, I never disable it. Victor From victor.stinner at gmail.com Thu Oct 31 13:20:48 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 31 Oct 2013 13:20:48 +0100 Subject: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear? In-Reply-To: References: Message-ID: 2013/10/31 Victor Stinner : > If I give access to this flag, it would be possible to disable > temporarily tracing in the current thread, but tracing would still be > enabled in other threads. Would it fit your requirement? It's probably not what you are looking for :-) As I wrote in the PEP, the API of tracemalloc was inspired by the faulthandler module. enable() / disable() makes sense in faulthandler because faulthandler is passive: it only do something on a trigger (synchonous signals like SIGFPE or SIGSEGV). I realized that tracemalloc is different: as written in the documentation, enable() *starts* tracing. After enable() has been called, tracemalloc becomes active. So tracemalloc should use names start() / stop() rather than enable() / disable(). I did another experiment. I replaced enable/disable/is_enabled with start/stop/is_tracing, and added enable/disable/is_enabled functions to disable temporarily tracing. API: - clear_traces(): clear traces - start(): start tracing (the old "enable") - stop(): stop tracing and clear traces (the old "disable") - disable(): disable temporarily tracing - enable(): reenable tracing - is_tracing(): True if tracemalloc is tracing, False otherwise (the old "is_enabled") - is_enabled(): True if tracemalloc is enabled, False otherwise All these functions are process-wide (affect all threads). tracemalloc is only tracing new allocations if is_tracing() and is_enabled() are True. If is_tracing() is True and is_enabled() is False, deallocations still remove traces (otherwise, the internal dictionary of traces would become inconsistent). Example: --------------- tracemalloc.start() # start your application ... useful = UsefulObject() huge = HugeObject() ... snapshot1 = take_snapshot() ... # oh no, I don't want to trace this ugly object, but please don't trash old traces tracemalloc.disable() ugly = ugly_object() ... # release memory of the huge object huge = None ... # restart tracing (ugly is still alive) tracemalloc.enable() ... snapshot2 = take_snapshot() tracemalloc.stop() --------------- snapshot1 contains traces of objects: - useful - huge snapshot2 contains traces of objects: - useful huge is missing from snapshot2 even if the module was disabled. ugly is missing from snapshot2 because tracing was disabled. Does it look better? I don't see the usecase of disable() / enable() yet, but it's cheap (it just add a flag). Victor From ethan at stoneleaf.us Thu Oct 31 15:32:42 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 31 Oct 2013 07:32:42 -0700 Subject: [Python-Dev] PEP 454 (tracemalloc) disable ==> clear? In-Reply-To: References: Message-ID: <52726A0A.8000808@stoneleaf.us> On 10/31/2013 05:20 AM, Victor Stinner wrote: > I did another experiment. I replaced enable/disable/is_enabled with > start/stop/is_tracing, and added enable/disable/is_enabled functions > to disable temporarily tracing. > > API: > > - clear_traces(): clear traces > - start(): start tracing (the old "enable") > - stop(): stop tracing and clear traces (the old "disable") > - disable(): disable temporarily tracing > - enable(): reenable tracing > - is_tracing(): True if tracemalloc is tracing, False otherwise (the > old "is_enabled") > - is_enabled(): True if tracemalloc is enabled, False otherwise These names make more sense. However, `stop` is still misleading as it both stops and destroys data. An easy fix for that is for stop to save the data somewhere so get_traces (or whatever) can still retrieve it. If `stop` really must destroy the data, perhaps it should be called `close` instead; StringIO has a similar close method that when called destroys any stored data, and get_value must be called first if that data is wanted. -- ~Ethan~ From python at mrabarnett.plus.com Thu Oct 31 15:48:04 2013 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 31 Oct 2013 14:48:04 +0000 Subject: [Python-Dev] Compiler security Message-ID: <52726DA4.3090502@mrabarnett.plus.com> Has anybody here heard about this, and, if so, is it anything we should be thinking about: How your compiler may be compromising application security http://www.itworld.com/security/380406/how-your-compiler-may-be-compromising-application-security From christian at python.org Thu Oct 31 15:54:23 2013 From: christian at python.org (Christian Heimes) Date: Thu, 31 Oct 2013 15:54:23 +0100 Subject: [Python-Dev] Compiler security In-Reply-To: <52726DA4.3090502@mrabarnett.plus.com> References: <52726DA4.3090502@mrabarnett.plus.com> Message-ID: Am 31.10.2013 15:48, schrieb MRAB: > Has anybody here heard about this, and, if so, is it anything we should > be thinking about: > > How your compiler may be compromising application security > http://www.itworld.com/security/380406/how-your-compiler-may-be-compromising-application-security http://bugs.python.org/issue17405 addresses one issue with dead code elimination. From guido at python.org Thu Oct 31 15:55:25 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 31 Oct 2013 07:55:25 -0700 Subject: [Python-Dev] Compiler security In-Reply-To: <52726DA4.3090502@mrabarnett.plus.com> References: <52726DA4.3090502@mrabarnett.plus.com> Message-ID: Interesting read. I'm surprised that the researchers didn't contact us, since the article mentions they found 5 bugs in Python. Regarding security: the article seems to use that term mostly to attract eyeballs; there are no specifics, just the implication that this *could* affect security. But it's hardly news -- as GCC versions became more aggressive we've had to fix our share of undefined code in Python. Usually the unittests catch these early. On Thu, Oct 31, 2013 at 7:48 AM, MRAB wrote: > Has anybody here heard about this, and, if so, is it anything we should > be thinking about: > > How your compiler may be compromising application security > http://www.itworld.com/**security/380406/how-your-** > compiler-may-be-compromising-**application-security > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/**mailman/options/python-dev/** > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Thu Oct 31 15:56:22 2013 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 31 Oct 2013 10:56:22 -0400 Subject: [Python-Dev] Compiler security In-Reply-To: <52726DA4.3090502@mrabarnett.plus.com> References: <52726DA4.3090502@mrabarnett.plus.com> Message-ID: I believe the 5 problems they found in Python were dealt with here http://bugs.python.org/issue17016 2013/10/31 MRAB : > Has anybody here heard about this, and, if so, is it anything we should > be thinking about: > > How your compiler may be compromising application security > http://www.itworld.com/security/380406/how-your-compiler-may-be-compromising-application-security > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org -- Regards, Benjamin From christian at python.org Thu Oct 31 15:57:32 2013 From: christian at python.org (Christian Heimes) Date: Thu, 31 Oct 2013 15:57:32 +0100 Subject: [Python-Dev] Compiler security In-Reply-To: <52726DA4.3090502@mrabarnett.plus.com> References: <52726DA4.3090502@mrabarnett.plus.com> Message-ID: Am 31.10.2013 15:48, schrieb MRAB: > Has anybody here heard about this, and, if so, is it anything we should > be thinking about: > > How your compiler may be compromising application security > http://www.itworld.com/security/380406/how-your-compiler-may-be-compromising-application-security > I didnt' see this at first: STACK was run against a number of systems written in C/C++ and it found 160 new bugs in the systems tested, including ... and Python (5). Has anybody contact us? I neither saw a bug report nor a mail to PSRT. Christian From pje at telecommunity.com Thu Oct 31 16:36:53 2013 From: pje at telecommunity.com (PJ Eby) Date: Thu, 31 Oct 2013 11:36:53 -0400 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On Thu, Oct 31, 2013 at 5:52 AM, Nick Coghlan wrote: > > On 31 Oct 2013 18:52, "Eric Snow" wrote: >> >> On Wed, Oct 30, 2013 at 10:24 PM, Nick Coghlan wrote: >> > There's also the option of implementing the constraint directly in the >> > finder, which *does* have the necessary info (with the change to pass >> > the >> > previous spec to find_spec). >> >> Yeah, I thought of that. I just prefer the more explicit >> supports_reload(). That said... >> >> > >> > I still think it makes more sense to leave this out for the moment - >> > it's >> > not at all clear we need the extra method, and adding it later would be >> > a >> > straightforward protocol update. >> >> ...I agree that makes the most sense for now. :) >> >> BTW, thanks for pushing these issues. I think the API has gotten >> pretty solid. I just need to make sure the PEP covers the cases and >> conclusions we're discussing. > > Thanks are also due to PJE for making me realise we were handwaving too much > when it came to the expected reload semantics :) You're welcome. ;-) But speaking of handwaving, I also want to be sure that loader developers know that "reloading" is only really "reloading" if there's a previous existing spec, or the module type is... Hm. Actually, I think I now know how to state what's bugging me every time I see this "supports_reload()" or "reload=True" or other reloading flags in this process. I think that references to reloading should be replaced with references to what's *actually* at issue, because "reloading" itself is vague and carries too many assumptions for a loader author to understand or get right. (Look how hard it is for *us*!) That is, I think we should clarify what use cases there are for knowing whether a "reload" is happening, and address those use cases explicitly rather than lumping them under a general heading. For example, if the reason a loader cares about reloading is because it's a C extension using a custom module type, and the existing module isn't of the right type, then we should just spell out how to handle it. (e.g. raise an exception) If the reason a loader cares about reloading is because of some sort of caching or reuse, then we should just spell out how to handle that, too. Lumping these cases together under a "reloading" flag or a check for "reloading" support is a nasty code smell, because it requires a loader developer to have the *same* vaguely-defined idea of "reloading" as the PEP authors. ;-) I also suspect, that if properly spelled out, those use cases are going to boil down to: 1. Throwing errors if you have an existing module object you can't load into, and 2. Passing in a previous spec object, if available In other words, loaders should not really have any responsibility for or concept of "reloading" -- they always load into a module object (that they may or may not have created), and they may get given a spec from a previous load. They should deal only in "module reuse" and "spec reuse". While a typical reload() might involve both reuses, there are cases where one sort of reuse could occur independently, and not all loaders care about both (or even either) condition. At any rate, it means a loader author doesn't have to figure out how to handle "reloading", all they have to figure out is whether they can load into a particular module object, and whether they can do something useful with a spec that was previously used to load a module with the same name -- a spec that may or may not refer to a similar previous loader. These are rather more well-defined endeavors than trying to determine in the abstract whether one "supports reload". ;-) From storchaka at gmail.com Thu Oct 31 17:01:05 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 31 Oct 2013 18:01:05 +0200 Subject: [Python-Dev] Compiler security In-Reply-To: References: <52726DA4.3090502@mrabarnett.plus.com> Message-ID: 31.10.13 16:56, Benjamin Peterson ???????(??): > I believe the 5 problems they found in Python were dealt with here > http://bugs.python.org/issue17016 Ah, now I have remembered author's name. http://bugs.python.org/issue18684 contains some other fixes of this kind. From tjreedy at udel.edu Thu Oct 31 21:49:34 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 31 Oct 2013 16:49:34 -0400 Subject: [Python-Dev] Compiler security In-Reply-To: References: <52726DA4.3090502@mrabarnett.plus.com> Message-ID: On 10/31/2013 10:57 AM, Christian Heimes wrote: > I didnt' see this at first: > > STACK was run against a number of systems written in C/C++ and > it found 160 new bugs in the systems tested, including ... > and Python (5). > > Has anybody contact us? I neither saw a bug report nor a mail to PSRT. from http://css.csail.mit.edu/stack/ Our software is hosted on Github at https://github.com/xiw/stack/. Obtain the latest code of STACK using: git clone git://github.com/xiw/stack See README and INSTALL for more information. -- Terry Jan Reedy From ncoghlan at gmail.com Thu Oct 31 23:10:44 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 1 Nov 2013 08:10:44 +1000 Subject: [Python-Dev] PEP 451 update In-Reply-To: References: Message-ID: On 1 Nov 2013 01:37, "PJ Eby" wrote: . ;-) > > I also suspect, that if properly spelled out, those use cases are > going to boil down to: > > 1. Throwing errors if you have an existing module object you can't > load into, and > 2. Passing in a previous spec object, if available > > In other words, loaders should not really have any responsibility for > or concept of "reloading" -- they always load into a module object > (that they may or may not have created), and they may get given a spec > from a previous load. They should deal only in "module reuse" and > "spec reuse". While a typical reload() might involve both reuses, > there are cases where one sort of reuse could occur independently, and > not all loaders care about both (or even either) condition. > > At any rate, it means a loader author doesn't have to figure out how > to handle "reloading", all they have to figure out is whether they can > load into a particular module object, and whether they can do > something useful with a spec that was previously used to load a module > with the same name -- a spec that may or may not refer to a similar > previous loader. These are rather more well-defined endeavors than > trying to determine in the abstract whether one "supports reload". > ;-) It may not have been clear from the email exchange, but that's basically where we ended up :) The change Eric is currently going to make to the PEP is to add an optional "previous" parameter to the various find_spec APIs. At the time when find_spec runs, sys.modules hasn't been touched yet, so the old module state is still available when reloading. Passing the spec in lets the loader decide whether or not it can actually load that module given the information about the previous load operation. However, perhaps it makes more sense to pass the entire existing module, rather than just the spec? I'd like to minimise the need for new-style loader authors to retrieve things directly from the sys module, and you're right that "can I reload into this particular module?" is a slightly different question from "can I reload a module previously loaded using this particular spec?". A spec-based API could still be used to answer the first question by looking up sys.modules, but so could the name based API. Passing in the module reference, on the other hand, should let loaders answer both questions without needing to touch the sys module. I also now think this pass-the-module approach when finding the spec approach could also clean up some of the messiness that is __main__ initialisation, where we repeatedly load things into the same module object. Let's say we be completely explicit and call the new parameter something like "load_target". If we do that then I would make the *same* change to runpy.run_path and runpy.run_module: let you pass in an existing module object under that name to say "hey, run in this namespace, rather than a fresh one". (Those APIs currently only support pre-populating a *fresh* namespace, rather than allowing you to directly specify an existing one) Most loaders won't care, but those that do will have all the info they need to throw an exception saying "I could provide a spec for this, but it's not compatible with that load target". Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: