From martin at v.loewis.de Sat May 1 00:00:18 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 01 May 2010 00:00:18 +0200 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <4BDB50C8.1090205@holdenweb.com> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> <4BDB4D26.3040904@v.loewis.de> <4BDB50C8.1090205@holdenweb.com> Message-ID: <4BDB52F2.6010508@v.loewis.de> Steve Holden wrote: > Martin v. L?wis wrote: >>>> Without a BDFL, I think we need a committee to make decisions, e.g. by >>>> majority vote amongst committers. >>> Couldn't we just go with the FLUFL? >> Not sure whether that's a serious proposal (April 1 is already some days >> back now). As a starting point, Barry would have to indicate whether he >> is interested in that role. >> > If he isn't then we can depose him and replace him with a puppet. Ah, ok. So you didn't mean that seriously; a smiley would have helped. Regards, Martin From martin at v.loewis.de Sat May 1 00:13:23 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 01 May 2010 00:13:23 +0200 Subject: [Python-Dev] patch for review: __import__ documentation In-Reply-To: References: Message-ID: <4BDB5603.6060907@v.loewis.de> > I see the confusion. I think Martin meant more about open issues that > required discussion, not simply issues that had a patch ready to go. I actually think it is perfectly fine to point out that specific issues are need committer action on this list. This is what the list is there for. Waiting some time to see whether some developer reacts is certainly a good idea: notice, however, that Chris had already waited a few days. Regards, Martin From jnoller at gmail.com Sat May 1 00:54:06 2010 From: jnoller at gmail.com (Jesse Noller) Date: Fri, 30 Apr 2010 18:54:06 -0400 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <4BDB34CE.9080804@v.loewis.de> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> Message-ID: <61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com> On Apr 30, 2010, at 3:51 PM, "Martin v. L?wis" wrote: >> As to Guido's point about the decision making process, Nick's >> right. I just >> want to make sure we can capture the resolution in the PEP, be it >> by BDFL >> pronouncement or "hey, silence is acceptance" email. > > I don't think "silence is acceptance" will work out in practice. For > issues where a PEP was written in the first place, somebody will > *always* object, and forever so, hoping that what he considers a > mistake > will not be done. The advantage of Guido acting as BDFL was that > somebody would make an decision ultimately, one which both proponents > and opponents of the PEP would accept. > > Without a BDFL, I think we need a committee to make decisions, e.g. by > majority vote amongst committers. > > Regards, > Martin > Consider this a plaintitive -1 to any sort of rule-or-decision based on committee. I'd much rather a 2x4 to the forehead. jesse From solipsis at pitrou.net Sat May 1 01:00:57 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 30 Apr 2010 23:00:57 +0000 (UTC) Subject: [Python-Dev] Two small PEP ideas References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com> Message-ID: Jesse Noller gmail.com> writes: > > Consider this a plaintitive -1 to any sort of rule-or-decision based > on committee. > > I'd much rather a 2x4 to the forehead. Oops, sorry but what does "a 2x4 to the forehead" mean? (and "plaintitive" by the way?) Regards Antoine. From benjamin at python.org Sat May 1 01:08:54 2010 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 30 Apr 2010 18:08:54 -0500 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com> Message-ID: 2010/4/30 Antoine Pitrou : > Jesse Noller gmail.com> writes: >> >> Consider this a plaintitive -1 to any sort of rule-or-decision based >> on committee. >> >> I'd much rather a 2x4 to the forehead. > > Oops, sorry but what does "a 2x4 to the forehead" mean? > (and "plaintitive" by the way?) The former means being hit by a board and by the latter, he probably means "plaintive" meaning melancholy. -- Regards, Benjamin From fijall at gmail.com Sat May 1 01:10:22 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 30 Apr 2010 17:10:22 -0600 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com> Message-ID: On Fri, Apr 30, 2010 at 5:08 PM, Benjamin Peterson wrote: > 2010/4/30 Antoine Pitrou : >> Jesse Noller gmail.com> writes: >>> >>> Consider this a plaintitive -1 to any sort of rule-or-decision based >>> on committee. >>> >>> I'd much rather a 2x4 to the forehead. >> >> Oops, sorry but what does "a 2x4 to the forehead" mean? >> (and "plaintitive" by the way?) > > The former means being hit by a board and by the latter, he probably > means "plaintive" meaning melancholy. > Does it only work for english speaking non-metric system users? (2x4 is probably something imperial) From fuzzyman at voidspace.org.uk Sat May 1 01:13:17 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 01 May 2010 00:13:17 +0100 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com> Message-ID: <4BDB640D.8000000@voidspace.org.uk> On 01/05/2010 00:08, Benjamin Peterson wrote: > 2010/4/30 Antoine Pitrou: > >> Jesse Noller gmail.com> writes: >> >>> Consider this a plaintitive -1 to any sort of rule-or-decision based >>> on committee. >>> >>> I'd much rather a 2x4 to the forehead. >>> >> Oops, sorry but what does "a 2x4 to the forehead" mean? >> (and "plaintitive" by the way?) >> > The former means being hit by a board and by the latter, he probably > means "plaintive" meaning melancholy. > > And 2x4 refers to a board with cross-section of 4 inches x 2 inches; the inch being an obsolete unit of measure from the old British imperial system now defunct throughout the civilized world. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From fuzzyman at voidspace.org.uk Sat May 1 01:13:59 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 01 May 2010 00:13:59 +0100 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com> Message-ID: <4BDB6437.90001@voidspace.org.uk> On 01/05/2010 00:10, Maciej Fijalkowski wrote: > On Fri, Apr 30, 2010 at 5:08 PM, Benjamin Peterson wrote: > >> 2010/4/30 Antoine Pitrou: >> >>> Jesse Noller gmail.com> writes: >>> >>>> Consider this a plaintitive -1 to any sort of rule-or-decision based >>>> on committee. >>>> >>>> I'd much rather a 2x4 to the forehead. >>>> >>> Oops, sorry but what does "a 2x4 to the forehead" mean? >>> (and "plaintitive" by the way?) >>> >> The former means being hit by a board and by the latter, he probably >> means "plaintive" meaning melancholy. >> >> > Does it only work for english speaking non-metric system users? (2x4 > is probably something imperial) > You we would probably strike with 100x50mm boards instead. Michael > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From benjamin at python.org Sat May 1 01:14:28 2010 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 30 Apr 2010 18:14:28 -0500 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com> Message-ID: 2010/4/30 Maciej Fijalkowski : > On Fri, Apr 30, 2010 at 5:08 PM, Benjamin Peterson wrote: >> 2010/4/30 Antoine Pitrou : >>> Jesse Noller gmail.com> writes: >>>> >>>> Consider this a plaintitive -1 to any sort of rule-or-decision based >>>> on committee. >>>> >>>> I'd much rather a 2x4 to the forehead. >>> >>> Oops, sorry but what does "a 2x4 to the forehead" mean? >>> (and "plaintitive" by the way?) >> >> The former means being hit by a board and by the latter, he probably >> means "plaintive" meaning melancholy. >> > > Does it only work for english speaking non-metric system users? (2x4 > is probably something imperial) Yes, that's a 5,04 by 10,08 for you. :) -- Regards, Benjamin From guido at python.org Sat May 1 03:11:19 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Apr 2010 18:11:19 -0700 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com> Message-ID: On Fri, Apr 30, 2010 at 4:14 PM, Benjamin Peterson wrote: > 2010/4/30 Maciej Fijalkowski : >> On Fri, Apr 30, 2010 at 5:08 PM, Benjamin Peterson wrote: >>> 2010/4/30 Antoine Pitrou : >>>> Jesse Noller gmail.com> writes: >>>>> >>>>> Consider this a plaintitive -1 to any sort of rule-or-decision based >>>>> on committee. >>>>> >>>>> I'd much rather a 2x4 to the forehead. >>>> >>>> Oops, sorry but what does "a 2x4 to the forehead" mean? >>>> (and "plaintitive" by the way?) >>> >>> The former means being hit by a board and by the latter, he probably >>> means "plaintive" meaning melancholy. >>> >> >> Does it only work for english speaking non-metric system users? (2x4 >> is probably something imperial) > > Yes, that's a 5,04 by 10,08 for you. :) Of course "2 by 4" is just the name. The actual measurements of such a piece of lumber in the store are about 1.75 by 3.75 inch. :-) -- --Guido van Rossum (python.org/~guido) From steve at holdenweb.com Sat May 1 04:46:19 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 30 Apr 2010 22:46:19 -0400 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <4BDB640D.8000000@voidspace.org.uk> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com> <4BDB640D.8000000@voidspace.org.uk> Message-ID: Michael Foord wrote: > On 01/05/2010 00:08, Benjamin Peterson wrote: >> 2010/4/30 Antoine Pitrou: >> >>> Jesse Noller gmail.com> writes: >>> >>>> Consider this a plaintitive -1 to any sort of rule-or-decision based >>>> on committee. >>>> >>>> I'd much rather a 2x4 to the forehead. >>>> >>> Oops, sorry but what does "a 2x4 to the forehead" mean? >>> (and "plaintitive" by the way?) >>> >> The former means being hit by a board and by the latter, he probably >> means "plaintive" meaning melancholy. >> >> > And 2x4 refers to a board with cross-section of 4 inches x 2 inches; the > inch being an obsolete unit of measure from the old British imperial > system now defunct throughout the civilized world. > The last time I was in a UK builders' yard I hear someone asking for "two meter pieces of two by four". At the time the UK was notionally metric (and the timber was planed to the nearest metric size) but the old names still survived. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From jnoller at gmail.com Sat May 1 05:41:14 2010 From: jnoller at gmail.com (Jesse Noller) Date: Fri, 30 Apr 2010 23:41:14 -0400 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com> Message-ID: On Fri, Apr 30, 2010 at 9:11 PM, Guido van Rossum wrote: > On Fri, Apr 30, 2010 at 4:14 PM, Benjamin Peterson wrote: >> 2010/4/30 Maciej Fijalkowski : >>> On Fri, Apr 30, 2010 at 5:08 PM, Benjamin Peterson wrote: >>>> 2010/4/30 Antoine Pitrou : >>>>> Jesse Noller gmail.com> writes: >>>>>> >>>>>> Consider this a plaintitive -1 to any sort of rule-or-decision based >>>>>> on committee. >>>>>> >>>>>> I'd much rather a 2x4 to the forehead. >>>>> >>>>> Oops, sorry but what does "a 2x4 to the forehead" mean? >>>>> (and "plaintitive" by the way?) >>>> >>>> The former means being hit by a board and by the latter, he probably >>>> means "plaintive" meaning melancholy. >>>> >>> >>> Does it only work for english speaking non-metric system users? (2x4 >>> is probably something imperial) >> >> Yes, that's a 5,04 by 10,08 for you. :) > > Of course "2 by 4" is just the name. The actual measurements of such a > piece of lumber in the store are about 1.75 by 3.75 inch. :-) > We americans round up! (all the time) From jnoller at gmail.com Sat May 1 05:42:14 2010 From: jnoller at gmail.com (Jesse Noller) Date: Fri, 30 Apr 2010 23:42:14 -0400 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com> Message-ID: On Fri, Apr 30, 2010 at 7:08 PM, Benjamin Peterson wrote: > 2010/4/30 Antoine Pitrou : >> Jesse Noller gmail.com> writes: >>> >>> Consider this a plaintitive -1 to any sort of rule-or-decision based >>> on committee. >>> >>> I'd much rather a 2x4 to the forehead. >> >> Oops, sorry but what does "a 2x4 to the forehead" mean? >> (and "plaintitive" by the way?) > > The former means being hit by a board and by the latter, he probably > means "plaintive" meaning melancholy. Yes, well, what Benjamin said. jesse From ncoghlan at gmail.com Sat May 1 06:01:25 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 01 May 2010 14:01:25 +1000 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com> <4BDB640D.8000000@voidspace.org.uk> Message-ID: <4BDBA795.7010204@gmail.com> Steve Holden wrote: > The last time I was in a UK builders' yard I hear someone asking for > "two meter pieces of two by four". At the time the UK was notionally > metric (and the timber was planed to the nearest metric size) but the > old names still survived. Yeah, a 2x4 is still a 2x4 here as well. It's English, even the native speakers don't really expect it to make sense all the time ;) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Sat May 1 07:12:38 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 01 May 2010 07:12:38 +0200 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> Message-ID: <4BDBB846.5020305@v.loewis.de> > IIRC in the IETF this is done by the committee chair. I think it's a > good idea to have this be a single person to avoid endless indecision. It then seems that this role should go to the release manager of the upcoming feature release. Assuming Georg can accept this additional responsibility. Regards, Martin From dirkjan at ochtman.nl Sat May 1 08:58:03 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sat, 1 May 2010 08:58:03 +0200 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <20100430150918.308984aa@heresy> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> Message-ID: On Fri, Apr 30, 2010 at 21:09, Barry Warsaw wrote: >>Though maybe it should be called Conclusion instead of Accepted and >>used for Rejected PEPs, as well? > > Good point. ?What do you think about 'Resolution'? Fine with me. Cheers, Dirkjan From ziade.tarek at gmail.com Sat May 1 11:00:10 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 1 May 2010 11:00:10 +0200 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> Message-ID: On Fri, Apr 30, 2010 at 11:25 PM, Guido van Rossum wrote: [..] >>> >>> Without a BDFL, I think we need a committee to make decisions, e.g. by >>> majority vote amongst committers. >> >> Couldn't we just go with the FLUFL? > > IIRC in the IETF this is done by the committee chair. I think it's a > good idea to have this be a single person to avoid endless indecision. BPD = Benevolent Pep Dictator BPC = Benevolent Pep Caliph (sounds good in French, not sure in English ;) ) What about naming several BPD + and have one BPC elected each year by all the core devs ? == BPD == I am not sure if this would work for all areas in Python-core, but looking at the maintainers.rst file, it looks like we could name for example Brett for all the import machinery things, Marc-Andr? for all the unicode things, I could be the one about packaging, etc. If we could manage to split the python-core development into categories, and add these categories in the PEP metadata, that would define who takes the final decision in case we can't reach consensus. == BPC == Of course some PEPs could concern several categories, so we would still need some kind of Pep dictator if there's no consensus. So what about electing a BPC every year ? Tarek -- Tarek Ziad? | http://ziade.org From martin at v.loewis.de Sat May 1 11:14:22 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 01 May 2010 11:14:22 +0200 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> Message-ID: <4BDBF0EE.8040100@v.loewis.de> > Of course some PEPs could concern several categories, so we would > still need some kind of Pep dictator if there's no consensus. So what > about electing a BPC every year ? I think having a single body/person pronounce on all PEPs is sufficient; as that person should certainly listen to the opinions of the respective experts. See also my proposal that the release manager of the next feature release automatically assumes that role. Regards, Martin From g.brandl at gmx.net Sat May 1 11:35:04 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 01 May 2010 11:35:04 +0200 Subject: [Python-Dev] Frequency of the dev docs autobuild In-Reply-To: <20100429114047.C031C20059D@kimball.webabinitio.net> References: <4BD95C6E.9080508@gmail.com> <20100429114047.C031C20059D@kimball.webabinitio.net> Message-ID: Am 29.04.2010 13:40, schrieb R. David Murray: > On Thu, 29 Apr 2010 20:16:14 +1000, Nick Coghlan wrote: >> Does the online dev version of the docs build in response to docs >> checkins, or just once a day? > > I believe it does it once a day. Georg recently changed how this is done, > so we should get official word from him. Yes, it builds once a day. The build process currently takes quite long, since PDFs are also built every time. We could change the procedure to rebuild the HTML more frequently or on checkin, but that would again make the PDFs mismatch the online docs, and I would say it's not necessary. >> (And is that written down somewhere and I've just forgotten where to >> look...) > > I don't think so, but it should be. It is documented at . The docs themselves bear the "last updated" date only. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From solipsis at pitrou.net Sat May 1 12:34:21 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 1 May 2010 10:34:21 +0000 (UTC) Subject: [Python-Dev] Two small PEP ideas References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> <4BDBF0EE.8040100@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > I think having a single body/person pronounce on all PEPs is sufficient; > as that person should certainly listen to the opinions of the respective > experts. The issue is more a question of personal bandwidth. Giving an informed decision requires reading and listening to a lot of advice, material etc. Then it's not obvious that we will have many PEPs in the future. The syntax is pretty much frozen (even after the moratorium, it seems unlikely anything big will happen -- except if we want to add first-class coroutines). Many enhancements (e.g. new APIs in existing library modules) can be done, and are done, without PEPs at all. Regards Antoine. From martin at v.loewis.de Sat May 1 13:03:16 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 01 May 2010 13:03:16 +0200 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> <4BDBF0EE.8040100@v.loewis.de> Message-ID: <4BDC0A74.6090106@v.loewis.de> > Then it's not obvious that we will have many PEPs in the future. Given Guido's Theorem: the PEPs yet to be written will hopefully outnumber the PEPs written so far. Regards, Martin From ziade.tarek at gmail.com Sat May 1 13:03:33 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 1 May 2010 13:03:33 +0200 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> <4BDBF0EE.8040100@v.loewis.de> Message-ID: On Sat, May 1, 2010 at 12:34 PM, Antoine Pitrou wrote: > Martin v. L?wis v.loewis.de> writes: >> >> I think having a single body/person pronounce on all PEPs is sufficient; >> as that person should certainly listen to the opinions of the respective >> experts. > > The issue is more a question of personal bandwidth. Giving an informed decision > requires reading and listening to a lot of advice, material etc. Yes that was the idea. For instance in packaging matters, I don't expect other core devs to have the time to learn about every aspect of a particular packaging problem to be able to make the best decision. Now if the release manager takes the decision, I guess the result will be the same at the end, as Martin says: he'll ask for the opinion of the respective experts so I guess it works too. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com > -- Tarek Ziad? | http://ziade.org From rdmurray at bitdance.com Sat May 1 16:28:57 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 01 May 2010 10:28:57 -0400 Subject: [Python-Dev] Frequency of the dev docs autobuild In-Reply-To: References: <4BD95C6E.9080508@gmail.com> <20100429114047.C031C20059D@kimball.webabinitio.net> Message-ID: <20100501142857.3CAD51FC3F5@kimball.webabinitio.net> On Sat, 01 May 2010 11:35:04 +0200, Georg Brandl wrote: > Am 29.04.2010 13:40, schrieb R. David Murray: > > On Thu, 29 Apr 2010 20:16:14 +1000, Nick Coghlan wrote: > >> Does the online dev version of the docs build in response to docs > >> checkins, or just once a day? > > > > I believe it does it once a day. Georg recently changed how this is done, > > so we should get official word from him. > > Yes, it builds once a day. The build process currently takes quite long, > since PDFs are also built every time. We could change the procedure to > rebuild the HTML more frequently or on checkin, but that would again > make the PDFs mismatch the online docs, and I would say it's not necessary. > > >> (And is that written down somewhere and I've just forgotten where to > >> look...) > > > > I don't think so, but it should be. > > It is documented at . The docs themselves > bear the "last updated" date only. Unless I'm missing something, I don't see any docs there about the automated build process and when and where it runs. (Note that the old automated build process, the build.sh script, wasn't documented in that sense anywhere that I know of, either. And I can never remember where its web status page is...at least for the new docs procedure we have the last build date in an obvious place :) -- R. David Murray www.bitdance.com From p.f.moore at gmail.com Sat May 1 17:18:19 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 1 May 2010 16:18:19 +0100 Subject: [Python-Dev] Frequency of the dev docs autobuild In-Reply-To: <20100501142857.3CAD51FC3F5@kimball.webabinitio.net> References: <4BD95C6E.9080508@gmail.com> <20100429114047.C031C20059D@kimball.webabinitio.net> <20100501142857.3CAD51FC3F5@kimball.webabinitio.net> Message-ID: On 1 May 2010 15:28, R. David Murray wrote: > Unless I'm missing something, I don't see any docs there about the > automated build process and when and where it runs. I assume Georg was referring to "Development versions of the Python documentation are available on-line and updated daily". Paul. From tseaver at palladion.com Sat May 1 19:12:12 2010 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 01 May 2010 13:12:12 -0400 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jesse Noller wrote: > On Fri, Apr 30, 2010 at 9:11 PM, Guido van Rossum wrote: >> On Fri, Apr 30, 2010 at 4:14 PM, Benjamin Peterson wrote: >>> 2010/4/30 Maciej Fijalkowski : >>>> On Fri, Apr 30, 2010 at 5:08 PM, Benjamin Peterson wrote: >>>>> 2010/4/30 Antoine Pitrou : >>>>>> Jesse Noller gmail.com> writes: >>>>>>> Consider this a plaintitive -1 to any sort of rule-or-decision based >>>>>>> on committee. >>>>>>> >>>>>>> I'd much rather a 2x4 to the forehead. >>>>>> Oops, sorry but what does "a 2x4 to the forehead" mean? >>>>>> (and "plaintitive" by the way?) >>>>> The former means being hit by a board and by the latter, he probably >>>>> means "plaintive" meaning melancholy. >>>>> >>>> Does it only work for english speaking non-metric system users? (2x4 >>>> is probably something imperial) >>> Yes, that's a 5,04 by 10,08 for you. :) >> Of course "2 by 4" is just the name. The actual measurements of such a >> piece of lumber in the store are about 1.75 by 3.75 inch. :-) >> > > We americans round up! (all the time) Lumber is, oddly, named for its "wet size" (when cut), rather than "dry size" (when you buy it at the lumber yard after it dries). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvcYOwACgkQ+gerLs4ltQ5SAgCdHnPJZ+UqTgjcTmW21iphpO1Y BQ8An07mX/cy3opSpNyt0FE2jtVMtLIu =Qqq1 -----END PGP SIGNATURE----- From rdmurray at bitdance.com Sun May 2 00:05:04 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 01 May 2010 18:05:04 -0400 Subject: [Python-Dev] Frequency of the dev docs autobuild In-Reply-To: References: <4BD95C6E.9080508@gmail.com> <20100429114047.C031C20059D@kimball.webabinitio.net> <20100501142857.3CAD51FC3F5@kimball.webabinitio.net> Message-ID: <20100501220504.D35AC200567@kimball.webabinitio.net> On Sat, 01 May 2010 16:18:19 +0100, Paul Moore wrote: > On 1 May 2010 15:28, R. David Murray wrote: > > Unless I'm missing something, I don't see any docs there about the > > automated build process and when and where it runs. > > I assume Georg was referring to "Development versions of the Python > documentation are available on-line and updated daily". Yes, most likely. I think I am asking for more documentation than you were :) -- R. David Murray www.bitdance.com From martin at v.loewis.de Sun May 2 00:44:22 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 02 May 2010 00:44:22 +0200 Subject: [Python-Dev] Frequency of the dev docs autobuild In-Reply-To: <20100501220504.D35AC200567@kimball.webabinitio.net> References: <4BD95C6E.9080508@gmail.com> <20100429114047.C031C20059D@kimball.webabinitio.net> <20100501142857.3CAD51FC3F5@kimball.webabinitio.net> <20100501220504.D35AC200567@kimball.webabinitio.net> Message-ID: <4BDCAEC6.1020202@v.loewis.de> R. David Murray wrote: > On Sat, 01 May 2010 16:18:19 +0100, Paul Moore wrote: >> On 1 May 2010 15:28, R. David Murray wrote: >>> Unless I'm missing something, I don't see any docs there about the >>> automated build process and when and where it runs. >> I assume Georg was referring to "Development versions of the Python >> documentation are available on-line and updated daily". > > Yes, most likely. I think I am asking for more documentation than you > were :) Then I think you have to propose specific wording. It runs on www.python.org (why would you expect it to run elsewhere???), and what about "daily" do you consider imprecise? Do you really need exact hour, minute, and second? What for? Regards, Martin From ncoghlan at gmail.com Sun May 2 05:38:02 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 02 May 2010 13:38:02 +1000 Subject: [Python-Dev] Frequency of the dev docs autobuild In-Reply-To: References: <4BD95C6E.9080508@gmail.com> <20100429114047.C031C20059D@kimball.webabinitio.net> <20100501142857.3CAD51FC3F5@kimball.webabinitio.net> Message-ID: <4BDCF39A.7060206@gmail.com> Paul Moore wrote: > On 1 May 2010 15:28, R. David Murray wrote: >> Unless I'm missing something, I don't see any docs there about the >> automated build process and when and where it runs. > > I assume Georg was referring to "Development versions of the Python > documentation are available on-line and updated daily". Thanks, that's what I was missing. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From brian at sweetapp.com Sun May 2 09:07:25 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sun, 2 May 2010 17:07:25 +1000 Subject: [Python-Dev] [PEP 3148] futures - execute computations asynchronously In-Reply-To: <4BA001E1.8080105@gmail.com> References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> <20100306065016.DBDF33A40A7@sparrow.telecommunity.com> <20100306160434.39FBD3A4077@sparrow.telecommunity.com> <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> <20100307035657.111563A4077@sparrow.telecommunity.com> <4B93303E.9050301@gmail.com> <20100307154809.742E73A4062@sparrow.telecommunity.com> <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com> <20100307175732.307513A4062@sparrow.telecommunity.com> <5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com> <20100307195614.BBD693A4062@sparrow.telecommunity.com> <4B94DC30.2030904@gmail.com> <19349.25574.880099.668647@montanaro.dyndns.org> <4B979292.4080906@gmail.com> <4BA001E1.8080105@gmail.com> Message-ID: I've updated the PEP to include: - completion callbacks (for interoperability with Twisted Deferreds) - a pointer to the discussion on stdlig-sig See: http://svn.python.org/view/peps/trunk/pep-3148.txt?r1=78618&r2=80679 Rejected ideas: - Having a registration system for executors Not yet addressed: - where the package should live (someone in a "concurrent" package seems fine) - having global executors with unbounded worker counts as a convenience [1] [1] There are a few issues with global executors that need to be thought thought through i.e. when should workers be created and when should they be terminated. I'd be happy to defer this idea unless someone is passionate about it (in which case it would be great if they'd step in with concrete ideas). Cheers, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From yaniv at aknin.name Sun May 2 14:39:01 2010 From: yaniv at aknin.name (Yaniv Aknin) Date: Sun, 2 May 2010 22:39:01 +1000 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: <20100428112647.A2D631FF378@kimball.webabinitio.net> References: <4BD4A69A.4080507@tummy.com> <87fx2hk3ch.fsf@uwakimon.sk.tsukuba.ac.jp> <201004281046.37939.steve@pearwood.info> <87bpd4800f.fsf@uwakimon.sk.tsukuba.ac.jp> <20100428112647.A2D631FF378@kimball.webabinitio.net> Message-ID: >> Yes, in the last year in particular there has been some excellent effort >> of maintaining the issue tracker content. But the question still remains >> - who are we worried about offending? > The people who are potential new contributors but don't currently know > anyone in the Python community. By definition these 'potential new and unknown contributors' are discrete and probably can't be characterized as a whole. However, seeing myself as one of the discrete elements in that group, I think it's worthwhile for me to pipe in here and say that I won't be 'offended' or think of it as nepotism if someone gets foo-privilege before I do because he happens to know core developer (some other 'potential new contributer' lurking here feels otherwise? - speak up!). I don't know the community (yet) and I can't say this for sure, but my current gut feeling about the Python community (and pretty much any OSS I can think of) is that in the long run, I'll be judged on merit just like any other guy, no matter who they know. - Yaniv From ncoghlan at gmail.com Sun May 2 18:55:59 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 03 May 2010 02:55:59 +1000 Subject: [Python-Dev] PEP or issue recording rationale for allowing multiple context managers? Message-ID: <4BDDAE9F.2060105@gmail.com> I was looking for a reference for the addition of multiple context manager support to with statements in 3.1 and 2.7 and came up empty (aside from the initial python-ideas thread [1] that I linked to from PEP 377). I was hoping to find something that clearly spelled out: - the two major flaws in contextlib.nested* - the parallel with import statements for the precise chosen syntax - Guido's blessing to go ahead and do it *Those flaws being - that __init__/__new__ methods of inner context managers aren't covered by outer context managers - that outer context managers can't suppress exceptions from inner __enter__ methods Note that I'm not complaining about the decision itself (that would be silly, since I agree with the outcome), I'm just trying to find something to point to about it that is a little more concrete than a python-ideas thread. Cheers, Nick. [1] http://mail.python.org/pipermail/python-ideas/2009-March/003188.html -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From olemis at gmail.com Sun May 2 20:15:09 2010 From: olemis at gmail.com (Olemis Lang) Date: Sun, 2 May 2010 13:15:09 -0500 Subject: [Python-Dev] Alias for distutils.file_util.write_file in e.g. shutils Message-ID: Hello ! Often I have the contents to be written in a file at a given path that I know as well. I recently tried to find a function in stdlib to do that and to my surprise this is what I found : - Such function exists - It's `distutils.file_util.write_file` IMO the last place where people'd look for such a function is inside `distutils` package. Besides I reviewed modules listed under `File and directory access` category in `Library Reference` and found nothing even similar. Q: - Is it a good idea to provide a similar function in e.g. shutils module ? Probably there's already a better way to do this and my comment is just irrelevant . Anyway is just a suggestion ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Soporte para AMF (RPC) en Trac - http://feedproxy.google.com/~r/simelo-es/~3/9dYgHeK5Be8/soporte-para-amf-rpc-en-trac.html From ziade.tarek at gmail.com Sun May 2 20:36:55 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 2 May 2010 20:36:55 +0200 Subject: [Python-Dev] Alias for distutils.file_util.write_file in e.g. shutils In-Reply-To: References: Message-ID: On Sun, May 2, 2010 at 8:15 PM, Olemis Lang wrote: > Hello ! > > Often I have the contents to be written in a file at a given path that > I know as well. I recently tried to find a function in stdlib to do > that and to my surprise this is what I found : > > ?- Such function exists > ?- It's `distutils.file_util.write_file` > > IMO the last place where people'd look for such a function is inside > `distutils` package. Besides I reviewed modules listed under `File and > directory access` category in `Library Reference` and found nothing > even similar. > > Q: > ?- Is it a good idea to provide a similar function in > ? ?e.g. shutils module ? A: Yes :) Basically, anything useful in distutils.file_util and distutils.dir_util can maove in Shutil. That's why I added make_archive (and unpack_archive) Please add an issue, I'll work on adding this function, Thanks Tarek -- Tarek Ziad? | http://ziade.org From rdmurray at bitdance.com Sun May 2 21:48:22 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 02 May 2010 15:48:22 -0400 Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage. In-Reply-To: References: <4BD4A69A.4080507@tummy.com> <87fx2hk3ch.fsf@uwakimon.sk.tsukuba.ac.jp> <201004281046.37939.steve@pearwood.info> <87bpd4800f.fsf@uwakimon.sk.tsukuba.ac.jp> <20100428112647.A2D631FF378@kimball.webabinitio.net> Message-ID: <20100502194823.171A8200778@kimball.webabinitio.net> On Sun, 02 May 2010 22:39:01 +1000, Yaniv Aknin wrote: > >> Yes, in the last year in particular there has been some excellent effort > >> of maintaining the issue tracker content. But the question still remains > >> - who are we worried about offending? > > > The people who are potential new contributors but don't currently know > > anyone in the Python community. > > By definition these 'potential new and unknown contributors' are discrete > and probably can't be characterized as a whole. However, seeing myself > as one of the discrete elements in that group, I think it's worthwhile for me > to pipe in here and say that I won't be 'offended' or think of it as nepotism > if someone gets foo-privilege before I do because he happens to know core > developer (some other 'potential new contributer' lurking here feels otherwise? > - speak up!). > > I don't know the community (yet) and I can't say this for sure, but my current > gut feeling about the Python community (and pretty much any OSS I can think > of) is that in the long run, I'll be judged on merit just like any other guy, no > matter who they know. Thank you very much for this feedback. -- R. David Murray www.bitdance.com From rdmurray at bitdance.com Sun May 2 22:21:28 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 02 May 2010 16:21:28 -0400 Subject: [Python-Dev] Frequency of the dev docs autobuild In-Reply-To: <4BDCAEC6.1020202@v.loewis.de> References: <4BD95C6E.9080508@gmail.com> <20100429114047.C031C20059D@kimball.webabinitio.net> <20100501142857.3CAD51FC3F5@kimball.webabinitio.net> <20100501220504.D35AC200567@kimball.webabinitio.net> <4BDCAEC6.1020202@v.loewis.de> Message-ID: <20100502202128.CEEC3200395@kimball.webabinitio.net> On Sun, 02 May 2010 00:44:22 +0200, =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= wrote: > R. David Murray wrote: > > On Sat, 01 May 2010 16:18:19 +0100, Paul Moore wrote: > >> On 1 May 2010 15:28, R. David Murray wrote: > >>> Unless I'm missing something, I don't see any docs there about the > >>> automated build process and when and where it runs. > >> I assume Georg was referring to "Development versions of the Python > >> documentation are available on-line and updated daily". > > > > Yes, most likely. I think I am asking for more documentation than you > > were :) > > Then I think you have to propose specific wording. It runs on > www.python.org (why would you expect it to run elsewhere???), I had in fact overlooked the indicated sentence when reading the page. It's not that I expect it to run somewhere else, but that it did indeed used to run somewhere else (if I'm reading the build.sh script correctly), so I think it is worth making it explicit. Also note that it isn't obvious (without resolving the host names) that docs.python.org and www.python.org are the same machine. > and what about "daily" do you consider imprecise? Do you really need > exact hour, minute, and second? What for? I wasn't asking for more precision than daily (I just hadn't seen it), but now that I think about it it would indeed be nice to know when the cron job starts, so that we know that if the checkin didn't happen before then, it won't show up in the online docs until the next day. I don't think this is particularly *important* to know, it would just be nice to know. Is it only the development versions that get updated, or do our updates to the next bug fix release also get posted daily (and those are the docs web site visitors normally see, I believe)? I was under the impression that it was the latter, but that Georg had also changed things so that a reference to a particular dot release got the snapshot of the docs that were shipped with that dot release. This impression seems to be confirmed by examination of the various pages involved. To answer your question about what wording I'd like, I think that it would be worthwhile to say somewhere (not necessarily on that page...maybe in the doc README.txt?...but it could be ont that page...) that the docs are auto-built by a cron job on the server hosting docs.python.org running 'make dist' in the doc directory of a freshly updated checkout and then....doing something with the resulting files? Or maybe the apache URLs point right at the dist directory in that autobuild checkout? Anyway, exactly how it all works is what I would like to see documented. Background: one of my tasks for one of my customers is helping them try to make it so that an outsider coming in could learn everything they needed to know to operate the system from the available docs...a goal that they are nowhere near achieving, but which I think is a worthwhile goal for which to strive. -- R. David Murray www.bitdance.com From martin at v.loewis.de Sun May 2 22:56:59 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 02 May 2010 22:56:59 +0200 Subject: [Python-Dev] Frequency of the dev docs autobuild In-Reply-To: <20100502202128.CEEC3200395@kimball.webabinitio.net> References: <4BD95C6E.9080508@gmail.com> <20100429114047.C031C20059D@kimball.webabinitio.net> <20100501142857.3CAD51FC3F5@kimball.webabinitio.net> <20100501220504.D35AC200567@kimball.webabinitio.net> <4BDCAEC6.1020202@v.loewis.de> <20100502202128.CEEC3200395@kimball.webabinitio.net> Message-ID: <4BDDE71B.9050104@v.loewis.de> > I wasn't asking for more precision than daily (I just hadn't seen it), but > now that I think about it it would indeed be nice to know when the cron > job starts, so that we know that if the checkin didn't happen before then, > it won't show up in the online docs until the next day. I don't think > this is particularly *important* to know, it would just be nice to know. That's different from asking that it be documented, though. I don't mind you knowing (it's at 15:00 local time for the build machine, which sits in the Europe/Amsterdam timezone). Just ask specific questions, and people may give specific answers. Now that you know, I still don't think it needs to be documented (else: where would that end? Would you want to know account name and uid of the build also, and the partition of the hard disk where the files are stored? Not even I know the rack slot in which the machine sits). > > Is it only the development versions that get updated, or do our updates > to the next bug fix release also get posted daily (and those are the docs > web site visitors normally see, I believe)? That's what the documentation claims, yes. The build script currently has these targets: BRANCHES = [ # checkout, target, isdev (BUILDROOT + '/python26', WWWROOT, False), (BUILDROOT + '/python31', WWWROOT + '/py3k', False), (BUILDROOT + '/python27', WWWROOT + '/dev', True), (BUILDROOT + '/python32', WWWROOT + '/dev/py3k', True), ] > To answer your question about what wording I'd like, I think that it would > be worthwhile to say somewhere (not necessarily on that page...maybe in > the doc README.txt?...but it could be ont that page...) that the docs are > auto-built by a cron job on the server hosting docs.python.org running > 'make dist' in the doc directory of a freshly updated checkout and > then....doing something with the resulting files? Actually, the command is rather like this: 'cd Doc; make autobuild-%s' % (isdev and 'dev' or 'stable') > Background: one of my tasks for one of my customers is helping them try to > make it so that an outsider coming in could learn everything they needed > to know to operate the system from the available docs...a goal that they > are nowhere near achieving, but which I think is a worthwhile goal for > which to strive. For this kind of work, I think looking at the actual installation is more productive to learn how things are done (perhaps opposed to how they were originally planned to be done). I didn't know how this all worked myself, either, but, using the root account, it took me only a minute to find out - much faster than finding the documentation that may have explained it in detail. The only starting point that you need is the machine that you know it runs on. Regards, Martin From brett at python.org Sun May 2 23:24:47 2010 From: brett at python.org (Brett Cannon) Date: Sun, 2 May 2010 14:24:47 -0700 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> Message-ID: On Sat, May 1, 2010 at 02:00, Tarek Ziad? wrote: > On Fri, Apr 30, 2010 at 11:25 PM, Guido van Rossum > wrote: > [..] > >>> > >>> Without a BDFL, I think we need a committee to make decisions, e.g. by > >>> majority vote amongst committers. > >> > >> Couldn't we just go with the FLUFL? > > > > IIRC in the IETF this is done by the committee chair. I think it's a > > good idea to have this be a single person to avoid endless indecision. > > BPD = Benevolent Pep Dictator > BPC = Benevolent Pep Caliph (sounds good in French, not sure in English > ;) ) > > What about naming several BPD + and have one BPC elected each year by > all the core devs ? > > == BPD == > > I am not sure if this would work for all areas in Python-core, but > looking at the maintainers.rst file, it looks like we could name for > example Brett for all the import machinery things, Marc-Andr? for all > the unicode things, I could be the one about packaging, etc. > > If we could manage to split the python-core development into > categories, and add these categories in the PEP metadata, that would > define who takes the final decision in case we can't reach consensus. > This sounds like the lieutenant setup they have for the Linux kernel. I have no clue how that works out for them. > > == BPC == > > Of course some PEPs could concern several categories, so we would > still need some kind of Pep dictator if there's no consensus. So what > about electing a BPC every year ? > So there is a "single voice" issue here (that also applies to Martin's idea of having the release manager make the call as that is a rotating position). One of the reasons, I think, the BDFL style of decision making has worked out is that it lets Guido keep Python consistent; the language is always striving to meet his mental model of the language more and more. Having this rotate amongst us will not allow us to have this benefit. It also raises the chance of arguing over who takes over the rotating position and that just falls down into the hellish hole of politics that I don't think any of us want to see happen. But even ignoring my worry/point, what are we even discussing here? Guido has said multiple time he is not retiring, simply scaling back his involvement. So are we trying to figure out how to make our own decisions about Python when Guido is not available to make one or simply doesn't care enough to pronounce? If that's the case then we should probably choose a vice BDFL (sort of a Benevolent Dicatator at Guido's Discretion) to keep the voice of Python as uniform as possible. I guess this person would become the assumed successor of the BDFL title if Guido ever does retire and the decision is made to continue on with active development of the language instead of just going into maintenance mode, but hopefully that problem will be a long ways off. If we are discussing something else, then I don't know what we are all talking about here other than measurements of standard pieces of wood. =) -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Mon May 3 06:25:15 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 03 May 2010 06:25:15 +0200 Subject: [Python-Dev] Frequency of the dev docs autobuild In-Reply-To: <20100502202128.CEEC3200395@kimball.webabinitio.net> References: <4BD95C6E.9080508@gmail.com> <20100429114047.C031C20059D@kimball.webabinitio.net> <20100501142857.3CAD51FC3F5@kimball.webabinitio.net> <20100501220504.D35AC200567@kimball.webabinitio.net> <4BDCAEC6.1020202@v.loewis.de> <20100502202128.CEEC3200395@kimball.webabinitio.net> Message-ID: Am 02.05.2010 22:21, schrieb R. David Murray: > On Sun, 02 May 2010 00:44:22 +0200, =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= wrote: >> R. David Murray wrote: >> > On Sat, 01 May 2010 16:18:19 +0100, Paul Moore wrote: >> >> On 1 May 2010 15:28, R. David Murray wrote: >> >>> Unless I'm missing something, I don't see any docs there about the >> >>> automated build process and when and where it runs. >> >> I assume Georg was referring to "Development versions of the Python >> >> documentation are available on-line and updated daily". >> > >> > Yes, most likely. I think I am asking for more documentation than you >> > were :) >> >> Then I think you have to propose specific wording. It runs on >> www.python.org (why would you expect it to run elsewhere???), > > I had in fact overlooked the indicated sentence when reading the page. And I realize it's less than you wanted to see :) But I also realized it's very easy to give more detail without crowding that page; since the script used for the daily build is in SVN under Doc/tools/dailybuild.py; and that also contains the hostname. I've updated the dev/doc/ page to include a link to the script text. cheers, Georg From jcea at jcea.es Mon May 3 06:57:44 2010 From: jcea at jcea.es (Jesus Cea) Date: Mon, 03 May 2010 06:57:44 +0200 Subject: [Python-Dev] Should we drop active support of OSF/1? In-Reply-To: <4BD5F0D8.4020007@v.loewis.de> References: <4BD5D3F5.7090503@jcea.es> <4BD5F0D8.4020007@v.loewis.de> Message-ID: <4BDE57C8.5010509@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 26/04/10 22:00, "Martin v. L?wis" wrote: >> Maybe we can drop OSF/1 safely supporting Tru64 yet, but we don't have >> any buildbot running any of this systems... > > Dropping support is fine with me, in the long term. If PEP 11 is truly > followed, we should deprecate support in one version, and drop it in the > next one. This would mean we add a easy-to-remove configure error in > 3.2, and remove the support in 3.3. Would be enough to raise an "ERROR" at configure time if OSF test is positive?. To delete that intentional "ERROR" would be trivial. In 3.3 I would remove the configure tests and the "#if" conditional compilation related to them. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS95XyJlgi5GaxT1NAQLQLQP/cE2E27g8hSAbH7XzewDDYzx1AZ11ja55 xuR0PLTg/yQgE+4oifMa56Rs54YVRtRkh+i7B5yR5+lcI2DJgfqiu2Q9Of8KbwHp SQ2BVlshUmjhImWh486Qj6aQ9sF3xMdaxOGVLG+SJOBAEVw4UWEkZYPpQOuKRTfG K0SYYLhERoY= =CUNo -----END PGP SIGNATURE----- From regebro at gmail.com Mon May 3 08:19:54 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 3 May 2010 08:19:54 +0200 Subject: [Python-Dev] Should we drop active support of OSF/1? In-Reply-To: <4BDE57C8.5010509@jcea.es> References: <4BD5D3F5.7090503@jcea.es> <4BD5F0D8.4020007@v.loewis.de> <4BDE57C8.5010509@jcea.es> Message-ID: On Mon, May 3, 2010 at 06:57, Jesus Cea wrote: > Would be enough to raise an "ERROR" at configure time if OSF test is > positive?. To delete that intentional "ERROR" would be trivial. Oh really? I have no clue of how to do that. Doesn't like like a good deprecation to me. :) Is printing a warning not easily feasible in ./configure? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From solipsis at pitrou.net Mon May 3 13:53:29 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 May 2010 11:53:29 +0000 (UTC) Subject: [Python-Dev] Should we drop active support of OSF/1? References: <4BD5D3F5.7090503@jcea.es> <4BD5F0D8.4020007@v.loewis.de> <4BDE57C8.5010509@jcea.es> Message-ID: Jesus Cea jcea.es> writes: > > Would be enough to raise an "ERROR" at configure time if OSF test is > positive?. To delete that intentional "ERROR" would be trivial. I think it's ok. Noone will probably notice anyway. Regards Antoine. From olemis at gmail.com Mon May 3 14:47:33 2010 From: olemis at gmail.com (Olemis Lang) Date: Mon, 3 May 2010 07:47:33 -0500 Subject: [Python-Dev] Alias for distutils.file_util.write_file in e.g. shutils In-Reply-To: References: Message-ID: On Sun, May 2, 2010 at 1:36 PM, Tarek Ziad? wrote: > a similar function in >> ? ?e.g. shutils module ? > > A: Yes :) > > Basically, anything useful in distutils.file_util and > distutils.dir_util can maove in Shutil. > That's why I added make_archive (and unpack_archive) > > Please add an issue, I'll work on adding this function, > http://bugs.python.org/issue8604 -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: From barry at python.org Mon May 3 17:30:22 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 3 May 2010 11:30:22 -0400 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> Message-ID: <20100503113022.59a2e609@heresy> On May 01, 2010, at 08:58 AM, Dirkjan Ochtman wrote: >On Fri, Apr 30, 2010 at 21:09, Barry Warsaw wrote: >>>Though maybe it should be called Conclusion instead of Accepted and >>>used for Rejected PEPs, as well? >> >> Good point. ?What do you think about 'Resolution'? > >Fine with me. I've updated PEP 1 and the supporting scripts to accept a Resolution: header. Note that in PEP 1 I wrote: *Note: The Resolution header is required for Standards Track PEPs only. It contains a URL that should point to an email message or other web resource where the pronouncement about the PEP is made.* but in fact, the scripts make Resolution optional (it's kind of a pain to make it required just for Standards Track PEPs - contributions welcome). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From fdrake at acm.org Mon May 3 17:38:31 2010 From: fdrake at acm.org (Fred Drake) Date: Mon, 3 May 2010 11:38:31 -0400 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <20100503113022.59a2e609@heresy> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <20100503113022.59a2e609@heresy> Message-ID: On Mon, May 3, 2010 at 11:30 AM, Barry Warsaw wrote: > but in fact, the scripts make Resolution optional (it's kind of a pain to make > it required just for Standards Track PEPs - contributions welcome). It will also be a pain to retroactively update older PEPs with the newly-required metadata; leaving it optional (in practice) seems acceptable. It should only apply to newly-resolved PEPs, which is also likely a pain to build into the script. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From barry at python.org Mon May 3 17:44:13 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 3 May 2010 11:44:13 -0400 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <4BDB3D6C.4050909@holdenweb.com> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> Message-ID: <20100503114413.5b5e9e3e@heresy> On Apr 30, 2010, at 04:28 PM, Steve Holden wrote: >Martin v. L?wis wrote: >>> As to Guido's point about the decision making process, Nick's right. I just >>> want to make sure we can capture the resolution in the PEP, be it by BDFL >>> pronouncement or "hey, silence is acceptance" email. >> >> I don't think "silence is acceptance" will work out in practice. For >> issues where a PEP was written in the first place, somebody will >> *always* object, and forever so, hoping that what he considers a mistake >> will not be done. The advantage of Guido acting as BDFL was that >> somebody would make an decision ultimately, one which both proponents >> and opponents of the PEP would accept. >> >> Without a BDFL, I think we need a committee to make decisions, e.g. by >> majority vote amongst committers. > >Couldn't we just go with the FLUFL? We could of course, and I would serve honorably . But Guido's not *that* much older than me, so you'd probably only get a few useful years out of me before I start getting increasingly eccentric, re-opening and approving PEPs like 313, 336, and 666. In the meantime, let's groom Benjamin to be the Sacred Next Uncle Galvanizing the Gamut of Language Evolution. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Mon May 3 17:57:52 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 3 May 2010 11:57:52 -0400 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <4BDBB846.5020305@v.loewis.de> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> <4BDBB846.5020305@v.loewis.de> Message-ID: <20100503115752.4f19f2e3@heresy> On May 01, 2010, at 07:12 AM, Martin v. L?wis wrote: >> IIRC in the IETF this is done by the committee chair. I think it's a >> good idea to have this be a single person to avoid endless indecision. > >It then seems that this role should go to the release manager of the >upcoming feature release. Assuming Georg can accept this additional >responsibility. I do think it makes sense for the RM to assume these responsibilities where Guido either can't or doesn't want to make the final decision. I think it will fairly substantially increase the workload on the RM, so perhaps there are ways to off-load some of the current responsibilities (e.g. updating the website for each release). I also think that RMs should be term-limited so that we can spread more experience within the community. And past-RMs can provide a sort of consultation group where contentious decisions can be discussed and advice gathered. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From guido at python.org Mon May 3 18:40:44 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 3 May 2010 09:40:44 -0700 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <20100503115752.4f19f2e3@heresy> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> <4BDBB846.5020305@v.loewis.de> <20100503115752.4f19f2e3@heresy> Message-ID: On Mon, May 3, 2010 at 8:57 AM, Barry Warsaw wrote: > On May 01, 2010, at 07:12 AM, Martin v. L?wis wrote: > >>> IIRC in the IETF this is done by the committee chair. I think it's a >>> good idea to have this be a single person to avoid endless indecision. >> >>It then seems that this role should go to the release manager of the >>upcoming feature release. Assuming Georg can accept this additional >>responsibility. > > I do think it makes sense for the RM to assume these responsibilities where > Guido either can't or doesn't want to make the final decision. ?I think it > will fairly substantially increase the workload on the RM, so perhaps there > are ways to off-load some of the current responsibilities (e.g. updating the > website for each release). ?I also think that RMs should be term-limited so > that we can spread more experience within the community. ?And past-RMs can > provide a sort of consultation group where contentious decisions can be > discussed and advice gathered. While I certainly won't object if a release manager volunteers to also be the final PEP arbiter, I don't want to make it a job requirement (or even an implied expectation). The responsibility of a release manager is very much in the here and now and the near future -- stability and consistency of the current release. Being PEP arbiter requires a somewhat different mindset, more focused on the long-term health of the language and its community. -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Mon May 3 21:40:02 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 03 May 2010 21:40:02 +0200 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <20100503114413.5b5e9e3e@heresy> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> <20100503114413.5b5e9e3e@heresy> Message-ID: <4BDF2692.3040006@v.loewis.de> > In the meantime, let's groom Benjamin to be the Sacred Next Uncle Galvanizing > the Gamut of Language Evolution. I don't think anybody having such a position permanently can really work. Regards, Martin From martin at v.loewis.de Mon May 3 21:55:42 2010 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 03 May 2010 21:55:42 +0200 Subject: [Python-Dev] Should we drop active support of OSF/1? In-Reply-To: <4BDE57C8.5010509@jcea.es> References: <4BD5D3F5.7090503@jcea.es> <4BD5F0D8.4020007@v.loewis.de> <4BDE57C8.5010509@jcea.es> Message-ID: <4BDF2A3E.2030609@v.loewis.de> Jesus Cea wrote: > On 26/04/10 22:00, "Martin v. L?wis" wrote: >>> Maybe we can drop OSF/1 safely supporting Tru64 yet, but we don't have >>> any buildbot running any of this systems... >> Dropping support is fine with me, in the long term. If PEP 11 is truly >> followed, we should deprecate support in one version, and drop it in the >> next one. This would mean we add a easy-to-remove configure error in >> 3.2, and remove the support in 3.3. > > Would be enough to raise an "ERROR" at configure time if OSF test is > positive?. To delete that intentional "ERROR" would be trivial. That's actually what the PEP suggests (or means to suggest). Users willing to take over maintenance would, at a minimum, know how to remove that error. If a user doesn't know how to remove it, they can ask for help, but more importantly, they probably consider it already unsupported - which is exactly the desired effect. Just to rephrase: the point of this deprecation cycle is to give users a chance to take over maintenance, and to declare, in public, that they are interested in that port. If nobody steps forward, we can safely assume that nobody cares. Regards, Martin From benjamin at python.org Mon May 3 22:58:46 2010 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 3 May 2010 15:58:46 -0500 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> <4BDBB846.5020305@v.loewis.de> <20100503115752.4f19f2e3@heresy> Message-ID: 2010/5/3 Guido van Rossum : > On Mon, May 3, 2010 at 8:57 AM, Barry Warsaw wrote: >> I do think it makes sense for the RM to assume these responsibilities where >> Guido either can't or doesn't want to make the final decision. I think it >> will fairly substantially increase the workload on the RM, so perhaps there >> are ways to off-load some of the current responsibilities (e.g. updating the >> website for each release). I also think that RMs should be term-limited so >> that we can spread more experience within the community. And past-RMs can >> provide a sort of consultation group where contentious decisions can be >> discussed and advice gathered. I'm very -1 on the release manager making decisions on PEPs. The RM right now basically makes a release schedule and cuts releases. The little power he has is mostly used for rejecting features in the beta period. This task should not be politicized. -- Regards, Benjamin From steve at holdenweb.com Mon May 3 23:38:12 2010 From: steve at holdenweb.com (Steve Holden) Date: Mon, 03 May 2010 17:38:12 -0400 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> <4BDBB846.5020305@v.loewis.de> <20100503115752.4f19f2e3@heresy> Message-ID: <4BDF4244.6060605@holdenweb.com> Benjamin Peterson wrote: > 2010/5/3 Guido van Rossum : >> On Mon, May 3, 2010 at 8:57 AM, Barry Warsaw wrote: >>> I do think it makes sense for the RM to assume these responsibilities where >>> Guido either can't or doesn't want to make the final decision. I think it >>> will fairly substantially increase the workload on the RM, so perhaps there >>> are ways to off-load some of the current responsibilities (e.g. updating the >>> website for each release). I also think that RMs should be term-limited so >>> that we can spread more experience within the community. And past-RMs can >>> provide a sort of consultation group where contentious decisions can be >>> discussed and advice gathered. > > I'm very -1 on the release manager making decisions on PEPs. The RM > right now basically makes a release schedule and cuts releases. The > little power he has is mostly used for rejecting features in the beta > period. This task should not be politicized. > That seems reasonable, but it also seems reasonable to try and simplify the task of updating the web site for each release. There's plenty to do without having to fight that issue too. I'm copying this message to Rich Leland, who is currently making a study of the PSF's web requirements, to make sure this gets folded in. Many of the tasks are essentially macrogeneration, and unless automated will remain error-prone. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From jcea at jcea.es Mon May 3 23:39:45 2010 From: jcea at jcea.es (Jesus Cea) Date: Mon, 03 May 2010 23:39:45 +0200 Subject: [Python-Dev] Should we drop active support of OSF/1? In-Reply-To: <4BDF2A3E.2030609@v.loewis.de> References: <4BD5D3F5.7090503@jcea.es> <4BD5F0D8.4020007@v.loewis.de> <4BDE57C8.5010509@jcea.es> <4BDF2A3E.2030609@v.loewis.de> Message-ID: <4BDF42A1.1060903@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/05/10 21:55, "Martin v. L?wis" wrote: > Just to rephrase: the point of this deprecation cycle is to give users a > chance to take over maintenance, and to declare, in public, that they > are interested in that port. If nobody steps forward, we can safely > assume that nobody cares. http://bugs.python.org/issue8606 . I will take care of the final cleanup in 3.3, if nobody stands up to keep OSF* alive. I am deprecating OSF*. If it is "too much", we can change it to "OSF1". - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS99CoJlgi5GaxT1NAQLSygQAiYD0UQAKl2vv1alwmMSP02AgHxN9e3Le bfMqysr2BE9fnG3WQlUKFn6pjbnnANFU2HRhTUJkaCNslewnUeg6pQ/s0xu321GL HQ6v6JHSOXxRoyyZ21Wq2ECNg+tYWGkM7Eohsif8XMjTeL0lP7ddYZSFaJbK0PfM BSK6DAV93/o= =nlYU -----END PGP SIGNATURE----- From brett at python.org Tue May 4 00:37:22 2010 From: brett at python.org (Brett Cannon) Date: Mon, 3 May 2010 15:37:22 -0700 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base Message-ID: Since 2.7 is probably going to exist for a while, I am running Clang 2.7's static analyzer (``clang --static``) over trunk. It's mostly just finding stuff like unneeded variable initialization or variables that are never used (compilation is picking up unused returned values, almost all from PyObject_INIT). When I check in these changes I will do it file by file, but my question is how to handle Misc/NEWS. I have gone through the underscores and the 'a's in Modules and already have six modified files, so the final count might be a little high. Do people want individual entries per file, or simply a single entry that lists each file modified? We should probably go through the C code and fix the whitespace before we hit 2.7 final (there is a ton of lines with extraneous spaces). -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Tue May 4 00:47:48 2010 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 3 May 2010 17:47:48 -0500 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: References: Message-ID: 2010/5/3 Brett Cannon : > Since 2.7 is probably going to exist for a while, I am running Clang 2.7's > static analyzer (``clang --static``) over trunk. It's mostly just finding > stuff like unneeded variable initialization or variables that are never used > (compilation is picking up unused returned values, almost all from > PyObject_INIT). > When I check in these changes I will do it file by file, but my question is > how to handle Misc/NEWS. I have gone through the underscores and the 'a's in > Modules and already have six modified files, so the final count might be a > little high. Do people want individual entries per file, or simply a single > entry that lists each file modified? Do these changes even warrant a NEWS entry? NEWS is more for user facing changes. > We should probably go through the C code and fix the whitespace before we > hit 2.7 final (there is a ton of lines with extraneous spaces). Why? > -Brett -- Regards, Benjamin From solipsis at pitrou.net Tue May 4 01:00:02 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 May 2010 23:00:02 +0000 (UTC) Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base References: Message-ID: Benjamin Peterson python.org> writes: > > 2010/5/3 Brett Cannon python.org>: > > When I check in these changes I will do it file by file, but my question is > > how to handle Misc/NEWS. I have gone through the underscores and the 'a's in > > Modules and already have six modified files, so the final count might be a > > little high. Do people want individual entries per file, or simply a single > > entry that lists each file modified? > > Do these changes even warrant a NEWS entry? NEWS is more for user > facing changes. Indeed. At most a single NEWS entry sounds sufficient. Regards Antoine. From brett at python.org Tue May 4 01:11:53 2010 From: brett at python.org (Brett Cannon) Date: Mon, 3 May 2010 16:11:53 -0700 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: References: Message-ID: I'll just do a single entry saying that the static analyzer was used and not list the files. And in reply to Benjamin's question about the whitespace, I guess it actually doesn't matter. More important to clean up in py3k. On May 3, 2010 4:00 PM, "Antoine Pitrou" wrote: Benjamin Peterson python.org> writes: > > 2010/5/3 Brett Cannon python.org>: > > When I check in these changes I will do it file by file, but my question is > > how to handle M... Indeed. At most a single NEWS entry sounds sufficient. Regards Antoine. _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue May 4 01:24:37 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 May 2010 23:24:37 +0000 (UTC) Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base References: Message-ID: Le Mon, 03 May 2010 16:11:53 -0700, Brett Cannon a ?crit?: > > And in reply to Benjamin's question about the whitespace, I guess it > actually doesn't matter. More important to clean up in py3k. Would it be finally time to standardize all C files on a 4-spaces indentation rule? I understand the "svn annotate" argument, but we edit code far more often than we annotate it. Also, it seems "svn ann -x -w" would ignore whitespace changes. Regards Antoine. From steve at holdenweb.com Tue May 4 04:12:25 2010 From: steve at holdenweb.com (Steve Holden) Date: Mon, 03 May 2010 22:12:25 -0400 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: References: Message-ID: <4BDF8289.2090306@holdenweb.com> Antoine Pitrou wrote: > Le Mon, 03 May 2010 16:11:53 -0700, Brett Cannon a ?crit : >> And in reply to Benjamin's question about the whitespace, I guess it >> actually doesn't matter. More important to clean up in py3k. > > Would it be finally time to standardize all C files on a 4-spaces > indentation rule? > > I understand the "svn annotate" argument, but we edit code far more often > than we annotate it. Also, it seems "svn ann -x -w" would ignore > whitespace changes. > Let's not forget to consider what Hg can do, since that represents the future. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From barry at python.org Tue May 4 04:34:13 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 3 May 2010 22:34:13 -0400 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: <4BDF8289.2090306@holdenweb.com> References: <4BDF8289.2090306@holdenweb.com> Message-ID: <20100503223413.17437de7@heresy> On May 03, 2010, at 10:12 PM, Steve Holden wrote: >Antoine Pitrou wrote: >> Le Mon, 03 May 2010 16:11:53 -0700, Brett Cannon a ?crit : >>> And in reply to Benjamin's question about the whitespace, I guess it >>> actually doesn't matter. More important to clean up in py3k. >> >> Would it be finally time to standardize all C files on a 4-spaces >> indentation rule? >> >> I understand the "svn annotate" argument, but we edit code far more often >> than we annotate it. Also, it seems "svn ann -x -w" would ignore >> whitespace changes. >> >Let's not forget to consider what Hg can do, since that represents the >future. Now would be a good time to convert the C files to 4 space indents. We've only been talking about it for a decade at least. While I'm sure history can be preserved across the svn->hg import, it's enough of a break to bite the bullet and fix the code. I think we only need to convert the py3k branch though. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From alexandre at peadrop.com Tue May 4 06:00:47 2010 From: alexandre at peadrop.com (Alexandre Vassalotti) Date: Mon, 3 May 2010 21:00:47 -0700 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: <20100503223413.17437de7@heresy> References: <4BDF8289.2090306@holdenweb.com> <20100503223413.17437de7@heresy> Message-ID: On Mon, May 3, 2010 at 7:34 PM, Barry Warsaw wrote: > Now would be a good time to convert the C files to 4 space indents. ?We've > only been talking about it for a decade at least. Will changing the indentation of source files to 4 space indents break patches on the bug tracker? -- Alexandre From martin at v.loewis.de Tue May 4 07:12:32 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 04 May 2010 07:12:32 +0200 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: References: <4BDF8289.2090306@holdenweb.com> <20100503223413.17437de7@heresy> Message-ID: <4BDFACC0.2010808@v.loewis.de> > Will changing the indentation of source files to 4 space indents break > patches on the bug tracker? Plain patch will choke, but "patch -l" might accept them. I have only read the documentation, though, and don't know whether it does in practice. Regards, Martin From martin at v.loewis.de Tue May 4 07:16:22 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 04 May 2010 07:16:22 +0200 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: <4BDF8289.2090306@holdenweb.com> References: <4BDF8289.2090306@holdenweb.com> Message-ID: <4BDFADA6.4040703@v.loewis.de> Steve Holden wrote: > Antoine Pitrou wrote: >> Le Mon, 03 May 2010 16:11:53 -0700, Brett Cannon a ?crit : >>> And in reply to Benjamin's question about the whitespace, I guess it >>> actually doesn't matter. More important to clean up in py3k. >> Would it be finally time to standardize all C files on a 4-spaces >> indentation rule? >> >> I understand the "svn annotate" argument, but we edit code far more often >> than we annotate it. Also, it seems "svn ann -x -w" would ignore >> whitespace changes. >> > Let's not forget to consider what Hg can do, since that represents the > future. I think Mercurial chokes in the light of white space inconsistencies just as much as Subversion. One reason for not converting in the past was also that it would break merging, unless that whitespace normalization is applied to all these branches. Regards, Martin From martin at v.loewis.de Tue May 4 08:50:11 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 04 May 2010 08:50:11 +0200 Subject: [Python-Dev] Daily DMGs Message-ID: <4BDFC3A3.4000205@v.loewis.de> David Bolen and I started producing daily OSX installers, at http://www.python.org/dev/daily-dmg/ If you find problems with these, please report them to bugs.python.org. Regards, Martin From victor.stinner at haypocalc.com Tue May 4 10:10:33 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Tue, 4 May 2010 10:10:33 +0200 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: <20100503223413.17437de7@heresy> References: <4BDF8289.2090306@holdenweb.com> <20100503223413.17437de7@heresy> Message-ID: <201005041010.33228.victor.stinner@haypocalc.com> Le mardi 04 mai 2010 04:34:13, Barry Warsaw a ?crit : > Now would be a good time to convert the C files to 4 space indents. (...) > I think we only need to convert the py3k branch though. It will make the port of patches (commits) between trunk and py3k much harder. Can you explain why do you want to only fix only py3k and not trunk? -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Tue May 4 10:14:19 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Tue, 4 May 2010 10:14:19 +0200 Subject: [Python-Dev] =?utf-8?q?Running_Clang_2=2E7=27s_static_analyzer_ov?= =?utf-8?q?er_the_code=09base?= In-Reply-To: References: Message-ID: <201005041014.19399.victor.stinner@haypocalc.com> Le mardi 04 mai 2010 01:24:37, Antoine Pitrou a ?crit : > Would it be finally time to standardize all C files on a 4-spaces > indentation rule? Indentation is painful on some files mixing tabs and spaces (sometimes in the same function!). > I understand the "svn annotate" argument, but we edit code far more often > than we annotate it. Also, it seems "svn ann -x -w" would ignore > whitespace changes. I use svn blame to find the commit which insered a line of code. When I get the commit number, I always read the commit to ensure that the patch really insered the line, and it's not a refactoring patch. Refactoring can be about the indentation, but there are a lot of other minor changes: op->ob_size => Py_SIZE(op), (a) => a, { with a new line => { without newline, ... -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Tue May 4 10:37:01 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Tue, 4 May 2010 10:37:01 +0200 Subject: [Python-Dev] =?utf-8?q?Running_Clang_2=2E7=27s_static_analyzer_ov?= =?utf-8?q?er_the_code=09base?= In-Reply-To: References: Message-ID: <201005041037.01778.victor.stinner@haypocalc.com> Le mardi 04 mai 2010 01:24:37, Antoine Pitrou a ?crit : > Le Mon, 03 May 2010 16:11:53 -0700, Brett Cannon a ?crit : > > And in reply to Benjamin's question about the whitespace, I guess it > > actually doesn't matter. More important to clean up in py3k. > > Would it be finally time to standardize all C files on a 4-spaces > indentation rule? Oh, if we decide to reindent all C files, would it be also possible to strip trailing spaces? -- Victor Stinner http://www.haypocalc.com/ From barry at python.org Tue May 4 14:23:31 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 4 May 2010 08:23:31 -0400 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: <4BDFADA6.4040703@v.loewis.de> References: <4BDF8289.2090306@holdenweb.com> <4BDFADA6.4040703@v.loewis.de> Message-ID: <20100504082331.2c7a6062@heresy> On May 04, 2010, at 07:16 AM, Martin v. L?wis wrote: >I think Mercurial chokes in the light of white space inconsistencies >just as much as Subversion. One reason for not converting in the past >was also that it would break merging, unless that whitespace >normalization is applied to all these branches. Then let's do py3k and release31-maint, or whatever they will be called under hg. Once 2.7 is released and we're on hg, how much back porting of revisions from Python 3 -> 2 is going to happen? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From solipsis at pitrou.net Tue May 4 14:41:42 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 4 May 2010 12:41:42 +0000 (UTC) Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base References: <4BDF8289.2090306@holdenweb.com> <4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy> Message-ID: Barry Warsaw python.org> writes: > > Then let's do py3k and release31-maint, or whatever they will be called under > hg. Once 2.7 is released and we're on hg, how much back porting of revisions > from Python 3 -> 2 is going to happen? Probably quite a bit still; all C extension bug fixes for example. I think we should reindent all 3 branches. Most of the work can probably be scripted (str.replace("\t", " " * 4)), and then a visual pass is necessary to fix vertical alignments and the like. Regards Antoine. From dirkjan at ochtman.nl Tue May 4 15:09:33 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Tue, 4 May 2010 15:09:33 +0200 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: References: <4BDF8289.2090306@holdenweb.com> <4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy> Message-ID: On Tue, May 4, 2010 at 14:41, Antoine Pitrou wrote: > I think we should reindent all 3 branches. Most of the work can probably be > scripted (str.replace("\t", " " * 4)), and then a visual pass is necessary to > fix vertical alignments and the like. If the script is robust enough, I can run it as part of the conversion, making sure that all branches get cleaned up... Cheers, Dirkjan From eric at trueblade.com Tue May 4 16:37:36 2010 From: eric at trueblade.com (Eric Smith) Date: Tue, 04 May 2010 10:37:36 -0400 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: References: <4BDF8289.2090306@holdenweb.com> <4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy> Message-ID: <4BE03130.9020700@trueblade.com> Dirkjan Ochtman wrote: > On Tue, May 4, 2010 at 14:41, Antoine Pitrou wrote: >> I think we should reindent all 3 branches. Most of the work can probably be >> scripted (str.replace("\t", " " * 4)), and then a visual pass is necessary to >> fix vertical alignments and the like. > > If the script is robust enough, I can run it as part of the > conversion, making sure that all branches get cleaned up... Could this be done as part of the conversion without affecting the history? -- Eric. From victor.stinner at haypocalc.com Tue May 4 17:26:22 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Tue, 4 May 2010 17:26:22 +0200 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: References: <20100504082331.2c7a6062@heresy> Message-ID: <201005041726.22300.victor.stinner@haypocalc.com> Le mardi 04 mai 2010 14:41:42, Antoine Pitrou a ?crit : > Barry Warsaw python.org> writes: > > Then let's do py3k and release31-maint, or whatever they will be called > > under hg. Once 2.7 is released and we're on hg, how much back porting of > > revisions from Python 3 -> 2 is going to happen? > > Probably quite a bit still; all C extension bug fixes for example. 2.7 will be maintained for long time, longer than any other 2.x version. > I think we should reindent all 3 branches. Most of the work can probably be > scripted (str.replace("\t", " " * 4)), and then a visual pass is necessary > to fix vertical alignments and the like. We should also add pre-commit scripts to avoid the reintroduction of tabulations in C (and Python?) files. -- Victor Stinner http://www.haypocalc.com/ From zvezdan at zope.com Tue May 4 17:27:58 2010 From: zvezdan at zope.com (Zvezdan Petkovic) Date: Tue, 4 May 2010 11:27:58 -0400 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: References: <4BDF8289.2090306@holdenweb.com> <4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy> Message-ID: On May 4, 2010, at 8:41 AM, Antoine Pitrou wrote: > I think we should reindent all 3 branches. Most of the work can probably be scripted (str.replace("\t", " " * 4)), and then a visual pass is necessary to fix vertical alignments and the like. > > I think we should reindent all 3 branches. Most of the work can probably be scripted (str.replace("\t", " " * 4)), and then a visual pass is necessary to fix vertical alignments and the like. Is it really that simple? People often do the following: - use tabs inside strings ("\t" should be used instead); - use tabs for alignment of inline comments on the right side; - mix tabs and spaces for indentation whitespace (continuation lines); - use tabs for alignment inside comments. A simple replacement of a tab with 4 spaces as you propose does not work on such a code. To do it properly, you may end up reinventing `indent` (UNIX program). What if there was a set of `indent` flags defined for PEP-7? It would be a nice tool to check whether a new code complies. However, running `indent` on an existing code would probably cause non-whitespace changes too as it rearranges the comments and code. I'm afraid that any other kind of "script + visual pass + post-edit" would incur similar changes if done properly (only more tedious). So, the question is what bothers developers more: - old C files with tab indentation, or - a lot of changes in version control to fix them? Both? :-) Zvezdan From brett at python.org Tue May 4 22:16:46 2010 From: brett at python.org (Brett Cannon) Date: Tue, 4 May 2010 13:16:46 -0700 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: References: <4BDF8289.2090306@holdenweb.com> <4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy> Message-ID: On Tue, May 4, 2010 at 08:27, Zvezdan Petkovic wrote: > On May 4, 2010, at 8:41 AM, Antoine Pitrou wrote: > > I think we should reindent all 3 branches. Most of the work can probably > be scripted (str.replace("\t", " " * 4)), and then a visual pass is > necessary to fix vertical alignments and the like. > > > > > I think we should reindent all 3 branches. Most of the work can probably > be scripted (str.replace("\t", " " * 4)), and then a visual pass is > necessary to fix vertical alignments and the like. > > Is it really that simple? > > People often do the following: > > - use tabs inside strings ("\t" should be used instead); > - use tabs for alignment of inline comments on the right side; > - mix tabs and spaces for indentation whitespace (continuation lines); > - use tabs for alignment inside comments. > > A simple replacement of a tab with 4 spaces as you propose does not work > on such a code. > > To do it properly, you may end up reinventing `indent` (UNIX program). > > What if there was a set of `indent` flags defined for PEP-7? > It would be a nice tool to check whether a new code complies. > `make patchcheck` already does this for Python code. Adding to the command to handle C code wouldn't be hard. -Brett > > However, running `indent` on an existing code would probably cause > non-whitespace changes too as it rearranges the comments and code. > > I'm afraid that any other kind of "script + visual pass + post-edit" would > incur similar changes if done properly (only more tedious). > > So, the question is what bothers developers more: > > - old C files with tab indentation, or > - a lot of changes in version control to fix them? > > Both? > :-) > > Zvezdan > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Tue May 4 22:46:51 2010 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 4 May 2010 15:46:51 -0500 Subject: [Python-Dev] slight release schedule change Message-ID: Hi, I'm sorry I neglected to mention this before. I'm delaying all 2.7 releases a week because of beta 1's lateness. I've now updated PEP 373. So, beta 2 will be this Saturday. -- Regards, Benjamin From solipsis at pitrou.net Wed May 5 00:02:06 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 4 May 2010 22:02:06 +0000 (UTC) Subject: [Python-Dev] reindenting References: <4BDF8289.2090306@holdenweb.com> <4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy> Message-ID: Le Tue, 04 May 2010 11:27:58 -0400, Zvezdan Petkovic a ?crit?: > > A simple replacement of a tab with 4 spaces as you propose does not work > on such a code. Right, I was simplifying a bit. I've just written a script which does a slightly more elaborate reindentation of C files, but still with the main objective of replacing tabs with 4-spaces indents. It shows good results on most of the source tree. In particular, it handles continuation lines and keeps vertical alignment correct in the majority of cases. This script could probably be applied with little a posteriori manual intervention. It can be found in trunk/sandbox/untabify/untabify.py. Regards Antoine. From ncoghlan at gmail.com Wed May 5 00:32:24 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 05 May 2010 08:32:24 +1000 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: <201005041726.22300.victor.stinner@haypocalc.com> References: <20100504082331.2c7a6062@heresy> <201005041726.22300.victor.stinner@haypocalc.com> Message-ID: <4BE0A078.90505@gmail.com> Victor Stinner wrote: > We should also add pre-commit scripts to avoid the reintroduction of > tabulations in C (and Python?) files. Python and ReST files are already covered (with reindent.py and reindent-rst.py to fix any files that get mixed up). "make patchcheck" includes the precommit checks for both of those file types. It's only C files that haven't been fixed yet (because their whitespace is such a mess overall, whereas the others were already pretty clean even before the precommit hooks were added). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed May 5 00:34:44 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 05 May 2010 08:34:44 +1000 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: References: <4BDF8289.2090306@holdenweb.com> <4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy> Message-ID: <4BE0A104.1020008@gmail.com> Zvezdan Petkovic wrote: > So, the question is what bothers developers more: > > - old C files with tab indentation, or > - a lot of changes in version control to fix them? > > Both? C files that mix tabs and spaces are actually the main source of pain when editing :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed May 5 00:41:04 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 05 May 2010 08:41:04 +1000 Subject: [Python-Dev] Standardising C file indentation (was Re: Running Clang 2.7's static analyzer over the code base) In-Reply-To: <4BE03130.9020700@trueblade.com> References: <4BDF8289.2090306@holdenweb.com> <4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy> <4BE03130.9020700@trueblade.com> Message-ID: <4BE0A280.8080903@gmail.com> Eric Smith wrote: > Dirkjan Ochtman wrote: >> On Tue, May 4, 2010 at 14:41, Antoine Pitrou wrote: >>> I think we should reindent all 3 branches. Most of the work can >>> probably be >>> scripted (str.replace("\t", " " * 4)), and then a visual pass is >>> necessary to >>> fix vertical alignments and the like. >> >> If the script is robust enough, I can run it as part of the >> conversion, making sure that all branches get cleaned up... > > Could this be done as part of the conversion without affecting the history? I don't think that would be a good idea - it's just whitespace, but I think we risk more problems by leaving it out of the history than we do by preserving. Howver, if we delay fixing the C file indentation until we're already on hg, the merge tools should offer the best chance of being able to apply pre-fix patches and have the software figure out where the whitespace changes need to be accounted for. If we do it before, or don't record the change in the history at all, then hg won't have any changeset information to work with and will almost certainly get just as confused by the whitespace changes as the basic "patch" command would. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Wed May 5 00:57:20 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 05 May 2010 00:57:20 +0200 Subject: [Python-Dev] Standardising C file indentation (was Re: Running Clang 2.7's static analyzer over the code base) In-Reply-To: <4BE0A280.8080903@gmail.com> References: <4BDF8289.2090306@holdenweb.com> <4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy> <4BE03130.9020700@trueblade.com> <4BE0A280.8080903@gmail.com> Message-ID: <1273013841.3688.9.camel@localhost.localdomain> Le mercredi 05 mai 2010 ? 08:41 +1000, Nick Coghlan a ?crit : > > Howver, if we delay fixing the C file indentation until we're already on > hg, the merge tools should offer the best chance of being able to apply > pre-fix patches and have the software figure out where the whitespace > changes need to be accounted for. I just tried using "patch -l" after having changed the indentation of a C source file and it worked fine. The only issue being that the patched lines of code have the wrong indentation (they get their indentation from the patch, not from the input file), but it's probably unavoidable. Regards Antoine. From victor.stinner at haypocalc.com Wed May 5 01:10:19 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 5 May 2010 01:10:19 +0200 Subject: [Python-Dev] Standardising C file indentation (was Re: Running Clang 2.7's static analyzer over the code base) In-Reply-To: <4BE0A280.8080903@gmail.com> References: <4BE03130.9020700@trueblade.com> <4BE0A280.8080903@gmail.com> Message-ID: <201005050110.19887.victor.stinner@haypocalc.com> Le mercredi 05 mai 2010 00:41:04, Nick Coghlan a ?crit : > Howver, if we delay fixing the C file indentation until we're already on > hg, the merge tools should offer the best chance of being able to apply > pre-fix patches and have the software figure out where the whitespace > changes need to be accounted for. If we wait for Hg to solve all of our problems, the migration from svn to hg will be harder and longer. -- Victor Stinner http://www.haypocalc.com/ From g.brandl at gmx.net Wed May 5 03:40:45 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 05 May 2010 03:40:45 +0200 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> <4BDBB846.5020305@v.loewis.de> <20100503115752.4f19f2e3@heresy> Message-ID: Am 03.05.2010 18:40, schrieb Guido van Rossum: > On Mon, May 3, 2010 at 8:57 AM, Barry Warsaw wrote: >> On May 01, 2010, at 07:12 AM, Martin v. L?wis wrote: >> >>>> IIRC in the IETF this is done by the committee chair. I think it's a >>>> good idea to have this be a single person to avoid endless indecision. >>> >>>It then seems that this role should go to the release manager of the >>>upcoming feature release. Assuming Georg can accept this additional >>>responsibility. >> >> I do think it makes sense for the RM to assume these responsibilities where >> Guido either can't or doesn't want to make the final decision. I think it >> will fairly substantially increase the workload on the RM, so perhaps there >> are ways to off-load some of the current responsibilities (e.g. updating the >> website for each release). I also think that RMs should be term-limited so >> that we can spread more experience within the community. And past-RMs can >> provide a sort of consultation group where contentious decisions can be >> discussed and advice gathered. > > While I certainly won't object if a release manager volunteers to also > be the final PEP arbiter, I don't want to make it a job requirement > (or even an implied expectation). The responsibility of a release > manager is very much in the here and now and the near future -- > stability and consistency of the current release. Being PEP arbiter > requires a somewhat different mindset, more focused on the long-term > health of the language and its community. I agree, and I wouldn't want to make these decisions. That person (or group) needs to have some weight in the community, or there will be a feeling of "... and who is he to decide anyway". We haven't emphasized RMship in the past; it's not a special position, except when something about a release goes wrong and blame must be placed ;) I think, being Guido, you are still the best single person to do this job. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From fuzzyman at voidspace.org.uk Wed May 5 12:43:45 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 05 May 2010 11:43:45 +0100 Subject: [Python-Dev] Fwd: broken mailing list links in PEP(s?) Message-ID: <4BE14BE1.6090200@voidspace.org.uk> Hello all, It looks like the changes to the python-dev mailman archives broke some of the links in PEPs. All the best, Michael Foord -------- Original Message -------- Subject: broken mailing list links in PEP(s?) Date: Tue, 4 May 2010 20:22:57 -0700 From: Bayle Shanks To: webmaster at python.org On http://www.python.org/dev/peps/pep-0225/ , in the section "Credits and archives", there are a bunch of links like http://mail.python.org/pipermail/python-list/2000-July/108893.html which are broken thanks, bayle -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phd.pp.ru Wed May 5 13:09:08 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Wed, 5 May 2010 15:09:08 +0400 Subject: [Python-Dev] Fwd: broken mailing list links in PEP(s?) In-Reply-To: <4BE14BE1.6090200@voidspace.org.uk> References: <4BE14BE1.6090200@voidspace.org.uk> Message-ID: <20100505110908.GA7160@phd.pp.ru> On Wed, May 05, 2010 at 11:43:45AM +0100, Michael Foord wrote: > http://mail.python.org/pipermail/python-list/2000-July/108893.html > > which are broken Pipermail's links aren't stable AFAIU. The numbering is changing over time. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From ncoghlan at gmail.com Wed May 5 13:15:56 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 05 May 2010 21:15:56 +1000 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> <4BDBB846.5020305@v.loewis.de> <20100503115752.4f19f2e3@heresy> Message-ID: <4BE1536C.6000206@gmail.com> Georg Brandl wrote: > I agree, and I wouldn't want to make these decisions. That person (or > group) needs to have some weight in the community, or there will be a > feeling of "... and who is he to decide anyway". We haven't emphasized > RMship in the past; it's not a special position, except when something > about a release goes wrong and blame must be placed ;) I think, being > Guido, you are still the best single person to do this job. Not to wish anything bad on Guido, but I think it does behoove us as responsible stewards of the reference interpreter to at least consider the "bus" factor (as in "hit by a..."). I suspect Martin is right that some form of committee (at least initially) would be inevitable. While I doubt that we could find any one person that everyone would agree to defer to, I suspect we could find a small group of 3 or 4 people who's consensus everyone else would respect. And if that small group couldn't agree on whether or not something was a good idea, then the old "status quo wins a stalemate" would kick in. That said, this is hopefully something that we won't have to worry about for real until some time well into the future when Guido decides he wants to retire. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed May 5 13:20:40 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 05 May 2010 21:20:40 +1000 Subject: [Python-Dev] Fwd: broken mailing list links in PEP(s?) In-Reply-To: <20100505110908.GA7160@phd.pp.ru> References: <4BE14BE1.6090200@voidspace.org.uk> <20100505110908.GA7160@phd.pp.ru> Message-ID: <4BE15488.2090103@gmail.com> Oleg Broytman wrote: > On Wed, May 05, 2010 at 11:43:45AM +0100, Michael Foord wrote: >> http://mail.python.org/pipermail/python-list/2000-July/108893.html >> >> which are broken > > Pipermail's links aren't stable AFAIU. The numbering is changing over > time. I don't think that's true in general. I do recall the archive getting FUBARed a few months back, and one of the consequences of fixing it was that some old messages were renumbered (since the whole archive basically had to be reindexed or something like that). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From fuzzyman at voidspace.org.uk Wed May 5 13:24:01 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 05 May 2010 12:24:01 +0100 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <4BE1536C.6000206@gmail.com> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> <4BDBB846.5020305@v.loewis.de> <20100503115752.4f19f2e3@heresy> <4BE1536C.6000206@gmail.com> Message-ID: <4BE15551.4020309@voidspace.org.uk> On 05/05/2010 12:15, Nick Coghlan wrote: > Georg Brandl wrote: > >> I agree, and I wouldn't want to make these decisions. That person (or >> group) needs to have some weight in the community, or there will be a >> feeling of "... and who is he to decide anyway". We haven't emphasized >> RMship in the past; it's not a special position, except when something >> about a release goes wrong and blame must be placed ;) I think, being >> Guido, you are still the best single person to do this job. >> > Not to wish anything bad on Guido, but I think it does behoove us as > responsible stewards of the reference interpreter to at least consider > the "bus" factor (as in "hit by a..."). > > I suspect Martin is right that some form of committee (at least > initially) would be inevitable. While I doubt that we could find any one > person that everyone would agree to defer to, I suspect we could find a > small group of 3 or 4 people who's consensus everyone else would > respect. And if that small group couldn't agree on whether or not > something was a good idea, then the old "status quo wins a stalemate" > would kick in. > > Sounds like a sensible arrangement. > That said, this is hopefully something that we won't have to worry about > for real until some time well into the future when Guido decides he > wants to retire. > Well, we (or more specifically Guido) may want to transition to some system where Guido doesn't have to make *every* decision. We only need this for PEPs where there isn't a community consensus. Michael > Regards, > Nick. > > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From g.brandl at gmx.net Wed May 5 13:30:04 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 05 May 2010 13:30:04 +0200 Subject: [Python-Dev] Two small PEP ideas In-Reply-To: <4BE15551.4020309@voidspace.org.uk> References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> <20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com> <4BDBB846.5020305@v.loewis.de> <20100503115752.4f19f2e3@heresy> <4BE1536C.6000206@gmail.com> <4BE15551.4020309@voidspace.org.uk> Message-ID: Am 05.05.2010 13:24, schrieb Michael Foord: > On 05/05/2010 12:15, Nick Coghlan wrote: >> Georg Brandl wrote: >> >>> I agree, and I wouldn't want to make these decisions. That person (or >>> group) needs to have some weight in the community, or there will be a >>> feeling of "... and who is he to decide anyway". We haven't emphasized >>> RMship in the past; it's not a special position, except when something >>> about a release goes wrong and blame must be placed ;) I think, being >>> Guido, you are still the best single person to do this job. >>> >> Not to wish anything bad on Guido, but I think it does behoove us as >> responsible stewards of the reference interpreter to at least consider >> the "bus" factor (as in "hit by a..."). Sure, but I think we wouldn't have too much trouble finding a replacement if and when that happens. (Let's hope it doesn't.) This is not something that requires weeks of preparation, and even if it did, no pronouncement on a PEP for a few weeks isn't the doom of Python. >> I suspect Martin is right that some form of committee (at least >> initially) would be inevitable. While I doubt that we could find any one >> person that everyone would agree to defer to, I suspect we could find a >> small group of 3 or 4 people who's consensus everyone else would >> respect. And if that small group couldn't agree on whether or not >> something was a good idea, then the old "status quo wins a stalemate" >> would kick in. > > Sounds like a sensible arrangement. Yes, I could imagine that too. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From barry at python.org Wed May 5 14:22:17 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 5 May 2010 08:22:17 -0400 Subject: [Python-Dev] Fwd: broken mailing list links in PEP(s?) In-Reply-To: <20100505110908.GA7160@phd.pp.ru> References: <4BE14BE1.6090200@voidspace.org.uk> <20100505110908.GA7160@phd.pp.ru> Message-ID: <60DA94AF-6444-435A-86DD-98CAF432197A@python.org> On May 5, 2010, at 7:09 AM, Oleg Broytman wrote: > On Wed, May 05, 2010 at 11:43:45AM +0100, Michael Foord wrote: >> http://mail.python.org/pipermail/python-list/2000-July/108893.html >> >> which are broken > > Pipermail's links aren't stable AFAIU. The numbering is changing over > time. They're only unstable if you regenerate the archives and the mbox file is old enough to have been a victim of a long-fixed delimiter bug. Which is true for python-dev. -Barry From solipsis at pitrou.net Wed May 5 15:10:16 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 5 May 2010 13:10:16 +0000 (UTC) Subject: [Python-Dev] Reindenting patches References: <4BDF8289.2090306@holdenweb.com> <4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy> <4BE03130.9020700@trueblade.com> <4BE0A280.8080903@gmail.com> Message-ID: Nick Coghlan gmail.com> writes: > > Howver, if we delay fixing the C file indentation until we're already on > hg, the merge tools should offer the best chance of being able to apply > pre-fix patches and have the software figure out where the whitespace > changes need to be accounted for. For the record, I've added to the untabify script a patch rewriting option ("-p") which reindents all patch hunks for C files containing tabs. It should minimize manual reformatting work with existing patches. Regards Antoine. From foom at fuhm.net Wed May 5 17:35:05 2010 From: foom at fuhm.net (James Y Knight) Date: Wed, 5 May 2010 11:35:05 -0400 Subject: [Python-Dev] Fwd: broken mailing list links in PEP(s?) In-Reply-To: <60DA94AF-6444-435A-86DD-98CAF432197A@python.org> References: <4BE14BE1.6090200@voidspace.org.uk> <20100505110908.GA7160@phd.pp.ru> <60DA94AF-6444-435A-86DD-98CAF432197A@python.org> Message-ID: On May 5, 2010, at 8:22 AM, Barry Warsaw wrote: > On May 5, 2010, at 7:09 AM, Oleg Broytman wrote: > >> On Wed, May 05, 2010 at 11:43:45AM +0100, Michael Foord wrote: >>> http://mail.python.org/pipermail/python-list/2000-July/108893.html >>> >>> which are broken >> >> Pipermail's links aren't stable AFAIU. The numbering is changing >> over >> time. > > They're only unstable if you regenerate the archives and the mbox > file is old enough to have been a victim of a long-fixed delimiter > bug. Which is true for python-dev. And of course if you're paying attention, you can fix the mbox file (quoting "From" etc) such that it generates the same numbers as it did the first time. James From brett at python.org Wed May 5 22:56:45 2010 From: brett at python.org (Brett Cannon) Date: Wed, 5 May 2010 13:56:45 -0700 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: References: Message-ID: I am done running the analysis over trunk. I will not svnmerge these changes into py3k as the amount of time and effort that would take equates to running the static analyzer again just before 3.2 is released and possibly catching more changes (and maybe even a newer version of Clang at that point). On Mon, May 3, 2010 at 15:37, Brett Cannon wrote: > Since 2.7 is probably going to exist for a while, I am running Clang 2.7's > static analyzer (``clang --static``) over trunk. It's mostly just finding > stuff like unneeded variable initialization or variables that are never used > (compilation is picking up unused returned values, almost all from > PyObject_INIT). > > When I check in these changes I will do it file by file, but my question is > how to handle Misc/NEWS. I have gone through the underscores and the 'a's in > Modules and already have six modified files, so the final count might be a > little high. Do people want individual entries per file, or simply a single > entry that lists each file modified? > > We should probably go through the C code and fix the whitespace before we > hit 2.7 final (there is a ton of lines with extraneous spaces). > > -Brett > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at haypocalc.com Wed May 5 23:01:50 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 5 May 2010 23:01:50 +0200 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: References: Message-ID: <201005052301.50504.victor.stinner@haypocalc.com> Le mardi 04 mai 2010 00:37:22, Brett Cannon a ?crit : > Since 2.7 is probably going to exist for a while, I am running Clang 2.7's > static analyzer (``clang --static``) over trunk. It's mostly just finding > stuff like unneeded variable initialization or variables that are never > used (compilation is picking up unused returned values, almost all from > PyObject_INIT). > > When I check in these changes I will do it file by file, ... Do you plan to port the changes to py3k? and what about 2.6 and 3.1? -- Please don't change whitespaces in the same commit. I think that we should fix the whitespaces of the whole project in one unique commit. Well, or least don't fix whitespaces file by file... -- Victor Stinner http://www.haypocalc.com/ From brett at python.org Thu May 6 01:21:44 2010 From: brett at python.org (Brett Cannon) Date: Wed, 5 May 2010 16:21:44 -0700 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: <201005052301.50504.victor.stinner@haypocalc.com> References: <201005052301.50504.victor.stinner@haypocalc.com> Message-ID: On Wed, May 5, 2010 at 14:01, Victor Stinner wrote: > Le mardi 04 mai 2010 00:37:22, Brett Cannon a ?crit : > > Since 2.7 is probably going to exist for a while, I am running Clang > 2.7's > > static analyzer (``clang --static``) over trunk. It's mostly just finding > > stuff like unneeded variable initialization or variables that are never > > used (compilation is picking up unused returned values, almost all from > > PyObject_INIT). > > > > When I check in these changes I will do it file by file, ... > > Do you plan to port the changes to py3k? In case you didn't see my follow-up email that I sent just before this email, I will most likely do py3k when 3.2 is closer. > and what about 2.6 and 3.1? Not doing 2.6 as almost all changes are too minor bother. I think I found like two potential Py_DECREF/Py_XDECREF changes, but that's about it. And 3.1 would require py3k which I am not planning on doing in the near future. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Thu May 6 01:58:29 2010 From: eric at trueblade.com (Eric Smith) Date: Wed, 05 May 2010 19:58:29 -0400 Subject: [Python-Dev] Did I miss the decision to untabify all of the C code? Message-ID: <4BE20625.9070203@trueblade.com> It looks like we're moving ahead with removing tabs. Was there consensus on this? Last I saw Antoine had written a script that might do what we want, but hadn't been thoroughly tested. Now I've seen a few checkins for files that have been run through the script. What gives? And why do this so close to 2.7? I don't think it will cause any problems, but it's hard to review commits to ensure they have no changes when there's a rush of large commits near a release. Eric. From benjamin at python.org Thu May 6 02:48:35 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 5 May 2010 19:48:35 -0500 Subject: [Python-Dev] Did I miss the decision to untabify all of the C code? In-Reply-To: <4BE20625.9070203@trueblade.com> References: <4BE20625.9070203@trueblade.com> Message-ID: 2010/5/5 Eric Smith : > It looks like we're moving ahead with removing tabs. Was there consensus on > this? > > Last I saw Antoine had written a script that might do what we want, but > hadn't been thoroughly tested. Now I've seen a few checkins for files that > have been run through the script. > > What gives? And why do this so close to 2.7? I don't think it will cause any > problems, but it's hard to review commits to ensure they have no changes > when there's a rush of large commits near a release. I presume Victor was acting on an older decision that said that files mixed with tabs and spaces can be reindented for consistency. -- Regards, Benjamin From solipsis at pitrou.net Thu May 6 02:51:20 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 6 May 2010 00:51:20 +0000 (UTC) Subject: [Python-Dev] Did I miss the decision to untabify all of the C code? References: <4BE20625.9070203@trueblade.com> Message-ID: Eric Smith trueblade.com> writes: > > Last I saw Antoine had written a script that might do what we want, but > hadn't been thoroughly tested. Now I've seen a few checkins for files > that have been run through the script. As far as I'm concerned, it was a case of eating my own dog food: running the script over a couple of files I'm interested in (_ssl.c, _fileio.c). I believe Victor processed posixmodule.c for the same reasons. > What gives? And why do this so close to 2.7? I don't think it will cause > any problems, but it's hard to review commits to ensure they have no > changes when there's a rush of large commits near a release. Well, however soon or late we do this, good luck reviewing multi-thousand line commits to check no mistake sneaked in :) By construction, these commits only adjust whitespace in some C files, which means the risk of breakage is very close to zero. (I guess you could do a "svn diff -x -w" between each two revisions to expose any potential non-whitespace changes) Really Antoine. From eric at trueblade.com Thu May 6 02:59:26 2010 From: eric at trueblade.com (Eric Smith) Date: Wed, 05 May 2010 20:59:26 -0400 Subject: [Python-Dev] Did I miss the decision to untabify all of the C code? In-Reply-To: References: <4BE20625.9070203@trueblade.com> Message-ID: <4BE2146E.6000702@trueblade.com> Antoine Pitrou wrote: > Eric Smith trueblade.com> writes: >> Last I saw Antoine had written a script that might do what we want, but >> hadn't been thoroughly tested. Now I've seen a few checkins for files >> that have been run through the script. > > As far as I'm concerned, it was a case of eating my own dog food: running the > script over a couple of files I'm interested in (_ssl.c, _fileio.c). I believe > Victor processed posixmodule.c for the same reasons. > >> What gives? And why do this so close to 2.7? I don't think it will cause >> any problems, but it's hard to review commits to ensure they have no >> changes when there's a rush of large commits near a release. > > Well, however soon or late we do this, good luck reviewing multi-thousand line > commits to check no mistake sneaked in :) That's my point. Since it's basically unreviewable, is it smart to do it during a beta? I grant you that it's a largely a mechanized change (except for the "a posteriori manual intervention" part), but still. Eric. From jsbueno at python.org.br Thu May 6 05:52:28 2010 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Thu, 6 May 2010 00:52:28 -0300 Subject: [Python-Dev] Did I miss the decision to untabify all of the C code? In-Reply-To: <4BE2146E.6000702@trueblade.com> References: <4BE20625.9070203@trueblade.com> <4BE2146E.6000702@trueblade.com> Message-ID: On Wed, May 5, 2010 at 9:59 PM, Eric Smith wrote: > Antoine Pitrou wrote: >> >> Eric Smith trueblade.com> writes: >>> >>> Last I saw Antoine had written a script that might do what we want, but >>> hadn't been thoroughly tested. Now I've seen a few checkins for files that >>> have been run through the script. >> >> As far as I'm concerned, it was a case of eating my own dog food: running >> the >> script over a couple of files I'm interested in (_ssl.c, _fileio.c). I >> believe >> Victor processed posixmodule.c for the same reasons. >> >>> What gives? And why do this so close to 2.7? I don't think it will cause >>> any problems, but it's hard to review commits to ensure they have no changes >>> when there's a rush of large commits near a release. >> >> Well, however soon or late we do this, good luck reviewing multi-thousand >> line >> commits to check no mistake sneaked in :) > > That's my point. Since it's basically unreviewable, is it smart to do it > during a beta? Hello folks - I don't think these modifications are that "unreviewable": the generated binaries have to be exactly the same with the untabified files don't they? So is a matter of stashing the binaries, applying the patches, rebuild and check to see if the binaries match. Any possible script defects undetected by this would be only (C code) indentation, which could be fixed later. Python 2.7 is in beta, but not applying such a fix now would probably mean that python 2.x would forever remain with the mixed tabs, since it would make much less sense for such a change in a minor revision (although I'd favor it even there). js -><- > > I grant you that it's a largely a mechanized change (except for the "a > posteriori manual intervention" part), but still. > > Eric. From alexandre at peadrop.com Thu May 6 08:17:14 2010 From: alexandre at peadrop.com (Alexandre Vassalotti) Date: Wed, 5 May 2010 23:17:14 -0700 Subject: [Python-Dev] Did I miss the decision to untabify all of the C code? In-Reply-To: References: <4BE20625.9070203@trueblade.com> <4BE2146E.6000702@trueblade.com> Message-ID: On Wed, May 5, 2010 at 8:52 PM, Joao S. O. Bueno wrote: > Python 2.7 is in beta, but not applying such a fix now would probably > mean that python 2.x would forever remain with the mixed tabs, since > it would make much less sense for such a change in a minor revision > (although I'd favor it even there). > Since 2.7 is likely the last release of the 2.x series, wouldn't it more productive to spend time improving it instead of wasting time on minor details like indentation? -- Alexandre From victor.stinner at haypocalc.com Thu May 6 10:43:44 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Thu, 6 May 2010 10:43:44 +0200 Subject: [Python-Dev] Did I miss the decision to untabify all of the C code? In-Reply-To: References: <4BE20625.9070203@trueblade.com> Message-ID: <201005061043.44725.victor.stinner@haypocalc.com> Le jeudi 06 mai 2010 08:17:14, Alexandre Vassalotti a ?crit : > On Wed, May 5, 2010 at 8:52 PM, Joao S. O. Bueno wrote: > > Python 2.7 is in beta, but not applying such a fix now would probably > > mean that python 2.x would forever remain with the mixed tabs, since > > it would make much less sense for such a change in a minor revision > > (although I'd favor it even there). > > Since 2.7 is likely the last release of the 2.x series, wouldn't it > more productive to spend time improving it instead of wasting time on > minor details like indentation? Because "2.7 is likely the last release of the 2.x series", I think that it will maintained more longer than any other 2.x release. If we change the indentation in 3.x but not in 2.x, it will be harder to port the commits from 3.x to 2.7. I consider that the maintenance is very important for this last release, more important than "improving it". I prefer to work on new features on 3.x and keep 2.7 as stable as we can. -- Victor Stinner http://www.haypocalc.com/ From victor.stinner at haypocalc.com Thu May 6 11:41:18 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Thu, 6 May 2010 11:41:18 +0200 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: <4BDFACC0.2010808@v.loewis.de> References: <4BDFACC0.2010808@v.loewis.de> Message-ID: <201005061141.19065.victor.stinner@haypocalc.com> Le mardi 04 mai 2010 07:12:32, Martin v. L?wis a ?crit : > > Will changing the indentation of source files to 4 space indents break > > patches on the bug tracker? > > Plain patch will choke, but "patch -l" might accept them. Tested on posixmodule.c: it works :-) -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Thu May 6 11:53:55 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 6 May 2010 09:53:55 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Did_I_miss_the_decision_to_untabify_all_of?= =?utf-8?q?_the_C=09code=3F?= References: <4BE20625.9070203@trueblade.com> <4BE2146E.6000702@trueblade.com> Message-ID: Alexandre Vassalotti peadrop.com> writes: > > Since 2.7 is likely the last release of the 2.x series, wouldn't it > more productive to spend time improving it instead of wasting time on > minor details like indentation? We occasionally waste time on minor details such as code indentation, documentation wording and punctuation, distinguishing "built-in" vs "builtin", etc. :) I don't think it prevents anyone from doing productive work. (besides, my own bug queue for 2.x currently appears to be empty) Then, as pointed out by Victor, if we want to solve the current indentation mixup, we have to do it in all branches so as to avoid making backports more difficult. Regards Antoine. From victor.stinner at haypocalc.com Thu May 6 10:38:01 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Thu, 6 May 2010 10:38:01 +0200 Subject: [Python-Dev] Did I miss the decision to untabify all of the C code? In-Reply-To: <4BE2146E.6000702@trueblade.com> References: <4BE20625.9070203@trueblade.com> <4BE2146E.6000702@trueblade.com> Message-ID: <201005061038.02285.victor.stinner@haypocalc.com> Le jeudi 06 mai 2010 02:59:26, Eric Smith a ?crit : > I grant you that it's a largely a mechanized change (except for the "a > posteriori manual intervention" part), but still. Attached patches are the "manual interventation" parts. 99% of the whole patch only changes the indentation. There is just *two* changes not related directly to indentation: - I moved rc and buffer variable (specific to OS/2) at the beginning of the function to avoid { and }. - I added a newline in "static PyObject * os2_error(int code)" The manual editions is mostly to "fix" the indentation. Diff on Python trunk (diff -uw): --------------------------- --- Modules/posixmodule.c.r80843 2010-05-06 10:18:47.000000000 +0200 +++ Modules/posixmodule.c 2010-05-06 10:18:51.000000000 +0200 @@ -470,6 +470,10 @@ { PyObject *d; char **e; +#if defined(PYOS_OS2) + APIRET rc; + char buffer[1024]; /* OS/2 Provides a Documented Max of 1024 Chars */ +#endif d = PyDict_New(); if (d == NULL) return NULL; @@ -505,10 +509,6 @@ Py_DECREF(v); } #if defined(PYOS_OS2) - { - APIRET rc; - char buffer[1024]; /* OS/2 Provides a Documented Max of 1024 Chars */ - rc = DosQueryExtLIBPATH(buffer, BEGIN_LIBPATH); if (rc == NO_ERROR) { /* (not a type, envname is NOT 'BEGIN_LIBPATH') */ PyObject *v = PyString_FromString(buffer); @@ -521,7 +521,6 @@ PyDict_SetItemString(d, "ENDLIBPATH", v); Py_DECREF(v); } - } #endif return d; } @@ -662,7 +661,8 @@ errors are not in a global variable e.g. 'errno' nor are they congruent with posix error numbers. */ -static PyObject * os2_error(int code) +static PyObject * +os2_error(int code) { char text[1024]; PyObject *v; --------------------------- Diff on Python py3k (diff -uw): --------------------------- --- Modules/posixmodule.c.r80845 2010-05-06 10:36:27.000000000 +0200 +++ Modules/posixmodule.c 2010-05-06 10:36:32.000000000 +0200 @@ -445,6 +445,11 @@ #else char **e; #endif +#if defined(PYOS_OS2) + APIRET rc; + char buffer[1024]; /* OS/2 Provides a Documented Max of 1024 Chars */ +#endif + d = PyDict_New(); if (d == NULL) return NULL; @@ -515,10 +520,6 @@ } #endif #if defined(PYOS_OS2) - { - APIRET rc; - char buffer[1024]; /* OS/2 Provides a Documented Max of 1024 Chars */ - rc = DosQueryExtLIBPATH(buffer, BEGIN_LIBPATH); if (rc == NO_ERROR) { /* (not a type, envname is NOT 'BEGIN_LIBPATH') */ PyObject *v = PyBytes_FromString(buffer); @@ -531,7 +532,6 @@ PyDict_SetItemString(d, "ENDLIBPATH", v); Py_DECREF(v); } - } #endif return d; } @@ -672,7 +672,8 @@ errors are not in a global variable e.g. 'errno' nor are they congruent with posix error numbers. */ -static PyObject * os2_error(int code) +static PyObject * +os2_error(int code) { char text[1024]; PyObject *v; --------------------------- -- Victor Stinner http://www.haypocalc.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: posix_tabify_manual-trunk.patch Type: text/x-patch Size: 22334 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: posix_untabify_manual-py3k.patch Type: text/x-patch Size: 18448 bytes Desc: not available URL: From barry at python.org Thu May 6 14:21:38 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 6 May 2010 08:21:38 -0400 Subject: [Python-Dev] Fwd: broken mailing list links in PEP(s?) In-Reply-To: References: <4BE14BE1.6090200@voidspace.org.uk> <20100505110908.GA7160@phd.pp.ru> <60DA94AF-6444-435A-86DD-98CAF432197A@python.org> Message-ID: <20100506082138.33d659d9@heresy> On May 05, 2010, at 11:35 AM, James Y Knight wrote: >And of course if you're paying attention, you can fix the mbox file >(quoting "From" etc) such that it generates the same numbers as it did >the first time. Mailman even has a command for this (I feel like an Apple commercial). You should check things though because it's based on heuristics that can sometimes be fooled. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From catherine.devlin at gmail.com Thu May 6 16:47:53 2010 From: catherine.devlin at gmail.com (Catherine Devlin) Date: Thu, 6 May 2010 10:47:53 -0400 Subject: [Python-Dev] audience-instructors for Teach Me Python Bugfixing needed Message-ID: Hey, everybody... I'm Catherine, a database administrator who makes up excuses to write Python instead. I'm not actually here as a core developer, but as somebody who hopes to become a developer and recruit some more, which brings me to my question: Who lives close enough to Ohio to make it to PyOhio this summer? I want to use PyOhio to create new Python devs (including myself), but I need some existing ones to seed the process. I need a few veterans (3?) who can commit to come to PyOhio and take part as audience/instructors in a "Teach Me [Python core / standard library] Bugfixing" session. (See http://catherinedevlin.blogspot.com/2010/04/bugfixing-at-pyohio.html.) The PyOhio Call for Proposals is up May 10 so I'd better find you quick! I'm pretty much ignorant enough to lead a Teach Me session. In a Teach Me session, the person at the projector *doesn't* know the material. Instead, she asks the audience questions ("How do I find a bug to work on?"), and they talk her through it. It's based on Teach Me Twisted, a mind-blowing session Steve Holden led at PyCon 2008 ( http://catherinedevlin.blogspot.com/2008/03/teach-me-twisted.html). I think it's a fantastic way to teach, but it depends on some veterans being in the audience. There are folks in the greater Python community eager to get hold of a video of such a session... if we do this well, it could become an important tool in keeping the quality of core Python code high. And all I need from you, my audience-instructors, is a promise to show up (no preparation necessary). Can you make it? Can you pass the appeal on to others you know of? Thanks! Hope to see you in July! -- - Catherine http://catherinedevlin.blogspot.com/ *** PyOhio 2010 * July 31 - Aug 1 * Columbus, OH * pyohio.org *** -------------- next part -------------- An HTML attachment was scrubbed... URL: From doug.hellmann at gmail.com Thu May 6 17:01:58 2010 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Thu, 6 May 2010 11:01:58 -0400 Subject: [Python-Dev] audience-instructors for Teach Me Python Bugfixing needed In-Reply-To: References: Message-ID: <8A2A6234-F4A1-4A3D-A9A8-A9BFDE0DB080@gmail.com> What an excellent idea! We should have these at *every* regional conference. Doug On May 6, 2010, at 10:47 AM, Catherine Devlin wrote: > Hey, everybody... I'm Catherine, a database administrator who makes > up excuses to write Python instead. > > I'm not actually here as a core developer, but as somebody who hopes > to become a developer and recruit some more, which brings me to my > question: > > Who lives close enough to Ohio to make it to PyOhio this summer? I > want to use PyOhio to create new Python devs (including myself), but > I need some existing ones to seed the process. > > I need a few veterans (3?) who can commit to come to PyOhio and take > part as audience/instructors in a "Teach Me [Python core / standard > library] Bugfixing" session. (See http://catherinedevlin.blogspot.com/2010/04/bugfixing-at-pyohio.html.) > The PyOhio Call for Proposals is up May 10 so I'd better find you > quick! > > I'm pretty much ignorant enough to lead a Teach Me session. In a > Teach Me session, the person at the projector *doesn't* know the > material. Instead, she asks the audience questions ("How do I find > a bug to work on?"), and they talk her through it. It's based on > Teach Me Twisted, a mind-blowing session Steve Holden led at PyCon > 2008 (http://catherinedevlin.blogspot.com/2008/03/teach-me-twisted.html > ). I think it's a fantastic way to teach, but it depends on some > veterans being in the audience. There are folks in the greater > Python community eager to get hold of a video of such a session... > if we do this well, it could become an important tool in keeping the > quality of core Python code high. > > And all I need from you, my audience-instructors, is a promise to > show up (no preparation necessary). Can you make it? Can you pass > the appeal on to others you know of? > > Thanks! Hope to see you in July! > > -- > - Catherine > http://catherinedevlin.blogspot.com/ > *** PyOhio 2010 * July 31 - Aug 1 * Columbus, OH * pyohio.org *** > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/doug.hellmann%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Thu May 6 17:09:11 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 06 May 2010 17:09:11 +0200 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: References: Message-ID: <27829881-2894-4A95-8D01-7BCD0B06FBB1@mac.com> On 5 May, 2010, at 22:56, Brett Cannon wrote: > I am done running the analysis over trunk. I will not svnmerge these changes into py3k as the amount of time and effort that would take equates to running the static analyzer again just before 3.2 is released and possibly catching more changes (and maybe even a newer version of Clang at that point). Have you looked into teaching clang's static analyser about Python's refcounting rules? Clang's analyser can tell you about problems related to reference count management for Objective-C code and doing the same for code using the CPython API would be usefull. Ronald > > On Mon, May 3, 2010 at 15:37, Brett Cannon wrote: > Since 2.7 is probably going to exist for a while, I am running Clang 2.7's static analyzer (``clang --static``) over trunk. It's mostly just finding stuff like unneeded variable initialization or variables that are never used (compilation is picking up unused returned values, almost all from PyObject_INIT). > > When I check in these changes I will do it file by file, but my question is how to handle Misc/NEWS. I have gone through the underscores and the 'a's in Modules and already have six modified files, so the final count might be a little high. Do people want individual entries per file, or simply a single entry that lists each file modified? > > We should probably go through the C code and fix the whitespace before we hit 2.7 final (there is a ton of lines with extraneous spaces). > > -Brett > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From solipsis at pitrou.net Thu May 6 17:29:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 6 May 2010 15:29:33 +0000 (UTC) Subject: [Python-Dev] audience-instructors for Teach Me Python Bugfixing needed References: Message-ID: Hello, > Who lives close enough to Ohio to make it to PyOhio this summer? I want > to use PyOhio to create new Python devs (including myself), but I need > some existing ones to seed the process. I'm not really answering your question (I'm very far from Ohio), but a good way to start up without physically meeting a core dev in your area would be to look for bugs in the issue tracker and start tackling those that tickle your interest. "Tackling" here can mean proposing a patch, or giving advice, or asking for guidance on how to produce a patch for that particular issue, or even reviewing an existing patch. You don't have to be a CPython expert to try this; just be prepared to spend some time exchanging with other people who will point out possible mistakes, or teach you some of the annoying idiosyncrasies. Scanning through open issues will also give you a general idea of what kind of functionalities are looking for improvement, or need fixing. (you can create a new issue and start tackling it yourself, too) Also, if you have enough free time, you can hang out on #python-dev, which can speed up the process, but it's not required. You can also find formal information about the development process here: http://www.python.org/dev/faq/ Regards Antoine. From brett at python.org Thu May 6 19:40:05 2010 From: brett at python.org (Brett Cannon) Date: Thu, 6 May 2010 10:40:05 -0700 Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base In-Reply-To: <27829881-2894-4A95-8D01-7BCD0B06FBB1@mac.com> References: <27829881-2894-4A95-8D01-7BCD0B06FBB1@mac.com> Message-ID: On Thu, May 6, 2010 at 08:09, Ronald Oussoren wrote: > > On 5 May, 2010, at 22:56, Brett Cannon wrote: > > I am done running the analysis over trunk. I will not svnmerge these > changes into py3k as the amount of time and effort that would take equates > to running the static analyzer again just before 3.2 is released and > possibly catching more changes (and maybe even a newer version of Clang at > that point). > > > Have you looked into teaching clang's static analyser about Python's > refcounting rules? Clang's analyser can tell you about problems related to > reference count management for Objective-C code and doing the same for code > using the CPython API would be usefull. > That's a thought, but I have not looked into it yet. As of right now the first thing I would do is fix its NULL de-reference analysis as it had a bunch of false-positives on that (I don't think it handles `!ptr` as equivalent to `ptr == NULL`). -Brett > > Ronald > > > On Mon, May 3, 2010 at 15:37, Brett Cannon wrote: > >> Since 2.7 is probably going to exist for a while, I am running Clang 2.7's >> static analyzer (``clang --static``) over trunk. It's mostly just finding >> stuff like unneeded variable initialization or variables that are never used >> (compilation is picking up unused returned values, almost all from >> PyObject_INIT). >> >> When I check in these changes I will do it file by file, but my question >> is how to handle Misc/NEWS. I have gone through the underscores and the 'a's >> in Modules and already have six modified files, so the final count might be >> a little high. Do people want individual entries per file, or simply a >> single entry that lists each file modified? >> >> We should probably go through the C code and fix the whitespace before we >> hit 2.7 final (there is a ton of lines with extraneous spaces). >> >> -Brett >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amk at amk.ca Fri May 7 03:50:35 2010 From: amk at amk.ca (A.M. Kuchling) Date: Thu, 6 May 2010 21:50:35 -0400 Subject: [Python-Dev] What's New text on future maintenance Message-ID: <20100507015035.GA50210@andrew-kuchlings-macbook.local> FYI: I've just added the text below to the "What's New" document for 2.7. I wanted to describe how 2.7 will probably be maintained, but didn't want to write anything that sounded like an iron-clad guarantee of a maintenance timespan. Does this text seem like a reasonable set of statements? --amk Python 2.7 is intended to be the last major release in the 2.x series. Though more major releases have not been absolutely ruled out, the Python maintainers are planning to focus their efforts on Python 3.x. This means that 2.7 will remain in place for a long time, running production systems that have not been ported to Python 3.x. Two consequences of the long-term significance of 2.7 are: * It's very likely the 2.7 release will have a longer period of maintenance compared to earlier 2.x versions. Python 2.7 will continue to be maintained while the transition to 3.x is in progress, and that transition will itself be lengthy. Most 2.x versions are maintained for about 4 years, from the first to the last bugfix release; patchlevel releases for Python 2.7 will probably be made for at least 6 years. * Because 2.7 will be running production applications, a policy decision was made to silence warnings only of interest to developers by default. Silencing :exc:`DeprecationWarning` and its descendants prevents users from seeing warnings triggered by an application. (Carried out in :issue:`7319`.) You can re-enable display of :exc:`DeprecationWarning` messages by running Python with the :option:`-Wdefault` (short form: :option:`-Wd`) switch, or you can add ``warnings.simplefilter('default')`` to your code. From guido at python.org Fri May 7 04:09:46 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 6 May 2010 19:09:46 -0700 Subject: [Python-Dev] What's New text on future maintenance In-Reply-To: <20100507015035.GA50210@andrew-kuchlings-macbook.local> References: <20100507015035.GA50210@andrew-kuchlings-macbook.local> Message-ID: On Thu, May 6, 2010 at 6:50 PM, A.M. Kuchling wrote: > FYI: I've just added the text below to the "What's New" document for > 2.7. ?I wanted to describe how 2.7 will probably be maintained, but > didn't want to write anything that sounded like an iron-clad guarantee > of a maintenance timespan. ?Does this text seem like a reasonable set > of statements? > > --amk > > Python 2.7 is intended to be the last major release in the 2.x series. > Though more major releases have not been absolutely ruled out, the I would scrap the "Though more ... ruled out" part. That just stokes unrealistic hopes. :-) > Python maintainers are planning to focus their efforts on Python 3.x. > > This means that 2.7 will remain in place for a long time, running > production systems that have not been ported to Python 3.x. > Two consequences of the long-term significance of 2.7 are: > > * It's very likely the 2.7 release will have a longer period of > ?maintenance compared to earlier 2.x versions. ?Python 2.7 will > ?continue to be maintained while the transition to 3.x is in > ?progress, and that transition will itself be lengthy. ?Most 2.x > ?versions are maintained for about 4 years, from the first to the > ?last bugfix release; patchlevel releases for Python 2.7 will > ?probably be made for at least 6 years. > > * Because 2.7 will be running production applications, a policy > ?decision was made to silence warnings only of interest to developers > ?by default. ?Silencing :exc:`DeprecationWarning` and its descendants > ?prevents users from seeing warnings triggered by an application. > ?(Carried out in :issue:`7319`.) > > ?You can re-enable display of :exc:`DeprecationWarning` messages by > ?running Python with the :option:`-Wdefault` (short form: > ?:option:`-Wd`) switch, or you can add > ?``warnings.simplefilter('default')`` to your code. All this sounds fine to me. Thanks for taking the time to write this! -- --Guido van Rossum (python.org/~guido) From ben+python at benfinney.id.au Fri May 7 04:10:45 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 07 May 2010 12:10:45 +1000 Subject: [Python-Dev] What's New text on future maintenance References: <20100507015035.GA50210@andrew-kuchlings-macbook.local> Message-ID: <877hngfom2.fsf@benfinney.id.au> "A.M. Kuchling" writes: > FYI: I've just added the text below to the "What's New" document for > 2.7. I wanted to describe how 2.7 will probably be maintained, but > didn't want to write anything that sounded like an iron-clad guarantee > of a maintenance timespan. Does this text seem like a reasonable set > of statements? If you give an actual time period, that's all that will actually be communicated to most people. This text will be read by a few people, and communicated as simply ?six years? to everyone else. It doesn't matter how many caveats and qualifiers you surround that with; those will all be lost in transmission from person to person. Would it make more sense to, instead of giving a time period, rather describe the *circumstances* that will determine when maintenance releases will cease for 2.7? -- \ ?Religious faith is the one species of human ignorance that | `\ will not admit of even the *possibility* of correction.? ?Sam | _o__) Harris, _The End of Faith_, 2004 | Ben Finney From benjamin at python.org Fri May 7 04:33:50 2010 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 6 May 2010 21:33:50 -0500 Subject: [Python-Dev] What's New text on future maintenance In-Reply-To: <20100507015035.GA50210@andrew-kuchlings-macbook.local> References: <20100507015035.GA50210@andrew-kuchlings-macbook.local> Message-ID: 2010/5/6 A.M. Kuchling : > FYI: I've just added the text below to the "What's New" document for > 2.7. ?I wanted to describe how 2.7 will probably be maintained, but > didn't want to write anything that sounded like an iron-clad guarantee > of a maintenance timespan. ?Does this text seem like a reasonable set > of statements? > > --amk > > Python 2.7 is intended to be the last major release in the 2.x series. > Though more major releases have not been absolutely ruled out, the > Python maintainers are planning to focus their efforts on Python 3.x. > > This means that 2.7 will remain in place for a long time, running > production systems that have not been ported to Python 3.x. > Two consequences of the long-term significance of 2.7 are: > > * It's very likely the 2.7 release will have a longer period of > ?maintenance compared to earlier 2.x versions. ?Python 2.7 will > ?continue to be maintained while the transition to 3.x is in > ?progress, and that transition will itself be lengthy. ?Most 2.x > ?versions are maintained for about 4 years, from the first to the > ?last bugfix release; patchlevel releases for Python 2.7 will > ?probably be made for at least 6 years. I don't think there's any point in being hypothetical about. I believe we've already said that maintence for 2.7 will last for at least 5 years, so let's proclaim it. -- Regards, Benjamin From tjreedy at udel.edu Fri May 7 06:43:20 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 07 May 2010 00:43:20 -0400 Subject: [Python-Dev] What's New text on future maintenance In-Reply-To: <20100507015035.GA50210@andrew-kuchlings-macbook.local> References: <20100507015035.GA50210@andrew-kuchlings-macbook.local> Message-ID: On 5/6/2010 9:50 PM, A.M. Kuchling wrote: > FYI: I've just added the text below to the "What's New" document for > 2.7. I wanted to describe how 2.7 will probably be maintained, but > didn't want to write anything that sounded like an iron-clad guarantee > of a maintenance timespan. Does this text seem like a reasonable set > of statements? > > --amk > > Python 2.7 is intended to be the last major release in the 2.x series. > Though more major releases have not been absolutely ruled out, the > Python maintainers are planning to focus their efforts on Python 3.x. > > This means that 2.7 will remain in place for a long time, running > production systems that have not been ported to Python 3.x. > Two consequences of the long-term significance of 2.7 are: > > * It's very likely the 2.7 release will have a longer period of > maintenance compared to earlier 2.x versions. Python 2.7 will > continue to be maintained while the transition to 3.x is in > progress, and that transition will itself be lengthy. Most 2.x > versions are maintained for about 4 years, from the first to the > last bugfix release; patchlevel releases for Python 2.7 will > probably be made for at least 6 years. Actually, bugfix releases generally stop with the next major release, with about a 2 year interval. I agree with others about condensing, to something like: "Python 2.7 is intended to be the last major release in the 2.x series. Python core developers plan to focus future efforts on Python 3.x. This means that 2.7 will remain in place for a long time, running production systems that have not been ported to Python 3.x. Bugfix releases will likely continue for 5 years." Then the warnings stuff > > * Because 2.7 will be running production applications, a policy Every major version (xcept 3.0) has run production application, and 3.1 may be and 3.2 certainly will be. So this reasoning is not clear to me. > decision was made to silence warnings only of interest to developers > by default. I believe this is meant to say "Warnings aimed only at those porting code to 3.x are silenced by default." > Silencing :exc:`DeprecationWarning` and its descendants > prevents users from seeing warnings triggered by an application. > (Carried out in :issue:`7319`.) > > You can re-enable display of :exc:`DeprecationWarning` messages by > running Python with the :option:`-Wdefault` (short form: > :option:`-Wd`) switch, or you can add > ``warnings.simplefilter('default')`` to your code. Terry Jan Reedy From martin at v.loewis.de Fri May 7 09:30:00 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 07 May 2010 09:30:00 +0200 Subject: [Python-Dev] What's New text on future maintenance In-Reply-To: <20100507015035.GA50210@andrew-kuchlings-macbook.local> References: <20100507015035.GA50210@andrew-kuchlings-macbook.local> Message-ID: <4BE3C178.2090800@v.loewis.de> > * It's very likely the 2.7 release will have a longer period of > maintenance compared to earlier 2.x versions. Python 2.7 will > continue to be maintained while the transition to 3.x is in > progress, and that transition will itself be lengthy. Most 2.x > versions are maintained for about 4 years, from the first to the > last bugfix release; patchlevel releases for Python 2.7 will > probably be made for at least 6 years. I agree with Terry: how did you arrive at the 4 years for 2.x releases? Bug fixes releases stopped after the next feature release being made, which gave (counting between initial release and last bug fix release): - 2.0 9 months - 2.1 12 months - 2.2 17 months - 2.3 19 months - 2.4 22 months - 2.5 27 months - 2.6 < 22 months (assuming the final 2.6 release is made before August) In addition, since 2.3, we were offering security fixes for a period of _five_ years. So for 2.7, the question really is how long we create bug fix releases and Windows binaries, rather than accepting only security fixes, and producing source-only releases. I'd personally expect that "longer" might mean 3 years (i.e. one more release cycle), and definitely less than 6. Regards, Martin From ncoghlan at gmail.com Fri May 7 11:52:49 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 07 May 2010 19:52:49 +1000 Subject: [Python-Dev] What's New text on future maintenance In-Reply-To: References: <20100507015035.GA50210@andrew-kuchlings-macbook.local> Message-ID: <4BE3E2F1.4040907@gmail.com> Terry Reedy wrote: > Then the warnings stuff >> >> * Because 2.7 will be running production applications, a policy > > Every major version (xcept 3.0) has run production application, and 3.1 > may be and 3.2 certainly will be. So this reasoning is not clear to me. > >> decision was made to silence warnings only of interest to developers >> by default. > > I believe this is meant to say "Warnings aimed only at those porting > code to 3.x are silenced by default." Actually, the decision was indeed to make all Deprecation Warnings silent by default. The rationale was a bit different from what AMK currently has though (otherwise we wouldn't have made the same change in 3.x). I'll take a stab at a more accurate rationale: """For previous releases, it has been the policy of the CPython core developers that :exc:`DeprecationWarning` should be enabled by default. This provides Python developers with a clear indication when their code has a substantial risk of breaking in the next major version of Python. However, the nature of Python usage has changed over the years, such that there are now a significantly larger number of Python application users that are not directly involved in the development of those applications. This has lead to a situation where users may be receiving irrelevant warnings from an application that is actually working correctly, creating unnecessary concern for affected end users and additional overhead for the developers of these applications in responding to these concerns. Accordingly, starting with Python 2.7, this policy has been revised and the CPython interpreter has been updated to silence all occurrences of :exc:`DeprecationWarning` by default. Developers wishing to see if their application is at significant risk of breaking on the next major Python release can re-enable display of :exc:`DeprecationWarning` messages by running Python with the :option:`-Wdefault` (short form: :option:`-Wd`) switch, or you can add ``warnings.simplefilter('default')`` to your code. (Note that the 'default' in these settings refers to the default warning behaviour of displaying warnings once for each location where they are encountered rather than to the overall default warning settings used by the interpreter)""" Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From victor.stinner at haypocalc.com Fri May 7 13:05:52 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 7 May 2010 13:05:52 +0200 Subject: [Python-Dev] Issue #8610: Set default file system encoding to ascii on error? Message-ID: <201005071305.52332.victor.stinner@haypocalc.com> Python 3.0 introduced PyUnicode_DecodeFSDefault() and PyUnicode_DecodeFSDefaultAndSize() functions. These functions fallback to UTF-8 if getting the file system encoding failed or the encoding is unknown (there is not nl_langinfo(CODESET) function). UTF-8 is not a good choice for the fallback because it's incompatible with other encodings like Latin1. I would like to fallback to ASCII on error which is compatible with all encodings (thanks to surrogateescape). I would like to ensure that sys.getfilesystemencoding() result cannot be None, because Python 3.2 uses it on Unix to decode os.environb and to encode filenames in subprocess. I can implement a fallback for os.environb and subprocess (and other functions calling sys.getfilesystemencoding()), but I prefer to have a reliable sys.getfilesystemencoding() function. This change doesn't concern Windows and Mac OS X because the encoding is hardcoded (mbcs, utf-8). On Unix, I don't know in which case nl_langinfo() can fail. Empty environment is not an error: nl_langinfo(CODESET) returns "ascii". I think that few (or no) user would notice this change. -- Victor Stinner http://www.haypocalc.com/ From solipsis at pitrou.net Fri May 7 13:24:18 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 7 May 2010 11:24:18 +0000 (UTC) Subject: [Python-Dev] Issue #8610: Set default file system encoding to ascii on error? References: <201005071305.52332.victor.stinner@haypocalc.com> Message-ID: Le Fri, 07 May 2010 13:05:52 +0200, Victor Stinner a ?crit?: > > UTF-8 is not a good choice for the fallback because it's incompatible > with other encodings like Latin1. I would like to fallback to ASCII on > error which is compatible with all encodings (thanks to > surrogateescape). What do you mean with "compatible with all encodings thanks to surrogateescape"? >>> "???".encode("ascii", "surrogateescape") Traceback (most recent call last): File "", line 1, in UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128) > I would like to ensure that sys.getfilesystemencoding() result cannot be > None, because Python 3.2 uses it on Unix to decode os.environb and to > encode filenames in subprocess. I can implement a fallback for > os.environb and subprocess (and other functions calling > sys.getfilesystemencoding()), but I prefer to have a reliable > sys.getfilesystemencoding() function. Having a reliable sys.getfilesystemencoding() would be a good thing indeed. > This change doesn't concern Windows and Mac OS X because the encoding is > hardcoded (mbcs, utf-8). On Unix, I don't know in which case > nl_langinfo() can fail. Empty environment is not an error: > nl_langinfo(CODESET) returns "ascii". I think that few (or no) user > would notice this change. Ok, it sounds like a good compromise. Regards Antoine. From victor.stinner at haypocalc.com Fri May 7 13:39:39 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 7 May 2010 13:39:39 +0200 Subject: [Python-Dev] =?utf-8?q?Issue_=238610=3A_Set_default_file_system_e?= =?utf-8?q?ncoding_to_ascii=09on_error=3F?= In-Reply-To: References: <201005071305.52332.victor.stinner@haypocalc.com> Message-ID: <201005071339.40070.victor.stinner@haypocalc.com> Le vendredi 07 mai 2010 13:24:18, Antoine Pitrou a ?crit : > > UTF-8 is not a good choice for the fallback because it's incompatible > > with other encodings like Latin1. I would like to fallback to ASCII on > > error which is compatible with all encodings (thanks to > > surrogateescape). > > What do you mean with "compatible with all encodings thanks to > surrogateescape"? > > >>> "???".encode("ascii", "surrogateescape") > ... > UnicodeEncodeError: 'ascii' codec can't encode characters ... ascii+surrogatescape can *decode* anything: >>> b"a\xc3\xff".decode('ascii', 'surrogateescape') 'a\udcc3\udcff' Encode with ascii+surrogatescape raise an UnicodeEncodeError for non-ASCII (except for surrogates). I think it's better to raise an error than creating utf8 filenames on a latin1 file system. -- I forgot to mention Marc Lemburg propositing of creating a PYTHONFSENCODING environment variable: #8622. -- Victor Stinner http://www.haypocalc.com/ From mal at egenix.com Fri May 7 13:55:48 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 07 May 2010 13:55:48 +0200 Subject: [Python-Dev] Issue #8610: Set default file system encoding to ascii on error? In-Reply-To: <201005071305.52332.victor.stinner@haypocalc.com> References: <201005071305.52332.victor.stinner@haypocalc.com> Message-ID: <4BE3FFC4.5000409@egenix.com> Victor Stinner wrote: > Python 3.0 introduced PyUnicode_DecodeFSDefault() and > PyUnicode_DecodeFSDefaultAndSize() functions. These functions fallback to > UTF-8 if getting the file system encoding failed or the encoding is unknown > (there is not nl_langinfo(CODESET) function). > > UTF-8 is not a good choice for the fallback because it's incompatible with > other encodings like Latin1. I would like to fallback to ASCII on error which > is compatible with all encodings (thanks to surrogateescape). > > I would like to ensure that sys.getfilesystemencoding() result cannot be None, > because Python 3.2 uses it on Unix to decode os.environb and to encode > filenames in subprocess. I can implement a fallback for os.environb and > subprocess (and other functions calling sys.getfilesystemencoding()), but I > prefer to have a reliable sys.getfilesystemencoding() function. > > This change doesn't concern Windows and Mac OS X because the encoding is > hardcoded (mbcs, utf-8). On Unix, I don't know in which case nl_langinfo() can > fail. Empty environment is not an error: nl_langinfo(CODESET) returns "ascii". > I think that few (or no) user would notice this change. +1 on that change. The UTF-8 fallback has these major problems: * it hides errors by always having the Unicode-bytes conversion succeed * it can cause applications to write files and create directories with wrongly encoded names (e.g. use UTF-8 on a Latin-1 file system) Together with [issue8622] Add PYTHONFSENCODING environment variable: http://bugs.python.org/issue8622 which reduces the Python3 reliance on encoding guess work, the change would make Python3 more user friendly and reduce the number of bummers user run into when waking up in the all-new Unicode world of Python3. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 07 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2010-04-23: Released mxODBC.Zope.DA 2.0.1 http://zope.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From van.lindberg at gmail.com Fri May 7 16:37:33 2010 From: van.lindberg at gmail.com (VanL) Date: Fri, 07 May 2010 09:37:33 -0500 Subject: [Python-Dev] Possible patch for functools partial - Interested? Message-ID: Howdy all - I have an app where I am using functools.partial to bundle up jobs to do, where a job is defined as a callable + args. In one case, I wanted to keep track of whether I had previously seen a job, so I started putting them into a set... only to find out that partials never test equal to each other: >>> import operator >>> from functools import partial >>> p1 = partial(operator.add) >>> p2 = partial(operator.add) >>> p1 == p2 False >>> seen = set();seen.add(p1) >>> p2 in seen False I created a subclass of functools.partial that provides appropriate __eq__ and __hash__ methods, so that this works as expected. I called the subclass a Job: >>> j1 = Job(operator.add) >>> j2 = Job(operator.add) >>> j1 == j2 True >>> seen = set();seen.add(j1) >>> j2 in seen True >>> j1 is j2 False While I was at it, I also added a nice repr. Would this group be interested in a patch, or is this not interesting? Thanks, Van From fuzzyman at voidspace.org.uk Fri May 7 16:51:17 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 07 May 2010 16:51:17 +0200 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: References: Message-ID: <4BE428E5.3050104@voidspace.org.uk> On 07/05/2010 16:37, VanL wrote: > Howdy all - > > I have an app where I am using functools.partial to bundle up jobs to > do, where a job is defined as a callable + args. In one case, I wanted > to keep track of whether I had previously seen a job, so I started > putting them into a set... only to find out that partials never test > equal to each other: > > >>>> import operator >>>> from functools import partial >>>> p1 = partial(operator.add) >>>> p2 = partial(operator.add) >>>> p1 == p2 >>>> > False > >>>> seen = set();seen.add(p1) >>>> p2 in seen >>>> > False > > I created a subclass of functools.partial that provides appropriate > __eq__ and __hash__ methods, so that this works as expected. I called > the subclass a Job: > >>>> j1 = Job(operator.add) >>>> j2 = Job(operator.add) >>>> j1 == j2 >>>> > True > >>>> seen = set();seen.add(j1) >>>> j2 in seen >>>> > True > >>>> j1 is j2 >>>> > False > > While I was at it, I also added a nice repr. Would this group be > interested in a patch, or is this not interesting? > > Sounds good to me. Could you post the patch to http://bugs.python.org please. Michael Foord > Thanks, > > Van > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ From solipsis at pitrou.net Fri May 7 17:02:14 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 7 May 2010 15:02:14 +0000 (UTC) Subject: [Python-Dev] Possible patch for functools partial - Interested? References: Message-ID: VanL gmail.com> writes: > > I created a subclass of functools.partial that provides appropriate > __eq__ and __hash__ methods, so that this works as expected. I called > the subclass a Job: [...] > > While I was at it, I also added a nice repr. Would this group be > interested in a patch, or is this not interesting? It would be more useful to provide equality, hashing and repr to partial itself, rather than a subclass. Feel free to propose a patch :) Regards Antoine. From amk at amk.ca Fri May 7 17:39:30 2010 From: amk at amk.ca (A.M. Kuchling) Date: Fri, 7 May 2010 11:39:30 -0400 Subject: [Python-Dev] What's New text on future maintenance In-Reply-To: <4BE3C178.2090800@v.loewis.de> References: <20100507015035.GA50210@andrew-kuchlings-macbook.local> <4BE3C178.2090800@v.loewis.de> Message-ID: <20100507153930.GA21021@amk-desktop.matrixgroup.net> On Fri, May 07, 2010 at 09:30:00AM +0200, "Martin v. L?wis" wrote: > I agree with Terry: how did you arrive at the 4 years for 2.x releases? > Bug fixes releases stopped after the next feature release being made, > which gave (counting between initial release and last bug fix release): I used the initial release date of 2.4 and 2.5 and the dates of the last patchlevel releases. That may well be distorted by the recent spasm of activity that led to 2.5.4. --amk From rob.cliffe at btinternet.com Fri May 7 18:07:55 2010 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 7 May 2010 17:07:55 +0100 Subject: [Python-Dev] Possible patch for functools partial - Interested? References: Message-ID: Sorry to grouse, but isn't this maybe being a bit too clever? Using your example, p1 = partial(operator.add) is creating a callable, p1, i.e. a sort of function. Yes I know technically it's not a function, but it behaves very much like one. Now, if I write def f1(x,y): return x+y def f2(x,y): return x+y I don't expect f1==f2 to be True, even though f1 and f2 behave in exactly the same way, and indeed it is not. If I wanted it to be true, I should have written def f1(x): return x+y f2=f1 I find this behaviour natural and expected, both in my example and yours (although maybe I've just got used to Python's behaviour). Similarly, if you wanted p1==p2, why not write p1 = partial(operator.add) p2 = p1 Maybe I could be persuaded otherwise by a convincing use case, but I rather doubt it. Rob Cliffe ----- Original Message ----- From: "VanL" To: Sent: Friday, May 07, 2010 3:37 PM Subject: [Python-Dev] Possible patch for functools partial - Interested? > Howdy all - > > I have an app where I am using functools.partial to bundle up jobs to > do, where a job is defined as a callable + args. In one case, I wanted > to keep track of whether I had previously seen a job, so I started > putting them into a set... only to find out that partials never test > equal to each other: > >>>> import operator >>>> from functools import partial >>>> p1 = partial(operator.add) >>>> p2 = partial(operator.add) >>>> p1 == p2 > False >>>> seen = set();seen.add(p1) >>>> p2 in seen > False > > I created a subclass of functools.partial that provides appropriate > __eq__ and __hash__ methods, so that this works as expected. I called > the subclass a Job: >>>> j1 = Job(operator.add) >>>> j2 = Job(operator.add) >>>> j1 == j2 > True >>>> seen = set();seen.add(j1) >>>> j2 in seen > True >>>> j1 is j2 > False > > While I was at it, I also added a nice repr. Would this group be > interested in a patch, or is this not interesting? > > Thanks, > > Van > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/rob.cliffe%40btinternet.com From amk at amk.ca Fri May 7 18:09:08 2010 From: amk at amk.ca (A.M. Kuchling) Date: Fri, 7 May 2010 12:09:08 -0400 Subject: [Python-Dev] What's New text on future maintenance In-Reply-To: <4BE3E2F1.4040907@gmail.com> References: <20100507015035.GA50210@andrew-kuchlings-macbook.local> <4BE3E2F1.4040907@gmail.com> Message-ID: <20100507160908.GB21021@amk-desktop.matrixgroup.net> On Fri, May 07, 2010 at 07:52:49PM +1000, Nick Coghlan wrote: > 3.x). I'll take a stab at a more accurate rationale: Thanks! I've applied the scalpel and reduced it to: * A policy decision was made to silence warnings only of interest to developers by default. :exc:`DeprecationWarning` and its descendants are now ignored unless otherwise requested, preventing users from seeing warnings triggered by an application. (Carried out in :issue:`7319`.) In previous releases, :exc:`DeprecationWarning` messages were enabled by default, providing Python developers with a clear indication of where their code may break in a future major version of Python. However, there are increasingly many users of Python-based applications who are not directly involved in the development of those applications. :exc:`DeprecationWarning` messages are irrelevant to such users, making them worry about an application that's actually working correctly and burdening the developers of these applications with responding to these concerns. You can re-enable display of :exc:`DeprecationWarning` messages by running Python with the :option:`-Wdefault` (short form: :option:`-Wd`) switch, or you can add ``warnings.simplefilter('default')`` to your code. Benjamin suggested being very definite about a 5-year maintenance period, but I don't want to write any checks our butt can't cash, so I've left the text as "Maintenance releases for Python 2.7 will probably be made for 5 years." An alternative formulation might say it will be maintained for the next two 3.x releases, not the next one as usual. I thought about Ben Finney's suggestion to not give a timespan and describe the conditions for 2.x maintenance continuing, but those conditions are complicated to describe -- if 3.x doesn't catch on? if the 3.x transition is slow? if there's a significant 2.x user base that remains? if someone starts a 2.x maintenance team? -- and might be a confusing tangle of what-if statements. --amk From status at bugs.python.org Fri May 7 18:09:22 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 7 May 2010 18:09:22 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100507160922.B719978141@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-04-30 - 2010-05-07) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2658 open (+44) / 17807 closed (+24) / 20465 total (+68) Open issues with patches: 1081 Average duration of open issues: 724 days. Median duration of open issues: 494 days. Open Issues Breakdown open 2639 (+44) languishing 12 ( +0) pending 6 ( +0) Issues Created Or Reopened (71) _______________________________ Bug in range() function for large values 2010-05-01 CLOSED http://bugs.python.org/issue1533 reopened mark.dickinson patch, 26backport configure: allow user-provided CFLAGS to override AC_PROG_CC d 2010-05-05 CLOSED http://bugs.python.org/issue8211 reopened mark.dickinson patch regrtest: use backslashreplace error handler for stdout 2010-05-05 http://bugs.python.org/issue8533 reopened haypo patch Add missing tests for FlushKey, LoadKey, and SaveKey in winreg 2010-04-30 http://bugs.python.org/issue8579 created brian.curtin Problem urllib2.URLError 2010-04-30 CLOSED http://bugs.python.org/issue8580 created m_enanos Logging handlers do not handle double-closing very well 2010-04-30 CLOSED http://bugs.python.org/issue8581 created Jason.Baker patch urllib.urlretrieve fails with ValueError: Invalid format strin 2010-04-30 CLOSED http://bugs.python.org/issue8582 created Jason.Gross patch Hardcoded namespace_separator in the cElementTree.XMLParser 2010-04-30 http://bugs.python.org/issue8583 created dmtr patch test_multiprocessing skips some tests 2010-04-30 http://bugs.python.org/issue8584 created pitrou zipimporter.find_module is untested 2010-05-01 http://bugs.python.org/issue8585 created alex test_imp.py test failures on Py3K Mac OS X 2010-05-01 CLOSED http://bugs.python.org/issue8586 created michael.foord test_import.py test failures on Py3K Mac OS X 2010-05-01 CLOSED http://bugs.python.org/issue8587 created michael.foord test_urllib2.py test failures on Py3K Mac OS X 2010-05-01 http://bugs.python.org/issue8588 created michael.foord test_warnings.CEnvironmentVariableTests.test_nonascii fails un 2010-05-01 http://bugs.python.org/issue8589 created michael.foord test_httpservers.CGIHTTPServerTestCase failure on 3.1-maint Ma 2010-05-01 http://bugs.python.org/issue8590 created michael.foord update mkpkg to latest coding standards 2010-05-01 http://bugs.python.org/issue8591 created meatballhat 'y' does not check for embedded NUL bytes 2010-05-01 http://bugs.python.org/issue8592 created pitrou Improve c-api/arg.rst 2010-05-01 CLOSED http://bugs.python.org/issue8593 created pitrou patch Add a "source_address" option to ftplib 2010-05-01 http://bugs.python.org/issue8594 created giampaolo.rodola patch, needs review Explain the default timeout in http-client-related libraries 2010-05-02 http://bugs.python.org/issue8595 created oddthinking crypt blowfish 'ignores' salt 2010-05-02 CLOSED http://bugs.python.org/issue8596 created pvo build out-of-line asm on Windows 2010-05-02 http://bugs.python.org/issue8597 created zooko patch test/support: don't use localhost as IPv6 host name 2010-05-02 http://bugs.python.org/issue8598 created haypo patch _execvpe behaves inconsistently when PATH includes a filename 2010-05-02 CLOSED http://bugs.python.org/issue8599 created Oren_Held test_gdb failures 2010-05-03 CLOSED http://bugs.python.org/issue8600 created pitrou bz2.BZ2File should support "with" protocol per PEP 343 2010-05-03 CLOSED http://bugs.python.org/issue8601 created Matt.Wartell documentation of bz2 module mildly erroneous 2010-05-03 http://bugs.python.org/issue8602 created Matt.Wartell Create a bytes version of os.environ and getenvb() 2010-05-03 CLOSED http://bugs.python.org/issue8603 created haypo patch Adding an atomic FS write API 2010-05-03 http://bugs.python.org/issue8604 created olemis test_gdb can fail with compiler opts 2010-05-03 http://bugs.python.org/issue8605 created pitrou patch OSF is not supported anymore 2010-05-03 CLOSED http://bugs.python.org/issue8606 created jcea OSX: duplicate -arch flags in CFLAGS breaks sysconfig 2010-05-04 CLOSED http://bugs.python.org/issue8607 created robind fix_import prefixes "." to already relative imports 2010-05-04 CLOSED http://bugs.python.org/issue8608 created shlomme patch itertools: problem with nested groupby, list() 2010-05-04 CLOSED http://bugs.python.org/issue8609 created nicki Python3/POSIX: errors if file system encoding is None 2010-05-04 http://bugs.python.org/issue8610 created haypo patch Python3 doesn't support locale different than utf8 and an non- 2010-05-04 http://bugs.python.org/issue8611 created haypo multiprocessing Queue module failes to send INIConfig objects 2010-05-04 http://bugs.python.org/issue8612 created Zaar.Hai Decimal module flags undetermined when a signal is trapped. 2010-05-04 http://bugs.python.org/issue8613 created mark.dickinson test_gzip fails on OS X 2010-05-04 CLOSED http://bugs.python.org/issue8614 created mark.dickinson turtle.py - backport of 3.1 features 2010-05-04 CLOSED http://bugs.python.org/issue8615 created gregorlingl patch Changes to content of Demo/turtle 2010-05-04 http://bugs.python.org/issue8616 created gregorlingl patch Non-existent variables documented 2010-05-04 http://bugs.python.org/issue8617 created dabrahams test_winsound failing on Windows Server 2008 2010-05-05 http://bugs.python.org/issue8618 created brian.curtin Doc bug for urllib.request._urlopener in Python 3.1+ 2010-05-05 CLOSED http://bugs.python.org/issue8619 created sri wrong truncation of line in Cmd.cmd 2010-05-05 http://bugs.python.org/issue8620 created omatt uuid.uuid4() generates non-unique values on OSX 2010-05-05 CLOSED http://bugs.python.org/issue8621 created yig patch Add PYTHONFSENCODING environment variable 2010-05-05 http://bugs.python.org/issue8622 created lemburg Aliasing warnings in socketmodule.c 2010-05-05 http://bugs.python.org/issue8623 created pitrou Aliasing warnings in multiprocessing.c 2010-05-05 http://bugs.python.org/issue8624 created pitrou --with-pydebug builds now include -O2 by default 2010-05-05 CLOSED http://bugs.python.org/issue8625 created mark.dickinson patch TypeError: rsplit() takes no keyword arguments 2010-05-05 http://bugs.python.org/issue8626 created dabrahams easy Unchecked PyErr_WarnPy3k return value in Objects/typeobject.c 2010-05-05 http://bugs.python.org/issue8627 created mark.dickinson Incorrect numbers.Complex.imag documentation 2010-05-05 CLOSED http://bugs.python.org/issue8628 created durban Fix test_ssl failures under 2.6/3.1 with OpenSSL 1.0.0 2010-05-05 CLOSED http://bugs.python.org/issue8629 created pitrou patch Keepends param in codec readline(s) 2010-05-05 http://bugs.python.org/issue8630 created amccampos subprocess.Popen.communicate(...) hangs on Windows 2010-05-05 http://bugs.python.org/issue8631 created Alex Quinn subprocess doesn't handle Windows built-in commands as os.syst 2010-05-05 CLOSED http://bugs.python.org/issue8632 created Alex Quinn tarfile doesn't support undecodable filename in PAX format 2010-05-05 http://bugs.python.org/issue8633 created haypo get method for dbm interface 2010-05-06 http://bugs.python.org/issue8634 created Kain94 enumerate() docstring doesn't cover optional start argument 2010-05-06 http://bugs.python.org/issue8635 created ncoghlan easy enumerate() test cases do not cover optional start argument 2010-05-06 http://bugs.python.org/issue8636 created scott.dial patch pydoc should respect MANPAGER over PAGER. 2010-05-06 http://bugs.python.org/issue8637 created jsbronder patch Remove suggestion for name mangling from the tutorial 2010-05-06 http://bugs.python.org/issue8638 created LucianU Allow callable objects in inspect.getargspec 2010-05-06 http://bugs.python.org/issue8639 created gsakkis subprocess: add envb argument to Popen constructor (Python3, P 2010-05-06 http://bugs.python.org/issue8640 created haypo IDLE 3 doesn't highlights b"", but u"" 2010-05-07 http://bugs.python.org/issue8641 created puzzlet json.loads description 2010-05-07 http://bugs.python.org/issue8642 created mft incorrect timedelta yielded by two on-the-fly nows subtraction 2010-05-07 CLOSED http://bugs.python.org/issue8643 created mykhal timedelta.total_seconds needlessly inaccurate, especially for 2010-05-07 http://bugs.python.org/issue8644 created mark.dickinson PyUnicode_AsEncodedObject is undocumented 2010-05-07 http://bugs.python.org/issue8645 created stutzbach PyUnicode_EncodeDecimal is undocumented 2010-05-07 http://bugs.python.org/issue8646 created stutzbach Issues Now Closed (55) ______________________ Bug in range() function for large values 6 days http://bugs.python.org/issue1533 mark.dickinson patch, 26backport Adding start to enumerate() 724 days http://bugs.python.org/issue2831 scott.dial patch MacOS X framework install to non-standard directory fails 622 days http://bugs.python.org/issue3646 ronaldoussoren patch shutil.copyfile() leaks file descriptors when disk fills 546 days http://bugs.python.org/issue4265 haypo patch GC stats not accurate because of debug overhead 501 days http://bugs.python.org/issue4687 pitrou patch Strange behavior when I logout() with IMAP4_SSL 403 days http://bugs.python.org/issue5565 r.david.murray email.encoders.encode_7or8bit(): typo "iso-2202". "iso-2022" 146 days http://bugs.python.org/issue7472 r.david.murray patch, easy random.choice should accept a set as input 141 days http://bugs.python.org/issue7522 Thomas.Dybdahl.Ahle copyright clarification for audiotest.au 103 days http://bugs.python.org/issue7755 barry needs review Minor bug in 2.6.4 related to cleanup at end of program 92 days http://bugs.python.org/issue7835 cburroughs patch, easy platform module doesn't detect Windows 7 89 days http://bugs.python.org/issue7863 lemburg patch, needs review io close() swallowing exceptions 89 days http://bugs.python.org/issue7865 pakal patch urlparse.urlsplit mishandles novel schemes 84 days http://bugs.python.org/issue7904 merwok remove leftover macos9 support code 83 days http://bugs.python.org/issue7908 ronaldoussoren patch, needs review Allow Arbitrary OpenID providers in this bug tracker 70 days http://bugs.python.org/issue8008 jcea configure: allow user-provided CFLAGS to override AC_PROG_CC d 0 days http://bugs.python.org/issue8211 lemburg patch message in unittest traceb 30 days http://bugs.python.org/issue8313 haypo patch None shouldn't be passed to traceback.format_exception 24 days http://bugs.python.org/issue8388 michael.foord tarfile: use surrogates for undecode fields 23 days http://bugs.python.org/issue8390 haypo patch Set operations don't work for dictionary views 22 days http://bugs.python.org/issue8404 rhettinger patch unittest Module Problem with different Kinds of Invocation 18 days http://bugs.python.org/issue8454 michael.foord raise test_support.TestSkipped() is used outside main() / test 17 days http://bugs.python.org/issue8462 haypo asyncore.dispatcher's __getattr__ method produces confusing ef 15 days http://bugs.python.org/issue8483 giampaolo.rodola patch regrtest: add a minimal "progress bar" 8 days http://bugs.python.org/issue8560 haypo patch Always run regrtest.py with -bb 2 days http://bugs.python.org/issue8565 haypo patch 2to3 should run under python 2.5 2 days http://bugs.python.org/issue8566 jyasskin patch, patch decimal module doesn't respect precedence rules for exceptiona 5 days http://bugs.python.org/issue8567 mark.dickinson patch Buggy _strerror in asyncore 7 days http://bugs.python.org/issue8573 giampaolo.rodola easy Problem urllib2.URLError 0 days http://bugs.python.org/issue8580 r.david.murray Logging handlers do not handle double-closing very well 3 days http://bugs.python.org/issue8581 vinay.sajip patch urllib.urlretrieve fails with ValueError: Invalid format strin 0 days http://bugs.python.org/issue8582 orsenthil patch test_imp.py test failures on Py3K Mac OS X 3 days http://bugs.python.org/issue8586 michael.foord test_import.py test failures on Py3K Mac OS X 0 days http://bugs.python.org/issue8587 michael.foord Improve c-api/arg.rst 2 days http://bugs.python.org/issue8593 pitrou patch crypt blowfish 'ignores' salt 1 days http://bugs.python.org/issue8596 mark.dickinson _execvpe behaves inconsistently when PATH includes a filename 0 days http://bugs.python.org/issue8599 r.david.murray test_gdb failures 3 days http://bugs.python.org/issue8600 pitrou bz2.BZ2File should support "with" protocol per PEP 343 0 days http://bugs.python.org/issue8601 Matt.Wartell Create a bytes version of os.environ and getenvb() 4 days http://bugs.python.org/issue8603 haypo patch OSF is not supported anymore 0 days http://bugs.python.org/issue8606 jcea OSX: duplicate -arch flags in CFLAGS breaks sysconfig 0 days http://bugs.python.org/issue8607 r.david.murray fix_import prefixes "." to already relative imports 0 days http://bugs.python.org/issue8608 r.david.murray patch itertools: problem with nested groupby, list() 0 days http://bugs.python.org/issue8609 nicki test_gzip fails on OS X 0 days http://bugs.python.org/issue8614 mark.dickinson turtle.py - backport of 3.1 features 0 days http://bugs.python.org/issue8615 benjamin.peterson patch Doc bug for urllib.request._urlopener in Python 3.1+ 1 days http://bugs.python.org/issue8619 r.david.murray uuid.uuid4() generates non-unique values on OSX 0 days http://bugs.python.org/issue8621 ronaldoussoren patch --with-pydebug builds now include -O2 by default 0 days http://bugs.python.org/issue8625 haypo patch Incorrect numbers.Complex.imag documentation 0 days http://bugs.python.org/issue8628 mark.dickinson Fix test_ssl failures under 2.6/3.1 with OpenSSL 1.0.0 1 days http://bugs.python.org/issue8629 pitrou patch subprocess doesn't handle Windows built-in commands as os.syst 0 days http://bugs.python.org/issue8632 Alex Quinn incorrect timedelta yielded by two on-the-fly nows subtraction 0 days http://bugs.python.org/issue8643 mark.dickinson bdist_deb - Debian packager 440 days http://bugs.python.org/issue1054967 astraw patch URI parsing library 8 days http://bugs.python.org/issue1462525 orsenthil patch Use Py_ssize_t for rangeobject members 1354 days http://bugs.python.org/issue1540617 mark.dickinson patch Top Issues Most Discussed (10) ______________________________ 42 Bug in range() function for large values 6 days closed http://bugs.python.org/issue1533 31 Create a bytes version of os.environ and getenvb() 4 days closed http://bugs.python.org/issue8603 21 test_support.find_unused_port can cause socket conflicts on Win 7 days open http://bugs.python.org/issue8576 16 test_imp.py test failures on Py3K Mac OS X 3 days closed http://bugs.python.org/issue8586 15 Create fsencode() and fsdecode() functions in os.path 14 days open http://bugs.python.org/issue8514 13 Python3/POSIX: errors if file system encoding is None 3 days open http://bugs.python.org/issue8610 12 Adding an atomic FS write API 4 days open http://bugs.python.org/issue8604 12 Add exception logging function to syslog module 45 days open http://bugs.python.org/issue8214 11 message in unittest traceba 30 days closed http://bugs.python.org/issue8313 9 httplib getheader() throws error instead of default 8 days open http://bugs.python.org/issue8572 From alexander.belopolsky at gmail.com Fri May 7 18:46:23 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 7 May 2010 12:46:23 -0400 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: References: Message-ID: On Fri, May 7, 2010 at 11:02 AM, Antoine Pitrou wrote: > VanL gmail.com> writes: .. >> While I was at it, I also added a nice repr. Would this group be >> interested in a patch, or is this not interesting? > > It would be more useful to provide equality, hashing and repr to partial itself, > rather than a subclass. Feel free to propose a patch :) While you are at it, you might want to take a look at http://bugs.python.org/issue7830, "Flatten nested functools.partial." For your purposes you will need to decide whether partial(partial(f, 1), 2) and partial(f, 1, 2) compare equal or not. Most likely with the current implementation the answer will be "no," which may not be ideal. From steve at pearwood.info Fri May 7 19:41:58 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 8 May 2010 03:41:58 +1000 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: References: Message-ID: <201005080342.00028.steve@pearwood.info> On Sat, 8 May 2010 02:07:55 am Rob Cliffe wrote: > Sorry to grouse, but isn't this maybe being a bit too clever? > Using your example, > p1 = partial(operator.add) > is creating a callable, p1, i.e. a sort of function. Yes I know > technically it's not a function, but it behaves very much like one. > > Now, if I write > > def f1(x,y): return x+y > def f2(x,y): return x+y > > I don't expect f1==f2 to be True, even though f1 and f2 behave in > exactly the same way, > and indeed it is not. I do expect f1==f2, and I'm (mildly) disappointed that they're not. [...] > Similarly, if you wanted p1==p2, why not write > > p1 = partial(operator.add) > p2 = p1 I thought the OP gave a use-case. He's generating "jobs" (partial applied to a callable and arguments), and wanted to avoid duplicated jobs. I think it is reasonable to expect that partial(operator.add, 2) compares equal to partial(operator.add, 2). I don't think he's suggesting it should compare equal to partial(lambda x,y: x+y, 2). +0.5 on comparing equal. +1 on a nicer repr for partial objects. -- Steven D'Aprano From steve at holdenweb.com Fri May 7 19:57:06 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 07 May 2010 13:57:06 -0400 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: <201005080342.00028.steve@pearwood.info> References: <201005080342.00028.steve@pearwood.info> Message-ID: <4BE45472.6040406@holdenweb.com> Steven D'Aprano wrote: > On Sat, 8 May 2010 02:07:55 am Rob Cliffe wrote: >> Sorry to grouse, but isn't this maybe being a bit too clever? >> Using your example, >> p1 = partial(operator.add) >> is creating a callable, p1, i.e. a sort of function. Yes I know >> technically it's not a function, but it behaves very much like one. >> >> Now, if I write >> >> def f1(x,y): return x+y >> def f2(x,y): return x+y >> >> I don't expect f1==f2 to be True, even though f1 and f2 behave in >> exactly the same way, >> and indeed it is not. > > I do expect f1==f2, and I'm (mildly) disappointed that they're not. > How about def f1(x, y): return x+y def f2(x, y): return y+x As you know, there are limits to everything. It seems to me that while pure mathematics can (sometime) easily determine functional equivalence, once you get to code it's a lot harder because there are semantic constraints that don't apply in pure mathematics. > > [...] >> Similarly, if you wanted p1==p2, why not write >> >> p1 = partial(operator.add) >> p2 = p1 > > I thought the OP gave a use-case. He's generating "jobs" (partial > applied to a callable and arguments), and wanted to avoid duplicated > jobs. > > I think it is reasonable to expect that partial(operator.add, 2) > compares equal to partial(operator.add, 2). I don't think he's > suggesting it should compare equal to partial(lambda x,y: x+y, 2). > Which absence, presumably, also mildly disappoints you? regards Steve > +0.5 on comparing equal. > +1 on a nicer repr for partial objects. > > > -- Steve Holden +1 571 484 6266 +1 800 494 3119 See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ From fuzzyman at voidspace.org.uk Fri May 7 20:01:46 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 07 May 2010 20:01:46 +0200 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: <4BE45472.6040406@holdenweb.com> References: <201005080342.00028.steve@pearwood.info> <4BE45472.6040406@holdenweb.com> Message-ID: <4BE4558A.3080001@voidspace.org.uk> On 07/05/2010 19:57, Steve Holden wrote: > Steven D'Aprano wrote: > >> On Sat, 8 May 2010 02:07:55 am Rob Cliffe wrote: >> >>> Sorry to grouse, but isn't this maybe being a bit too clever? >>> Using your example, >>> p1 = partial(operator.add) >>> is creating a callable, p1, i.e. a sort of function. Yes I know >>> technically it's not a function, but it behaves very much like one. >>> >>> Now, if I write >>> >>> def f1(x,y): return x+y >>> def f2(x,y): return x+y >>> >>> I don't expect f1==f2 to be True, even though f1 and f2 behave in >>> exactly the same way, >>> and indeed it is not. >>> >> I do expect f1==f2, and I'm (mildly) disappointed that they're not. >> >> > How about > > def f1(x, y): return x+y > def f2(x, y): return y+x > > As you know, there are limits to everything. It seems to me that while > pure mathematics can (sometime) easily determine functional equivalence, > once you get to code it's a lot harder because there are semantic > constraints that don't apply in pure mathematics. > Sure, but in CPython at least you *could* detect *identical* functions (but not functions that do the same thing in a different way). All the information is exposed - it would mean comparing bytecode though (plus signature etc) and is not likely to be portable to other implementations. Partials that wrap the *same* function (based on identity) comparing equal seems useful enough to me. Michael -- http://www.ironpythoninaction.com/ From brett at python.org Fri May 7 20:38:07 2010 From: brett at python.org (Brett Cannon) Date: Fri, 7 May 2010 11:38:07 -0700 Subject: [Python-Dev] What's New text on future maintenance In-Reply-To: <20100507160908.GB21021@amk-desktop.matrixgroup.net> References: <20100507015035.GA50210@andrew-kuchlings-macbook.local> <4BE3E2F1.4040907@gmail.com> <20100507160908.GB21021@amk-desktop.matrixgroup.net> Message-ID: On Fri, May 7, 2010 at 09:09, A.M. Kuchling wrote: > On Fri, May 07, 2010 at 07:52:49PM +1000, Nick Coghlan wrote: > > 3.x). I'll take a stab at a more accurate rationale: > > Thanks! I've applied the scalpel and reduced it to: > > * A policy decision was made to silence warnings only of interest to > developers by default. :exc:`DeprecationWarning` and its > descendants are now ignored unless otherwise requested, preventing > users from seeing warnings triggered by an application. (Carried > out in :issue:`7319`.) > > In previous releases, :exc:`DeprecationWarning` messages were > enabled by default, providing Python developers with a clear > indication of where their code may break in a future major version > of Python. > > However, there are increasingly many users of Python-based > applications who are not directly involved in the development of > those applications. :exc:`DeprecationWarning` messages are > irrelevant to such users, making them worry about an application > that's actually working correctly and burdening the developers of > these applications with responding to these concerns. > > You can re-enable display of :exc:`DeprecationWarning` messages by > running Python with the :option:`-Wdefault` (short form: > :option:`-Wd`) switch, or you can add > ``warnings.simplefilter('default')`` to your code. > > That sounds good to me. > Benjamin suggested being very definite about a 5-year maintenance > period, but I don't want to write any checks our butt can't cash, so > I've left the text as "Maintenance releases for Python 2.7 will > probably be made for 5 years." An alternative formulation might say > it will be maintained for the next two 3.x releases, not the next one > as usual. > > I thought about Ben Finney's suggestion to not give a timespan and > describe the conditions for 2.x maintenance continuing, but those > conditions are complicated to describe -- if 3.x doesn't catch on? if > the 3.x transition is slow? if there's a significant 2.x user base > that remains? if someone starts a 2.x maintenance team? -- and might > be a confusing tangle of what-if statements. Why can't we simply say that "we plan to support Python 2.7 beyond the typical two years for bugfix releases"? It doesn't tie us to anything but still lets people know our intentions. We don't have to worry about every possible scenario now (e.g. 3.x gets no more traction or some other rare event) and saying we plan on long term support but don't know for how long is completely truthful; we have no timeline on how long we are willing to keep 2.7 afloat beyond the fact that we plan to do it longer than normal. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri May 7 21:13:50 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 8 May 2010 05:13:50 +1000 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: <4BE45472.6040406@holdenweb.com> References: <201005080342.00028.steve@pearwood.info> <4BE45472.6040406@holdenweb.com> Message-ID: <201005080513.50741.steve@pearwood.info> On Sat, 8 May 2010 03:57:06 am Steve Holden wrote: > Steven D'Aprano wrote: > > On Sat, 8 May 2010 02:07:55 am Rob Cliffe wrote: > >> Sorry to grouse, but isn't this maybe being a bit too clever? > >> Using your example, > >> p1 = partial(operator.add) > >> is creating a callable, p1, i.e. a sort of function. Yes I know > >> technically it's not a function, but it behaves very much like > >> one. > >> > >> Now, if I write > >> > >> def f1(x,y): return x+y > >> def f2(x,y): return x+y > >> > >> I don't expect f1==f2 to be True, even though f1 and f2 behave in > >> exactly the same way, > >> and indeed it is not. > > > > I do expect f1==f2, and I'm (mildly) disappointed that they're not. > > How about > > def f1(x, y): return x+y > def f2(x, y): return y+x > > As you know, there are limits to everything. It seems to me that > while pure mathematics can (sometime) easily determine functional > equivalence, once you get to code it's a lot harder because there are > semantic constraints that don't apply in pure mathematics. This being Python, we can't assume x+y is always equal to y+x, so in your example I wouldn't expect them to be equal. And thus I avoid your trap :) What you say is correct in general. I understand the arguments against making function equality more sophisticated than just identity testing: there aren't many use-cases for it, and it is potentially a lot of work depending on how clever you try to be. But if it came for free, it would be a sweet trick to impress your friends and scare your enemies. [...] > > I think it is reasonable to expect that partial(operator.add, 2) > > compares equal to partial(operator.add, 2). I don't think he's > > suggesting it should compare equal to partial(lambda x,y: x+y, 2). > > Which absence, presumably, also mildly disappoints you? Only in the sense that in a perfect world where language features had benefits but no costs, I would expect nothing less. -- Steven D'Aprano From pjenvey at underboss.org Fri May 7 21:26:10 2010 From: pjenvey at underboss.org (Philip Jenvey) Date: Fri, 7 May 2010 12:26:10 -0700 Subject: [Python-Dev] What's New text on future maintenance In-Reply-To: <20100507160908.GB21021@amk-desktop.matrixgroup.net> References: <20100507015035.GA50210@andrew-kuchlings-macbook.local> <4BE3E2F1.4040907@gmail.com> <20100507160908.GB21021@amk-desktop.matrixgroup.net> Message-ID: On May 7, 2010, at 9:09 AM, A.M. Kuchling wrote: > You can re-enable display of :exc:`DeprecationWarning` messages by > running Python with the :option:`-Wdefault` (short form: > :option:`-Wd`) switch, or you can add > ``warnings.simplefilter('default')`` to your code. The new PYTHONWARNINGS env var also needs mentioning here, how's about: You can re-enable display of :exc:`DeprecationWarning` messages by either running Python with the :option:`-Wdefault` (short form: :option:`-Wd`) switch or by setting the :envvar:`PYTHONWARNINGS` environment variable to "default" (or "d") before running Python. Python code can also re-enable them by calling ``warnings.simplefilter('default')``. -- Philip Jenvey From martin at v.loewis.de Fri May 7 21:45:23 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 07 May 2010 21:45:23 +0200 Subject: [Python-Dev] What's New text on future maintenance In-Reply-To: <20100507153930.GA21021@amk-desktop.matrixgroup.net> References: <20100507015035.GA50210@andrew-kuchlings-macbook.local> <4BE3C178.2090800@v.loewis.de> <20100507153930.GA21021@amk-desktop.matrixgroup.net> Message-ID: <4BE46DD3.6060806@v.loewis.de> A.M. Kuchling wrote: > On Fri, May 07, 2010 at 09:30:00AM +0200, "Martin v. L?wis" wrote: >> I agree with Terry: how did you arrive at the 4 years for 2.x releases? >> Bug fixes releases stopped after the next feature release being made, >> which gave (counting between initial release and last bug fix release): > > I used the initial release date of 2.4 and 2.5 and the dates of the > last patchlevel releases. That may well be distorted by the recent > spasm of activity that led to 2.5.4. Ah. But the most recent 2.4 and 2.5 patchlevel releases are *not* bugfix releases. They only provide security fixes, and I plan to provide them for a period of five years (after the initial release). Regards, Martin From van.lindberg at gmail.com Fri May 7 21:53:01 2010 From: van.lindberg at gmail.com (VanL) Date: Fri, 07 May 2010 14:53:01 -0500 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: <201005080342.00028.steve@pearwood.info> References: <201005080342.00028.steve@pearwood.info> Message-ID: On 5/7/2010 12:41 PM, Steven D'Aprano wrote: >> Now, if I write >> >> def f1(x,y): return x+y >> def f2(x,y): return x+y >> >> I don't expect f1==f2 to be True, even though f1 and f2 behave in >> exactly the same way, and indeed it is not. This is not what I am getting after; these (IMO) should compare unequal. I took a fairly conservative line, testing for identity of functions and equality of arguments when converted to a normal form: def __eq__(self, other): try: return ((self.func == other.func) and (self.args == other.args) and (self.keywords == other.keywords)) except AttributeError: return False def __hash__(self): if self.keywords: sorted_keys = sorted(self.keywords.keys()) values = [self.keywords[k] for k in sorted_keys] kwhash = tuple(sorted_keys + values) else: kwhash = None return hash(tuple((self.func, self.args, kwhash))) It may be slightly controversial that I read out all the keys and values and convert them to a normal form for the hash. This is both to get around hashing issues (keywords is a dict, even though it is read-only) and to allow for a dict-like object to be provided as an argument (which happens in my application sometimes). The repr is slightly long, too, but it was designed to support (weakly) the idea that obj == exec(repr(obj)). Its not necessary, but I always like it when that is true. def __repr__(self): if self.args: argstr = '*%s' % repr(tuple(self.args)) else: argstr = '' if self.keywords: kwstr = '**%s' % self.keywords else: kwstr = '' if argstr and kwstr: argstr += ', ' classname = self.__class__.__name__ funcname = self.func.__name__ if argstr or kwstr: funcname += ', ' return '%s(%s%s%s)' % (classname, funcname, argstr, kwstr) I make no effort to look for functional equality between two different functions. too many side effects could go off trying to make that work, and it would still be only inconsistently right. (IIRC, doing it completely is NP-hard.) Thanks, Van From ijmorlan at uwaterloo.ca Fri May 7 22:15:35 2010 From: ijmorlan at uwaterloo.ca (Isaac Morland) Date: Fri, 7 May 2010 16:15:35 -0400 (EDT) Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: References: <201005080342.00028.steve@pearwood.info> Message-ID: On Fri, 7 May 2010, VanL wrote: > I make no effort to look for functional equality between two different > functions. too many side effects could go off trying to make that work, > and it would still be only inconsistently right. (IIRC, doing it > completely is NP-hard.) Actually, doing it completely is flat-out impossible. Godel, Halting Problem, and all that. So you don't need to apologize for not doing it ;-) Isaac Morland CSCF Web Guru DC 2554C, x36650 WWW Software Specialist From tjreedy at udel.edu Fri May 7 22:36:04 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 07 May 2010 16:36:04 -0400 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: References: <201005080342.00028.steve@pearwood.info> Message-ID: There is obviously interest in your proposal. I suggest you open an issue on the tracker. Make it a library feature request for 3.2. Upload the patch as a separate file. I would include the subclass form in the post to show the intneded effect, for discussion, and also so people can test and use with current installations without patching. When you have done so, post here in a followup with the link. Terry Jan Reedy From vincent at vincentdavis.net Sat May 8 00:02:00 2010 From: vincent at vincentdavis.net (Vincent Davis) Date: Fri, 7 May 2010 16:02:00 -0600 Subject: [Python-Dev] audience-instructors for Teach Me Python Bugfixing needed In-Reply-To: References: Message-ID: On Thu, May 6, 2010 at 9:29 AM, Antoine Pitrou wrote: > > Scanning through open issues will also give you a general idea of what > kind of functionalities are looking for improvement, or need fixing. > > (you can create a new issue and start tackling it yourself, too) > > As a wanabe Dev I think the hardest thing is to find an open issue I can actually fix and to have a mentor to help make sure I don't miss something I did not know about. Please record the "Teach Me" session if it happens. (audio and/or video) > > *Vincent Davis 720-301-3003 * vincent at vincentdavis.net my blog | LinkedIn -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat May 8 01:50:42 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 08 May 2010 09:50:42 +1000 Subject: [Python-Dev] What's New text on future maintenance In-Reply-To: <20100507160908.GB21021@amk-desktop.matrixgroup.net> References: <20100507015035.GA50210@andrew-kuchlings-macbook.local> <4BE3E2F1.4040907@gmail.com> <20100507160908.GB21021@amk-desktop.matrixgroup.net> Message-ID: <4BE4A752.2010302@gmail.com> A.M. Kuchling wrote: > On Fri, May 07, 2010 at 07:52:49PM +1000, Nick Coghlan wrote: >> 3.x). I'll take a stab at a more accurate rationale: > > Thanks! I've applied the scalpel and reduced it to: I saw this go by on the checkin list - looks good (although Philip is right about mentioning PYTHONWARNINGS). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sat May 8 01:54:52 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 08 May 2010 09:54:52 +1000 Subject: [Python-Dev] audience-instructors for Teach Me Python Bugfixing needed In-Reply-To: References: Message-ID: <4BE4A84C.9080805@gmail.com> Vincent Davis wrote: > As a wanabe Dev I think the hardest thing is to find an open issue I can > actually fix and to have a mentor to help make sure I don't miss > something I did not know about. Does the "easy" tag help with that at all? It's intended to mark issues that aren't delving into deep dark corners of the interpreter or standard library, but when you've been doing this for a long time one's judgment of what's easy and what's difficult can get a little askew. (e.g. I recently marked the task of fixing the enumerate docstring as easy, but that does still require knowing how docstrings work for C code) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From dave at boostpro.com Sat May 8 04:49:06 2010 From: dave at boostpro.com (David Abrahams) Date: Fri, 07 May 2010 22:49:06 -0400 Subject: [Python-Dev] urlparse.urlunsplit should be smarter about + Message-ID: This is a bug report. bugs.python.org seems to be down. >>> from urlparse import * >>> urlunsplit(urlsplit('git+file:///foo/bar/baz')) git+file:/foo/bar/baz Note the dropped slashes after the colon. -- Dave Abrahams Meet me at BoostCon: http://www.boostcon.com BoostPro Computing http://www.boostpro.com From orsenthil at gmail.com Sat May 8 06:04:04 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Sat, 8 May 2010 09:34:04 +0530 Subject: [Python-Dev] urlparse.urlunsplit should be smarter about + In-Reply-To: References: Message-ID: On Sat, May 8, 2010 at 8:19 AM, David Abrahams wrote: > > This is a bug report. ?bugs.python.org seems to be down. Tracked here: http://bugs.python.org/issue8656 > ?>>> urlunsplit(urlsplit('git+file:///foo/bar/baz')) Is 'git+file' a valid protocol? Or was it just your example? I don't see any reason for it to be invalid but I don't find authoritative references either. -- Senthil From ddborowitz at gmail.com Sat May 8 06:12:28 2010 From: ddborowitz at gmail.com (David Borowitz) Date: Fri, 7 May 2010 21:12:28 -0700 Subject: [Python-Dev] urlparse.urlunsplit should be smarter about + In-Reply-To: References: Message-ID: On Fri, May 7, 2010 at 21:04, Senthil Kumaran wrote: > On Sat, May 8, 2010 at 8:19 AM, David Abrahams wrote: > > > > This is a bug report. bugs.python.org seems to be down. > > Tracked here: http://bugs.python.org/issue8656 > > > >>> urlunsplit(urlsplit('git+file:///foo/bar/baz')) > > Is 'git+file' a valid protocol? Or was it just your example? > I don't see any reason for it to be invalid but I don't find > authoritative references either. > RFC 3986 is pretty clear on allowing '+' in scheme names, in principle if not necessarily in practice: http://tools.ietf.org/html/rfc3986#section-3.1 > -- > Senthil > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ddborowitz%40gmail.com > -- It is better to be quotable than to be honest. -Tom Stoppard Borowitz -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sat May 8 16:47:09 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 08 May 2010 23:47:09 +0900 Subject: [Python-Dev] urlparse.urlunsplit should be smarter about + In-Reply-To: References: Message-ID: <87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp> David Abrahams writes: > > This is a bug report. bugs.python.org seems to be down. > > >>> from urlparse import * > >>> urlunsplit(urlsplit('git+file:///foo/bar/baz')) > git+file:/foo/bar/baz > > Note the dropped slashes after the colon. That's clearly wrong, but what does "+" have to to do with it? AFAIK, the only thing special about + in scheme names is that it's not allowed as the first character. From john.arbash.meinel at gmail.com Sat May 8 18:04:47 2010 From: john.arbash.meinel at gmail.com (John Arbash Meinel) Date: Sat, 08 May 2010 11:04:47 -0500 Subject: [Python-Dev] urlparse.urlunsplit should be smarter about + In-Reply-To: <87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4BE58B9F.6080604@gmail.com> Stephen J. Turnbull wrote: > David Abrahams writes: > > > > This is a bug report. bugs.python.org seems to be down. > > > > >>> from urlparse import * > > >>> urlunsplit(urlsplit('git+file:///foo/bar/baz')) > > git+file:/foo/bar/baz > > > > Note the dropped slashes after the colon. > > That's clearly wrong, but what does "+" have to to do with it? AFAIK, > the only thing special about + in scheme names is that it's not > allowed as the first character. Don't you need to register the "git+file:///" url for urlparse to properly split it? if protocol not in urlparse.uses_netloc: urlparse.uses_netloc.append(protocol) John =:-> From exarkun at twistedmatrix.com Sat May 8 19:16:57 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sat, 08 May 2010 17:16:57 -0000 Subject: [Python-Dev] Parallel test execution on buildbot Message-ID: <20100508171657.1660.1545567012.divmod.xquotient.35@localhost.localdomain> Hi all, Has anyone considered using regrtest's -j option in the buildbot configuration to speed up the test runs? Antoine Pitrou pointed out that even for single CPU slaves, this could be a win due to the number of tests that spend time sleeping or waiting on I/O. And on slaves with multiple CPUs it would clearly be even better. eg, I don't know what hardware is actually in the Solaris slave (bot loewis-sun), but if it has the full 4 UltraSPARCs that it could, then running with -j4 or -j5 there might bring its runtime down from nearly 100 minutes to 20 or 25 - competitive with some of the more reasonable slaves. Jean-Paul From solipsis at pitrou.net Sat May 8 19:48:36 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 8 May 2010 19:48:36 +0200 Subject: [Python-Dev] Parallel test execution on buildbot References: <20100508171657.1660.1545567012.divmod.xquotient.35@localhost.localdomain> Message-ID: <20100508194836.40160211@pitrou.net> Hi, > Has anyone considered using regrtest's -j option in the buildbot > configuration to speed up the test runs? Perhaps some buildbots are doing other useful tasks, in addition to simply building Python. This should probably be a case by case setting. I don't know how easy it is to add specific options to a buildslave. One small issue would be that, currently, when a buildbot hangs, you know which test it is hung in since it's the last one displayed. This isn't true with -j (but perhaps we can improve the runner to fix this). Regards Antoine. From benjamin at python.org Sat May 8 20:56:01 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 8 May 2010 13:56:01 -0500 Subject: [Python-Dev] [RELEASED] Python 2.7 beta 2 Message-ID: On behalf of the Python development team, I'm elated to announce the second beta release of Python 2.7. Python 2.7 is scheduled (by Guido and Python-dev) to be the last major version in the 2.x series. 2.7 will have an extended period of bugfix maintenance. 2.7 includes many features that were first released in Python 3.1. The faster io module, the new nested with statement syntax, improved float repr, set literals, dictionary views, and the memoryview object have been backported from 3.1. Other features include an ordered dictionary implementation, unittests improvements, a new sysconfig module, and support for ttk Tile in Tkinter. For a more extensive list of changes in 2.7, see http://doc.python.org/dev/whatsnew/2.7.html or Misc/NEWS in the Python distribution. To download Python 2.7 visit: http://www.python.org/download/releases/2.7/ While this is a development release and is thus not suitable for production use, we encourage Python application and library developers to test the release with their code and report any bugs they encounter to: http://bugs.python.org/ 2.7 documentation can be found at: http://docs.python.org/2.7/ Enjoy! -- Benjamin Peterson 2.7 Release Manager benjamin at python.org (on behalf of the entire python-dev team and 2.7's contributors) From martin at v.loewis.de Sat May 8 22:08:20 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 08 May 2010 22:08:20 +0200 Subject: [Python-Dev] Parallel test execution on buildbot In-Reply-To: <20100508171657.1660.1545567012.divmod.xquotient.35@localhost.localdomain> References: <20100508171657.1660.1545567012.divmod.xquotient.35@localhost.localdomain> Message-ID: <4BE5C4B4.7090703@v.loewis.de> > Has anyone considered using regrtest's -j option in the buildbot > configuration to speed up the test runs? Yes, I did. I turned it off again when the tests started failing because of it. Regards, Martin From tjreedy at udel.edu Sun May 9 00:38:21 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 08 May 2010 18:38:21 -0400 Subject: [Python-Dev] bugs.python.org intermittently not working. Message-ID: Saturday eve (us, eastern). Last night I had intermittent problems with bugs.python.org: issues not being fetched, submissions not being recorded. David Abrahams reported 'bugs.python.org seems to be down' is his urlparse thread. Right not, I have a fetch and submission 'loading' for over a minute, after fetching an issue a minute before. The new beta2 page came up with no problems. The submission just stopped with Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request POST /issue8641. Reason: Error reading from remote server Apache/2.2.9 (Debian) mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.9 OpenSSL/0.9.8g Server at bugs.python.org Port 80 From ncoghlan at gmail.com Sun May 9 03:32:42 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 09 May 2010 11:32:42 +1000 Subject: [Python-Dev] Parallel test execution on buildbot In-Reply-To: <4BE5C4B4.7090703@v.loewis.de> References: <20100508171657.1660.1545567012.divmod.xquotient.35@localhost.localdomain> <4BE5C4B4.7090703@v.loewis.de> Message-ID: <4BE610BA.7060305@gmail.com> Martin v. L?wis wrote: >> Has anyone considered using regrtest's -j option in the buildbot >> configuration to speed up the test runs? > > Yes, I did. I turned it off again when the tests started failing because > of it. Yeah, a lot of our tests weren't written with parallel execution in mind (e.g. the existence of test_support.TESTFN, using specific ports for test servers). While they've been getting better (e.g. increased use of randomly generated temporary directories over specific filenames, letting the OS assign a free port to servers), I believe there is still a fair bit of work to be done to make them all "parallel execution friendly" (of course, some tests, such as those that try to trigger MemoryError, are inherently parallel execution unfriendly). So yes, we've thought about it, but there's still work to be done before that option can be used without having to worry about false alarms. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From fuzzyman at voidspace.org.uk Sun May 9 11:08:29 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 09 May 2010 11:08:29 +0200 Subject: [Python-Dev] Parallel test execution on buildbot In-Reply-To: <4BE610BA.7060305@gmail.com> References: <20100508171657.1660.1545567012.divmod.xquotient.35@localhost.localdomain> <4BE5C4B4.7090703@v.loewis.de> <4BE610BA.7060305@gmail.com> Message-ID: <4BE67B8D.9030504@voidspace.org.uk> On 09/05/2010 03:32, Nick Coghlan wrote: > Martin v. L?wis wrote: > >>> Has anyone considered using regrtest's -j option in the buildbot >>> configuration to speed up the test runs? >>> >> Yes, I did. I turned it off again when the tests started failing because >> of it. >> > Yeah, a lot of our tests weren't written with parallel execution in mind > (e.g. the existence of test_support.TESTFN, using specific ports for > test servers). > > While they've been getting better (e.g. increased use of randomly > generated temporary directories over specific filenames, letting the OS > assign a free port to servers), I believe there is still a fair bit of > work to be done to make them all "parallel execution friendly" (of > course, some tests, such as those that try to trigger MemoryError, are > inherently parallel execution unfriendly). > > So yes, we've thought about it, but there's still work to be done before > that option can be used without having to worry about false alarms. > FWIW I *usually* run the test suite with parallelization (it is just so much quicker) and these days *rarely* see spurious failures as a result. This is on Mac OS X, YMMV. Michael Foord > Cheers, > Nick. > > -- http://www.ironpythoninaction.com/ From victor.stinner at haypocalc.com Sun May 9 13:49:37 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 9 May 2010 13:49:37 +0200 Subject: [Python-Dev] Parallel test execution on buildbot In-Reply-To: <4BE67B8D.9030504@voidspace.org.uk> References: <20100508171657.1660.1545567012.divmod.xquotient.35@localhost.localdomain> <4BE610BA.7060305@gmail.com> <4BE67B8D.9030504@voidspace.org.uk> Message-ID: <201005091349.37463.victor.stinner@haypocalc.com> Le dimanche 09 mai 2010 11:08:29, Michael Foord a ?crit : > FWIW I *usually* run the test suite with parallelization (it is just so > much quicker) and these days *rarely* see spurious failures as a result. > This is on Mac OS X, YMMV. I use regrtest.py -j 4 on a Intel Quad Core on Linux: 4 tests are really running at the same time, and I only get only one spurious failure: test_ioctl. It should be easy to fix this test. Or we can maybe write a blacklist of tests not compatible with multiprocessing mode (or the opposite, a whitelist of compatible tests). -- Victor Stinner http://www.haypocalc.com/ From martin at v.loewis.de Sun May 9 13:50:47 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 09 May 2010 13:50:47 +0200 Subject: [Python-Dev] Parallel test execution on buildbot In-Reply-To: <4BE67B8D.9030504@voidspace.org.uk> References: <20100508171657.1660.1545567012.divmod.xquotient.35@localhost.localdomain> <4BE5C4B4.7090703@v.loewis.de> <4BE610BA.7060305@gmail.com> <4BE67B8D.9030504@voidspace.org.uk> Message-ID: <4BE6A197.9090808@v.loewis.de> > FWIW I *usually* run the test suite with parallelization (it is just so > much quicker) and these days *rarely* see spurious failures as a result. > This is on Mac OS X, YMMV. I may misremember the details, but IIRC, the multiprocessing tests would fail to terminate on Solaris. This made it unsuitable for buildbot usage. Regards, Martin From stephen at xemacs.org Sun May 9 14:15:38 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 09 May 2010 21:15:38 +0900 Subject: [Python-Dev] urlparse.urlunsplit should be smarter about + In-Reply-To: <4BE58B9F.6080604@gmail.com> References: <87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp> <4BE58B9F.6080604@gmail.com> Message-ID: <87fx21s239.fsf@uwakimon.sk.tsukuba.ac.jp> John Arbash Meinel writes: > Stephen J. Turnbull wrote: > > David Abrahams writes: > > > > > > This is a bug report. bugs.python.org seems to be down. > > > > > > >>> from urlparse import * > > > >>> urlunsplit(urlsplit('git+file:///foo/bar/baz')) > > > git+file:/foo/bar/baz > > > > > > Note the dropped slashes after the colon. > > > > That's clearly wrong, but what does "+" have to to do with it? AFAIK, > > the only thing special about + in scheme names is that it's not > > allowed as the first character. > > Don't you need to register the "git+file:///" url for urlparse to > properly split it? > > if protocol not in urlparse.uses_netloc: > urlparse.uses_netloc.append(protocol) I don't know about the urlparse implementation, but from the point of view of the RFC I think not. Either BCP 35 or RFC 3986 (or maybe both) makes it plain that if the scheme name is followed by "://", the scheme is a hierarchical one. So that URL should parse with an empty authority, and be recomposed the same. I would do this by parsing 'git+file:///foo/bar/baz' to ('git+file', '', '/foo/bar/baz') or something like than, and 'git+file:/foo/bar/baz' to ('git+file', None, '/foo/bar/baz'). I don't see any reason why implementations should abbreviate the empty authority by removing the double slashes, unless specified in the scheme definition. Although my reading of RFC 3986 is that a missing authority (no "//") *should* be dereferenced in the same way as an empty one: If the URI scheme defines a default for host, then that default applies when the host subcomponent is undefined or when the registered name is empty (zero length). (Sec. 3.2.2) I don't see why urlparse should try to enforce that by converting from one to the other. From dickinsm at gmail.com Sun May 9 15:54:06 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sun, 9 May 2010 14:54:06 +0100 Subject: [Python-Dev] Did I miss the decision to untabify all of the C code? In-Reply-To: References: <4BE20625.9070203@trueblade.com> <4BE2146E.6000702@trueblade.com> Message-ID: On Thu, May 6, 2010 at 4:52 AM, Joao S. O. Bueno wrote: > On Wed, May 5, 2010 at 9:59 PM, Eric Smith wrote: >> That's my point. Since it's basically unreviewable, is it smart to do it >> during a beta? > > Hello folks - > I don't think these modifications are that "unreviewable": the > generated binaries have to be exactly the same with the untabified > files don't they? So is a matter of stashing the binaries, applying > the patches, rebuild and check to see if the binaries match. Any > possible script defects undetected by this would be only (C code) > indentation, which could be fixed later. That's not foolproof, though: there are lots of sections of code that will only get compiled on certain platforms, or with certain configure options, etc. Mark From solipsis at pitrou.net Sun May 9 20:33:46 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 9 May 2010 20:33:46 +0200 Subject: [Python-Dev] PEP 7 updated Message-ID: <20100509203346.169facf0@pitrou.net> Hello, The untabification of C files didn't produce any noticeable problem on the buildbots. I've updated PEP 7 with the mention that all C files should be 4-space indented, and removed the obsolete wording about some files being indented with tabs. Regards Antoine. From fw at deneb.enyo.de Sun May 9 21:51:25 2010 From: fw at deneb.enyo.de (Florian Weimer) Date: Sun, 09 May 2010 21:51:25 +0200 Subject: [Python-Dev] [RELEASED] Python 2.7 beta 2 In-Reply-To: (Benjamin Peterson's message of "Sat, 8 May 2010 13:56:01 -0500") References: Message-ID: <87632wx39e.fsf@mid.deneb.enyo.de> * Benjamin Peterson: > http://doc.python.org/dev/whatsnew/2.7.html or Misc/NEWS in the Python > distribution. Something is missing here: "* Multiple context managers in" From dave at boostpro.com Sun May 9 23:19:40 2010 From: dave at boostpro.com (David Abrahams) Date: Sun, 09 May 2010 15:19:40 -0600 Subject: [Python-Dev] urlparse.urlunsplit should be smarter about + In-Reply-To: <4BE58B9F.6080604@gmail.com> References: <87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp> <4BE58B9F.6080604@gmail.com> Message-ID: At Sat, 08 May 2010 11:04:47 -0500, John Arbash Meinel wrote: > > Stephen J. Turnbull wrote: > > David Abrahams writes: > > > > > > This is a bug report. bugs.python.org seems to be down. > > > > > > >>> from urlparse import * > > > >>> urlunsplit(urlsplit('git+file:///foo/bar/baz')) > > > git+file:/foo/bar/baz > > > > > > Note the dropped slashes after the colon. > > > > That's clearly wrong, but what does "+" have to to do with it? AFAIK, > > the only thing special about + in scheme names is that it's not > > allowed as the first character. > > Don't you need to register the "git+file:///" url for urlparse to > properly split it? Yes. But the question is whether urlparse should really be so fragile that every hierarchical scheme needs to be explicitly registered. Surely ending with ?+file? should be sufficient to have it recognized as a file-based scheme -- Dave Abrahams Meet me at BoostCon: http://www.boostcon.com BoostPro Computing http://www.boostpro.com From jon+python-dev at unequivocal.co.uk Mon May 10 02:17:06 2010 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Mon, 10 May 2010 01:17:06 +0100 Subject: [Python-Dev] urlparse.urlunsplit should be smarter about + In-Reply-To: References: <87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp> <4BE58B9F.6080604@gmail.com> Message-ID: <20100510001706.GJ6976@snowy.squish.net> On Sun, May 09, 2010 at 03:19:40PM -0600, David Abrahams wrote: > Yes. But the question is whether urlparse should really be so fragile > that every hierarchical scheme needs to be explicitly registered. > Surely ending with ?+file? should be sufficient to have it recognized > as a file-based scheme How do you figure? From meadori at gmail.com Mon May 10 03:33:14 2010 From: meadori at gmail.com (Meador Inge) Date: Sun, 9 May 2010 20:33:14 -0500 Subject: [Python-Dev] Reindenting patches In-Reply-To: References: <4BDF8289.2090306@holdenweb.com> <4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy> <4BE03130.9020700@trueblade.com> <4BE0A280.8080903@gmail.com> Message-ID: On Wed, May 5, 2010 at 8:10 AM, Antoine Pitrou wrote: > For the record, I've added to the untabify script a patch rewriting option > ("-p") which reindents all patch hunks for C files containing tabs. It > should > minimize manual reformatting work with existing patches. > I just tried '-p' with a local patch that I had created pre-whitespace-fixes. It worked like a charm. Thanks Antoine. -- Meador -------------- next part -------------- An HTML attachment was scrubbed... URL: From orsenthil at gmail.com Mon May 10 07:38:14 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Mon, 10 May 2010 11:08:14 +0530 Subject: [Python-Dev] urlparse.urlunsplit should be smarter about + In-Reply-To: References: <87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp> <4BE58B9F.6080604@gmail.com> Message-ID: <20100510053814.GA3608@remy> On Sun, May 09, 2010 at 03:19:40PM -0600, David Abrahams wrote: > John Arbash Meinel wrote: > > Don't you need to register the "git+file:///" url for urlparse to > > properly split it? > > Yes. But the question is whether urlparse should really be so fragile > that every hierarchical scheme needs to be explicitly registered. > Surely ending with ?+file? should be sufficient to have it recognized > as a file-based scheme Not all urls have the 'authority' component after the scheme. (sip based urls for e.g) urlparse differentiates those by maintaining a list of scheme names which will follow the pattern of parsing, and joining for the urls which have a netloc (or authority component). This is in general according to RFC 3986 itself. Yes,'+' is a valid char in url schemes and svn, svn+ssh will be as per your expectations. But git and git+ssh was missing in there and I attached a patch in issue8657 to include the same. It is rightly a bug in the module. But for any general scheme and assuming '+file' would follow valid authority component, is not something I am sure that should be in urlparse's expected behavior. -- Senthil Do not seek death; death will find you. But seek the road which makes death a fulfillment. -- Dag Hammarskjold From stephen at xemacs.org Mon May 10 08:51:36 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 10 May 2010 15:51:36 +0900 Subject: [Python-Dev] urlparse.urlunsplit should be smarter about + In-Reply-To: References: <87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp> <4BE58B9F.6080604@gmail.com> Message-ID: <878w7ss0zr.fsf@uwakimon.sk.tsukuba.ac.jp> David Abrahams writes: > At Sat, 08 May 2010 11:04:47 -0500, > John Arbash Meinel wrote: > > Don't you need to register the "git+file:///" url for urlparse to > > properly split it? > > Yes. But the question is whether urlparse should really be so fragile > that every hierarchical scheme needs to be explicitly registered. Exactly. And the answer is "no". The RFCs are quite clear that hierarchical schemes are expected to be extremely common, and provide several requirements for how they should be parsed, even by nonvalidating parsers. It's pretty clear to me that urlunsplit(urlsplit('git+file:///foo/bar/baz')) should be the identity. The remaining question is, "Should urlunsplit(urlsplit('git+file:/foo/bar/baz')) be the identity?" I would argue that if git+file is *not* registered, it should be the identity, while there should be an optional registry of schemes which may (or perhaps should) be canonicalized (ie, a *missing* authority would be unsplit as an *empty* authority). > Surely ending with "+file" should be sufficient to have it recognized > as a file-based scheme What's a "file-based scheme"? If you mean an RFC 3986 hierarchical scheme, that is recognized by the presence of the authority portion, which is syntactically defined by the presence of "//" immediately after the scheme ":" terms. No need for any implicit magic. In general, EIBTI applies here. If a registry as described above is implemented, I would argue that canonicalization should not happen implicitly. Nor should validation (eg, error or warning on a URI with a scheme registered as hierarchical but lacking authority, or vice versa). The API should require an explicit statement from the user to invoke those functionalities. It might be useful for the OOWTDI API to canonicalize/validate by default (especially given XSS attacks and the like), but it should be simple for consenting adults to turn that feature off. From stephen at xemacs.org Mon May 10 10:11:12 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 10 May 2010 17:11:12 +0900 Subject: [Python-Dev] urlparse.urlunsplit should be smarter about + In-Reply-To: <20100510053814.GA3608@remy> References: <87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp> <4BE58B9F.6080604@gmail.com> <20100510053814.GA3608@remy> Message-ID: <87632wrxb3.fsf@uwakimon.sk.tsukuba.ac.jp> Senthil Kumaran writes: > Not all urls have the 'authority' component after the scheme. (sip > based urls for e.g) urlparse differentiates those by maintaining a > list of scheme names which will follow the pattern of parsing, and > joining for the urls which have a netloc (or authority component). > This is in general according to RFC 3986 itself. This actually quite at variance with the RFC. The grammar in section 3 doesn't make any reference to schemes as being significant in parsing. Whether an authority component is to be parsed or not is entirely dependent on the presence or absence of the "//" delimiter following the scheme and its colon delimiter. AFAICS, if the "//" delimiter is present, an authority component (possibly empty) *must* be present in the parse. Presumably an unparse should then include that empty component in the generated URI (ie, a "scheme:///..." URI). Thus, it seems that by the RFC, regardless of any registration, urlparse.unsplit(urlparse.split('git+file:///foo/bar')) should produce 'git+file:///foo/bar' (or perhaps raise an error in "validation" mode). The only question is whether registration of 'git+file' as a use_netloc scheme should force urlparse.unsplit(urlparse.split('git+file:/foo/bar')) to return 'git+file:///foo/bar', or whether 'git+file:/foo/bar' would be acceptable (or better). None of what I wrote here or elsewhere takes account of backward compatibility, it is true. I'm only talking about the letter of the RFC. From orsenthil at gmail.com Mon May 10 10:27:33 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Mon, 10 May 2010 13:57:33 +0530 Subject: [Python-Dev] urlparse.urlunsplit should be smarter about + In-Reply-To: <87632wrxb3.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp> <4BE58B9F.6080604@gmail.com> <20100510053814.GA3608@remy> <87632wrxb3.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20100510082732.GA8163@remy> On Mon, May 10, 2010 at 05:11:12PM +0900, Stephen J. Turnbull wrote: > > Not all urls have the 'authority' component after the scheme. (sip > > based urls for e.g) urlparse differentiates those by maintaining a > > list of scheme names which will follow the pattern of parsing, and > > joining for the urls which have a netloc (or authority component). > > This is in general according to RFC 3986 itself. > > This actually quite at variance with the RFC. The grammar in section I should have said, 'treatment of urls with authority' and 'treatment of urls without authority' in terms of parsing and joining is as per RFC. How it is doing practically is by maintaining a list of urls with known scheme names which use_netloc. -- Senthil Many Myths are based on truth -- Spock, "The Way to Eden", stardate 5832.3 From stephen at xemacs.org Mon May 10 10:56:29 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 10 May 2010 17:56:29 +0900 Subject: [Python-Dev] urlparse.urlunsplit should be smarter about + In-Reply-To: <20100510082732.GA8163@remy> References: <87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp> <4BE58B9F.6080604@gmail.com> <20100510053814.GA3608@remy> <87632wrxb3.fsf@uwakimon.sk.tsukuba.ac.jp> <20100510082732.GA8163@remy> Message-ID: <874oigrv7m.fsf@uwakimon.sk.tsukuba.ac.jp> Senthil Kumaran writes: > I should have said, 'treatment of urls with authority' and 'treatment > of urls without authority' in terms of parsing and joining is as per > RFC. How it is doing practically is by maintaining a list of urls > with known scheme names which use_netloc. Why do that if you can get better behavior based purely on syntactic analysis? From barry at python.org Mon May 10 11:54:36 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 10 May 2010 11:54:36 +0200 Subject: [Python-Dev] What's New text on future maintenance In-Reply-To: References: <20100507015035.GA50210@andrew-kuchlings-macbook.local> Message-ID: <20100510115436.6ae3b399@heresy> On May 06, 2010, at 09:33 PM, Benjamin Peterson wrote: >I don't think there's any point in being hypothetical about. I believe >we've already said that maintence for 2.7 will last for at least 5 >years, so let's proclaim it. +1. If our goal is to move our community to Python 3, then I think we do them the biggest service to be explicit about our intentions for maintenance of 2.7 and the future of Python 2. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ronaldoussoren at mac.com Mon May 10 16:09:06 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 10 May 2010 16:09:06 +0200 Subject: [Python-Dev] PEP 7 updated In-Reply-To: <20100509203346.169facf0@pitrou.net> References: <20100509203346.169facf0@pitrou.net> Message-ID: <68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com> On 9 May, 2010, at 20:33, Antoine Pitrou wrote: > > Hello, > > The untabification of C files didn't produce any noticeable problem on > the buildbots. I've updated PEP 7 with the mention that all C files > should be 4-space indented, and removed the obsolete wording about > some files being indented with tabs. Does anyone know of a way to teach vim that C sources in a python checkout should have 4-space indents without changing the defaults for other C files? Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From phd at phd.pp.ru Mon May 10 18:40:00 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Mon, 10 May 2010 20:40:00 +0400 Subject: [Python-Dev] vim (was: PEP 7 updated) In-Reply-To: <68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com> References: <20100509203346.169facf0@pitrou.net> <68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com> Message-ID: <20100510164000.GA16432@phd.pp.ru> On Mon, May 10, 2010 at 04:09:06PM +0200, Ronald Oussoren wrote: > Does anyone know of a way to teach vim that C sources in a python checkout should have 4-space indents without changing the defaults for other C files? BufReadPre /usr/local/src/Python/*/*.c set sts=4 sw=4 Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From rdmurray at bitdance.com Mon May 10 20:10:46 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 10 May 2010 14:10:46 -0400 Subject: [Python-Dev] vim (was: PEP 7 updated) In-Reply-To: <20100510164000.GA16432@phd.pp.ru> References: <20100509203346.169facf0@pitrou.net> <68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com> <20100510164000.GA16432@phd.pp.ru> Message-ID: <20100510181046.2E6BF2093EE@kimball.webabinitio.net> On Mon, 10 May 2010 20:40:00 +0400, Oleg Broytman wrote: > On Mon, May 10, 2010 at 04:09:06PM +0200, Ronald Oussoren wrote: > > Does anyone know of a way to teach vim that C sources in a python checkout should have 4-space indents without changing the defaults for other C files? > > BufReadPre /usr/local/src/Python/*/*.c set sts=4 sw=4 Alternatively you could sniff the file for leading tabs and set the "filetype" or the tab setting based on that. I think Brett has a vimrc somewhere that does something like that. --David From brett at python.org Mon May 10 23:14:52 2010 From: brett at python.org (Brett Cannon) Date: Mon, 10 May 2010 14:14:52 -0700 Subject: [Python-Dev] vim (was: PEP 7 updated) In-Reply-To: <20100510181046.2E6BF2093EE@kimball.webabinitio.net> References: <20100509203346.169facf0@pitrou.net> <68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com> <20100510164000.GA16432@phd.pp.ru> <20100510181046.2E6BF2093EE@kimball.webabinitio.net> Message-ID: On Mon, May 10, 2010 at 11:10, R. David Murray wrote: > On Mon, 10 May 2010 20:40:00 +0400, Oleg Broytman wrote: > > On Mon, May 10, 2010 at 04:09:06PM +0200, Ronald Oussoren wrote: > > > Does anyone know of a way to teach vim that C sources in a python > checkout should have 4-space indents without changing the defaults for other > C files? > > > > BufReadPre /usr/local/src/Python/*/*.c set sts=4 sw=4 > > Alternatively you could sniff the file for leading tabs and set the > "filetype" or the tab setting based on that. I think Brett has a vimrc > somewhere that does something like that. > Misc/Vim/python.vim does what you need to do. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon May 10 23:14:52 2010 From: brett at python.org (Brett Cannon) Date: Mon, 10 May 2010 14:14:52 -0700 Subject: [Python-Dev] vim (was: PEP 7 updated) In-Reply-To: <20100510181046.2E6BF2093EE@kimball.webabinitio.net> References: <20100509203346.169facf0@pitrou.net> <68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com> <20100510164000.GA16432@phd.pp.ru> <20100510181046.2E6BF2093EE@kimball.webabinitio.net> Message-ID: On Mon, May 10, 2010 at 11:10, R. David Murray wrote: > On Mon, 10 May 2010 20:40:00 +0400, Oleg Broytman wrote: > > On Mon, May 10, 2010 at 04:09:06PM +0200, Ronald Oussoren wrote: > > > Does anyone know of a way to teach vim that C sources in a python > checkout should have 4-space indents without changing the defaults for other > C files? > > > > BufReadPre /usr/local/src/Python/*/*.c set sts=4 sw=4 > > Alternatively you could sniff the file for leading tabs and set the > "filetype" or the tab setting based on that. I think Brett has a vimrc > somewhere that does something like that. > Misc/Vim/python.vim does what you need to do. -------------- next part -------------- An HTML attachment was scrubbed... URL: From borowitz at stanford.edu Mon May 10 23:34:23 2010 From: borowitz at stanford.edu (David Borowitz) Date: Mon, 10 May 2010 14:34:23 -0700 Subject: [Python-Dev] PEP 7 updated In-Reply-To: <68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com> References: <20100509203346.169facf0@pitrou.net> <68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com> Message-ID: On Mon, May 10, 2010 at 07:09, Ronald Oussoren wrote: > > On 9 May, 2010, at 20:33, Antoine Pitrou wrote: > > > > > Hello, > > > > The untabification of C files didn't produce any noticeable problem on > > the buildbots. I've updated PEP 7 with the mention that all C files > > should be 4-space indented, and removed the obsolete wording about > > some files being indented with tabs. > > Does anyone know of a way to teach vim that C sources in a python checkout > should have 4-space indents without changing the defaults for other C files? > :help autocmd-patterns has this example, which should be easy to adapt: :autocmd BufRead /vim/src/*.c set cindent Set the 'cindent' option for C files in the /vim/src directory. > Ronald > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ddborowitz%40gmail.com > > -- It is better to be quotable than to be honest. -Tom Stoppard Borowitz -------------- next part -------------- An HTML attachment was scrubbed... URL: From orsenthil at gmail.com Tue May 11 06:55:30 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Tue, 11 May 2010 10:25:30 +0530 Subject: [Python-Dev] urlparse.urlunsplit should be smarter about + In-Reply-To: <874oigrv7m.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp> <4BE58B9F.6080604@gmail.com> <20100510053814.GA3608@remy> <87632wrxb3.fsf@uwakimon.sk.tsukuba.ac.jp> <20100510082732.GA8163@remy> <874oigrv7m.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20100511045530.GA6340@remy> On Mon, May 10, 2010 at 05:56:29PM +0900, Stephen J. Turnbull wrote: > Senthil Kumaran writes: > > > I should have said, 'treatment of urls with authority' and 'treatment > > of urls without authority' in terms of parsing and joining is as per > > RFC. How it is doing practically is by maintaining a list of urls > > with known scheme names which use_netloc. > > Why do that if you can get better behavior based purely on syntactic > analysis? For the cases for just parsing and splitting, the syntactic behaviours are fine enough. I agree with your comments and reinstatement of RFC rules in the previous emails. The problem as we know off, comes while unparsing and joining, ( also I have not yet looked at the relative url joining behaviour where redundant /'s can be ignored). As you may already know, when the data is ParseResult(scheme='file', netloc='', path='/tmp/junk.txt', params='', query='', fragment='') You might expect the output to be file:///tmp/junk.txt Original might be same too. But for: ParseResult(scheme='x', netloc='', path='/y', params='', query='', fragment='') One can expect a valid output to be: x:/y Your suggestion of netloc/authority being differentiate by '' and None seems a good one to analyze. Also, by keeping a registry of valid schemes, are you not proposing something very similar to uses_netloc? But with a different API to handle parsing based on registry values. Is my understanding of your proposal correct? FWIW, I looked at the history of uses_netloc list and it seems that it been there from the first version when urlparse module followed different rfc specs for different protocols (telnet, sip etc), so any changes should be carefully incorporated as not to break the existing solutions. -- Senthil From solipsis at pitrou.net Wed May 12 01:16:15 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 12 May 2010 01:16:15 +0200 Subject: [Python-Dev] bugs.python.org down? Message-ID: <20100512011615.59ee8c7f@pitrou.net> Apparently the tracker has been unresponding for some time now, although port 80 still accepts connections. Regards Antoine. From sridharr at activestate.com Wed May 12 01:44:36 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Tue, 11 May 2010 16:44:36 -0700 Subject: [Python-Dev] PEP 7 updated In-Reply-To: <20100509203346.169facf0@pitrou.net> References: <20100509203346.169facf0@pitrou.net> Message-ID: <4B06CB5F-6304-42B1-8B2D-F386E74DB63D@activestate.com> Nor did it break any of our ActivePython 2.7 (Python trunk) builds ... though I had to hand-edit the patches to use 4 spaces now. Will this untabification change be made to the `release2.6-maint` branch too? -srid On 2010-05-09, at 11:33 AM, Antoine Pitrou wrote: > > Hello, > > The untabification of C files didn't produce any noticeable problem on > the buildbots. I've updated PEP 7 with the mention that all C files > should be 4-space indented, and removed the obsolete wording about > some files being indented with tabs. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/sridharr%40activestate.com From solipsis at pitrou.net Wed May 12 01:51:12 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 12 May 2010 01:51:12 +0200 Subject: [Python-Dev] PEP 7 updated In-Reply-To: <4B06CB5F-6304-42B1-8B2D-F386E74DB63D@activestate.com> References: <20100509203346.169facf0@pitrou.net> <4B06CB5F-6304-42B1-8B2D-F386E74DB63D@activestate.com> Message-ID: <1273621872.3248.2.camel@localhost.localdomain> Le mardi 11 mai 2010 ? 16:44 -0700, Sridhar Ratnakumar a ?crit : > Nor did it break any of our ActivePython 2.7 (Python trunk) builds ... > though I had to hand-edit the patches to use 4 spaces now. Will this > untabification change be made to the `release2.6-maint` branch too? It has already been made to the 2.6 branch. By the way, you don't have to untabify patches by hand, you can just use "untabify.py -p". Regards Antoine. From cs at zip.com.au Wed May 12 06:54:20 2010 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 12 May 2010 14:54:20 +1000 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: References: Message-ID: <20100512045420.GA27885@cskk.homeip.net> On 07May2010 14:53, VanL wrote: | On 5/7/2010 12:41 PM, Steven D'Aprano wrote: | >> Now, if I write | >> | >> def f1(x,y): return x+y | >> def f2(x,y): return x+y | >> | >> I don't expect f1==f2 to be True, even though f1 and f2 behave in | >> exactly the same way, and indeed it is not. | | This is not what I am getting after; these (IMO) should compare unequal. | I took a fairly conservative line, testing for identity of functions and | equality of arguments when converted to a normal form: | | def __eq__(self, other): | try: | return ((self.func == other.func) and | (self.args == other.args) and | (self.keywords == other.keywords)) [...] I think that if you're going to say "identity" above you should have: self.func is other.func in your code. If you want "==" in your code (and I think you do, since you're implementing __eq__) you should say "equality" instead of "identity". I know for functions "==" and "is" currently are equivalent, but we should be really finicky here about intent, especially since a few messages in the thread is contemplate testing function for equivalence to one degree or other. At which point "==" and "is" aren't the same any more. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ Careful and correct use of language is a powerful aid to straight thinking, for putting into words precisely what we mean necessitates getting our own minds quite clear on what we mean. - W.I.B. Beveridge From lie.1296 at gmail.com Wed May 12 09:49:48 2010 From: lie.1296 at gmail.com (Lie Ryan) Date: Wed, 12 May 2010 17:49:48 +1000 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: <4BE45472.6040406@holdenweb.com> References: <201005080342.00028.steve@pearwood.info> <4BE45472.6040406@holdenweb.com> Message-ID: On 05/08/10 03:57, Steve Holden wrote: > Steven D'Aprano wrote: >> >> [...] >>> Similarly, if you wanted p1==p2, why not write >>> >>> p1 = partial(operator.add) >>> p2 = p1 >> >> I thought the OP gave a use-case. He's generating "jobs" (partial >> applied to a callable and arguments), and wanted to avoid duplicated >> jobs. >> >> I think it is reasonable to expect that partial(operator.add, 2) >> compares equal to partial(operator.add, 2). I don't think he's >> suggesting it should compare equal to partial(lambda x,y: x+y, 2). >> > Which absence, presumably, also mildly disappoints you? > it disappoints me this does not compare equal: add3 = lambda a, b, c: a + b + c a = partial(partial(add3, 1), 2) b = partial(partial(add3, 2), 1) print a == b :-) From stephen at xemacs.org Wed May 12 10:34:50 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 12 May 2010 17:34:50 +0900 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: References: <201005080342.00028.steve@pearwood.info> <4BE45472.6040406@holdenweb.com> Message-ID: <8739xxr00l.fsf@uwakimon.sk.tsukuba.ac.jp> Lie Ryan writes: > it disappoints me this does not compare equal: > > add3 = lambda a, b, c: a + b + c > a = partial(partial(add3, 1), 2) > b = partial(partial(add3, 2), 1) > print a == b > > :-) But it's not even true for floating point. From steve at pearwood.info Wed May 12 12:37:51 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 12 May 2010 20:37:51 +1000 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: <8739xxr00l.fsf@uwakimon.sk.tsukuba.ac.jp> References: <8739xxr00l.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <201005122037.51976.steve@pearwood.info> On Wed, 12 May 2010 06:34:50 pm Stephen J. Turnbull wrote: > Lie Ryan writes: > > it disappoints me this does not compare equal: > > > > add3 = lambda a, b, c: a + b + c > > a = partial(partial(add3, 1), 2) > > b = partial(partial(add3, 2), 1) > > print a == b > > > > :-) > > But it's not even true for floating point. All humour aside, let's not ruin the chances of this patch being accepted by overloading it. I think there is a good use-case for partial objects to compare equal if they were constructed with arguments that compare equal. That's a nice, conservative change that is unlikely to lead to bugs, unlike some of the more "clever" proposals that rely on mathematical equivalences that don't hold for Python objects. -- Steven D'Aprano From van.lindberg at gmail.com Wed May 12 14:33:58 2010 From: van.lindberg at gmail.com (VanL) Date: Wed, 12 May 2010 07:33:58 -0500 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: <20100512045420.GA27885@cskk.homeip.net> References: <20100512045420.GA27885@cskk.homeip.net> Message-ID: <4BEAA036.3070800@gmail.com> On 5/11/2010 11:54 PM, Cameron Simpson wrote: > I know for functions "==" and "is" currently are equivalent, but we should be > really finicky here about intent, especially since a few messages in the > thread is contemplate testing function for equivalence to one degree or > other. At which point "==" and "is" aren't the same any more. > As I stated above, I make no effort to address equivalence of the functions, nor of the arguments. I am addressing identity of the partial object, which I am defining as identity of the functions + identity of the args + equivalence of the keyword arguments after a deterministic process has been applied. From jcea at jcea.es Wed May 12 15:13:21 2010 From: jcea at jcea.es (Jesus Cea) Date: Wed, 12 May 2010 15:13:21 +0200 Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0 Message-ID: <4BEAA971.3040001@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 [Zlib-devel] HEADS UP: Apparent bad compilation under (just released) GCC 4.5.0 GCC Bugzilla Bug 40838 gcc shouldn't assume that the stack is aligned http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40838 Short history: new GCC 4.5.0 (released a month ago), when compiling with - -O3, is adding MMX/SSE instructions that requires stack aligned to 16 byte. This is wrong, since x86 ABI only requires stack aligned to 4 bytes. If you compile EVERYTHING with GCC 4.5.0, you are safe (I guess!), but if your environment has mixed compiled code (for instance, the OS libraries), you can possibly "core dump". If you have an old compiled Python and you update libs compiled with GCC 4.5.0, you can crash in the process. Psyco is showing the issue, but it is not the culprit. It only leaves - -correctly- the stack in not 16-byte alignment. But there are plenty of examples of crashes not related to python+psyco. Proposal: add "-fno-tree-vectorize" to compilation options for 2.7/3.2. Warm 2.3/2.4/2.5/2.6/3.0/3.1 users. Or warm users compiling with GCC 4.5.0. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS+qpcZlgi5GaxT1NAQLAjgP/anZ5C2lED190++ffcluApF3ASF20iSnH t21ynHxrz2fgkPeOxeGRAqGEGCc3jZ0UuXchECcLLzFhI8WQS8K766EOgOKdwZLV WCt0ohWZ0FfsJZX4J5Y73x39uhjShbnl6SSek2uEvPi/tua/R4W/kVXrAZ0ZZR6/ vRqSUXpdolM= =5K+S -----END PGP SIGNATURE----- From ncoghlan at gmail.com Wed May 12 15:32:55 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 12 May 2010 23:32:55 +1000 Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0 In-Reply-To: <4BEAA971.3040001@jcea.es> References: <4BEAA971.3040001@jcea.es> Message-ID: <4BEAAE07.90903@gmail.com> Jesus Cea wrote: > Proposal: add "-fno-tree-vectorize" to compilation options for 2.7/3.2. Will this actually help? Won't there still be a problem if any extension module is compiled with GCC 4.5.0 without that option, regardless of the options used to build Python itself? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From foom at fuhm.net Wed May 12 15:39:56 2010 From: foom at fuhm.net (James Y Knight) Date: Wed, 12 May 2010 09:39:56 -0400 Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0 In-Reply-To: <4BEAA971.3040001@jcea.es> References: <4BEAA971.3040001@jcea.es> Message-ID: On May 12, 2010, at 9:13 AM, Jesus Cea wrote: > Short history: new GCC 4.5.0 (released a month ago), when compiling > with > - -O3, is adding MMX/SSE instructions that requires stack aligned to > 16 > byte. This is wrong, since x86 ABI only requires stack aligned to 4 > bytes. > > If you compile EVERYTHING with GCC 4.5.0, you are safe (I guess!), but > if your environment has mixed compiled code (for instance, the OS > libraries), you can possibly "core dump". If you have an old compiled > Python and you update libs compiled with GCC 4.5.0, you can crash in > the > process. > > Psyco is showing the issue, but it is not the culprit. It only leaves > - -correctly- the stack in not 16-byte alignment. But there are > plenty of > examples of crashes not related to python+psyco. > > Proposal: add "-fno-tree-vectorize" to compilation options for > 2.7/3.2. > Warm 2.3/2.4/2.5/2.6/3.0/3.1 users. Or warm users compiling with GCC > 4.5.0. While assuming the stack is 16byte aligned is undeniably an ABI- violation in GCC, at this point, it's surely simpler to just go along: the new unofficial ABI for x86 is that the stack must always be left in 16-byte alignment... So, just change psyco to always use 16-byte-aligned stackframes. GCC has used 16byte-aligned stackframes for a heck of a long time now (so if the stack starts 16byte aligned on entry to a function it will stay that way on calls). So usually the only way people run into unaligned stacks is via hand-written assembly code or JIT compilers. I think you'll be a lot happier just modifying Psyco than making everyone else in the world change their compiler flags. James From jcea at jcea.es Wed May 12 15:56:48 2010 From: jcea at jcea.es (Jesus Cea) Date: Wed, 12 May 2010 15:56:48 +0200 Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0 In-Reply-To: <4BEAAE07.90903@gmail.com> References: <4BEAA971.3040001@jcea.es> <4BEAAE07.90903@gmail.com> Message-ID: <4BEAB3A0.9010302@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/05/10 15:32, Nick Coghlan wrote: > Jesus Cea wrote: >> Proposal: add "-fno-tree-vectorize" to compilation options for 2.7/3.2. > > Will this actually help? Won't there still be a problem if any extension > module is compiled with GCC 4.5.0 without that option, regardless of the > options used to build Python itself? When I do a "python setup.py install", the options used to compile the module are the same that used to compile python itself. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS+qzn5lgi5GaxT1NAQJ5BgQAimvcMssT57iuuLpaI9P9GXKGZf9VUAzj F4XJM/lT4Qtzu22jecIvEej0MyiMR/4qHvns8lgqLXn/30pkYrkmYcjxFigpM7Bl rOKMYeAr3ki8dN87APkoiMKOJHByXlEUZu+BowCOXcUCtGYcENLwQ+STr1NyCsUB IINfh1pJ2Nc= =GiUG -----END PGP SIGNATURE----- From jcea at jcea.es Wed May 12 16:01:19 2010 From: jcea at jcea.es (Jesus Cea) Date: Wed, 12 May 2010 16:01:19 +0200 Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0 In-Reply-To: References: <4BEAA971.3040001@jcea.es> Message-ID: <4BEAB4AF.80908@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/05/10 15:39, James Y Knight wrote: > While assuming the stack is 16byte aligned is undeniably an > ABI-violation in GCC, at this point, it's surely simpler to just go > along: the new unofficial ABI for x86 is that the stack must always be > left in 16-byte alignment... You can not rule out other software embedding python inside, or callbacks from foreign code. For instance, Berkeley DB library can do callbacks to Python code. > So, just change psyco to always use 16-byte-aligned stackframes. GCC has > used 16byte-aligned stackframes for a heck of a long time now (so if the > stack starts 16byte aligned on entry to a function it will stay that way > on calls). So usually the only way people run into unaligned stacks is > via hand-written assembly code or JIT compilers. Not all the universe is GCC based. For instance, Solaris system libraries are not compiled using GCC. The world is bigger that Linux/GCC. > I think you'll be a lot happier just modifying Psyco than making > everyone else in the world change their compiler flags. Would be nice if GCC 4.5.1 would solve this :). They are objectivelly breaking the x86 ABI. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBS+q0rplgi5GaxT1NAQK98wP+NJoqNCpvjemP4Gv7y1G/iPkQgjuidslT uiPxDcN9Eprcluc+mGTBu6N+fCTj09xYhUCD1wWhoJq2dRyoA8b+XC1fCSyL4VXc mzsy0rGmKeQh4lyAw+7agFCqryd6n+/oyl+9aOT6YkzyLFjQd4KDEcGNZ0h+6PAf 4jtx1+p3k0c= =TzTY -----END PGP SIGNATURE----- From foom at fuhm.net Wed May 12 16:27:09 2010 From: foom at fuhm.net (James Y Knight) Date: Wed, 12 May 2010 10:27:09 -0400 Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0 In-Reply-To: <4BEAB4AF.80908@jcea.es> References: <4BEAA971.3040001@jcea.es> <4BEAB4AF.80908@jcea.es> Message-ID: On May 12, 2010, at 10:01 AM, Jesus Cea wrote: > On 12/05/10 15:39, James Y Knight wrote: >> While assuming the stack is 16byte aligned is undeniably an >> ABI-violation in GCC, at this point, it's surely simpler to just go >> along: the new unofficial ABI for x86 is that the stack must always >> be >> left in 16-byte alignment... > > You can not rule out other software embedding python inside, or > callbacks from foreign code. For instance, Berkeley DB library can do > callbacks to Python code. So? When calling callback functions, the Berkeley DB library won't un-16byte-align the stack, will it? (Assuming it's been compiled with gcc in the last 10 years) > Not all the universe is GCC based. For instance, Solaris system > libraries are not compiled using GCC. The world is bigger that Linux/ > GCC. If the Solaris compilers don't use 16byte-aligned stackframes, and GCC on Solaris/x86 also assumes 16byte-aligned stacks, I guess GCC on Solaris/x86 is pretty broken indeed. But for Linux/x86, stacks have been de-facto 16byte aligned for so long, you can *almost* excuse the ABI violation as unimportant. But anyways, psyco should keep the stackframes 16byte aligned regardless, for performance reasons: even when accessing datatypes for which unaligned access doesn't crash, it's faster when it's aligned. James From martin at v.loewis.de Wed May 12 17:14:25 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 12 May 2010 17:14:25 +0200 Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0 In-Reply-To: <4BEAA971.3040001@jcea.es> References: <4BEAA971.3040001@jcea.es> Message-ID: <4BEAC5D1.2050906@v.loewis.de> > Short history: new GCC 4.5.0 (released a month ago), when compiling with > -O3, is adding MMX/SSE instructions that requires stack aligned to 16 > byte. This is wrong, since x86 ABI only requires stack aligned to 4 bytes. I think this is debatable. It depends on the operating system also; ultimately, it is the OS vendor who specifies the C ABI for their systems. On Linux, in absence of a vendor, the ABI is what the kernel and gcc define it to be. Regards, Martin From urban.dani at gmail.com Wed May 12 17:11:21 2010 From: urban.dani at gmail.com (Daniel Urban) Date: Wed, 12 May 2010 17:11:21 +0200 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: References: Message-ID: On Fri, May 7, 2010 at 17:02, Antoine Pitrou wrote: > It would be more useful to provide equality, hashing and repr to partial itself, > rather than a subclass. Feel free to propose a patch :) Hi! I've done that. I've opened a feature request: http://bugs.python.org/issue8699 The patch is also at Rietveld: http://codereview.appspot.com/1179044 I'm a beginner, so my patch is probably far from perfect, but I'd appreciate any help, and will try to correct my mistakes. Thanks, Daniel Urban From jyasskin at gmail.com Wed May 12 17:58:19 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Wed, 12 May 2010 08:58:19 -0700 Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0 In-Reply-To: References: <4BEAA971.3040001@jcea.es> Message-ID: On Wed, May 12, 2010 at 6:39 AM, James Y Knight wrote: > I think you'll be a lot happier just modifying Psyco than making everyone > else in the world change their compiler flags. Aye, there's the rub. Nobody's happier modifying Psyco. :) But that just means people will gradually have to stop using psyco in favor of maintainable JITs. Gcc's not going to change its stack requirements just because some people think they know what the ABI should be better than the people defining the ABI. Btw, this has been a problem since at least gcc-4.4. From tjreedy at udel.edu Wed May 12 19:09:11 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 12 May 2010 13:09:11 -0400 Subject: [Python-Dev] bugs.python.org down? In-Reply-To: <20100512011615.59ee8c7f@pitrou.net> References: <20100512011615.59ee8c7f@pitrou.net> Message-ID: On 5/11/2010 7:16 PM, Antoine Pitrou wrote: > > Apparently the tracker has been unresponding for some time now, > although port 80 still accepts connections. As I reported before, there have been on and off problems for days. Messages like (minutes ago) upon trying to login: "Service Temporarily Unavailable The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later. Apache/2.2.9 (Debian) mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.9 OpenSSL/0.9.8g Server at bugs.python.org Port 80" Now I retry and it works. From rdmurray at bitdance.com Wed May 12 20:09:22 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 12 May 2010 14:09:22 -0400 Subject: [Python-Dev] bugs.python.org down? In-Reply-To: References: <20100512011615.59ee8c7f@pitrou.net> Message-ID: <20100512180923.00A20200560@kimball.webabinitio.net> On Wed, 12 May 2010 13:09:11 -0400, Terry Reedy wrote: > On 5/11/2010 7:16 PM, Antoine Pitrou wrote: > > > > Apparently the tracker has been unresponding for some time now, > > although port 80 still accepts connections. > > As I reported before, there have been on and off problems for days. > Messages like (minutes ago) upon trying to login: > > "Service Temporarily Unavailable > > The server is temporarily unable to service your request due to > maintenance downtime or capacity problems. Please try again later. > Apache/2.2.9 (Debian) mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.9 > OpenSSL/0.9.8g Server at bugs.python.org Port 80" > > Now I retry and it works. That last was us restarting roundup, which cured the problem at least temporarily. We've added a robots.txt file to hopefully slow down the bot hits (which are the bulk of the traffic on the tracker) once they bother to notice it. I'm not convinced that hit count load is the problem, but it might be. -- R. David Murray www.bitdance.com From janssen at parc.com Thu May 13 05:17:59 2010 From: janssen at parc.com (Bill Janssen) Date: Wed, 12 May 2010 20:17:59 PDT Subject: [Python-Dev] configuring the buildbot to skip some tests? Message-ID: <84766.1273720679@parc.com> I've got parc-tiger-1 up and running again. It's failing on test_tk, which makes sense, because it's running as a background twisted process, and thus can't access the window server. I should configure that out. I'm looking for documentation on how to configure the build slave so that it skips this test. Bill From trent at snakebite.org Thu May 13 05:38:48 2010 From: trent at snakebite.org (Trent Nelson) Date: Wed, 12 May 2010 23:38:48 -0400 Subject: [Python-Dev] Python on Windows with CoApp Message-ID: <4BEB7448.8080204@snakebite.org> Howdy folks, Quick e-mail at 34,000ft (aren't wifi-enabled flights great?) to mention a new initiative that's been started by Microsoft called CoApp (Common Opensource Application Publishing Platform). The aim is simple: make open source software rock on Windows ;-) It's probably easiest to think of it as a Microsoft-endorsed-but-community-run open source distribution for Windows, akin to all the various package managers for Linux distributions and ports/packages for the *BSDs. There are specific user and developer experiences we'll be addressing -- like making it easy to install and use open source software, or use it within your own project (open source or not). CoApp will affect Python in one of two ways. Once there's a clear-cut specification for open source projects to follow, Python can either decide to follow it, or not. The same applies to all open source packages, actually. For those that follow it, great! If not, no problem -- the plan is to shallow-fork such projects via launchpad and the CoApp community will take responsibility for getting releases of open source projects into CoApp shape. It's in its infancy at the moment -- it took the chap (Garrett Serack) who's spearheading it at Microsoft about six months to get it all signed off by the lawyers and platform/server VPs. So, for those of you out there who are Windows-inclined, now's a perfect time to get involved to help shape the direction of CoApp going forward. The website/wiki is http://coapp.org/ and the launchpad project site is http://launchpad.net/coapp (which is where the mailing list is hosted). We're actually having a 'CoApp Development Summit' tomorrow and Friday in Seattle (that Microsoft's graciously sponsored). The event will be accessible via Live Meeting for those that are interested: http://coapp.org/Project_Planning/CoApp_Design_and_Development_Summit Regards, Trent. From trent at snakebite.org Thu May 13 05:47:59 2010 From: trent at snakebite.org (Trent Nelson) Date: Wed, 12 May 2010 23:47:59 -0400 Subject: [Python-Dev] PEP 7 updated In-Reply-To: <68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com> References: <20100509203346.169facf0@pitrou.net> <68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com> Message-ID: <4BEB766F.7070100@snakebite.org> > Does anyone know of a way to teach vim that C sources in a python checkout should have 4-space indents without changing the defaults for other C files? I use this in my vimrc: "----------------------------------------------------------------------------" " indentation: use detectindent plugin if possible "----------------------------------------------------------------------------" set autoindent set smartindent try let g:detectindent_preferred_expandtab = 1 let g:detectindent_preferred_tabsize = 8 let g:detectindent_preferred_indent = 4 source $VIMRUNTIME/plugin/detectindent.vim au BufNewFile,BufRead * .* DetectIndent catch set smarttab set expandtab set tabstop=8 set shiftwidth=4 set softtabstop=4 set textwidth=80 endtry *** And this is plugin/detectindent.vim: " Name: detectindent (global plugin) " Version: 1.0 " Author: Ciaran McCreesh " Updates: http://dev.gentoo.org/~ciaranm/vim/ " Purpose: Detect file indent settings " " License: You may redistribute this plugin under the same terms as Vim " itself. " " Usage: :DetectIndent " " " to prefer expandtab to noexpandtab when detection is " " impossible: " :let g:detectindent_preferred_expandtab = 1 " " " to set a preferred indent level when detection is " " impossible: " :let g:detectindent_preferred_indent = 4 " " Requirements: Untested on Vim versions below 6.2 fun! IsCommentStart(line) " &comments isn't reliable if &ft == "c" || &ft == "cpp" return -1 != match(a:line, '/\*') else return 0 endif endfun fun! IsCommentEnd(line) if &ft == "c" || &ft == "cpp" return -1 != match(a:line, '\*/') else return 0 endif endfun fun! DetectIndent() let l:has_leading_tabs = 0 let l:has_leading_spaces = 0 let l:shortest_leading_spaces_run = 0 let l:longest_leading_spaces_run = 0 let l:idx_end = line("$") let l:idx = 1 while l:idx <= l:idx_end let l:line = getline(l:idx) " try to skip over comment blocks, they can give really screwy indent " settings in c/c++ files especially if IsCommentStart(l:line) while l:idx <= l:idx_end && ! IsCommentEnd(l:line) let l:line = getline(l:idx) let l:idx = l:idx + 1 endwhile let l:idx = l:idx + 1 continue endif let l:leading_char = strpart(l:line, 0, 1) if l:leading_char == "\t" let l:has_leading_tabs = 1 elseif l:leading_char == " " " only interested if we don't have a run of spaces followed by a " tab. if -1 == match(l:line, '^ \+\t') let l:has_leading_spaces = 1 let l:spaces = strlen(matchstr(l:line, '^ \+')) if l:shortest_leading_spaces_run == 0 || \ l:spaces < l:shortest_leading_spaces_run let l:shortest_leading_spaces_run = l:spaces endif if l:spaces > l:longest_leading_spaces_run let l:longest_leading_spaces_run = l:spaces endif endif endif let l:idx = l:idx + 1 endwhile if l:has_leading_tabs && ! l:has_leading_spaces " tabs only, no spaces set noexpandtab if exists("g:detectindent_preferred_tabsize") let &shiftwidth = g:detectindent_preferred_indent let &tabstop = g:detectindent_preferred_indent endif elseif l:has_leading_spaces && ! l:has_leading_tabs " spaces only, no tabs set expandtab let &shiftwidth = l:shortest_leading_spaces_run elseif l:has_leading_spaces && l:has_leading_tabs " spaces and tabs set noexpandtab let &shiftwidth = l:shortest_leading_spaces_run " mmmm, time to guess how big tabs are if l:longest_leading_spaces_run < 2 let &tabstop = 2 elseif l:longest_leading_spaces_run < 4 let &tabstop = 4 else let &tabstop = 8 endif else " no spaces, no tabs if exists("g:detectindent_preferred_tabsize") let &shiftwidth = g:detectindent_preferred_indent let &tabstop = g:detectindent_preferred_indent endif if exists("g:detectindent_preferred_expandtab") set expandtab endif endif endfun command! -nargs=0 DetectIndent call DetectIndent() From martin at v.loewis.de Thu May 13 09:20:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 13 May 2010 09:20:07 +0200 Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <84766.1273720679@parc.com> References: <84766.1273720679@parc.com> Message-ID: <4BEBA827.5040000@v.loewis.de> Bill Janssen wrote: > I've got parc-tiger-1 up and running again. It's failing on test_tk, > which makes sense, because it's running as a background twisted process, > and thus can't access the window server. It doesn't really make sense. It should skip the test, instead of failing it. I.e. aborting the Python process is definitely not a good response. Regards, Martin From yaniv at aknin.name Thu May 13 10:50:02 2010 From: yaniv at aknin.name (Yaniv Aknin) Date: Thu, 13 May 2010 18:50:02 +1000 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: References: Message-ID: I'm never certain where to reply in such a case, on the list or on the issue, but since no one is nosy yet to Daniel's patch, I thought I'd ask here. While a partial object should reasonably never change, you could change it: >>> from functools import partial >>> p = partial(lambda *a, **kw: kw, 1, 2, spam='eggs') >>> p() {'spam': 'eggs'} >>> p.keywords['spam'] = 'bacon' >>> p() {'spam': 'bacon'} >>> I realize touching p.keywords voids your warranty, but if we can stop people from doing it, maybe we should (or at least put a warning in the documentation, no?). So I'm thinking either we make an immutable/hashable dict while we're at it, or store the keyword arguments as a tuple (which guarantees immutability), and only convert them back to a dict when you want to call the partial object (simpler, slower). Your thoughts? Should we continue this discussion at issue8699? - Yaniv On Thu, May 13, 2010 at 1:11 AM, Daniel Urban wrote: > On Fri, May 7, 2010 at 17:02, Antoine Pitrou wrote: > > It would be more useful to provide equality, hashing and repr to partial > itself, > > rather than a subclass. Feel free to propose a patch :) > > Hi! > > I've done that. > I've opened a feature request: http://bugs.python.org/issue8699 > The patch is also at Rietveld: http://codereview.appspot.com/1179044 > > I'm a beginner, so my patch is probably far from perfect, but I'd > appreciate any help, and will try to correct my mistakes. > > Thanks, > Daniel Urban > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/yaniv%40aknin.name > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu May 13 12:39:13 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 13 May 2010 20:39:13 +1000 Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <84766.1273720679@parc.com> References: <84766.1273720679@parc.com> Message-ID: <4BEBD6D1.8020702@gmail.com> Bill Janssen wrote: > I've got parc-tiger-1 up and running again. It's failing on test_tk, > which makes sense, because it's running as a background twisted process, > and thus can't access the window server. I should configure that out. > > I'm looking for documentation on how to configure the build slave so > that it skips this test. It may be better to try to detect the "no window server" case and skip it in the test itself rather than in the build slave configuration. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From steve at pearwood.info Thu May 13 13:30:29 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 13 May 2010 21:30:29 +1000 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: References: Message-ID: <201005132130.30619.steve@pearwood.info> On Thu, 13 May 2010 06:50:02 pm Yaniv Aknin wrote: > I'm never certain where to reply in such a case, on the list or on > the issue, but since no one is nosy yet to Daniel's patch, I thought > I'd ask here. > > While a partial object should reasonably never change, you could change it: [...] > I realize touching p.keywords voids your warranty, but if we can stop > people from doing it, maybe we should (or at least put a warning in > the documentation, no?). Modifying partial.keywords will almost certainly effect hashing, so I think this is relevant to the patch. > So I'm thinking either we make an > immutable/hashable dict while we're at it, or store the keyword > arguments as a tuple (which guarantees immutability), and only > convert them back to a dict when you want to call the partial object > (simpler, slower). I'd support an immutable dict. partial objects already impose a significant (~ 30%) performance penalty: >>> from timeit import Timer >>> min(Timer('f(5)', 'f = lambda x: x').repeat()) 0.93580079078674316 >>> min(Timer('p(5)', 'from functools import partial; p = partial(lambda x: x)').repeat()) 1.2715129852294922 No need to make that worse if that can be avoided. -- Steven D'Aprano From urban.dani at gmail.com Thu May 13 11:51:23 2010 From: urban.dani at gmail.com (Daniel Urban) Date: Thu, 13 May 2010 11:51:23 +0200 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: References: Message-ID: > While a partial object should reasonably never change, you could change it: >>>> from functools import partial >>>> p = partial(lambda *a, **kw: kw, 1, 2, spam='eggs') >>>> p() > {'spam': 'eggs'} >>>> p.keywords['spam'] = 'bacon' >>>> p() > {'spam': 'bacon'} >>>> > I realize touching p.keywords voids your warranty, but if we can stop people > from doing it, maybe we should (or at least put a warning in the > documentation, no?). So I'm thinking either we make an immutable/hashable > dict while we're at it, or?store the keyword arguments as a tuple (which > guarantees immutability), and only convert them back to a dict when you want > to call the partial object (simpler, slower). You're right. I think it is possible to stop people from modifying p.keywords. If p.keywords wouldn't be a dict, but a read-only proxy for that dict, that may solve this problem. I think that is possible, with PyDictProxy_New [1]. But I'm not certain if that is a good idea, it may have side effects I'm not aware of. Any thoughts about this? > Your thoughts? Should we continue this discussion at issue8699? I don't know, I'm new here... [1] http://docs.python.org/py3k/c-api/dict.html#PyDictProxy_New Thanks, Daniel Urban From exarkun at twistedmatrix.com Thu May 13 15:41:36 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 13 May 2010 13:41:36 -0000 Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <84766.1273720679@parc.com> References: <84766.1273720679@parc.com> Message-ID: <20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain> On 03:17 am, janssen at parc.com wrote: >I've got parc-tiger-1 up and running again. It's failing on test_tk, >which makes sense, because it's running as a background twisted >process, >and thus can't access the window server. I should configure that out. You can run it in an xvfb. Jean-Paul From martin at v.loewis.de Thu May 13 16:42:18 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 13 May 2010 16:42:18 +0200 Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <4BEBD6D1.8020702@gmail.com> References: <84766.1273720679@parc.com> <4BEBD6D1.8020702@gmail.com> Message-ID: <4BEC0FCA.40601@v.loewis.de> >> I've got parc-tiger-1 up and running again. It's failing on test_tk, >> which makes sense, because it's running as a background twisted process, >> and thus can't access the window server. I should configure that out. >> >> I'm looking for documentation on how to configure the build slave so >> that it skips this test. > > It may be better to try to detect the "no window server" case and skip > it in the test itself rather than in the build slave configuration. Even better would be if Python wouldn't crash when you try to run Tk commands without a window server. Instead of aborting Python, that should raise an exception (which can then be detected as a test skip). Regards, Martin From martin at v.loewis.de Thu May 13 16:43:49 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 13 May 2010 16:43:49 +0200 Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain> References: <84766.1273720679@parc.com> <20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain> Message-ID: <4BEC1025.2050205@v.loewis.de> exarkun at twistedmatrix.com wrote: > On 03:17 am, janssen at parc.com wrote: >> I've got parc-tiger-1 up and running again. It's failing on test_tk, >> which makes sense, because it's running as a background twisted process, >> and thus can't access the window server. I should configure that out. > > You can run it in an xvfb. But that's beside the point! The slave configuration detected a bug in Python, so rather than working around the bug by modifying the slave configuration, the bug should get fixed. Of course, the slave is then useless until somebody contributes such a fix. Regards, Martin From p.f.moore at gmail.com Thu May 13 17:49:48 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 13 May 2010 16:49:48 +0100 Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <4BEC1025.2050205@v.loewis.de> References: <84766.1273720679@parc.com> <20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain> <4BEC1025.2050205@v.loewis.de> Message-ID: On 13 May 2010 15:43, "Martin v. L?wis" wrote: > Of course, the slave is then useless until somebody contributes such a fix. That's the sad part. If there was a means of temporarily marking the test on a particular slave as a known issue, it would avoid a single bug rendering a buildslave useless... (Having said that, a similar situation with my buildslave prompted me to spend the time fixing the bug so I didn't have to keep restarting the slave, so maybe it's a good thing after all :-)) Paul. From martin at v.loewis.de Thu May 13 17:57:45 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 13 May 2010 17:57:45 +0200 Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: References: <84766.1273720679@parc.com> <20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain> <4BEC1025.2050205@v.loewis.de> Message-ID: <4BEC2179.9070507@v.loewis.de> > (Having said that, a similar situation with my buildslave prompted me > to spend the time fixing the bug so I didn't have to keep restarting > the slave, so maybe it's a good thing after all :-)) Indeed. More generally, I'd question the point of automated testing if people try to work around serious problems rather than fixing them. And an interpreter crash in the test suite *is* a serious problem, IMO. Of course, it may turn out to be an unfixable Tcl or Apple bug, in which case working around would become more interesting. Regards, Martin From janssen at parc.com Thu May 13 19:42:23 2010 From: janssen at parc.com (Bill Janssen) Date: Thu, 13 May 2010 10:42:23 PDT Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <4BEBA827.5040000@v.loewis.de> References: <84766.1273720679@parc.com> <4BEBA827.5040000@v.loewis.de> Message-ID: <81551.1273772543@parc.com> Martin v. L?wis wrote: > Bill Janssen wrote: > > I've got parc-tiger-1 up and running again. It's failing on test_tk, > > which makes sense, because it's running as a background twisted process, > > and thus can't access the window server. > > It doesn't really make sense. It should skip the test, instead of > failing it. I.e. aborting the Python process is definitely not a good > response. Yes, you're right. It's a bug in the test. Bill From glyph at twistedmatrix.com Thu May 13 19:45:36 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 13 May 2010 13:45:36 -0400 Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain> References: <84766.1273720679@parc.com> <20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain> Message-ID: On May 13, 2010, at 9:41 AM, exarkun at twistedmatrix.com wrote: > On 03:17 am, janssen at parc.com wrote: >> I've got parc-tiger-1 up and running again. It's failing on test_tk, >> which makes sense, because it's running as a background twisted process, >> and thus can't access the window server. I should configure that out. > > You can run it in an xvfb. See : this isn't an X server that he's talking about, it's "WindowServer", the OS X windowing system, so Xvfb won't help. -------------- next part -------------- An HTML attachment was scrubbed... URL: From janssen at parc.com Thu May 13 19:47:56 2010 From: janssen at parc.com (Bill Janssen) Date: Thu, 13 May 2010 10:47:56 PDT Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain> References: <84766.1273720679@parc.com> <20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain> Message-ID: <81726.1273772876@parc.com> exarkun at twistedmatrix.com wrote: > On 03:17 am, janssen at parc.com wrote: > >I've got parc-tiger-1 up and running again. It's failing on test_tk, > > which makes sense, because it's running as a background twisted > > process, > >and thus can't access the window server. I should configure that out. > > You can run it in an xvfb. I don't think that would work -- it's a Mac. Bill From brett at python.org Thu May 13 20:39:23 2010 From: brett at python.org (Brett Cannon) Date: Thu, 13 May 2010 11:39:23 -0700 Subject: [Python-Dev] PEP 7 updated In-Reply-To: <4BEB766F.7070100@snakebite.org> References: <20100509203346.169facf0@pitrou.net> <68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com> <4BEB766F.7070100@snakebite.org> Message-ID: Feel free to look at Misc/Vim/python.vim and see if this works better than what is already there. On Wed, May 12, 2010 at 20:47, Trent Nelson wrote: > >> Does anyone know of a way to teach vim that C sources in a python checkout >> should have 4-space indents without changing the defaults for other C files? > > I use this in my vimrc: > > "----------------------------------------------------------------------------" > " indentation: use detectindent plugin if possible > "----------------------------------------------------------------------------" > set autoindent > set smartindent > try > ? ?let g:detectindent_preferred_expandtab = 1 > ? ?let g:detectindent_preferred_tabsize = 8 > ? ?let g:detectindent_preferred_indent = 4 > > ? ?source $VIMRUNTIME/plugin/detectindent.vim > ? ?au BufNewFile,BufRead * .* DetectIndent > catch > ? ?set smarttab > ? ?set expandtab > ? ?set tabstop=8 > ? ?set shiftwidth=4 > ? ?set softtabstop=4 > ? ?set textwidth=80 > endtry > > *** And this is plugin/detectindent.vim: > > " Name: ? ? ? ? ?detectindent (global plugin) > " Version: ? ? ? 1.0 > " Author: ? ? ? ?Ciaran McCreesh > " Updates: ? ? ? http://dev.gentoo.org/~ciaranm/vim/ > " Purpose: ? ? ? Detect file indent settings > " > " License: ? ? ? You may redistribute this plugin under the same terms as > Vim > " ? ? ? ? ? ? ? ?itself. > " > " Usage: ? ? ? ? :DetectIndent > " > " ? ? ? ? ? ? ? ?" to prefer expandtab to noexpandtab when detection is > " ? ? ? ? ? ? ? ?" impossible: > " ? ? ? ? ? ? ? ?:let g:detectindent_preferred_expandtab = 1 > " > " ? ? ? ? ? ? ? ?" to set a preferred indent level when detection is > " ? ? ? ? ? ? ? ?" impossible: > " ? ? ? ? ? ? ? ?:let g:detectindent_preferred_indent = 4 > " > " Requirements: ?Untested on Vim versions below 6.2 > > fun! IsCommentStart(line) > ? ?" &comments isn't reliable > ? ?if &ft == "c" || &ft == "cpp" > ? ? ? ?return -1 != match(a:line, '/\*') > ? ?else > ? ? ? ?return 0 > ? ?endif > endfun > > fun! IsCommentEnd(line) > ? ?if &ft == "c" || &ft == "cpp" > ? ? ? ?return -1 != match(a:line, '\*/') > ? ?else > ? ? ? ?return 0 > ? ?endif > endfun > > fun! DetectIndent() > ? ?let l:has_leading_tabs ? ? ? ? ? ?= 0 > ? ?let l:has_leading_spaces ? ? ? ? ?= 0 > ? ?let l:shortest_leading_spaces_run = 0 > ? ?let l:longest_leading_spaces_run ?= 0 > > ? ?let l:idx_end = line("$") > ? ?let l:idx = 1 > ? ?while l:idx <= l:idx_end > ? ? ? ?let l:line = getline(l:idx) > > ? ? ? ?" try to skip over comment blocks, they can give really screwy indent > ? ? ? ?" settings in c/c++ files especially > ? ? ? ?if IsCommentStart(l:line) > ? ? ? ? ? ?while l:idx <= l:idx_end && ! IsCommentEnd(l:line) > ? ? ? ? ? ? ? ?let l:line = getline(l:idx) > ? ? ? ? ? ? ? ?let l:idx = l:idx + 1 > ? ? ? ? ? ?endwhile > ? ? ? ? ? ?let l:idx = l:idx + 1 > ? ? ? ? ? ?continue > ? ? ? ?endif > > ? ? ? ?let l:leading_char = strpart(l:line, 0, 1) > > ? ? ? ?if l:leading_char == "\t" > ? ? ? ? ? ?let l:has_leading_tabs = 1 > > ? ? ? ?elseif l:leading_char == " " > ? ? ? ? ? ?" only interested if we don't have a run of spaces followed by a > ? ? ? ? ? ?" tab. > ? ? ? ? ? ?if -1 == match(l:line, '^ \+\t') > ? ? ? ? ? ? ? ?let l:has_leading_spaces = 1 > ? ? ? ? ? ? ? ?let l:spaces = strlen(matchstr(l:line, '^ \+')) > ? ? ? ? ? ? ? ?if l:shortest_leading_spaces_run == 0 || > ? ? ? ? ? ? ? ? ? ? ? ? ? ?\ l:spaces < l:shortest_leading_spaces_run > ? ? ? ? ? ? ? ? ? ?let l:shortest_leading_spaces_run = l:spaces > ? ? ? ? ? ? ? ?endif > ? ? ? ? ? ? ? ?if l:spaces > l:longest_leading_spaces_run > ? ? ? ? ? ? ? ? ? ?let l:longest_leading_spaces_run = l:spaces > ? ? ? ? ? ? ? ?endif > ? ? ? ? ? ?endif > > ? ? ? ?endif > > ? ? ? ?let l:idx = l:idx + 1 > ? ?endwhile > > ? ?if l:has_leading_tabs && ! l:has_leading_spaces > ? ? ? ?" tabs only, no spaces > ? ? ? ?set noexpandtab > ? ? ? ?if exists("g:detectindent_preferred_tabsize") > ? ? ? ? ? ?let &shiftwidth ?= g:detectindent_preferred_indent > ? ? ? ? ? ?let &tabstop ? ? = g:detectindent_preferred_indent > ? ? ? ?endif > > ? ?elseif l:has_leading_spaces && ! l:has_leading_tabs > ? ? ? ?" spaces only, no tabs > ? ? ? ?set expandtab > ? ? ? ?let &shiftwidth ?= l:shortest_leading_spaces_run > > ? ?elseif l:has_leading_spaces && l:has_leading_tabs > ? ? ? ?" spaces and tabs > ? ? ? ?set noexpandtab > ? ? ? ?let &shiftwidth = l:shortest_leading_spaces_run > > ? ? ? ?" mmmm, time to guess how big tabs are > ? ? ? ?if l:longest_leading_spaces_run < 2 > ? ? ? ? ? ?let &tabstop = 2 > ? ? ? ?elseif l:longest_leading_spaces_run < 4 > ? ? ? ? ? ?let &tabstop = 4 > ? ? ? ?else > ? ? ? ? ? ?let &tabstop = 8 > ? ? ? ?endif > > ? ?else > ? ? ? ?" no spaces, no tabs > ? ? ? ?if exists("g:detectindent_preferred_tabsize") > ? ? ? ? ? ?let &shiftwidth ?= g:detectindent_preferred_indent > ? ? ? ? ? ?let &tabstop ? ? = g:detectindent_preferred_indent > ? ? ? ?endif > ? ? ? ?if exists("g:detectindent_preferred_expandtab") > ? ? ? ? ? ?set expandtab > ? ? ? ?endif > > ? ?endif > endfun > > command! -nargs=0 DetectIndent call DetectIndent() > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > From martin at v.loewis.de Thu May 13 20:41:45 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 13 May 2010 20:41:45 +0200 Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <81551.1273772543@parc.com> References: <84766.1273720679@parc.com> <4BEBA827.5040000@v.loewis.de> <81551.1273772543@parc.com> Message-ID: <4BEC47E9.20804@v.loewis.de> Bill Janssen wrote: > Martin v. L?wis wrote: > >> Bill Janssen wrote: >>> I've got parc-tiger-1 up and running again. It's failing on test_tk, >>> which makes sense, because it's running as a background twisted process, >>> and thus can't access the window server. >> It doesn't really make sense. It should skip the test, instead of >> failing it. I.e. aborting the Python process is definitely not a good >> response. > > Yes, you're right. It's a bug in the test. No, I'd say it's even deeper, in the Tcl integration. There shouldn't be a way to cause an interpreter abort, except by calling os.abort(). Regards, Martin From ronaldoussoren at mac.com Thu May 13 20:50:24 2010 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 13 May 2010 20:50:24 +0200 Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <4BEC47E9.20804@v.loewis.de> References: <84766.1273720679@parc.com> <4BEBA827.5040000@v.loewis.de> <81551.1273772543@parc.com> <4BEC47E9.20804@v.loewis.de> Message-ID: <05A4049A-BBDA-4149-AB65-6DD77716B8DA@mac.com> On 13 May, 2010, at 20:41, Martin v. L?wis wrote: > Bill Janssen wrote: >> Martin v. L?wis wrote: >> >>> Bill Janssen wrote: >>>> I've got parc-tiger-1 up and running again. It's failing on test_tk, >>>> which makes sense, because it's running as a background twisted process, >>>> and thus can't access the window server. >>> It doesn't really make sense. It should skip the test, instead of >>> failing it. I.e. aborting the Python process is definitely not a good >>> response. >> >> Yes, you're right. It's a bug in the test. > > No, I'd say it's even deeper, in the Tcl integration. > > There shouldn't be a way to cause an interpreter abort, except by > calling os.abort(). This is a bug in Tk: >>> root = Tkinter.Tk() Thu May 13 20:45:13 Rivendell.local python[84887] : kCGErrorFailure: Set a breakpoint @ CGErrorBreakpoint() to catch errors as they are logged. _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL. >>> 2010-05-13 20:45:16.751 Python[84887:d07] An uncaught exception was raised 2010-05-13 20:45:16.762 Python[84887:d07] Error (1002) creating CGSWindow 2010-05-13 20:45:16.955 Python[84887:d07] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Error (1002) creating CGSWindow' *** Call stack at first throw: ( 0 CoreFoundation 0x00007fff85e31d24 __exceptionPreprocess + 180 1 libobjc.A.dylib 0x00007fff860000f3 objc_exception_throw + 45 2 CoreFoundation 0x00007fff85e31b47 +[NSException raise:format:arguments:] + 103 3 CoreFoundation 0x00007fff85e31ad4 +[NSException raise:format:] + 148 4 AppKit 0x00007fff84614aba _NSCreateWindowWithOpaqueShape2 + 473 5 AppKit 0x00007fff845a9055 -[NSWindow _commonAwake] + 1214 6 AppKit 0x00007fff845c6d3d -[NSWindow _makeKeyRegardlessOfVisibility] + 96 7 AppKit 0x00007fff845c6cb2 -[NSWindow makeKeyAndOrderFront:] + 24 8 Tk 0x000000010075b86c XMapWindow + 155 9 Tk 0x00000001006ca6d0 Tk_MapWindow + 89 10 Tk 0x00000001006d35e6 TkToplevelWindowForCommand + 2658 11 Tcl 0x00000001006300d3 TclServiceIdle + 76 12 Tcl 0x00000001006162ce Tcl_DoOneEvent + 329 13 _tkinter.so 0x0000000100595683 Tkapp_CallDeallocArgs + 277 14 readline.so 0x00000001001f1f9a initreadline + 1280 15 Python 0x00000001000076a1 PyOS_Readline + 239 16 Python 0x0000000100008a57 PyTokenizer_FromString + 1322 17 Python 0x00000001000090a0 PyTokenizer_Get + 154 18 Python 0x0000000100005698 PyParser_AddToken + 1018 19 Python 0x00000001000a2320 PyParser_ASTFromFile + 146 20 Python 0x00000001000a443f PyRun_InteractiveOneFlags + 345 21 Python 0x00000001000a4615 PyRun_InteractiveLoopFlags + 206 22 Python 0x00000001000a4685 PyRun_AnyFileExFlags + 76 23 Python 0x00000001000b0286 Py_Main + 2718 24 Python 0x0000000100000e6c start + 52 25 ??? 0x0000000000000001 0x0 + 1 ) terminate called after throwing an instance of 'NSException' Abort trap This is running /usr/bin/python in a session as a user that doesn't have access to the GUI. The text above says that there is an uncaught ObjC exception, caused by the lack of a connection to the window server. Tk should have converted that to its own style of errors but didn't. Bill: could you please file an issue for this in the python tracker, it should be possible to add a workaround for this to the Tkinter extension. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3567 bytes Desc: not available URL: From martin at v.loewis.de Thu May 13 21:10:00 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 13 May 2010 21:10:00 +0200 Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <05A4049A-BBDA-4149-AB65-6DD77716B8DA@mac.com> References: <84766.1273720679@parc.com> <4BEBA827.5040000@v.loewis.de> <81551.1273772543@parc.com> <4BEC47E9.20804@v.loewis.de> <05A4049A-BBDA-4149-AB65-6DD77716B8DA@mac.com> Message-ID: <4BEC4E88.6070705@v.loewis.de> > This is running /usr/bin/python in a session as a user that doesn't > have access to the GUI. The text above says that there is an > uncaught ObjC exception, caused by the lack of a connection to the > window server. Tk should have converted that to its own style of > errors but didn't. That makes sense to me; thanks for investigating it. If we are kind, we could also file a Tk bug, then. Regards, Martin From janssen at parc.com Thu May 13 22:44:11 2010 From: janssen at parc.com (Bill Janssen) Date: Thu, 13 May 2010 13:44:11 PDT Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <4BEC47E9.20804@v.loewis.de> References: <84766.1273720679@parc.com> <4BEBA827.5040000@v.loewis.de> <81551.1273772543@parc.com> <4BEC47E9.20804@v.loewis.de> Message-ID: <83788.1273783451@parc.com> Martin v. L?wis wrote: > Bill Janssen wrote: > > Martin v. L?wis wrote: > > > >> Bill Janssen wrote: > >>> I've got parc-tiger-1 up and running again. It's failing on test_tk, > >>> which makes sense, because it's running as a background twisted process, > >>> and thus can't access the window server. > >> It doesn't really make sense. It should skip the test, instead of > >> failing it. I.e. aborting the Python process is definitely not a good > >> response. > > > > Yes, you're right. It's a bug in the test. > > No, I'd say it's even deeper, in the Tcl integration. > > There shouldn't be a way to cause an interpreter abort, except by > calling os.abort(). There are some cases where the OS X security mechanism interprets an invalid attempt to connect to the window server as a privilege violation, and just terminates the process. I've seen that before. This is basically a resource issue in the testing framework, too, like having network access. The problem is, for lot of external libraries, it's not clear just what resources they assume are available. If we believe that Tk doesn't intend to require access to the window server, it's a Tk bug. If we know that it does, it's a resource issue. Bill From janssen at parc.com Thu May 13 23:01:15 2010 From: janssen at parc.com (Bill Janssen) Date: Thu, 13 May 2010 14:01:15 PDT Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <4BEC47E9.20804@v.loewis.de> References: <84766.1273720679@parc.com> <4BEBA827.5040000@v.loewis.de> <81551.1273772543@parc.com> <4BEC47E9.20804@v.loewis.de> Message-ID: <83959.1273784475@parc.com> Martin v. L?wis wrote: > Bill Janssen wrote: > > Martin v. L?wis wrote: > > > >> Bill Janssen wrote: > >>> I've got parc-tiger-1 up and running again. It's failing on test_tk, > >>> which makes sense, because it's running as a background twisted process, > >>> and thus can't access the window server. > >> It doesn't really make sense. It should skip the test, instead of > >> failing it. I.e. aborting the Python process is definitely not a good > >> response. > > > > Yes, you're right. It's a bug in the test. > > No, I'd say it's even deeper, in the Tcl integration. > > There shouldn't be a way to cause an interpreter abort, except by > calling os.abort(). Yes, I agree. It's an undesirable design bug in the Apple OS, IMO. When some Apple libraries ask for some things that they're not allowed to have, the library calls abort() instead of signalling an error. Google for "FAILED TO GET ASN FROM CORESERVICES" sometime. Bill From ncoghlan at gmail.com Thu May 13 23:44:07 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 May 2010 07:44:07 +1000 Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <4BEC0FCA.40601@v.loewis.de> References: <84766.1273720679@parc.com> <4BEBD6D1.8020702@gmail.com> <4BEC0FCA.40601@v.loewis.de> Message-ID: <4BEC72A7.2020107@gmail.com> Martin v. L?wis wrote: >>> I've got parc-tiger-1 up and running again. It's failing on test_tk, >>> which makes sense, because it's running as a background twisted process, >>> and thus can't access the window server. I should configure that out. >>> >>> I'm looking for documentation on how to configure the build slave so >>> that it skips this test. >> It may be better to try to detect the "no window server" case and skip >> it in the test itself rather than in the build slave configuration. > > Even better would be if Python wouldn't crash when you try to run Tk > commands without a window server. Instead of aborting Python, that > should raise an exception (which can then be detected as a test skip). Yes, when I commented I didn't realise that "failing" in this case actually meant "crashing" :P Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From urban.dani+py at gmail.com Fri May 14 06:39:12 2010 From: urban.dani+py at gmail.com (Daniel Urban) Date: Fri, 14 May 2010 06:39:12 +0200 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: References: <201005132130.30619.steve@pearwood.info> Message-ID: I wrote an e-mail yesterday, but it seems, it didn't reach python-dev. Here it is again: On Thu, May 13, 2010 at 13:30, Steven D'Aprano wrote: > I'd support an immutable dict. partial objects already impose a > significant (~ 30%) performance penalty: > >>>> from timeit import Timer >>>> min(Timer('f(5)', 'f = lambda x: x').repeat()) > 0.93580079078674316 >>>> min(Timer('p(5)', 'from functools import partial; p = partial(lambda > x: x)').repeat()) > 1.2715129852294922 > > No need to make that worse if that can be avoided. I've made a new patch, in which the keywords attribute is a read-only proxy of the dictionary. I've used your benchmark, and I haven't found any significant difference in execution times. The patch is in the tracker (http://bugs.python.org/issue8699) and Rietveld (http://codereview.appspot.com/1179044). Daniel Urban From vincent at vincentdavis.net Fri May 14 06:47:18 2010 From: vincent at vincentdavis.net (Vincent Davis) Date: Thu, 13 May 2010 22:47:18 -0600 Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <4BEC72A7.2020107@gmail.com> References: <84766.1273720679@parc.com> <4BEBD6D1.8020702@gmail.com> <4BEC0FCA.40601@v.loewis.de> <4BEC72A7.2020107@gmail.com> Message-ID: Not to interrupt you you conversation but I am interested in setting up a buildbot on one of my Macs. Is there any documentations or advise that is different from that of a linux machine? Any advise would be appreciated. Thanks Vincent On Thu, May 13, 2010 at 3:44 PM, Nick Coghlan wrote: > Martin v. L?wis wrote: >>>> I've got parc-tiger-1 up and running again. ?It's failing on test_tk, >>>> which makes sense, because it's running as a background twisted process, >>>> and thus can't access the window server. ?I should configure that out. >>>> >>>> I'm looking for documentation on how to configure the build slave so >>>> that it skips this test. >>> It may be better to try to detect the "no window server" case and skip >>> it in the test itself rather than in the build slave configuration. >> >> Even better would be if Python wouldn't crash when you try to run Tk >> commands without a window server. Instead of aborting Python, that >> should raise an exception (which can then be detected as a test skip). > > Yes, when I commented I didn't realise that "failing" in this case > actually meant "crashing" :P > > Regards, > Nick. > > -- > Nick Coghlan ? | ? ncoghlan at gmail.com ? | ? Brisbane, Australia > --------------------------------------------------------------- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/vincent%40vincentdavis.net > From martin at v.loewis.de Fri May 14 09:28:29 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 14 May 2010 09:28:29 +0200 Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: References: <84766.1273720679@parc.com> <4BEBD6D1.8020702@gmail.com> <4BEC0FCA.40601@v.loewis.de> <4BEC72A7.2020107@gmail.com> Message-ID: <4BECFB9D.8060502@v.loewis.de> Vincent Davis wrote: > Not to interrupt you you conversation but I am interested in setting > up a buildbot on one of my Macs. Is there any documentations or advise > that is different from that of a linux machine? Any advise would be > appreciated. This is a little bit out of context: what exactly do you want to set up? A buildbot master, or a buildbot slave? For running what tests? Buildbot is available from http://buildbot.net/trac Regards, Martin From status at bugs.python.org Fri May 14 18:07:37 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 14 May 2010 18:07:37 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100514160737.2E53C7814D@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-05-07 - 2010-05-14) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2678 open (+43) / 17853 closed (+23) / 20531 total (+66) Open issues with patches: 1090 Average duration of open issues: 722 days. Median duration of open issues: 495 days. Open Issues Breakdown open 2658 (+43) languishing 12 ( +0) pending 7 ( +0) Issues Created Or Reopened (72) _______________________________ __import__ with fromlist= 2010-05-08 http://bugs.python.org/issue2090 reopened brett.cannon patch SSL handshake fails after TCP connection in getpeername() 2010-05-07 CLOSED http://bugs.python.org/issue4171 reopened ddvoinikov A number tests "crash" if python is compiled --without-threads 2010-05-13 http://bugs.python.org/issue7449 reopened skrah patch, easy, needs review pep-0370 on osx duplicates existing functionality 2010-05-08 http://bugs.python.org/issue8084 reopened mark.dickinson patch, needs review, buildbot tiger buildbot: test_abspath_issue3426 failure (test_genericpa 2010-05-13 CLOSED http://bugs.python.org/issue8422 reopened flox buildbot PyUnicode_GetMax is undocumented 2010-05-07 http://bugs.python.org/issue8647 created stutzbach The UTF-7 codec functions are undocumented 2010-05-07 http://bugs.python.org/issue8648 created stutzbach Py_UNICODE_* functions are undocumented 2010-05-07 http://bugs.python.org/issue8649 created stutzbach zlibmodule.c isn't 64-bit clean 2010-05-07 http://bugs.python.org/issue8650 created pitrou "s#" and friends can silently truncate buffer length 2010-05-07 http://bugs.python.org/issue8651 created pitrou Minor improvements to the "Handling Exceptions" part of the tu 2010-05-07 http://bugs.python.org/issue8652 created marienz urlparse.urlparse/urlsplit doc missing 2010-05-07 http://bugs.python.org/issue8653 created dabrahams Improve ABI compatibility between UCS2 and UCS4 builds 2010-05-07 http://bugs.python.org/issue8654 created stutzbach patch Problem with getpeercert in the ssl module when retrieving cli 2010-05-07 http://bugs.python.org/issue8655 created Westly.Ward urllib2 mangles file://-scheme URLs 2010-05-08 CLOSED http://bugs.python.org/issue8656 created dabrahams urlparse.urlunsplit should be smarter about + 2010-05-08 CLOSED http://bugs.python.org/issue8657 created dabrahams patch urlparse.urlunsplit should be smarter about + and file urls 2010-05-08 CLOSED http://bugs.python.org/issue8658 created orsenthil patch ABS(x) == 0 can be simplified to x == 0 2010-05-08 CLOSED http://bugs.python.org/issue8659 created stutzbach patch, needs review py3k weakref documentation mentions the long built-in type 2010-05-08 CLOSED http://bugs.python.org/issue8660 created durban FAQ item 2.25 is unclear 2010-05-08 http://bugs.python.org/issue8661 created exarkun bytes join cannot join bytes 2010-05-08 CLOSED http://bugs.python.org/issue8662 created perey Failed compile in a non-ASCII path 2010-05-08 http://bugs.python.org/issue8663 created pitrou patch py_compile.compile() should consistently create parent directo 2010-05-08 CLOSED http://bugs.python.org/issue8664 created Arfrever patch "make pycremoval" fails 2010-05-08 http://bugs.python.org/issue8665 created pitrou Allow ConfigParser.get*() to take a default value 2010-05-08 http://bugs.python.org/issue8666 created Gumnos patch Link to PEP 3147 from importlib docs 2010-05-08 http://bugs.python.org/issue8667 created brett.cannon easy add a 'develop' command 2010-05-09 http://bugs.python.org/issue8668 created meatballhat lack of bdist_rpm module raises error on 'setup.py --help-comm 2010-05-09 http://bugs.python.org/issue8669 created meatballhat c_types.c_wchar should not assume that sizeof(wchar_t) == size 2010-05-09 http://bugs.python.org/issue8670 created stutzbach patch A small erorr on http://docs.python.org/library/re.html 2010-05-09 CLOSED http://bugs.python.org/issue8671 created bones7456 Error decompressing valid zlib data 2010-05-09 CLOSED http://bugs.python.org/issue8672 created matthew.brett patch configure script doesn't recognize 10.5 SDK correctly 2010-05-10 CLOSED http://bugs.python.org/issue8673 created ronaldoussoren audioop: incorrect integer overflow checks 2010-05-10 CLOSED http://bugs.python.org/issue8674 created thoger patch audioop module needs an int -> Py_ssize_t upgrade 2010-05-10 CLOSED http://bugs.python.org/issue8675 created mark.dickinson patch, easy Py3k Built-in Exceptions documentation mentions the raise stat 2010-05-10 CLOSED http://bugs.python.org/issue8676 created durban Modules needing PY_SSIZE_T_CLEAN 2010-05-10 http://bugs.python.org/issue8677 created pitrou crashers in rgbimg 2010-05-10 http://bugs.python.org/issue8678 created brett.cannon write a distutils to distutils2 converter 2010-05-10 http://bugs.python.org/issue8679 created tarek Add a sandbox in Distutils2 2010-05-10 http://bugs.python.org/issue8680 created tarek Make the zlib module emit more detailed error messages 2010-05-10 CLOSED http://bugs.python.org/issue8681 created pitrou patch _ssl.c uses PyWeakref_GetObject but doesn't incref result 2010-05-10 http://bugs.python.org/issue8682 created pitrou HPUX Segmentation fault in Modules/gcmodule.c -- if (!gc_list_ 2010-05-11 CLOSED http://bugs.python.org/issue8683 created srid improvements to sched.py 2010-05-11 http://bugs.python.org/issue8684 created josiahcarlson patch, patch, needs review set(range(100000)).difference(set()) is slow 2010-05-11 http://bugs.python.org/issue8685 created spiv patch "This isn't defined beyond that" phrase is not friendly to non 2010-05-11 http://bugs.python.org/issue8686 created naoki sched.py module doesn't have a test suite 2010-05-11 http://bugs.python.org/issue8687 created giampaolo.rodola patch distutils sdist is too laze w.r.t. recalculating MANIFEST 2010-05-11 http://bugs.python.org/issue8688 created ronaldoussoren sqlite3 parameter substitution breaks with multiple parameters 2010-05-11 http://bugs.python.org/issue8689 created azriphale multiprocessing.dummy.Queue does not expose same interface as 2010-05-11 http://bugs.python.org/issue8690 created Jason.Baker Doc: left alignment is not the default for numbers 2010-05-11 http://bugs.python.org/issue8691 created tjreedy Use divide-and-conquer for faster factorials 2010-05-11 http://bugs.python.org/issue8692 created stutzbach patch, patch, needs review Py_Initialize: can't initialize sys standard streams 2010-05-12 CLOSED http://bugs.python.org/issue8693 reopened mbnoimi python3 FAQ mentions unicode() 2010-05-12 http://bugs.python.org/issue8694 created ohnobinki Issue while installing Python 2.6.5 in IBM AIX 6.1 2010-05-12 http://bugs.python.org/issue8695 created senthil_l PEP 391: Adding documentation of logging.config.dictConfig 2010-05-12 CLOSED http://bugs.python.org/issue8696 created akuchling patch PEP 391: Adding documentation of logging.config.dictConfig 2010-05-12 CLOSED http://bugs.python.org/issue8697 created akuchling patch, needs review PEP 391: Adding documentation of logging.config.dictConfig 2010-05-12 CLOSED http://bugs.python.org/issue8698 created akuchling patch, needs review Equality and hashing for functools.partial 2010-05-12 http://bugs.python.org/issue8699 created durban patch strip() is removing an extra character if the strip pattern co 2010-05-12 CLOSED http://bugs.python.org/issue8700 created Balachander.Ganesan tarfile: first character of member names doubled 2010-05-12 CLOSED http://bugs.python.org/issue8701 created pooryorick difflib: unified_diff produces wrong patches (again) 2010-05-13 CLOSED http://bugs.python.org/issue8702 created techtonik easy Py3k PyList_Type documentation mentions types.ListType 2010-05-13 CLOSED http://bugs.python.org/issue8703 created durban cgitb sends a bogus HTTP header if the app crashes before fini 2010-05-13 http://bugs.python.org/issue8704 created stutzbach easy shutil.rmtree with empty filepath 2010-05-13 http://bugs.python.org/issue8705 created dbkoch accept keyword arguments on all base type methods and builtins 2010-05-13 http://bugs.python.org/issue8706 created gregory.p.smith Duplicated document in telnetlib. 2010-05-13 http://bugs.python.org/issue8707 created naoki OpenID blunder 2010-05-14 http://bugs.python.org/issue8708 created Jack.Diederich mention explicitly Windows support for os.devnull 2010-05-14 http://bugs.python.org/issue8709 created akira patch test_xpickle: compat tests: workaround for missing test_suppor 2010-05-14 http://bugs.python.org/issue8710 created skrah Document PyUnicode_DecodeFSDefault() and PyUnicode_DecodeFSDef 2010-05-14 http://bugs.python.org/issue8711 created haypo patch Skip libpthread related test failures on OpenBSD 2010-05-14 http://bugs.python.org/issue8712 created skrah patch, needs review bdist_deb - Debian packager 2010-05-07 CLOSED http://bugs.python.org/issue1054967 reopened merwok patch Issues Now Closed (54) ______________________ cmath is numerically unsound 920 days http://bugs.python.org/issue1381 bins patch test_uuid is warning about unreliable functions 904 days http://bugs.python.org/issue1481 skrah patch Add a factorial function 786 days http://bugs.python.org/issue2138 mark.dickinson Py_FatalError causes infinite loop 632 days http://bugs.python.org/issue3605 jyasskin needs review SSL handshake fails after TCP connection in getpeername() 1 days http://bugs.python.org/issue4171 pitrou {context,unified}_diff add spurious trailing whitespace if fro 490 days http://bugs.python.org/issue4898 r.david.murray patch Implement PEP 376 484 days http://bugs.python.org/issue4908 tarek patch subprocess.Popen.__del__ causes AttributeError (os module == N 469 days http://bugs.python.org/issue5099 brett.cannon patch, needs review test_smtplib fails on os x 10.6 220 days http://bugs.python.org/issue7040 giampaolo.rodola difflib should separate filename from timestamp with tab 137 days http://bugs.python.org/issue7585 techtonik patch unittest: allow an 'import_path' as an alternative to 'top_lev 103 days http://bugs.python.org/issue7780 michael.foord Provide unittest.TestCase.assertNotRegexpMatches 67 days http://bugs.python.org/issue8038 michael.foord patch Python 3.1.2rc1 doesn't compile using the 10.4 sdk on a 10.6 M 62 days http://bugs.python.org/issue8126 ronaldoussoren Putting a function in a unittest.TestSuite doesn't work 35 days http://bugs.python.org/issue8301 michael.foord python -m unittest -h and python -m unittest discover -h messa 37 days http://bugs.python.org/issue8303 michael.foord siginterrupt with flag=False is reset when signal received 30 days http://bugs.python.org/issue8354 exarkun patch, needs review tiger buildbot: test_abspath_issue3426 failure (test_genericpa 0 days http://bugs.python.org/issue8422 haypo buildbot openssl version detection doesn't work properly when using OSX 20 days http://bugs.python.org/issue8444 ronaldoussoren asyncore test suite 19 days http://bugs.python.org/issue8490 giampaolo.rodola patch, patch update to autoconf2.65 18 days http://bugs.python.org/issue8510 mark.dickinson patch, needs review Add fsencode() functions to os module 15 days http://bugs.python.org/issue8514 haypo patch unittest test discovery can fail when package under test is al 11 days http://bugs.python.org/issue8547 michael.foord zlib causes a SystemError when decompressing a chunk >1GB 8 days http://bugs.python.org/issue8571 pitrou patch Update/reorganize _winreg documentation 12 days http://bugs.python.org/issue8575 brian.curtin patch, patch, needs review test_urllib2.py test failures on Py3K Mac OS X 8 days http://bugs.python.org/issue8588 mark.dickinson enumerate() test cases do not cover optional start argument 2 days http://bugs.python.org/issue8636 benjamin.peterson patch json.loads description 3 days http://bugs.python.org/issue8642 georg.brandl patch timedelta.total_seconds needlessly inaccurate, especially for 2 days http://bugs.python.org/issue8644 mark.dickinson patch urllib2 mangles file://-scheme URLs 0 days http://bugs.python.org/issue8656 orsenthil urlparse.urlunsplit should be smarter about + 5 days http://bugs.python.org/issue8657 orsenthil patch urlparse.urlunsplit should be smarter about + and file urls 0 days http://bugs.python.org/issue8658 orsenthil patch ABS(x) == 0 can be simplified to x == 0 0 days http://bugs.python.org/issue8659 mark.dickinson patch, needs review py3k weakref documentation mentions the long built-in type 0 days http://bugs.python.org/issue8660 benjamin.peterson bytes join cannot join bytes 0 days http://bugs.python.org/issue8662 perey py_compile.compile() should consistently create parent directo 0 days http://bugs.python.org/issue8664 benjamin.peterson patch A small erorr on http://docs.python.org/library/re.html 0 days http://bugs.python.org/issue8671 eric.smith Error decompressing valid zlib data 2 days http://bugs.python.org/issue8672 pitrou patch configure script doesn't recognize 10.5 SDK correctly 3 days http://bugs.python.org/issue8673 ronaldoussoren audioop: incorrect integer overflow checks 1 days http://bugs.python.org/issue8674 mark.dickinson patch audioop module needs an int -> Py_ssize_t upgrade 1 days http://bugs.python.org/issue8675 mark.dickinson patch, easy Py3k Built-in Exceptions documentation mentions the raise stat 0 days http://bugs.python.org/issue8676 benjamin.peterson Make the zlib module emit more detailed error messages 1 days http://bugs.python.org/issue8681 pitrou patch HPUX Segmentation fault in Modules/gcmodule.c -- if (!gc_list_ 1 days http://bugs.python.org/issue8683 pitrou Py_Initialize: can't initialize sys standard streams 0 days http://bugs.python.org/issue8693 mbnoimi PEP 391: Adding documentation of logging.config.dictConfig 0 days http://bugs.python.org/issue8696 akuchling patch PEP 391: Adding documentation of logging.config.dictConfig 0 days http://bugs.python.org/issue8697 akuchling patch, needs review PEP 391: Adding documentation of logging.config.dictConfig 1 days http://bugs.python.org/issue8698 meatballhat patch, needs review strip() is removing an extra character if the strip pattern co 0 days http://bugs.python.org/issue8700 r.david.murray tarfile: first character of member names doubled 1 days http://bugs.python.org/issue8701 pooryorick difflib: unified_diff produces wrong patches (again) 0 days http://bugs.python.org/issue8702 mark.dickinson easy Py3k PyList_Type documentation mentions types.ListType 0 days http://bugs.python.org/issue8703 benjamin.peterson Carbon Event ReceiveNextEvent 2479 days http://bugs.python.org/issue779285 tjreedy bdist_deb - Debian packager 0 days http://bugs.python.org/issue1054967 tarek patch update urlparse to RFC 3986 1280 days http://bugs.python.org/issue1591035 orsenthil easy Top Issues Most Discussed (10) ______________________________ 45 Use divide-and-conquer for faster factorials 3 days open http://bugs.python.org/issue8692 25 Improve ABI compatibility between UCS2 and UCS4 builds 7 days open http://bugs.python.org/issue8654 14 Failed compile in a non-ASCII path 6 days open http://bugs.python.org/issue8663 11 let's equip ftplib.FTP with __enter__ and __exit__ 482 days open http://bugs.python.org/issue4972 10 add a distutils test command 39 days open http://bugs.python.org/issue8324 10 cookielib doesn't handle URLs with / in parameters 625 days open http://bugs.python.org/issue3704 9 Py_Initialize: can't initialize sys standard streams 0 days closed http://bugs.python.org/issue8693 9 timedelta.total_seconds needlessly inaccurate, especially for n 2 days closed http://bugs.python.org/issue8644 9 update to autoconf2.65 18 days closed http://bugs.python.org/issue8510 9 input() doesn't catch _PyUnicode_AsString() exception; io.Strin 47 days open http://bugs.python.org/issue8256 From janssen at parc.com Fri May 14 18:15:06 2010 From: janssen at parc.com (Bill Janssen) Date: Fri, 14 May 2010 09:15:06 PDT Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: References: <84766.1273720679@parc.com> <4BEBD6D1.8020702@gmail.com> <4BEC0FCA.40601@v.loewis.de> <4BEC72A7.2020107@gmail.com> Message-ID: <83407.1273853706@parc.com> Vincent Davis wrote: > Not to interrupt you you conversation but I am interested in setting > up a buildbot on one of my Macs. Is there any documentations or advise > that is different from that of a linux machine? Any advise would be > appreciated. Assuming you mean "set up a build slave for Python testing"... Here's what I did. I was installing on a Tiger machine, which comes with Python 2.3.5, so the first thing I did was to install Xcode 2.5, the latest release for Tiger. I also needed Subversion, so I downloaded and installed that. Later versions of OS X Xcode include Subversion, so you won't need to do it separately if you're using those. I then installed Python 2.6.5 from python.org, which winds up in /usr/local/bin, and modified my PATH to put /usr/local/bin on it. I downloaded the source bundles for Twisted, zope.interface, and buildbot, and installed them in that order -- "sudo python setup.py install" works fine for each of them. After that, I followed the instructions on the Wiki at http://wiki.python.org/moin/BuildBot. Create a buildbot account, make sure /usr/local/bin is on the PATH of the buildbot account, and issue the commands shown there. Bill From vincent at vincentdavis.net Fri May 14 18:28:12 2010 From: vincent at vincentdavis.net (Vincent Davis) Date: Fri, 14 May 2010 10:28:12 -0600 Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <83407.1273853706@parc.com> References: <84766.1273720679@parc.com> <4BEBD6D1.8020702@gmail.com> <4BEC0FCA.40601@v.loewis.de> <4BEC72A7.2020107@gmail.com> <83407.1273853706@parc.com> Message-ID: On Fri, May 14, 2010 at 10:15 AM, Bill Janssen wrote: > Vincent Davis wrote: > > > Not to interrupt you you conversation but I am interested in setting > > up a buildbot on one of my Macs. Is there any documentations or advise > > that is different from that of a linux machine? Any advise would be > > appreciated. > > Assuming you mean "set up a build slave for Python testing"... > > Here's what I did. I was installing on a Tiger machine, which comes > with Python 2.3.5, so the first thing I did was to install Xcode 2.5, > the latest release for Tiger. I also needed Subversion, so I downloaded > and installed that. Later versions of OS X Xcode include Subversion, so > you won't need to do it separately if you're using those. > > I then installed Python 2.6.5 from python.org, which winds up in > /usr/local/bin, and modified my PATH to put /usr/local/bin on it. I > downloaded the source bundles for Twisted, zope.interface, and buildbot, > and installed them in that order -- "sudo python setup.py install" works > fine for each of them. > > After that, I followed the instructions on the Wiki at > http://wiki.python.org/moin/BuildBot. Create a buildbot account, make > sure /usr/local/bin is on the PATH of the buildbot account, and issue > the commands shown there. Yes thanks this is what I was thinking "set up a build slave for Python testing", No reason this would not work on a leopard 10.6 machine? Thanks *Vincent Davis 720-301-3003 * vincent at vincentdavis.net my blog | LinkedIn -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Fri May 14 19:03:49 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 14 May 2010 11:03:49 -0600 Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0 In-Reply-To: References: <4BEAA971.3040001@jcea.es> Message-ID: On Wed, May 12, 2010 at 9:58 AM, Jeffrey Yasskin wrote: > On Wed, May 12, 2010 at 6:39 AM, James Y Knight wrote: >> I think you'll be a lot happier just modifying Psyco than making everyone >> else in the world change their compiler flags. > > Aye, there's the rub. Nobody's happier modifying Psyco. :) ?But that > just means people will gradually have to stop using psyco in favor of > maintainable JITs. Gcc's not going to change its stack requirements > just because some people think they know what the ABI should be better > than the people defining the ABI. Btw, this has been a problem since > at least gcc-4.4. Heh. psyco already does it for Mac OS X (which defines 16 byte stack alignment as ABI), so should be super trivial. Good rant Jeffrey though. Cheers, fijal From martin at v.loewis.de Fri May 14 19:56:29 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 14 May 2010 19:56:29 +0200 Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: References: <84766.1273720679@parc.com> <4BEBD6D1.8020702@gmail.com> <4BEC0FCA.40601@v.loewis.de> <4BEC72A7.2020107@gmail.com> <83407.1273853706@parc.com> Message-ID: <4BED8ECD.5000800@v.loewis.de> > Yes thanks this is what I was thinking "set up a build slave for Python > testing", No reason this would not work on a leopard 10.6 machine? In my experience, it is mandatory that the slave admin has really good understanding of Python, and of the operating system that the slave runs on. Otherwise, the slave will be down most of the time, and just not function correctly; it was then a waste of time to set it up in the first place. Regards, Martin From janssen at parc.com Fri May 14 20:23:32 2010 From: janssen at parc.com (Bill Janssen) Date: Fri, 14 May 2010 11:23:32 PDT Subject: [Python-Dev] configuring the buildbot to skip some tests? In-Reply-To: <05A4049A-BBDA-4149-AB65-6DD77716B8DA@mac.com> References: <84766.1273720679@parc.com> <4BEBA827.5040000@v.loewis.de> <81551.1273772543@parc.com> <4BEC47E9.20804@v.loewis.de> <05A4049A-BBDA-4149-AB65-6DD77716B8DA@mac.com> Message-ID: <85682.1273861412@parc.com> Ronald Oussoren wrote: > Bill: could you please file an issue for this in the python tracker, http://bugs.python.org/issue8716 > it should be possible to add a workaround for this to the Tkinter > extension. That would be good. Bill From urban.dani at gmail.com Thu May 13 18:48:15 2010 From: urban.dani at gmail.com (Daniel Urban) Date: Thu, 13 May 2010 18:48:15 +0200 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: <201005132130.30619.steve@pearwood.info> References: <201005132130.30619.steve@pearwood.info> Message-ID: On Thu, May 13, 2010 at 13:30, Steven D'Aprano wrote: > I'd support an immutable dict. partial objects already impose a > significant (~ 30%) performance penalty: > >>>> from timeit import Timer >>>> min(Timer('f(5)', 'f = lambda x: x').repeat()) > 0.93580079078674316 >>>> min(Timer('p(5)', 'from functools import partial; p = partial(lambda > x: x)').repeat()) > 1.2715129852294922 > > No need to make that worse if that can be avoided. I've made a new patch, in which the keywords attribute is a read-only proxy of the dictionary. I've used your benchmark, and I haven't found any significant difference in execution times. The patch is in the tracker (http://bugs.python.org/issue8699) and Rietveld (http://codereview.appspot.com/1179044). Daniel Urban From kiko at meebo-inc.com Fri May 14 01:26:17 2010 From: kiko at meebo-inc.com (Kiko Griffin) Date: Thu, 13 May 2010 16:26:17 -0700 Subject: [Python-Dev] Python Web Developer @ Meebo (Perm in Mountain View, CA.) Message-ID: Hello All, Meebo is looking for a bright, fun, dedicated software engineer who's interested in supporting a growing revenue team, loves a good challenge, and wants to learn about how advertising operations works from the inside out. In this role, you will be responsible for implementing and maintaining critical tools that support the ongoing adops infrastructure, ranging from content acquisition to ad inventory management. As a developer on the ads team, you'll be exposed to many business functions and you'll support and develop tools and systems to streamline the entire process. Our team is fun, friendly, and willing to work really hard to meet our goals. We love a great user experience, internal and external. *Responsibilities:* - Work with ads team to create and maintain mission critical support tools - Work closely with Sales and Marketing teams to elicit feedback from partners - Design and implement partner facing tools - Participate in product meetings to determine necessary systems - Scale and optimize with a growing revenue stream - Interface with both server and client side teams to leverage existing tools *What you will need:* - 3-5+ years of software engineering experience in Linux/Unix - BS, MS in Computer Science (or equivalent) - Strong expertise in Python, Perl, php, or other scripting languages - Knowledge of Django, PIG a plus - Database experience a plus - Ability to work with technical and non-technical teams, in an extremely fast-paced environment - Strong UI design sense with the ability to adapt, take critical feedback, and execute quickly on tasks *Big pluses:* - Experience working with large scale systems - Experience working with ad ops industry tools such as: Doubleclick, Atlas, 24/7, Salesforce, Dart Sales Manager, Solbright, Operative If at all interested OR know someone who might be interested please reply here Kiko at meebo-inc.com. Thanks, Kiko Griffin kiko at meebo-inc.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat May 15 16:42:12 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 15 May 2010 10:42:12 -0400 Subject: [Python-Dev] Python Web Developer @ Meebo (Perm in Mountain View, CA.) In-Reply-To: References: Message-ID: On 5/13/2010 7:26 PM, Kiko Griffin wrote: Dear Kiko - The python-dev mailing list and the gmane.comp.python.devel mirror are for development of Python and CPython. Job announcements are considered off-topic spam and constitute a dis-promotion for your firm. Please do not repeat. Please redirect your efforts to the python.org job board where you can post such announcements. http://www.python.org/community/jobs/ From solipsis at pitrou.net Sat May 15 19:48:39 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 15 May 2010 19:48:39 +0200 Subject: [Python-Dev] robots exclusion file on the buildbot pages? Message-ID: <20100515194839.22b71aa6@pitrou.net> Hello, The buildbots are sometimes subject to a flood of "svn exception" errors. It has been conjectured that these errors are caused by Web crawlers pressing "force build" buttons without filling any of the fields (of course, the fact that we get such ugly errors in the buildbot results, rather than a clean error message when pressing the button, is a buildbot bug in itself). Couldn't we simply exclude all crawlers from the buildbot Web pages? Regards Antoine. From exarkun at twistedmatrix.com Sat May 15 19:57:28 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sat, 15 May 2010 17:57:28 -0000 Subject: [Python-Dev] robots exclusion file on the buildbot pages? In-Reply-To: <20100515194839.22b71aa6@pitrou.net> References: <20100515194839.22b71aa6@pitrou.net> Message-ID: <20100515175728.1632.1573034403.divmod.xquotient.3@localhost.localdomain> On 05:48 pm, solipsis at pitrou.net wrote: > >Hello, > >The buildbots are sometimes subject to a flood of "svn exception" >errors. It has been conjectured that these errors are caused by Web >crawlers pressing "force build" buttons without filling any of the >fields (of course, the fact that we get such ugly errors in the >buildbot results, rather than a clean error message when pressing >the button, is a buildbot bug in itself). Couldn't we simply exclude >all >crawlers from the buildbot Web pages? Most (all?) legitimate crawlers won't submit forms. Do you think there's a non-form link to the force build URL (which _will_ accept a GET request to mean the same thing as a POST)? One thing I have noticed is that spammers find these forms and submit them with garbage. We can probably suppose that such people are going to ignore a robots.txt file. Jean-Paul From solipsis at pitrou.net Sat May 15 20:07:09 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 15 May 2010 20:07:09 +0200 Subject: [Python-Dev] robots exclusion file on the buildbot pages? References: <20100515194839.22b71aa6@pitrou.net> <20100515175728.1632.1573034403.divmod.xquotient.3@localhost.localdomain> Message-ID: <20100515200709.64be2103@pitrou.net> On Sat, 15 May 2010 17:57:28 -0000 exarkun at twistedmatrix.com wrote: > > One thing I have noticed is that spammers find these forms and submit > them with garbage. We can probably suppose that such people are going > to ignore a robots.txt file. So we could "just" fix the buggy buildbot code. Not that I want to do it myself :S From fijall at gmail.com Sat May 15 21:06:15 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 15 May 2010 13:06:15 -0600 Subject: [Python-Dev] robots exclusion file on the buildbot pages? In-Reply-To: <20100515200709.64be2103@pitrou.net> References: <20100515194839.22b71aa6@pitrou.net> <20100515175728.1632.1573034403.divmod.xquotient.3@localhost.localdomain> <20100515200709.64be2103@pitrou.net> Message-ID: On Sat, May 15, 2010 at 12:07 PM, Antoine Pitrou wrote: > On Sat, 15 May 2010 17:57:28 -0000 > exarkun at twistedmatrix.com wrote: >> >> One thing I have noticed is that spammers find these forms and submit >> them with garbage. ?We can probably suppose that such people are going >> to ignore a robots.txt file. > > So we could "just" fix the buggy buildbot code. > Not that I want to do it myself :S > I can help modify buildbot if you want, but I suppose I need a specification what precisely is a bug here. Not accepting forms with garbage? By default buildbot "force build" does not require forms to be filled and that's on purpose. From solipsis at pitrou.net Sat May 15 21:12:14 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 15 May 2010 21:12:14 +0200 Subject: [Python-Dev] robots exclusion file on the buildbot pages? In-Reply-To: References: <20100515194839.22b71aa6@pitrou.net> <20100515175728.1632.1573034403.divmod.xquotient.3@localhost.localdomain> <20100515200709.64be2103@pitrou.net> Message-ID: <20100515211214.0d001162@pitrou.net> Hi, > I can help modify buildbot if you want, but I suppose I need a > specification what precisely is a bug here. Not accepting forms with > garbage? By default buildbot "force build" does not require forms to > be filled and that's on purpose. Well, the "fix" would be to forbid an empty "Branch to build" since it doesn't point to anything buildable (even worse, the fact that the branch then ends up as None rather than an empty string produces an exception in buildbot code). I'm not sure what the process to get the fix in would be, but it probably involves discussing it with Martin :) Regards Antoine. From martin at v.loewis.de Sat May 15 21:49:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 15 May 2010 21:49:07 +0200 Subject: [Python-Dev] robots exclusion file on the buildbot pages? In-Reply-To: <20100515194839.22b71aa6@pitrou.net> References: <20100515194839.22b71aa6@pitrou.net> Message-ID: <4BEEFAB3.4010907@v.loewis.de> > The buildbots are sometimes subject to a flood of "svn exception" > errors. It has been conjectured that these errors are caused by Web > crawlers pressing "force build" buttons without filling any of the > fields (of course, the fact that we get such ugly errors in the > buildbot results, rather than a clean error message when pressing > the button, is a buildbot bug in itself). Couldn't we simply exclude all > crawlers from the buildbot Web pages? Hmm. Before doing any modifications, I'd rather have a definite analysis on this. Are you absolutely certain that, when that happened, the individual builds that caused this svn exception where actually triggered over the web, rather than by checkin? When it happens next, please report exact date and time, and the build log URL. Due to log rotation, it would then be necessary to investigate that in a timely manner. Without any reference to the specific case, I'd guess that a flood of svn exceptions is caused due to an svn outage, which in turn might be caused when a build is triggered while the daily Apache restart happens (i.e. around 6:30 UTC+2). That said: /dev/buildbot has been disallowed for all robots for quite some time now: http://www.python.org/robots.txt There is really no point robots crawling the build logs, as they don't contain much useful information for a search engine. Regards, Martin From solipsis at pitrou.net Sat May 15 21:55:13 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 15 May 2010 21:55:13 +0200 Subject: [Python-Dev] robots exclusion file on the buildbot pages? In-Reply-To: <4BEEFAB3.4010907@v.loewis.de> References: <20100515194839.22b71aa6@pitrou.net> <4BEEFAB3.4010907@v.loewis.de> Message-ID: <20100515215513.24991b4d@pitrou.net> On Sat, 15 May 2010 21:49:07 +0200 "Martin v. L?wis" wrote: > > Hmm. Before doing any modifications, I'd rather have a definite analysis > on this. Are you absolutely certain that, when that happened, the > individual builds that caused this svn exception where actually > triggered over the web, rather than by checkin? How can I be "absolutely certain"? As I said, it's a conjecture, and the suggested fix is just that: a suggestion. > When it happens next, please report exact date and time, and the build > log URL. Due to log rotation, it would then be necessary to investigate > that in a timely manner. Please take a look at http://www.python.org/dev/buildbot/trunk/. There are still a bunch of violet buildbots there. For example: http://www.python.org/dev/buildbot/builders/sparc%20Ubuntu%20trunk/builds/175 http://www.python.org/dev/buildbot/builders/sparc%20Ubuntu%20trunk/builds/176 Regards Antoine. From fijall at gmail.com Sat May 15 22:01:26 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 15 May 2010 14:01:26 -0600 Subject: [Python-Dev] robots exclusion file on the buildbot pages? In-Reply-To: <20100515215513.24991b4d@pitrou.net> References: <20100515194839.22b71aa6@pitrou.net> <4BEEFAB3.4010907@v.loewis.de> <20100515215513.24991b4d@pitrou.net> Message-ID: For us at least no branch specified builds the default branch (trunk) and does not end up with exception in buildbot code. How about specifying the default branch in config file? On Sat, May 15, 2010 at 1:55 PM, Antoine Pitrou wrote: > On Sat, 15 May 2010 21:49:07 +0200 > "Martin v. L?wis" wrote: >> >> Hmm. Before doing any modifications, I'd rather have a definite analysis >> on this. Are you absolutely certain that, when that happened, the >> individual builds that caused this svn exception where actually >> triggered over the web, rather than by checkin? > > How can I be "absolutely certain"? As I said, it's a conjecture, and > the suggested fix is just that: a suggestion. > >> When it happens next, please report exact date and time, and the build >> log URL. Due to log rotation, it would then be necessary to investigate >> that in a timely manner. > > Please take a look at http://www.python.org/dev/buildbot/trunk/. There > are still a bunch of violet buildbots there. > For example: > http://www.python.org/dev/buildbot/builders/sparc%20Ubuntu%20trunk/builds/175 > http://www.python.org/dev/buildbot/builders/sparc%20Ubuntu%20trunk/builds/176 > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > From janssen at parc.com Sat May 15 22:01:46 2010 From: janssen at parc.com (Bill Janssen) Date: Sat, 15 May 2010 13:01:46 PDT Subject: [Python-Dev] robots exclusion file on the buildbot pages? In-Reply-To: <20100515194839.22b71aa6@pitrou.net> References: <20100515194839.22b71aa6@pitrou.net> Message-ID: <829.1273953706@parc.com> Antoine Pitrou wrote: > The buildbots are sometimes subject to a flood of "svn exception" > errors. It has been conjectured that these errors are caused by Web > crawlers pressing "force build" buttons without filling any of the > fields (of course, the fact that we get such ugly errors in the > buildbot results, rather than a clean error message when pressing > the button, is a buildbot bug in itself). Couldn't we simply exclude all > crawlers from the buildbot Web pages? I caused a few of those myself yesterday updating my PPC buildbots. Apologies! Bill From janssen at parc.com Sat May 15 22:03:59 2010 From: janssen at parc.com (Bill Janssen) Date: Sat, 15 May 2010 13:03:59 PDT Subject: [Python-Dev] robots exclusion file on the buildbot pages? In-Reply-To: References: <20100515194839.22b71aa6@pitrou.net> <20100515175728.1632.1573034403.divmod.xquotient.3@localhost.localdomain> <20100515200709.64be2103@pitrou.net> Message-ID: <848.1273953839@parc.com> Maciej Fijalkowski wrote: > On Sat, May 15, 2010 at 12:07 PM, Antoine Pitrou wrote: > > On Sat, 15 May 2010 17:57:28 -0000 > > exarkun at twistedmatrix.com wrote: > >> > >> One thing I have noticed is that spammers find these forms and submit > >> them with garbage. ?We can probably suppose that such people are going > >> to ignore a robots.txt file. > > > > So we could "just" fix the buggy buildbot code. > > Not that I want to do it myself :S > > > > I can help modify buildbot if you want, but I suppose I need a > specification what precisely is a bug here. Not accepting forms with > garbage? By default buildbot "force build" does not require forms to > be filled and that's on purpose. I'd find it useful if the "branch" field was a choice pull-down listing valid branches, rather than a plain text field, and if the "revision" field always defaulted to "HEAD". Seems to me that since the form is coming from the buildmaster, that should be possible. Bill From janssen at parc.com Sat May 15 22:09:12 2010 From: janssen at parc.com (Bill Janssen) Date: Sat, 15 May 2010 13:09:12 PDT Subject: [Python-Dev] buildbot svn exceptions again... Message-ID: <898.1273954152@parc.com> parc-leopard-1 (and most of the other builders) are failing the svn checkout with the following error: svn: PROPFIND of '/projects/python/trunk': Could not resolve hostname `svn.python.org': Temporary failure in name resolution (http://svn.python.org) When I log into that machine as "buildbot" and do the svn checkout manually using the following command /usr/bin/svn checkout --non-interactive --no-auth-cache --revision HEAD http://svn.python.org/projects/python/trunk build it works fine. Bill From solipsis at pitrou.net Sat May 15 22:17:15 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 15 May 2010 22:17:15 +0200 Subject: [Python-Dev] robots exclusion file on the buildbot pages? In-Reply-To: <848.1273953839@parc.com> References: <20100515194839.22b71aa6@pitrou.net> <20100515175728.1632.1573034403.divmod.xquotient.3@localhost.localdomain> <20100515200709.64be2103@pitrou.net> <848.1273953839@parc.com> Message-ID: <20100515221715.5b76caaa@pitrou.net> On Sat, 15 May 2010 13:03:59 PDT Bill Janssen wrote: > I'd find it useful if the "branch" field was a choice pull-down listing > valid branches, rather than a plain text field, and if the "revision" > field always defaulted to "HEAD". Seems to me that since the form is > coming from the buildmaster, that should be possible. Actually, I think that it does already default to HEAD if you leave it empty. Regards Antoine. From martin at v.loewis.de Sat May 15 22:28:23 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 15 May 2010 22:28:23 +0200 Subject: [Python-Dev] robots exclusion file on the buildbot pages? In-Reply-To: <20100515215513.24991b4d@pitrou.net> References: <20100515194839.22b71aa6@pitrou.net> <4BEEFAB3.4010907@v.loewis.de> <20100515215513.24991b4d@pitrou.net> Message-ID: <4BEF03E7.9000400@v.loewis.de> >> Hmm. Before doing any modifications, I'd rather have a definite analysis >> on this. Are you absolutely certain that, when that happened, the >> individual builds that caused this svn exception where actually >> triggered over the web, rather than by checkin? > > How can I be "absolutely certain"? In the example you gave, the build log says "The web-page 'force build' button was pressed by '': " So ISTM that it's indeed certain that the build was triggered over the web, rather than by a checkin. > http://www.python.org/dev/buildbot/builders/sparc%20Ubuntu%20trunk/builds/175 AFAICT from the twistd logs, the user agent triggering this build was "Mozilla/4.7 [ja] (Win98; I)". It still may have been a bot; it was using a GET request, even though the form asks for a POST. The IP address points to some Japanese dialup network (reverse lookup reports address.dy.bbexcite.jp.) The bot probably has malicious intent: it has been using about 10 different user-agent strings, on various parts of the site. I have now blackholed this IP address (although it stopped contacting python.org around 8 hours ago, anyway). If desired, we could password-protect the "force build" forms. If that is to be done, some help from a buildbot expert on what to change specifically would be appreciated. Regards, Martin From martin at v.loewis.de Sat May 15 22:31:21 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 15 May 2010 22:31:21 +0200 Subject: [Python-Dev] robots exclusion file on the buildbot pages? In-Reply-To: <829.1273953706@parc.com> References: <20100515194839.22b71aa6@pitrou.net> <829.1273953706@parc.com> Message-ID: <4BEF0499.2020905@v.loewis.de> > I caused a few of those myself yesterday updating my PPC buildbots. > > Apologies! No need to apologize! these are not the ones Antoine is talking about. By convention, filling out the "Your name" field in a web build is recommended, so people know that this was an intentional build. I usually also fill out a reason. Regards, Martin From martin at v.loewis.de Sat May 15 22:32:54 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 15 May 2010 22:32:54 +0200 Subject: [Python-Dev] robots exclusion file on the buildbot pages? In-Reply-To: <848.1273953839@parc.com> References: <20100515194839.22b71aa6@pitrou.net> <20100515175728.1632.1573034403.divmod.xquotient.3@localhost.localdomain> <20100515200709.64be2103@pitrou.net> <848.1273953839@parc.com> Message-ID: <4BEF04F6.3090308@v.loewis.de> > I'd find it useful if the "branch" field was a choice pull-down listing > valid branches, rather than a plain text field, and if the "revision" > field always defaulted to "HEAD". Seems to me that since the form is > coming from the buildmaster, that should be possible. Unfortunately, these forms are deeply hidden in the buildbot code. So I'd rather avoid editing them, or else upgrading to the next buildbot version becomes even more tedious. Regards, Martin From martin at v.loewis.de Sat May 15 22:45:35 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 15 May 2010 22:45:35 +0200 Subject: [Python-Dev] buildbot svn exceptions again... In-Reply-To: <898.1273954152@parc.com> References: <898.1273954152@parc.com> Message-ID: <4BEF07EF.8070900@v.loewis.de> Bill Janssen wrote: > parc-leopard-1 (and most of the other builders) are failing the svn > checkout with the following error: > > svn: PROPFIND of '/projects/python/trunk': Could not resolve hostname > `svn.python.org': Temporary failure in name resolution > (http://svn.python.org) > > When I log into that machine as "buildbot" and do the svn checkout > manually using the following command > > /usr/bin/svn checkout --non-interactive --no-auth-cache --revision > HEAD http://svn.python.org/projects/python/trunk build > > it works fine. That's a common OSX problem/bug. Processes occasionally lose the ability to resolve host names. Various theories float around what's causing this (most commonly, people expect that a "controlling terminal" must be present); my theory is this: There is a Mach port in the system that lets a process talk to the local resolver (lookupd, or whatever Apple's naming infrastructure is called today). For some reason, this port gets closed in a "background" process, perhaps because the parent process closed it when you logged out. Or the Kerberos credentials expired, or some such... In short, if some OS X expert could shed some light on this, it would be appreciated. It would then be good to add this to the wiki. See also http://buildbot.net/trac/wiki/UsingLaunchd http://developer.apple.com/mac/library/technotes/tn2005/tn2083.html Regards, Martin From exarkun at twistedmatrix.com Sun May 16 01:35:31 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sat, 15 May 2010 23:35:31 -0000 Subject: [Python-Dev] robots exclusion file on the buildbot pages? In-Reply-To: <4BEF04F6.3090308@v.loewis.de> References: <20100515194839.22b71aa6@pitrou.net> <20100515175728.1632.1573034403.divmod.xquotient.3@localhost.localdomain> <20100515200709.64be2103@pitrou.net> <848.1273953839@parc.com> <4BEF04F6.3090308@v.loewis.de> Message-ID: <20100515233531.1632.315497902.divmod.xquotient.5@localhost.localdomain> On 08:32 pm, martin at v.loewis.de wrote: >>I'd find it useful if the "branch" field was a choice pull-down >>listing >>valid branches, rather than a plain text field, and if the "revision" >>field always defaulted to "HEAD". Seems to me that since the form is >>coming from the buildmaster, that should be possible. > >Unfortunately, these forms are deeply hidden in the buildbot code. So >I'd rather avoid editing them, or else upgrading to the next buildbot >version becomes even more tedious. Someone sufficiently interested in this feature could work with buildbot upstream to get the feature added to an upcoming release, though (obviously). Jean-Paul From yaniv at aknin.name Sun May 16 03:55:14 2010 From: yaniv at aknin.name (Yaniv Aknin) Date: Sun, 16 May 2010 11:55:14 +1000 Subject: [Python-Dev] frozendict (was: Possible patch for functools partial...) Message-ID: > > > So I'm thinking either we make an > > immutable/hashable dict while we're at it, or store the keyword > > arguments as a tuple (which guarantees immutability), and only > > convert them back to a dict when you want to call the partial object > > (simpler, slower). > > I'd support an immutable dict. [...] I've set out to implement a frozendict (like frozenset) for use to store functools.keywords' attribute, and quickly realized I didn't think of an obvious flaw in that idea. A frozenset will only accept hashable members, but a frozendict can't afford this luxury for its values. I'm not sure how should I go about handling that, if at all. Should I implement a frozendict which will remain unhashable but support equality testing like a regular dict? A frozendict that is only hashable if all its values are hashable, like a tuple? Is the whole notion of a frozendict worthy, given this limitation? I'd be happy to hear python-dev's guidance on this. Cheers, - Yaniv -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun May 16 12:53:58 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 16 May 2010 12:53:58 +0200 Subject: [Python-Dev] frozendict (was: Possible patch for functools partial...) References: Message-ID: <20100516125358.2f08e90f@pitrou.net> On Sun, 16 May 2010 11:55:14 +1000 Yaniv Aknin wrote: > > Is the whole notion of a frozendict > worthy, given this limitation? I don't think so. Of course we how have a more general problem: - if we choose to implement only equality testing for partials and leave hashing as is, then hashing isn't consistent with equality anymore -- which is unacceptable - if we choose to implement only equality testing for partials and make them unhashable, we are breaking programs which store partials in a set or a dict So we are left with the following choice: - implement hashing consistently with equality testing, even though the keyword arguments can be changed later. I think this is acceptable from a practicality standpoint - abandon the whole idea of improving equality testing Regards Antoine. From ncoghlan at gmail.com Sun May 16 13:10:02 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 16 May 2010 21:10:02 +1000 Subject: [Python-Dev] frozendict (was: Possible patch for functools partial...) In-Reply-To: <20100516125358.2f08e90f@pitrou.net> References: <20100516125358.2f08e90f@pitrou.net> Message-ID: <4BEFD28A.70209@gmail.com> Antoine Pitrou wrote: > On Sun, 16 May 2010 11:55:14 +1000 > Yaniv Aknin wrote: >> Is the whole notion of a frozendict >> worthy, given this limitation? > > I don't think so. Of course we how have a more general problem: > - if we choose to implement only equality testing for partials and > leave hashing as is, then hashing isn't consistent with equality > anymore -- which is unacceptable > - if we choose to implement only equality testing for partials and make > them unhashable, we are breaking programs which store partials in a > set or a dict > > So we are left with the following choice: > - implement hashing consistently with equality testing, even though the > keyword arguments can be changed later. I think this is acceptable > from a practicality standpoint Another option is to compare the keywords by identity rather than value. This is likely to be more correct anyway, since items may compare equal without giving the same result for all functions: >>> dict(a=1) == dict(a=1.0) True >>> partial(type, 1)() >>> partial(type, 1.0)() Once you're comparing the supplied arguments by identity rather than value, the question of hashability goes away. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From janssen at parc.com Sun May 16 19:53:23 2010 From: janssen at parc.com (Bill Janssen) Date: Sun, 16 May 2010 10:53:23 PDT Subject: [Python-Dev] buildbot svn exceptions again... In-Reply-To: <4BEF07EF.8070900@v.loewis.de> References: <898.1273954152@parc.com> <4BEF07EF.8070900@v.loewis.de> Message-ID: <99371.1274032403@parc.com> Martin v. L?wis wrote: > Bill Janssen wrote: > > parc-leopard-1 (and most of the other builders) are failing the svn > > checkout with the following error: > > > > svn: PROPFIND of '/projects/python/trunk': Could not resolve hostname > > `svn.python.org': Temporary failure in name resolution > > (http://svn.python.org) > > > > When I log into that machine as "buildbot" and do the svn checkout > > manually using the following command > > > > /usr/bin/svn checkout --non-interactive --no-auth-cache --revision > > HEAD http://svn.python.org/projects/python/trunk build > > > > it works fine. > > That's a common OSX problem/bug. Processes occasionally lose the ability > to resolve host names. A workaround is to just put them in the /etc/hosts file. Bill From janssen at parc.com Sun May 16 19:54:11 2010 From: janssen at parc.com (Bill Janssen) Date: Sun, 16 May 2010 10:54:11 PDT Subject: [Python-Dev] buildbot svn exceptions again... In-Reply-To: <4BEF07EF.8070900@v.loewis.de> References: <898.1273954152@parc.com> <4BEF07EF.8070900@v.loewis.de> Message-ID: <99382.1274032451@parc.com> Martin v. L?wis wrote: > That's a common OSX problem/bug. Processes occasionally lose the ability > to resolve host names. Various theories float around what's causing this > (most commonly, people expect that a "controlling terminal" must be > present); my theory is this: > > There is a Mach port in the system that lets a process talk to the local > resolver (lookupd, or whatever Apple's naming infrastructure is called > today). For some reason, this port gets closed in a "background" > process, perhaps because the parent process closed it when you logged > out. Or the Kerberos credentials expired, or some such... > > In short, if some OS X expert could shed some light on this, it would be > appreciated. It would then be good to add this to the wiki. See also This is as close to an explanation as I've been able to get from Apple: http://lists.apple.com/archives/Darwin-kernel/2009/Apr/msg00016.html > > http://buildbot.net/trac/wiki/UsingLaunchd > http://developer.apple.com/mac/library/technotes/tn2005/tn2083.html > > Regards, > Martin > > From martin at v.loewis.de Sun May 16 20:27:23 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 16 May 2010 20:27:23 +0200 Subject: [Python-Dev] buildbot svn exceptions again... In-Reply-To: <99371.1274032403@parc.com> References: <898.1273954152@parc.com> <4BEF07EF.8070900@v.loewis.de> <99371.1274032403@parc.com> Message-ID: <4BF0390B.9@v.loewis.de> > A workaround is to just put them in the /etc/hosts file. That doesn't really help: the test suite also relies on a number of host names. Regards, Martin From janssen at parc.com Sun May 16 21:33:20 2010 From: janssen at parc.com (Bill Janssen) Date: Sun, 16 May 2010 12:33:20 PDT Subject: [Python-Dev] OS X buildbots and launchd Message-ID: <566.1274038400@parc.com> I can find no evidence that the buildbot installation process given on the wiki will cause the buildbot slave to be restarted after a reboot of the machine. To accomplish this, you should also undertake the work described in http://buildbot.net/trac/wiki/UsingLaunchd On my Leopard slave, I created the file /Library/LaunchAgents/org.python.buildbot.slave.plist, and put in it this XML: StandardOutPath twistd.log StandardErrorPath twistd-err.log EnvironmentVariables PATH /usr/local/bin:/sbin:/usr/sbin:/bin:/usr/bin GroupName daemon KeepAlive SuccessfulExit Label org.python.buildbot.slave ProgramArguments /usr/bin/twistd -no -y /Users/buildbot/buildarea/buildbot.tac RunAtLoad UserName buildbot WorkingDirectory /Users/buildbot/buildarea Note that I am using the system Python 2.5, and that I have configured the buildbot slave machine to login the "buildbot" account automatically upon boot, which login will cause this daemon to start, and will provide access to the GUI context for testing tk. I've configured it with a locking screensaver that starts immediately; let's see if that works OK. I've also put "dinsdale.python.org" and "svn.python.org" in /etc/hosts. Bill From janssen at parc.com Sun May 16 22:02:07 2010 From: janssen at parc.com (Bill Janssen) Date: Sun, 16 May 2010 13:02:07 PDT Subject: [Python-Dev] what ports are needed for a buildbot? Message-ID: <1132.1274040127@parc.com> I'm running my buildbots in a captive firewalled subnet, and I have to enumerate all the ports that the buildbot is allowed to open to the outside world. Right now, I'm getting this error: ERROR: test_connect (test.test_smtpnet.SmtpSSLTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/test/test_smtpnet.py", line 15, in test_connect server = smtplib.SMTP_SSL(self.testServer, self.remotePort) File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/smtplib.py", line 757, in __init__ SMTP.__init__(self, host, port, local_hostname, timeout) File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/smtplib.py", line 239, in __init__ (code, msg) = self.connect(host, port) File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/smtplib.py", line 295, in connect self.sock = self._get_socket(host, port, self.timeout) File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/smtplib.py", line 762, in _get_socket new_socket = socket.create_connection((host, port), timeout) File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/socket.py", line 321, in create_connection raise error(msg) socket.error: [Errno 61] Connection refused This make me think that the list of ports enumerated at http://wiki.python.org/moin/BuildBot is incomplete. We need a complete list. If you know of additional ports, please add them to the wiki page. Bill From nir at winpdb.org Sun May 16 22:07:06 2010 From: nir at winpdb.org (Nir Aides) Date: Sun, 16 May 2010 23:07:06 +0300 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) Message-ID: Hi all, Here is a second (last?) attempt at getting traction on fixing the GIL (is it broken?) with BFS (WTF?). So don't be shy (don't be too rude either) since ignoring counts as down voting. Relevant Python issue: http://bugs.python.org/issue7946 *Bottom line first* I submitted an implementation of BFS ( http://ck.kolivas.org/patches/bfs/sched-BFS.txt) as a patch to the GIL, which to the extent I have tested it, behaves nicely on Windows XP, Windows 7, GNU/Linux with either CFS or O(1) schedulers, 1/2/4 cores, laptop, desktop and VirtualBox VM guest (some data below). The patch is still work in progress and requires work in terms of style, moving code where it belongs, test code, etc... nevertheless, Python core developers recommended I already (re)post to python-dev for discussion. *So is the GIL broken?* There seems to be some disagreement on that question among Python core developers (unless you all agree it is not broken :) ). Some developers maintain the effects described by David Beazley do not affect real world systems. Even I took the role of a devil's advocate in a previous discussion, but in fact I think that Python, being a general purpose language, is similar to the OS in that regard. It is used across many application domains, platforms, and development paradigms, just as OS schedulers are, and therefore accepting thread scheduling with such properties as a fact of life is not a good idea. I was first bitten by the original GIL last year while testing a system, and found David's research while looking for answers, and later had to work around that problem in another system. Here are other real world cases: 1) Zope people hit this back in 2002 and documented the problem with interesting insight: http://www.zope.org/Members/glpb/solaris/multiproc "I have directly observed a 30% penalty under MP constraints when the sys.setcheckinterval value was too low (and there was too much GIL thrashing)." http://www.zope.org/Members/glpb/solaris/report_ps "A machine that's going full-throttle isn't as bad, curiously enough -- because the other CPU's are busy doing real work, the GIL doesn't have as much opportunity to get shuffled between CPUs. On a MP box it's very important to set sys.setcheckinterval() up to a fairly large number, I recommend pystones / 50 or so." 2) Python mailing list - 2005 http://mail.python.org/pipermail/python-list/2005-August/336286.html "The app suffers from serious performance degradation (compared to pure c/C++) and high context switches that I suspect the GIL unlocking may be aggravating ?" 3) Python mailing list - 2008 http://mail.python.org/pipermail/python-list/2008-June/1143217.html "When I start the server, it sometimes eats up 100% of the CPU for a good minute or so... though none of the threads are CPU-intensive" 4) Twisted http://twistedmatrix.com/pipermail/twisted-python/2005-July/011048.html "When I run a CPU intensive method via threads.deferToThread it takes all the CPU away and renders the twisted process unresponsive." Admittedly, it is not easy to dig reports up in Google. Finally, I think David explained the relevance of this problem quite nicely: http://mail.python.org/pipermail/python-dev/2010-March/098416.html *What about the new GIL?* There is no real world experience with the new GIL since it is under development. What we have is David's analysis and a few benchmarks from the bug report. *Evolving the GIL into a scheduler* The problem addressed by the GIL has always been *scheduling* threads to the interpreter, not just controlling access to it. The patches by Antoine and David essentially evolve the GIL into a scheduler, however both cause thread starvation or high rate of context switching in some scenarios (see data below). *BFS* Enter BFS, a new scheduler designed by Con Kolivas, a Linux kernel hacker who is an expert in this field: http://ck.kolivas.org/patches/bfs/sched-BFS.txt "The goal of the Brain Fuck Scheduler, referred to as BFS from here on, is to completely do away with the complex designs of the past for the cpu process scheduler and instead implement one that is very simple in basic design. The main focus of BFS is to achieve excellent desktop interactivity and responsiveness without heuristics and tuning knobs that are difficult to understand, impossible to model and predict the effect of, and when tuned to one workload cause massive detriment to another." I submitted an implementation of BFS (bfs.patch) which on my machines gives comparable performance to gilinter2.patch (Antoine's) and seems to schedule threads more fairly, predictably, and with lower rate of context switching (see data below). There are however, some issues in bfs.patch: 1) It works on top of the OS scheduler, which means (for all GIL patches!): a) It does not control and is not aware of information such as OS thread preemption, CPU core to run on, etc... b) There may be hard to predict interaction between BFS and the underlying OS scheduler, which needs to be tested on each target platform. 2) It works best when TSC (http://en.wikipedia.org/wiki/Time_Stamp_Counter) is available and otherwise falls back to gettimeofday(). I expect the scheduler to misbehave to some degree or affect performance when TSC is not available and either of the following is true: a) if gettimeofday() is very expensive to read (impacts release/acquire overhead). b) if gettimeofday() has very low precision ~10ms. By design of BFS, once CPU load crosses a given threshold (about 8 CPU bound tasks which need the CPU at once), the scheduler falls back to FIFO behavior and latency goes up sharply. I have no data on how bfs.patch behaves on ARM, AMD, old CPU models, OSX, FreeBSD, Solaris, or mobile. The patch may require some tuning to work properly on those systems, so data is welcome (make sure TSC code in Include/cycle.h works on those systems before benching). All that said, to the extent I have tested it, bfs.patch behaves nicely on Windows XP, Windows 7, GNU/Linux with either CFS or O(1) schedulers, 1/2/4 cores, laptop, desktop and VirtualBox VM guest. *Data* Comparison of proposed patches running ccbench on Windows XP: http://bugs.python.org/issue7946#msg104899 Comparison of proposed patches running Florent's write.py test on Ubuntu Karmic: http://bugs.python.org/issue7946#msg105687 Comparison of old GIL, new GIL and BFS running ccbench on Ubuntu Karmic: http://bugs.python.org/issue7946#msg105874 Last comparison includes a run of old GIL with sys.setcheckinterval(2500) as Zope people do. IO latency shoots up to ~1000ms as result. *What can be done with it?* Here are some options: 1) Abandon it - no one is interested, yawn. 2) Take ideas and workarounds from its code and apply to other patches. 3) Include it in the interpreter as an auxiliary (turn on with a runtime switch) scheduler. 4) Adopt it as the Python scheduler. *Opinion?* Your opinion is needed (however, please submit code review comments which are not likely to interest other people, e.g. "why did you use volatile for X?", at the issue page: http://bugs.python.org/issue7946). Thanks, Nir -------------- next part -------------- An HTML attachment was scrubbed... URL: From janssen at parc.com Sun May 16 22:32:19 2010 From: janssen at parc.com (Bill Janssen) Date: Sun, 16 May 2010 13:32:19 PDT Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: Message-ID: <1743.1274041939@parc.com> I'm interested in having *something*, but I'm not particularly interested in the 3.x branch. I expect it to be at least 5 years before the various Python subsystems I depend on are all ported to 3.x, and I expect it to be at least that long before I can move onto a version of OS X that ships with Python 3 as the default system Python. Right now, I'm faced with the prospect of Apple's next OS including Python 2.7, and being saddled with the current Python 2.x terrible multicore behavior for the next 5 years or so. We badly need some kind of patch for this in the 2.x branch. Bill From janssen at parc.com Sun May 16 22:35:49 2010 From: janssen at parc.com (Bill Janssen) Date: Sun, 16 May 2010 13:35:49 PDT Subject: [Python-Dev] OS X buildbots and launchd In-Reply-To: <566.1274038400@parc.com> References: <566.1274038400@parc.com> Message-ID: <2100.1274042149@parc.com> Bill Janssen wrote: > I can find no evidence that the buildbot installation process given on > the wiki will cause the buildbot slave to be restarted after a reboot of > the machine. To accomplish this, you should also undertake the work > described in > > http://buildbot.net/trac/wiki/UsingLaunchd > > On my Leopard slave, I created the file > /Library/LaunchAgents/org.python.buildbot.slave.plist, and put in it > this XML: > > [...] > > Note that I am using the system Python 2.5, and that I have configured > the buildbot slave machine to login the "buildbot" account automatically > upon boot, which login will cause this daemon to start, and will provide > access to the GUI context for testing tk. I've configured it with a > locking screensaver that starts immediately; let's see if that works OK. This seems to work. test_tk now passes. Bill From ncoghlan at gmail.com Sun May 16 22:45:12 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 17 May 2010 06:45:12 +1000 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <1743.1274041939@parc.com> References: <1743.1274041939@parc.com> Message-ID: <4BF05958.7080206@gmail.com> Bill Janssen wrote: > Right now, I'm faced with the prospect of Apple's next OS including > Python 2.7, and being saddled with the current Python 2.x terrible > multicore behavior for the next 5 years or so. We badly need some kind > of patch for this in the 2.x branch. The matter of the GIL seems far less urgent to those of us that don't see threading as a particularly good way to exploit multiple cores. Either way, with the first 2.7 release candidate out soon, it's already too late to contemplate significant changes to the GIL for that release. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From victor.stinner at haypocalc.com Sun May 16 22:52:59 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 16 May 2010 22:52:59 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: Message-ID: <201005162253.00229.victor.stinner@haypocalc.com> Hi, Le dimanche 16 mai 2010 22:07:06, Nir Aides a ?crit : > *Evolving the GIL into a scheduler* > > The problem addressed by the GIL has always been *scheduling* threads to > the interpreter, not just controlling access to it. The patches by Antoine > and David essentially evolve the GIL into a scheduler, however both cause > thread starvation or high rate of context switching in some scenarios (see > data below). I didn't followed last development around the GIL. Can you explain me why Python should have its own scheduler whereas each OS has already its own scheduler? The OS has useful informations to help scheduling that Python doesn't have. Linux and FreeBSD schedulers are faster each year since... 5 years?, especially with multiple CPU/cores. -- Victor Stinner http://www.haypocalc.com/ From janssen at parc.com Sun May 16 22:57:06 2010 From: janssen at parc.com (Bill Janssen) Date: Sun, 16 May 2010 13:57:06 PDT Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <4BF05958.7080206@gmail.com> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> Message-ID: <3430.1274043426@parc.com> Nick Coghlan wrote: > Bill Janssen wrote: > > Right now, I'm faced with the prospect of Apple's next OS including > > Python 2.7, and being saddled with the current Python 2.x terrible > > multicore behavior for the next 5 years or so. We badly need some kind > > of patch for this in the 2.x branch. > > The matter of the GIL seems far less urgent to those of us that don't > see threading as a particularly good way to exploit multiple cores. Nick, this isn't about exploiting cores. This is about Python programs that used to work fine on single-core machines suddenly becoming horrible resource hogs when moved to a more modern machine with a more modern version of Python. As far as I'm concerned, just tying all of the program's threads to a single core would be fine, though I imagine others would differ. > Either way, with the first 2.7 release candidate out soon, it's already > too late to contemplate significant changes to the GIL for that release. The release schedule, and labelling things as "release candidates" or not, are all under our control. Nothing is "too late". And there's always Python 2.8 :-). But I'd consider this a bug in the threading library, not some unmotivated blue-sky change to the GIL. Bill From ncoghlan at gmail.com Sun May 16 23:12:14 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 17 May 2010 07:12:14 +1000 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <3430.1274043426@parc.com> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> Message-ID: <4BF05FAE.7020809@gmail.com> Bill Janssen wrote: > As far as I'm concerned, just tying all of > the program's threads to a single core would be fine, though I imagine > others would differ. Which can be done through the OS tools for setting an application's CPU affinity. Fixing the Python thread scheduling is only necessary if we want to be able to exploit the extra power of those cores rather than forcing reversion to single core behaviour. Note that I'm not *opposed* to fixing it, and the discussion in the tracker issue over Nir and Dave's different approaches to the problem looks interesting. > The release schedule, and labelling things as "release candidates" or > not, are all under our control. Nothing is "too late". And there's > always Python 2.8 :-) . But I'd consider this a bug in the threading > library, not some unmotivated blue-sky change to the GIL. Yes, but if we never said "too late" we'd never ship anything :) And you do have a reasonable case for considering this a bug, but it wouldn't be the first time we've escalated bug fixes to "new feature" level simply because they had a relatively high impact on core parts of the code. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From janssen at parc.com Sun May 16 23:27:09 2010 From: janssen at parc.com (Bill Janssen) Date: Sun, 16 May 2010 14:27:09 PDT Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <4BF05FAE.7020809@gmail.com> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> Message-ID: <4328.1274045229@parc.com> Nick Coghlan wrote: > Bill Janssen wrote: > > As far as I'm concerned, just tying all of > > the program's threads to a single core would be fine, though I imagine > > others would differ. > > Which can be done through the OS tools for setting an application's CPU > affinity. Yes, as I say, if the initialization of the threading module called those tools appropriately and automatically, I'd be happy. Well, less unhappy :-). I have to admit, I don't know how to do this on OS X. What's the tool and the process, if we're not getting too far afield? Bill From martin at v.loewis.de Sun May 16 23:38:35 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 16 May 2010 23:38:35 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <4328.1274045229@parc.com> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> Message-ID: <4BF065DB.2090703@v.loewis.de> Bill Janssen wrote: > Nick Coghlan wrote: > >> Bill Janssen wrote: >>> As far as I'm concerned, just tying all of >>> the program's threads to a single core would be fine, though I imagine >>> others would differ. >> Which can be done through the OS tools for setting an application's CPU >> affinity. > > Yes, as I say, if the initialization of the threading module called > those tools appropriately and automatically, I'd be happy. Well, less > unhappy :-). > > I have to admit, I don't know how to do this on OS X. What's the > tool and the process, if we're not getting too far afield? OSX doesn't really support thread affinity. The affinity API that they have is described on http://developer.apple.com/mac/library/releasenotes/Performance/RN-AffinityAPI/index.html You can't bind a thread to specific core with it, though, but you can requests that multiple threads all run on the same core (leaving the choice "which core" to the system). IIUC, an affinity preference does not survive exec(2), so you can't write a tool that binds all threads of its child processes to a single core (such a tool is available on many unices, though). Regards, Martin From janssen at parc.com Mon May 17 00:13:44 2010 From: janssen at parc.com (Bill Janssen) Date: Sun, 16 May 2010 15:13:44 PDT Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <4BF065DB.2090703@v.loewis.de> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> Message-ID: <5806.1274048024@parc.com> Martin v. L?wis wrote: > OSX doesn't really support thread affinity. The affinity API that they > have is described on > > http://developer.apple.com/mac/library/releasenotes/Performance/RN-AffinityAPI/index.html > > You can't bind a thread to specific core with it, though, but you can > requests that multiple threads all run on the same core (leaving the > choice "which core" to the system). I believe that would be sufficient to fix the problem with Python, though I wonder about the effect on JCC-generated modules like pylucene, where the threads are really Java threads as well as Python threads. So the patch to the threading code would presumably, for those OSs where the capability exists, try to put all created threads in the same affinity set. Presumably there would also be a way to clear that binding, for folks who know what they're doing. Bill From janssen at parc.com Mon May 17 01:46:23 2010 From: janssen at parc.com (Bill Janssen) Date: Sun, 16 May 2010 16:46:23 PDT Subject: [Python-Dev] OS X buildbots and launchd In-Reply-To: <566.1274038400@parc.com> References: <566.1274038400@parc.com> Message-ID: <8192.1274053583@parc.com> I've repeated this setup with OS X 10.4 (Tiger). Some changes to the XML file were necessary, because I'd installed a non-system Python, twisted, and zope.interface on this machine, and because on Tiger, launchd daemon were not allowed to set their group ID. Here's the Tiger version, which again goes in /Library/LaunchAgents/org.python.buildbot.slave.plist. Bill StandardOutPath twistd.log StandardErrorPath twistd-err.log EnvironmentVariables PATH /Library/Frameworks/Python.framework/Versions/2.6/bin:/sbin:/usr/sbin:/bin:/usr/bin PYTHONPATH /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages KeepAlive SuccessfulExit Label org.python.buildbot.slave ProgramArguments /Library/Frameworks/Python.framework/Versions/2.6/bin/twistd -no -y /Users/buildbot/buildarea/buildbot.tac RunAtLoad UserName buildbot WorkingDirectory /Users/buildbot/buildarea From regebro at gmail.com Mon May 17 08:28:08 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 17 May 2010 08:28:08 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <201005162253.00229.victor.stinner@haypocalc.com> References: <201005162253.00229.victor.stinner@haypocalc.com> Message-ID: On Sun, May 16, 2010 at 22:52, Victor Stinner wrote: > I didn't followed last development around the GIL. Can you explain me why > Python should have its own scheduler whereas each OS has already its own > scheduler? Because the GIL locks and unlocks threads, in practice, it already have. But the scheduler is so simplistic it ends up fighting with the OS scheduler, and a large amount of CPU time is used up switching instead of executing. Having a proper scheduler fixes this. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From hrvoje.niksic at avl.com Mon May 17 09:47:10 2010 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Mon, 17 May 2010 09:47:10 +0200 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: References: <201005132130.30619.steve@pearwood.info> Message-ID: <4BF0F47E.6060604@avl.com> On 05/14/2010 06:39 AM, Daniel Urban wrote: > I've made a new patch, in which the keywords attribute is a read-only > proxy of the dictionary. What about backward compatibility? This looks like an incompatible change. From urban.dani+py at gmail.com Mon May 17 13:51:40 2010 From: urban.dani+py at gmail.com (Daniel Urban) Date: Mon, 17 May 2010 13:51:40 +0200 Subject: [Python-Dev] Possible patch for functools partial - Interested? In-Reply-To: <4BF0F47E.6060604@avl.com> References: <201005132130.30619.steve@pearwood.info> <4BF0F47E.6060604@avl.com> Message-ID: On Mon, May 17, 2010 at 09:47, Hrvoje Niksic wrote: > On 05/14/2010 06:39 AM, Daniel Urban wrote: >> >> I've made a new patch, in which the keywords attribute is a read-only >> proxy of the dictionary. > > What about backward compatibility? ?This looks like an incompatible change. You're probably right. I'd be happy to make patch which doesn't replace the dict with the proxy. But that would mean, that the hash value isn't really immutable. Daniel Urban From solipsis at pitrou.net Mon May 17 14:12:29 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 17 May 2010 14:12:29 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) References: <201005162253.00229.victor.stinner@haypocalc.com> Message-ID: <20100517141229.13a2d66c@pitrou.net> On Mon, 17 May 2010 08:28:08 +0200 Lennart Regebro wrote: > But the scheduler is so simplistic it ends up fighting with the > OS scheduler, and a large amount of CPU time is used up switching > instead of executing. This is already fixed with py3k. Antoine. From solipsis at pitrou.net Mon May 17 14:14:55 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 17 May 2010 14:14:55 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> Message-ID: <20100517141455.6e1b6f64@pitrou.net> On Sun, 16 May 2010 15:13:44 PDT Bill Janssen wrote: > > So the patch to the threading code would presumably, for those OSs where > the capability exists, try to put all created threads in the same > affinity set. This is not really a good idea. There's some code which releases the GIL, precisely so that you can run several threads (computations) at once. If you aren't too concerned with interactivity, you can increase sys.setcheckinterval() to alleviate the problem on 2.x. Regards Antoine. From regebro at gmail.com Mon May 17 14:47:25 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 17 May 2010 14:47:25 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <20100517141229.13a2d66c@pitrou.net> References: <201005162253.00229.victor.stinner@haypocalc.com> <20100517141229.13a2d66c@pitrou.net> Message-ID: On Mon, May 17, 2010 at 14:12, Antoine Pitrou wrote: > On Mon, 17 May 2010 08:28:08 +0200 > Lennart Regebro wrote: >> But the scheduler is so simplistic it ends up fighting with the >> OS scheduler, and a large amount of CPU time is used up switching >> instead of executing. > > This is already fixed with py3k. Are you referring to the "New GIL"? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From solipsis at pitrou.net Mon May 17 15:05:26 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 17 May 2010 15:05:26 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: <201005162253.00229.victor.stinner@haypocalc.com> <20100517141229.13a2d66c@pitrou.net> Message-ID: <20100517150526.0295af9e@pitrou.net> On Mon, 17 May 2010 14:47:25 +0200 Lennart Regebro wrote: > On Mon, May 17, 2010 at 14:12, Antoine Pitrou wrote: > > On Mon, 17 May 2010 08:28:08 +0200 > > Lennart Regebro wrote: > >> But the scheduler is so simplistic it ends up fighting with the > >> OS scheduler, and a large amount of CPU time is used up switching > >> instead of executing. > > > > This is already fixed with py3k. > > Are you referring to the "New GIL"? Yes. From janssen at parc.com Mon May 17 18:05:03 2010 From: janssen at parc.com (Bill Janssen) Date: Mon, 17 May 2010 09:05:03 PDT Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <20100517141455.6e1b6f64@pitrou.net> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> Message-ID: <6887.1274112303@parc.com> Antoine Pitrou wrote: > On Sun, 16 May 2010 15:13:44 PDT > Bill Janssen wrote: > > > > So the patch to the threading code would presumably, for those OSs where > > the capability exists, try to put all created threads in the same > > affinity set. > > This is not really a good idea. Yes, fixing the GIL for multicore in 2.7 would be a better idea, IMO. But that might be too large a change. > There's some code which releases the GIL, precisely so that you can > run several threads (computations) at once. If they can get hold of the GIL in the first place! Yes, you'd want to be able to "unbind" threads if you knew what you were doing, so that they could run on other cores, and you'd want a switch to disable the affinity mechanism entirely. But I'd be happy to have things in the naive case run as well as they do on single-core machines, and let experts do optimizations. > If you aren't too concerned with interactivity, you can increase > sys.setcheckinterval() to alleviate the problem on 2.x. Unfortunately, many use cases might well be concerned with interactivity. Things like event-processing loops. Bill From solipsis at pitrou.net Mon May 17 18:16:52 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 17 May 2010 18:16:52 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <6887.1274112303@parc.com> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> <6887.1274112303@parc.com> Message-ID: <1274113012.3050.8.camel@localhost.localdomain> Le lundi 17 mai 2010 ? 09:05 -0700, Bill Janssen a ?crit : > Antoine Pitrou wrote: > > > On Sun, 16 May 2010 15:13:44 PDT > > Bill Janssen wrote: > > > > > > So the patch to the threading code would presumably, for those OSs where > > > the capability exists, try to put all created threads in the same > > > affinity set. > > > > This is not really a good idea. > > Yes, fixing the GIL for multicore in 2.7 would be a better idea, IMO. > But that might be too large a change. Well, 2.7rc1 is scheduled in less than three weeks now. IMO any patch changing fundamental threading properties is a no-no (even the processor affinity proposal). Someone had tried to backport the new GIL to 2.7 (there's a tracker issue for it, I'm too lazy to search right now), but it was concluded that compatibility requirements for Python 2.x (compatibility with various legacy threading libraries) made things too complicated and tedious. > > There's some code which releases the GIL, precisely so that you can > > run several threads (computations) at once. > > If they can get hold of the GIL in the first place! Yes, you'd want to > be able to "unbind" threads if you knew what you were doing, so that > they could run on other cores, and you'd want a switch to disable the > affinity mechanism entirely. But I'd be happy to have things in the > naive case run as well as they do on single-core machines, and let > experts do optimizations. "Letting experts do optimizations" is a regression, though, because right now you don't have to be an expert to take advantage of such optimizations (just run a separate thread doing e.g. some zlib compression). Regards Antoine. From janssen at parc.com Mon May 17 19:11:03 2010 From: janssen at parc.com (Bill Janssen) Date: Mon, 17 May 2010 10:11:03 PDT Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <1274113012.3050.8.camel@localhost.localdomain> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> <6887.1274112303@parc.com> <1274113012.3050.8.camel@localhost.localdomain> Message-ID: <9017.1274116263@parc.com> Antoine Pitrou wrote: > Well, 2.7rc1 is scheduled in less than three weeks now. IMO any patch > changing fundamental threading properties is a no-no (even the processor > affinity proposal). Unfortunately, our "fundamental threading properties" are broken for multicore machines. And multicore seems to be the wave of the future. Not fixing this is a big problem. It relegates the only Python which will be usable by many many people for many years, 2.x, to a toy language status on modern machines. It will have threads, but the use of them, either directly or indirectly by modules you may import, may cause unpredictable performance problems. I'd hate to let a fundamental flaw like this go through simply because someone somewhere somewhen set a completely synthetic deadline. > Someone had tried to backport the new GIL to 2.7 (there's a tracker > issue for it, I'm too lazy to search right now), but it was concluded > that compatibility requirements for Python 2.x (compatibility with > various legacy threading libraries) made things too complicated and > tedious. http://bugs.python.org/issue7753, from January. I've read through that issue (several times), and I have to say that I wind up gnashing my teeth each time. Here's a fix that's rejected because it "only" supports NT and POSIX threads. What percentage of Python use cases do those two threading systems cover? Do we know? If by "compatibility requirements" you mean that new releases of Python should run on antique systems just as the older releases did, that's satisfied by the issue's current state, you're right. On the other hand, to me that seems an odd goal to prioritize. After all, the older releases are still there, for that antique system. Nor do I see an answer to Russ' final question: ``What if, as you proposed earlier, the patch were to leave the old behavior if the threading model on the given platform were not supported?'' > > > There's some code which releanses the GIL, precisely so that you can > > > run several threads (computations) at once. > > > > If they can get hold of the GIL in the first place! Yes, you'd want to > > be able to "unbind" threads if you knew what you were doing, so that > > they could run on other cores, and you'd want a switch to disable the > > affinity mechanism entirely. But I'd be happy to have things in the > > naive case run as well as they do on single-core machines, and let > > experts do optimizations. > > "Letting experts do optimizations" is a regression, though, because > right now you don't have to be an expert to take advantage of such > optimizations (just run a separate thread doing e.g. some zlib > compression). If threading performance wasn't broken on multicore, I'd agree with you. But right now, *everyone* has to be an expert just to use Python 2.x effectively on modern multicore hardware -- you have to find the right patch in issue 7753, apply it to the sources, build a custom python, and use it. Whether or not use explicitly know you are using threads (because some other package may be using them under the covers). Bill From solipsis at pitrou.net Mon May 17 19:39:52 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 17 May 2010 19:39:52 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <9017.1274116263@parc.com> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> <6887.1274112303@parc.com> <1274113012.3050.8.camel@localhost.localdomain> <9017.1274116263@parc.com> Message-ID: <20100517193952.48875717@pitrou.net> On Mon, 17 May 2010 10:11:03 PDT Bill Janssen wrote: > > I'd hate to let a fundamental flaw like this go through simply because > someone somewhere somewhen set a completely synthetic deadline. [...] > I've read through that issue (several times), and I have to say that I > wind up gnashing my teeth each time. Here's a fix that's rejected > because it "only" supports NT and POSIX threads. What percentage of > Python use cases do those two threading systems cover? Do we know? Well, if instead of gnashing your teeth, you had contributed to the issue, perhaps a patch would have been committed by now (or perhaps not, but who knows?). If you stay silent, you cannot assume that someone else will stand up for *your* opinion (and the fact that nobody did could indicate that not many people care about the issue, actually). > But right now, *everyone* has to be an expert just to use Python 2.x > effectively on modern multicore hardware Python works reasonably well on multicore hardware, especially if you don't run spinning loops and if you're not on Mac OS X. It may lose *at most* 10-20% performance compared to a single-core run but that's hardly the end of the world. And some workloads won't suffer any degradation. Besides, today's multicore CPUs have far better single-threaded performance than yesteryear's single-core CPUs, which makes the performance regression issue more theoretical than practical. In real life, you have very little risk of witnessing a performance regression when switching your Python from a single-core to a multicore machine. Regards Antoine. From janssen at parc.com Mon May 17 19:44:19 2010 From: janssen at parc.com (Bill Janssen) Date: Mon, 17 May 2010 10:44:19 PDT Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <9017.1274116263@parc.com> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> <6887.1274112303@parc.com> <1274113012.3050.8.camel@localhost.localdomain> <9017.1274116263@parc.com> Message-ID: <10221.1274118259@parc.com> Bill Janssen wrote: > use it. Whether or not use explicitly know you are using threads > (because some other package may be using them under the covers). Of course, I meant to say, "Whether or not *youse* explicitly know you are using threads (because some other package may be using them under the covers)." :-). Bill From janssen at parc.com Mon May 17 20:15:49 2010 From: janssen at parc.com (Bill Janssen) Date: Mon, 17 May 2010 11:15:49 PDT Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <20100517193952.48875717@pitrou.net> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> <6887.1274112303@parc.com> <1274113012.3050.8.camel@localhost.localdomain> <9017.1274116263@parc.com> <20100517193952.48875717@pitrou.net> Message-ID: <10810.1274120149@parc.com> Antoine Pitrou wrote: > Well, if instead of gnashing your teeth, you had contributed to the > issue, perhaps a patch would have been committed by now (or perhaps > not, but who knows?). If you stay silent, you cannot assume that > someone else will stand up for *your* opinion (and the fact that nobody > did could indicate that not many people care about the issue, actually). Unfortunately, I somehow did not even *know* about the issue until February, after the issue had been closed. What I did know was that some of our big complicated Python multi-threaded daemons had shown puzzling resource hogging when moved from small Macs to large 8-core machines with hardware RAID and lots of memory. But, simpleton that I am, I'd presumed that threading in Python wasn't broken, and was looking elsewhere for the cause. > Python works reasonably well on multicore hardware, especially if you > don't run spinning loops and if you're not on Mac OS X. I'm not sure what you mean by "spinning loops". But I *am* on Mac OS X, along with an increasing percentage of the world. And I'm dismayed that there's no momentum to fix this problem. Not a good sign. Bill From solipsis at pitrou.net Mon May 17 20:28:47 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 17 May 2010 20:28:47 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <10810.1274120149@parc.com> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> <6887.1274112303@parc.com> <1274113012.3050.8.camel@localhost.localdomain> <9017.1274116263@parc.com> <20100517193952.48875717@pitrou.net> <10810.1274120149@parc.com> Message-ID: <20100517202847.6d9aa8d3@pitrou.net> On Mon, 17 May 2010 11:15:49 PDT Bill Janssen wrote: > > What I did know was that > some of our big complicated Python multi-threaded daemons had shown > puzzling resource hogging when moved from small Macs to large 8-core > machines with hardware RAID and lots of memory. Could you give detailed information about this? Since you're talking about a "big complicated Python multi-threaded daemon", I presume you can't port it to Python 3 very quickly, but it would be nice to know if the problem disappears with 3.2. > I'm not sure what you mean by "spinning loops". It was an allusion to Dave Beazley's first benchmarks, which merely ran a spinning loop over several threads, and showed catastrophic degradation under OS X. > But I *am* on Mac OS X, along with an increasing percentage of the > world. And I'm dismayed that there's no momentum to fix this problem. There /has/ been momentum in fixing it. In py3k. Regards Antoine. From florent.xicluna at gmail.com Mon May 17 20:30:27 2010 From: florent.xicluna at gmail.com (Florent Xicluna) Date: Mon, 17 May 2010 20:30:27 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: Message-ID: 2010/5/16 Nir Aides > > *What can be done with it?* > > Here are some options: > 1) Abandon it - no one is interested, yawn. > 2) Take ideas and workarounds from its code and apply to other patches. > 3) Include it in the interpreter as an auxiliary (turn on with a runtime > switch) scheduler. > 4) Adopt it as the Python scheduler. > > > *Opinion?* > > I would like to have the possibility to "./configure --without-broken-GIL" or "./configure --with-bfs-scheduler" in Python 2.7 like we "./configure --with-computed-gotos" in 3.2. It will let the opportunity for the experienced user (or the distribution packager) to enable the threading optimizations on its platform, while it preserves the current behaviour by default. It will give more chance that people test this experimental configure option and report any issue they find, so it can be fixed and improved in the next bugfix version (2.7.1). Since the 2.7 branch will have long term support, it makes sense to support multi-core platforms. And more users of the fixed GIL means more bugfixes (even for 3.x branch). -- Florent -------------- next part -------------- An HTML attachment was scrubbed... URL: From janssen at parc.com Mon May 17 20:59:16 2010 From: janssen at parc.com (Bill Janssen) Date: Mon, 17 May 2010 11:59:16 PDT Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <20100517202847.6d9aa8d3@pitrou.net> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> <6887.1274112303@parc.com> <1274113012.3050.8.camel@localhost.localdomain> <9017.1274116263@parc.com> <20100517193952.48875717@pitrou.net> <10810.1274120149@parc.com> <20100517202847.6d9aa8d3@pitrou.net> Message-ID: <12368.1274122756@parc.com> Antoine Pitrou wrote: > On Mon, 17 May 2010 11:15:49 PDT > Bill Janssen wrote: > > > > What I did know was that > > some of our big complicated Python multi-threaded daemons had shown > > puzzling resource hogging when moved from small Macs to large 8-core > > machines with hardware RAID and lots of memory. > > Could you give detailed information about this? Probably not detailed enough. IP issues. It's a version of UpLib. > Since you're talking about a "big complicated Python multi-threaded > daemon", I presume you can't port it to Python 3 very quickly, but it > would be nice to know if the problem disappears with 3.2. Yes, it would. As soon as I have working 3.x versions of BeautifulSoup, PIL, ReportLab, JCC, pylucene, pyglet, nltk, email, epydoc, feedparser, dictclient, docutils, hachoir, mutagen, medusa, python-dateutil, and vobject, I'll let you know. :-) > There /has/ been momentum in fixing it. In py3k. Yes, I specifically meant in the 2.x branch. I'm guessing I'll have to stay on 2.x for at least 5 more years, due to the other package dependencies. Bill From tjreedy at udel.edu Mon May 17 21:39:36 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 17 May 2010 15:39:36 -0400 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <12368.1274122756@parc.com> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> <6887.1274112303@parc.com> <1274113012.3050.8.camel@localhost.localdomain> <9017.1274116263@parc.com> <20100517193952.48875717@pitrou.net> <10810.1274120149@parc.com> <20100517202847.6d9aa8d3@pitrou.net> <12368.1274122756@parc.com> Message-ID: On 5/17/2010 2:59 PM, Bill Janssen wrote: > Yes, it would. As soon as I have working 3.x versions of BeautifulSoup, > PIL, ReportLab, JCC, pylucene, pyglet, nltk, email, epydoc, feedparser, > dictclient, docutils, hachoir, mutagen, medusa, python-dateutil, and > vobject, I'll let you know. :-) > >> There /has/ been momentum in fixing it. In py3k. > > Yes, I specifically meant in the 2.x branch. I'm guessing I'll have to > stay on 2.x for at least 5 more years, due to the other package > dependencies. I suspect it will be sooner than that, especially if users like you ask/beg/plead with the maintainers of libraries like those you listed to make them work with 3.2. Give your particular reason, that Python3 will work increasingly well with multicore machines. I an sure a couple of such people have posted that they see no reason to upgrade until users start requesting them to. Terry Jan Reedy From martin at v.loewis.de Mon May 17 22:20:14 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 17 May 2010 22:20:14 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <9017.1274116263@parc.com> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> <6887.1274112303@parc.com> <1274113012.3050.8.camel@localhost.localdomain> <9017.1274116263@parc.com> Message-ID: <4BF1A4FE.3030505@v.loewis.de> > Not fixing this is a big problem. It relegates the only Python which > will be usable by many many people for many years, 2.x, to a toy > language status on modern machines. It will have threads, but the use > of them, either directly or indirectly by modules you may import, may > cause unpredictable performance problems. People may disagree with this characterization, but if we take that for granted, then, yes, we are willing to accept that as the state of things for the coming years. People running into these problems will have a number of choices to them: switch operating systems (i.e. drop OSX for something that actually works), switch programming languages (i.e. drop Python for something that actually works), switch application architectures (i.e. drop threads for something that actually works), switch to 3.x, or just accept the problem, and hope that the system will find something else to do while switching Python threads. > I'd hate to let a fundamental flaw like this go through simply because > someone somewhere somewhen set a completely synthetic deadline. No, it's not like that. We set the deadline so that we are able to cancel discussions like this one. It would be possible to change the schedule, if we would agree that it was for a good cause - which we don't. > If threading performance wasn't broken on multicore, I'd agree with you. > But right now, *everyone* has to be an expert just to use Python 2.x > effectively on modern multicore hardware Not at all. Just use the multiprocessing module instead, and be done. It's really easy to use if you already managed to understand threads. Regards, Martin From janssen at parc.com Mon May 17 23:09:07 2010 From: janssen at parc.com (Bill Janssen) Date: Mon, 17 May 2010 14:09:07 PDT Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <4BF1A4FE.3030505@v.loewis.de> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> <6887.1274112303@parc.com> <1274113012.3050.8.camel@localhost.localdomain> <9017.1274116263@parc.com> <4BF1A4FE.3030505@v.loewis.de> Message-ID: <14331.1274130547@parc.com> Martin v. L?wis wrote: > > I'd hate to let a fundamental flaw like this go through simply because > > someone somewhere somewhen set a completely synthetic deadline. > > No, it's not like that. We set the deadline so that we are able to > cancel discussions like this one. It would be possible to change the > schedule, if we would agree that it was for a good cause - which we don't. I do appreciate that, and also what you and Antoine are saying. > > If threading performance wasn't broken on multicore, I'd agree with you. > > But right now, *everyone* has to be an expert just to use Python 2.x > > effectively on modern multicore hardware > > Not at all. Just use the multiprocessing module instead, and be done. > It's really easy to use if you already managed to understand threads. But that's just the problem. Most folks don't use "threads", they use a higher-level abstraction like the nltk library. Does it use threads? Has its owner ported it to py3k? Has its owner ported it to the multiprocessing module? I have to be an expert to know. I'll stop talking about this now... At least, here. Apparently we only need to fix this for OS X. Bill From ncoghlan at gmail.com Mon May 17 23:27:59 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 18 May 2010 07:27:59 +1000 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <4BF1A4FE.3030505@v.loewis.de> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> <6887.1274112303@parc.com> <1274113012.3050.8.camel@localhost.localdomain> <9017.1274116263@parc.com> <4BF1A4FE.3030505@v.loewis.de> Message-ID: <4BF1B4DF.9010908@gmail.com> Martin v. L?wis wrote: > People running into these problems will have a number of choices to > them: switch operating systems (i.e. drop OSX for something that > actually works), switch programming languages (i.e. drop Python for > something that actually works), switch application architectures (i.e. > drop threads for something that actually works), switch to 3.x, or > just accept the problem, and hope that the system will find something > else to do while switching Python threads. There's even another option: if the new-GIL backport patch in http://bugs.python.org/issue7753 works for you, apply it and run with it (and advocate for a Python 2.8 release to make it more widely available, possibly even contributing the fallback behaviour discussed in the issue to make that situation more likely). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From janssen at parc.com Mon May 17 23:28:22 2010 From: janssen at parc.com (Bill Janssen) Date: Mon, 17 May 2010 14:28:22 PDT Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> <6887.1274112303@parc.com> <1274113012.3050.8.camel@localhost.localdomain> <9017.1274116263@parc.com> <20100517193952.48875717@pitrou.net> <10810.1274120149@parc.com> <20100517202847.6d9aa8d3@pitrou.net> <12368.1274122756@parc.com> Message-ID: <14980.1274131702@parc.com> Terry Reedy wrote: > On 5/17/2010 2:59 PM, Bill Janssen wrote: > > Yes, it would. As soon as I have working 3.x versions of BeautifulSoup, > > PIL, ReportLab, JCC, pylucene, pyglet, nltk, email, epydoc, feedparser, > > dictclient, docutils, hachoir, mutagen, medusa, python-dateutil, and > > vobject, I'll let you know. :-) > > > >> There /has/ been momentum in fixing it. In py3k. > > > > Yes, I specifically meant in the 2.x branch. I'm guessing I'll have to > > stay on 2.x for at least 5 more years, due to the other package > > dependencies. > > I suspect it will be sooner than that, especially if users like you > ask/beg/plead with the maintainers of libraries like those you listed > to make them work with 3.2. Oh, that's the way I like to spend my day (and, as you can tell from this conversation, I'm really good at it :-). Though I will of course do that. But some of these, like JCC+pylucene, nltk, and vobject, were developed with idiosyncratic funding resources which no longer exist. Others, like pyglet, were developed for a completely different purpose, and I doubt the developers care what I want. So, realistically, I doubt it will be less than five years. Bill From ncoghlan at gmail.com Mon May 17 23:31:22 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 18 May 2010 07:31:22 +1000 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <20100517141455.6e1b6f64@pitrou.net> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> Message-ID: <4BF1B5AA.8060208@gmail.com> Antoine Pitrou wrote: > On Sun, 16 May 2010 15:13:44 PDT > Bill Janssen wrote: >> So the patch to the threading code would presumably, for those OSs where >> the capability exists, try to put all created threads in the same >> affinity set. > > This is not really a good idea. There's some code which releases the > GIL, precisely so that you can run several threads (computations) at > once. Somewhat irrelevant given the rest of this thread, but you could potentially deal with that by playing CPU affinity games in the BEGIN/END_ALLOW_THREADS macros. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Mon May 17 23:33:37 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 18 May 2010 07:33:37 +1000 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: Message-ID: <4BF1B631.1040503@gmail.com> Florent Xicluna wrote: > I would like to have the possibility to "./configure > --without-broken-GIL" or "./configure --with-bfs-scheduler" in Python > 2.7 like we "./configure --with-computed-gotos" in 3.2. > > It will let the opportunity for the experienced user (or the > distribution packager) to enable the threading optimizations on its > platform, while it preserves the current behaviour by default. > > It will give more chance that people test this experimental configure > option and report any issue they find, so it can be fixed and improved > in the next bugfix version (2.7.1). > Since the 2.7 branch will have long term support, it makes sense to > support multi-core platforms. > > And more users of the fixed GIL means more bugfixes (even for 3.x branch). Would you suggest extending this as far as providing "new GIL" binaries for Windows and Mac OS X? (If that was the case, it begins to sound a lot like http://bugs.python.org/issue7753 with reversion to the old GIL on non-NT/POSIX platforms) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From janssen at parc.com Mon May 17 23:36:44 2010 From: janssen at parc.com (Bill Janssen) Date: Mon, 17 May 2010 14:36:44 PDT Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <20100517141455.6e1b6f64@pitrou.net> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> Message-ID: <15678.1274132204@parc.com> Antoine Pitrou wrote: > On Sun, 16 May 2010 15:13:44 PDT > Bill Janssen wrote: > > > > So the patch to the threading code would presumably, for those OSs where > > the capability exists, try to put all created threads in the same > > affinity set. > > This is not really a good idea. There's some code which releases the > GIL, precisely so that you can run several threads (computations) at > once. Could the macro that releases the GIL also release the thread affinity? And the macro that acquires it again set the affinity tag back again? Bill From solipsis at pitrou.net Mon May 17 23:44:59 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 17 May 2010 23:44:59 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <15678.1274132204@parc.com> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> <15678.1274132204@parc.com> Message-ID: <1274132699.3054.3.camel@localhost.localdomain> Le lundi 17 mai 2010 ? 14:36 -0700, Bill Janssen a ?crit : > > Could the macro that releases the GIL also release the thread affinity? We release the GIL in a variety of situations which don't necessarily involve heavy computations (such as: waiting for IO or sleeping). Furthermore, having several threads taking / dropping the GIL would make the affinity setting / unsetting pattern quite chaotic. Really, I think the processor affinity solution can only be application-specific. There doesn't seem to be an easy, generic way of guessing whether some kind of processor affinity should be requested or not. Regards Antoine. From martin at v.loewis.de Mon May 17 23:48:55 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 17 May 2010 23:48:55 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <14980.1274131702@parc.com> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> <6887.1274112303@parc.com> <1274113012.3050.8.camel@localhost.localdomain> <9017.1274116263@parc.com> <20100517193952.48875717@pitrou.net> <10810.1274120149@parc.com> <20100517202847.6d9aa8d3@pitrou.net> <12368.1274122756@parc.com> <14980.1274131702@parc.com> Message-ID: <4BF1B9C7.9030200@v.loewis.de> > But some of these, like JCC+pylucene, nltk, and vobject, were > developed with idiosyncratic funding resources which no longer exist. > Others, like pyglet, were developed for a completely different > purpose, and I doubt the developers care what I want. So, > realistically, I doubt it will be less than five years. Make it a GSoC project next summer, and you have them ported next fall. Regards, Martin From nir at winpdb.org Tue May 18 01:01:32 2010 From: nir at winpdb.org (Nir Aides) Date: Tue, 18 May 2010 02:01:32 +0300 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: Message-ID: I would like to restart this thread with 2 notes from the lively discussion: a) Issue 7946 (and this discussion?) concerns Python 3.2 b) The GIL problems are not specific to OSX. The old and new GIL misbehave on GNU/Linux and Windows too. [Putting on anti-frying-pan helmet] Nir -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Tue May 18 08:45:41 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 18 May 2010 08:45:41 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <20100517150526.0295af9e@pitrou.net> References: <201005162253.00229.victor.stinner@haypocalc.com> <20100517141229.13a2d66c@pitrou.net> <20100517150526.0295af9e@pitrou.net> Message-ID: On Mon, May 17, 2010 at 15:05, Antoine Pitrou wrote: > On Mon, 17 May 2010 14:47:25 +0200 > Lennart Regebro wrote: >> On Mon, May 17, 2010 at 14:12, Antoine Pitrou wrote: >> > On Mon, 17 May 2010 08:28:08 +0200 >> > Lennart Regebro wrote: >> >> But the scheduler is so simplistic it ends up fighting with the >> >> OS scheduler, and a large amount of CPU time is used up switching >> >> instead of executing. >> > >> > This is already fixed with py3k. >> >> Are you referring to the "New GIL"? > > Yes. At has been shown, it also in certain cases will race with the OS scheduler, so this is not already fixed, although apparently improved, if I understand correctly. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From solipsis at pitrou.net Tue May 18 12:53:12 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 18 May 2010 12:53:12 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: <201005162253.00229.victor.stinner@haypocalc.com> <20100517141229.13a2d66c@pitrou.net> <20100517150526.0295af9e@pitrou.net> Message-ID: <20100518125312.404b0206@pitrou.net> On Tue, 18 May 2010 08:45:41 +0200 Lennart Regebro wrote: > >> > >> Are you referring to the "New GIL"? > > > > Yes. > > At has been shown, it also in certain cases will race with the OS > scheduler, so this is not already fixed, although apparently improved, > if I understand correctly. "Race" is a strange term here and I'm not sure what you mean. The issue found out by Dave Beazley can't be reasonably described by this word, I think. Please read and understand the issue report mentioned by Nir before trying to make statements based on rumours heard here and there. Antoine. From ncoghlan at gmail.com Tue May 18 13:22:55 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 18 May 2010 21:22:55 +1000 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: Message-ID: <4BF2788F.901@gmail.com> Nir Aides wrote: > I would like to restart this thread with 2 notes from the lively discussion: > > a) Issue 7946 (and this discussion?) concerns Python 3.2 > b) The GIL problems are not specific to OSX. The old and new GIL > misbehave on GNU/Linux and Windows too. I think Antoine and Bill went off an a bit of a tangent that *is* specific to Mac OS X and the old GIL (where a Python application not only fails to take advantage of additional cores, but actually runs slower than it does on a less powerful single core machine). The convoying problem identified in issue 7946 does indeed apply to the new GIL on multiple platforms. Without reviewing either proposed patch in detail, I personally am slightly inclined to favour David's suggested solution, both because I better understand the explanation of how it works (and simplicity is a virtue from a maintainability point of view), and because the BFS approach appears to run into trouble when it comes to identifying a suitable cross platform time reference. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From stefan_ml at behnel.de Tue May 18 13:38:05 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 18 May 2010 13:38:05 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <14331.1274130547@parc.com> References: <1743.1274041939@parc.com> <4BF05958.7080206@gmail.com> <3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de> <5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net> <6887.1274112303@parc.com> <1274113012.3050.8.camel@localhost.localdomain> <9017.1274116263@parc.com> <4BF1A4FE.3030505@v.loewis.de> <14331.1274130547@parc.com> Message-ID: Bill Janssen, 17.05.2010 23:09: > Most folks don't use "threads" Seems like a somewhat reasonable assumption to me. > they use a higher-level abstraction like the nltk library. I have my doubts that this applies to "most folks" - likely not even to most of those who use threads. Stefan From regebro at gmail.com Tue May 18 14:16:37 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 18 May 2010 14:16:37 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <20100518125312.404b0206@pitrou.net> References: <201005162253.00229.victor.stinner@haypocalc.com> <20100517141229.13a2d66c@pitrou.net> <20100517150526.0295af9e@pitrou.net> <20100518125312.404b0206@pitrou.net> Message-ID: On Tue, May 18, 2010 at 12:53, Antoine Pitrou wrote: > "Race" is a strange term here and I'm not sure what you mean. > The issue found out by Dave Beazley can't be reasonably described by > this word, I think. OK, maybe "race" is the wrong word. But that doesn't mean the issue doesn't exist. > Please read and understand the issue report mentioned by Nir before > trying to make statements based on rumours heard here and there. Oh, so Dave Beazleys reports is a rumour now. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From solipsis at pitrou.net Tue May 18 14:52:32 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 18 May 2010 14:52:32 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: <201005162253.00229.victor.stinner@haypocalc.com> <20100517141229.13a2d66c@pitrou.net> <20100517150526.0295af9e@pitrou.net> <20100518125312.404b0206@pitrou.net> Message-ID: <1274187152.3083.13.camel@localhost.localdomain> Le mardi 18 mai 2010 ? 14:16 +0200, Lennart Regebro a ?crit : > > Please read and understand the issue report mentioned by Nir before > > trying to make statements based on rumours heard here and there. > > Oh, so Dave Beazleys reports is a rumour now. Your and other people's grandiloquent interpretation of them is. Now let me ask you a question: did you witness some of the effects mentioned here? Did it disturb the proper functioning one of your applications, programs or services? If yes, please be so kind as to explain how; it will be an useful datapoint. Bonus points if the issue affects Python 3.2, since that's really what Nir is talking about. If not, then do you have any valuable information to contribute to this discussion? From regebro at gmail.com Tue May 18 18:10:24 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 18 May 2010 18:10:24 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <1274187152.3083.13.camel@localhost.localdomain> References: <201005162253.00229.victor.stinner@haypocalc.com> <20100517141229.13a2d66c@pitrou.net> <20100517150526.0295af9e@pitrou.net> <20100518125312.404b0206@pitrou.net> <1274187152.3083.13.camel@localhost.localdomain> Message-ID: On Tue, May 18, 2010 at 14:52, Antoine Pitrou wrote: > > Le mardi 18 mai 2010 ? 14:16 +0200, Lennart Regebro a ?crit : >> > Please read and understand the issue report mentioned by Nir before >> > trying to make statements based on rumours heard here and there. >> >> Oh, so Dave Beazleys reports is a rumour now. > > Your and other people's grandiloquent interpretation of them is. > > Now let me ask you a question: did you witness some of the effects > mentioned here? Did it disturb the proper functioning one of your > applications, programs or services? > If yes, please be so kind as to explain how; it will be an useful > datapoint. Bonus points if the issue affects Python 3.2, since that's > really what Nir is talking about. > > If not, then do you have any valuable information to contribute to this > discussion? I doubt anything I say can be less constructive than your rude comments. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From regebro at gmail.com Tue May 18 18:10:24 2010 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 18 May 2010 18:10:24 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <1274187152.3083.13.camel@localhost.localdomain> References: <201005162253.00229.victor.stinner@haypocalc.com> <20100517141229.13a2d66c@pitrou.net> <20100517150526.0295af9e@pitrou.net> <20100518125312.404b0206@pitrou.net> <1274187152.3083.13.camel@localhost.localdomain> Message-ID: On Tue, May 18, 2010 at 14:52, Antoine Pitrou wrote: > > Le mardi 18 mai 2010 ? 14:16 +0200, Lennart Regebro a ?crit : >> > Please read and understand the issue report mentioned by Nir before >> > trying to make statements based on rumours heard here and there. >> >> Oh, so Dave Beazleys reports is a rumour now. > > Your and other people's grandiloquent interpretation of them is. > > Now let me ask you a question: did you witness some of the effects > mentioned here? Did it disturb the proper functioning one of your > applications, programs or services? > If yes, please be so kind as to explain how; it will be an useful > datapoint. Bonus points if the issue affects Python 3.2, since that's > really what Nir is talking about. > > If not, then do you have any valuable information to contribute to this > discussion? I doubt anything I say can be less constructive than your rude comments. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From martin at v.loewis.de Tue May 18 21:43:30 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 18 May 2010 21:43:30 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: <201005162253.00229.victor.stinner@haypocalc.com> <20100517141229.13a2d66c@pitrou.net> <20100517150526.0295af9e@pitrou.net> <20100518125312.404b0206@pitrou.net> <1274187152.3083.13.camel@localhost.localdomain> Message-ID: <4BF2EDE2.8090906@v.loewis.de> Lennart Regebro wrote: > On Tue, May 18, 2010 at 14:52, Antoine Pitrou wrote: >> Le mardi 18 mai 2010 ? 14:16 +0200, Lennart Regebro a ?crit : >>>> Please read and understand the issue report mentioned by Nir before >>>> trying to make statements based on rumours heard here and there. >>> Oh, so Dave Beazleys reports is a rumour now. >> Your and other people's grandiloquent interpretation of them is. >> >> Now let me ask you a question: did you witness some of the effects >> mentioned here? Did it disturb the proper functioning one of your >> applications, programs or services? >> If yes, please be so kind as to explain how; it will be an useful >> datapoint. Bonus points if the issue affects Python 3.2, since that's >> really what Nir is talking about. >> >> If not, then do you have any valuable information to contribute to this >> discussion? > > I doubt anything I say can be less constructive than your rude comments. I can understand why Antoine is being offended: it's his implementation that you attacked. You literally said "At has been shown, it also in certain cases will race with the OS scheduler, so this is not already fixed", claiming that it is not fixed I believe Antoine does consider it fixed, on the grounds that all counter-examples provided so far are made-up toy examples, rather than actual applications that still suffer from the original problems. So please join us in considering the issue fixed unless you can provide a really world example that demonstrates the contrary. Regards, Martin From digitalxero at gmail.com Tue May 18 22:26:19 2010 From: digitalxero at gmail.com (Dj Gilcrease) Date: Tue, 18 May 2010 16:26:19 -0400 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <4BF2EDE2.8090906@v.loewis.de> References: <20100517141229.13a2d66c@pitrou.net> <20100517150526.0295af9e@pitrou.net> <20100518125312.404b0206@pitrou.net> <1274187152.3083.13.camel@localhost.localdomain> <4BF2EDE2.8090906@v.loewis.de> Message-ID: On Tue, May 18, 2010 at 3:43 PM, "Martin v. L?wis" wrote: > So please join us in considering the issue fixed unless you can provide > a really world example that demonstrates the contrary. The server software I maintain (openrpg) experiences this issue with when I tried porting the server code to 3.2. Granted it was only a 5 to 7% speed drop over single core, though with the old GIL (py2.5) there was a 25% to 30% speed drop (It can be running upto 300 IO bound threads & 30 CPU bound threads) so a net improvement of 20% so I am more than happy with the new GIL. I think the new GIL should be given a year or so in the wild before you start trying to optimize theoretical issues you may run into. If in a year people come back and have some examples of where a proper scheduler would help improve speed on multi-core systems even more, then we can address the issue at that time. From mike.klaas at gmail.com Tue May 18 23:39:43 2010 From: mike.klaas at gmail.com (Mike Klaas) Date: Tue, 18 May 2010 14:39:43 -0700 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: Message-ID: On Sun, May 16, 2010 at 1:07 PM, Nir Aides wrote: > Relevant Python issue:?http://bugs.python.org/issue7946 Is there any chance Antoine's "gilinter" patch from that issue might be applied to python 2.7? I have been experiencing rare long delays in simple io operations in multithreaded python applications, and I suspect that they might be related to this issue. -Mike From solipsis at pitrou.net Tue May 18 23:50:26 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 18 May 2010 23:50:26 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) References: Message-ID: <20100518235026.03cad25d@pitrou.net> On Tue, 18 May 2010 14:39:43 -0700 Mike Klaas wrote: > On Sun, May 16, 2010 at 1:07 PM, Nir Aides wrote: > > > Relevant Python issue:?http://bugs.python.org/issue7946 > > Is there any chance Antoine's "gilinter" patch from that issue might > be applied to python 2.7? I have been experiencing rare long delays > in simple io operations in multithreaded python applications, and I > suspect that they might be related to this issue. There's no chance for this since the patch relies on the new GIL. (that's unless there's a rush to backport the new GIL in 2.7, of course) I think your "rare long delays" might be related to the old GIL's own problems, though. How long are they? Regards Antoine. From barry at python.org Wed May 19 00:07:16 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 18 May 2010 18:07:16 -0400 Subject: [Python-Dev] Python versions for Ubuntu 10.10 (Maverick Meerkat) Message-ID: <20100518180716.0bba61df@heresy> I just wanted to let the python-dev community know about some tracks we had at the recently concluded Ubuntu Developer Summit in Brussels. Among the several Python-related discussions, we talked about what versions of Python will be supported and default in the next version of Ubuntu (10.10, code name Maverick Meerkat, to be released in October). If you're interested in following and participating in this discussion, I've started a wiki page as the central place to collect information: https://wiki.ubuntu.com/MaverickMeerkat/TechnicalOverview/Python While we don't have consensus yet, and we have a lot of footwork to do before we can make a decision, my goal is to include Python 2.6, 2.7, and 3.2 (beta) in Ubuntu 10.10. I'd like to make Python 2.7 the default if possible, and we will need to get pre-approval to do stable release upgrades to Python 3.2 to track the move to its final release, which will happen after Ubuntu 10.10. is released. It seems that most discussions are happening on the debian-python mailing list right now. they-don't-call-it-maverick-for-nuthin'-ly y'rs, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From me at gustavonarea.net Wed May 19 00:00:20 2010 From: me at gustavonarea.net (Gustavo Narea) Date: Tue, 18 May 2010 23:00:20 +0100 Subject: [Python-Dev] Incorrect length of collections.Counter objects / Multiplicity function Message-ID: <201005182300.21079.me@gustavonarea.net> Hello, everyone. I've checked the new collections.Counter class and I think I've found a bug: > >>> from collections import Counter > >>> c1 = Counter([1, 2, 1, 3, 2]) > >>> c2 = Counter([1, 1, 2, 2, 3]) > >>> c3 = Counter([1, 1, 2, 3]) > >>> c1 == c2 and c3 not in (c1, c2) > True > >>> # Perfect, so far. But... There's always a "but": > ... > >>> len(c1) > 3 The length of a Counter is the amount of unique elements. But the length must be the cardinality, and the cardinality of a multiset is the total number of elements (including duplicates) [1] [2]. The source code mentions that the recipe on ActiveState [3] was one of the references, but that recipe has this right. Also, why is it indexed? The indexes of a multiset call to mind the position of its elements, but there's no such thing in sets. I think this is inconsistent with the built-in set. I would have implemented the multiplicity function as a method instead of the indexes: c1.get_multiplicity(element) # instead of c1[element] Is this the intended behavior? If so, I'd like to propose a proper multiset implementation for the standard library (preferably called "Multiset"; should I create a PEP?). If not, I can write a patch to fix it, although I'm afraid it'd be a backwards incompatible change. Cheers, [1] http://en.wikipedia.org/wiki/Multiset#Overview [2] http://preview.tinyurl.com/smalltalk-bag [3] http://code.activestate.com/recipes/259174/ -- Gustavo Narea . | Tech blog: =Gustavo/(+blog)/tech ~ About me: =Gustavo/about | From me at gustavonarea.net Wed May 19 00:13:42 2010 From: me at gustavonarea.net (Gustavo Narea) Date: Tue, 18 May 2010 23:13:42 +0100 Subject: [Python-Dev] Unordered tuples/lists Message-ID: <201005182313.42935.me@gustavonarea.net> Hello, everybody. I've been searching for a data structure like a tuple/list *but* unordered -- like a set, but duplicated elements shouldn't be removed. I have not even found a recipe, so I'd like to write an implementation and contribute it to the "collections" module in the standard library. This is the situation I have: I have a data structure that represents an arithmetic/boolean operation. Operations can be commutative, which means that the order of their operands don't change the result of the operation. This is, the following operations are equivalent: operation(a, b, c) == operation(c, b, a) == operation(b, a, c) operation(a, b, a) == operation(a, a, b) == operation(b, a, a) operation(a, a) == operation(a, a) So, I need a type to store the arguments/operands so that if two of these collections have the same elements with the same multiplicity, they are equivalent, regardless of the order. A multiset is not exactly what I need: I still need to use the elements in the order they were given. For example, the logical conjuction (aka "and" operator) has a left and right operands; I need to evaluate the first/left one and, if it returned True, then call the second/right one. They must not be evaluated in a random order. To sum up, it would behave like a tuple or a list, except when it's compared with another object: They would be equivalent if they're both unordered tuples/lists, and have the same elements. There can be mutable and immutable editions (UnorderedList and UnorderedTuple, respectively). I will write a PEP to elaborate on this if you think it'd be nice to have. Or, should I have written the PEP first? Cheers, -- Gustavo Narea . | Tech blog: =Gustavo/(+blog)/tech ~ About me: =Gustavo/about | From solipsis at pitrou.net Wed May 19 01:10:24 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 May 2010 01:10:24 +0200 Subject: [Python-Dev] Summing up In-Reply-To: <4BF2EDE2.8090906@v.loewis.de> References: <201005162253.00229.victor.stinner@haypocalc.com> <20100517141229.13a2d66c@pitrou.net> <20100517150526.0295af9e@pitrou.net> <20100518125312.404b0206@pitrou.net> <1274187152.3083.13.camel@localhost.localdomain> <4BF2EDE2.8090906@v.loewis.de> Message-ID: <20100519011024.52deb1da@pitrou.net> On Tue, 18 May 2010 21:43:30 +0200 "Martin v. L?wis" wrote: > > I can understand why Antoine is being offended: it's his implementation > that you attacked. You literally said "At has been shown, it also in > certain cases will race with the OS scheduler, so this is not already > fixed", claiming that it is not fixed > > I believe Antoine does consider it fixed, on the grounds that all > counter-examples provided so far are made-up toy examples, rather than > actual applications that still suffer from the original problems. Ok, this is a good opportunity to try to sum up, from my point of view. The main problem of the old GIL, which was evidenced in Dave's original study (not this year's, but the previous one) *is* fixed unless someone demonstrates otherwise. It should be noted that witnessing a slight performance degradation on a multi-core machine is not enough to demonstrate such a thing. The degradation could be caused by other factors, such as thread migration, bad OS behaviour, or even locking peculiarities in your own application, which are not related to the GIL. A good test is whether performance improves if you play with sys.setswitchinterval(). Dave's newer study regards another issue, which I must stress is also present in the old GIL algorithm, and therefore must have affected, if it is serious, real-world applications in 2.x. And indeed, the test I recently added to ccbench evidences the huge drop in socket I/Os per second when there's a background CPU thread; this test exercises the same situation as Dave's demos, only with a less trivial CPU workload: == CPython 2.7b2+.0 (trunk:81274M) == == x86_64 Linux on 'x86_64' == --- I/O bandwidth --- Background CPU task: Pi calculation (Python) CPU threads=0: 23034.5 packets/s. CPU threads=1: 6.4 ( 0 %) CPU threads=2: 15.7 ( 0 %) CPU threads=3: 13.9 ( 0 %) CPU threads=4: 20.8 ( 0 %) (note: I've just changed my desktop machine, so these figures are different from what I've posted weeks or months ago) Regardless of the fact that apparently noone reported it in real-world conditions, we *could* decide that the issue needs fixing. If we decide so, Nir's approach is the most rigorous one: it tries to fix the problem thoroughly, rather than graft an additional heuristic. Nir also has tested his patch on a variety of machines, more so than Dave and I did with our own patches; he is obviously willing to go forward. Right now, there are two problems with Nir's proposal: - first, what Nick said: the difficulty of having reliable high-precision cross-platform time sources, which are necessary for the BFS algorithm. Ironically, timestamp counters have their own problems on multi-core machines (they can go out of sync between CPUs). gettimeofday() and clock_gettime() may be precise enough on most Unices, though. - second, the BFS algorithm is not that well-studied, since AFAIK it was refused for inclusion in the Linux kernel; someone in the python-dev community would therefore have to make sense of, and evaluate, its heuristic. I also don't consider my own patch a very satisfactory "solution", although it has the reassuring quality of being simple and short (and easy to revert!). That said, most of us are programmers and we love to invent ways of fixing technical issues. It sometimes leads us to consider some things issues even when they are mostly theoretical. This is why I am lukewarm on this. I think interested people should focus on real-world testing (rather than Dave and I's synthetic tests) of the new GIL, with or without the various patches, and share the results. Otherwise, Dj Gilcrease's suggestion of waiting for third-party reports is also a very good one. Regards Antoine. From guido at python.org Wed May 19 01:11:58 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 18 May 2010 16:11:58 -0700 Subject: [Python-Dev] Unordered tuples/lists In-Reply-To: <201005182313.42935.me@gustavonarea.net> References: <201005182313.42935.me@gustavonarea.net> Message-ID: This is typically called a "bag". Maybe searching for that will help you find a recipe? On Tue, May 18, 2010 at 3:13 PM, Gustavo Narea wrote: > Hello, everybody. > > I've been searching for a data structure like a tuple/list *but* unordered -- > like a set, but duplicated elements shouldn't be removed. I have not even > found a recipe, so I'd like to write an implementation and contribute it to > the "collections" module in the standard library. > > This is the situation I have: I have a data structure that represents an > arithmetic/boolean operation. Operations can be commutative, which means that > the order of their operands don't change the result of the operation. This is, > the following operations are equivalent: > ? ?operation(a, b, c) == operation(c, b, a) == operation(b, a, c) > ? ?operation(a, b, a) == operation(a, a, b) == operation(b, a, a) > ? ?operation(a, a) == operation(a, a) > > So, I need a type to store the arguments/operands so that if two of these > collections have the same elements with the same multiplicity, they are > equivalent, regardless of the order. > > A multiset is not exactly what I need: I still need to use the elements in the > order they were given. For example, the logical conjuction (aka "and" > operator) has a left and right operands; I need to evaluate the first/left one > and, if it returned True, then call the second/right one. They must not be > evaluated in a random order. > > To sum up, it would behave like a tuple or a list, except when it's compared > with another object: They would be equivalent if they're both unordered > tuples/lists, and have the same elements. There can be mutable and immutable > editions (UnorderedList and UnorderedTuple, respectively). > > I will write a PEP to elaborate on this if you think it'd be nice to have. Or, > should I have written the PEP first? > > Cheers, > -- > Gustavo Narea . > | Tech blog: =Gustavo/(+blog)/tech ?~ ?About me: =Gustavo/about | > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From benjamin at python.org Wed May 19 01:22:49 2010 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 18 May 2010 19:22:49 -0400 Subject: [Python-Dev] Unordered tuples/lists In-Reply-To: References: <201005182313.42935.me@gustavonarea.net> Message-ID: 2010/5/18 Guido van Rossum : > This is typically called a "bag". Maybe searching for that will help > you find a recipe? Yes, and we have one in Python 2.7+ called collections.Counter. -- Regards, Benjamin From steve at pearwood.info Wed May 19 01:28:29 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 19 May 2010 09:28:29 +1000 Subject: [Python-Dev] Unordered tuples/lists In-Reply-To: <201005182313.42935.me@gustavonarea.net> References: <201005182313.42935.me@gustavonarea.net> Message-ID: <201005190928.30079.steve@pearwood.info> On Wed, 19 May 2010 08:13:42 am Gustavo Narea wrote: > Hello, everybody. > > I've been searching for a data structure like a tuple/list *but* > unordered -- like a set, but duplicated elements shouldn't be > removed. I have not even found a recipe, so I'd like to write an > implementation and contribute it to the "collections" module in the > standard library. [...] > So, I need a type to store the arguments/operands so that if two of > these collections have the same elements with the same multiplicity, > they are equivalent, regardless of the order. If I've understood your requirements, this should probably do it: # Untested. class MyCounter(collections.Counter): def __eq__(self, other): try: other = other.items() except AttributeError: return False return sorted(self.items()) == sorted(other) def __ne__(self, other): return not self == other > A multiset is not exactly what I need: I still need to use the > elements in the order they were given. For example, the logical > conjuction (aka "and" operator) has a left and right operands; I need > to evaluate the first/left one and, if it returned True, then call > the second/right one. They must not be evaluated in a random order. These requirements seem specialised enough to me that I expect it would be wasteful to add your class to the standard library. It's not quite an OrderedMultiSet (perhaps built on an OrderedDict), but a MultiSet that sometimes behaves as ordered and sometimes doesn't. > To sum up, it would behave like a tuple or a list, except when it's > compared with another object: They would be equivalent if they're > both unordered tuples/lists, and have the same elements. There can be > mutable and immutable editions (UnorderedList and UnorderedTuple, > respectively). > > I will write a PEP to elaborate on this if you think it'd be nice to > have. Or, should I have written the PEP first? Neither. I think you will need to demonstrate the need for these first. The Python standard library doesn't generally add data types on the basis of theoretical nice-to-have, or even for a single person's use-case (unless that person is Guido *wink*). Your first step should be to publish the classes somewhere else. If they are small enough, the Python recipes on ActiveState would be good, otherwise PyPI. If there is a demonstrated need for these classes, and you (or somebody else) is willing to maintain them, then they may be added to the collections module (perhaps for Python 3.3, since I think 3.2 and 2.7 are in feature-freeze). I suggest you also take this idea to python-list at python.org or comp.lang.python first, to see whether there is any enthusiasm from the wider Python community. -- Steven D'Aprano From phd at phd.pp.ru Wed May 19 01:53:33 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Wed, 19 May 2010 03:53:33 +0400 Subject: [Python-Dev] Unordered tuples/lists In-Reply-To: <201005182313.42935.me@gustavonarea.net> References: <201005182313.42935.me@gustavonarea.net> Message-ID: <20100518235333.GA31031@phd.pp.ru> On Tue, May 18, 2010 at 11:13:42PM +0100, Gustavo Narea wrote: > To sum up, it would behave like a tuple or a list, except when it's compared > with another object: They would be equivalent if they're both unordered > tuples/lists, and have the same elements. There can be mutable and immutable > editions (UnorderedList and UnorderedTuple, respectively). class UnorderedList(list): def __eq__(self, other): if not isinstance(other, UnorderedList): return False return sorted(self) == sorted(other) def __ne__(self, other): return not self.__eq__(other) Do you need more than that? Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From ben+python at benfinney.id.au Wed May 19 02:19:07 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 19 May 2010 10:19:07 +1000 Subject: [Python-Dev] Unordered tuples/lists References: <201005182313.42935.me@gustavonarea.net> Message-ID: <87y6fgiw04.fsf@benfinney.id.au> Gustavo Narea writes: > I've been searching for a data structure like a tuple/list *but* > unordered -- like a set, but duplicated elements shouldn't be removed. By that description, you're looking for the ?Bag? pattern. [?] > A multiset is not exactly what I need: I still need to use the > elements in the order they were given. This appears to contradict your explicit request that the collection type be unordered. [?] > To sum up, it would behave like a tuple or a list, except when it's > compared with another object: They would be equivalent if they're both > unordered tuples/lists, and have the same elements. There can be > mutable and immutable editions (UnorderedList and UnorderedTuple, > respectively). Okay, so you want something rather special; it would be misleading to call this an unordered type, since you *do* want ordering preserved. It's only the comparison operators that would ignore ordering. AFAIK you'll need to design and implement this yourself. > I will write a PEP to elaborate on this if you think it'd be nice to > have. Or, should I have written the PEP first? Either way; but I think this discussion belongs on ?python-ideas? while the behaviour is still being designed. -- \ ?On the internet you simply can't outsource parenting.? ?Eliza | `\ Cussen, _Top 10 Internet Filter Lies_, The Punch, 2010-03-25 | _o__) | Ben Finney From mike.klaas at gmail.com Wed May 19 02:26:44 2010 From: mike.klaas at gmail.com (Mike Klaas) Date: Tue, 18 May 2010 17:26:44 -0700 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <20100518235026.03cad25d@pitrou.net> References: <20100518235026.03cad25d@pitrou.net> Message-ID: On Tue, May 18, 2010 at 2:50 PM, Antoine Pitrou wrote: > There's no chance for this since the patch relies on the new GIL. > (that's unless there's a rush to backport the new GIL in 2.7, of course) Thanks I missed that detail. > I think your "rare long delays" might be related to the old GIL's own > problems, though. How long are they? Typically between 20 and 60s. This is the time it takes to send and receive a single small packet on an already-active tcp connection to ensure it is still alive. Most of the time it is < 1ms. I don't have strong evidence that GIL issues are causing the problem, because I can't reliably reproduce the issue. But the general setup is similar (one thread doing light io experiencing odd delays in a process with multiple threads that are often cpu-bound, on a multi-core machine) thanks, -Mike From dave at dabeaz.com Wed May 19 02:35:48 2010 From: dave at dabeaz.com (David Beazley) Date: Tue, 18 May 2010 19:35:48 -0500 Subject: [Python-Dev] Summing up In-Reply-To: References: Message-ID: Antoine, This is a pretty good summary that mirrors my thoughts on the GIL matter as well. In the big picture, I do think it's desirable for Python to address the multicore performance issue--namely to not have the performance needlessly thrashed in that environment. The original new GIL addressed this. The I/O convoy effect problem is more subtle. Personally, I think it's an issue that at least merits further study because trying to overlap I/O with computation is a known programming technique that might be useful for people using Python to do message passing, distributed computation, etc. As an example, the multiprocessing module uses threads as part of its queue implementation. Is it impacted by convoying? I honestly don't know. I agree that getting some more real-world experience would be useful. Cheers, Dave > From: Antoine Pitrou > > Ok, this is a good opportunity to try to sum up, from my point of view. > > The main problem of the old GIL, which was evidenced in Dave's original > study (not this year's, but the previous one) *is* fixed unless someone > demonstrates otherwise. > > It should be noted that witnessing a slight performance degradation on > a multi-core machine is not enough to demonstrate such a thing. The > degradation could be caused by other factors, such as thread migration, > bad OS behaviour, or even locking peculiarities in your own > application, which are not related to the GIL. A good test is whether > performance improves if you play with sys.setswitchinterval(). > > > Dave's newer study regards another issue, which I must stress is also > present in the old GIL algorithm, and therefore must have affected, if > it is serious, real-world applications in 2.x. And indeed, the test I > recently added to ccbench evidences the huge drop in socket I/Os per > second when there's a background CPU thread; this test exercises the > same situation as Dave's demos, only with a less trivial CPU workload: > > == CPython 2.7b2+.0 (trunk:81274M) == > == x86_64 Linux on 'x86_64' == > > --- I/O bandwidth --- > > Background CPU task: Pi calculation (Python) > > CPU threads=0: 23034.5 packets/s. > CPU threads=1: 6.4 ( 0 %) > CPU threads=2: 15.7 ( 0 %) > CPU threads=3: 13.9 ( 0 %) > CPU threads=4: 20.8 ( 0 %) > > (note: I've just changed my desktop machine, so these figures are > different from what I've posted weeks or months ago) > > > Regardless of the fact that apparently noone reported it in real-world > conditions, we *could* decide that the issue needs fixing. If we > decide so, Nir's approach is the most rigorous one: it tries to fix > the problem thoroughly, rather than graft an additional heuristic. Nir > also has tested his patch on a variety of machines, more so than Dave > and I did with our own patches; he is obviously willing to go forward. > > Right now, there are two problems with Nir's proposal: > > - first, what Nick said: the difficulty of having reliable > high-precision cross-platform time sources, which are necessary for > the BFS algorithm. Ironically, timestamp counters have their own > problems on multi-core machines (they can go out of sync between > CPUs). gettimeofday() and clock_gettime() may be precise enough on > most Unices, though. > > - second, the BFS algorithm is not that well-studied, since AFAIK it > was refused for inclusion in the Linux kernel; someone in the > python-dev community would therefore have to make sense of, and > evaluate, its heuristic. > > I also don't consider my own patch a very satisfactory "solution", > although it has the reassuring quality of being simple and short (and > easy to revert!). > > > That said, most of us are programmers and we love to invent ways > of fixing technical issues. It sometimes leads us to consider some > things issues even when they are mostly theoretical. This is why I > am lukewarm on this. I think interested people should focus on > real-world testing (rather than Dave and I's synthetic tests) of the new > GIL, with or without the various patches, and share the results. > > Otherwise, Dj Gilcrease's suggestion of waiting for third-party reports > is also a very good one. > > Regards > > Antoine. From solipsis at pitrou.net Wed May 19 02:43:15 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 May 2010 02:43:15 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: <20100518235026.03cad25d@pitrou.net> Message-ID: <20100519024315.28f6f19c@pitrou.net> On Tue, 18 May 2010 17:26:44 -0700 Mike Klaas wrote: > > > I think your "rare long delays" might be related to the old GIL's own > > problems, though. How long are they? > > Typically between 20 and 60s. You mean milliseconds I suppose? If it's the case, then you may simply be witnessing garbage collection runs. I've measured garbage collection runs of about 50 ms each on a Web application, with the full framework loaded and a bunch of objects in memory. If you really meant seconds, it looks a bit too high to be GIL-related. What kind of things are the CPU threads doing? Regards Antoine. From ncoghlan at gmail.com Wed May 19 05:20:36 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 May 2010 13:20:36 +1000 Subject: [Python-Dev] Summing up In-Reply-To: References: Message-ID: <4BF35904.50503@gmail.com> On 19/05/10 10:35, David Beazley wrote: > Antoine, > > This is a pretty good summary that mirrors my thoughts on the GIL > matter as well. In the big picture, I do think it's desirable for > Python to address the multicore performance issue--namely to not have > the performance needlessly thrashed in that environment. The > original new GIL addressed this. > > The I/O convoy effect problem is more subtle. Personally, I think > it's an issue that at least merits further study because trying to > overlap I/O with computation is a known programming technique that > might be useful for people using Python to do message passing, > distributed computation, etc. As an example, the multiprocessing > module uses threads as part of its queue implementation. Is it > impacted by convoying? I honestly don't know. I agree that getting > some more real-world experience would be useful. My takeaway from this discussion is that: A. we should leave the new GIL in 3.2 in its current (relatively) simple form for now, keeping the various patches in issue 7946 in our back pocket if someone finds real world examples of the convoying effect discussed there. The idea here being that we shouldn't complicate the implementation without some solid evidence that doing so is actually necessary for real world workloads. B. some more thought should be given to incorporating the new GIL into 2.7. However, this requires two things: - an update to the patch in 7753 to either retain the old GIL for platforms not supported by the new GIL or else to make the new GIL a configure option - Benjamin accepting that patch (as it would likely mean adding another beta release to the cycle) In the absence of an updated version of the 7753 patch, backporting the new GIL to 2.7 isn't really a serious option. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Wed May 19 08:32:03 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 19 May 2010 08:32:03 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: <20100517141229.13a2d66c@pitrou.net> <20100517150526.0295af9e@pitrou.net> <20100518125312.404b0206@pitrou.net> <1274187152.3083.13.camel@localhost.localdomain> <4BF2EDE2.8090906@v.loewis.de> Message-ID: <4BF385E3.9030903@v.loewis.de> > I think the new GIL should be given a year or so in the wild before > you start trying to optimize theoretical issues you may run into. If > in a year people come back and have some examples of where a proper > scheduler would help improve speed on multi-core systems even more, > then we can address the issue at that time. Exactly my feelings. Regards, Martin From dave at dabeaz.com Wed May 19 13:00:04 2010 From: dave at dabeaz.com (David Beazley) Date: Wed, 19 May 2010 06:00:04 -0500 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: Message-ID: > From: "Martin v. L?wis" > To: Dj Gilcrease > Cc: python-dev at python.org > Subject: Re: [Python-Dev] Fixing the GIL (with a BFS scheduler) > Message-ID: <4BF385E3.9030903 at v.loewis.de> > Content-Type: text/plain; charset=ISO-8859-1 > >> I think the new GIL should be given a year or so in the wild before >> you start trying to optimize theoretical issues you may run into. If >> in a year people come back and have some examples of where a proper >> scheduler would help improve speed on multi-core systems even more, >> then we can address the issue at that time. > > Exactly my feelings. > Although I don't agree that the problem of I/O convoying is merely some "theoretical issue", I would agree with a go-slow approach---after all, the new GIL hasn't even appeared in any actual release yet. It might be a good idea to prominently document the fact that the new GIL has some known performance issues (e.g., possible I/O convoying), but that feedback concerning the performance of real-world applications is desired. Cheers, Dave From janssen at parc.com Wed May 19 18:04:26 2010 From: janssen at parc.com (Bill Janssen) Date: Wed, 19 May 2010 09:04:26 PDT Subject: [Python-Dev] Summing up In-Reply-To: <4BF35904.50503@gmail.com> References: <4BF35904.50503@gmail.com> Message-ID: <38562.1274285066@parc.com> Nick Coghlan wrote: > B. some more thought should be given to incorporating the new GIL into > 2.7. However, this requires two things: > - an update to the patch in 7753 to either retain the old GIL for > platforms not supported by the new GIL or else to make the new GIL a > configure option > - Benjamin accepting that patch (as it would likely mean adding > another beta release to the cycle) I think I agree with that, at least for 2.7.0. My fallback plan is to write an extension that does the thread affinity hack, monkeypatching threading.py, only for OS X. People like me who are really annoyed with the continued presence of the multicore bug can just add that to site.py or sitecustomize.py (does that still work?). I'd like to find out first whether you can actually change the thread affinity on OS X after the thread has been started. > In the absence of an updated version of the 7753 patch, backporting > the new GIL to 2.7 isn't really a serious option. Right. At least, not for this revision level. Bill From janssen at parc.com Wed May 19 18:09:16 2010 From: janssen at parc.com (Bill Janssen) Date: Wed, 19 May 2010 09:09:16 PDT Subject: [Python-Dev] pybuildbot.identify? Message-ID: <38589.1274285356@parc.com> The PPC buildbots are running pretty well, now that I've opened a few more ports, but I'd like to find this script "pybuildbot.identify" that they keep complaining about, and install it. I've poked around the Python sources, but haven't found it. Anyone know where to get it from? Thanks. Bill From debatem1 at gmail.com Wed May 19 18:27:36 2010 From: debatem1 at gmail.com (geremy condra) Date: Wed, 19 May 2010 09:27:36 -0700 Subject: [Python-Dev] Summing up In-Reply-To: <20100519011024.52deb1da@pitrou.net> References: <20100517150526.0295af9e@pitrou.net> <20100518125312.404b0206@pitrou.net> <1274187152.3083.13.camel@localhost.localdomain> <4BF2EDE2.8090906@v.loewis.de> <20100519011024.52deb1da@pitrou.net> Message-ID: On Tue, May 18, 2010 at 4:10 PM, Antoine Pitrou wrote: > On Tue, 18 May 2010 21:43:30 +0200 > Regardless of the fact that apparently noone reported it in real-world > conditions, we *could* decide that the issue needs fixing. If we > decide so, Nir's approach is the most rigorous one: it tries to fix > the problem thoroughly, rather than graft an additional heuristic. Nir > also has tested his patch on a variety of machines, more so than Dave > and I did with our own patches; he is obviously willing to go forward. > > Right now, there are two problems with Nir's proposal: > > - first, what Nick said: the difficulty of having reliable > ?high-precision cross-platform time sources, which are necessary for > ?the BFS algorithm. Ironically, timestamp counters have their own > ?problems on multi-core machines (they can go out of sync between > ?CPUs). gettimeofday() and clock_gettime() may be precise enough on > ?most Unices, though. > > - second, the BFS algorithm is not that well-studied, since AFAIK it > ?was refused for inclusion in the Linux kernel; someone in the > ?python-dev community would therefore have to make sense of, and > ?evaluate, its heuristic. I don't have the expertise to do this, but I'll be playing with the patch over the next few weeks, so if there's a specific piece of data you want, let me know. Geremy Condra From doko at ubuntu.com Wed May 19 18:27:49 2010 From: doko at ubuntu.com (Matthias Klose) Date: Wed, 19 May 2010 18:27:49 +0200 Subject: [Python-Dev] pybuildbot.identify? In-Reply-To: <38589.1274285356@parc.com> References: <38589.1274285356@parc.com> Message-ID: <4BF41185.3090600@ubuntu.com> On 19.05.2010 18:09, Bill Janssen wrote: > The PPC buildbots are running pretty well, now that I've opened a few > more ports, but I'd like to find this script "pybuildbot.identify" that > they keep complaining about, and install it. I've poked around the > Python sources, but haven't found it. > > Anyone know where to get it from? it's just a custom script, currently very linux specific. attached here. maybe it's worth to check it in and extend it for other configurations. Matthias -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pybuildbot.identify URL: From martin at v.loewis.de Wed May 19 19:03:13 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 19 May 2010 19:03:13 +0200 Subject: [Python-Dev] pybuildbot.identify? In-Reply-To: <38589.1274285356@parc.com> References: <38589.1274285356@parc.com> Message-ID: <4BF419D1.3060000@v.loewis.de> Bill Janssen wrote: > The PPC buildbots are running pretty well, now that I've opened a few > more ports, but I'd like to find this script "pybuildbot.identify" that > they keep complaining about, and install it. I've poked around the > Python sources, but haven't found it. You don't need to bother about it - it's fine that it's missing. Regards, Martin From janssen at parc.com Wed May 19 19:58:44 2010 From: janssen at parc.com (Bill Janssen) Date: Wed, 19 May 2010 10:58:44 PDT Subject: [Python-Dev] pybuildbot.identify? In-Reply-To: <4BF41185.3090600@ubuntu.com> References: <38589.1274285356@parc.com> <4BF41185.3090600@ubuntu.com> Message-ID: <43886.1274291924@parc.com> Matthias Klose wrote: > On 19.05.2010 18:09, Bill Janssen wrote: > > The PPC buildbots are running pretty well, now that I've opened a few > > more ports, but I'd like to find this script "pybuildbot.identify" that > > they keep complaining about, and install it. I've poked around the > > Python sources, but haven't found it. > > > > Anyone know where to get it from? > > it's just a custom script, currently very linux specific. attached > here. maybe it's worth to check it in and extend it for other > configurations. > > Matthias Well, please check it in somewhere, at least. Thanks. Bill From martin at v.loewis.de Wed May 19 21:45:39 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 19 May 2010 21:45:39 +0200 Subject: [Python-Dev] pybuildbot.identify? In-Reply-To: <43886.1274291924@parc.com> References: <38589.1274285356@parc.com> <4BF41185.3090600@ubuntu.com> <43886.1274291924@parc.com> Message-ID: <4BF43FE3.8040803@v.loewis.de> Bill Janssen wrote: > Matthias Klose wrote: > >> On 19.05.2010 18:09, Bill Janssen wrote: >>> The PPC buildbots are running pretty well, now that I've opened a few >>> more ports, but I'd like to find this script "pybuildbot.identify" that >>> they keep complaining about, and install it. I've poked around the >>> Python sources, but haven't found it. >>> >>> Anyone know where to get it from? >> it's just a custom script, currently very linux specific. attached >> here. maybe it's worth to check it in and extend it for other >> configurations. >> >> Matthias > > Well, please check it in somewhere, at least. Thanks. Can you please investigate why it was causing an error message in the first place? The which test should have determined that the script doesn't exist, and then it shouldn't have been invoked. So I'm puzzled why there is an error message at all. I slightly object to checking it in. It's *not* a mandatory part of a buildbot slave setup, and if you don't know what it is, you don't need to know. Regards, Martin From peter.a.portante at gmail.com Wed May 19 22:17:22 2010 From: peter.a.portante at gmail.com (Peter Portante) Date: Wed, 19 May 2010 16:17:22 -0400 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: Message-ID: Does anybody think that by having problems with the new GIL that it might further weaken the adoption rate for 3k? -peter On 5/19/10 7:00 AM, "David Beazley" wrote: >> From: "Martin v. L?wis" >> To: Dj Gilcrease >> Cc: python-dev at python.org >> Subject: Re: [Python-Dev] Fixing the GIL (with a BFS scheduler) >> Message-ID: <4BF385E3.9030903 at v.loewis.de> >> Content-Type: text/plain; charset=ISO-8859-1 >> >>> I think the new GIL should be given a year or so in the wild before >>> you start trying to optimize theoretical issues you may run into. If >>> in a year people come back and have some examples of where a proper >>> scheduler would help improve speed on multi-core systems even more, >>> then we can address the issue at that time. >> >> Exactly my feelings. >> > > Although I don't agree that the problem of I/O convoying is merely some > "theoretical issue", I would agree with a go-slow approach---after all, the > new GIL hasn't even appeared in any actual release yet. It might be a good > idea to prominently document the fact that the new GIL has some known > performance issues (e.g., possible I/O convoying), but that feedback > concerning the performance of real-world applications is desired. > > Cheers, > Dave > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com From martin at v.loewis.de Wed May 19 22:34:50 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 19 May 2010 22:34:50 +0200 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: Message-ID: <4BF44B6A.5010805@v.loewis.de> Peter Portante wrote: > Does anybody think that by having problems with the new GIL that it might > further weaken the adoption rate for 3k? -peter No, to the contrary. By having the new GIL being superior to the old implementation, the adoption rate for 3k will increase. Regards, Martin From digitalxero at gmail.com Wed May 19 22:39:14 2010 From: digitalxero at gmail.com (Dj Gilcrease) Date: Wed, 19 May 2010 16:39:14 -0400 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: References: Message-ID: On Wed, May 19, 2010 at 4:17 PM, Peter Portante wrote: > Does anybody think that by having problems with the new GIL that it might > further weaken the adoption rate for 3k? -peter Nope, because the remaining issues with the new GIL affect the old GIL as well, and have yet to be proven to affect real world apps. My server is the only real world example I have seen and the modest 5% slow down it experiences on the new GIL & multi-core vs new GIL & single core can potentially be explained by other issues. The old GIL has a 25 to 30% slowdown on multi vs single core so over all the new GIL gives my server a 20% speed boost. Incremental upgrades to core functionality like the GIL is the better way to go then to try an optimize every facet of it before hand, because some of those optimizations may have unintended side effects in real world apps and if you attempt all the optimizations at once it makes it harder to figure out which is the cause of the unintended side effects. From dave at dabeaz.com Wed May 19 22:49:02 2010 From: dave at dabeaz.com (David Beazley) Date: Wed, 19 May 2010 15:49:02 -0500 Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <4BF44B6A.5010805@v.loewis.de> References: <4BF44B6A.5010805@v.loewis.de> Message-ID: <44A67E74-8D06-4A04-972C-B36EDCC1447E@dabeaz.com> Yes, but the million dollar question is whether or not it really is superior with the I/O convoying problem in the current implementation (an effect that is substantially worse with the new GIL than with the old one by the way). Personally, I think the convoying issue is something that will have to be fixed eventually. Although, I also agree that it merits more study--especially with real-world applications. Cheers, Dave On May 19, 2010, at 3:34 PM, Martin v. L?wis wrote: > Peter Portante wrote: >> Does anybody think that by having problems with the new GIL that it might >> further weaken the adoption rate for 3k? -peter > > No, to the contrary. By having the new GIL being superior to the old > implementation, the adoption rate for 3k will increase. > > Regards, > Martin From janssen at parc.com Wed May 19 23:27:27 2010 From: janssen at parc.com (Bill Janssen) Date: Wed, 19 May 2010 14:27:27 PDT Subject: [Python-Dev] pybuildbot.identify? In-Reply-To: <4BF43FE3.8040803@v.loewis.de> References: <38589.1274285356@parc.com> <4BF41185.3090600@ubuntu.com> <43886.1274291924@parc.com> <4BF43FE3.8040803@v.loewis.de> Message-ID: <46492.1274304447@parc.com> Martin v. L?wis wrote: > Bill Janssen wrote: > > Matthias Klose wrote: > > > >> On 19.05.2010 18:09, Bill Janssen wrote: > >>> The PPC buildbots are running pretty well, now that I've opened a few > >>> more ports, but I'd like to find this script "pybuildbot.identify" that > >>> they keep complaining about, and install it. I've poked around the > >>> Python sources, but haven't found it. > >>> > >>> Anyone know where to get it from? > >> it's just a custom script, currently very linux specific. attached > >> here. maybe it's worth to check it in and extend it for other > >> configurations. > >> > >> Matthias > > > > Well, please check it in somewhere, at least. Thanks. > > Can you please investigate why it was causing an error message in the > first place? The which test should have determined that the script > doesn't exist, and then it shouldn't have been invoked. So I'm puzzled > why there is an error message at all. I agree. Must be something in the buildbot code. > I slightly object to checking it in. It's *not* a mandatory part of a > buildbot slave setup, and if you don't know what it is, you don't need > to know. Well, if it failed silently, I would agree. As it is, I created a Darwin version and sent it off to Matthias. Bill From janssen at parc.com Wed May 19 23:35:25 2010 From: janssen at parc.com (Bill Janssen) Date: Wed, 19 May 2010 14:35:25 PDT Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler) In-Reply-To: <4BF44B6A.5010805@v.loewis.de> References: <4BF44B6A.5010805@v.loewis.de> Message-ID: <46623.1274304925@parc.com> Martin v. L?wis wrote: > Peter Portante wrote: > > Does anybody think that by having problems with the new GIL that it might > > further weaken the adoption rate for 3k? -peter > > No, to the contrary. By having the new GIL being superior to the old > implementation, the adoption rate for 3k will increase. Well, if that's a strategy, there's still time to make 2+2 equal 5 in the "old" implementation. Bill From yaniv at aknin.name Thu May 20 00:13:03 2010 From: yaniv at aknin.name (Yaniv Aknin) Date: Thu, 20 May 2010 08:13:03 +1000 Subject: [Python-Dev] Documenting [C]Python's Internals Message-ID: Hi, I wanted to let python-dev know about a series of articles about CPython's internals I'm publishing under the collective title "Guido's Python"* ( http://tech.blog.aknin.name/tag/guidos-python/). Three articles already were published already, more are planned (mainly focused on CPython/py3k, but comparisons with other implementations may also be covered; we'll see). So far I've done an introduction/whirlwind tour of Py_Main and a two-article in-depth review of the (new-style) object system. I'm sharing this with you (and hope you care) due to three reasons, probably in escalating importance: (a) Maybe some of python-dev's readers would be interested (possibly the newer and more silent members). (b) Maybe my scales are wrong, but I was a bit surprised by the number of readers (>20,000 in the past two weeks); I wouldn't want to mislead such a reader base and would be happy if a veteran here would be interested in aiding by technically proofing the material (shan't be too hard I hope, feel free to contact me directly if qualified and interested). (c) While the content is currently geared to be blog-oriented, if it's found worthy by the group I'd be delighted to formulate it into something more 'reference-material-ish' and give it back to the community. I found no centrally organized CPython-internals material other than bits and pieces (descrintro, eclectic blog posts, lectures, C-API reference, etc), and I hope maybe something like this could be featured more officially on python.org, with the relevant 'this is subject to change' disclaimers (can be a document for new contributors, for pure Python programmers who're just interested, or for whatever we decide). Questions? Comments? - Yaniv * think "Tim Berners-Lee's Web" or "Keanu Reeves' Green Gibberish", see the first post for details -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Thu May 20 00:35:07 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 19 May 2010 23:35:07 +0100 Subject: [Python-Dev] Documenting [C]Python's Internals In-Reply-To: References: Message-ID: <4BF4679B.3000809@voidspace.org.uk> On 19/05/2010 23:13, Yaniv Aknin wrote: > Hi, > > I wanted to let python-dev know about a series of articles about > CPython's internals I'm publishing under the collective title "Guido's > Python"* (http://tech.blog.aknin.name/tag/guidos-python/). Three > articles already were published already, more are planned (mainly > focused on CPython/py3k, but comparisons with other implementations > may also be covered; we'll see). So far I've done an > introduction/whirlwind tour of Py_Main and a two-article in-depth > review of the (new-style) object system. Whether or not they become part of the Python documentation I have very much enjoyed and appreciated this series of blog entries. I still covet the ability to contribute to Python in C and these articles are a great introduction to the underlying Python interpreter and object system. Please continue! All the best, Michael Foord > > I'm sharing this with you (and hope you care) due to three reasons, > probably in escalating importance: > (a) Maybe some of python-dev's readers would be interested (possibly > the newer and more silent members). > > (b) Maybe my scales are wrong, but I was a bit surprised by the number > of readers (>20,000 in the past two weeks); I wouldn't want to mislead > such a reader base and would be happy if a veteran here would be > interested in aiding by technically proofing the material (shan't be > too hard I hope, feel free to contact me directly if qualified and > interested). > > (c) While the content is currently geared to be blog-oriented, if it's > found worthy by the group I'd be delighted to formulate it into > something more 'reference-material-ish' and give it back to the > community. I found no centrally organized CPython-internals material > other than bits and pieces (descrintro, eclectic blog posts, lectures, > C-API reference, etc), and I hope maybe something like this could be > featured more officially on python.org , with the > relevant 'this is subject to change' disclaimers (can be a document > for new contributors, for pure Python programmers who're just > interested, or for whatever we decide). > > Questions? Comments? > - Yaniv > > * think "Tim Berners-Lee's Web" or "Keanu Reeves' Green Gibberish", > see the first post for details > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Thu May 20 01:08:22 2010 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Thu, 20 May 2010 01:08:22 +0200 Subject: [Python-Dev] Documenting [C]Python's Internals In-Reply-To: References: Message-ID: 2010/5/20 Yaniv Aknin : > Hi, > I wanted to let python-dev know about a series of articles about CPython's > internals I'm publishing under the collective title "Guido's Python"* > (http://tech.blog.aknin.name/tag/guidos-python/). Three articles already > were published already, more are planned (mainly focused on CPython/py3k, > but comparisons with other implementations may also be covered; we'll > see).?So far I've done an introduction/whirlwind?tour of Py_Main and a > two-article in-depth review of the (new-style) object system. > I'm sharing this with you (and hope you care) due to three reasons, probably > in escalating importance: > (a) Maybe some of python-dev's readers would be interested (possibly the > newer and more silent members). > (b) Maybe my scales are wrong, but I was a bit surprised by the number of > readers (>20,000 in the past two weeks); I wouldn't want to mislead such a > reader base and would be happy if a veteran here would be interested in > aiding by technically proofing the material (shan't be too hard I hope, feel > free to contact me directly if qualified and interested). > (c) While the content is currently geared to be blog-oriented, if it's found > worthy by the group I'd be delighted to formulate it into something more > 'reference-material-ish' and give it back to the community. I found no > centrally organized CPython-internals material other than bits and pieces > (descrintro, eclectic blog posts, lectures, C-API reference, etc), and I > hope maybe something like this could be featured more officially > on?python.org, with the relevant 'this is subject to change' disclaimers > (can be a document for new contributors, for pure Python programmers who're > just interested, or?for whatever we decide). > Questions? Comments? > ?- Yaniv > *?think "Tim Berners-Lee's Web" or "Keanu Reeves' Green Gibberish", see the > first post for details > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com Great! This can be *extremely* useful for new developers like me who still haven't took a look at cPython internals. Thanks for the effort. --- Giampaolo http://code.google.com/p/pyftpdlib http://code.google.com/p/psutil From g.rodola at gmail.com Thu May 20 01:42:08 2010 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Thu, 20 May 2010 01:42:08 +0200 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method Message-ID: >>> class A: ... def echo(self, x): ... return x ... >>> a = A() >>> a.echo() Traceback (most recent call last): File "", line 1, in TypeError: echo() takes exactly 2 arguments (1 given) >>> I bet my last 2 cents this has already been raised in past but I want to give it a try and revamp the subject anyway. Is there a reason why the error shouldn't be adjusted to state that *1* argument is actually required instead of 2? --- Giampaolo http://code.google.com/p/pyftpdlib http://code.google.com/p/psutil From fuzzyman at voidspace.org.uk Thu May 20 01:47:33 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 20 May 2010 00:47:33 +0100 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: References: Message-ID: <4BF47895.3000700@voidspace.org.uk> On 20/05/2010 00:42, Giampaolo Rodol? wrote: >>>> class A: >>>> > ... def echo(self, x): > ... return x > ... > >>>> a = A() >>>> a.echo() >>>> > Traceback (most recent call last): > File "", line 1, in > TypeError: echo() takes exactly 2 arguments (1 given) > >>>> > I bet my last 2 cents this has already been raised in past but I want > to give it a try and revamp the subject anyway. > Is there a reason why the error shouldn't be adjusted to state that > *1* argument is actually required instead of 2? > > +1 - I've seen many newbies confused by this error and I sometimes do a double-take myself. All the best, Michael > --- Giampaolo > http://code.google.com/p/pyftpdlib > http://code.google.com/p/psutil > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From john.arbash.meinel at gmail.com Thu May 20 01:53:03 2010 From: john.arbash.meinel at gmail.com (John Arbash Meinel) Date: Wed, 19 May 2010 18:53:03 -0500 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: References: Message-ID: <4BF479DF.4020102@gmail.com> Giampaolo Rodol? wrote: >>>> class A: > ... def echo(self, x): > ... return x > ... >>>> a = A() >>>> a.echo() > Traceback (most recent call last): > File "", line 1, in > TypeError: echo() takes exactly 2 arguments (1 given) > > I bet my last 2 cents this has already been raised in past but I want > to give it a try and revamp the subject anyway. > Is there a reason why the error shouldn't be adjusted to state that > *1* argument is actually required instead of 2? > Because you wouldn't want to have A.echo() Say that it takes 1 argument and (-1 given) ? John =:-> From debatem1 at gmail.com Thu May 20 02:15:43 2010 From: debatem1 at gmail.com (geremy condra) Date: Wed, 19 May 2010 17:15:43 -0700 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: References: Message-ID: On Wed, May 19, 2010 at 4:42 PM, Giampaolo Rodol? wrote: >>>> class A: > ... ? ? def echo(self, x): > ... ? ? ? ? ? ? return x > ... >>>> a = A() >>>> a.echo() > Traceback (most recent call last): > ?File "", line 1, in > TypeError: echo() takes exactly 2 arguments (1 given) >>>> > > I bet my last 2 cents this has already been raised in past but I want > to give it a try and revamp the subject anyway. > Is there a reason why the error shouldn't be adjusted to state that > *1* argument is actually required instead of 2? > > > --- Giampaolo Because it actually does take two arguments (self and x) and it only got one (self). I understand the confusion (and was bitten by it myself when I was a newbie) but the interpreter is only telling you the truth. Geremy Condra From rdmurray at bitdance.com Thu May 20 02:20:04 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 19 May 2010 20:20:04 -0400 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: References: Message-ID: <20100520002004.9A3C1209B76@kimball.webabinitio.net> On Thu, 20 May 2010 01:42:08 +0200, =?ISO-8859-1?Q?Giampaolo_Rodol=E0?= wrote: > >>> class A: > ... def echo(self, x): > ... return x > ... > >>> a = A() > >>> a.echo() > Traceback (most recent call last): > File "", line 1, in > TypeError: echo() takes exactly 2 arguments (1 given) > >>> > > I bet my last 2 cents this has already been raised in past but I want > to give it a try and revamp the subject anyway. > Is there a reason why the error shouldn't be adjusted to state that > *1* argument is actually required instead of 2? You can contribute to issue 2516 if you like :) http://bugs.python.org/issue2516 -- R. David Murray www.bitdance.com From stephen at xemacs.org Thu May 20 04:55:02 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 20 May 2010 11:55:02 +0900 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: References: Message-ID: <87k4qzgu49.fsf@uwakimon.sk.tsukuba.ac.jp> Giampaolo Rodol? writes: > >>> class A: > ... def echo(self, x): > ... return x > ... > >>> a = A() > >>> a.echo() > Traceback (most recent call last): > File "", line 1, in > TypeError: echo() takes exactly 2 arguments (1 given) > >>> > > I bet my last 2 cents this has already been raised in past but I want > to give it a try and revamp the subject anyway. > Is there a reason why the error shouldn't be adjusted to state that > *1* argument is actually required instead of 2? As a function, it does take two arguments, and can be called explicitly that way, no? Adjustment is not enough, the message needs to be substantially rewritten. Something like TypeError: invoked as a method, echo() takes exactly 1 argument (0 given) captures the semantics, but is perhaps too verbose. From tjreedy at udel.edu Thu May 20 06:01:16 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 20 May 2010 00:01:16 -0400 Subject: [Python-Dev] Documenting [C]Python's Internals In-Reply-To: References: Message-ID: On 5/19/2010 6:13 PM, Yaniv Aknin wrote: > Hi, > > I wanted to let python-dev know about a series of articles about > CPython's internals I'm publishing under the collective title "Guido's > Python"* (http://tech.blog.aknin.name/tag/guidos-python/). This link has all post concatenated together in reverse order of how they should be read. The tags link returns the same page. Does your blog software allow you to make a master post and update with new links as available? Three > articles already were published already, more are planned (mainly > focused on CPython/py3k, but comparisons with other implementations may > also be covered; we'll see). So far I've done an > introduction/whirlwind tour of Py_Main and a two-article in-depth review > of the (new-style) object system. > > I'm sharing this with you (and hope you care) due to three reasons, > probably in escalating importance: > (a) Maybe some of python-dev's readers would be interested (possibly the > newer and more silent members). > (b) Maybe my scales are wrong, but I was a bit surprised by the number > of readers (>20,000 in the past two weeks); I wouldn't want to mislead > such a reader base and would be happy if a veteran here would be > interested in aiding by technically proofing the material (shan't be too > hard I hope, feel free to contact me directly if qualified and interested). I would if I were qualified, but I an mot. One way to get people to help with details is to publish mistakes. This happens all the time on python-list ;-). Pre-review would be nice though. > (c) While the content is currently geared to be blog-oriented, if it's > found worthy by the group I'd be delighted to formulate it into > something more 'reference-material-ish' and give it back to the > community. I found no centrally organized CPython-internals material > other than bits and pieces (descrintro, eclectic blog posts, lectures, > C-API reference, etc), and I hope maybe something like this could be > featured more officially on python.org , with the > relevant 'this is subject to change' disclaimers (can be a document for > new contributors, for pure Python programmers who're just interested, > or for whatever we decide). People have asked for an internals-doc since I started over a decade ago. A coherent CPython3.2 Internals would be nice to have with the 3.2 release next December or so, whether or not it was made 'official'. Terry Jan Reedy From greg.ewing at canterbury.ac.nz Thu May 20 09:26:22 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 20 May 2010 19:26:22 +1200 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: <4BF479DF.4020102@gmail.com> References: <4BF479DF.4020102@gmail.com> Message-ID: <4BF4E41E.3050800@canterbury.ac.nz> John Arbash Meinel wrote: > Because you wouldn't want to have > > A.echo() > > Say that it takes 1 argument and (-1 given) ? Something like "1 argument in addition to 'self'" would be reasonably clear and would cover both situations. +1 on fixing this from me, too. Even when you understand exactly what's going on, it's annoying having to make the mental adjustment every time. -- Greg From ben+python at benfinney.id.au Thu May 20 09:46:54 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 20 May 2010 17:46:54 +1000 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method References: <4BF479DF.4020102@gmail.com> <4BF4E41E.3050800@canterbury.ac.nz> Message-ID: <87iq6jgglt.fsf@benfinney.id.au> Greg Ewing writes: > John Arbash Meinel wrote: > > > Because you wouldn't want to have > > > > A.echo() > > > > Say that it takes 1 argument and (-1 given) ? > > Something like "1 argument in addition to 'self'" would be reasonably > clear and would cover both situations. Except that there's nothing special to the syntax or parser about the name ?self?. > +1 on fixing this from me, too. Even when you understand exactly > what's going on, it's annoying having to make the mental adjustment > every time. Would it help if the traceback showed the ?repr()? of each of the arguments received? That way it would be much clearer when the instance was received as the first argument. -- \ ?Any sufficiently advanced bug is indistinguishable from a | `\ feature.? ?Rich Kulawiec | _o__) | Ben Finney From floris.bruynooghe at gmail.com Thu May 20 10:02:07 2010 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Thu, 20 May 2010 09:02:07 +0100 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: <87k4qzgu49.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87k4qzgu49.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20100520080207.GA16837@laurie.devork> On Thu, May 20, 2010 at 11:55:02AM +0900, Stephen J. Turnbull wrote: > Giampaolo Rodol? writes: > > >>> class A: > > ... def echo(self, x): > > ... return x > > ... > > >>> a = A() > > >>> a.echo() > > Traceback (most recent call last): > > File "", line 1, in > > TypeError: echo() takes exactly 2 arguments (1 given) > > >>> > > > > I bet my last 2 cents this has already been raised in past but I want > > to give it a try and revamp the subject anyway. > > Is there a reason why the error shouldn't be adjusted to state that > > *1* argument is actually required instead of 2? > > As a function, it does take two arguments, and can be called > explicitly that way, no? Adjustment is not enough, the message needs > to be substantially rewritten. Something like > > TypeError: invoked as a method, echo() takes exactly 1 argument (0 given) > > captures the semantics, but is perhaps too verbose. How about: TypeError: bound method echo() takes exactly 1 argument (0 given) That way you can also have: "unbound method echo() ...". And it's as semantically correct as the short "echo() takes ..." Not having looked at the code I don't know how hard it is for the code that raises this traceback to notice if it's a bound or unbound method tough. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From me at gustavonarea.net Wed May 19 10:28:28 2010 From: me at gustavonarea.net (Gustavo Narea) Date: Wed, 19 May 2010 09:28:28 +0100 Subject: [Python-Dev] Unordered tuples/lists In-Reply-To: References: <201005182313.42935.me@gustavonarea.net> Message-ID: Hi, Guido. On Wed, May 19, 2010 at 12:11 AM, Guido van Rossum wrote: > This is typically called a "bag". Maybe searching for that will help > you find a recipe? > A bag/multiset is close to what I need, except for one thing: I need to iterate over the elements in the original order, not in a random order. The data structure I'm proposing is basically a list/tuple, and the only thing that changes is comparison with another unordered list/tuple: If they both have the same elements with the same multiplicity, they are equivalent (regardless of the order). Cheers, - Gustavo. -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at gustavonarea.net Wed May 19 10:56:03 2010 From: me at gustavonarea.net (Gustavo Narea) Date: Wed, 19 May 2010 09:56:03 +0100 Subject: [Python-Dev] Unordered tuples/lists In-Reply-To: <201005182313.42935.me@gustavonarea.net> References: <201005182313.42935.me@gustavonarea.net> Message-ID: Hello, Oleg. > class UnorderedList(list): > def __eq__(self, other): > if not isinstance(other, UnorderedList): > return False > return sorted(self) == sorted(other) > > def __ne__(self, other): > return not self.__eq__(other) > > Do you need more than that? > > Oleg. That's what I had in mind. I think it'd be useful enough to go in the standard library. Now that there's a sample implementation, should I still try to demonstrate why I believe it's worth adding to the stdlib and get support? Cheers, - Gustavo. On Tue, May 18, 2010 at 11:13 PM, Gustavo Narea wrote: > Hello, everybody. > > I've been searching for a data structure like a tuple/list *but* unordered > -- > like a set, but duplicated elements shouldn't be removed. I have not even > found a recipe, so I'd like to write an implementation and contribute it to > the "collections" module in the standard library. > > This is the situation I have: I have a data structure that represents an > arithmetic/boolean operation. Operations can be commutative, which means > that > the order of their operands don't change the result of the operation. This > is, > the following operations are equivalent: > operation(a, b, c) == operation(c, b, a) == operation(b, a, c) > operation(a, b, a) == operation(a, a, b) == operation(b, a, a) > operation(a, a) == operation(a, a) > > So, I need a type to store the arguments/operands so that if two of these > collections have the same elements with the same multiplicity, they are > equivalent, regardless of the order. > > A multiset is not exactly what I need: I still need to use the elements in > the > order they were given. For example, the logical conjuction (aka "and" > operator) has a left and right operands; I need to evaluate the first/left > one > and, if it returned True, then call the second/right one. They must not be > evaluated in a random order. > > To sum up, it would behave like a tuple or a list, except when it's > compared > with another object: They would be equivalent if they're both unordered > tuples/lists, and have the same elements. There can be mutable and > immutable > editions (UnorderedList and UnorderedTuple, respectively). > > I will write a PEP to elaborate on this if you think it'd be nice to have. > Or, > should I have written the PEP first? > > Cheers, > -- > Gustavo Narea . > | Tech blog: =Gustavo/(+blog)/tech ~ About me: =Gustavo/about | > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Thu May 20 11:49:02 2010 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Thu, 20 May 2010 11:49:02 +0200 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: <4BF479DF.4020102@gmail.com> References: <4BF479DF.4020102@gmail.com> Message-ID: 2010/5/20 John Arbash Meinel : > Giampaolo Rodol? wrote: >>>>> class A: >> ... ? ? def echo(self, x): >> ... ? ? ? ? ? ? return x >> ... >>>>> a = A() >>>>> a.echo() >> Traceback (most recent call last): >> ? File "", line 1, in >> TypeError: echo() takes exactly 2 arguments (1 given) >> >> I bet my last 2 cents this has already been raised in past but I want >> to give it a try and revamp the subject anyway. >> Is there a reason why the error shouldn't be adjusted to state that >> *1* argument is actually required instead of 2? >> > > Because you wouldn't want to have > > A.echo() > > Say that it takes 1 argument and (-1 given) ? > > John > =:-> > > I see that as a different error type: what you're doing there is calling a method of a class which you haven't instantiated in the first place. Actually the error message returned in this other case is not very clear as well: "unbound method echo() must be called with A instance as first argument (got nothing instead)" It talks about "arguments" while no arguments are actually involved in the problem: just a class I forgot to initialize. --- Giampaolo http://code.google.com/p/pyftpdlib http://code.google.com/p/psutil From phd at phd.pp.ru Thu May 20 12:40:25 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Thu, 20 May 2010 14:40:25 +0400 Subject: [Python-Dev] Unordered tuples/lists In-Reply-To: References: <201005182313.42935.me@gustavonarea.net> Message-ID: <20100520104025.GA9757@phd.pp.ru> On Wed, May 19, 2010 at 09:56:03AM +0100, Gustavo Narea wrote: > I think it'd be useful enough to go in the standard library. Now that > there's a sample implementation, should I still try to demonstrate why I > believe it's worth adding to the stdlib and get support? I think yes. How many developers would find it useful?.. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From fuzzyman at voidspace.org.uk Thu May 20 12:40:51 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 20 May 2010 11:40:51 +0100 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: References: <4BF479DF.4020102@gmail.com> Message-ID: <4BF511B3.1090203@voidspace.org.uk> On 20/05/2010 10:49, Giampaolo Rodol? wrote: > 2010/5/20 John Arbash Meinel: > >> Giampaolo Rodol? wrote: >> >>>>>> class A: >>>>>> >>> ... def echo(self, x): >>> ... return x >>> ... >>> >>>>>> a = A() >>>>>> a.echo() >>>>>> >>> Traceback (most recent call last): >>> File "", line 1, in >>> TypeError: echo() takes exactly 2 arguments (1 given) >>> >>> I bet my last 2 cents this has already been raised in past but I want >>> to give it a try and revamp the subject anyway. >>> Is there a reason why the error shouldn't be adjusted to state that >>> *1* argument is actually required instead of 2? >>> >>> >> Because you wouldn't want to have >> >> A.echo() >> >> Say that it takes 1 argument and (-1 given) ? >> >> John >> =:-> >> >> >> > I see that as a different error type: what you're doing there is > calling a method of a class which you haven't instantiated in the > first place. > Actually the error message returned in this other case is not very > clear as well: > > "unbound method echo() must be called with A instance as first > argument (got nothing instead)" > > It talks about "arguments" while no arguments are actually involved in > the problem: just a class I forgot to initialize. > Although the pattern of calling an unbound method with an instance as the first argument (self) is not uncommon - and if you're doing Class.method() that not only looks like what you're doing but is also an accurate error message. I would also expect that accidentally calling unbound methods is a much less common error than calling bound methods with the wrong number of arguments... All the best, Michael Foord > > --- Giampaolo > http://code.google.com/p/pyftpdlib > http://code.google.com/p/psutil > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From steve at pearwood.info Thu May 20 13:37:08 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 20 May 2010 21:37:08 +1000 Subject: [Python-Dev] Unordered tuples/lists In-Reply-To: <20100520104025.GA9757@phd.pp.ru> References: <201005182313.42935.me@gustavonarea.net> <20100520104025.GA9757@phd.pp.ru> Message-ID: <201005202137.08852.steve@pearwood.info> On Thu, 20 May 2010 08:40:25 pm Oleg Broytman wrote: > On Wed, May 19, 2010 at 09:56:03AM +0100, Gustavo Narea wrote: > > I think it'd be useful enough to go in the standard library. Now > > that there's a sample implementation, should I still try to > > demonstrate why I believe it's worth adding to the stdlib and get > > support? > > I think yes. How many developers would find it useful?.. Adding classes to the standard library doesn't come without cost. There's the maintenance and testing burden, and the cognitive burden of having to choose between similar classes with slight variations of behaviour. I don't think the benefit of this proposed subclass is great enough to outweigh the costs. It is a trivial subclass -- Oleg gave a seven line subclass of list -- with a fairly specialist use-case (it is a regular ordered list except for the purposes of equality comparisons). I don't believe it is a burden on developers to add it to their own application should they need it. -1 for this. If anyone wishes to support this proposal, please come up with a less misleading name than "UnorderedList". -- Steven D'Aprano From debatem1 at gmail.com Thu May 20 18:02:47 2010 From: debatem1 at gmail.com (geremy condra) Date: Thu, 20 May 2010 09:02:47 -0700 Subject: [Python-Dev] Unordered tuples/lists In-Reply-To: References: <201005182313.42935.me@gustavonarea.net> Message-ID: On Wed, May 19, 2010 at 1:56 AM, Gustavo Narea wrote: > Hello, Oleg. > >> >> class UnorderedList(list): >> ? ?def __eq__(self, other): >> ? ? ? ?if not isinstance(other, UnorderedList): >> ? ? ? ? ? ?return False >> ? ? ? ?return sorted(self) == sorted(other) >> >> ? ?def __ne__(self, other): >> ? ? ? ?return not self.__eq__(other) >> >> ? Do you need more than that? >> >> Oleg. > > That's what I had in mind. > > I think it'd be useful enough to go in the standard library. Now that > there's a sample implementation, should I still try to demonstrate why I > believe it's worth adding to the stdlib and get support? > > Cheers, > > ?- Gustavo. I'm generally in favor of adding more data structures to Python, but I'm at best -0 on this. Besides being trivial to code and questionably useful, a much better implementation could be written using heap. Maybe with a better implementation I would go +0, but I'm hard pressed to see a case where this would be needed and could not be trivially written. Geremy Condra From rdmurray at bitdance.com Thu May 20 18:06:32 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 20 May 2010 12:06:32 -0400 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: References: <4BF479DF.4020102@gmail.com> Message-ID: <20100520160633.0301A1F9F81@kimball.webabinitio.net> On Thu, 20 May 2010 11:49:02 +0200, =?ISO-8859-1?Q?Giampaolo_Rodol=E0?= wrote: > 2010/5/20 John Arbash Meinel : > >>>>> a.echo() > >> Traceback (most recent call last): > >> =A0 File "", line 1, in > >> TypeError: echo() takes exactly 2 arguments (1 given) > >> > >> I bet my last 2 cents this has already been raised in past but I want > >> to give it a try and revamp the subject anyway. > >> Is there a reason why the error shouldn't be adjusted to state that > >> *1* argument is actually required instead of 2? > > > > Because you wouldn't want to have > > > > A.echo() > > > > Say that it takes 1 argument and (-1 given) ? > > I see that as a different error type: what you're doing there is > calling a method of a class which you haven't instantiated in the > first place. > Actually the error message returned in this other case is not very > clear as well: > > "unbound method echo() must be called with A instance as first > argument (got nothing instead)" > > It talks about "arguments" while no arguments are actually involved in > the problem: just a class I forgot to initialize. That second message is entirely accurate and IMO should not be changed. As Michael said, calling an unbound method is not that uncommon. The problem with fixing the first message (as you will see if you read issue 2516) is that it turns out to be non-trivial to do it right. Otherwise I think it would have been fixed already. -- R. David Murray www.bitdance.com From fuzzyman at voidspace.org.uk Thu May 20 18:14:24 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 20 May 2010 17:14:24 +0100 Subject: [Python-Dev] Unordered tuples/lists In-Reply-To: References: <201005182313.42935.me@gustavonarea.net> Message-ID: <4BF55FE0.8070707@voidspace.org.uk> On 20/05/2010 17:02, geremy condra wrote: > On Wed, May 19, 2010 at 1:56 AM, Gustavo Narea wrote: > >> Hello, Oleg. >> >> >>> class UnorderedList(list): >>> def __eq__(self, other): >>> if not isinstance(other, UnorderedList): >>> return False >>> return sorted(self) == sorted(other) >>> >>> def __ne__(self, other): >>> return not self.__eq__(other) >>> >>> Do you need more than that? >>> >>> Oleg. >>> >> That's what I had in mind. >> >> I think it'd be useful enough to go in the standard library. Now that >> there's a sample implementation, should I still try to demonstrate why I >> believe it's worth adding to the stdlib and get support? >> >> Cheers, >> >> - Gustavo. >> > I'm generally in favor of adding more data structures to Python, > but I'm at best -0 on this. Besides being trivial to code and > questionably useful, a much better implementation could be > written using heap. > > -1 from me, a trivial implementation of an uncommon use case. At this point the discussion belongs on python-ideas anyway. (The trouble with sorted is how it breaks with un-orderable types. There are at least two places now in the py3k standard library that have techniques for comparing sequences of heterogenous types - both prettyprint and unittest I believe. Perhaps a standard recipe for comparing or sorting such containers *does* belong somewhere more obvious.) All the best, Michael > Maybe with a better implementation I would go +0, but I'm > hard pressed to see a case where this would be needed and > could not be trivially written. > > Geremy Condra > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From steven.bethard at gmail.com Thu May 20 18:18:45 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Thu, 20 May 2010 09:18:45 -0700 Subject: [Python-Dev] bug or feature? fixing argparse's default help value for version actions Message-ID: Sorry I haven't had time to get around to the argparse issues. I should have time this weekend. I need a release manager call on one of the issues though. Two things I assume are fine to fix at this stage: * In the documentation, the '--version' example should either not use a shorthand, or should use the conventional '-V' * In the documentation, the difference between the argparse and optparse ways of specifying versions needs to be mentioned in the section on migrating from optparse. One thing I'm not sure about: * Right now, the default help value for the version action is None, which is pretty useless since it means that everyone has to type in the same help="show program's version number and exit" when they use the version action. Instead, the string "show program's version number and exit" should have been the default value for the help argument. To be explicit, right now everyone who adds a version to their argument parser has to write: parser.add_argument('-V', action='version', version='', help="show program's version number and exit") With the fixed default value you would only have to write: parser.add_argument('-V', action='version', version='') Can this be considered a bug, i.e. something that can be committed before the RC? Or does it need to be considered a feature, i.e. at this point can never be added to Python 2.7? Thanks, and sorry again for not having time for this until now, Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From tjreedy at udel.edu Thu May 20 18:55:13 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 20 May 2010 12:55:13 -0400 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: <20100520080207.GA16837@laurie.devork> References: <87k4qzgu49.fsf@uwakimon.sk.tsukuba.ac.jp> <20100520080207.GA16837@laurie.devork> Message-ID: On 5/20/2010 4:02 AM, Floris Bruynooghe wrote: >> TypeError: invoked as a method, echo() takes exactly 1 argument (0 given) >> >> captures the semantics, but is perhaps too verbose. > > How about: > > TypeError: bound method echo() takes exactly 1 argument (0 given) > > That way you can also have: "unbound method echo() ...". And it's as > semantically correct as the short "echo() takes ..." > > Not having looked at the code I don't know how hard it is for the code > that raises this traceback to notice if it's a bound or unbound method > tough. In 3.x, there are no unbound method objects, just functions. But that should make the difference *easier* to detect, as bound/unbound method objects were actually the same class, differing only in an attribute. I notice >>> list.append() Traceback (most recent call last): File "", line 1, in list.append() TypeError: descriptor 'append' of 'list' object needs an argument So here the message is specific to the type of the callable. From brett at python.org Thu May 20 18:56:29 2010 From: brett at python.org (Brett Cannon) Date: Thu, 20 May 2010 09:56:29 -0700 Subject: [Python-Dev] bug or feature? fixing argparse's default help value for version actions In-Reply-To: References: Message-ID: On Thu, May 20, 2010 at 09:18, Steven Bethard wrote: > Sorry I haven't had time to get around to the argparse issues. I > should have time this weekend. I need a release manager call on one of > the issues though. Two things I assume are fine to fix at this stage: > > * In the documentation, the '--version' example should either not use > a shorthand, or should use the conventional '-V' > * In the documentation, the difference between the argparse and > optparse ways of specifying versions needs to be mentioned in the > section on migrating from optparse. > > One thing I'm not sure about: > > * Right now, the default help value for the version action is None, > which is pretty useless since it means that everyone has to type in > the same help="show program's version number and exit" when they use > the version action. Instead, the string "show program's version number > and exit" should have been the default value for the help argument. To > be explicit, right now everyone who adds a version to their argument > parser has to write: > > ?parser.add_argument('-V', action='version', version='', > help="show program's version number and exit") > > With the fixed default value you would only have to write: > > ?parser.add_argument('-V', action='version', version='') > > Can this be considered a bug, i.e. something that can be committed > before the RC? Or does it need to be considered a feature, i.e. at > this point can never be added to Python 2.7? In the end it's Benjamin's call, but my vote is to make the change. The chances someone wanted None as their help message is so bloody small and this is such a good UX change that I'm +1 on making the change. -Brett From eric at trueblade.com Thu May 20 19:13:47 2010 From: eric at trueblade.com (Eric Smith) Date: Thu, 20 May 2010 13:13:47 -0400 Subject: [Python-Dev] bug or feature? fixing argparse's default help value for version actions In-Reply-To: References: Message-ID: <4BF56DCB.40207@trueblade.com> Brett Cannon wrote: > In the end it's Benjamin's call, but my vote is to make the change. > The chances someone wanted None as their help message is so bloody > small and this is such a good UX change that I'm +1 on making the > change. I completely agree. -- Eric. From martin at v.loewis.de Thu May 20 19:35:03 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 20 May 2010 19:35:03 +0200 Subject: [Python-Dev] Unordered tuples/lists In-Reply-To: References: <201005182313.42935.me@gustavonarea.net> Message-ID: <4BF572C7.9040703@v.loewis.de> > I think it'd be useful enough to go in the standard library. Now that > there's a sample implementation, should I still try to demonstrate why I > believe it's worth adding to the stdlib and get support? Most definitely. Just in case it isn't clear: nobody else seems to think this is useful (let alone useful enough to go into the standard library). In addition, it's trivial to implement, more reason not to add it. Regards, Martin From me at gustavonarea.net Thu May 20 21:26:36 2010 From: me at gustavonarea.net (Gustavo Narea) Date: Thu, 20 May 2010 20:26:36 +0100 Subject: [Python-Dev] Unordered tuples/lists In-Reply-To: <4BF572C7.9040703@v.loewis.de> References: <201005182313.42935.me@gustavonarea.net> <4BF572C7.9040703@v.loewis.de> Message-ID: <201005202026.36640.me@gustavonarea.net> Martin said: > Most definitely. Just in case it isn't clear: nobody else seems to think > this is useful (let alone useful enough to go into the standard > library). In addition, it's trivial to implement, more reason not to add > it. Yeah, fair enough. Thanks for your responses! :) -- Gustavo Narea . | Tech blog: =Gustavo/(+blog)/tech ~ About me: =Gustavo/about | From solipsis at pitrou.net Thu May 20 21:32:53 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 20 May 2010 21:32:53 +0200 Subject: [Python-Dev] Adding a new C API function in 2.6 Message-ID: <20100520213253.48f7dfdd@pitrou.net> Hello, I would like to check that it's possible to a new C API function in the 2.6 branch, on the basis that it would help solve what seems to be reported as a security problem by several vendors (including Linux distributions) -- see http://bugs.python.org/issue5753 for a thorough discussion. The change is rather minimal at the code level; it adds a new function PySys_SetArgvEx which has an additional flag telling it whether to update sys.path or not. The existing PySys_SetArgv function unconditionally updates sys.path, which can allow shadowing of stdlib or third-party library modules by an attacker. Thank you Antoine. From guido at python.org Thu May 20 21:53:31 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 20 May 2010 12:53:31 -0700 Subject: [Python-Dev] Adding a new C API function in 2.6 In-Reply-To: <20100520213253.48f7dfdd@pitrou.net> References: <20100520213253.48f7dfdd@pitrou.net> Message-ID: Sounds good to me, since this is (a) a security fix that will make some vendors happy, and (b) only a C-level API. I expect that some apps embedding Python will use this API unconditionally and this break with earlier Python versions; this could be intentional because of the vulnerability (else why would they change their code to call the new API), or they can use an #if to check for a version >= 2.6.6. --Guido On Thu, May 20, 2010 at 12:32 PM, Antoine Pitrou wrote: > > Hello, > > I would like to check that it's possible to a new C API function in the > 2.6 branch, on the basis that it would help solve what seems to be > reported as a security problem by several vendors (including Linux > distributions) -- see http://bugs.python.org/issue5753 for a thorough > discussion. > > The change is rather minimal at the code level; it adds a new function > PySys_SetArgvEx which has an additional flag telling it whether to > update sys.path or not. The existing PySys_SetArgv function > unconditionally updates sys.path, which can allow shadowing of stdlib > or third-party library modules by an attacker. > > Thank you > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From me at gustavonarea.net Thu May 20 22:59:00 2010 From: me at gustavonarea.net (Gustavo Narea) Date: Thu, 20 May 2010 21:59:00 +0100 Subject: [Python-Dev] Incorrect length of collections.Counter objects / Multiplicity function In-Reply-To: <201005182300.21079.me@gustavonarea.net> References: <201005182300.21079.me@gustavonarea.net> Message-ID: <201005202159.01277.me@gustavonarea.net> Anyone? Gustavo said: > Hello, everyone. > > I've checked the new collections.Counter class and I think I've found a bug: > > >>> from collections import Counter > > >>> c1 = Counter([1, 2, 1, 3, 2]) > > >>> c2 = Counter([1, 1, 2, 2, 3]) > > >>> c3 = Counter([1, 1, 2, 3]) > > >>> c1 == c2 and c3 not in (c1, c2) > > > > True > > > > >>> # Perfect, so far. But... There's always a "but": > > ... > > > > >>> len(c1) > > > > 3 > > The length of a Counter is the amount of unique elements. But the length > must be the cardinality, and the cardinality of a multiset is the total > number of elements (including duplicates) [1] [2]. The source code > mentions that the recipe on ActiveState [3] was one of the references, but > that recipe has this right. > > Also, why is it indexed? The indexes of a multiset call to mind the > position of its elements, but there's no such thing in sets. I think this > is inconsistent with the built-in set. I would have implemented the > multiplicity function as a method instead of the indexes: > c1.get_multiplicity(element) > # instead of > c1[element] > > Is this the intended behavior? If so, I'd like to propose a proper multiset > implementation for the standard library (preferably called "Multiset"; > should I create a PEP?). If not, I can write a patch to fix it, although > I'm afraid it'd be a backwards incompatible change. > > Cheers, > > [1] http://en.wikipedia.org/wiki/Multiset#Overview > [2] http://preview.tinyurl.com/smalltalk-bag > [3] http://code.activestate.com/recipes/259174/ -- Gustavo Narea . | Tech blog: =Gustavo/(+blog)/tech ~ About me: =Gustavo/about | From facundobatista at gmail.com Thu May 20 23:11:51 2010 From: facundobatista at gmail.com (Facundo Batista) Date: Thu, 20 May 2010 18:11:51 -0300 Subject: [Python-Dev] Incorrect length of collections.Counter objects / Multiplicity function In-Reply-To: <201005202159.01277.me@gustavonarea.net> References: <201005182300.21079.me@gustavonarea.net> <201005202159.01277.me@gustavonarea.net> Message-ID: On Thu, May 20, 2010 at 5:59 PM, Gustavo Narea wrote: > Anyone? The best place to post a bug is the bug tracker [0]: you'll surely receive proper attention there. Regards, [0] http://bugs.python.org/ -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From dickinsm at gmail.com Thu May 20 23:18:56 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 20 May 2010 22:18:56 +0100 Subject: [Python-Dev] Incorrect length of collections.Counter objects / Multiplicity function In-Reply-To: <201005182300.21079.me@gustavonarea.net> References: <201005182300.21079.me@gustavonarea.net> Message-ID: On Tue, May 18, 2010 at 11:00 PM, Gustavo Narea wrote: > I've checked the new collections.Counter class and I think I've found a bug: > >> >>> from collections import Counter >> >>> c1 = Counter([1, 2, 1, 3, 2]) >> >>> c2 = Counter([1, 1, 2, 2, 3]) >> >>> c3 = Counter([1, 1, 2, 3]) >> >>> c1 == c2 and c3 not in (c1, c2) >> True >> >>> # Perfect, so far. But... There's always a "but": >> ... >> >>> len(c1) >> 3 This is the intended behaviour; it also agrees with what you get when you iterate over a Counter object: >>> list(c1) [1, 2, 3] As I understand it, there are other uses for Counter objects besides treating them as multisets; I think the choices for len() and iter() reflected those other uses. > Is this the intended behavior? If so, I'd like to propose a proper multiset > implementation for the standard library (preferably called "Multiset"; should > I create a PEP?). Feel free! The proposal should probably go to python-list or python-ideas rather than here, though. See also this recent thread on python-list, and in particular the messages from Raymond Hettinger in that thread: http://mail.python.org/pipermail/python-list/2010-March/thread.html Mark From dickinsm at gmail.com Thu May 20 23:20:40 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 20 May 2010 22:20:40 +0100 Subject: [Python-Dev] Incorrect length of collections.Counter objects / Multiplicity function In-Reply-To: References: <201005182300.21079.me@gustavonarea.net> Message-ID: On Thu, May 20, 2010 at 10:18 PM, Mark Dickinson wrote: > See also this recent thread on python-list, and in particular the messages > from Raymond Hettinger in that thread: > > http://mail.python.org/pipermail/python-list/2010-March/thread.html Sorry, bad thread link. Try: http://mail.python.org/pipermail/python-list/2010-March/1238730.html instead. From barry at python.org Thu May 20 23:52:25 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 20 May 2010 17:52:25 -0400 Subject: [Python-Dev] Adding a new C API function in 2.6 In-Reply-To: References: <20100520213253.48f7dfdd@pitrou.net> Message-ID: <20100520175225.54611105@heresy> On May 20, 2010, at 12:53 PM, Guido van Rossum wrote: >Sounds good to me, since this is (a) a security fix that will make >some vendors happy, and (b) only a C-level API. I expect that some >apps embedding Python will use this API unconditionally and this break >with earlier Python versions; this could be intentional because of the >vulnerability (else why would they change their code to call the new >API), or they can use an #if to check for a version >= 2.6.6. I concur, as long as the new API availability is clearly spelled out in the release notes, NEWS, and C API documentation. Maybe even provide an example #ifdef in the latter. Should we start thinking about releasing 2.6.6 soonish? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From tjreedy at udel.edu Fri May 21 00:01:52 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 20 May 2010 18:01:52 -0400 Subject: [Python-Dev] Adding a new C API function in 2.6 In-Reply-To: <20100520175225.54611105@heresy> References: <20100520213253.48f7dfdd@pitrou.net> <20100520175225.54611105@heresy> Message-ID: On 5/20/2010 5:52 PM, Barry Warsaw wrote: > On May 20, 2010, at 12:53 PM, Guido van Rossum wrote: > >> Sounds good to me, since this is (a) a security fix that will make >> some vendors happy, and (b) only a C-level API. I expect that some >> apps embedding Python will use this API unconditionally and this break >> with earlier Python versions; this could be intentional because of the >> vulnerability (else why would they change their code to call the new >> API), or they can use an #if to check for a version>= 2.6.6. > > I concur, as long as the new API availability is clearly spelled out in the > release notes, NEWS, and C API documentation. Maybe even provide an example > #ifdef in the latter. > > Should we start thinking about releasing 2.6.6 soonish? By tradition, it should come out soon after 2.7 and be the last bugfix (except for security patches). From barry at python.org Fri May 21 00:11:42 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 20 May 2010 18:11:42 -0400 Subject: [Python-Dev] Adding a new C API function in 2.6 In-Reply-To: References: <20100520213253.48f7dfdd@pitrou.net> <20100520175225.54611105@heresy> Message-ID: <20100520181142.75e3c51b@heresy> On May 20, 2010, at 06:01 PM, Terry Reedy wrote: >On 5/20/2010 5:52 PM, Barry Warsaw wrote: >> On May 20, 2010, at 12:53 PM, Guido van Rossum wrote: >> >>> Sounds good to me, since this is (a) a security fix that will make >>> some vendors happy, and (b) only a C-level API. I expect that some >>> apps embedding Python will use this API unconditionally and this break >>> with earlier Python versions; this could be intentional because of the >>> vulnerability (else why would they change their code to call the new >>> API), or they can use an #if to check for a version>= 2.6.6. >> >> I concur, as long as the new API availability is clearly spelled out in the >> release notes, NEWS, and C API documentation. Maybe even provide an example >> #ifdef in the latter. >> >> Should we start thinking about releasing 2.6.6 soonish? > >By tradition, it should come out soon after 2.7 and be the last bugfix >(except for security patches). I guess what I mean is, should we have (at least) one more point release before the post-2.7 last-bug-fix-release? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From martin at v.loewis.de Fri May 21 01:05:02 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 21 May 2010 01:05:02 +0200 Subject: [Python-Dev] Adding a new C API function in 2.6 In-Reply-To: <20100520181142.75e3c51b@heresy> References: <20100520213253.48f7dfdd@pitrou.net> <20100520175225.54611105@heresy> <20100520181142.75e3c51b@heresy> Message-ID: <4BF5C01E.5000406@v.loewis.de> >>> Should we start thinking about releasing 2.6.6 soonish? >> >> By tradition, it should come out soon after 2.7 and be the last bugfix >> (except for security patches). > > I guess what I mean is, should we have (at least) one more point release > before the post-2.7 last-bug-fix-release? Because it's a security fix? No. This issue has been sitting in the tracker for more than a year now, so it's not really relevant (IMO) whether the bug fix arrives two weeks earlier. Regards, Martin From benjamin at python.org Fri May 21 01:14:24 2010 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 20 May 2010 18:14:24 -0500 Subject: [Python-Dev] bug or feature? fixing argparse's default help value for version actions In-Reply-To: References: Message-ID: 2010/5/20 Steven Bethard : > Sorry I haven't had time to get around to the argparse issues. I > should have time this weekend. I need a release manager call on one of > the issues though. Two things I assume are fine to fix at this stage: > > * In the documentation, the '--version' example should either not use > a shorthand, or should use the conventional '-V' > * In the documentation, the difference between the argparse and > optparse ways of specifying versions needs to be mentioned in the > section on migrating from optparse. Documentation changes are always okay. > Can this be considered a bug, i.e. something that can be committed > before the RC? Or does it need to be considered a feature, i.e. at > this point can never be added to Python 2.7? That sounds fine to me. > > Thanks, and sorry again for not having time for this until now, -- Regards, Benjamin From greg.ewing at canterbury.ac.nz Fri May 21 02:53:16 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 21 May 2010 12:53:16 +1200 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: <87iq6jgglt.fsf@benfinney.id.au> References: <4BF479DF.4020102@gmail.com> <4BF4E41E.3050800@canterbury.ac.nz> <87iq6jgglt.fsf@benfinney.id.au> Message-ID: <4BF5D97C.6080607@canterbury.ac.nz> Ben Finney wrote: >>Something like "1 argument in addition to 'self'" would be reasonably >>clear and would cover both situations. > > Except that there's nothing special to the syntax or parser about the > name ?self?. That's true, but the use of the word 'self' here isn't meant to refer to the name of a parameter. The message is aimed at the caller of the function, who doesn't necessarily know or care what the parameter is actually called. The important thing is that it represents the object the method is being called for, and 'self' is the traditional term used when talking about that. I can't think of anything that would be more accurate without being excessively verbose or pedantic. -- Greg From greg.ewing at canterbury.ac.nz Fri May 21 03:23:15 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 21 May 2010 13:23:15 +1200 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: <20100520080207.GA16837@laurie.devork> References: <87k4qzgu49.fsf@uwakimon.sk.tsukuba.ac.jp> <20100520080207.GA16837@laurie.devork> Message-ID: <4BF5E083.6000201@canterbury.ac.nz> Floris Bruynooghe wrote: > Not having looked at the code I don't know how hard it is for the code > that raises this traceback to notice if it's a bound or unbound method > tough. The way things currently work, it would be quite difficult. The exception is raised when attempting to call the function underlying the bound method, at which point it can't be distinguished from any other way of calling it. However, it could be made to raise a special subclass of TypeError that the bound method wrapper could catch and replace with its own exception. -- Greg From greg.ewing at canterbury.ac.nz Fri May 21 03:31:33 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 21 May 2010 13:31:33 +1200 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: References: <4BF479DF.4020102@gmail.com> Message-ID: <4BF5E275.6000902@canterbury.ac.nz> Giampaolo Rodol? wrote: > "unbound method echo() must be called with A instance as first > argument (got nothing instead)" > > It talks about "arguments" while no arguments are actually involved in > the problem: just a class I forgot to initialize. It's hard to see how this could be improved. If you had been intending to call an unbound method, the comment about arguments would have been relevant -- you were supposed to supply one but didn't. Python is unable to read your mind and figure out that you really meant something else altogether. BTW, unbound methods no longer exist in Py3, so you would get the standard message about incorrect number of arguments instead. Not sure whether that's better or worse in this situation. -- Greg From cs at zip.com.au Fri May 21 03:46:38 2010 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 21 May 2010 11:46:38 +1000 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: <87iq6jgglt.fsf@benfinney.id.au> References: <87iq6jgglt.fsf@benfinney.id.au> Message-ID: <20100521014638.GA28209@cskk.homeip.net> On 20May2010 17:46, Ben Finney wrote: | Would it help if the traceback showed the ?repr()? of each of the | arguments received? That way it would be much clearer when the instance | was received as the first argument. I've occasionally passed large or deep dicts etc to functions and foolishly printed their repr in debug statements. They can be... wordy. Maybe a cropped repr, or the .__class__ attribute? -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ Every technical corrigendum is met by an equally troublesome new defect report. - Norman Diamond From ben+python at benfinney.id.au Fri May 21 04:24:37 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 21 May 2010 12:24:37 +1000 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method References: <87iq6jgglt.fsf@benfinney.id.au> <20100521014638.GA28209@cskk.homeip.net> Message-ID: <87aarugffe.fsf@benfinney.id.au> Cameron Simpson writes: > On 20May2010 17:46, Ben Finney wrote: > | Would it help if the traceback showed the ?repr()? of each of the > | arguments received? That way it would be much clearer when the instance > | was received as the first argument. > > I've occasionally passed large or deep dicts etc to functions and > foolishly printed their repr in debug statements. They can be... > wordy. Maybe a cropped repr, or the .__class__ attribute? Okay. My main point is that I'm offering this as a way of avoiding a special-case message for method calls versus non-method calls. In Python, all function calls are essentially the same, and I think it's too complicated for the message to differ depending on the syntax used. Emitting the arguments in some form, or even the types of each argument, in the message would make it clearer without a special case. -- \ ?We now have access to so much information that we can find | `\ support for any prejudice or opinion.? ?David Suzuki, 2008-06-27 | _o__) | Ben Finney From brian at sweetapp.com Fri May 21 05:00:35 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Fri, 21 May 2010 13:00:35 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement Message-ID: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> The PEP is here: http://www.python.org/dev/peps/pep-3148/ I think the PEP is ready for pronouncement, and the code is pretty much ready for submission into py3k (I will have to make some minor changes in the patch like changing the copyright assignment): http://code.google.com/p/pythonfutures/source/browse/#svn/branches/ feedback/python3/futures%3Fstate%3Dclosed The tests are here and pass on W2K, Mac OS X and Linux: http://code.google.com/p/pythonfutures/source/browse/branches/feedback/python3/test_futures.py The docs (which also need some minor changes) are here: http://code.google.com/p/pythonfutures/source/browse/branches/feedback/docs/index.rst Cheers, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri May 21 05:54:56 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 21 May 2010 13:54:56 +1000 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: <4BF5D97C.6080607@canterbury.ac.nz> References: <87iq6jgglt.fsf@benfinney.id.au> <4BF5D97C.6080607@canterbury.ac.nz> Message-ID: <201005211354.57237.steve@pearwood.info> On Fri, 21 May 2010 10:53:16 am Greg Ewing wrote: > Ben Finney wrote: > >>Something like "1 argument in addition to 'self'" would be > >> reasonably clear and would cover both situations. > > > > Except that there's nothing special to the syntax or parser about > > the name ?self?. > > That's true, but the use of the word 'self' here isn't meant > to refer to the name of a parameter. The message is aimed at > the caller of the function, who doesn't necessarily know or > care what the parameter is actually called. And who doesn't necessarily know that: > The important thing is that it represents the object the method > is being called for, and 'self' is the traditional term used > when talking about that. Referring to self in any such error message is going to confuse newbies, and occasional Python programmers who use it without learning about object oriented code. E.g. sys admins who use Python as a scripting language without ever writing their own classes. Error messages should be aimed at the least capable users (within reason!) rather than the most capable, since the most capable are: (1) less likely to make the error in the first place; (2) more able to interpret the error message correctly; (3) more likely to understand if the error relates to something "under the hood"; and (4) better able to debug the error even in the absence of a useful error message. Given this, it is better for the error message to relate to what the "average" user most likely has in his source code (i.e. a bound method) rather than what the better-than-average user occasionally uses (an unbound method). > I can't think of anything that would > be more accurate without being excessively verbose or pedantic. In Python 3.1, strings seem to behave more sensibly IMO: >>> 'abc'.find() Traceback (most recent call last): File "", line 1, in TypeError: find() takes at least 1 argument (0 given) That looks right to me. I'm calling a *method* with zero arguments inside the parentheses when I need to call it with one. If I call it as an unbound method (yes, I know it's actually a function) I get something more confusing: >>> str.find('abc') Traceback (most recent call last): File "", line 1, in TypeError: find() takes at least 1 argument (0 given) While calling unbound methods isn't exactly rare, it is a more advanced thing to do than calling bound methods. If we're stuck with confusing messages, better to have it occur for the users who are more likely to be able to interpret it, that is, those advanced enough to use unbound methods rather than "ordinary" users. Note that str avoids the "-1 argument" trap: >>> str.find() Traceback (most recent call last): File "", line 1, in TypeError: descriptor 'find' of 'str' object needs an argument Classes behave in the opposite manner: the error message is technically correct for the less common, more advanced case, but misleading for the more common, simple case: >>> class C: ... def method(self, x): ... return x+1 ... >>> C.method() Traceback (most recent call last): File "", line 1, in TypeError: method() takes exactly 2 positional arguments (0 given) This is exactly right, since C.method is a function object which takes two arguments, self and x. Because this is a function, we can't change this (not that we want to!) without impacting the errors you get when calling functions not associated with a class. >>> C().method() Traceback (most recent call last): File "", line 1, in TypeError: method() takes exactly 2 positional arguments (1 given) This is misleading, since C().method is a bound method which takes one argument, x, not two. I find myself wishing that Python distinguished between ArgumentError and other TypeErrors, so that the method wrapper of bound methods could simply catch ArgumentError and subtract 1 from each argument count. -- Steven D'Aprano From yaniv at aknin.name Fri May 21 07:18:43 2010 From: yaniv at aknin.name (Yaniv Aknin) Date: Fri, 21 May 2010 15:18:43 +1000 Subject: [Python-Dev] Documenting [C]Python's Internals In-Reply-To: References: Message-ID: > > This link has all post concatenated together in reverse order of how they >> should be read. The tags link returns the same page. Does your blog software >> allow you to make a master post and update with new links as available? > > Ugh, either it doesn't or I couldn't find the feature (I'm using wordpress.com, if someone has advice, let me know). I can clumsily suggest scrolling from the end. Also see below about reworking this into a single multi-chapter document with coherent form. I would if I were qualified, but I an mot. One way to get people to help > with details is to publish mistakes. This happens all the time on > python-list ;-). Pre-review would be nice though. > I don't mind so much the 'humiliation' of published mistakes, but since I want this to be perceived as reference grade material, I prefer pre-review. Yesterday my first mistake was found (ugh), I published an 'Errata Policy' and will stick to it from now on (see the blog itself for details of the mistake). Thankfully, I've been approached already about pre-review, we'll see how this develops (this doesn't mean other people can't also offer themselves, six eyeballs are better than four). People have asked for an internals-doc since I started over a decade ago. A > coherent CPython3.2 Internals would be nice to have with the 3.2 release > next December or so, whether or not it was made 'official'. > I'm targeting py3k anyway, and while I expect a bug lull in my writing between early June and early September, I think December is a realistic date for me to have good coverage CPython 3.2's core and rework the content into a more reference-material-ish form. That said, working things into reference-material form could be significant work, so if python-dev doesn't show interest in this I think the blog posts are good enough. Other people, this is your queue to chime in and state your opinion about this appearing on python.org somewhere. Cheers! - Yaniv -------------- next part -------------- An HTML attachment was scrubbed... URL: From list at qtrac.plus.com Fri May 21 13:29:58 2010 From: list at qtrac.plus.com (Mark Summerfield) Date: Fri, 21 May 2010 12:29:58 +0100 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> Message-ID: <201005211229.58970.list@qtrac.plus.com> On 2010-05-21, Brian Quinlan wrote: > The PEP is here: > http://www.python.org/dev/peps/pep-3148/ [snip] Hi Brian, Could I suggest a small subtle changing in naming: replace "executor" with "executer"? I guess this suggestion is doomed though since Java uses executor:-( I'd also be tempted to rename "submit()" to "apply()" in view of Python's history. Also, maybe change "done()" to "finished()" since the function returns True if the call was cancelled (so the job can't have been "done"), as well as if the call was finished. Actually, having read further, maybe the best name would be "completed()" since that's a term used throughout. Perhaps call the "not_finished" set "pending" since presumably these are still in progress? (My understanding is that if they were cancelled or finished they'd be in the "finished" set. I'd also rename "finished" to "completed" if you have a "completed()" method.) I think FIRST_COMPLETED is misleading since it implies (to me anyway) the first one passed. How about ONE_COMPLETED; and similarly ONE_EXCEPTION? I think it would be helpful to clarify whether the timout value (which you specify as being in seconds) can meaningfully accept a float, e.g., 0.5? Anyway, it looks like it will be a really nice addition to the standard library:-) -- Mark Summerfield, Qtrac Ltd, www.qtrac.eu C++, Python, Qt, PyQt - training and consultancy "C++ GUI Programming with Qt 4" - ISBN 0132354160 From john.arbash.meinel at gmail.com Fri May 21 13:44:35 2010 From: john.arbash.meinel at gmail.com (John Arbash Meinel) Date: Fri, 21 May 2010 06:44:35 -0500 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> Message-ID: <4BF67223.5080304@gmail.com> Brian Quinlan wrote: > The PEP is here: > http://www.python.org/dev/peps/pep-3148/ > > I think the PEP is ready for pronouncement, and the code is pretty much > ready for submission into py3k (I will have to make some minor changes > in the patch like changing the copyright assignment): > http://code.google.com/p/pythonfutures/source/browse/#svn/branches/feedback/python3/futures%3Fstate%3Dclosed > Your example here: for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)): print('%d is prime: %s' % (number, is_prime)) Overwrites the 'is_prime' function with the return value of the function. Probably better to use a different variable name. John =:-> From brian at sweetapp.com Fri May 21 14:13:11 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Fri, 21 May 2010 22:13:11 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <201005211229.58970.list@qtrac.plus.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <201005211229.58970.list@qtrac.plus.com> Message-ID: <785B69C5-9DBE-4049-A1F5-D36BA5EACCFC@sweetapp.com> Hey Mark, This really isn't the time to propose changes. The PEP has been discussed extensively on stdlib-sig and python-dev. On May 21, 2010, at 9:29 PM, Mark Summerfield wrote: > On 2010-05-21, Brian Quinlan wrote: >> The PEP is here: >> http://www.python.org/dev/peps/pep-3148/ > [snip] > > Hi Brian, > > Could I suggest a small subtle changing in naming: replace "executor" > with "executer"? I guess this suggestion is doomed though since Java > uses executor:-( > > I'd also be tempted to rename "submit()" to "apply()" in view of > Python's history. > > Also, maybe change "done()" to "finished()" since the function returns > True if the call was cancelled (so the job can't have been "done"), as > well as if the call was finished. Actually, having read further, maybe > the best name would be "completed()" since that's a term used > throughout. > > Perhaps call the "not_finished" set "pending" since presumably these > are > still in progress? (My understanding is that if they were cancelled or > finished they'd be in the "finished" set. I'd also rename "finished" > to > "completed" if you have a "completed()" method.) > > I think FIRST_COMPLETED is misleading since it implies (to me anyway) > the first one passed. How about ONE_COMPLETED; and similarly > ONE_EXCEPTION? > > I think it would be helpful to clarify whether the timout value (which > you specify as being in seconds) can meaningfully accept a float, > e.g., > 0.5? I've updated the docs to clarify that float args are acceptable. Cheers, Brian > Anyway, it looks like it will be a really nice addition to the > standard > library:-) > > -- > Mark Summerfield, Qtrac Ltd, www.qtrac.eu > C++, Python, Qt, PyQt - training and consultancy > "C++ GUI Programming with Qt 4" - ISBN 0132354160 From brian at sweetapp.com Fri May 21 14:14:11 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Fri, 21 May 2010 22:14:11 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BF67223.5080304@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF67223.5080304@gmail.com> Message-ID: On May 21, 2010, at 9:44 PM, John Arbash Meinel wrote: > Brian Quinlan wrote: >> The PEP is here: >> http://www.python.org/dev/peps/pep-3148/ >> >> I think the PEP is ready for pronouncement, and the code is pretty >> much >> ready for submission into py3k (I will have to make some minor >> changes >> in the patch like changing the copyright assignment): >> http://code.google.com/p/pythonfutures/source/browse/#svn/branches/ >> feedback/python3/futures%3Fstate%3Dclosed >> > > Your example here: > for number, is_prime in zip(PRIMES, executor.map(is_prime, > PRIMES)): > print('%d is prime: %s' % (number, is_prime)) > > Overwrites the 'is_prime' function with the return value of the > function. Probably better to use a different variable name. Good catch. I've updated the example. Cheers, Brian From john.meinel at canonical.com Fri May 21 13:47:52 2010 From: john.meinel at canonical.com (John Arbash Meinel) Date: Fri, 21 May 2010 06:47:52 -0500 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> Message-ID: <4BF672E8.1080001@canonical.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Brian Quinlan wrote: > The PEP is here: > http://www.python.org/dev/peps/pep-3148/ > > I think the PEP is ready for pronouncement, and the code is pretty much > ready for submission into py3k (I will have to make some minor changes > in the patch like changing the copyright assignment): > http://code.google.com/p/pythonfutures/source/browse/#svn/branches/feedback/python3/futures%3Fstate%3Dclosed > > The tests are here and pass on W2K, Mac OS X and Linux: > http://code.google.com/p/pythonfutures/source/browse/branches/feedback/python3/test_futures.py > > The docs (which also need some minor changes) are here: > http://code.google.com/p/pythonfutures/source/browse/branches/feedback/docs/index.rst > > Cheers, > Brian > I also just noticed that your example uses: zip(PRIMES, executor.map(is_prime, PRIMES)) But your doc explicitly says: map(func, *iterables, timeout=None) Equivalent to map(func, *iterables) but executed asynchronously and possibly out-of-order. So it isn't safe to zip() against something that can return out of order. Which opens up a discussion about how these things should be used. Given that your other example uses a dict to get back to the original arguments, and this example uses zip() [incorrectly], it seems that the Futures object should have the arguments easily accessible. It certainly seems like a common use case that if things are going to be returned in arbitrary order, you'll want an easy way to distinguish which one you have. Having to write a dict map before each call can be done, but seems unoptimal. John =:-> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Cygwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkv2cugACgkQJdeBCYSNAAPWzACdE6KepgEmjwhCD1M4bSSVrI97 NIYAn1z5U3CJqZnBSn5XgQ/DyLvcKtbf =TKO7 -----END PGP SIGNATURE----- From murman at gmail.com Fri May 21 14:25:33 2010 From: murman at gmail.com (Michael Urman) Date: Fri, 21 May 2010 07:25:33 -0500 Subject: [Python-Dev] Reasons behind misleading TypeError message when passing the wrong number of arguments to a method In-Reply-To: <201005211354.57237.steve@pearwood.info> References: <87iq6jgglt.fsf@benfinney.id.au> <4BF5D97C.6080607@canterbury.ac.nz> <201005211354.57237.steve@pearwood.info> Message-ID: On Thu, May 20, 2010 at 22:54, Steven D'Aprano wrote: > This is misleading, since C().method is a bound method which takes one > argument, x, not two. I find myself wishing that Python distinguished > between ArgumentError and other TypeErrors, so that the method wrapper > of bound methods could simply catch ArgumentError and subtract 1 from > each argument count. But how exactly could it distinguish between various levels of call nesting? class C: def a(self, i): return self.b(i) def b(self, j): return self.c(j) def c(self, k): return self.a() What should C.a(), C().a(), and C().a(1) each yield? Does it change if c(self, k) calls C.a(self)? -- Michael Urman From brian at sweetapp.com Fri May 21 14:26:13 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Fri, 21 May 2010 22:26:13 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BF672E8.1080001@canonical.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> Message-ID: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> On May 21, 2010, at 9:47 PM, John Arbash Meinel wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Brian Quinlan wrote: >> The PEP is here: >> http://www.python.org/dev/peps/pep-3148/ >> >> I think the PEP is ready for pronouncement, and the code is pretty >> much >> ready for submission into py3k (I will have to make some minor >> changes >> in the patch like changing the copyright assignment): >> http://code.google.com/p/pythonfutures/source/browse/#svn/branches/ >> feedback/python3/futures%3Fstate%3Dclosed >> >> The tests are here and pass on W2K, Mac OS X and Linux: >> http://code.google.com/p/pythonfutures/source/browse/branches/feedback/python3/test_futures.py >> >> The docs (which also need some minor changes) are here: >> http://code.google.com/p/pythonfutures/source/browse/branches/feedback/docs/index.rst >> >> Cheers, >> Brian >> > > I also just noticed that your example uses: > zip(PRIMES, executor.map(is_prime, PRIMES)) > > But your doc explicitly says: > map(func, *iterables, timeout=None) > > Equivalent to map(func, *iterables) but executed asynchronously and > possibly out-of-order. > > So it isn't safe to zip() against something that can return out of > order. The docs don't say that the return value can be out-of-order, just that execution can be out-of-order. But I agree that the phrasing is confusing so I've changed it to: Equivalent to ``map(func, *iterables)`` but *func* is executed asynchronously and several calls to *func *may be made concurrently. > Which opens up a discussion about how these things should be used. Except that now isn't the time for that discussion. This PEP has discussed on-and-off for several months on both stdlib-sig and python- dev. If you think that storing the args (e.g. with the future) is a good idea then you can propose a patch after the PEP is integrated (if it is rejected then it probably isn't worth discussing ;-)). Cheers, Brian > Given that your other example uses a dict to get back to the original > arguments, and this example uses zip() [incorrectly], it seems that > the > Futures object should have the arguments easily accessible. It > certainly > seems like a common use case that if things are going to be returned > in > arbitrary order, you'll want an easy way to distinguish which one you > have. Having to write a dict map before each call can be done, but > seems unoptimal. > > John > =:-> > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.9 (Cygwin) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAkv2cugACgkQJdeBCYSNAAPWzACdE6KepgEmjwhCD1M4bSSVrI97 > NIYAn1z5U3CJqZnBSn5XgQ/DyLvcKtbf > =TKO7 > -----END PGP SIGNATURE----- From lie.1296 at gmail.com Fri May 21 14:42:09 2010 From: lie.1296 at gmail.com (Lie Ryan) Date: Fri, 21 May 2010 22:42:09 +1000 Subject: [Python-Dev] Documenting [C]Python's Internals In-Reply-To: References: Message-ID: On 05/21/10 15:18, Yaniv Aknin wrote: > I would if I were qualified, but I an mot. One way to get people to help >> with details is to publish mistakes. This happens all the time on >> python-list ;-). Pre-review would be nice though. >> > > I don't mind so much the 'humiliation' of published mistakes, but since I > want this to be perceived as reference grade material, I prefer pre-review. > Yesterday my first mistake was found (ugh), I published an 'Errata Policy' > and will stick to it from now on (see the blog itself for details of the > mistake). Thankfully, I've been approached already about pre-review, we'll > see how this develops (this doesn't mean other people can't also offer > themselves, six eyeballs are better than four). How about a separate blog (or wiki) for alpha-quality articles? After an article is written, it is first posted to the alpha blog, and after some time and eyeballs, moved to the original blog. Of course with an open comment system, so people can easily suggest corrections. From fuzzyman at voidspace.org.uk Fri May 21 14:52:09 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 21 May 2010 13:52:09 +0100 Subject: [Python-Dev] Documenting [C]Python's Internals In-Reply-To: References: Message-ID: <4BF681F9.1070305@voidspace.org.uk> On 21/05/2010 13:42, Lie Ryan wrote: > On 05/21/10 15:18, Yaniv Aknin wrote: > >> I would if I were qualified, but I an mot. One way to get people to help >> >>> with details is to publish mistakes. This happens all the time on >>> python-list ;-). Pre-review would be nice though. >>> >>> >> I don't mind so much the 'humiliation' of published mistakes, but since I >> want this to be perceived as reference grade material, I prefer pre-review. >> Yesterday my first mistake was found (ugh), I published an 'Errata Policy' >> and will stick to it from now on (see the blog itself for details of the >> mistake). Thankfully, I've been approached already about pre-review, we'll >> see how this develops (this doesn't mean other people can't also offer >> themselves, six eyeballs are better than four). >> > How about a separate blog (or wiki) for alpha-quality articles? After an > article is written, it is first posted to the alpha blog, and after some > time and eyeballs, moved to the original blog. Of course with an open > comment system, so people can easily suggest corrections. > Separate blog is confusing I think - you then duplicate your content and people are just as likely to be referred to the "alpha quality" version as the final version. Just publish and improve the articles based on feedback - I think your current approach with an established errata policy is well beyond what most people do or expect. When you have established the sort of coverage of the topic you are aiming for you can then take your blog articles, along with all feedback, and turn them into documentation. All the best, Michael > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From loitiere at gmail.com Fri May 21 17:25:45 2010 From: loitiere at gmail.com (=?utf-8?Q?Yannick_Loiti=C3=A8re?=) Date: Fri, 21 May 2010 17:25:45 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement Message-ID: <66D23619-7D57-4B87-AE43-DA6E0C5A0EFD@gmail.com> The description of Future.add_done_callback() does not specify whether the callbacks are done in any particular order (such as the order they were added) or not. I'm guessing they are not, but believe it should be documented either way. -- Yannick Loiti?re From status at bugs.python.org Fri May 21 18:07:59 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 21 May 2010 18:07:59 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100521160759.4DB6C78139@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-05-14 - 2010-05-21) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2693 open (+46) / 17908 closed (+24) / 20601 total (+70) Open issues with patches: 1098 Average duration of open issues: 719 days. Median duration of open issues: 497 days. Open Issues Breakdown open 2672 (+46) languishing 12 ( +0) pending 8 ( +0) Issues Created Or Reopened (73) _______________________________ heapq change breaking compatibility 2010-05-17 CLOSED http://bugs.python.org/issue3051 reopened exarkun patch subprocess.Popen.__del__ causes AttributeError (os module == N 2010-05-14 CLOSED http://bugs.python.org/issue5099 reopened haypo patch, needs review Buggy _strerror in asyncore 2010-05-18 CLOSED http://bugs.python.org/issue8573 reopened merwok easy multiprocessing needs option to eschew fork() under Linux 2010-05-14 http://bugs.python.org/issue8713 created brandon-rhodes Delayed signals in the REPL on OpenBSD (possibly libpthread re 2010-05-14 http://bugs.python.org/issue8714 created skrah Create PyUnicode_EncodeFSDefault() function 2010-05-14 CLOSED http://bugs.python.org/issue8715 created haypo patch test_tk fails on OS X if run from buildbot slave daemon -- cra 2010-05-14 http://bugs.python.org/issue8716 created janssen patch Subprocess.py broken in trunk 2010-05-14 CLOSED http://bugs.python.org/issue8717 created pakal patch subprocess: NameError: name 'WaitForSingleObject' is not def 2010-05-14 CLOSED http://bugs.python.org/issue8718 created srid buildbot: segfault on FreeBSD (signal 11) 2010-05-14 http://bugs.python.org/issue8719 created haypo undo findsource regression/change 2010-05-14 http://bugs.python.org/issue8720 created hpk patch urlparse.urlsplit regression in 2.7 2010-05-15 CLOSED http://bugs.python.org/issue8721 created srid Documentation for __getattr__ 2010-05-15 http://bugs.python.org/issue8722 created Paul.Davis IDLE won't start import os error 2010-05-15 CLOSED http://bugs.python.org/issue8723 created gromij bind_and_activate parameter is missed from directive 2010-05-15 http://bugs.python.org/issue8724 created naoki Python3: use ASCII for the file system encoding on initfsencod 2010-05-15 http://bugs.python.org/issue8725 created haypo patch test_capi failure 2010-05-15 CLOSED http://bugs.python.org/issue8726 created pitrou test_import failure 2010-05-15 CLOSED http://bugs.python.org/issue8727 created pitrou 2.7 regression in httplib.py: AttributeError: 'NoneType' objec 2010-05-15 http://bugs.python.org/issue8728 created srid The Mapping ABC's __eq__ method should return NotImplemented i 2010-05-15 http://bugs.python.org/issue8729 created stutzbach patch Spurious test failure in distutils 2010-05-16 http://bugs.python.org/issue8730 created ronaldoussoren BeOS for 2.7 - add to unsupported 2010-05-16 http://bugs.python.org/issue8731 created rpetrov patch Should urrllib2.urlopen send an Accept-Encoding header? 2010-05-16 http://bugs.python.org/issue8732 created dabrahams list type and UserList do not call super in __init__ and there 2010-05-16 http://bugs.python.org/issue8733 created alonho msvcrt get_osfhandle crash on bad FD 2010-05-16 http://bugs.python.org/issue8734 created pakal patch optparse: parse_args(values=...) does not set up default value 2010-05-16 http://bugs.python.org/issue8735 created john at nmt.edu *= is undocumented for lists 2010-05-16 CLOSED http://bugs.python.org/issue8736 created stutzbach easy ssl.RAND_add() should only accept bytes (not str) 2010-05-16 CLOSED http://bugs.python.org/issue8737 created haypo patch cPickle dumps(tuple) != dumps(loads(dumps(tuple))) 2010-05-17 http://bugs.python.org/issue8738 created Alberto.Planas.Dom??nguez Update to smtpd.py to RFC 5321 2010-05-17 http://bugs.python.org/issue8739 created alfmel patch infinite recursion with setfilesystemencoding and pdb 2010-05-17 CLOSED http://bugs.python.org/issue8740 created ccomb 2.7 regression in tarfile: IOError: link could not be created 2010-05-17 http://bugs.python.org/issue8741 created srid broken asdl link in Parser/asdl.py 2010-05-17 CLOSED http://bugs.python.org/issue8742 created pyfex set() operators don't work with collections.Set instances 2010-05-17 http://bugs.python.org/issue8743 created stutzbach Maybe typo in doc 2010-05-18 CLOSED http://bugs.python.org/issue8744 created naoki zipimport is a bit slow 2010-05-18 http://bugs.python.org/issue8745 created Goplat patch *chflags detection broken on FreeBSD 9-CURRENT 2010-05-18 http://bugs.python.org/issue8746 created yaneurabeya Autoconf tests in python not portably correct 2010-05-18 http://bugs.python.org/issue8747 created yaneurabeya integer-to-complex comparisons give incorrect results 2010-05-18 http://bugs.python.org/issue8748 created mark.dickinson patch Cruft in object.c: PyObject_GenericGetAttr 2010-05-18 http://bugs.python.org/issue8749 created Yaniv.Aknin Many of MutableSet's methods assume that the other parameter i 2010-05-18 http://bugs.python.org/issue8750 created stutzbach patch, needs review Threading and KeyError: 51 2010-05-18 CLOSED http://bugs.python.org/issue8751 created Vignesh.K set_contains, etc. should check for collections.Set in additio 2010-05-18 CLOSED http://bugs.python.org/issue8752 created stutzbach Py_ReprEnter and Py_ReprLeave are undocumented 2010-05-18 http://bugs.python.org/issue8753 created stutzbach ImportError: quote bad module name in message 2010-05-18 http://bugs.python.org/issue8754 created tjreedy easy idle crash UnicodeDecodeError utf-8 codec 2010-05-18 http://bugs.python.org/issue8755 created andyharrington Multiprocessing and UUID bug on Mac OSX 2010-05-18 CLOSED http://bugs.python.org/issue8756 created Gavin.Roy Race condition when checking for set in set 2010-05-18 http://bugs.python.org/issue8757 created stutzbach BUILD_LIST followed by BINARY_SUBSCR can be optimized to a BUI 2010-05-18 http://bugs.python.org/issue8758 created alex 2.7: wrong user site directory on Linux 2010-05-18 CLOSED http://bugs.python.org/issue8759 created srid Python 2.6.5 fails to build on AIX 5.3 2010-05-18 CLOSED http://bugs.python.org/issue8760 created oirraza PyUnicode_CompareWithASCIIString name is not mangled (UCS2, UC 2010-05-18 http://bugs.python.org/issue8761 created haypo patch default value in constructor not unique across objects 2010-05-19 CLOSED http://bugs.python.org/issue8762 created bolt py3K bdist_msi wrongly installs itself in ALL python versions 2010-05-19 http://bugs.python.org/issue8763 created pakal logging RotatingFileHandler cause too long to fininsh 2010-05-19 CLOSED http://bugs.python.org/issue8764 created davy.zhang Tests unwillingly writing unicocde to raw streams 2010-05-19 http://bugs.python.org/issue8765 created pakal patch Segmentation fault with empty "encodings" subdirectory of dire 2010-05-19 CLOSED http://bugs.python.org/issue8766 created Arfrever patch Configure: Cannot disable unicode 2010-05-19 CLOSED http://bugs.python.org/issue8767 created redcomet The checkempty_symmetric_difference test is never called 2010-05-19 http://bugs.python.org/issue8768 created stutzbach patch Straightforward usage of email package fails to round-trip 2010-05-19 http://bugs.python.org/issue8769 created akuchling patch Make 'python -m sysconfig' print something useful 2010-05-19 http://bugs.python.org/issue8770 created srid Socket freezing under load issue on Mac. 2010-05-19 http://bugs.python.org/issue8771 created amcgregor sysconfig: _get_default_scheme can be made public? 2010-05-20 http://bugs.python.org/issue8772 created srid mailbox module is needlessly executable 2010-05-20 http://bugs.python.org/issue8773 created meatballhat tabnanny improperly handles non-ascii source files 2010-05-20 CLOSED http://bugs.python.org/issue8774 created meatballhat patch Use locale encoding to decode sys.argv, not the file system en 2010-05-20 http://bugs.python.org/issue8775 created haypo Bytes version of sys.argv 2010-05-20 http://bugs.python.org/issue8776 created haypo Add threading.Barrier 2010-05-20 http://bugs.python.org/issue8777 created krisvale patch, patch typo in docs for symtable.SymbolTable.has_import_star 2010-05-20 CLOSED http://bugs.python.org/issue8778 created terrence patch utf8 codec fails to parse a character 2010-05-20 CLOSED http://bugs.python.org/issue8779 created Roman.Gershman py3k: child process don't inherit stdout / stdout on Windows 2010-05-21 http://bugs.python.org/issue8780 created haypo patch 32-bit wchar_t doesn't need to be unsigned to be usable (I thi 2010-05-21 http://bugs.python.org/issue8781 created stutzbach inspect.getsource returns invalid source for non-newline-endin 2010-05-21 http://bugs.python.org/issue8782 created hpk patch Issues Now Closed (66) ______________________ Removal of stale code in pyconfig.h 904 days http://bugs.python.org/issue1512 loewis Codepage unset in msilib.init_database() 855 days http://bugs.python.org/issue1855 loewis msilib.SetProperty(msilib.PID_CODEPAGE, '1252') raises 0x65d = 850 days http://bugs.python.org/issue1884 loewis add ftp-tls support to ftplib - RFC 4217 832 days http://bugs.python.org/issue2054 giampaolo.rodola patch Make sysmodule.c compatible with Bazaar 786 days http://bugs.python.org/issue2456 barry heapq change breaking compatibility 2 days http://bugs.python.org/issue3051 r.david.murray patch Py_FatalError causes infinite loop 633 days http://bugs.python.org/issue3605 haypo needs review PKG-INFO file should differentiate between authors and maintai 628 days http://bugs.python.org/issue3686 tarek urllib2 sends Basic auth across redirects 617 days http://bugs.python.org/issue3819 kiilerix patch, easy ssl module is missing SSL_OP_NO_SSLv2 499 days http://bugs.python.org/issue4870 pitrou patch subprocess.Popen.__del__ causes AttributeError (os module == N 0 days http://bugs.python.org/issue5099 haypo patch, needs review csv sniffer 454 days http://bugs.python.org/issue5332 r.david.murray patch tarfile normalizes arcname 364 days http://bugs.python.org/issue6054 srid 2to3 fails to fix test.test_support 295 days http://bugs.python.org/issue6583 pakal file_close() ignores return value of close_the_file 222 days http://bugs.python.org/issue7079 pitrou patch, needs review distutils makefile from python.org installer doesn't work on O 144 days http://bugs.python.org/issue7580 loewis buffered io seek() buggy 130 days http://bugs.python.org/issue7640 pitrou patch newgil backport 117 days http://bugs.python.org/issue7753 jcea patch, needs review relative import broken 99 days http://bugs.python.org/issue7902 brett.cannon patch, needs review _ssl: support surrogates in filenames, and bytes/bytearray fil 26 days http://bugs.python.org/issue8477 haypo patch subprocess: support bytes program name (POSIX) 25 days http://bugs.python.org/issue8513 haypo patch, needs review Expose SSL contexts 19 days http://bugs.python.org/issue8550 pitrou patch test_gdb: test_strings() fails with ASCII locale 22 days http://bugs.python.org/issue8559 haypo patch Buggy _strerror in asyncore 0 days http://bugs.python.org/issue8573 giampaolo.rodola easy test_warnings.CEnvironmentVariableTests.test_nonascii fails un 18 days http://bugs.python.org/issue8589 haypo patch Python3/POSIX: errors if file system encoding is None 11 days http://bugs.python.org/issue8610 haypo patch Doc bug for urllib.request._urlopener in Python 3.1+ 13 days http://bugs.python.org/issue8619 orsenthil tarfile doesn't support undecodable filename in PAX format 12 days http://bugs.python.org/issue8633 lars.gustaebel subprocess: canonicalize env to bytes on Unix (Python3) 12 days http://bugs.python.org/issue8640 haypo patch Failed compile in a non-ASCII path 11 days http://bugs.python.org/issue8663 haypo patch "make pycremoval" fails 8 days http://bugs.python.org/issue8665 pitrou distutils sdist is too laze w.r.t. recalculating MANIFEST 6 days http://bugs.python.org/issue8688 tarek patch, needs review sqlite3 parameter substitution breaks with multiple parameters 3 days http://bugs.python.org/issue8689 tjreedy Use divide-and-conquer for faster factorials 5 days http://bugs.python.org/issue8692 mark.dickinson patch, patch, needs review OpenID blunder 2 days http://bugs.python.org/issue8708 brett.cannon Document PyUnicode_DecodeFSDefault() and PyUnicode_DecodeFSDef 0 days http://bugs.python.org/issue8711 haypo patch Create PyUnicode_EncodeFSDefault() function 1 days http://bugs.python.org/issue8715 haypo patch Subprocess.py broken in trunk 0 days http://bugs.python.org/issue8717 haypo patch subprocess: NameError: name 'WaitForSingleObject' is not def 0 days http://bugs.python.org/issue8718 haypo urlparse.urlsplit regression in 2.7 5 days http://bugs.python.org/issue8721 r.david.murray IDLE won't start import os error 0 days http://bugs.python.org/issue8723 gromij test_capi failure 2 days http://bugs.python.org/issue8726 jyasskin test_import failure 3 days http://bugs.python.org/issue8727 barry *= is undocumented for lists 1 days http://bugs.python.org/issue8736 stutzbach easy ssl.RAND_add() should only accept bytes (not str) 0 days http://bugs.python.org/issue8737 haypo patch infinite recursion with setfilesystemencoding and pdb 2 days http://bugs.python.org/issue8740 ccomb broken asdl link in Parser/asdl.py 1 days http://bugs.python.org/issue8742 orsenthil Maybe typo in doc 0 days http://bugs.python.org/issue8744 orsenthil Threading and KeyError: 51 0 days http://bugs.python.org/issue8751 draghuram set_contains, etc. should check for collections.Set in additio 0 days http://bugs.python.org/issue8752 rhettinger Multiprocessing and UUID bug on Mac OSX 0 days http://bugs.python.org/issue8756 r.david.murray 2.7: wrong user site directory on Linux 2 days http://bugs.python.org/issue8759 r.david.murray Python 2.6.5 fails to build on AIX 5.3 3 days http://bugs.python.org/issue8760 amaury.forgeotdarc default value in constructor not unique across objects 0 days http://bugs.python.org/issue8762 ysj.ray logging RotatingFileHandler cause too long to fininsh 0 days http://bugs.python.org/issue8764 vinay.sajip Segmentation fault with empty "encodings" subdirectory of dire 1 days http://bugs.python.org/issue8766 haypo patch Configure: Cannot disable unicode 0 days http://bugs.python.org/issue8767 redcomet tabnanny improperly handles non-ascii source files 1 days http://bugs.python.org/issue8774 haypo patch typo in docs for symtable.SymbolTable.has_import_star 0 days http://bugs.python.org/issue8778 benjamin.peterson patch utf8 codec fails to parse a character 0 days http://bugs.python.org/issue8779 benjamin.peterson csv module cannot handle embedded \r 2174 days http://bugs.python.org/issue967934 r.david.murray ftplib: add support for MDTM command 2033 days http://bugs.python.org/issue1053369 giampaolo.rodola patch Bugs in _csv module - lineterminator 2004 days http://bugs.python.org/issue1072404 r.david.murray Missing trailing newline with comment raises SyntaxError 1860 days http://bugs.python.org/issue1184112 benjamin.peterson easy urllib.quote is too slow 1712 days http://bugs.python.org/issue1285086 flox patch Py_FileSystemDefaultEncoding can be non-canonical 1262 days http://bugs.python.org/issue1608805 haypo patch Top Issues Most Discussed (10) ______________________________ 26 Use divide-and-conquer for faster factorials 5 days closed http://bugs.python.org/issue8692 19 implement PEP 3118 struct changes 703 days open http://bugs.python.org/issue3132 16 integer-to-complex comparisons give incorrect results 3 days open http://bugs.python.org/issue8748 10 tabnanny improperly handles non-ascii source files 1 days closed http://bugs.python.org/issue8774 10 urlparse.urlsplit regression in 2.7 5 days closed http://bugs.python.org/issue8721 10 Check that _PyUnicode_AsString() result is not NULL 281 days open http://bugs.python.org/issue6697 9 Python 2.6.5 fails to build on AIX 5.3 3 days closed http://bugs.python.org/issue8760 9 Failed compile in a non-ASCII path 11 days closed http://bugs.python.org/issue8663 8 2.7: wrong user site directory on Linux 2 days closed http://bugs.python.org/issue8759 8 test_import failure 3 days closed http://bugs.python.org/issue8727 From andrew.svetlov at gmail.com Fri May 21 20:18:00 2010 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Fri, 21 May 2010 22:18:00 +0400 Subject: [Python-Dev] __package__ attribute Message-ID: What are expected values for module.__package__? I see two different behaviors: importlib and standard python import. importlib processes __package__ pretty simple and clean: - for toplevel modules __package__ is empty string - for packages it's package name - for modules inside packages it's again package name Standard import follows another strategy: - it tries to setup __package__ only in case of relative import (see first 'if' in import.c:get_parent) - otherwise __package__ is untouched and None by default. For me importlib's way is better. I don't see any PEP specifying __package__ behavior. From barry at python.org Fri May 21 21:19:04 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 21 May 2010 15:19:04 -0400 Subject: [Python-Dev] Adding a new C API function in 2.6 In-Reply-To: <4BF5C01E.5000406@v.loewis.de> References: <20100520213253.48f7dfdd@pitrou.net> <20100520175225.54611105@heresy> <20100520181142.75e3c51b@heresy> <4BF5C01E.5000406@v.loewis.de> Message-ID: <20100521151904.6c1ef180@heresy> On May 21, 2010, at 01:05 AM, Martin v. L?wis wrote: >>>> Should we start thinking about releasing 2.6.6 soonish? >>> >>> By tradition, it should come out soon after 2.7 and be the last bugfix >>> (except for security patches). >> >> I guess what I mean is, should we have (at least) one more point release >> before the post-2.7 last-bug-fix-release? > >Because it's a security fix? No. This issue has been sitting in the >tracker for more than a year now, so it's not really relevant (IMO) >whether the bug fix arrives two weeks earlier. Partly that, yes. But also, 2.7 final is not scheduled until July, so we could fit one more release in I think. If there's no clamor for it, I'm also happy to just wait for 2.6.6 until after Python 2.7 is released. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From digitalxero at gmail.com Fri May 21 21:30:13 2010 From: digitalxero at gmail.com (Dj Gilcrease) Date: Fri, 21 May 2010 15:30:13 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> Message-ID: On Fri, May 21, 2010 at 8:26 AM, Brian Quinlan wrote: > Except that now isn't the time for that discussion. This PEP has discussed > on-and-off for several months on both stdlib-sig and python-dev. I think any time till the PEP is accepted is a good time to discuss changes to the API Issues with the PEP: 1) Examples as written fail on windows. Patch to fix @ http://code.google.com/p/pythonfutures/issues/detail?id=5 Issues with Implementation: 1) Globals are used for tracking running threads (but not processes) and shutdown state, but are not accessed as a globals every where they are modified so it could be inconsistent. 2) The atexit handle randomly throws an exception on exit on windows when running the tests or examples. Error in atexit._run_exitfuncs: TypeError: print_exception(): Exception expected for value, str found Issues 1 & 2 would be solved by moving thread tracking back into the executor responsible for the threads, or making a singleton that tracked threads / processes for all executors. http://code.google.com/p/pythonfutures/issues/detail?id=6 is one such implementation From andrew.svetlov at gmail.com Fri May 21 23:35:20 2010 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sat, 22 May 2010 01:35:20 +0400 Subject: [Python-Dev] __package__ attribute In-Reply-To: References: Message-ID: For me it's the real bug in standard python import machinery. I don't see any backward incompatibilities. There are very hard to write any import-depended code based on decision: was module imported in absolute or relative way. If it's a bug I can create issue in python bugtracker and provide a patch (for Python 3.2 and for 2.7 if later is required). On Fri, May 21, 2010 at 10:18 PM, Andrew Svetlov wrote: > What are expected values for module.__package__? > I see two different behaviors: importlib and standard python import. > importlib processes __package__ pretty simple and clean: > - for toplevel modules __package__ is empty string > - for packages it's package name > - for modules inside packages it's again package name > > Standard import follows another strategy: > - it tries to setup __package__ only in case of relative import (see > first 'if' in import.c:get_parent) > - otherwise __package__ is untouched and None by default. > > For me importlib's way is better. > I don't see any PEP specifying __package__ behavior. > From luhon at healthcare.uiowa.edu Sat May 22 01:32:23 2010 From: luhon at healthcare.uiowa.edu (Lu, Hongchao (UI Health Care)) Date: Fri, 21 May 2010 18:32:23 -0500 Subject: [Python-Dev] Ask a question for a script about re.findall Modlue Message-ID: <368F25BB4F425B4087B4B4C5285559FB06476857@HC-MAIL11.healthcare.uiowa.edu> Hi All, I am working on a script to use re.findall, But the result seems wield. -------------------------------------------------------------------- import re p=re.compile("\S+",re.M) pirfile=""" >DL;mm9| >DL;petMar1| >DL;cavPor3| >DL;mm9|: """ for m in p.findall(pirfile,re.M): print m Results: | >DL;petMar1| >DL;cavPor3| >DL;mm9| The first result is missing some characters, Could you explain why? Any wrong with the script? I tried Python2.4/2.6/2.7, the result is same. Thank you. Hongchao Lu -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat May 22 02:09:37 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 21 May 2010 17:09:37 -0700 Subject: [Python-Dev] Ask a question for a script about re.findall Modlue In-Reply-To: <368F25BB4F425B4087B4B4C5285559FB06476857@HC-MAIL11.healthcare.uiowa.edu> References: <368F25BB4F425B4087B4B4C5285559FB06476857@HC-MAIL11.healthcare.uiowa.edu> Message-ID: You shouldn't be asking questions about using Python on python-dev -- this list is for development of Python. Your problem is easily explained however: the second argument to p.findall() should be an offset, not a flag set. (You are confusing re.findall() and p.findall().) --Guido On Fri, May 21, 2010 at 4:32 PM, Lu, Hongchao (UI Health Care) < luhon at healthcare.uiowa.edu> wrote: > Hi All, > > I am working on a script to use re.findall, > > But the result seems wield. > > -------------------------------------------------------------------- > > import re > > p=re.compile("\S+",re.M) > > pirfile=""" > > >DL;mm9| > > >DL;petMar1| > > >DL;cavPor3| > > >DL;mm9|: > > """ > > for m in p.findall(pirfile,re.M): > > print m > > Results: > > | > > >DL;petMar1| > > >DL;cavPor3| > > >DL;mm9| > > The first result is missing some characters, Could you explain why? Any > wrong with the script? > > I tried Python2.4/2.6/2.7, the result is same. > > Thank you. > > Hongchao Lu > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From hagen at zhuliguan.net Sat May 22 10:07:41 2010 From: hagen at zhuliguan.net (=?ISO-8859-1?Q?Hagen_F=FCrstenau?=) Date: Sat, 22 May 2010 10:07:41 +0200 Subject: [Python-Dev] Ask a question for a script about re.findall Modlue In-Reply-To: References: <368F25BB4F425B4087B4B4C5285559FB06476857@HC-MAIL11.healthcare.uiowa.edu> Message-ID: <4BF790CD.1060909@zhuliguan.net> > Your problem is easily explained however: the second argument to > p.findall() should be an offset, not a flag set. (You are confusing > re.findall() and p.findall().) I filed a doc bug for this: http://bugs.python.org/issue8785 Cheers, Hagen -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 489 bytes Desc: OpenPGP digital signature URL: From brian at sweetapp.com Sat May 22 11:12:05 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sat, 22 May 2010 19:12:05 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> Message-ID: <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> On May 22, 2010, at 5:30 AM, Dj Gilcrease wrote: > On Fri, May 21, 2010 at 8:26 AM, Brian Quinlan > wrote: >> Except that now isn't the time for that discussion. This PEP has >> discussed >> on-and-off for several months on both stdlib-sig and python-dev. > > I think any time till the PEP is accepted is a good time to discuss > changes to the API I disagree. If a PEP is being updated continuously then there is nothing stable to pronounce on. > > Issues with the PEP: > 1) Examples as written fail on windows. Patch to fix @ > http://code.google.com/p/pythonfutures/issues/detail?id=5 Updated, thanks! > Issues with Implementation: > 1) Globals are used for tracking running threads (but not processes) > and shutdown state, but are not accessed as a globals every where they > are modified so it could be inconsistent. > 2) The atexit handle randomly throws an exception on exit on windows > when running the tests or examples. > > Error in atexit._run_exitfuncs: > TypeError: print_exception(): Exception expected for value, str found Lets take this off-list. Cheers, Brian > Issues 1 & 2 would be solved by moving thread tracking back into the > executor responsible for the threads, or making a singleton that > tracked threads / processes for all executors. > http://code.google.com/p/pythonfutures/issues/detail?id=6 is one such > implementation > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com From rdmurray at bitdance.com Sat May 22 15:59:46 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 22 May 2010 09:59:46 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> Message-ID: <20100522135946.8F5451F960D@kimball.webabinitio.net> On Sat, 22 May 2010 19:12:05 +1000, Brian Quinlan wrote: > On May 22, 2010, at 5:30 AM, Dj Gilcrease wrote: > > On Fri, May 21, 2010 at 8:26 AM, Brian Quinlan > > wrote: > >> Except that now isn't the time for that discussion. This PEP has > >> discussed > >> on-and-off for several months on both stdlib-sig and python-dev. > > > > I think any time till the PEP is accepted is a good time to discuss > > changes to the API > > I disagree. If a PEP is being updated continuously then there is > nothing stable to pronounce on. Well, you've been making updates as a result of this round of discussion. If there is still discussion then perhaps the PEP isn't ready for pronouncement yet. At some point someone can decide it is all bikeshedding and ask for pronouncement on that basis, but I don't think it is appropriate to cut off discussion by saying "it's ready for pronouncement" unless you want increase the chances of its getting rejected. The usual way of doing this (at least so far as I have observed, which granted hasn't been too many cases) is to say something like "I think this PEP is ready for pronouncement" and then wait for feedback on that assertion or for the pronouncement. It's especially good if you can answer any concerns that are raised with "that was discussed already and we concluded X". Bonus points for finding a thread reference and adding it to the PEP :) -- R. David Murray www.bitdance.com From jnoller at gmail.com Sat May 22 16:09:37 2010 From: jnoller at gmail.com (Jesse Noller) Date: Sat, 22 May 2010 10:09:37 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <20100522135946.8F5451F960D@kimball.webabinitio.net> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> Message-ID: On Sat, May 22, 2010 at 9:59 AM, R. David Murray wrote: > On Sat, 22 May 2010 19:12:05 +1000, Brian Quinlan wrote: >> On May 22, 2010, at 5:30 AM, Dj Gilcrease wrote: >> > On Fri, May 21, 2010 at 8:26 AM, Brian Quinlan >> > wrote: >> >> Except that now isn't the time for that discussion. This PEP has >> >> discussed >> >> on-and-off for several months on both stdlib-sig and python-dev. >> > >> > I think any time till the PEP is accepted is a good time to discuss >> > changes to the API >> >> I disagree. If a PEP is being updated continuously then there is >> nothing stable to pronounce on. > > Well, you've been making updates as a result of this round of > discussion. > > If there is still discussion then perhaps the PEP isn't ready for > pronouncement yet. ?At some point someone can decide it is all > bikeshedding and ask for pronouncement on that basis, but I don't > think it is appropriate to cut off discussion by saying "it's ready for > pronouncement" unless you want increase the chances of its getting > rejected. I commiserate with Brian here - he's been very patient, and has been working on things, taking in input, etc for awhile now on this. In his mind, it is done (or at least incredibly close to done) and opening the door in the conversation for more API nit picking and debate about the exact verbiage on method names means we're never going to be done splashing paint. > The usual way of doing this (at least so far as I have observed, which > granted hasn't been too many cases) is to say something like "I think > this PEP is ready for pronouncement" and then wait for feedback on that > assertion or for the pronouncement. ?It's especially good if you can answer > any concerns that are raised with "that was discussed already and we > concluded X". ?Bonus points for finding a thread reference and adding > it to the PEP :) While sure, this is true - I'd actually back Brian up on trying to avoid more "why didn't you call it a banana" style discussions. At some point the constant back and forth has to stop, and to his credit, Brian has made a lot of changes, listened to a lot of feedback, etc. It's fair for him to just ask that a decision be made. jesse From guido at python.org Sat May 22 17:38:27 2010 From: guido at python.org (Guido van Rossum) Date: Sat, 22 May 2010 08:38:27 -0700 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> Message-ID: On Sat, May 22, 2010 at 7:09 AM, Jesse Noller wrote: > On Sat, May 22, 2010 at 9:59 AM, R. David Murray wrote: >> On Sat, 22 May 2010 19:12:05 +1000, Brian Quinlan wrote: >>> On May 22, 2010, at 5:30 AM, Dj Gilcrease wrote: >>> > On Fri, May 21, 2010 at 8:26 AM, Brian Quinlan >>> > wrote: >>> >> Except that now isn't the time for that discussion. This PEP has >>> >> discussed >>> >> on-and-off for several months on both stdlib-sig and python-dev. >>> > >>> > I think any time till the PEP is accepted is a good time to discuss >>> > changes to the API >>> >>> I disagree. If a PEP is being updated continuously then there is >>> nothing stable to pronounce on. >> >> Well, you've been making updates as a result of this round of >> discussion. >> >> If there is still discussion then perhaps the PEP isn't ready for >> pronouncement yet. ?At some point someone can decide it is all >> bikeshedding and ask for pronouncement on that basis, but I don't >> think it is appropriate to cut off discussion by saying "it's ready for >> pronouncement" unless you want increase the chances of its getting >> rejected. > > I commiserate with Brian here - he's been very patient, and has been > working on things, taking in input, etc for awhile now on this. In his > mind, it is done (or at least incredibly close to done) and opening > the door in the conversation for more API nit picking and debate about > the exact verbiage on method names means we're never going to be done > splashing paint. > >> The usual way of doing this (at least so far as I have observed, which >> granted hasn't been too many cases) is to say something like "I think >> this PEP is ready for pronouncement" and then wait for feedback on that >> assertion or for the pronouncement. ?It's especially good if you can answer >> any concerns that are raised with "that was discussed already and we >> concluded X". ?Bonus points for finding a thread reference and adding >> it to the PEP :) > > While sure, this is true - I'd actually back Brian up on trying to > avoid more "why didn't you call it a banana" style discussions. At > some point the constant back and forth has to stop, and to his credit, > Brian has made a lot of changes, listened to a lot of feedback, etc. > It's fair for him to just ask that a decision be made. Great points Jesse! Since I really don't have the time or expertise to make a judgment on this PEP, I hereby appoint you chair of the approval process for this PEP. That basically means that when you think it's ready to be approved, you say so, and it's a done deal. The remaining feedback cycle is up to you now -- it sounds like you're ready for closure, which sounds good to me (again, without having read the PEP or tried to write something using the proposed code). You can do it however you like: you can declare it approved now, or read it over once more yourself and suggest some final changes, or set a period (e.g. 48 hours) during which final comments have to be received, which you then will judge by merit or by your whim, or you can flip a coin or say a prayer... (I've tried most of those myself in the past and haven't done too badly if I say so myself. :-) You're the boss now. I know you will do the right thing for this PEP. -- --Guido van Rossum (python.org/~guido) From rdmurray at bitdance.com Sat May 22 22:40:16 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 22 May 2010 16:40:16 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> Message-ID: <20100522204016.DA8541F9FA2@kimball.webabinitio.net> On Sat, 22 May 2010 10:09:37 -0400, Jesse Noller wrote: > On Sat, May 22, 2010 at 9:59 AM, R. David Murray wr= > ote: > > On Sat, 22 May 2010 19:12:05 +1000, Brian Quinlan wr= > ote: > >> On May 22, 2010, at 5:30 AM, Dj Gilcrease wrote: > >> > On Fri, May 21, 2010 at 8:26 AM, Brian Quinlan > >> > wrote: > >> >> Except that now isn't the time for that discussion. This PEP has > >> >> discussed > >> >> on-and-off for several months on both stdlib-sig and python-dev. > >> > > >> > I think any time till the PEP is accepted is a good time to discuss > >> > changes to the API > >> > >> I disagree. If a PEP is being updated continuously then there is > >> nothing stable to pronounce on. > > > > Well, you've been making updates as a result of this round of > > discussion. > > > > If there is still discussion then perhaps the PEP isn't ready for > > pronouncement yet. =A0At some point someone can decide it is all > > bikeshedding and ask for pronouncement on that basis, but I don't > > think it is appropriate to cut off discussion by saying "it's ready for > > pronouncement" unless you want increase the chances of its getting > > rejected. > > I commiserate with Brian here - he's been very patient, and has been > working on things, taking in input, etc for awhile now on this. In his > mind, it is done (or at least incredibly close to done) and opening > the door in the conversation for more API nit picking and debate about > the exact verbiage on method names means we're never going to be done > splashing paint. OK, so you are saying that the comments in question are bikeshedding. I can accept that easily. What I was trying to point out was that saying "discussion is closed" is not the best way to nurture community consensus. Saying "we've reached the bikesheedding point, let's pronounce" is a very different thing to my mind, even if it is only a matter of tone. Have fun pronouncing ;) -- R. David Murray www.bitdance.com From brian at sweetapp.com Sun May 23 01:12:46 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sun, 23 May 2010 09:12:46 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <20100522135946.8F5451F960D@kimball.webabinitio.net> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> Message-ID: <2EF114F8-585B-4204-A7ED-1FE7F8226D2D@sweetapp.com> On 22 May 2010, at 23:59, R. David Murray wrote: > On Sat, 22 May 2010 19:12:05 +1000, Brian Quinlan > wrote: >> On May 22, 2010, at 5:30 AM, Dj Gilcrease wrote: >>> On Fri, May 21, 2010 at 8:26 AM, Brian Quinlan >>> wrote: >>>> Except that now isn't the time for that discussion. This PEP has >>>> discussed >>>> on-and-off for several months on both stdlib-sig and python-dev. >>> >>> I think any time till the PEP is accepted is a good time to discuss >>> changes to the API >> >> I disagree. If a PEP is being updated continuously then there is >> nothing stable to pronounce on. > > Well, you've been making updates as a result of this round of > discussion. Yes, I've been making documentation and PEP updates to clarify points that people found confusing and will continue to do so. > If there is still discussion then perhaps the PEP isn't ready for > pronouncement yet. At some point someone can decide it is all > bikeshedding and ask for pronouncement on that basis, but I don't > think it is appropriate to cut off discussion by saying "it's ready > for > pronouncement" unless you want increase the chances of its getting > rejected. Here are the new proposed non-documentation changes that I've collected (let me know if I've missed any): Rename "executor" => "executer" Rename "submit" to "apply" Rename "done" to "finished" Rename "not_finished" to "pending" Rename "FIRST_COMPLETED" to "ONE_COMPLETED" We can discuss naming for all eternity and never reach a point where even half of the participants are satisfied. Since naming has been discussed extensively here and in stdlib-sig, I think that we have to decide that it is good enough and move on. Or decide that it isn't good enough and reject the PEP. Cheers, Brian > The usual way of doing this (at least so far as I have observed, which > granted hasn't been too many cases) is to say something like "I think > this PEP is ready for pronouncement" and then wait for feedback on > that > assertion or for the pronouncement. It's especially good if you can > answer > any concerns that are raised with "that was discussed already and we > concluded X". Bonus points for finding a thread reference and adding > it to the PEP :) > > -- > R. David Murray www.bitdance.com From jyasskin at gmail.com Sun May 23 01:43:41 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Sat, 22 May 2010 16:43:41 -0700 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <2EF114F8-585B-4204-A7ED-1FE7F8226D2D@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <2EF114F8-585B-4204-A7ED-1FE7F8226D2D@sweetapp.com> Message-ID: I think the PEP's overall API is good to go. On Sat, May 22, 2010 at 4:12 PM, Brian Quinlan wrote: > > On 22 May 2010, at 23:59, R. David Murray wrote: >> If there is still discussion then perhaps the PEP isn't ready for >> pronouncement yet. ?At some point someone can decide it is all >> bikeshedding and ask for pronouncement on that basis, but I don't >> think it is appropriate to cut off discussion by saying "it's ready for >> pronouncement" unless you want increase the chances of its getting >> rejected. > > Here are the new proposed non-documentation changes that I've collected (let > me know if I've missed any): > > ... I propose to rename the Future.result method to Future.get. "get" is what Java (http://java.sun.com/javase/7/docs/api/java/util/concurrent/Future.html) and C++ (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3092.pdf section 30.6.6 para 12) use, and the word "result" doesn't seem particularly better or worse than "get" for our purposes, which inclines me to stay consistent. > We can discuss naming for all eternity and never reach a point where even > half of the participants are satisfied. Agreed. To reduce the length of the discussion, I'm not going to reply to counter-arguments to my proposal, but I think it'll be useful to Jesse if people who agree or disagree speak up briefly. I'll reply the other naming proposals in another message. Jeffrey From jyasskin at gmail.com Sun May 23 02:06:30 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Sat, 22 May 2010 17:06:30 -0700 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <2EF114F8-585B-4204-A7ED-1FE7F8226D2D@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <2EF114F8-585B-4204-A7ED-1FE7F8226D2D@sweetapp.com> Message-ID: On Sat, May 22, 2010 at 4:12 PM, Brian Quinlan wrote: > Rename "executor" => "executer" -1 for consistency with Java. > Rename "submit" to "apply" "apply" focuses attention on the function object, while "submit" focuses attention, properly I think, on the fact that you're handing something to the executor to run. So -1. > Rename "done" to "finished" "done" is nice and short, and I don't think "finished" or "completed" will be any less prone to people thinking the task actually ran. So -1. > Rename "not_finished" to "pending" +0.5. Doesn't matter that much, but pending is used elsewhere in the proposal for this concept. On the other hand, "pending" could be thought to refer to the state before "running". Possibly "finished" should be renamed to "done" here, since it's described as '"finished", contains the futures that completed (finished or were cancelled)', which uses "finished" for two different concepts. > Rename "FIRST_COMPLETED" to "ONE_COMPLETED" "ONE_COMPLETED" could imply that the first result set must contain exactly one element, but in fact, if multiple tasks finish before the waiting thread has a chance to wake up, multiple futures could be returned as done. So -1. And like my other post, I won't argue about these, leaving the actual decision up to Brian and Jesse. From brian at sweetapp.com Sun May 23 02:47:08 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sun, 23 May 2010 10:47:08 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> Message-ID: <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> Hey all, Jesse, the designated pronouncer for this PEP, has decided to keep discussion open for a few more days. So fire away! Cheers, Brian From brian at sweetapp.com Sun May 23 03:04:49 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sun, 23 May 2010 11:04:49 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <2EF114F8-585B-4204-A7ED-1FE7F8226D2D@sweetapp.com> Message-ID: <54EFD5A6-295A-4622-947F-FC57EBA554A5@sweetapp.com> On May 23, 2010, at 9:43 AM, Jeffrey Yasskin wrote: > I think the PEP's overall API is good to go. > > On Sat, May 22, 2010 at 4:12 PM, Brian Quinlan > wrote: >> >> On 22 May 2010, at 23:59, R. David Murray wrote: >>> If there is still discussion then perhaps the PEP isn't ready for >>> pronouncement yet. At some point someone can decide it is all >>> bikeshedding and ask for pronouncement on that basis, but I don't >>> think it is appropriate to cut off discussion by saying "it's >>> ready for >>> pronouncement" unless you want increase the chances of its getting >>> rejected. >> >> Here are the new proposed non-documentation changes that I've >> collected (let >> me know if I've missed any): >> >> ... > > I propose to rename the Future.result method to Future.get. "get" is > what Java (http://java.sun.com/javase/7/docs/api/java/util/concurrent/Future.html > ) > and C++ (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3092.pdf > section 30.6.6 para 12) use, and the word "result" doesn't seem > particularly better or worse than "get" for our purposes, which > inclines me to stay consistent. In C++ and Java, there is only one result-retrieving method so "get" seems like a reasonable name. My implementation has a second method .exception(), which returns the exception raised by the submitted function (or None if no exception was raised). I thought that having multiple getter methods, where one is called .get() would be a bit confusing. But I don't really care so I'm -0. Cheers, Brian From jnoller at gmail.com Sun May 23 03:07:19 2010 From: jnoller at gmail.com (Jesse Noller) Date: Sat, 22 May 2010 21:07:19 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> Message-ID: On Sat, May 22, 2010 at 8:47 PM, Brian Quinlan wrote: > Hey all, > > Jesse, the designated pronouncer for this PEP, has decided to keep > discussion open for a few more days. > > So fire away! Man, everyone's faster on the email thing lately than me :) Yes, I spoke to Brian, and since we're not in a rush - please do bring up serious issues you might have, assume for the moment that the names, unless they're "def lol_python" are going to stay pretty consistent unless everyone breaks out the pitchforks, and also assume the API won't see much changing. So please, do bring up issues. I'm obviously biased towards accepting it - however, nothing is ever set in stone. jesse From brian at sweetapp.com Sun May 23 03:21:51 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sun, 23 May 2010 11:21:51 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <2EF114F8-585B-4204-A7ED-1FE7F8226D2D@sweetapp.com> Message-ID: On May 23, 2010, at 10:06 AM, Jeffrey Yasskin wrote: > On Sat, May 22, 2010 at 4:12 PM, Brian Quinlan > wrote: >> Rename "executor" => "executer" > > -1 for consistency with Java. > -1 pending an explanation of why "executer" is better >> Rename "submit" to "apply" > > "apply" focuses attention on the function object, while "submit" > focuses attention, properly I think, on the fact that you're handing > something to the executor to run. So -1. -1 >> Rename "done" to "finished" > > "done" is nice and short, and I don't think "finished" or "completed" > will be any less prone to people thinking the task actually ran. So > -1. -0 >> Rename "not_finished" to "pending" > > +0.5. Doesn't matter that much, but pending is used elsewhere in the > proposal for this concept. On the other hand, "pending" could be > thought to refer to the state before "running". Possibly "finished" > should be renamed to "done" here, since it's described as '"finished", > contains the futures that completed (finished or were cancelled)', > which uses "finished" for two different concepts. I think that using "finished" is bad terminology here. So +1 to "finished" => "done". I don't have a preference for "not_done" vs. "pending". >> Rename "FIRST_COMPLETED" to "ONE_COMPLETED" > > "ONE_COMPLETED" could imply that the first result set must contain > exactly one element, but in fact, if multiple tasks finish before the > waiting thread has a chance to wake up, multiple futures could be > returned as done. So -1. A logician would probably call it "SOME_COMPLETED". What about "ANY_COMPLETED"? Though I think that "FIRST_COMPLETED" still reads better. Cheers, Brian From glyph at twistedmatrix.com Sun May 23 06:44:50 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Sun, 23 May 2010 00:44:50 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> Message-ID: <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> On May 22, 2010, at 8:47 PM, Brian Quinlan wrote: > Jesse, the designated pronouncer for this PEP, has decided to keep discussion open for a few more days. > > So fire away! As you wish! The PEP should be consistent in its usage of terminology about callables. It alternately calls them "callables", "functions", and "functions or methods". It would be nice to clean this up and be consistent about what can be called where. I personally like "callables". The execution context of callable code is not made clear. Implicitly, submit() or map() would run the code in threads or processes as defined by the executor, but that's not spelled out clearly. More relevant to my own interests, the execution context of the callables passed to add_done_callback and remove_done_callback is left almost completely to the imagination. If I'm reading the sample implementation correctly, , it looks like in the multiprocessing implementation, the done callbacks are invoked in a random local thread. The fact that they are passed the future itself *sort* of implies that this is the case, but the multiprocessing module plays fast and loose with object identity all over the place, so it would be good to be explicit and say that it's *not* a pickled copy of the future sitting in some arbitrary process (or even on some arbitrary machine). This is really minor, I know, but why does it say "NOTE: This method can be used to create adapters from Futures to Twisted Deferreds"? First of all, what's the deal with "NOTE"; it's the only "NOTE" in the whole PEP, and it doesn't seem to add anything. This sentence would read exactly the same if that word were deleted. Without more clarity on the required execution context of the callbacks, this claim might not actually be true anyway; Deferred callbacks can only be invoked in the main reactor thread in Twisted. But even if it is perfectly possible, why leave so much of the adapter implementation up to the imagination? If it's important enough to mention, why not have a reference to such an adapter in the reference Futures implementation, since it *should* be fairly trivial to write? The fact that add_done_callback is implemented using a set is weird, since it means you can't add the same callback more than once. The set implementation also means that the callbacks get called in a semi-random order, potentially creating even _more_ hard-to-debug order of execution issues than you'd normally have with futures. And I think that this documentation will be unclear to a lot of novice developers: many people have trouble with the idea that "a = Foo(); b = Foo(); a.bar_method != b.bar_method", but "import foo_module; foo_module.bar_function == foo_module.bar_function". It's also weird that you can remove callbacks - what's the use case? Deferreds have no callback-removal mechanism and nobody has ever complained of the need for one, as far as I know. (But lots of people do add the same callback multiple times.) I suggest having have add_done_callback, implementing it with a list so that callbacks are always invoked in the order that they're added, and getting rid of remove_done_callback. futures._base.Executor isn't exposed publicly, but it needs to be. The PEP kinda makes it sound like it is ("Executor is an abstract class..."). Plus, A third party library wanting to implement an executor of its own shouldn't have to copy and paste the implementation of Executor.map. One minor suggestion on the "internal future methods" bit - something I wish we'd done with Deferreds was to put 'callback()' and 'addCallbacks()' on separate objects, so that it was very explicit whether you were on the emitting side of a Deferred or the consuming side. That seems to be the case with these internal methods - they are not so much "internal" as they are for the producer of the Future (whether a unit test or executor) so you might want to put them on a different object that it's easy for the thing creating a Future() to get at but hard for any subsequent application code to fiddle with by accident. Off the top of my head, I suggest naming it "Invoker()". A good way to do this would be to have an Invoker class which can't be instantiated (raises an exception from __init__ or somesuch), then a Future.create() method which returns an Invoker, which itself has a '.future' attribute. Finally, why isn't this just a module on PyPI? It doesn't seem like there's any particular benefit to making this a stdlib module and going through the whole PEP process - except maybe to prompt feedback like this :). Issues like the ones I'm bringing up could be fixed pretty straightforwardly if it were just a matter of filing a bug on a small package, but fixing a stdlib module is a major undertaking. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at sweetapp.com Sun May 23 08:37:31 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sun, 23 May 2010 16:37:31 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> Message-ID: On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote: > > On May 22, 2010, at 8:47 PM, Brian Quinlan wrote: > >> Jesse, the designated pronouncer for this PEP, has decided to keep >> discussion open for a few more days. >> >> So fire away! > > As you wish! I retract my request ;-) > The PEP should be consistent in its usage of terminology about > callables. It alternately calls them "callables", "functions", and > "functions or methods". It would be nice to clean this up and be > consistent about what can be called where. I personally like > "callables". Did you find the terminology confusing? If not then I propose not changing it. But changing it in the user docs is probably a good idea. I like "callables" too. > > The execution context of callable code is not made clear. > Implicitly, submit() or map() would run the code in threads or > processes as defined by the executor, but that's not spelled out > clearly. > > More relevant to my own interests, the execution context of the > callables passed to add_done_callback and remove_done_callback is > left almost completely to the imagination. If I'm reading the > sample implementation correctly, >, it looks like in the multiprocessing implementation, the done > callbacks are invoked in a random local thread. The fact that they > are passed the future itself *sort* of implies that this is the > case, but the multiprocessing module plays fast and loose with > object identity all over the place, so it would be good to be > explicit and say that it's *not* a pickled copy of the future > sitting in some arbitrary process (or even on some arbitrary machine). The callbacks will always be called in a thread other than the main thread in the process that created the executor. Is that a strong enough contract? > This is really minor, I know, but why does it say "NOTE: This method > can be used to create adapters from Futures to Twisted Deferreds"? > First of all, what's the deal with "NOTE"; it's the only "NOTE" in > the whole PEP, and it doesn't seem to add anything. This sentence > would read exactly the same if that word were deleted. Without more > clarity on the required execution context of the callbacks, this > claim might not actually be true anyway; Deferred callbacks can only > be invoked in the main reactor thread in Twisted. But even if it is > perfectly possible, why leave so much of the adapter implementation > up to the imagination? If it's important enough to mention, why not > have a reference to such an adapter in the reference Futures > implementation, since it *should* be fairly trivial to write? I'm a bit surprised that this doesn't allow for better interoperability with Deferreds given this discussion: At 02:36 PM 3/16/2010 -0700, Brian Quinlan wrote: """ From P.J Eby: > On Mar 7, 2010, at 11:56 AM, P.J. Eby wrote: > >> At 10:59 AM 3/7/2010 -0800, Jeffrey Yasskin wrote: >>> Given a way to register "on-done" callbacks with the future, it >>> would be straightforward to wait for a future without blocking, too. >> >> Yes, and with a few more additions besides that one, you might be >> on the way to an actual competitor for Deferreds. For example: >> retry support, chaining, logging, API for transparent result >> processing, coroutine support, co-ordination tools like locks, >> sempaphores and queues, etc. > > OK, but lets just think about making the APIs compatible e.g. you > have some code that uses Futures and now you want to integrate it > with some code that uses Deferreds. > > I think Jeff's suggestion of having a completion callback on Futures > would make it possible to write a Future-to-Deferred adapter. Is > that correct? As long as the callback signature included a way to pass in an error, then yes, that'd probably be sufficient. """ If add_done_callback doesn't help with twisted interoperability then I'd suggest removing it to allow for something that may be more useful to be added later. > > The fact that add_done_callback is implemented using a set is weird, > since it means you can't add the same callback more than once. The > set implementation also means that the callbacks get called in a > semi-random order, potentially creating even _more_ hard-to-debug > order of execution issues than you'd normally have with futures. > And I think that this documentation will be unclear to a lot of > novice developers: many people have trouble with the idea that "a = > Foo(); b = Foo(); a.bar_method != b.bar_method", but "import > foo_module; foo_module.bar_function == foo_module.bar_function". > > It's also weird that you can remove callbacks - what's the use > case? Deferreds have no callback-removal mechanism and nobody has > ever complained of the need for one, as far as I know. (But lots of > people do add the same callback multiple times.) > > I suggest having have add_done_callback, implementing it with a list > so that callbacks are always invoked in the order that they're > added, and getting rid of remove_done_callback. Sounds good to me! > futures._base.Executor isn't exposed publicly, but it needs to be. > The PEP kinda makes it sound like it is ("Executor is an abstract > class..."). Plus, A third party library wanting to implement an > executor of its own shouldn't have to copy and paste the > implementation of Executor.map. That was a bug that I've fixed. Thanks! > > One minor suggestion on the "internal future methods" bit - > something I wish we'd done with Deferreds was to put 'callback()' > and 'addCallbacks()' on separate objects, so that it was very > explicit whether you were on the emitting side of a Deferred or the > consuming side. That seems to be the case with these internal > methods - they are not so much "internal" as they are for the > producer of the Future (whether a unit test or executor) so you > might want to put them on a different object that it's easy for the > thing creating a Future() to get at but hard for any subsequent > application code to fiddle with by accident. Off the top of my > head, I suggest naming it "Invoker()". A good way to do this would > be to have an Invoker class which can't be instantiated (raises an > exception from __init__ or somesuch), then a Future.create() method > which returns an Invoker, which itself has a '.future' attribute. > > Finally, why isn't this just a module on PyPI? It doesn't seem like > there's any particular benefit to making this a stdlib module and > going through the whole PEP process - except maybe to prompt > feedback like this :). We've already had this discussion before. Could you explain why this module should *not* be in the stdlib e.g. does it have significantly less utility than other modules in stdlib? Is it significantly higher risk? etc? > Issues like the ones I'm bringing up could be fixed pretty > straightforwardly if it were just a matter of filing a bug on a > small package, but fixing a stdlib module is a major undertaking. True but I don't think that is a convincing argument. A subset of the functionality provided by this module is already available in Java and C++ and (at least in Java) it is used extensively and without too much trouble. If there are implementation bugs then we can fix them just like we would with any other module. Cheers, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From debatem1 at gmail.com Sun May 23 11:15:12 2010 From: debatem1 at gmail.com (geremy condra) Date: Sun, 23 May 2010 05:15:12 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> Message-ID: On Sun, May 23, 2010 at 2:37 AM, Brian Quinlan wrote: > Finally, why isn't this just a module on PyPI? ?It doesn't seem like there's > any particular benefit to making this a stdlib module and going through the > whole PEP process - except maybe to prompt feedback like this :). > > We've already had this discussion before. Could you explain why this module > should *not* be in the stdlib e.g. does it have significantly less utility > than other modules in stdlib? Is it significantly higher risk? etc? Inclusion in the stdlib is the exception, not the rule, and every exception should be issued for a good reason. I'd like to know what that reason is in this case, if only to get a clearer understanding of why the PEP was accepted. > Issues like the ones I'm bringing up could be fixed pretty straightforwardly > if it were just a matter of filing a bug on a small package, but fixing a > stdlib module is a major undertaking. > > True but I don't think that is a convincing argument. A subset of the > functionality provided by this module is already available in Java and C++ > and (at least in Java) it is used extensively and without too much trouble. > If there are implementation bugs then we can fix them just like we would > with any other module. Guido made exactly the opposite argument during his keynote at PyCon. It seemed fairly reasonable at the time- why do you think it doesn't apply here? Geremy Condra From brian at sweetapp.com Sun May 23 11:39:11 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sun, 23 May 2010 19:39:11 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> Message-ID: <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> On May 23, 2010, at 7:15 PM, geremy condra wrote: > On Sun, May 23, 2010 at 2:37 AM, Brian Quinlan > wrote: > > > >> Finally, why isn't this just a module on PyPI? It doesn't seem >> like there's >> any particular benefit to making this a stdlib module and going >> through the >> whole PEP process - except maybe to prompt feedback like this :). >> >> We've already had this discussion before. Could you explain why >> this module >> should *not* be in the stdlib e.g. does it have significantly less >> utility >> than other modules in stdlib? Is it significantly higher risk? etc? > > Inclusion in the stdlib is the exception, not the rule, and every > exception should be issued for a good reason. I'd like to know > what that reason is in this case, This package eliminates the need to construct the boilerplate present in many Python applications i.e. a thread or process pool, a work queue and result queue. It also makes it easy to take an existing Python application that executes (e.g. IO operations) in sequence and execute them in parallel. It package provides common idioms for two existing modules i.e. multiprocessing offers map functionality while threading doesn't. Those idioms are well understood and already present in Java and C++. > if only to get a clearer > understanding of why the PEP was accepted. It hasn't been accepted. >> Issues like the ones I'm bringing up could be fixed pretty >> straightforwardly >> if it were just a matter of filing a bug on a small package, but >> fixing a >> stdlib module is a major undertaking. >> >> True but I don't think that is a convincing argument. A subset of the >> functionality provided by this module is already available in Java >> and C++ >> and (at least in Java) it is used extensively and without too much >> trouble. >> If there are implementation bugs then we can fix them just like we >> would >> with any other module. > > Guido made exactly the opposite argument during his keynote at PyCon. > It seemed fairly reasonable at the time- why do you think it doesn't > apply > here? Could you be a little more specific about Guido's argument at PyCon? Cheers, Brian From regebro at gmail.com Sun May 23 11:54:27 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 23 May 2010 11:54:27 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> Message-ID: On Sun, May 23, 2010 at 11:39, Brian Quinlan wrote: > This package eliminates the need to construct the boilerplate present in > many Python applications i.e. a thread or process pool, a work queue and > result queue. ?It also makes it easy to take an existing Python application > that executes (e.g. IO operations) in sequence and execute them in parallel. > It package provides common idioms for two existing modules i.e. > multiprocessing offers map functionality while threading doesn't. Those > idioms are well understood and already present in Java and C++. It can do that as a separate package as well. And not only that, it could then be available on PyPI for earlier versions of Python as well, making it much more likely to gain widespread acceptance. > Could you be a little more specific about Guido's argument at PyCon? A module in stdlib has to be "dead". After it's included in the stdlib it can not go through any major changes since that would mean loss of backwards incompatibility. Also, you can't fix bugs except by releasing new versions of Python. Therefore the API must be completely stable, and the product virtually bugfree before it should be in stdlib. The best way of ensuring that is to release it as a separate module on PyPI, and let it stabilize for a couple of years. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From brian at sweetapp.com Sun May 23 12:15:39 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sun, 23 May 2010 20:15:39 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> Message-ID: <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> On May 23, 2010, at 7:54 PM, Lennart Regebro wrote: > On Sun, May 23, 2010 at 11:39, Brian Quinlan > wrote: >> This package eliminates the need to construct the boilerplate >> present in >> many Python applications i.e. a thread or process pool, a work >> queue and >> result queue. It also makes it easy to take an existing Python >> application >> that executes (e.g. IO operations) in sequence and execute them in >> parallel. >> It package provides common idioms for two existing modules i.e. >> multiprocessing offers map functionality while threading doesn't. >> Those >> idioms are well understood and already present in Java and C++. > > It can do that as a separate package as well. You could make the same argument about any module in the stdlib. > And not only that, it > could then be available on PyPI for earlier versions of Python as > well, making it much more likely to gain widespread acceptance. I doubt it. Simple modules are unlikely to develop a following because it is too easy to partially replicate their functionality. urlparse and os.path are very useful modules but I doubt that they would have been successful on PyPI. >> Could you be a little more specific about Guido's argument at PyCon? > > A module in stdlib has to be "dead". After it's included in the stdlib > it can not go through any major changes since that would mean loss of > backwards incompatibility. The good news in this case is that the same API has been used successfully in Java and C++ for years so it is unlikely that any major changes will need to be made. > Also, you can't fix bugs except by > releasing new versions of Python. Therefore the API must be completely > stable, and the product virtually bugfree before it should be in > stdlib. The best way of ensuring that is to release it as a separate > module on PyPI, and let it stabilize for a couple of years. Yeah but that model isn't likely to work with this package. Cheers, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Sun May 23 12:43:22 2010 From: robertc at robertcollins.net (Robert Collins) Date: Sun, 23 May 2010 22:43:22 +1200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> Message-ID: On Sun, May 23, 2010 at 10:15 PM, Brian Quinlan wrote: >> Also, you can't fix bugs except by >> releasing new versions of Python. Therefore the API must be completely >> stable, and the product virtually bugfree before it should be in >> stdlib. The best way of ensuring that is to release it as a separate >> module on PyPI, and let it stabilize for a couple of years. > > Yeah but that model isn't likely to work with this package. > Cheers, > Brian Forgive my ignorance, but why do you say that that model won't work with this package? From dirkjan at ochtman.nl Sun May 23 12:43:57 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sun, 23 May 2010 12:43:57 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> Message-ID: On Sun, May 23, 2010 at 12:15, Brian Quinlan wrote: > I doubt it. Simple modules are unlikely to develop a following because it is > too easy to partially replicate their functionality. urlparse and os.path > are very useful modules but I doubt that they would have been successful on > PyPI. simplejson was also fairly simple, but still developed a following. > The good news in this case is that the same API has been used successfully > in Java and C++ for years so it is unlikely that any major changes will need > to be made. I would agree that having prior versions in other languages should make the API more stable, but I wouldn't agree that it doesn't need changes (and even minor changes can be a PITA in the stdlib). >> Also, you can't fix bugs except by >> releasing new versions of Python. Therefore the API must be completely >> stable, and the product virtually bugfree before it should be in >> stdlib. The best way of ensuring that is to release it as a separate >> module on PyPI, and let it stabilize for a couple of years. > > Yeah but that model isn't likely to work with this package. Okay, I'll bite: why is your package different? In general, this reminds me of the ipaddr discussions. I read through the thread from March real quick to see if there was reasoning there why this package should be an exception from the "normal" standards track (that is, ripen on PyPI, then moving it in the stdlib when it's mature -- where "mature" is another word for dead, really). But then this is just another instance of the fat-stdlib vs lean-stdlib discussion, I guess, so we can go on at length. Cheers, Dirkjan From brian at sweetapp.com Sun May 23 12:47:27 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sun, 23 May 2010 20:47:27 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> Message-ID: <7251439E-F687-490C-A7A5-29018ECC3861@sweetapp.com> On May 23, 2010, at 8:43 PM, Robert Collins wrote: > On Sun, May 23, 2010 at 10:15 PM, Brian Quinlan > wrote: > >>> Also, you can't fix bugs except by >>> releasing new versions of Python. Therefore the API must be >>> completely >>> stable, and the product virtually bugfree before it should be in >>> stdlib. The best way of ensuring that is to release it as a separate >>> module on PyPI, and let it stabilize for a couple of years. >> >> Yeah but that model isn't likely to work with this package. >> Cheers, >> Brian > > Forgive my ignorance, but why do you say that that model won't work > with this package? As I said in my last message: """Simple modules are unlikely to develop a following because it is too easy to partially replicate their functionality. urlparse and os.path are very useful modules but I doubt that they would have been successful on PyPI.""" Cheers, Brian From solipsis at pitrou.net Sun May 23 12:51:55 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 May 2010 12:51:55 +0200 Subject: [Python-Dev] Dead modules References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> Message-ID: <20100523125155.53959f2f@pitrou.net> On Sun, 23 May 2010 12:43:57 +0200 Dirkjan Ochtman wrote: > > In general, this reminds me of the ipaddr discussions. I read through > the thread from March real quick to see if there was reasoning there > why this package should be an exception from the "normal" standards > track (that is, ripen on PyPI, then moving it in the stdlib when it's > mature -- where "mature" is another word for dead, really). I disagree that a stdlib module is a dead module. It is perfectly possible to augment the API with new functionality without breaking compatibility. You can also deprecate old APIs if you want. Regards Antoine. From brian at sweetapp.com Sun May 23 13:02:48 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sun, 23 May 2010 21:02:48 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> Message-ID: On May 23, 2010, at 8:43 PM, Dirkjan Ochtman wrote: > On Sun, May 23, 2010 at 12:15, Brian Quinlan > wrote: >> I doubt it. Simple modules are unlikely to develop a following >> because it is >> too easy to partially replicate their functionality. urlparse and >> os.path >> are very useful modules but I doubt that they would have been >> successful on >> PyPI. > > simplejson was also fairly simple, but still developed a following. The API is simple but writing a JSON parser is hard enough that people will check to see if someone has already done the work for them (especially since JSON is fairly topical). If you are familiar with threads then writing a "good enough" solution without futures probably won't take you very long. Also, unless you are familiar with another futures implementation, you aren't likely to know where to look. >> The good news in this case is that the same API has been used >> successfully >> in Java and C++ for years so it is unlikely that any major changes >> will need >> to be made. > > I would agree that having prior versions in other languages should > make the API more stable, but I wouldn't agree that it doesn't need > changes (and even minor changes can be a PITA in the stdlib). Some changes are hard (i.e. changing the semantics of existing method) but some are pretty easy (i.e. adding new methods). Cheers, Brian > >>> Also, you can't fix bugs except by >>> releasing new versions of Python. Therefore the API must be >>> completely >>> stable, and the product virtually bugfree before it should be in >>> stdlib. The best way of ensuring that is to release it as a separate >>> module on PyPI, and let it stabilize for a couple of years. >> >> Yeah but that model isn't likely to work with this package. > > Okay, I'll bite: why is your package different? > > In general, this reminds me of the ipaddr discussions. I read through > the thread from March real quick to see if there was reasoning there > why this package should be an exception from the "normal" standards > track (that is, ripen on PyPI, then moving it in the stdlib when it's > mature -- where "mature" is another word for dead, really). But then > this is just another instance of the fat-stdlib vs lean-stdlib > discussion, I guess, so we can go on at length. From dirkjan at ochtman.nl Sun May 23 13:00:05 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sun, 23 May 2010 13:00:05 +0200 Subject: [Python-Dev] Dead modules In-Reply-To: <20100523125155.53959f2f@pitrou.net> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <20100523125155.53959f2f@pitrou.net> Message-ID: On Sun, May 23, 2010 at 12:51, Antoine Pitrou wrote: > I disagree that a stdlib module is a dead module. It is perfectly > possible to augment the API with new functionality without breaking > compatibility. You can also deprecate old APIs if you want. Right, it wasn't intended as that harsh... but it does come with a rather impressive set of constraints in terms of what you can do with the API. Cheers, Dirkjan From andrew.svetlov at gmail.com Sun May 23 13:16:27 2010 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sun, 23 May 2010 15:16:27 +0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <7251439E-F687-490C-A7A5-29018ECC3861@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <7251439E-F687-490C-A7A5-29018ECC3861@sweetapp.com> Message-ID: Is there any reason to have Future .cancelled, .done, .running as methods? >From my perspective they are really readonly properties. BTW, is 'cancelled' correct name? Spell-checkers likes only single 'l' form: 'canceled'. On Sun, May 23, 2010 at 2:47 PM, Brian Quinlan wrote: > > On May 23, 2010, at 8:43 PM, Robert Collins wrote: > >> On Sun, May 23, 2010 at 10:15 PM, Brian Quinlan >> wrote: >> >>>> Also, you can't fix bugs except by >>>> releasing new versions of Python. Therefore the API must be completely >>>> stable, and the product virtually bugfree before it should be in >>>> stdlib. The best way of ensuring that is to release it as a separate >>>> module on PyPI, and let it stabilize for a couple of years. >>> >>> Yeah but that model isn't likely to work with this package. >>> Cheers, >>> Brian >> >> Forgive my ignorance, but why do you say that that model won't work >> with this package? > > As I said in my last message: > > """Simple modules are unlikely to develop a following because it is too easy > to partially replicate their functionality. urlparse and os.path are very > useful modules but I doubt that they would have been successful on PyPI.""" > > Cheers, > Brian > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com > From regebro at gmail.com Sun May 23 13:17:51 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 23 May 2010 13:17:51 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> Message-ID: On Sun, May 23, 2010 at 12:15, Brian Quinlan wrote: > You could make the same argument about any module in the stdlib. Yeah, and that's exactly what I did. > I doubt it. Simple modules are unlikely to develop a following because it is > too easy to partially replicate their functionality. urlparse and os.path > are very useful modules but I doubt that they would have been successful on > PyPI. Are you saying your proposed module is so simple that anyone can easily replicate it with just a couple of lines of code? > The good news in this case is that the same API has been used successfully > in Java and C++ for years so it is unlikely that any major changes will need > to be made. Good. Then the time it takes to "mature" on PyPI would be very short. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From jon+python-dev at unequivocal.co.uk Sun May 23 13:20:26 2010 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Sun, 23 May 2010 12:20:26 +0100 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <7251439E-F687-490C-A7A5-29018ECC3861@sweetapp.com> Message-ID: <20100523112026.GM6976@snowy.squish.net> On Sun, May 23, 2010 at 03:16:27PM +0400, Andrew Svetlov wrote: > Is there any reason to have Future .cancelled, .done, .running as methods? > >From my perspective they are really readonly properties. > > BTW, is 'cancelled' correct name? Spell-checkers likes only single 'l' > form: 'canceled'. In English, only "cancelled" is correct. In American, either is correct. From brian at sweetapp.com Sun May 23 13:29:45 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sun, 23 May 2010 21:29:45 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> Message-ID: On 23 May 2010, at 21:17, Lennart Regebro wrote: > On Sun, May 23, 2010 at 12:15, Brian Quinlan > wrote: >> You could make the same argument about any module in the stdlib. > > Yeah, and that's exactly what I did. > >> I doubt it. Simple modules are unlikely to develop a following >> because it is >> too easy to partially replicate their functionality. urlparse and >> os.path >> are very useful modules but I doubt that they would have been >> successful on >> PyPI. > > Are you saying your proposed module is so simple that anyone can > easily replicate it with just a couple of lines of code? Parts of it, yes. Just like I can replace most operations in os.path and urlparse with a few lines of code. >> The good news in this case is that the same API has been used >> successfully >> in Java and C++ for years so it is unlikely that any major changes >> will need >> to be made. > > Good. Then the time it takes to "mature" on PyPI would be very short. How would you define "very short"? I've had the project on PyPI for about a year now: http://pypi.python.org/pypi/futures3 Cheers, Brian From steve at holdenweb.com Sun May 23 13:52:10 2010 From: steve at holdenweb.com (Steve Holden) Date: Sun, 23 May 2010 07:52:10 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> Message-ID: <4BF916EA.2090507@holdenweb.com> Brian Quinlan wrote: > > On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote: > [...] >> One minor suggestion on the "internal future methods" bit - something >> I wish we'd done with Deferreds was to put 'callback()' and >> 'addCallbacks()' on separate objects, so that it was very explicit >> whether you were on the emitting side of a Deferred or the consuming >> side. That seems to be the case with these internal methods - they >> are not so much "internal" as they are for the producer of the Future >> (whether a unit test or executor) so you might want to put them on a >> different object that it's easy for the thing creating a Future() to >> get at but hard for any subsequent application code to fiddle with by >> accident. Off the top of my head, I suggest naming it "Invoker()". A >> good way to do this would be to have an Invoker class which can't be >> instantiated (raises an exception from __init__ or somesuch), then a >> Future.create() method which returns an Invoker, which itself has a >> '.future' attribute. >> >> Finally, why isn't this just a module on PyPI? It doesn't seem like >> there's any particular benefit to making this a stdlib module and >> going through the whole PEP process - except maybe to prompt feedback >> like this :). > > We've already had this discussion before. Could you explain why this > module should *not* be in the stdlib e.g. does it have significantly > less utility than other modules in stdlib? Is it significantly higher > risk? etc? > Given that its author was ready to go for pronouncement and is still responding to pretty serious philosophical questions about the API I'd say that it was at least worth talking about. The thing that's needed (isn't it?) of stdlib modules is API stability. > >> Issues like the ones I'm bringing up could be fixed pretty >> straightforwardly if it were just a matter of filing a bug on a small >> package, but fixing a stdlib module is a major undertaking. > > True but I don't think that is a convincing argument. A subset of the > functionality provided by this module is already available in Java and > C++ and (at least in Java) it is used extensively and without too much > trouble. If there are implementation bugs then we can fix them just like > we would with any other module. > I don't see the availability of this functionality in those languages as any kind of reason why this needs to go into the stdlib now. Is there some desperate rush to get it in? If it were used extensively from PyPi *that* would be a recommendation ... regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ "All I want for my birthday is another birthday" - Ian Dury, 1942-2000 From regebro at gmail.com Sun May 23 13:56:05 2010 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 23 May 2010 13:56:05 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> Message-ID: On Sun, May 23, 2010 at 13:29, Brian Quinlan wrote: > Parts of it, yes. Just like I can replace most operations in os.path and > urlparse with a few lines of code. Yeah, but "parts of" is not the question. I've read the PEP, and I do *not* know how to implement it. That means it's not a trivial module, so that argument doesn't hold up here, even if we accept it as valid (which I actually don't). I don't think any module in the stdlib is entirely trivial. Yes, even parsing an URL is non-trivial, as shown by the fact that the urlparse module apparently has a bug in it for urls like svn+ssh://foo.bar/frotz. ;-) Also, even trivial modules can be useful if you use them a lot. > How would you define "very short"? That's not up to me to decide. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From jnoller at gmail.com Sun May 23 14:34:22 2010 From: jnoller at gmail.com (Jesse Noller) Date: Sun, 23 May 2010 08:34:22 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BF916EA.2090507@holdenweb.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4BF916EA.2090507@holdenweb.com> Message-ID: On Sun, May 23, 2010 at 7:52 AM, Steve Holden wrote: ...snip... >> >>> Issues like the ones I'm bringing up could be fixed pretty >>> straightforwardly if it were just a matter of filing a bug on a small >>> package, but fixing a stdlib module is a major undertaking. >> >> True but I don't think that is a convincing argument. A subset of the >> functionality provided by this module is already available in Java and >> C++ and (at least in Java) it is used extensively and without too much >> trouble. If there are implementation bugs then we can fix them just like >> we would with any other module. >> > I don't see the availability of this functionality in those languages as > any kind of reason why this needs to go into the stdlib now. Is there > some desperate rush to get it in? If it were used extensively from PyPi > *that* would be a recommendation ... > Not picking Steve's particular comments out - but Brian cites the previous discussions in the PEP itself: http://www.python.org/dev/peps/pep-3148/ All of you questioning "Why should this be in the standard library" should go read those old threads, where that question was answered numerous times. Now I suddenly regret leaving the floodgates open, as we're rapidly rehashing discussions from months ago. For this same mailing list only a few months ago (brian, I think this link should be added to the PEP, I didn't see it): http://mail.python.org/pipermail/python-dev/2010-March/098169.html Specifically: http://mail.python.org/pipermail/python-dev/2010-March/098173.html Quote: "Baloney. A young library providing some syntactic sugar which uses primitives in the standard library to implement a common pattern is fine for a PEP. We've hashed this out pretty heavily on the stdlib-sig list prior to bringing it here. By the same argument, we should shunt all of the recent unittest changes and improvements into space, since golly, other people did it, why should we. This is something relatively simple, which I would gladly add in an instant to the multiprocessing package - but Brian's one-upped me in that regard and is providing something which works with both threads and processes handily. Take a look at multiprocessing.Pool for example - all that is some sugar on top of the primitives, but it's good sugar, and is used by a fair number of people. Let me also state - "my" vision of where futures would live would be in a concurrent package - for example: from concurrent import futures The reason *why* is that I would like to also move the abstractions I have in multiprocessing *out* of that module, make them work with both threads and processes (if it makes sense) and reduce the multiprocessing module to the base primitive Process object. A concurrent package which implements common patterns built on top of the primitives we support is an objectively Good Thing. For example, how many of us have sat down and implemented a thread pool on top of threading, I would hazard to say that most of us who use threading have done this, and probably more than once. It stands to reason that this is a common enough pattern to include in the standard library. " Brian has already agreed to name spacing it to "concurrent.futures" - this means it will be a small part to a much larger concurrent.* implementation ala Java. So, in short - given we've already hashed the reasoning out. jesse From solipsis at pitrou.net Sun May 23 14:47:54 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 May 2010 14:47:54 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4BF916EA.2090507@holdenweb.com> Message-ID: <20100523144754.1c6f1afc@pitrou.net> On Sun, 23 May 2010 08:34:22 -0400 Jesse Noller wrote: > > Brian has already agreed to name spacing it to "concurrent.futures" - > this means it will be a small part to a much larger concurrent.* > implementation ala Java. What I would question here is what other things will be part of the "concurrent" package, and who will implement them. Are there plans for that? (or even tracker issues open?) Apart from that, it seems to me that the only serious issues blocking PEP approval are Glyph's interesting remarks. Regards Antoine. From steve at holdenweb.com Sun May 23 15:11:00 2010 From: steve at holdenweb.com (Steve Holden) Date: Sun, 23 May 2010 09:11:00 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4BF916EA.2090507@holdenweb.com> Message-ID: <4BF92964.1040101@holdenweb.com> Jesse Noller wrote: > On Sun, May 23, 2010 at 7:52 AM, Steve Holden wrote: > ...snip... > >>>> Issues like the ones I'm bringing up could be fixed pretty >>>> straightforwardly if it were just a matter of filing a bug on a small >>>> package, but fixing a stdlib module is a major undertaking. >>> True but I don't think that is a convincing argument. A subset of the >>> functionality provided by this module is already available in Java and >>> C++ and (at least in Java) it is used extensively and without too much >>> trouble. If there are implementation bugs then we can fix them just like >>> we would with any other module. >>> >> I don't see the availability of this functionality in those languages as >> any kind of reason why this needs to go into the stdlib now. Is there >> some desperate rush to get it in? If it were used extensively from PyPi >> *that* would be a recommendation ... >> > > Not picking Steve's particular comments out - but Brian cites the > previous discussions in the PEP itself: > > http://www.python.org/dev/peps/pep-3148/ > > All of you questioning "Why should this be in the standard library" > should go read those old threads, where that question was answered > numerous times. Now I suddenly regret leaving the floodgates open, as > we're rapidly rehashing discussions from months ago. > Yes, it might have been better to call for participation from those who had contributed to the original discussion, and therefore knew what they were talking about. No flood from me, though, all my questions have been answered. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ "All I want for my birthday is another birthday" - Ian Dury, 1942-2000 From tjreedy at udel.edu Sun May 23 21:01:59 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 23 May 2010 15:01:59 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <2EF114F8-585B-4204-A7ED-1FE7F8226D2D@sweetapp.com> Message-ID: On 5/22/2010 8:06 PM, Jeffrey Yasskin wrote: > On Sat, May 22, 2010 at 4:12 PM, Brian Quinlan wrote: >> Rename "executor" => "executer" > > -1 for consistency with Java. -10 for 'executer'. As far as I can find out, it is a misspelling of 'executor'. If the designers of some other language made a stupid mistake, let them correct it instead of us following them over a cliff. Unlike this one, the other name suggestions look at least plausible and worth a couple of minutes of consideration. As for the module itself, part of the justification to me for accepting it would be if it is part of a larger plan, even if currently vague, to refactor and improve Python's support for concurrent execution, as implied by the name 'concurrent.futures'. If Jesse accepts it, I would take it as some kind of commitment to help with at least one other concurrent.x module so this one is not an orphan. While concurrent execution does not *currently* affect me, I am convinced that better support will be important for Python's future. Terry Jan Reedy From glyph at twistedmatrix.com Sun May 23 21:16:35 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Sun, 23 May 2010 15:16:35 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> Message-ID: <4FD0AFC2-F0FF-4190-867D-3A2C35FD850E@twistedmatrix.com> On May 23, 2010, at 2:37 AM, Brian Quinlan wrote: > On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote: > >> >> On May 22, 2010, at 8:47 PM, Brian Quinlan wrote: >> >>> Jesse, the designated pronouncer for this PEP, has decided to keep discussion open for a few more days. >>> >>> So fire away! >> >> As you wish! > > I retract my request ;-) May you get what you wish for, may you find what you are seeking :). >> The PEP should be consistent in its usage of terminology about callables. It alternately calls them "callables", "functions", and "functions or methods". It would be nice to clean this up and be consistent about what can be called where. I personally like "callables". > > Did you find the terminology confusing? If not then I propose not changing it. Yes, actually. Whenever I see references to the multiprocessing module, I picture a giant "HERE BE (serialization) DRAGONS" sign. When I saw that some things were documented as being "functions", I thought that maybe there was intended to be a restriction like the "these can only be top-level functions so they're easy for different executors to locate and serialize". I didn't realize that the intent was "arbitrary callables" until I carefully re-read the document and noticed that the terminology was inconsistent. > But changing it in the user docs is probably a good idea. I like "callables" too. Great. Still, users will inevitably find the PEP and use it as documentation too. >> The execution context of callable code is not made clear. Implicitly, submit() or map() would run the code in threads or processes as defined by the executor, but that's not spelled out clearly. Any response to this bit? Did I miss something in the PEP? >> More relevant to my own interests, the execution context of the callables passed to add_done_callback and remove_done_callback is left almost completely to the imagination. If I'm reading the sample implementation correctly, , it looks like in the multiprocessing implementation, the done callbacks are invoked in a random local thread. The fact that they are passed the future itself *sort* of implies that this is the case, but the multiprocessing module plays fast and loose with object identity all over the place, so it would be good to be explicit and say that it's *not* a pickled copy of the future sitting in some arbitrary process (or even on some arbitrary machine). > > The callbacks will always be called in a thread other than the main thread in the process that created the executor. Is that a strong enough contract? Sure. Really, almost any contract would work, it just needs to be spelled out. It might be nice to know whether the thread invoking the callbacks is a daemon thread or not, but I suppose it's not strictly necessary. >> This is really minor, I know, but why does it say "NOTE: This method can be used to create adapters from Futures to Twisted Deferreds"? First of all, what's the deal with "NOTE"; it's the only "NOTE" in the whole PEP, and it doesn't seem to add anything. This sentence would read exactly the same if that word were deleted. Without more clarity on the required execution context of the callbacks, this claim might not actually be true anyway; Deferred callbacks can only be invoked in the main reactor thread in Twisted. But even if it is perfectly possible, why leave so much of the adapter implementation up to the imagination? If it's important enough to mention, why not have a reference to such an adapter in the reference Futures implementation, since it *should* be fairly trivial to write? > > I'm a bit surprised that this doesn't allow for better interoperability with Deferreds given this discussion: > I did not communicate that well. As implemented, it's quite possible to implement a translation layer which turns a Future into a Deferred. What I meant by that comment was, the specification in the PEP was to loose to be sure that such a layer would work with arbitrary executors. For what it's worth, the Deferred translator would look like this, if you want to include it in the PEP (untested though, you may want to run it first): from twisted.internet.defer import Deferred from twisted.internet.reactor import callFromThread def future2deferred(future): d = Deferred() def invoke_deferred(): try: result = future.result() except: d.errback() else: d.callback(result) def done_callback(same_future): callFromThread(invoke_deferred) future.add_done_callback(done_callback) return d This does beg the question of what the traceback will look like in that except: block though. I guess the multi-threaded executor will use python3 exception chaining so Deferred should be able to show a sane traceback in case of an error, but what about exceptions in other processes? >> I suggest having have add_done_callback, implementing it with a list so that callbacks are always invoked in the order that they're added, and getting rid of remove_done_callback. > > Sounds good to me! Great! :-) >> futures._base.Executor isn't exposed publicly, but it needs to be. The PEP kinda makes it sound like it is ("Executor is an abstract class..."). Plus, A third party library wanting to implement an executor of its own shouldn't have to copy and paste the implementation of Executor.map. > > That was a bug that I've fixed. Thanks! Double-great! >> One minor suggestion on the "internal future methods" bit - something I wish we'd done with Deferreds was to put 'callback()' and 'addCallbacks()' on separate objects, so that it was very explicit whether you were on the emitting side of a Deferred or the consuming side. That seems to be the case with these internal methods - they are not so much "internal" as they are for the producer of the Future (whether a unit test or executor) so you might want to put them on a different object that it's easy for the thing creating a Future() to get at but hard for any subsequent application code to fiddle with by accident. Off the top of my head, I suggest naming it "Invoker()". A good way to do this would be to have an Invoker class which can't be instantiated (raises an exception from __init__ or somesuch), then a Future.create() method which returns an Invoker, which itself has a '.future' attribute. No reaction on this part? I think you'll wish you did this in a couple of years when you start bumping into application code that calls "set_result" :). >> Finally, why isn't this just a module on PyPI? It doesn't seem like there's any particular benefit to making this a stdlib module and going through the whole PEP process - except maybe to prompt feedback like this :). > > We've already had this discussion before. Could you explain why this module should *not* be in the stdlib e.g. does it have significantly less utility than other modules in stdlib? Is it significantly higher risk? etc? You've convinced me, mainly because I noticed later on in the discussion that it *has* been released to pypi for several months, and does have a bunch of downloads. It doesn't have quite the popularity I'd personally like to see for stdlib modules, but it's not like you didn't try, and you do (sort of) have a point about small modules being hard to get adoption. I'm sorry that this, my least interesting point in my opinion, is what has seen the most discussion so far. I'd appreciate it if you could do a release to pypi with the bugfixes you mentioned here, to make sure that the released version is consistent with what eventually gets into Python. Oh, and one final nitpick: says you really should not put real domain names into your "web crawl example", especially not "some-made-up-domain.com". -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon May 24 01:33:47 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 24 May 2010 11:33:47 +1200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> Message-ID: <4BF9BB5B.50809@canterbury.ac.nz> Glyph Lefkowitz wrote: > Finally, why isn't this just a module on PyPI? It doesn't seem like > there's any particular benefit to making this a stdlib module and going > through the whole PEP process I'm inclined to agree. This needs to be field-tested before being considered for stdlib inclusion. -- Greg From greg.ewing at canterbury.ac.nz Mon May 24 01:51:26 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 24 May 2010 11:51:26 +1200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> Message-ID: <4BF9BF7E.5030300@canterbury.ac.nz> Brian Quinlan wrote: > The good news in this case is that the same API has been used > successfully in Java and C++ for years so it is unlikely that any major > changes will need to be made. That doesn't follow. An API that's appropriate for Java or C++ is not necessarily appropriate for Python. Slavishly copying an API from another language is often not the best approach when designing an API for a Python module. -- Greg From greg.ewing at canterbury.ac.nz Mon May 24 02:00:05 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 24 May 2010 12:00:05 +1200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <7251439E-F687-490C-A7A5-29018ECC3861@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <7251439E-F687-490C-A7A5-29018ECC3861@sweetapp.com> Message-ID: <4BF9C185.5010200@canterbury.ac.nz> Brian Quinlan wrote: > Simple modules are unlikely to develop a following because it is too > easy to partially replicate their functionality. I don't think it needs a particularly large following. What it does need is at least a few people using it in some real projects. No matter how much discussion there is and how much apparent agreement is reached, it's no substitute for practical experience. Often API design mistakes are only found when trying to use the library for real. -- Greg From greg.ewing at canterbury.ac.nz Mon May 24 02:06:33 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 24 May 2010 12:06:33 +1200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <7251439E-F687-490C-A7A5-29018ECC3861@sweetapp.com> Message-ID: <4BF9C309.1020300@canterbury.ac.nz> Andrew Svetlov wrote: > BTW, is 'cancelled' correct name? Spell-checkers likes only single 'l' > form: 'canceled'. I think this is an English vs. American thing. Double 'l' looks right to me, but then I was brought up as a loyal subject of the antipodean branch of the British Empire. :-) -- Greg From stephen at xemacs.org Mon May 24 03:47:34 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 24 May 2010 10:47:34 +0900 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> Message-ID: <874ohym5op.fsf@uwakimon.sk.tsukuba.ac.jp> Brian Quinlan writes: > If you are familiar with threads then writing a "good enough" solution > without futures probably won't take you very long. Also, unless you > are familiar with another futures implementation, you aren't likely to > know where to look. That looks like an argument *against* your module, to me. Why would people look for it in the stdlib if they're not looking for it at all, and specifically because anybody who would know enough to look for "something like" it is also able to devise a good-enough solution? You're describing a solution in search of a user, not a user in search of a solution, and it would appear to violate "not every three-line function" as well as TOOWTDI. I personally plan to defer to the people who know and use such constructs (specifically Glyph and Jesse), and who seem to be in favor (at least +0) of stabilizing an API for this in the stdlib. But you may want to rethink your sales pitch if you want to avoid giving ammo to the opposition. It sounds like you agree with them, except on the vote you cast. From cs at zip.com.au Mon May 24 04:58:05 2010 From: cs at zip.com.au (Cameron Simpson) Date: Mon, 24 May 2010 12:58:05 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <874ohym5op.fsf@uwakimon.sk.tsukuba.ac.jp> References: <874ohym5op.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20100524025805.GA29197@cskk.homeip.net> On 24May2010 10:47, Stephen J. Turnbull wrote: | Brian Quinlan writes: | > If you are familiar with threads then writing a "good enough" solution | > without futures probably won't take you very long. Also, unless you | > are familiar with another futures implementation, you aren't likely to | > know where to look. | | That looks like an argument *against* your module, to me. Why would | people look for it in the stdlib if they're not looking for it at all, | and specifically because anybody who would know enough to look for | "something like" it is also able to devise a good-enough solution? | You're describing a solution in search of a user, not a user in search | of a solution, and it would appear to violate "not every three-line | function" as well as TOOWTDI. This might be a terminology problem. I think, above, Brian means "good enough" to mean "looks ok at first cut but doesn't handle the corner cases". Which usually means obscure breakage later. I almost am Brian's hypothetical user. I've got a "FuncMultiQueue" that accepts callables in synchronous and asynchronous modes for future possibly-concurrent execution, just as the futures module does. I've spent a _lot_ of time debugging it. There's a lot to be said for a robust implementation of a well defined problem. Brian's module, had it been present and presuming it robust and debugged, would have been quite welcome. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ I am a Bear of Very Little Brain and long words Bother Me. - Winnie-the-Pooh From steven.bethard at gmail.com Mon May 24 05:52:19 2010 From: steven.bethard at gmail.com (Steven Bethard) Date: Sun, 23 May 2010 20:52:19 -0700 Subject: [Python-Dev] moving issues from argparse tracker to python tracker? Message-ID: Before I go and add about 30 open issues to the Python tracker, I figured I should ask. What's the normal process for the bug trackers of modules that move to the standard library? I have a few feature requests, etc. for argparse, and I was planning to just copy them over to the Python bug tracker (and close them on the Google code tracker). Is this what people normally do? (It should be easy enough to do - I just don't want to mess up the tracker if this is usually done some other way.) Thanks, Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus From orsenthil at gmail.com Mon May 24 06:08:33 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Mon, 24 May 2010 09:38:33 +0530 Subject: [Python-Dev] moving issues from argparse tracker to python tracker? In-Reply-To: References: Message-ID: <20100524040832.GA5722@remy> On Sun, May 23, 2010 at 08:52:19PM -0700, Steven Bethard wrote: > requests, etc. for argparse, and I was planning to just copy them over > to the Python bug tracker (and close them on the Google code tracker). I think, this is a good idea. +1 from me. There is only one module, bsddb, which is maintained outside due to release constraints and other issues, but other than anything part of stdlib, better be tracked through python tracker. An additional advantage of moving to python tracker is, people who are subscribed to python-bugs will be notified and can pitch in their ideas/efforts too. -- Senthil OK, enough hype. -- Larry Wall in the perl man page From p.f.moore at gmail.com Mon May 24 09:17:21 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 24 May 2010 08:17:21 +0100 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <20100524025805.GA29197@cskk.homeip.net> References: <874ohym5op.fsf@uwakimon.sk.tsukuba.ac.jp> <20100524025805.GA29197@cskk.homeip.net> Message-ID: On 24 May 2010 03:58, Cameron Simpson wrote: > I almost am Brian's hypothetical user. I've got a "FuncMultiQueue" that > accepts callables in synchronous and asynchronous modes for future > possibly-concurrent execution, just as the futures module does. I've > spent a _lot_ of time debugging it. I pretty much am that user as well (whether or not I am hypothetical, I'll leave to others to determine...) I have a set of scripts that needed to do precisely the sort of thing that the futures module offers. I searched for a fair while for a suitable offering (this was before futures had been published) and found nothing suitable. So in the end I implemented my own - and I hit corner cases, and they needed a lot of work to fix. I now have a working solution, but it's too tangled in the application logic to be reusable :-( If futures had been in the stdlib, I'd have used it like a shot, and saved myself a lot of wasted time. > There's a lot to be said for a robust implementation of a well defined > problem. Brian's module, had it been present and presuming it robust and > debugged, would have been quite welcome. Precisely my view. Paul. From g.brandl at gmx.net Mon May 24 09:24:32 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 24 May 2010 09:24:32 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BF9BF7E.5030300@canterbury.ac.nz> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BF9BF7E.5030300@canterbury.ac.nz> Message-ID: Am 24.05.2010 01:51, schrieb Greg Ewing: > Brian Quinlan wrote: > >> The good news in this case is that the same API has been used >> successfully in Java and C++ for years so it is unlikely that any major >> changes will need to be made. > > That doesn't follow. An API that's appropriate for Java or > C++ is not necessarily appropriate for Python. Slavishly > copying an API from another language is often not the best > approach when designing an API for a Python module. *cough* unittest *cough* Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From vinay_sajip at yahoo.co.uk Mon May 24 10:49:56 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 24 May 2010 08:49:56 +0000 (UTC) Subject: [Python-Dev] urllib2/urllib incompatibility? Message-ID: > Not top-posting, but gmane seems too finicky in this area. I encountered what seems like an incompatibility between urllib and urllib2 in the way they handle file:// URLs, is this a bug? I had a look on the bug tracker and via Google to see if there were prior reports, but perhaps my search-fu is deficient. Here's a console session to illustrate: vinay at eta-karmic:/tmp$ echo Hello, world! >hello.txt vinay at eta-karmic:/tmp$ cat hello.txt Hello, world! vinay at eta-karmic:/tmp$ python Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15) [GCC 4.4.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import urllib,urllib2 >>> s = 'file:////tmp/hello.txt' >>> f1 = urllib.urlopen(s) >>> f1.read() 'Hello, world!\n' >>> f2 = urllib2.urlopen(s) Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.6/urllib2.py", line 124, in urlopen return _opener.open(url, data, timeout) File "/usr/lib/python2.6/urllib2.py", line 389, in open response = self._open(req, data) File "/usr/lib/python2.6/urllib2.py", line 407, in _open '_open', req) File "/usr/lib/python2.6/urllib2.py", line 367, in _call_chain result = func(*args) File "/usr/lib/python2.6/urllib2.py", line 1240, in file_open return self.parent.open(req) File "/usr/lib/python2.6/urllib2.py", line 389, in open response = self._open(req, data) File "/usr/lib/python2.6/urllib2.py", line 407, in _open '_open', req) File "/usr/lib/python2.6/urllib2.py", line 367, in _call_chain result = func(*args) File "/usr/lib/python2.6/urllib2.py", line 1287, in ftp_open raise URLError('ftp error: no host given') urllib2.URLError: >>> Anyone seen this before? Should I file an issue on the tracker? If I've missed something obvious, sorry for the noise, but please do tell! Regards, Vinay Sajip From list at qtrac.plus.com Mon May 24 11:30:59 2010 From: list at qtrac.plus.com (Mark Summerfield) Date: Mon, 24 May 2010 10:30:59 +0100 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> Message-ID: <201005241030.59713.list@qtrac.plus.com> On 2010-05-23, Terry Reedy wrote: > On 5/22/2010 8:06 PM, Jeffrey Yasskin wrote: > > On Sat, May 22, 2010 at 4:12 PM, Brian Quinlan wrote: > >> Rename "executor" => "executer" > > > > -1 for consistency with Java. > > -10 for 'executer'. As far as I can find out, it is a misspelling of > 'executor'. If the designers of some other language made a stupid > mistake, let them correct it instead of us following them over a cliff. I'd suggested this because it seemed obvious to me, but clearly not. Compare: http://www.thefreedictionary.com/executor http://www.thefreedictionary.com/executer However, as I mentioned in the first place I didn't expect any change of this since Java uses the first spelling. [snip] -- Mark Summerfield, Qtrac Ltd, www.qtrac.eu C++, Python, Qt, PyQt - training and consultancy "Advanced Qt Programming" - ISBN 0321635906 From brian at sweetapp.com Mon May 24 11:36:50 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Mon, 24 May 2010 19:36:50 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4FD0AFC2-F0FF-4190-867D-3A2C35FD850E@twistedmatrix.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4FD0AFC2-F0FF-4190-867D-3A2C35FD850E@twistedmatrix.com> Message-ID: <64B5ACEE-0B5C-40A8-AEC8-4D40BC7FE65B@sweetapp.com> On May 24, 2010, at 5:16 AM, Glyph Lefkowitz wrote: > > On May 23, 2010, at 2:37 AM, Brian Quinlan wrote: > >> On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote: >> >>> >>> On May 22, 2010, at 8:47 PM, Brian Quinlan wrote: >>> >>>> Jesse, the designated pronouncer for this PEP, has decided to >>>> keep discussion open for a few more days. >>>> >>>> So fire away! >>> >>> As you wish! >> >> I retract my request ;-) > > May you get what you wish for, may you find what you are seeking :). > >>> The PEP should be consistent in its usage of terminology about >>> callables. It alternately calls them "callables", "functions", >>> and "functions or methods". It would be nice to clean this up and >>> be consistent about what can be called where. I personally like >>> "callables". >> >> Did you find the terminology confusing? If not then I propose not >> changing it. > > Yes, actually. Whenever I see references to the multiprocessing > module, I picture a giant "HERE BE (serialization) DRAGONS" sign. > When I saw that some things were documented as being "functions", I > thought that maybe there was intended to be a restriction like the > "these can only be top-level functions so they're easy for different > executors to locate and serialize". I didn't realize that the > intent was "arbitrary callables" until I carefully re-read the > document and noticed that the terminology was inconsistent. ProcessPoolExecutor has the same serialization perils that multiprocessing does. My original plan was to link to the multiprocessing docs to explain them but I couldn't find them listed. >> But changing it in the user docs is probably a good idea. I like >> "callables" too. > > Great. Still, users will inevitably find the PEP and use it as > documentation too. > >>> The execution context of callable code is not made clear. >>> Implicitly, submit() or map() would run the code in threads or >>> processes as defined by the executor, but that's not spelled out >>> clearly. > > Any response to this bit? Did I miss something in the PEP? Yes, the execution context is Executor-dependent. The section under ProcessPoolExecutor and ThreadPoolExecutor spells this out, I think. >>> More relevant to my own interests, the execution context of the >>> callables passed to add_done_callback and remove_done_callback is >>> left almost completely to the imagination. If I'm reading the >>> sample implementation correctly, >> >, it looks like in the multiprocessing implementation, the done >>> callbacks are invoked in a random local thread. The fact that >>> they are passed the future itself *sort* of implies that this is >>> the case, but the multiprocessing module plays fast and loose with >>> object identity all over the place, so it would be good to be >>> explicit and say that it's *not* a pickled copy of the future >>> sitting in some arbitrary process (or even on some arbitrary >>> machine). >> >> The callbacks will always be called in a thread other than the main >> thread in the process that created the executor. Is that a strong >> enough contract? > > Sure. Really, almost any contract would work, it just needs to be > spelled out. It might be nice to know whether the thread invoking > the callbacks is a daemon thread or not, but I suppose it's not > strictly necessary. Your concerns is that the thread will be killed when the interpreter exits? It won't be. >>> This is really minor, I know, but why does it say "NOTE: This >>> method can be used to create adapters from Futures to Twisted >>> Deferreds"? First of all, what's the deal with "NOTE"; it's the >>> only "NOTE" in the whole PEP, and it doesn't seem to add >>> anything. This sentence would read exactly the same if that word >>> were deleted. Without more clarity on the required execution >>> context of the callbacks, this claim might not actually be true >>> anyway; Deferred callbacks can only be invoked in the main reactor >>> thread in Twisted. But even if it is perfectly possible, why >>> leave so much of the adapter implementation up to the >>> imagination? If it's important enough to mention, why not have a >>> reference to such an adapter in the reference Futures >>> implementation, since it *should* be fairly trivial to write? >> >> I'm a bit surprised that this doesn't allow for better >> interoperability with Deferreds given this discussion: > >> > > I did not communicate that well. As implemented, it's quite > possible to implement a translation layer which turns a Future into > a Deferred. What I meant by that comment was, the specification in > the PEP was to loose to be sure that such a layer would work with > arbitrary executors. > > For what it's worth, the Deferred translator would look like this, > if you want to include it in the PEP (untested though, you may want > to run it first): > > from twisted.internet.defer import Deferred > from twisted.internet.reactor import callFromThread > > def future2deferred(future): > d = Deferred() > def invoke_deferred(): > try: > result = future.result() > except: > d.errback() > else: > d.callback(result) > def done_callback(same_future): > callFromThread(invoke_deferred) > future.add_done_callback(done_callback) > return d > > This does beg the question of what the traceback will look like in > that except: block though. I guess the multi-threaded executor will > use python3 exception chaining so Deferred should be able to show a > sane traceback in case of an error, but what about exceptions in > other processes? > >>> I suggest having have add_done_callback, implementing it with a >>> list so that callbacks are always invoked in the order that >>> they're added, and getting rid of remove_done_callback. >> >> Sounds good to me! > > Great! :-) > >>> futures._base.Executor isn't exposed publicly, but it needs to >>> be. The PEP kinda makes it sound like it is ("Executor is an >>> abstract class..."). Plus, A third party library wanting to >>> implement an executor of its own shouldn't have to copy and paste >>> the implementation of Executor.map. >> >> That was a bug that I've fixed. Thanks! > > Double-great! > >>> One minor suggestion on the "internal future methods" bit - >>> something I wish we'd done with Deferreds was to put 'callback()' >>> and 'addCallbacks()' on separate objects, so that it was very >>> explicit whether you were on the emitting side of a Deferred or >>> the consuming side. That seems to be the case with these internal >>> methods - they are not so much "internal" as they are for the >>> producer of the Future (whether a unit test or executor) so you >>> might want to put them on a different object that it's easy for >>> the thing creating a Future() to get at but hard for any >>> subsequent application code to fiddle with by accident. Off the >>> top of my head, I suggest naming it "Invoker()". A good way to do >>> this would be to have an Invoker class which can't be instantiated >>> (raises an exception from __init__ or somesuch), then a >>> Future.create() method which returns an Invoker, which itself has >>> a '.future' attribute. > > No reaction on this part? I think you'll wish you did this in a > couple of years when you start bumping into application code that > calls "set_result" :). My reactions are mixed ;-) Your proposal is to add a level of indirection to make it harder for people to call implementation methods. The downside is that it makes it a bit harder to write tests and Executors. I also can't see a big problem in letting people call set_result in client code though it is documented as being only for Executor implementations and tests. On the implementation side, I don't see why an Invoker needs a reference to the future. Each Invoker could own one Future. A reference to the Invoker is kept by the Executor and its future is returned to the client i.e. class Invoker(object): def __init__(self): """Should only be called by Executor implementations.""" self.future = Future() def set_running_or_notify_cancel(self): # Messes with self.future's internals def set_result(self): # Messes with self.future's internals def set_exception(self): # Messes with self.future's internals Cheers, Brian >>> Finally, why isn't this just a module on PyPI? It doesn't seem >>> like there's any particular benefit to making this a stdlib module >>> and going through the whole PEP process - except maybe to prompt >>> feedback like this :). >> >> We've already had this discussion before. Could you explain why >> this module should *not* be in the stdlib e.g. does it have >> significantly less utility than other modules in stdlib? Is it >> significantly higher risk? etc? > > You've convinced me, mainly because I noticed later on in the > discussion that it *has* been released to pypi for several months, > and does have a bunch of downloads. It doesn't have quite the > popularity I'd personally like to see for stdlib modules, but it's > not like you didn't try, and you do (sort of) have a point about > small modules being hard to get adoption. I'm sorry that this, my > least interesting point in my opinion, is what has seen the most > discussion so far. > > I'd appreciate it if you could do a release to pypi with the > bugfixes you mentioned here, to make sure that the released version > is consistent with what eventually gets into Python. > > Oh, and one final nitpick: rfc2606.txt> says you really should not put real domain names into > your "web crawl example", especially not "some-made-up-domain.com". > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From orsenthil at gmail.com Mon May 24 11:37:16 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Mon, 24 May 2010 15:07:16 +0530 Subject: [Python-Dev] urllib2/urllib incompatibility? In-Reply-To: References: Message-ID: <20100524093716.GA5284@remy> On Mon, May 24, 2010 at 08:49:56AM +0000, Vinay Sajip wrote: > I encountered what seems like an incompatibility between urllib and urllib2 in > the way they handle file:// URLs, is this a bug? I had a look on the bug tracker > >>> s = 'file:////tmp/hello.txt' > >>> f1 = urllib.urlopen(s) The actual (and Valid) file:// url in your case is 'file:///tmp/hello.txt' And this was fine and consistent. >>> s = 'file:///tmp/hello.txt' >>> import urllib2 >>> import urllib >>> o1 = urllib.urlopen(s) >>> o2 = urllib2.urlopen(s) The extra '/' is making it in invalid url in urllib2, I think that be tracked as bug at least to show a consistent behaviour. The local file open methods of urllib2 and urllib are different. -- Senthil You may my glories and my state dispose, But not my griefs; still am I king of those. -- William Shakespeare, "Richard II" From vinay_sajip at yahoo.co.uk Mon May 24 12:00:07 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 24 May 2010 10:00:07 +0000 (UTC) Subject: [Python-Dev] urllib2/urllib incompatibility? References: <20100524093716.GA5284@remy> Message-ID: Senthil Kumaran gmail.com> writes: > > The actual (and Valid) file:// url in your case is 'file:///tmp/hello.txt' > And this was fine and consistent. > > The extra '/' is making it in invalid url in urllib2, I think that be > tracked as bug at least to show a consistent behaviour. The local > file open methods of urllib2 and urllib are different. > Yes, it seems to be a bug in urllib which permits an invalid URL to pass. I came across it by accident, I normally use urls like file://localhost/tmp/hello.txt so I don't trip over the /// hurdles :-) I'll raise an issue on the tracker. Regards, Vinay Sajip From stephen at xemacs.org Mon May 24 12:46:02 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 24 May 2010 19:46:02 +0900 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <20100524025805.GA29197@cskk.homeip.net> References: <874ohym5op.fsf@uwakimon.sk.tsukuba.ac.jp> <20100524025805.GA29197@cskk.homeip.net> Message-ID: <871vd1mvbp.fsf@uwakimon.sk.tsukuba.ac.jp> Cameron Simpson writes: > There's a lot to be said for a robust implementation of a well defined > problem. Brian's module, had it been present and presuming it robust and > debugged, would have been quite welcome. That, of course, is the consensus view, both in general and with respect to this particular module. The difference is over what constitutes sufficient evidence for your presumption of "robust and debugged" from the point of view of the users of the stdlib. From solipsis at pitrou.net Mon May 24 13:01:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 24 May 2010 13:01:33 +0200 Subject: [Python-Dev] moving issues from argparse tracker to python tracker? References: Message-ID: <20100524130133.43438a0e@pitrou.net> On Sun, 23 May 2010 20:52:19 -0700 Steven Bethard wrote: > I have a few feature > requests, etc. for argparse, and I was planning to just copy them over > to the Python bug tracker (and close them on the Google code tracker). Yes, I think this is desireable. You should also maintain argparse directly in the Python SVN (even if you plan to do separate releases). Regards Antoine. From brett at python.org Tue May 25 04:57:56 2010 From: brett at python.org (Brett Cannon) Date: Mon, 24 May 2010 19:57:56 -0700 Subject: [Python-Dev] __package__ attribute In-Reply-To: References: Message-ID: On Fri, May 21, 2010 at 14:35, Andrew Svetlov wrote: > For me it's the real bug in standard python import machinery. > I don't see any backward incompatibilities. > There are very hard to write any import-depended code based on > decision: was module imported in absolute or relative way. > > If it's a bug I can create issue in python bugtracker and provide a > patch (for Python 3.2 and for 2.7 if later is required). It's definitely not a bug. The definition of __package__ is such that not bothering to set it is still valid (see PEP 366 for the exact details of the attribute). It just happens that importlib puts in the extra effort to add/set the attribute as much as possible. Now if you want to write a patch for import.c to add the attribute properly then that could get reviewed and added as a feature request, but as it stands the current semantics are correct. -Brett > > On Fri, May 21, 2010 at 10:18 PM, Andrew Svetlov > wrote: >> What are expected values for module.__package__? >> I see two different behaviors: importlib and standard python import. >> importlib processes __package__ pretty simple and clean: >> - for toplevel modules __package__ is empty string >> - for packages it's package name >> - for modules inside packages it's again package name >> >> Standard import follows another strategy: >> - it tries to setup __package__ only in case of relative import (see >> first 'if' in import.c:get_parent) >> - otherwise __package__ is untouched and None by default. >> >> For me importlib's way is better. >> I don't see any PEP specifying __package__ behavior. >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From ncoghlan at gmail.com Wed May 26 01:54:13 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 May 2010 09:54:13 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <20100523144754.1c6f1afc@pitrou.net> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4BF916EA.2090507@holdenweb.com> <20100523144754.1c6f1afc@pitrou.net> Message-ID: <4BFC6325.5050700@gmail.com> On 23/05/10 22:47, Antoine Pitrou wrote: > On Sun, 23 May 2010 08:34:22 -0400 > Jesse Noller wrote: >> >> Brian has already agreed to name spacing it to "concurrent.futures" - >> this means it will be a small part to a much larger concurrent.* >> implementation ala Java. > > What I would question here is what other things will be part > of the "concurrent" package, and who will implement them. Are there > plans for that? (or even tracker issues open?) I'm not sure it is called out explicitly in the PEP, but the specific example that came up in the previous discussions was something like "concurrent.pool" to hold a thread vs process agnostic worker pool interface based on the existing Pool interface in multiprocessing (with concrete implementations for both threading and multiprocessing). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed May 26 02:10:25 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 May 2010 10:10:25 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> Message-ID: <4BFC66F1.3060107@gmail.com> On 23/05/10 21:56, Lennart Regebro wrote: > On Sun, May 23, 2010 at 13:29, Brian Quinlan wrote: >> Parts of it, yes. Just like I can replace most operations in os.path and >> urlparse with a few lines of code. > > Yeah, but "parts of" is not the question. I've read the PEP, and I do > *not* know how to implement it. That means it's not a trivial module, > so that argument doesn't hold up here, even if we accept it as valid > (which I actually don't). I don't think any module in the stdlib is > entirely trivial. Yes, even parsing an URL is non-trivial, as shown by > the fact that the urlparse module apparently has a bug in it for urls > like svn+ssh://foo.bar/frotz. ;-) In this case, the "trivial" refers to being able to write something that will get the job done for a specific task or application, but that isn't as solid from a reliability/maintainability/portability/scalability point of view. By providing solid infrastructure in the standard library, we can remove that choice between "do it fast" vs "do it right", by providing ready-made robust infrastructure. Those that say "just put it on PyPI" may not recognise the additional overhead that can be involved in identifying, obtaining approval to use and ongoing management of additional dependencies in a corporate environment that is actually performing appropriate due diligence in regards to IP licensing. This overhead can be especially significant (and, depending on licence and contract details, even a dealbreaker) for companies with specific IP licensing provisions in their contracts with their clients. It doesn't matter *how* easy we make it to download PyPI packages, we can't do anything about such company IP management policies (except for making it easier for programmers to violate them thoughtlessly, of course). To use myself as an example, I have several utilities that I probably would have written differently if the futures module had been available in the standard library at the time I wrote them. As it is, they work well enough, but their usage of the threading module is fairly ad hoc (and migrating them to use multiprocessing would be a fairly complex task, and making that decision run-time selectable even more complicated). In the near-term, backports of future standard library modules are much easier to get through a corporate review process as the licensing is typically similar to the PSF license (or is even the PSF license itself) and the modules come with a clear roadmap for eliminating the dependency (i.e. once the baseline Python version employed by the company includes the module in the standard library, the external dependency is no longer needed). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed May 26 02:12:43 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 May 2010 10:12:43 +1000 Subject: [Python-Dev] Dead modules In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <20100523125155.53959f2f@pitrou.net> Message-ID: <4BFC677B.9060107@gmail.com> (Sending again - I didn't mean to drop python-dev from the cc list when I originally sent this via the gmail web interface) On Sun, May 23, 2010 at 9:00 PM, Dirkjan Ochtman > wrote: Right, it wasn't intended as that harsh... but it does come with a rather impressive set of constraints in terms of what you can do with the API. True, but in some cases (especially low level infrastructure), it is worth accepting those constraints in order to achieve other aims (such as standardisation of techniques). Things like itertools, collections, functools, unittest owe their existence largely to the choice of gains in standardisation over flexibility of API updates. Besides, popular PyPI modules don't have that much more freedom than the stdlib when it comes to API changes. The only real difference is that the 18-24 month release cycle for the stdlib is a lot *slower* than that of many PyPI packages, so feedback on any changes we make is correspondingly delayed. Hence the existence of projects like distutils2 and unittest2 to enable that faster feedback cycle to inform the updates passed back into the more slowly evolving stdlib modules, as well as the desire to copy prior art wherever it makes sense to do so (whether that is other languages, existing PyPI modules or the internal code bases of large corporate contributors). Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed May 26 02:25:22 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 May 2010 10:25:22 +1000 Subject: [Python-Dev] Documenting [C]Python's Internals In-Reply-To: References: Message-ID: <4BFC6A72.2060608@gmail.com> On 20/05/10 08:13, Yaniv Aknin wrote: > Hi, > > I wanted to let python-dev know about a series of articles about > CPython's internals I'm publishing under the collective title "Guido's > Python"* (http://tech.blog.aknin.name/tag/guidos-python/). A resource that may be useful to you is a 2.5 focused manuscript I put together a few years ago trying to bridge the gap between the library reference and the language reference: http://svn.python.org/projects/sandbox/trunk/userref/ It's obviously a little dated in some areas and doesn't delve as deeply into the source code as you apparently plan to, but hopefully it may prove useful as a resource (I still have vague intentions of exporting that document to ReST markup and updating it to current Python, but that doesn't look like actually happening any time soon) Cheers, Nick. P.S. For the record, the relevant URL is now http://tech.blog.aknin.name/tag/pythons-innards/ -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From jnoller at gmail.com Wed May 26 02:42:05 2010 From: jnoller at gmail.com (Jesse Noller) Date: Tue, 25 May 2010 20:42:05 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFC6325.5050700@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4BF916EA.2090507@holdenweb.com> <20100523144754.1c6f1afc@pitrou.net> <4BFC6325.5050700@gmail.com> Message-ID: On Tue, May 25, 2010 at 7:54 PM, Nick Coghlan wrote: > On 23/05/10 22:47, Antoine Pitrou wrote: >> >> On Sun, 23 May 2010 08:34:22 -0400 >> Jesse Noller ?wrote: >>> >>> Brian has already agreed to name spacing it to "concurrent.futures" - >>> this means it will be a small part to a much larger concurrent.* >>> implementation ala Java. >> >> What I would question here is what other things will be part >> of the "concurrent" package, and who will implement them. Are there >> plans for that? (or even tracker issues open?) > > I'm not sure it is called out explicitly in the PEP, but the specific > example that came up in the previous discussions was something like > "concurrent.pool" to hold a thread vs process agnostic worker pool interface > based on the existing Pool interface in multiprocessing (with concrete > implementations for both threading and multiprocessing). > Nick is correct - there's plenty of things in multiprocessing which belong in a more abstract package as they're useful for more things than just multiprocessing. I don't think they need to be called out as part of the PEP though. jesse From regebro at gmail.com Wed May 26 04:29:05 2010 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 26 May 2010 04:29:05 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFC66F1.3060107@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> Message-ID: On Wed, May 26, 2010 at 02:10, Nick Coghlan wrote: > Those that say "just put it on PyPI" may not recognise the additional ... Just a note, so we don't get sidelined by misunderstandings: I don't think anybody said that. ;-) There are two issues here, one generic and one specific: Generic: Modules should go on PyPI first, for a time, to stabilize (and so they can be used in earlier versions of Python) before they end up in stdlib. I suspect everyone actually agrees on that (but I could be wrong). Specific:Has futures been long enough on PyPI, and is it stable? I'm staying out of that discussion. :-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From ncoghlan at gmail.com Wed May 26 05:34:35 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 May 2010 13:34:35 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> Message-ID: <4BFC96CB.9090600@gmail.com> On 26/05/10 12:29, Lennart Regebro wrote: > On Wed, May 26, 2010 at 02:10, Nick Coghlan wrote: >> Those that say "just put it on PyPI" may not recognise the additional ... > > Just a note, so we don't get sidelined by misunderstandings: I don't > think anybody said that. ;-) Nah, that pseudo-quote wasn't from this discussion in particular. It's a reference to the ongoing tension between the "batteries included" advocates and the "make the standard library as streamlined as possible" crowd. Both sides have valid points, so the "included battery" vs "optional download" question needs to be decided on a case-by-case basis. > There are two issues here, one generic and one specific: > > Generic: Modules should go on PyPI first, for a time, to stabilize > (and so they can be used in earlier versions of Python) before they > end up in stdlib. I suspect everyone actually agrees on that (but I > could be wrong). That's the point I'm disagreeing with. For most modules it makes sense to do things that way, but for some low-level infrastructure elements, it is going to be less effective (because people will quickly throw together their own solutions instead of adding a new dependency for something "simple"). Other times we'll invent a new module because *we* need it for something (e.g. runpy). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed May 26 05:38:33 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 May 2010 13:38:33 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <871vd1mvbp.fsf@uwakimon.sk.tsukuba.ac.jp> References: <874ohym5op.fsf@uwakimon.sk.tsukuba.ac.jp> <20100524025805.GA29197@cskk.homeip.net> <871vd1mvbp.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4BFC97B9.3080901@gmail.com> On 24/05/10 20:46, Stephen J. Turnbull wrote: > Cameron Simpson writes: > > > There's a lot to be said for a robust implementation of a well defined > > problem. Brian's module, had it been present and presuming it robust and > > debugged, would have been quite welcome. > > That, of course, is the consensus view, both in general and with > respect to this particular module. > > The difference is over what constitutes sufficient evidence for your > presumption of "robust and debugged" from the point of view of the > users of the stdlib. At the very least, we'll be offering a promise to be "more robust and more debugged than what you came up with in that coding marathon last night" ;) Having a decent test suite that is regularly executed on multiple platforms (which will be the case for any accepted module by the time it is included in a Python release) also places anything we release a cut above a *lot* of in-house code. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From stephen at xemacs.org Wed May 26 05:51:07 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 26 May 2010 12:51:07 +0900 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFC66F1.3060107@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> Message-ID: <87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > Those that say "just put it on PyPI" Nobody is saying that, AFAICS. Nobody is saying that *some* futures module shouldn't *eventually* go into the stdlib. The question is whether *this* futures module should go into the stdlib *now*. And it is clear that more time on PyPI would provide valuable information. This is a general principle that has served us well: put best current practice backed up by actual widespread usage in the stdlib, not theoretical wins based on the developer's experience. PyPI is a way to broaden usage to determine BCP, not an end in itself. People have been asking "what's special about this module, to violate the BCP principle?" There's nothing special about the fact that several people would use a "robust and debugged" futures module if it were in the stdlib. That's true of *every* module that is worth a PEP. But remember, in the case of ipaddr it was the people who wanted some such module badly who were also the most vocal opponents, because they could see offhand that it was going to serve their use cases badly. (It turned out that this was equally trivial to fix despite a week of hot debate, and everyone lived happily ever after. But that was smiling Luck, not inexorable Fate.) For this module, three people have said "I 'would have' used it if it were available," but none of you has announced that you've started refactoring and the PEP 3148 API meets all expectations. I call that "damning with faint praise". OTOH, Glyph has changed from "why not more time on PyPI?" to "let's see if we can improve this a bit, then let's do it". He has published code (showing how to turn futures into Twisted Deferreds), and argues that based on download stats to date and the nature of the use cases it would take a lot of time on PyPI to demonstrate a BCP. Those are good arguments for an exception, IMHO. From ncoghlan at gmail.com Wed May 26 06:22:01 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 May 2010 14:22:01 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4BFCA1E9.1070808@gmail.com> On 26/05/10 13:51, Stephen J. Turnbull wrote: > People have been asking "what's special about this module, to violate > the BCP principle?" There's nothing special about the fact that > several people would use a "robust and debugged" futures module if it > were in the stdlib. That's true of *every* module that is worth a > PEP. I actually wrote a reply to that question earlier in the week, but failed at using gmail's web interface correctly and only sent it to Steve Holden. =================== The trick with futures and executor pools is that they're a *better* way of programming with threads in many cases. However, given the choices of: - hack together something simple with some worker Threads and a Queue (or two) - write my own futures and executor infrastructure - download a futures module from PyPI and live with the additional dependency I'll choose the first option every time, and my programs will be the worse for it. Put the capability to use futures and an executor into the stdlib, and it becomes something I can reach for without having to worry about additional dependencies beyond specifying a minimal Python version. It provides a higher level API that can be more readily switched between threading and multiprocessing back ends. It becomes something that can be taught as a standard Python technique for enabling concurrency in imperative code. This is something that is irrelevant to me as a module on PyPI, but has the potential to significantly affect my programming in the future as a standard library module. Even in the near term, backports of future standard library modules are often perceived differently when being discussed as potential additional dependencies for an application (i.e. I believe it would be worthwhile for a backport of the module to earlier Python versions than 3.2 to be made available on PyPI). =================== Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From regebro at gmail.com Wed May 26 09:11:58 2010 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 26 May 2010 09:11:58 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFCA1E9.1070808@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp> <4BFCA1E9.1070808@gmail.com> Message-ID: On Wed, May 26, 2010 at 06:22, Nick Coghlan wrote: > - download a futures module from PyPI and live with the additional > dependency Why would that be a problem? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From p.f.moore at gmail.com Wed May 26 09:37:04 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 26 May 2010 08:37:04 +0100 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp> <4BFCA1E9.1070808@gmail.com> Message-ID: On 26 May 2010 08:11, Lennart Regebro wrote: > On Wed, May 26, 2010 at 06:22, Nick Coghlan wrote: >> - download a futures module from PyPI and live with the additional >> dependency > > Why would that be a problem? That has been hashed out repeatedly on this and other lists. Can it please be stipulated that for *some* people, in *some* cases, it is a problem? It seems to me that if you've experienced the sort of culture that makes it a problem, you understand the point immediately, but if you haven't, you never will (that's not disparaging anyone, the idiosyncracies of corporate culture are widespread and bizarre - if it helps, just remember that Dilbert is a documentary :-)) Paul. From solipsis at pitrou.net Wed May 26 09:38:20 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 May 2010 09:38:20 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4BF916EA.2090507@holdenweb.com> <20100523144754.1c6f1afc@pitrou.net> <4BFC6325.5050700@gmail.com> Message-ID: <20100526093820.4fbf2f87@pitrou.net> On Wed, 26 May 2010 09:54:13 +1000 Nick Coghlan wrote: > > > > What I would question here is what other things will be part > > of the "concurrent" package, and who will implement them. Are there > > plans for that? (or even tracker issues open?) > > I'm not sure it is called out explicitly in the PEP, but the specific > example that came up in the previous discussions was something like > "concurrent.pool" to hold a thread vs process agnostic worker pool > interface based on the existing Pool interface in multiprocessing > (with concrete implementations for both threading and > multiprocessing). Ha, I'm a bit surprised. Isn't it what "futures" already provides? (except that for some reason it insists on the "SomeExecutor" naming scheme) http://www.python.org/dev/peps/pep-3148/#processpoolexecutor Regards Antoine. From glyph at twistedmatrix.com Wed May 26 10:09:17 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 26 May 2010 04:09:17 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <64B5ACEE-0B5C-40A8-AEC8-4D40BC7FE65B@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4FD0AFC2-F0FF-4190-867D-3A2C35FD850E@twistedmatrix.com> <64B5ACEE-0B5C-40A8-AEC8-4D40BC7FE65B@sweetapp.com> Message-ID: <1AE1A9E7-F821-45FF-8396-62EE8297ADE3@twistedmatrix.com> On May 24, 2010, at 5:36 AM, Brian Quinlan wrote: > On May 24, 2010, at 5:16 AM, Glyph Lefkowitz wrote: >> On May 23, 2010, at 2:37 AM, Brian Quinlan wrote: >>> On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote: > ProcessPoolExecutor has the same serialization perils that multiprocessing does. My original plan was to link to the multiprocessing docs to explain them but I couldn't find them listed. Linking to the pickle documentation might be a good start. > Yes, the execution context is Executor-dependent. The section under ProcessPoolExecutor and ThreadPoolExecutor spells this out, I think. I suppose so. I guess I'm just looking for more precise usage of terminology. (This is a PEP, after all. It's a specification that multiple VMs may have to follow, not just some user documentation for a package, even if they'll *probably* be using your code in all cases.) I'd be happier if there were a clearer term than "calls" for the things being scheduled ("submissions"?), since the done callbacks aren't called in the subprocess for ProcessPoolExecutor, as we just discussed. >> Sure. Really, almost any contract would work, it just needs to be spelled out. It might be nice to know whether the thread invoking the callbacks is a daemon thread or not, but I suppose it's not strictly necessary. > > Your concerns is that the thread will be killed when the interpreter exits? It won't be. Good to know. Tell it to the PEP though, not me ;). >> No reaction on [invoker vs. future]? I think you'll wish you did this in a couple of years when you start bumping into application code that calls "set_result" :). > > My reactions are mixed ;-) Well, you are not obliged to take my advice, as long as I am not obliged to refrain from mocking you mercilessly if it happens that I was right in a couple of years ;-). > Your proposal is to add a level of indirection to make it harder for people to call implementation methods. The downside is that it makes it a bit harder to write tests and Executors. Both tests and executors will still create and invoke methods directly on one object; the only additional difficulty seems to be the need to type '.future' every so often on the executor/testing side of things, and that seems a cost well worth paying to avoid confusion over who is allowed to call those methods and when. > I also can't see a big problem in letting people call set_result in client code though it is documented as being only for Executor implementations and tests. > > On the implementation side, I don't see why an Invoker needs a reference to the future. Well, uh... > class Invoker(object): > def __init__(self): > """Should only be called by Executor implementations.""" > self.future = Future() ^ this is what I'd call a "reference to the future" -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Wed May 26 10:15:25 2010 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 26 May 2010 10:15:25 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp> <4BFCA1E9.1070808@gmail.com> Message-ID: On Wed, May 26, 2010 at 09:37, Paul Moore wrote: > It seems to me that if you've experienced the sort of culture that > makes it a problem, Ah, it's a culture problem. In a heterogenous world, every action will benefit some and hurt some. Another arbitrary corporate ruleset could also mean you might be stuck on ancient python versions, and might not see a new module added to stdlib in 3.2 until 2015 or so. Some corporations go through a lot of trouble to prevent their employees from doing their job. Pythons core developers can not and should not let that hinder *them* from doing what is best for Python. Decisions on inclusion in stdlib must be made on what benefits Python and it's users in general. Since even small mistakes in a stdlib module will hurt far more people than having to having the module mature on PyPI until the worst API issues and bugs are ironed out, it's clear to me that letting a module mature on PyPI before inclusion is the better policy here, although how long obviously must be decided on a case by case basis. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From glyph at twistedmatrix.com Wed May 26 10:25:18 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 26 May 2010 04:25:18 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp> <4BFCA1E9.1070808@gmail.com> Message-ID: On May 26, 2010, at 3:37 AM, Paul Moore wrote: > On 26 May 2010 08:11, Lennart Regebro wrote: >> On Wed, May 26, 2010 at 06:22, Nick Coghlan wrote: >>> - download a futures module from PyPI and live with the additional >>> dependency >> >> Why would that be a problem? > > That has been hashed out repeatedly on this and other lists. Can it > please be stipulated that for *some* people, in *some* cases, it is a > problem? Sure, but I for one fully support Lennart asking the question, because while in the short term this *is* a problem with packaging tools in the Python ecosystem, in the long term (as you do note) it's an organizational dysfunction that can be addressed with better tools. I think it would be bad to ever concede the point that sane factoring of dependencies and code re-use aren't worth it because some jerk in Accounting or System Operations wants you to fill out a requisition form for a software component that's free and liberally licensed anyway. To support the unfortunate reality that such jerks in such departments really do in fact exist, there should be simple tools to glom a set of small, nicely factored dependencies into a giant monolithic ball of crud that installs all at once, and slap a sticker on the side of it that says "I am only filling out your stupid form once, okay". This should be as distant as possible from the actual decision to package things in sensibly-sized chunks. In other words, while I kinda-sorta buy Brian's argument that having this module in easy reach will motivate more people to use a standard, tested idiom for parallelization, I *don't* think that the stdlib should be expanded simply to accommodate those who just don't want to install additional packages for anything. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Wed May 26 10:44:02 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 26 May 2010 17:44:02 +0900 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFCA1E9.1070808@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp> <4BFCA1E9.1070808@gmail.com> Message-ID: <87hblvkq7h.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > On 26/05/10 13:51, Stephen J. Turnbull wrote: > > People have been asking "what's special about this module, to violate > > the BCP principle?" There's nothing special about the fact that > > several people would use a "robust and debugged" futures module if it > > were in the stdlib. That's true of *every* module that is worth a > > PEP. > > The trick with futures and executor pools is that they're a *better* way > of programming with threads in many cases. and > However, given the choices of [...]. I'll choose the first option > every time, and my programs will be the worse for it. Again, nothing all that special about those; lots of proposed changes satisfy similar conditions. I don't think anyone denies the truth or applicability of those arguments. But are they enough? Really, what you're arguing is "now is better than never." Indeed, that is so. But you shouldn't forget that is immediately followed by "although never is often better than *right* now." From brian at sweetapp.com Wed May 26 10:52:44 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Wed, 26 May 2010 18:52:44 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <87hblvkq7h.fsf@uwakimon.sk.tsukuba.ac.jp> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp> <4BFCA1E9.1070808@gmail.com> <87hblvkq7h.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 26 May 2010, at 18:44, Stephen J. Turnbull wrote: > Nick Coghlan writes: >> On 26/05/10 13:51, Stephen J. Turnbull wrote: > >>> People have been asking "what's special about this module, to >>> violate >>> the BCP principle?" There's nothing special about the fact that >>> several people would use a "robust and debugged" futures module if >>> it >>> were in the stdlib. That's true of *every* module that is worth a >>> PEP. >> >> The trick with futures and executor pools is that they're a >> *better* way >> of programming with threads in many cases. > > and > >> However, given the choices of [...]. I'll choose the first option >> every time, and my programs will be the worse for it. > > Again, nothing all that special about those; lots of proposed changes > satisfy similar conditions. I don't think anyone denies the truth or > applicability of those arguments. But are they enough? > > Really, what you're arguing is "now is better than never." Indeed, > that is so. But you shouldn't forget that is immediately followed by > "although never is often better than *right* now." I've been trying to stay out of the meta-discussions but "*right* now" would be >6 months if it applies in this context. If that is what "*right* now" means to you then I hope that I never have a heart attack in your presence and need an ambulance *right* now :-) Cheers, Brian From brian at sweetapp.com Wed May 26 10:55:42 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Wed, 26 May 2010 18:55:42 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <1AE1A9E7-F821-45FF-8396-62EE8297ADE3@twistedmatrix.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4FD0AFC2-F0FF-4190-867D-3A2C35FD850E@twistedmatrix.com> <64B5ACEE-0B5C-40A8-AEC8-4D40BC7FE65B@sweetapp.com> <1AE1A9E7-F821-45FF-8396-62EE8297ADE3@twistedmatrix.com> Message-ID: <27B98FA1-A2FC-41D6-88AD-B44B9BF90DDC@sweetapp.com> On 26 May 2010, at 18:09, Glyph Lefkowitz wrote: > > On May 24, 2010, at 5:36 AM, Brian Quinlan wrote: >> On May 24, 2010, at 5:16 AM, Glyph Lefkowitz wrote: >>> On May 23, 2010, at 2:37 AM, Brian Quinlan wrote: >>>> On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote: > >> ProcessPoolExecutor has the same serialization perils that >> multiprocessing does. My original plan was to link to the >> multiprocessing docs to explain them but I couldn't find them listed. > > Linking to the pickle documentation might be a good start. Will do. >> Yes, the execution context is Executor-dependent. The section under >> ProcessPoolExecutor and ThreadPoolExecutor spells this out, I think. > > I suppose so. I guess I'm just looking for more precise usage of > terminology. (This is a PEP, after all. It's a specification that > multiple VMs may have to follow, not just some user documentation > for a package, even if they'll *probably* be using your code in all > cases.) I'd be happier if there were a clearer term than "calls" > for the things being scheduled ("submissions"?), since the done > callbacks aren't called in the subprocess for ProcessPoolExecutor, > as we just discussed. > >>> Sure. Really, almost any contract would work, it just needs to be >>> spelled out. It might be nice to know whether the thread invoking >>> the callbacks is a daemon thread or not, but I suppose it's not >>> strictly necessary. >> >> Your concerns is that the thread will be killed when the >> interpreter exits? It won't be. > > Good to know. Tell it to the PEP though, not me ;). Will do. >>> No reaction on [invoker vs. future]? I think you'll wish you did >>> this in a couple of years when you start bumping into application >>> code that calls "set_result" :). >> >> My reactions are mixed ;-) > > Well, you are not obliged to take my advice, as long as I am not > obliged to refrain from mocking you mercilessly if it happens that I > was right in a couple of years ;-). I was looking for your reasoning rather than trying to negotiate the circumstances under which you would mock me. > >> Your proposal is to add a level of indirection to make it harder >> for people to call implementation methods. The downside is that it >> makes it a bit harder to write tests and Executors. > > Both tests and executors will still create and invoke methods > directly on one object; the only additional difficulty seems to be > the need to type '.future' every so often on the executor/testing > side of things, and that seems a cost well worth paying to avoid > confusion over who is allowed to call those methods and when. > >> I also can't see a big problem in letting people call set_result in >> client code though it is documented as being only for Executor >> implementations and tests. >> >> On the implementation side, I don't see why an Invoker needs a >> reference to the future. > > Well, uh... > >> class Invoker(object): >> def __init__(self): >> """Should only be called by Executor implementations.""" >> self.future = Future() > ^ this is what I'd call a "reference to the future" I said exactly the opposite of what I meant: futures don't need a reference to the invoker. Cheers, Brian From glyph at twistedmatrix.com Wed May 26 11:02:33 2010 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 26 May 2010 05:02:33 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <27B98FA1-A2FC-41D6-88AD-B44B9BF90DDC@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4FD0AFC2-F0FF-4190-867D-3A2C35FD850E@twistedmatrix.com> <64B5ACEE-0B5C-40A8-AEC8-4D40BC7FE65B@sweetapp.com> <1AE1A9E7-F821-45FF-8396-62EE8297ADE3@twistedmatrix.com> <27B98FA1-A2FC-41D6-88AD-B44B9BF90DDC@sweetapp.com> Message-ID: On May 26, 2010, at 4:55 AM, Brian Quinlan wrote: > I said exactly the opposite of what I meant: futures don't need a reference to the invoker. Indeed they don't, and they really shouldn't have one. If I wrote that they did, then it was an error. ... and that appears to be it! Thank you for your very gracious handling of a pretty huge pile of criticism :). Good luck with the PEP, -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From hawkett at gmail.com Wed May 26 11:15:02 2010 From: hawkett at gmail.com (Colin H) Date: Wed, 26 May 2010 10:15:02 +0100 Subject: [Python-Dev] variable name resolution in exec is incorrect Message-ID: Hi, issue991196 was closed being described as intentional. I've added a comment in that issue which argues that this is a serious bug (also aserted by a previous commenter - Armin Rigo), because it creates a unique, undocumented, oddly behaving scope that doesn't apply closures correctly. At the very least I think this should be acknowledged as a plain old bug (rather than a feature), and then a discussion about whether it will be fixed or not. Appreciate your thoughts - cheers, Colin From solipsis at pitrou.net Wed May 26 11:39:15 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 May 2010 11:39:15 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp> <4BFCA1E9.1070808@gmail.com> Message-ID: <20100526113915.5bf46665@pitrou.net> On Wed, 26 May 2010 04:25:18 -0400 Glyph Lefkowitz wrote: > > In other words, while I kinda-sorta buy Brian's argument that having this module in easy reach > will motivate more people to use a standard, tested idiom for parallelization, I *don't* think > that the stdlib should be expanded simply to accommodate those who just don't want to install > additional packages for anything. +1. Why don't the castrated-by-the-corporation people offer to maintain a "Sumo" distribution of Python on python.org instead? The rest of the world shouldn't have to be impacted by their corporate culture woes. cheers Antoine. From dickinsm at gmail.com Wed May 26 11:48:51 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 26 May 2010 10:48:51 +0100 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: Message-ID: On Wed, May 26, 2010 at 10:15 AM, Colin H wrote: > ? issue991196 was closed being described as intentional. ?I've added > a comment in that issue which argues that this is a serious bug (also > aserted by a previous commenter - Armin Rigo), because it creates a > unique, undocumented, oddly behaving scope that doesn't apply closures > correctly. At the very least I think this should be acknowledged as a > plain old bug (rather than a feature), and then a discussion about > whether it will be fixed or not. Here's a quick recap of the issue so that people don't have to go searching through the bug archive. In Python 2.x, we get the following behaviour: >>> code = """\ ... y = 3 ... def f(): ... return y ... f() ... """ >>> exec code in {} # works fine >>> exec code in {}, {} # dies with a NameError Traceback (most recent call last): File "", line 1, in File "", line 4, in File "", line 3, in f NameError: global name 'y' is not defined The issue is whether the second example should work, given that two different dictionaries have been passed. The cause of the NameError can be seen by looking at the bytecode: y is bound using STORE_NAME, which stores y into the locals dictionary (which here is *not* the same as the globals dictionary) but the attempt to retrieve the value of y uses LOAD_GLOBAL, which only looks in the globals. >>> co = compile(code, 'mycode', 'exec') >>> dis.dis(co) 1 0 LOAD_CONST 0 (3) 3 STORE_NAME 0 (y) 2 6 LOAD_CONST 1 () 9 MAKE_FUNCTION 0 12 STORE_NAME 1 (f) 4 15 LOAD_NAME 1 (f) 18 CALL_FUNCTION 0 21 POP_TOP 22 LOAD_CONST 2 (None) 25 RETURN_VALUE >>> dis.dis(co.co_consts[1]) # disassembly of 'f' 3 0 LOAD_GLOBAL 0 (y) 3 RETURN_VALUE This is a long way from my area of expertise (I'm commenting here because it was me who sent Colin here in the first place), and it's not clear to me whether this is a bug, and if it is a bug, how it could be resolved. What would the impact be of having the compiler produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here? Mark From steve at pearwood.info Wed May 26 12:42:12 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 26 May 2010 20:42:12 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <20100526113915.5bf46665@pitrou.net> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> Message-ID: <201005262042.12825.steve@pearwood.info> On Wed, 26 May 2010 07:39:15 pm Antoine Pitrou wrote: > On Wed, 26 May 2010 04:25:18 -0400 > > Glyph Lefkowitz wrote: > > In other words, while I kinda-sorta buy Brian's argument that > > having this module in easy reach will motivate more people to use a > > standard, tested idiom for parallelization, I *don't* think that > > the stdlib should be expanded simply to accommodate those who just > > don't want to install additional packages for anything. > > +1. Why don't the castrated-by-the-corporation people offer to > maintain a "Sumo" distribution of Python on python.org instead? The > rest of the world shouldn't have to be impacted by their corporate > culture woes. It's not just the corporate culture. For many people, the standard library is the first introduction to even the existence of a particular technique or technology. You can't go looking for something on PyPI if you don't know that there's a something to look for. And for many beginners and not-so-beginners, the idea and practice of installing additional packages is simply problematic. I'm not saying that Python-Dev should bend over backwards to accommodate such people to the exclusion of all else, but these folks are stakeholders too, and their wants and needs are just as worthy as the wants and needs of those who prefer a more conservative approach to the standard library. This is a Python implementation of a stable Java API, Brian has said the futures package has been on PyPI for about a year, and it's been flagged as a production/stable release since October last year. http://pypi.python.org/pypi/futures3 Given that there does seem to be a general agreement that futures should go into the std lib at some point, is this not sufficient exposure? -- Steven D'Aprano From greg.ewing at canterbury.ac.nz Wed May 26 12:57:15 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 26 May 2010 22:57:15 +1200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> Message-ID: <4BFCFE8B.1080205@canterbury.ac.nz> Having read through the PEP again, here are my thoughts. * I'm bothered by the term "future". To my mind, it's too long on cleverness and too short on explanativeness. I think that the standard library is no place for cuteness of naming. The name of a stdlib module should reflect its functionality in some straightforward and obvious way. If I were looking for a thread pool or process pool implementation, the word "future" is not something that would spring readily to mind. The stated purpose of the module is to "execute computations asynchronously", so perhaps a name such as "asyntask" would be appropriate, following the pattern of existing modules dealing with ansynchronous matters, ansyncore and asynchat. For the Future object itself, I'd suggest something like "Task" or "Job". * It seems unnecessarily verbose to tack "Executor" onto the end of every Executor subclass. They could simply be called ThreadPool and ProcessPool without losing anything. * I don't see a strong reason to put this module inside a newly-created namespace. If there were a namespace called "concurrent", I would expect to find other existing concurrency-related modules there as well, such as threading and multiprocessing. But we can't move them there without breaking existing code. (More generally, I'm inclined to think that introducing a namespace package for a category of modules having existing members in the stdlib is an anti-pattern, unless it's done during the kind of namespace refactoring that we won't get another chance to perform until Py4k.) Concerning the structure of the PEP: * A section titled 'Specification' should *not* start with a bunch of examples. It may be appropriate to include short examples *following* items in the specification in order to illustrate the features concerned. Extended examples such as these belong in a section of their own. * I found the examples included to be rather difficult to follow, and they served more to confuse than elucidate. I think this is partly because they are written in a fairly compressed style, burying important things being illustrated inside complicated expressions. Rewriting them in a more straightforward style might help. Concerning details of the specification: * Is it possible to have more than one Executor active at a time? The fact that as_completed() is a module-level function rather than an Executor method suggests that it is, but it would be good to have this spelled out one way or the other in the PEP. -- Greg From solipsis at pitrou.net Wed May 26 12:56:14 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 May 2010 12:56:14 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> Message-ID: <20100526125614.0a7ae470@pitrou.net> On Wed, 26 May 2010 20:42:12 +1000 Steven D'Aprano wrote: > > I'm not saying that Python-Dev should bend over backwards to accommodate > such people to the exclusion of all else, but these folks are > stakeholders too, and their wants and needs are just as worthy as the > wants and needs of those who prefer a more conservative approach to the > standard library. Well, my "Sumo" proposal was a serious one. (not serious in that I would offer to give a hand, but in that I think it could help those people; also, wouldn't it be sensible for users in a corporate environment to share their efforts and produce something that can benefit all of them? it's the free software spirit after all) > This is a Python implementation of a stable Java API, Brian has said the > futures package has been on PyPI for about a year, and it's been > flagged as a production/stable release since October last year. I'm not against futures being in the stdlib, I was just pointing out that I don't agree with the "corporate culture issues should be accomodated by including more modules in the stdlib" argument. Regards Antoine. From brian at sweetapp.com Wed May 26 13:05:45 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Wed, 26 May 2010 21:05:45 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFCFE8B.1080205@canterbury.ac.nz> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> Message-ID: <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> On May 26, 2010, at 8:57 PM, Greg Ewing wrote: > Having read through the PEP again, here are my thoughts. > > * I'm bothered by the term "future". To my mind, it's > too long on cleverness and too short on explanativeness. > > I think that the standard library is no place for > cuteness of naming. The name of a stdlib module should > reflect its functionality in some straightforward and > obvious way. If I were looking for a thread pool or > process pool implementation, the word "future" is not > something that would spring readily to mind. > > The stated purpose of the module is to "execute > computations asynchronously", so perhaps a name such > as "asyntask" would be appropriate, following the > pattern of existing modules dealing with ansynchronous > matters, ansyncore and asynchat. For the Future object > itself, I'd suggest something like "Task" or "Job". "future" is a computing science term of art, like "thread". Anyway, this has been discussed in the past and Guido was happy with the name. > * It seems unnecessarily verbose to tack "Executor" > onto the end of every Executor subclass. They could > simply be called ThreadPool and ProcessPool without > losing anything. You could have general thread pools that aren't related to executors (actually, it would be great if Python had a good built-in thread pool implementation) and I'd like to avoid using an overly generic name. > * I don't see a strong reason to put this module > inside a newly-created namespace. If there were a > namespace called "concurrent", I would expect to find > other existing concurrency-related modules there as > well, such as threading and multiprocessing. But we > can't move them there without breaking existing code. I think that Jesse was planning to add some functionality to this namespace. I don't really have an opinion on this. > (More generally, I'm inclined to think that introducing > a namespace package for a category of modules having > existing members in the stdlib is an anti-pattern, > unless it's done during the kind of namespace refactoring > that we won't get another chance to perform until Py4k.) > > Concerning the structure of the PEP: > > * A section titled 'Specification' should *not* start > with a bunch of examples. It may be appropriate to include > short examples *following* items in the specification in > order to illustrate the features concerned. Extended > examples such as these belong in a section of their own. I thought that the specification would be difficult to follow without examples to pave the way. Anyone else have an opinion on this? > * I found the examples included to be rather difficult > to follow, and they served more to confuse than elucidate. > I think this is partly because they are written in a > fairly compressed style, burying important things being > illustrated inside complicated expressions. Rewriting > them in a more straightforward style might help. Do you think starting with a simpler example would help? I think that idiomatic future use will end up looking similar to my examples. If that is too complex for most users then we have a problem. > Concerning details of the specification: > > * Is it possible to have more than one Executor active > at a time? Of course. > The fact that as_completed() is a module-level > function rather than an Executor method suggests that it > is, but it would be good to have this spelled out one > way or the other in the PEP. I'll add a note to the global functions that they can accept futures from different in the same call. Cheers, Brian > > -- > Greg > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com From steve at pearwood.info Wed May 26 13:18:22 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 26 May 2010 21:18:22 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFCFE8B.1080205@canterbury.ac.nz> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BFCFE8B.1080205@canterbury.ac.nz> Message-ID: <201005262118.23006.steve@pearwood.info> On Wed, 26 May 2010 08:57:15 pm Greg Ewing wrote: > * I'm bothered by the term "future". To my mind, it's > too long on cleverness and too short on explanativeness. "Futures" is a standard term in computer science which has been around for 33 years. http://en.wikipedia.org/wiki/Futures_and_promises > I think that the standard library is no place for > cuteness of naming. You mean like pickle, marshal, shelve, turtle, and even dict? > * I don't see a strong reason to put this module > inside a newly-created namespace. If there were a > namespace called "concurrent", I would expect to find > other existing concurrency-related modules there as > well, such as threading and multiprocessing. But we > can't move them there without breaking existing code. I'm sure that it can be done easily, although not quickly. For instance, we move threading into the concurrent namespace, and leave behind in its place a stub: from concurrent.threading import * Then for (say) 3.3 the stub could gain a PendingDeprecation warning, then in 3.4 a Deprecation warning, and finally in 3.5 or 3.6 it could be removed. -- Steven D'Aprano From floris.bruynooghe at gmail.com Wed May 26 14:06:51 2010 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Wed, 26 May 2010 13:06:51 +0100 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> Message-ID: <20100526120651.GA12751@laurie.devork.be> Hi On Sun, May 23, 2010 at 10:47:08AM +1000, Brian Quinlan wrote: > Jesse, the designated pronouncer for this PEP, has decided to keep > discussion open for a few more days. > > So fire away! In thread.py the module automatically registers a handler with atexit. I don't think I'm alone in thinking libraries should not be doing this sort of thing unconditionally behind a user's back. I'm also not so sure how comfortable I am with the module-level globals. Would it not be possible to have an exit handler on each thread pool which the documentation reccomends you register with atexit if it suits your application? I think that would get rid of the global singletons and hidden atexit in a fairly elegant way. Lastly _base.py creates a LOGGER (connected to sys.stderr if I understand correctly) and only logs a critical message to it at the same time as a RuntimeError is raised. While I don't necessarily dislike that it uses a logger, I don't like that it's wired up to sys.stderr I rather think it's the application's duty to create a handler if it wants one. But given that it's only used at the same time as a RuntimeError it does seem redundant. Regards Floris PS: I've only looked at the threading part of the implementation. -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From p.f.moore at gmail.com Wed May 26 14:19:01 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 26 May 2010 13:19:01 +0100 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <20100526125614.0a7ae470@pitrou.net> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> Message-ID: On 26 May 2010 11:56, Antoine Pitrou wrote: > On Wed, 26 May 2010 20:42:12 +1000 > Steven D'Aprano wrote: >> >> I'm not saying that Python-Dev should bend over backwards to accommodate >> such people to the exclusion of all else, but these folks are >> stakeholders too, and their wants and needs are just as worthy as the >> wants and needs of those who prefer a more conservative approach to the >> standard library. > > Well, my "Sumo" proposal was a serious one. > (not serious in that I would offer to give a hand, but in that I think > it could help those people; also, wouldn't it be sensible for users in > a corporate environment to share their efforts and produce something > that can benefit all of them? it's the free software spirit after all) I'm not sure how a "Sumo" approach would work in practical terms, and this thread isn't really the place to discuss, but there's a couple of points I think are worth making: * For a "Sumo" distribution to make sense, some relatively substantial chunk of the standard library would need to be moved *out* to reside in the sumo distribution. Otherwise it's not really a "sumo", just a couple of modules that "nearly made it into the stdlib", at least for the near-to-medium term. I've yet to see any sort of consensus that python-dev is willing to undertake that decoupling work. (Which would include extracting the various tests, migrating bugs out of the pythion tracker, etc etc). * If the decoupled modules aren't simply being abandoned, python-dev needs to continue to commit to supporting them "in the wild" (i.e., on PyPI and in the sumo distribution). Otherwise we're just abandoning existing users and saying "support it yourself". I've seen no indication that python-dev members would expect to follow bug trackers for various decoupled modules - so in practice, this sounds more like abandonment than decoupling. Until a stdlib-decoupling proposal which takes these aspects into account is on the table, I'm afraid that suggesting there's a "Sumo distribution" style middle ground between stdlib and PyPI isn't really true... Paul. From jnoller at gmail.com Wed May 26 14:25:08 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 26 May 2010 08:25:08 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> Message-ID: On Wed, May 26, 2010 at 8:19 AM, Paul Moore wrote: > On 26 May 2010 11:56, Antoine Pitrou wrote: >> On Wed, 26 May 2010 20:42:12 +1000 >> Steven D'Aprano wrote: >>> >>> I'm not saying that Python-Dev should bend over backwards to accommodate >>> such people to the exclusion of all else, but these folks are >>> stakeholders too, and their wants and needs are just as worthy as the >>> wants and needs of those who prefer a more conservative approach to the >>> standard library. >> >> Well, my "Sumo" proposal was a serious one. >> (not serious in that I would offer to give a hand, but in that I think >> it could help those people; also, wouldn't it be sensible for users in >> a corporate environment to share their efforts and produce something >> that can benefit all of them? it's the free software spirit after all) > > I'm not sure how a "Sumo" approach would work in practical terms, and > this thread isn't really the place to discuss, but there's a couple of > points I think are worth making: > > * For a "Sumo" distribution to make sense, some relatively substantial > chunk of the standard library would need to be moved *out* to reside > in the sumo distribution. Otherwise it's not really a "sumo", just a > couple of modules that "nearly made it into the stdlib", at least for > the near-to-medium term. I've yet to see any sort of consensus that > python-dev is willing to undertake that decoupling work. (Which would > include extracting the various tests, migrating bugs out of the > pythion tracker, etc etc). > > * If the decoupled modules aren't simply being abandoned, python-dev > needs to continue to commit to supporting them "in the wild" (i.e., on > PyPI and in the sumo distribution). Otherwise we're just abandoning > existing users and saying "support it yourself". I've seen no > indication that python-dev members would expect to follow bug trackers > for various decoupled modules - so in practice, this sounds more like > abandonment than decoupling. > > Until a stdlib-decoupling proposal which takes these aspects into > account is on the table, I'm afraid that suggesting there's a "Sumo > distribution" style middle ground between stdlib and PyPI isn't really > true... > > Paul. The fat vs. thin stdlib was discussed on stdlib-sig some time ago (I am generally +1 to having a thin dist and a secondary "fatter" dist), however right now, it doesn't make sense packaging and dependency management is still a mess (but getting better), and there's a ton of other things to take into consideration, some of which has been iterated in this thread. That being said, we've now evolved into meta-meta-meta-discussion - if people seriously want to discuss the fat vs. thin subject, it should probably go to stdlib-sig. jesse From ncoghlan at gmail.com Wed May 26 14:26:17 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 May 2010 22:26:17 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <20100526125614.0a7ae470@pitrou.net> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> Message-ID: <4BFD1369.8090803@gmail.com> On 26/05/10 20:56, Antoine Pitrou wrote: > On Wed, 26 May 2010 20:42:12 +1000 > Steven D'Aprano wrote: >> >> I'm not saying that Python-Dev should bend over backwards to accommodate >> such people to the exclusion of all else, but these folks are >> stakeholders too, and their wants and needs are just as worthy as the >> wants and needs of those who prefer a more conservative approach to the >> standard library. > > Well, my "Sumo" proposal was a serious one. > (not serious in that I would offer to give a hand, but in that I think > it could help those people; also, wouldn't it be sensible for users in > a corporate environment to share their efforts and produce something > that can benefit all of them? it's the free software spirit after all) That's actually what happens with groups like Enthought and ActiveState - bundles with extra batteries. However, note that this isn't just a dysfunctional corporate culture issue, and I object to it being characterised as such (although dysfunctional cultures can certainly make it much, much worse). Vetting licenses for due diligence reasons, tracking releases of an external module, familiarising yourself with an additional API and code base, the risk of encountering bugs in that code base... these are all real costs that don't go away no matter how good the Python packaging ecosystem becomes. There is a trade off between "do the simplest thing that could possibly work (but may cause you problems later)" and spending the time to investigate third party solutions (with the risk that you end up rolling your own later anyway if you don't find anything suitable or, worse, find something that initially seems suitable but proves unworkable in practice). A module that makes it into the standard library, however, carries python-dev's stamp of approval. Except for some older legacy libraries, that means a module will have at least half decent documentation and an automated test suite that is regularly run on multiple platforms. Its design will also have run the gauntlet of python-dev approval. If we identify a good solution to a standard problem, and we have reason to believe that posting it in on PyPI as a separate module won't lead to a significant amount of additional real world testing, then it makes sense for it to go straight into the standard library. Such modules are going to be rare (since most non-trivial modules *will* benefit from some time on PyPI, and most trivial modules won't be added to the standard library at all), but they do exist (runpy, contextlib, collections, itertools and abc spring to mind). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From fuzzyman at voidspace.org.uk Wed May 26 14:28:38 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 26 May 2010 13:28:38 +0100 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> Message-ID: <4BFD13F6.1080300@voidspace.org.uk> On 26/05/2010 13:19, Paul Moore wrote: > On 26 May 2010 11:56, Antoine Pitrou wrote: > >> On Wed, 26 May 2010 20:42:12 +1000 >> Steven D'Aprano wrote: >> >>> I'm not saying that Python-Dev should bend over backwards to accommodate >>> such people to the exclusion of all else, but these folks are >>> stakeholders too, and their wants and needs are just as worthy as the >>> wants and needs of those who prefer a more conservative approach to the >>> standard library. >>> >> Well, my "Sumo" proposal was a serious one. >> (not serious in that I would offer to give a hand, but in that I think >> it could help those people; also, wouldn't it be sensible for users in >> a corporate environment to share their efforts and produce something >> that can benefit all of them? it's the free software spirit after all) >> > I'm not sure how a "Sumo" approach would work in practical terms, and > this thread isn't really the place to discuss, but there's a couple of > points I think are worth making: > > * For a "Sumo" distribution to make sense, some relatively substantial > chunk of the standard library would need to be moved *out* to reside > in the sumo distribution. Otherwise it's not really a "sumo", just a > couple of modules that "nearly made it into the stdlib", at least for > the near-to-medium term. I've yet to see any sort of consensus that > python-dev is willing to undertake that decoupling work. (Which would > include extracting the various tests, migrating bugs out of the > pythion tracker, etc etc). > > * If the decoupled modules aren't simply being abandoned, python-dev > needs to continue to commit to supporting them "in the wild" (i.e., on > PyPI and in the sumo distribution). Otherwise we're just abandoning > existing users and saying "support it yourself". I've seen no > indication that python-dev members would expect to follow bug trackers > for various decoupled modules - so in practice, this sounds more like > abandonment than decoupling. > > Until a stdlib-decoupling proposal which takes these aspects into > account is on the table, I'm afraid that suggesting there's a "Sumo > distribution" style middle ground between stdlib and PyPI isn't really > true... > Well... a middle ground certainly could exist; perhaps in the form of an "Extended Standard Library" (community distribution), with simple installation and management tools. It could be "blessed" by python-dev and maintain a high standard (only well established best-of-breed modules with a commitment of ongoing maintenance and more than one maintainer - something that the stdlib itself doesn't stick to). A common license could even be chosen, potentially allowing corporations to approve the extended package in a single pass. Lot of details to flesh out obviously - but it would be great to see something like this come into being. Obviously this would need to be a community initiative and would take some time to establish. A "fat" distribution like this, based on tools like pip and distribute would be great for both newbies and for experienced programmers in making it easier to find "best" solutions for standard problems. It could also act as an incubator for the standard library (perhaps with stable and experimental streams where stable has a more conservative update policy). All the best, Michael Foord > Paul. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From brian at sweetapp.com Wed May 26 14:31:06 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Wed, 26 May 2010 22:31:06 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <20100526120651.GA12751@laurie.devork.be> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <20100526120651.GA12751@laurie.devork.be> Message-ID: <8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com> On 26 May 2010, at 22:06, Floris Bruynooghe wrote: > Hi > > On Sun, May 23, 2010 at 10:47:08AM +1000, Brian Quinlan wrote: >> Jesse, the designated pronouncer for this PEP, has decided to keep >> discussion open for a few more days. >> >> So fire away! > > In thread.py the module automatically registers a handler with atexit. > I don't think I'm alone in thinking libraries should not be doing this > sort of thing unconditionally behind a user's back. I'm also not so > sure how comfortable I am with the module-level globals. > > Would it not be possible to have an exit handler on each thread pool > which the documentation reccomends you register with atexit if it > suits your application? I think that would get rid of the global > singletons and hidden atexit in a fairly elegant way. First let me explain why I install at atexit handler. Imagine that the you write a script like this: t = ThreadPoolExecutor(1) t.submit(lambda url: print(urllib.open(url).read()), 'http://www.apple.com/') You have two semantic choices here: 1. let the interpreter exit with the future still running 2. wait until the future finishes and then exit I chose (2) but can be convinced otherwise. The obvious way to accomplish this is to make the worker thread non-daemon so the interpreter won't exit while it is running. But since the worker thread is part of a pool, it won't stop while it's executor is alive. So my approach was to make worker threads daemon and install an atexit handler that sets a global indicating that the interpreter is exiting so any workers should exit when when their work queues are empty. It then calls join on each worker thread so the interpreter will not exit until they are finished. I think that this approach is reasonable assuming that you want (2). I also don't have the aversion to globals that you do :-) > Lastly _base.py creates a LOGGER (connected to sys.stderr if I > understand correctly) and only logs a critical message to it at the > same time as a RuntimeError is raised. While I don't necessarily > dislike that it uses a logger, I don't like that it's wired up to > sys.stderr I rather think it's the application's duty to create a > handler if it wants one. But given that it's only used at the same > time as a RuntimeError it does seem redundant. The LOGGER is only use for "impossible" exceptions (exceptions in the machinery of the module itself) that won't be propagated because they occur in worker threads. Cheers, Brian > Regards > Floris > > PS: I've only looked at the threading part of the implementation. > > -- > Debian GNU/Linux -- The Power of Freedom > www.debian.org | www.gnu.org | www.kernel.org > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com From ncoghlan at gmail.com Wed May 26 14:32:33 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 May 2010 22:32:33 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <20100526093820.4fbf2f87@pitrou.net> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4BF916EA.2090507@holdenweb.com> <20100523144754.1c6f1afc@pitrou.net> <4BFC6325.5050700@gmail.com> <20100526093820.4fbf2f87@pitrou.net> Message-ID: <4BFD14E1.6000305@gmail.com> On 26/05/10 17:38, Antoine Pitrou wrote: > On Wed, 26 May 2010 09:54:13 +1000 > Nick Coghlan wrote: >>> >>> What I would question here is what other things will be part >>> of the "concurrent" package, and who will implement them. Are there >>> plans for that? (or even tracker issues open?) >> >> I'm not sure it is called out explicitly in the PEP, but the specific >> example that came up in the previous discussions was something like >> "concurrent.pool" to hold a thread vs process agnostic worker pool >> interface based on the existing Pool interface in multiprocessing >> (with concrete implementations for both threading and >> multiprocessing). > > Ha, I'm a bit surprised. Isn't it what "futures" already provides? > (except that for some reason it insists on the "SomeExecutor" naming > scheme) > http://www.python.org/dev/peps/pep-3148/#processpoolexecutor Not really - a general purpose pool would be a lot more agnostic about how you give the pooled threads/processes work to do and get the results back. Executors are the kind of thing you would build on top of one though. If concurrent.pool was added, then the existing processing pools in multiprocessing and the executors in concurrent.future would be the first use cases for it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed May 26 14:42:35 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 May 2010 22:42:35 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFCFE8B.1080205@canterbury.ac.nz> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> Message-ID: <4BFD173B.9010703@gmail.com> On 26/05/10 20:57, Greg Ewing wrote: > Having read through the PEP again, here are my thoughts. > * It seems unnecessarily verbose to tack "Executor" > onto the end of every Executor subclass. They could > simply be called ThreadPool and ProcessPool without > losing anything. We would lose the ability to add general purpose thread and process pools under the obvious names later. > * I don't see a strong reason to put this module > inside a newly-created namespace. If there were a > namespace called "concurrent", I would expect to find > other existing concurrency-related modules there as > well, such as threading and multiprocessing. But we > can't move them there without breaking existing code. > > (More generally, I'm inclined to think that introducing > a namespace package for a category of modules having > existing members in the stdlib is an anti-pattern, > unless it's done during the kind of namespace refactoring > that we won't get another chance to perform until Py4k.) _thread, threading, Queue and multiprocessing do likely belong here, but moving them isn't likely to be worth the pain. Does it help to know that at least Jesse and I (and probably others) would like to see concurrent.pool added eventually with robust general purpose ThreadPool and ProcessPool implementations? The specific reason the new package namespace was added was to help avoid confusion with stock market futures without using an unduly cumbersome module name, but I don't know how well the PEP explains that. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Wed May 26 14:46:03 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 May 2010 14:46:03 +0200 Subject: [Python-Dev] Sumo In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> Message-ID: <1274877963.3162.20.camel@localhost.localdomain> Le mercredi 26 mai 2010 ? 13:19 +0100, Paul Moore a ?crit : > > I'm not sure how a "Sumo" approach would work in practical terms, and > this thread isn't really the place to discuss, but there's a couple of > points I think are worth making: > > * For a "Sumo" distribution to make sense, some relatively substantial > chunk of the standard library would need to be moved *out* to reside > in the sumo distribution. Otherwise it's not really a "sumo", just a > couple of modules that "nearly made it into the stdlib", at least for > the near-to-medium term. This is not what I'm suggesting at all. The stdlib wouldn't shrink (well, we could dump outdated modules but that's a separate decision). A hypothetical "Sumo" distribution would be made of those more or less useful modules that many people (application or framework developers; for example, take a look at the dozens of direct and indirect dependencies TurboGears pulls in) need and install. The whole point is that it would *not* be supported by python-dev itself but by a separate group of people (such as you :-)). And it would have its own rules (surely more relaxed) over inclusion or deprecation delays, the ability to make compatibility-breaking changes, release timings, multiple obvious ways to do the same thing, etc. And it means it could really bundle a lot of third-party stuff: small helpers (things like the decorator module), event loops, template engines, network address abstractions, argument parsers, ORMs, UI toolkits, etc. (a side-effect would be that it could, if it works well, behave as a good intermediate stress test - a purgatory, if you want - for modules before they integrate the stdlib) If you want an existing analogy, you could look at EasyPHP. Or think of Python as the Gnome or KDE project (consistent and aiming at providing the most important everyday tools, but quite focused), and "Sumo" as an entire distribution of disparate Linux GUI apps. Regards Antoine. From solipsis at pitrou.net Wed May 26 14:50:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 May 2010 14:50:33 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4BF916EA.2090507@holdenweb.com> <20100523144754.1c6f1afc@pitrou.net> <4BFC6325.5050700@gmail.com> <20100526093820.4fbf2f87@pitrou.net> <4BFD14E1.6000305@gmail.com> Message-ID: <20100526145033.3b61d95b@pitrou.net> On Wed, 26 May 2010 22:32:33 +1000 Nick Coghlan wrote: > > > > Ha, I'm a bit surprised. Isn't it what "futures" already provides? > > (except that for some reason it insists on the "SomeExecutor" naming > > scheme) > > http://www.python.org/dev/peps/pep-3148/#processpoolexecutor > > Not really - a general purpose pool would be a lot more agnostic about > how you give the pooled threads/processes work to do and get the results > back. > > Executors are the kind of thing you would build on top of one though. If > concurrent.pool was added, then the existing processing pools in > multiprocessing and the executors in concurrent.future would be the > first use cases for it. I think I'm a bit ignorant, but how is the Executor abstraction (and its proposed implementations) not generic enough? You have a pool, submit one or several tasks, and can either repeatedly poll for completion or do a blocking wait. (after all, Glyph pointed out that it should be quite easy to wrap the resulting Futures into Deferred objects) cheers Antoine. From ncoghlan at gmail.com Wed May 26 14:51:42 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 May 2010 22:51:42 +1000 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: Message-ID: <4BFD195E.2060007@gmail.com> On 26/05/10 19:48, Mark Dickinson wrote: > This is a long way from my area of expertise (I'm commenting here > because it was me who sent Colin here in the first place), and it's > not clear to me whether this is a bug, and if it is a bug, how it > could be resolved. What would the impact be of having the compiler > produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here? exec with a single argument = module namespace exec with two arguments = class namespace Class namespaces are deliberately exempted from lexical scoping so that methods can't see class attributes, hence the example in the tracker issue works exactly as it would if the code was written as a class body. class C: y = 3 def execfunc(): print y execfunc() With this code, y would end up in C.__dict__ rather than the module globals (at least, it would if it wasn't for the exception) and the call to execfunc fails with a NameError when attempting to find y. I know I've closed other bug reports that were based on the same misunderstanding, and I didn't understand it myself until Guido explained it to me a few years back, so suggestions for improving the exec documentation in this area would be appreciated. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From jnoller at gmail.com Wed May 26 15:01:18 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 26 May 2010 09:01:18 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFD173B.9010703@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com> Message-ID: On Wed, May 26, 2010 at 8:42 AM, Nick Coghlan wrote: > On 26/05/10 20:57, Greg Ewing wrote: >> >> Having read through the PEP again, here are my thoughts. >> * It seems unnecessarily verbose to tack "Executor" >> onto the end of every Executor subclass. They could >> simply be called ThreadPool and ProcessPool without >> losing anything. > > We would lose the ability to add general purpose thread and process pools > under the obvious names later. > >> * I don't see a strong reason to put this module >> inside a newly-created namespace. If there were a >> namespace called "concurrent", I would expect to find >> other existing concurrency-related modules there as >> well, such as threading and multiprocessing. But we >> can't move them there without breaking existing code. >> >> (More generally, I'm inclined to think that introducing >> a namespace package for a category of modules having >> existing members in the stdlib is an anti-pattern, >> unless it's done during the kind of namespace refactoring >> that we won't get another chance to perform until Py4k.) > > _thread, threading, Queue and multiprocessing do likely belong here, but > moving them isn't likely to be worth the pain. Does it help to know that at > least Jesse and I (and probably others) would like to see concurrent.pool > added eventually with robust general purpose ThreadPool and ProcessPool > implementations? > > The specific reason the new package namespace was added was to help avoid > confusion with stock market futures without using an unduly cumbersome > module name, but I don't know how well the PEP explains that. I assume(d) it's sufficient to link to the mailing list threads where we hashed this out already ;) The namespace serves a few purposes: 1 > We put futures where it makes sense - into a concurrent package. Futures are a concurrency construct; therefore it simply makes sense to put them within a sub package rather on the top level. 2 > We carve out a box to move to, and add other concurrent things, such as generic pools, Actor implementations, etc. See java.util.concurrent. Things within multiprocessing that don't start with P and rhyme with "rocess" can go here too. Admittedly, it's mainly my own long-term vision to see python-core grow more concurrency tidbits - although I don't know too many people who would complain about it. jesse From brian at sweetapp.com Wed May 26 15:01:15 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Wed, 26 May 2010 23:01:15 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFD173B.9010703@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com> Message-ID: <1F22277E-150D-43D1-A332-3DD1BC2B11A0@sweetapp.com> On 26 May 2010, at 22:42, Nick Coghlan wrote: > On 26/05/10 20:57, Greg Ewing wrote: >> Having read through the PEP again, here are my thoughts. >> * It seems unnecessarily verbose to tack "Executor" >> onto the end of every Executor subclass. They could >> simply be called ThreadPool and ProcessPool without >> losing anything. > > We would lose the ability to add general purpose thread and process > pools under the obvious names later. > >> * I don't see a strong reason to put this module >> inside a newly-created namespace. If there were a >> namespace called "concurrent", I would expect to find >> other existing concurrency-related modules there as >> well, such as threading and multiprocessing. But we >> can't move them there without breaking existing code. >> >> (More generally, I'm inclined to think that introducing >> a namespace package for a category of modules having >> existing members in the stdlib is an anti-pattern, >> unless it's done during the kind of namespace refactoring >> that we won't get another chance to perform until Py4k.) > > _thread, threading, Queue and multiprocessing do likely belong here, > but moving them isn't likely to be worth the pain. Does it help to > know that at least Jesse and I (and probably others) would like to > see concurrent.pool added eventually with robust general purpose > ThreadPool and ProcessPool implementations? > > The specific reason the new package namespace was added was to help > avoid confusion with stock market futures without using an unduly > cumbersome module name, but I don't know how well the PEP explains > that. It doesn't at all. Are these plans formalized anywhere that I can link to? Cheers, Brian From thomas at python.org Wed May 26 15:02:41 2010 From: thomas at python.org (Thomas Wouters) Date: Wed, 26 May 2010 15:02:41 +0200 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: Message-ID: On Wed, May 26, 2010 at 11:48, Mark Dickinson wrote: > On Wed, May 26, 2010 at 10:15 AM, Colin H wrote: > > issue991196 was closed being described as intentional. I've added > > a comment in that issue which argues that this is a serious bug (also > > aserted by a previous commenter - Armin Rigo), because it creates a > > unique, undocumented, oddly behaving scope that doesn't apply closures > > correctly. At the very least I think this should be acknowledged as a > > plain old bug (rather than a feature), and then a discussion about > > whether it will be fixed or not. > > Here's a quick recap of the issue so that people don't have to go > searching through the bug archive. In Python 2.x, we get the > following behaviour: > > >>> code = """\ > ... y = 3 > ... def f(): > ... return y > ... f() > ... """ > >>> exec code in {} # works fine > >>> exec code in {}, {} # dies with a NameError > Traceback (most recent call last): > File "", line 1, in > File "", line 4, in > File "", line 3, in f > NameError: global name 'y' is not defined > > The issue is whether the second example should work, given that two > different dictionaries have been passed. > > The cause of the NameError can be seen by looking at the bytecode: y > is bound using STORE_NAME, which stores y into the locals dictionary > (which here is *not* the same as the globals dictionary) but the > attempt to retrieve the value of y uses LOAD_GLOBAL, which only looks > in the globals. > > >>> co = compile(code, 'mycode', 'exec') > >>> dis.dis(co) > 1 0 LOAD_CONST 0 (3) > 3 STORE_NAME 0 (y) > > 2 6 LOAD_CONST 1 ( 0xa22b40, file "mycode", line 2>) > 9 MAKE_FUNCTION 0 > 12 STORE_NAME 1 (f) > > 4 15 LOAD_NAME 1 (f) > 18 CALL_FUNCTION 0 > 21 POP_TOP > 22 LOAD_CONST 2 (None) > 25 RETURN_VALUE > >>> dis.dis(co.co_consts[1]) # disassembly of 'f' > 3 0 LOAD_GLOBAL 0 (y) > 3 RETURN_VALUE > > This is a long way from my area of expertise (I'm commenting here > because it was me who sent Colin here in the first place), and it's > not clear to me whether this is a bug, and if it is a bug, how it > could be resolved. What would the impact be of having the compiler > produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here? It wouldn't matter. The 'f' function only knows about its own namespace (separate from the surrounding code's local namespace), and the global namespace. LOAD_NAME is only different from LOAD_GLOBAL in that it looks in the local namespace first, but in this case the local namespace contains nothing. Here's what happens: 'exec code in d1, d2' executes code with 'd1' as locals and 'd2' as globals. Assignment always operates on the local namespace (barring the 'global' declaration.) The function definition also assigns to the local namespace, but the created function knows nothing about that local namespace -- it only cares about its own namespace and the global namespace, 'd1'. 'exec code in d1' does the same thing as 'exec code in d1, d1': it uses the same dict for the locals and the globals. The execution of the code doesn't change -- the assignment to 'y' still assigns to the locals, and the 'f' function still looks it up in globals, but now *they are the same dict*. Using the same dict for locals and globals is how modules work, as well. The main confusion here is the fact that 'exec' doesn't generate closures. (Nobody was ever confused about this behaviour back in Python 2.0-and-earlier! :-) The reason for that is the disconnect between the compiler and the exec statement: the compiler sees no enclosing function, so it doesn't generate a closure. The exec statement, because it gets two different namespaces, then executes it like a function, with a distinct local namespace. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Wed May 26 15:08:35 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 26 May 2010 14:08:35 +0100 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: <4BFD195E.2060007@gmail.com> References: <4BFD195E.2060007@gmail.com> Message-ID: <4BFD1D53.2080502@voidspace.org.uk> On 26/05/2010 13:51, Nick Coghlan wrote: > On 26/05/10 19:48, Mark Dickinson wrote: >> This is a long way from my area of expertise (I'm commenting here >> because it was me who sent Colin here in the first place), and it's >> not clear to me whether this is a bug, and if it is a bug, how it >> could be resolved. What would the impact be of having the compiler >> produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here? > > exec with a single argument = module namespace > exec with two arguments = class namespace > > Class namespaces are deliberately exempted from lexical scoping so > that methods can't see class attributes, hence the example in the > tracker issue works exactly as it would if the code was written as a > class body. > > class C: > y = 3 > def execfunc(): > print y > execfunc() > > With this code, y would end up in C.__dict__ rather than the module > globals (at least, it would if it wasn't for the exception) and the > call to execfunc fails with a NameError when attempting to find y. > > I know I've closed other bug reports that were based on the same > misunderstanding, and I didn't understand it myself until Guido > explained it to me a few years back, so suggestions for improving the > exec documentation in this area would be appreciated. Your explanation here is very clear. Is this in the documentation? Michael > > Cheers, > Nick. > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From jnoller at gmail.com Wed May 26 15:17:54 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 26 May 2010 09:17:54 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <1F22277E-150D-43D1-A332-3DD1BC2B11A0@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com> <1F22277E-150D-43D1-A332-3DD1BC2B11A0@sweetapp.com> Message-ID: On Wed, May 26, 2010 at 9:01 AM, Brian Quinlan wrote: > > On 26 May 2010, at 22:42, Nick Coghlan wrote: > >> On 26/05/10 20:57, Greg Ewing wrote: >>> >>> Having read through the PEP again, here are my thoughts. >>> * It seems unnecessarily verbose to tack "Executor" >>> onto the end of every Executor subclass. They could >>> simply be called ThreadPool and ProcessPool without >>> losing anything. >> >> We would lose the ability to add general purpose thread and process pools >> under the obvious names later. >> >>> * I don't see a strong reason to put this module >>> inside a newly-created namespace. If there were a >>> namespace called "concurrent", I would expect to find >>> other existing concurrency-related modules there as >>> well, such as threading and multiprocessing. But we >>> can't move them there without breaking existing code. >>> >>> (More generally, I'm inclined to think that introducing >>> a namespace package for a category of modules having >>> existing members in the stdlib is an anti-pattern, >>> unless it's done during the kind of namespace refactoring >>> that we won't get another chance to perform until Py4k.) >> >> _thread, threading, Queue and multiprocessing do likely belong here, but >> moving them isn't likely to be worth the pain. Does it help to know that at >> least Jesse and I (and probably others) would like to see concurrent.pool >> added eventually with robust general purpose ThreadPool and ProcessPool >> implementations? >> >> The specific reason the new package namespace was added was to help avoid >> confusion with stock market futures without using an unduly cumbersome >> module name, but I don't know how well the PEP explains that. > > It doesn't at all. Are these plans formalized anywhere that I can link to? > > Cheers, > Brian Nope; and I don't think we need to worry about it right now. From brian at sweetapp.com Wed May 26 15:29:05 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Wed, 26 May 2010 23:29:05 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <20100526145033.3b61d95b@pitrou.net> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4BF916EA.2090507@holdenweb.com> <20100523144754.1c6f1afc@pitrou.net> <4BFC6325.5050700@gmail.com> <20100526093820.4fbf2f87@pitrou.net> <4BFD14E1.6000305@gmail.com> <20100526145033.3b61d95b@pitrou.net> Message-ID: On 26 May 2010, at 22:50, Antoine Pitrou wrote: > On Wed, 26 May 2010 22:32:33 +1000 > Nick Coghlan wrote: >>> >>> Ha, I'm a bit surprised. Isn't it what "futures" already provides? >>> (except that for some reason it insists on the "SomeExecutor" naming >>> scheme) >>> http://www.python.org/dev/peps/pep-3148/#processpoolexecutor >> >> Not really - a general purpose pool would be a lot more agnostic >> about >> how you give the pooled threads/processes work to do and get the >> results >> back. >> >> Executors are the kind of thing you would build on top of one >> though. If >> concurrent.pool was added, then the existing processing pools in >> multiprocessing and the executors in concurrent.future would be the >> first use cases for it. > > I think I'm a bit ignorant, but how is the Executor abstraction (and > its proposed implementations) not generic enough? You have a pool, > submit one or several tasks, and can either repeatedly poll for > completion or do a blocking wait. > > (after all, Glyph pointed out that it should be quite easy to wrap the > resulting Futures into Deferred objects) Interesting. Executor.submit() return a Future, which might not be useful in some ThreadPool fire-and-forget use cases but having them doesn't seem harmful. Java does take this approach and it gives you a lot more ways to customize the Executor thread pool i.e. the minimum number of threads running, the maximum number, the amount of time that a thread can be idle before it is killed, the queueing strategy to use (e.g. LIFO, FIFO, priority). Cheers, Brian From ncoghlan at gmail.com Wed May 26 15:33:03 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 May 2010 23:33:03 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <1F22277E-150D-43D1-A332-3DD1BC2B11A0@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com> <1F22277E-150D-43D1-A332-3DD1BC2B11A0@sweetapp.com> Message-ID: <4BFD230F.8010100@gmail.com> On 26/05/10 23:01, Brian Quinlan wrote: >> _thread, threading, Queue and multiprocessing do likely belong here, >> but moving them isn't likely to be worth the pain. Does it help to >> know that at least Jesse and I (and probably others) would like to see >> concurrent.pool added eventually with robust general purpose >> ThreadPool and ProcessPool implementations? >> >> The specific reason the new package namespace was added was to help >> avoid confusion with stock market futures without using an unduly >> cumbersome module name, but I don't know how well the PEP explains that. > > It doesn't at all. Are these plans formalized anywhere that I can link to? Just the previous lot of discussions. The main point that should be mentioned in the PEP is that "futures" on its own was ambiguous as to the applicable domain, but "concurrent.futures" was perfectly clear, without causing any readability problems the way a longer name could. Moving the general purpose pools out to their own module was just an example that occurred to us as something else that could go in that package rather than a concrete plan for implementation. Yes, we're setting ourselves up for inevitable questions as to why the existing modules are top level rather than part of this package, but the minimal pain response there would be to link to them from the package documentation with a note along the lines of "for historical reasons, some modules you might reasonably expect to find in this package are instead provided as top level modules". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed May 26 15:43:35 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 May 2010 23:43:35 +1000 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: <4BFD1D53.2080502@voidspace.org.uk> References: <4BFD195E.2060007@gmail.com> <4BFD1D53.2080502@voidspace.org.uk> Message-ID: <4BFD2587.7060903@gmail.com> On 26/05/10 23:08, Michael Foord wrote: > On 26/05/2010 13:51, Nick Coghlan wrote: >> On 26/05/10 19:48, Mark Dickinson wrote: >>> This is a long way from my area of expertise (I'm commenting here >>> because it was me who sent Colin here in the first place), and it's >>> not clear to me whether this is a bug, and if it is a bug, how it >>> could be resolved. What would the impact be of having the compiler >>> produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here? >> >> exec with a single argument = module namespace >> exec with two arguments = class namespace >> >> Class namespaces are deliberately exempted from lexical scoping so >> that methods can't see class attributes, hence the example in the >> tracker issue works exactly as it would if the code was written as a >> class body. >> >> class C: >> y = 3 >> def execfunc(): >> print y >> execfunc() >> >> With this code, y would end up in C.__dict__ rather than the module >> globals (at least, it would if it wasn't for the exception) and the >> call to execfunc fails with a NameError when attempting to find y. >> >> I know I've closed other bug reports that were based on the same >> misunderstanding, and I didn't understand it myself until Guido >> explained it to me a few years back, so suggestions for improving the >> exec documentation in this area would be appreciated. > > Your explanation here is very clear. Is this in the documentation? It isn't actually - I think Thomas may be right that the current exec documentation was largely written prior to the introduction of lexical scoping. So adding something along these lines would probably be a good place to start. Unfortunately, even my explanation above isn't 100% correct - it misses a subtle distinction where lexical scoping will pass through a class definition nested inside a function definition and see the function scope, but the use of strings for exec code means that the scopes of any containing functions are ignored completely. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed May 26 15:48:24 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 May 2010 23:48:24 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4BF916EA.2090507@holdenweb.com> <20100523144754.1c6f1afc@pitrou.net> <4BFC6325.5050700@gmail.com> <20100526093820.4fbf2f87@pitrou.net> <4BFD14E1.6000305@gmail.com> <20100526145033.3b61d95b@pitrou.net> Message-ID: <4BFD26A8.9050802@gmail.com> On 26/05/10 23:29, Brian Quinlan wrote: > On 26 May 2010, at 22:50, Antoine Pitrou wrote: >> I think I'm a bit ignorant, but how is the Executor abstraction (and >> its proposed implementations) not generic enough? You have a pool, >> submit one or several tasks, and can either repeatedly poll for >> completion or do a blocking wait. >> >> (after all, Glyph pointed out that it should be quite easy to wrap the >> resulting Futures into Deferred objects) > > Interesting. Executor.submit() return a Future, which might not be > useful in some ThreadPool fire-and-forget use cases but having them > doesn't seem harmful. > > Java does take this approach and it gives you a lot more ways to > customize the Executor thread pool i.e. the minimum number of threads > running, the maximum number, the amount of time that a thread can be > idle before it is killed, the queueing strategy to use (e.g. LIFO, FIFO, > priority). I would say it is precisely that extra configurability which separates the executor pools in the PEP implementation from more flexible general purpose pools. It's something to investigate somewhere along the line, but, as Jesse pointed out, not something we need to worry about specifically for this PEP (except as an example of another module that may eventually end up in the concurrent package) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From hawkett at gmail.com Wed May 26 16:03:35 2010 From: hawkett at gmail.com (Colin H) Date: Wed, 26 May 2010 15:03:35 +0100 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: Message-ID: Thanks for the details on why the observed behaviour occurs - very clear. My only query would be why this is considered correct? Why is it running as a class namespace, when it is not a class? Is there any reason why this is not considered a mistake? Slightly concerned that this is being considered not a bug because 'it is how it is'. A really good reason why you would want to provide a separate locals dictionary is to get access to the stuff that was defined in the exec()'d code block. Unfortunately this use case is broken by the current behaviour. The only way to get the definitions from the exec()'d code block is to supply a single dictionary, and then try to weed out the definitions from amongst all the other globals, which is very difficult if you don't know in advance what was in the code block you exec()'d. So put simply - the bug is that a class namespace is used, but its not a class. On 26/05/2010 13:51, Nick Coghlan wrote: > On 26/05/10 19:48, Mark Dickinson wrote: >> This is a long way from my area of expertise (I'm commenting here >> because it was me who sent Colin here in the first place), and it's >> not clear to me whether this is a bug, and if it is a bug, how it >> could be resolved. What would the impact be of having the compiler >> produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here? > > exec with a single argument = module namespace > exec with two arguments = class namespace > > Class namespaces are deliberately exempted from lexical scoping so > that methods can't see class attributes, hence the example in the > tracker issue works exactly as it would if the code was written as a > class body. > > class C: > y = 3 > def execfunc(): > print y > execfunc() > > With this code, y would end up in C.__dict__ rather than the module > globals (at least, it would if it wasn't for the exception) and the > call to execfunc fails with a NameError when attempting to find y. > > I know I've closed other bug reports that were based on the same > misunderstanding, and I didn't understand it myself until Guido > explained it to me a few years back, so suggestions for improving the > exec documentation in this area would be appreciated. From debatem1 at gmail.com Wed May 26 17:40:56 2010 From: debatem1 at gmail.com (geremy condra) Date: Wed, 26 May 2010 11:40:56 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <20100526125614.0a7ae470@pitrou.net> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> Message-ID: On Wed, May 26, 2010 at 6:56 AM, Antoine Pitrou wrote: > On Wed, 26 May 2010 20:42:12 +1000 > Steven D'Aprano wrote: >> >> I'm not saying that Python-Dev should bend over backwards to accommodate >> such people to the exclusion of all else, but these folks are >> stakeholders too, and their wants and needs are just as worthy as the >> wants and needs of those who prefer a more conservative approach to the >> standard library. > > Well, my "Sumo" proposal was a serious one. > (not serious in that I would offer to give a hand, but in that I think > it could help those people; also, wouldn't it be sensible for users in > a corporate environment to share their efforts and produce something > that can benefit all of them? it's the free software spirit after all) Not in a corporate environment, but I would gladly help with this. Geremy Condra From debatem1 at gmail.com Wed May 26 18:22:04 2010 From: debatem1 at gmail.com (geremy condra) Date: Wed, 26 May 2010 09:22:04 -0700 Subject: [Python-Dev] Sumo In-Reply-To: <1274877963.3162.20.camel@localhost.localdomain> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> Message-ID: On Wed, May 26, 2010 at 5:46 AM, Antoine Pitrou wrote: > Le mercredi 26 mai 2010 ? 13:19 +0100, Paul Moore a ?crit : >> >> I'm not sure how a "Sumo" approach would work in practical terms, and >> this thread isn't really the place to discuss, but there's a couple of >> points I think are worth making: >> >> * For a "Sumo" distribution to make sense, some relatively substantial >> chunk of the standard library would need to be moved *out* to reside >> in the sumo distribution. Otherwise it's not really a "sumo", just a >> couple of modules that "nearly made it into the stdlib", at least for >> the near-to-medium term. > > This is not what I'm suggesting at all. The stdlib wouldn't shrink > (well, we could dump outdated modules but that's a separate decision). > > A hypothetical "Sumo" distribution would be made of those more or less > useful modules that many people (application or framework developers; > for example, take a look at the dozens of direct and indirect > dependencies TurboGears pulls in) need and install. > > The whole point is that it would *not* be supported by python-dev itself > but by a separate group of people (such as you :-)). > And it would have its own rules (surely more relaxed) over inclusion or > deprecation delays, the ability to make compatibility-breaking changes, > release timings, multiple obvious ways to do the same thing, etc. > > And it means it could really bundle a lot of third-party stuff: small > helpers (things like the decorator module), event loops, template > engines, network address abstractions, argument parsers, ORMs, UI > toolkits, etc. > > (a side-effect would be that it could, if it works well, behave as a > good intermediate stress test - a purgatory, if you want - for modules > before they integrate the stdlib) > > If you want an existing analogy, you could look at EasyPHP. Or think of > Python as the Gnome or KDE project (consistent and aiming at providing > the most important everyday tools, but quite focused), and "Sumo" as an > entire distribution of disparate Linux GUI apps. > > Regards > > Antoine. I'd also point out that creating a sumo distribution would focus attention on the need to port those libraries which were a part of it to python3, which would help to weaken the argument that there aren't any packages written for it. Geremy Condra From tjreedy at udel.edu Wed May 26 18:27:17 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 26 May 2010 12:27:17 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFD173B.9010703@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com> Message-ID: > On 26/05/10 20:57, Greg Ewing wrote: >> (More generally, I'm inclined to think that introducing >> a namespace package for a category of modules having >> existing members in the stdlib is an anti-pattern, As a user, I agree with this. >> unless it's done during the kind of namespace refactoring >> that we won't get another chance to perform until Py4k.) Is Steven D'Aprano's suggestion (in another post) possible? > I'm sure that it can be done easily, although not quickly. For > instance, we move threading into the concurrent namespace, and leave > behind in its place a stub: > > from concurrent.threading import * > > Then for (say) 3.3 the stub could gain a PendingDeprecation warning, > then in 3.4 a Deprecation warning, and finally in 3.5 or 3.6 it > could be removed. On 5/26/2010 8:42 AM, Nick Coghlan wrote: > _thread, threading, Queue and multiprocessing do likely belong here, but > moving them isn't likely to be worth the pain. As a user, I disagree and think collecting together these and future modules (pun intended) would be beneficial. There are, from my viewpoint, too many top level modules already. [and in another thread] > Yes, we're setting ourselves up for inevitable questions as to why > the existing modules are top level rather than part of this package, Yes, forever until they are put in the package. > but the minimal pain responsethere For the developers, for the short term > would be to link to them from the > package documentation with a note along the lines of "for historical > reasons, some modules you might reasonably expect to find in this > package are instead provided as top level modules". You are, in my opinion, overly discounting the cognitive load and confusion on the part of new users. It would be much better to link *to* subpackage documentation *from* a top level entries (until deleted) and just say that the top level synonyms are present for the obvious historical reason that there once no package, just modules. I am suggesting that if we add a package, we do it right, from the beginning. Terry Jan Reedy From jyasskin at gmail.com Wed May 26 18:25:53 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Wed, 26 May 2010 09:25:53 -0700 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFCFE8B.1080205@canterbury.ac.nz> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> Message-ID: On Wed, May 26, 2010 at 3:57 AM, Greg Ewing wrote: > Having read through the PEP again, here are my thoughts. > > * I'm bothered by the term "future". To my mind, it's > too long on cleverness and too short on explanativeness. > > I think that the standard library is no place for > cuteness of naming. The name of a stdlib module should > reflect its functionality in some straightforward and > obvious way. If I were looking for a thread pool or > process pool implementation, the word "future" is not > something that would spring readily to mind. Please go re-read the older threads on this. For example, http://mail.python.org/pipermail/python-dev/2010-March/098279.html. > * It seems unnecessarily verbose to tack "Executor" > onto the end of every Executor subclass. They could > simply be called ThreadPool and ProcessPool without > losing anything. +0 > * I don't see a strong reason to put this module > inside a newly-created namespace. If there were a > namespace called "concurrent", I would expect to find > other existing concurrency-related modules there as > well, such as threading and multiprocessing. But we > can't move them there without breaking existing code. Again, please re-read the older thread (which you participated in). For example, http://mail.python.org/pipermail/python-dev/2010-March/098200.html. Jeffrey From tjreedy at udel.edu Wed May 26 19:40:43 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 26 May 2010 13:40:43 -0400 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: Message-ID: Mark Dickinson wrote (with interactice prompts removed) code = """\ y = 3 def f(): return y . f() """ exec code in {} # works fine exec code in {}, {} # dies with a NameError Traceback (most recent call last): File "", line 1, in File "", line 4, in File "", line 3, in f NameError: global name 'y' is not defined I verified that 3.1 (with exec adjusted to be a function) operates the same. On 5/26/2010 8:51 AM, Nick Coghlan wrote: >exec with a single argument = module namespace >exec with two arguments = class namespace I verified in 3.1 that indenting 'code' and prepending 'class C():\n' gives the same error and that prepending 'def f():\n' now, with nexted function namespaces, does not give an error, although it would have been an error before Python 2.2, when there were no nested function namespaces. On 5/26/2010 10:03 AM, Colin H wrote: > Thanks for the details on why the observed behaviour occurs - very > clear. My only query would be why this is considered correct? Why is > it running as a class namespace, when it is not a class? You are expecting that it run as a function namespace (with post 2.2 nesting), when it is not a function. Why is that any better? > Is there any > reason why this is not considered a mistake? Slightly concerned that > this is being considered not a bug because 'it is how it is'. In original Python, the snippet would have given an error whether you thought of it as being in a class or function context, which is how anyone who knew Python then would have expected. Consistency is not a bug. When nested function namespaces were introduced, the behavior of exec was left unchanged. Backward compatibility is not a bug. A change could have been proposed for 3.x, but I do not remember such a discussion and expect it would have been rejected. One can get the nested function behavior by doing what I did in the test mentioned above. One could easily write a nested_exec function to do the wrapping automatically. ----- In http://bugs.python.org/issue8824 I suggest that "In all cases, if the optional parts are omitted, the code is executed in the current scope. If only globals is provided, it must be a dictionary, which will be used for both the global and the local variables. If globals and locals are given, they are used for the global and local variables, respectively. If provided, locals can be any mapping object." be followed by "If only globals is provided or if onedict is provided as both globals and locals, the code is executed in a new top-level scope. If different objects are given as globals and locals, the code is executed as if it were in a class statement in a new top-level scope." to make the behavior clearer. Terry Jan Reedy From hawkett at gmail.com Wed May 26 22:07:52 2010 From: hawkett at gmail.com (Colin H) Date: Wed, 26 May 2010 21:07:52 +0100 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: Message-ID: The changes to the docs will definitely help in understanding why this behaves as it does. I would like like to take one last stab though at justifying why this behaviour isn't correct - will leave it alone if these arguments don't stack up :) Appreciate the input and discussion. Terry Jan Reedy wrote > You are expecting that it run as a function namespace (with post 2.2 > nesting), when it is not a function. Why is that any better? Because a class namespace (as I see it) was implemented to treat a specific situation - i.e. that functions in classes cannot see class variables. exec() is a far more generic instrument that has no such explicit requirement - i.e. it feels like hijacking an edge case to meet a requirement that doesn't exist. However 'all locals in an enclosing scope are made available in the function namespace' is generally understood as python's generic closure implementation, and would match more effectively the generic nature of the exec() statement. A litmus test for this sort of thing - if you polled 100 knowledgeable python devs who hadn't encountered this problem or this thread and asked if they would expect exec() to run as a class or function namespace, I think you'd struggle to get 1 of them to expect a class namespace. Functions are the more generic construct, and thus more appropriate for the generic nature of exec() (IMHO). It would appear that the only actual requirement not to make locals in an enclosing scope available in a nested function scope is for a class. The situation we are discussing seems have created a similar requirement for exec(), but with no reason. > In original Python, the snippet would have given an error whether you > thought of it as being in a class or function context, which is how > anyone who knew Python then would have expected. Consistency is not a bug. > When nested function namespaces were introduced, the behavior of exec > was left unchanged. Backward compatibility is not a bug. Generally, most other behaviour did change - locals in enclosing scopes *did* become available in the nested function namespace, which was not backward compatible. Why is a special case made to retain consistency and backward compatibility for code run using exec()? It's all python code. Inconsistent backward compatibility might be considered a bug. Cheers, Colin From p.f.moore at gmail.com Thu May 27 00:41:51 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 26 May 2010 23:41:51 +0100 Subject: [Python-Dev] Sumo In-Reply-To: <1274877963.3162.20.camel@localhost.localdomain> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> Message-ID: On 26 May 2010 13:46, Antoine Pitrou wrote: > This is not what I'm suggesting at all. The stdlib wouldn't shrink > (well, we could dump outdated modules but that's a separate decision). Ah, OK. In that case, I see the argument for a "Sumo" distribution as weak for a different reason - for general use, the standard library is (nearly!) sufficient, and ignoring specialised use cases, there aren't enough generally useful modules to warrant a "Sumo" distribution (you'd essentially be talking about stuff that "nearly made it into the stdlib", and there's not a huge amount of that). Specialised distributions are another matter - I can see a "web stack" distribution comprising your TurboGears example (or should it be Django, or...?). Enthought essentially do that for a "Scientific Python" distribution. There could easily be others. But a general purpose "Sumo" distribution *on top of* the stdlib? I'm skeptical. (Personally, my "essential extras" are pywin32, cx_Oracle and that's about it - futures might make it if it doesn't get into the stdlib, but that's about all). I'm genuinely struggling to see how a Sumo distribution ever comes into being under your proposal. There's no evidence that anyone wants it (otherwise it would have been created by now!!) and until it exists, it's not a plausible "place" to put modules that don't make it into the stdlib. So (unless I'm missing something) your argument seems to be that if enough good stuff is rejected for stdlib inclusion, this will prompt the people who wanted that stuff included to create a sumo distribution, which addresses the "too many dependencies is bad" argument for inclusion in the stdlib. That sounds like a suspiciously circular argument to me... Paul. From solipsis at pitrou.net Thu May 27 00:51:19 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 27 May 2010 00:51:19 +0200 Subject: [Python-Dev] Sumo In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> Message-ID: <1274914279.3162.76.camel@localhost.localdomain> Le mercredi 26 mai 2010 ? 23:41 +0100, Paul Moore a ?crit : > > But a general > purpose "Sumo" distribution *on top of* the stdlib? I'm skeptical. > (Personally, my "essential extras" are pywin32, cx_Oracle and that's > about it - futures might make it if it doesn't get into the stdlib, > but that's about all). Well, unless you are short on megabytes or on download bandwidth, would you really care to get some modules you never use? (there are probably some in the stdlib too!) > I'm genuinely struggling to see how a Sumo distribution ever comes > into being under your proposal. There's no evidence that anyone wants > it (otherwise it would have been created by now!!) and until it > exists, it's not a plausible "place" to put modules that don't make it > into the stdlib. I don't think all package owners dream of putting their software in the stdlib. It's likely some don't care, and it's likely some would actively refuse it (because e.g. they don't want to lose control). So the suggestion is not to somehow salvage packages which "don't make it into the stdlib", but to build a broader distribution of packages without necessarily having people bow to the constraints of stdlib inclusion. Of course, I agree with you that someone has to do it if they want it to happen :-) Regards Antoine. From debatem1 at gmail.com Thu May 27 01:11:27 2010 From: debatem1 at gmail.com (geremy condra) Date: Wed, 26 May 2010 16:11:27 -0700 Subject: [Python-Dev] Sumo In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> Message-ID: On Wed, May 26, 2010 at 3:41 PM, Paul Moore wrote: > On 26 May 2010 13:46, Antoine Pitrou wrote: >> This is not what I'm suggesting at all. The stdlib wouldn't shrink >> (well, we could dump outdated modules but that's a separate decision). > > Ah, OK. In that case, I see the argument for a "Sumo" distribution as > weak for a different reason - for general use, the standard library is > (nearly!) sufficient, and ignoring specialised use cases, there aren't > enough generally useful modules to warrant a "Sumo" distribution > (you'd essentially be talking about stuff that "nearly made it into > the stdlib", and there's not a huge amount of that). > > Specialised distributions are another matter - I can see a "web stack" > distribution comprising your TurboGears example (or should it be > Django, or...?). Enthought essentially do that for a "Scientific > Python" distribution. There could easily be others. But a general > purpose "Sumo" distribution *on top of* the stdlib? I'm skeptical. > (Personally, my "essential extras" are pywin32, cx_Oracle and that's > about it - futures might make it if it doesn't get into the stdlib, > but that's about all). I'm not clear, you seem to be arguing that there's a market for many augmented python distributions but not one. Why not just have one that includes the best from each domain? > I'm genuinely struggling to see how a Sumo distribution ever comes > into being under your proposal. There's no evidence that anyone wants > it (otherwise it would have been created by now!!) Everything worth making has already been made? > and until it exists, it's not a plausible "place" to put modules that don't > make it into the stdlib. Of course its implausible to put something somewhere that doesn't exist... until it does. > So (unless I'm missing something) your argument seems > to be that if enough good stuff is rejected for stdlib inclusion, this > will prompt the people who wanted that stuff included to create a sumo > distribution, which addresses the "too many dependencies is bad" > argument for inclusion in the stdlib. That sounds like a suspiciously > circular argument to me... I'd say rather that there are a large number of specialized tools which aren't individually popular enough to be included in Python, but which when taken together greatly increase its utility, and that sumo offers a way to provide that additional utility to python's users without forcing python core devs to shoulder the maintenance burden. Geremy Condra From greg.ewing at canterbury.ac.nz Thu May 27 01:36:07 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 27 May 2010 11:36:07 +1200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> Message-ID: <4BFDB067.4060800@canterbury.ac.nz> Brian Quinlan wrote: > I think that Jesse was planning to add some functionality to this > namespace. Even if that happens, the existing threading and multiprocessing modules would remain outside of it. > You could have general thread pools that aren't related to executors Yes, but it should be fairly obvious that the ones defined in the futures module have to do with futures. Namespaces are only a honking great idea if you actually let them do the job they're designed for. > I thought that the specification would be difficult to follow without > examples to pave the way. Well, for me, what happened was that I saw the examples and thought "WTF is going on here?" Then I read the specification to figure out how the examples worked. It might be better to have a tutorial section preceeding the specification section, containing explanation interspersed with examples. > I think that > idiomatic future use will end up looking similar to my examples. Maybe, but code written for pedagogical purposes needs to meet a particularly high standard of clarity. Remember that the reader isn't yet familiar with the idioms, so idiomatic code isn't necessarily going to be easy for him to follow. >> * Is it possible to have more than one Executor active >> at a time? > > Of course. That's good, but I don't think that the "of course" is at all obvious, considering that things such as GUI event loops generally can't be mixed easily. -- Greg From greg.ewing at canterbury.ac.nz Thu May 27 01:38:50 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 27 May 2010 11:38:50 +1200 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: Message-ID: <4BFDB10A.4080002@canterbury.ac.nz> Mark Dickinson wrote: >>>>code = """\ > > ... y = 3 > ... def f(): > ... return y > ... f() > ... """ > >>>>exec code in {} # works fine >>>>exec code in {}, {} # dies with a NameError Seems to me the whole idea of being able to specify separate global and local scopes for top-level code is screwy in the first place. Are there any use cases for it? Maybe the second scope argument to exec() should be deprecated? -- Greg From fuzzyman at voidspace.org.uk Thu May 27 01:33:46 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 27 May 2010 00:33:46 +0100 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: <4BFDB10A.4080002@canterbury.ac.nz> References: <4BFDB10A.4080002@canterbury.ac.nz> Message-ID: <4BFDAFDA.7060404@voidspace.org.uk> On 27/05/2010 00:38, Greg Ewing wrote: > Mark Dickinson wrote: > >>>>> code = """\ >> >> ... y = 3 >> ... def f(): >> ... return y >> ... f() >> ... """ >> >>>>> exec code in {} # works fine >>>>> exec code in {}, {} # dies with a NameError > > Seems to me the whole idea of being able to specify > separate global and local scopes for top-level code is > screwy in the first place. Are there any use cases for > it? Maybe the second scope argument to exec() should > be deprecated? > Sounds good to me, certainly ends the confusion over this undoubtedly unintuitive behaviour. :-) Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From guido at python.org Thu May 27 01:44:02 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 26 May 2010 16:44:02 -0700 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: <4BFDAFDA.7060404@voidspace.org.uk> References: <4BFDB10A.4080002@canterbury.ac.nz> <4BFDAFDA.7060404@voidspace.org.uk> Message-ID: Let me quickly jump in before someone actually deletes the feature. Nick Coghlan and Thomas Wouters had it right; there is still a use case. Don't kill it -- documenting it better is of course fine. It *might* be possible to add a closure to the definition of f in the case where globals != locals, but I doubt that that would be worth it, and there's probably some code out there that would actually break, so I'd say this is not a priority. --Guido On Wed, May 26, 2010 at 4:33 PM, Michael Foord wrote: > On 27/05/2010 00:38, Greg Ewing wrote: >> >> Mark Dickinson wrote: >> >>>>>> code = """\ >>> >>> ... y = 3 >>> ... def f(): >>> ... return y >>> ... f() >>> ... """ >>> >>>>>> exec code in {} # works fine >>>>>> exec code in {}, {} # dies with a NameError >> >> Seems to me the whole idea of being able to specify >> separate global and local scopes for top-level code is >> screwy in the first place. Are there any use cases for >> it? Maybe the second scope argument to exec() should >> be deprecated? >> > Sounds good to me, certainly ends the confusion over this undoubtedly > unintuitive behaviour. :-) > > Michael > > > -- > http://www.ironpythoninaction.com/ > http://www.voidspace.org.uk/blog > > READ CAREFULLY. By accepting and reading this email you agree, on behalf of > your employer, to release me from all obligations and waivers arising from > any and all NON-NEGOTIATED agreements, licenses, terms-of-service, > shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, > non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have > entered into with your employer, its partners, licensors, agents and > assigns, in perpetuity, without prejudice to my ongoing rights and > privileges. You further represent that you have the authority to release me > from any BOGUS AGREEMENTS on behalf of your employer. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Thu May 27 01:57:26 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 May 2010 09:57:26 +1000 Subject: [Python-Dev] Sumo In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> Message-ID: <4BFDB566.2060502@gmail.com> On 27/05/10 09:11, geremy condra wrote: >> Specialised distributions are another matter - I can see a "web stack" >> distribution comprising your TurboGears example (or should it be >> Django, or...?). Enthought essentially do that for a "Scientific >> Python" distribution. There could easily be others. But a general >> purpose "Sumo" distribution *on top of* the stdlib? I'm skeptical. >> (Personally, my "essential extras" are pywin32, cx_Oracle and that's >> about it - futures might make it if it doesn't get into the stdlib, >> but that's about all). > > I'm not clear, you seem to be arguing that there's a market for many > augmented python distributions but not one. Why not just have one > that includes the best from each domain? Because scientists, financial analysts, web designers, etc all have different needs. A targeted distribution like Scientific Python will include nearly all the stuff a scientist is likely to need, but a financial analyst or web designer would find it lacking. As Paul points out, the current size of the set of modules that are sufficiently general purpose and of high enough quality to qualify for python-dev's blessing, but wouldn't be suitable for inclusion in the normal standard library is fairly small. Particular when most developers are able to get sufficiently valuable modules from PyPI if they genuinely need them. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu May 27 02:03:23 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 May 2010 10:03:23 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com> Message-ID: <4BFDB6CB.7030104@gmail.com> On 27/05/10 02:27, Terry Reedy wrote: > I am suggesting that if we add a package, we do it right, from the > beginning. This is a reasonable point of view, but I wouldn't want to hold up PEP 3148 over it (call it a +0 for the idea in general, but a -1 for linking it to the acceptance of PEP 3148). A separate short PEP proposing a migration plan that could be accepted or rejected independently of PEP 3148 would likely be valuable. E.g. - no change in 2.x (obviously) - add concurrent.* alternate names in 3.x - rearrange documentation in 3.x, with pointers from old names to new names - put a PendingDeprecationWarning on the old names, but otherwise leave them alone indefinitely - add 2to3 fixers to translate from the old names to the new names in import statements Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From jnoller at gmail.com Thu May 27 02:04:46 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 26 May 2010 20:04:46 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFDB067.4060800@canterbury.ac.nz> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> <4BFDB067.4060800@canterbury.ac.nz> Message-ID: On Wed, May 26, 2010 at 7:36 PM, Greg Ewing wrote: > Brian Quinlan wrote: > >> I think that Jesse was planning to add some functionality to this >> ?namespace. > > Even if that happens, the existing threading and multiprocessing > modules would remain outside of it. Not entirely; once concurrent.* comes into existence, I will seriously begin looking at what we can move out of multiprocessing, into concurrent.* alongside futures. >> You could have general thread pools that aren't related to executors > > Yes, but it should be fairly obvious that the ones defined > in the futures module have to do with futures. Namespaces are > only a honking great idea if you actually let them do the job > they're designed for. concurrent.* is the namespace, futures is the package within the namespace - concurrent.futures is highly descriptive of the items contained therein. jesse From jnoller at gmail.com Thu May 27 02:05:28 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 26 May 2010 20:05:28 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFDB6CB.7030104@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com> <4BFDB6CB.7030104@gmail.com> Message-ID: On Wed, May 26, 2010 at 8:03 PM, Nick Coghlan wrote: > On 27/05/10 02:27, Terry Reedy wrote: >> >> I am suggesting that if we add a package, we do it right, from the >> beginning. > > This is a reasonable point of view, but I wouldn't want to hold up PEP 3148 > over it (call it a +0 for the idea in general, but a -1 for linking it to > the acceptance of PEP 3148). > > A separate short PEP proposing a migration plan that could be accepted or > rejected independently of PEP 3148 would likely be valuable. > > E.g. > ?- no change in 2.x (obviously) > ?- add concurrent.* alternate names in 3.x > ?- rearrange documentation in 3.x, with pointers from old names to new names > ?- put a PendingDeprecationWarning on the old names, but otherwise leave > them alone indefinitely > ?- add 2to3 fixers to translate from the old names to the new names in > import statements > > Cheers, > Nick. Agreed; and intended as a different PEP. From ncoghlan at gmail.com Thu May 27 02:12:30 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 May 2010 10:12:30 +1000 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: Message-ID: <4BFDB8EE.4010307@gmail.com> On 27/05/10 06:07, Colin H wrote: >> In original Python, the snippet would have given an error whether you >> thought of it as being in a class or function context, which is how >> anyone who knew Python then would have expected. Consistency is not a bug. > >> When nested function namespaces were introduced, the behavior of exec >> was left unchanged. Backward compatibility is not a bug. > > Generally, most other behaviour did change - locals in enclosing > scopes *did* become available in the nested function namespace, which > was not backward compatible. Why is a special case made to retain > consistency and backward compatibility for code run using exec()? It's > all python code. Inconsistent backward compatibility might be > considered a bug. Because strings are opaque to the compiler. The lexical scoping has *no idea* what is inside the string, and the exec operation only has 3 things available to it: - the code object compiled from the string - the supplied globals namespace - the supplied locals namespace It isn't a special case, it's the only way it can possible work. Consider a more complex example: def get_exec_str(): y = 3 return "print(y)" exec(get_exec_str()) Should that code work? Or consider this one: def get_exec_str(): y = 3 return "print y" def run_exec_str(str_to_run): y = 5 exec(str_to_run) run_exec_str(get_exec_str()) Should that work? If yes, should it print 3 or 5? Lexical scoping only works for code that is compiled as part of a single operation - the separation between the compilation of the individual string and the code defining that string means that the symbol table analysis needed for lexical scoping can't cross the boundary. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu May 27 02:19:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 May 2010 10:19:50 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFDB067.4060800@canterbury.ac.nz> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> <4BFDB067.4060800@canterbury.ac.nz> Message-ID: <4BFDBAA6.2010104@gmail.com> On 27/05/10 09:36, Greg Ewing wrote: > Brian Quinlan wrote: > >> I think that Jesse was planning to add some functionality to this >> namespace. > > Even if that happens, the existing threading and multiprocessing > modules would remain outside of it. > >> You could have general thread pools that aren't related to executors > > Yes, but it should be fairly obvious that the ones defined > in the futures module have to do with futures. Namespaces are > only a honking great idea if you actually let them do the job > they're designed for. futures.ThreadPoolExecutor would likely be refactored to inherit from the mooted pool.ThreadPool. I'd like to leave that option open, and having two classes with the same name from different modules in a single inheritance tree is one of the places where module namespacing still isn't quite as tidy as we might wish. I'd also consider a simple thread pool and an actual executor different things. I'm fine with the longer names, but if I was going to drop a word from the names, it would actually be "Pool" (i.e. ThreadExecutor, ProcessExecutor). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From debatem1 at gmail.com Thu May 27 02:15:41 2010 From: debatem1 at gmail.com (geremy condra) Date: Wed, 26 May 2010 17:15:41 -0700 Subject: [Python-Dev] Sumo In-Reply-To: <4BFDB566.2060502@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> <4BFDB566.2060502@gmail.com> Message-ID: On Wed, May 26, 2010 at 4:57 PM, Nick Coghlan wrote: > On 27/05/10 09:11, geremy condra wrote: >>> >>> Specialised distributions are another matter - I can see a "web stack" >>> distribution comprising your TurboGears example (or should it be >>> Django, or...?). Enthought essentially do that for a "Scientific >>> Python" distribution. There could easily be others. But a general >>> purpose "Sumo" distribution *on top of* the stdlib? I'm skeptical. >>> (Personally, my "essential extras" are pywin32, cx_Oracle and that's >>> about it - futures might make it if it doesn't get into the stdlib, >>> but that's about all). >> >> I'm not clear, you seem to be arguing that there's a market for many >> augmented python distributions but not one. Why not just have one >> that includes the best from each domain? > > Because scientists, financial analysts, web designers, etc all have > different needs. My point is just that a web designer probably doesn't care if he's got numpy, nor does a mathematician care if he has cherrypy onboard. They only care when the tools they need aren't there, which is where sumo can help. > A targeted distribution like Scientific Python will include nearly all the > stuff a scientist is likely to need, but a financial analyst or web designer > would find it lacking. Seems like the same point as above, I might be missing something. > As Paul points out, the current size of the set of modules that are > sufficiently general purpose and of high enough quality to qualify for > python-dev's blessing, but wouldn't be suitable for inclusion in the normal > standard library is fairly small. Particular when most developers are able > to get sufficiently valuable modules from PyPI if they genuinely need them. Seems like the point is not to focus on the general purpose ones, but rather to include domain or task specific libraries, and libraries that are close to (but not at) the level where they would be considered for inclusion. Geremy Condra From solipsis at pitrou.net Thu May 27 02:29:35 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 27 May 2010 02:29:35 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> <4BFDB067.4060800@canterbury.ac.nz> <4BFDBAA6.2010104@gmail.com> Message-ID: <20100527022935.2cfd04c5@pitrou.net> On Thu, 27 May 2010 10:19:50 +1000 Nick Coghlan wrote: > > futures.ThreadPoolExecutor would likely be refactored to inherit from > the mooted pool.ThreadPool. There still doesn't seem to be reason to have two different thread pool APIs, though. Shouldn't there be one obvious way to do it? > I'd also consider a simple thread pool and an actual executor different > things. I'm fine with the longer names, but if I was going to drop a > word from the names, it would actually be "Pool" (i.e. ThreadExecutor, > ProcessExecutor). To me, ThreadPool looks a lot more obvious than ThreadExecutor ("obvious" in that I can easily find it again, and I don't need to read some documentation to know what it is). Regards Antoine. From hawkett at gmail.com Thu May 27 02:37:52 2010 From: hawkett at gmail.com (Colin H) Date: Thu, 27 May 2010 01:37:52 +0100 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: Message-ID: Mark Dickinson wrote: > Seems to me the whole idea of being able to specify > separate global and local scopes for top-level code is > screwy in the first place. Are there any use cases for > it? Maybe the second scope argument to exec() should > be deprecated? It is running as class namespace that makes the argument that there's no use case. I agree - I can't think of any decent use cases for exec() as class namespace - defining functions and classes only works for a subset of function and class definitions However, if exec() ran as function namespace instead, then the locals dictionary will contain all the definitions from the exec()'d code block, and only those definitions. Very useful. This is a major use case for exec() - defining code from strings (e.g. enabling you to store python code in the database), and using it at runtime. It seems to me this must have been the point of locals in the first place. If you just use globals, then your definitions exist amongst a whole bunch of other python stuff, and unless you know in advance what was defined in your code block, its very difficult to extract them. From guido at python.org Thu May 27 02:38:04 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 26 May 2010 17:38:04 -0700 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: <4BFDB8EE.4010307@gmail.com> References: <4BFDB8EE.4010307@gmail.com> Message-ID: On Wed, May 26, 2010 at 5:12 PM, Nick Coghlan wrote: > On 27/05/10 06:07, Colin H wrote: >>> >>> In original Python, the snippet would have given an error whether you >>> thought of it as being in a class or function context, which is how >>> anyone who knew Python then would have expected. Consistency is not a >>> bug. >> >>> When nested function namespaces were introduced, the behavior of exec >>> was left unchanged. Backward compatibility is not a bug. >> >> Generally, most other behaviour did change - locals in enclosing >> scopes *did* become available in the nested function namespace, which >> was not backward compatible. ?Why is a special case made to retain >> consistency and backward compatibility for code run using exec()? It's >> all python code. Inconsistent backward compatibility might be >> considered a bug. > > Because strings are opaque to the compiler. The lexical scoping has *no > idea* what is inside the string, and the exec operation only has 3 things > available to it: > ?- the code object compiled from the string > ?- the supplied globals namespace > ?- the supplied locals namespace > > It isn't a special case, it's the only way it can possible work. > > Consider a more complex example: > > ?def get_exec_str(): > ? ?y = 3 > ? ?return "print(y)" > > ?exec(get_exec_str()) > > Should that code work? > > Or consider this one: > > ?def get_exec_str(): > ? ?y = 3 > ? ?return "print y" > > ?def run_exec_str(str_to_run): > ? ?y = 5 > ? ?exec(str_to_run) > > ?run_exec_str(get_exec_str()) > > Should that work? If yes, should it print 3 or 5? > > Lexical scoping only works for code that is compiled as part of a single > operation - the separation between the compilation of the individual string > and the code defining that string means that the symbol table analysis > needed for lexical scoping can't cross the boundary. Hi Nick, I don't think Colin was asking for such things. His use case is clarify by this post of his: On Wed, May 26, 2010 at 7:03 AM, Colin H wrote: > A really good reason why you would want to provide a separate locals > dictionary is to get access to the stuff that was defined in the > exec()'d code block. Unfortunately this use case is broken by the > current behaviour. The only way to get the definitions from the > exec()'d code block is to supply a single dictionary, and then try to > weed out the definitions from amongst all the other globals, which is > very difficult if you don't know in advance what was in the code block > you exec()'d. I think here's an example of what he's asking for def define_stuff(user_code): context = {'FOO': 42, 'BAR': 10**100} stuff = {} exec(user_code, context, stuff) return stuff In some other place, define_stuff() is called like this: user_code = """ EXTRA = 1.1 def func(): return FOO * BAR * EXTRA """ stuff = define_stuff(user_code) func = stuff['func'] print(func()) This can't find the EXTRA variable found. (Another example would be defining a recursive function -- the function can't find itself.) The alternative (which Colin complains about) is for define_stuff() to use a single namespace, initialized with the context, and to return that -- but he's right that in that case FOO and BAR would be exported as part of stuff, which may not be the intention. (Another solution would be to put FOO and BAR in the builtins dict -- but that has other problems of course.) > So put simply - the bug is that a class namespace is used, but its not a class. Well, really, the bug is that no closure is created even though there are separate globals and locals. Unfortunately this is because while at the point where the code is *executed* the two different contexts are clearly present, at the point where it is *compiled* this is not the case -- the compiler normally uses syntactic clues to decide whether to generate code using closures, in particular, the presence of nested functions. Note that define_stuff() could be equivalently written like this, where it is more obvious that the compiler doesn't know about the separate namespaces: def define_stuff(user_code): context = {...} stuff = {} compiled_code = compile(user_code, "", "exec") exec(user_code, context, stuff) return stuff This is not easy to fix. The best short-term work-around is probably a hack like this: def define_stuff(user_code): context = {...} stuff = {} stuff.update(context) exec(user_code, stuff) for key in context: if key in stuff and stuff[key] == context[key]: del stuff[key] return stuff -- --Guido van Rossum (python.org/~guido) From guido at python.org Thu May 27 02:44:21 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 26 May 2010 17:44:21 -0700 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: Message-ID: On Wed, May 26, 2010 at 5:37 PM, Colin H wrote: > Mark Dickinson wrote: > >> Seems to me the whole idea of being able to specify >> separate global and local scopes for top-level code is >> screwy in the first place. Are there any use cases for >> it? Maybe the second scope argument to exec() should >> be deprecated? > > It is running as class namespace that makes the argument that there's > no use case. I agree - I can't think of any decent use cases for > exec() as class namespace - defining functions and classes only works > for a subset of function and class definitions > > However, if exec() ran as function namespace instead, then the locals > dictionary will contain all the definitions from the exec()'d code > block, and only those definitions. Very useful. Hmm... I see your point, but fear that the implementation is tricky, since it would mean every exec'ed block would have to be compiled as if it were a function body, and that probably violates some other invariants. > This is a major use > case for exec() - defining code from strings (e.g. enabling you to > store python code in the database), and using it at runtime. It seems > to me this must have been the point of locals in the first place. > > If you just use globals, then your definitions exist amongst a whole > bunch of other python stuff, and unless you know in advance what was > defined in your code block, its very difficult to extract them. Check the reconstruction of your use case I just posted. I think you re thinking context == globals(), which makes sense. -- --Guido van Rossum (python.org/~guido) From hawkett at gmail.com Thu May 27 02:53:15 2010 From: hawkett at gmail.com (Colin H) Date: Thu, 27 May 2010 01:53:15 +0100 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: <4BFDB8EE.4010307@gmail.com> Message-ID: Hi Guido, Thanks for the possible workaround - unfortunately 'stuff' will contain a whole stack of things that are not in 'context', and were not defined in 'user_code' - things that python embeds - a (very small) selection - {..., 'NameError': , 'BytesWarning': , 'dict': , 'input': , 'oct': , 'bin': , ...} It makes sense why this happens of course, but upon return, the globals dict is very large, and finding the stuff you defined in your user_code amongst it is a very difficult task. Avoiding this problem is the 'locals' use-case for me. Cheers, Colin On Thu, May 27, 2010 at 1:38 AM, Guido van Rossum wrote: > This is not easy to fix. The best short-term work-around is probably a > hack like this: > > def define_stuff(user_code): > ?context = {...} > ?stuff = {} > ?stuff.update(context) > ?exec(user_code, stuff) > ?for key in context: > ? ?if key in stuff and stuff[key] == context[key]: > ? ? ?del stuff[key] > ?return stuff > > -- > --Guido van Rossum (python.org/~guido) > From pje at telecommunity.com Thu May 27 03:05:40 2010 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 26 May 2010 21:05:40 -0400 Subject: [Python-Dev] Sumo In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> Message-ID: <20100527010538.C7AA83A40A5@sparrow.telecommunity.com> At 11:41 PM 5/26/2010 +0100, Paul Moore wrote: >I'm genuinely struggling to see how a Sumo distribution ever comes >into being under your proposal. There's no evidence that anyone wants >it (otherwise it would have been created by now!!) Actually, sumo distributions *have* been created; it's just that there are so *many* of them: * Linux distributions * BSD & Mac "ports" systems * ActivePython, the Enthought Distribution, etc. * certain distutils extensions that I shall not name When the idea of "sumo" distributions were first being batted around (10-12 years ago?), I'm not sure even *distutils* existed yet, let alone any of this other stuff. I mean, yeah, the port systems and distros existed, but their coverage wasn't nearly as good as it is today. From guido at python.org Thu May 27 03:05:46 2010 From: guido at python.org (Guido van Rossum) Date: Wed, 26 May 2010 18:05:46 -0700 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: <4BFDB8EE.4010307@gmail.com> Message-ID: On Wed, May 26, 2010 at 5:53 PM, Colin H wrote: > ? Thanks for the possible workaround - unfortunately 'stuff' will > contain a whole stack of things that are not in 'context', and were > not defined in 'user_code' - things that python embeds - a (very > small) selection - > > {..., 'NameError': , 'BytesWarning': > , 'dict': , 'input': > , 'oct': , > 'bin': , ...} > > It makes sense why this happens of course, but upon return, the > globals dict is very large, and finding the stuff you defined in your > user_code amongst it is a very difficult task. ?Avoiding this problem > is the 'locals' use-case for me. ?Cheers, No, if taken literally that doesn't make sense. Those are builtins. I think you are mistaken that each of those (e.g. NameError) is in stuff -- they are in stuff['__builtins__'] which represents the built-in namespace. You should remove that key from stuff as well. --Guido > Colin > > On Thu, May 27, 2010 at 1:38 AM, Guido van Rossum wrote: >> This is not easy to fix. The best short-term work-around is probably a >> hack like this: >> >> def define_stuff(user_code): >> ?context = {...} >> ?stuff = {} >> ?stuff.update(context) >> ?exec(user_code, stuff) >> ?for key in context: >> ? ?if key in stuff and stuff[key] == context[key]: >> ? ? ?del stuff[key] >> ?return stuff >> >> -- >> --Guido van Rossum (python.org/~guido) >> > -- --Guido van Rossum (python.org/~guido) From hawkett at gmail.com Thu May 27 03:13:42 2010 From: hawkett at gmail.com (Colin H) Date: Thu, 27 May 2010 02:13:42 +0100 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: <4BFDB8EE.4010307@gmail.com> Message-ID: Of course :) - I need to pay more attention. Your workaround should do the trick. It would make sense if locals could be used for this purpose, but the workaround doesn't add so much overhead in most situations. Thanks for the help, much appreciated, Colin On Thu, May 27, 2010 at 2:05 AM, Guido van Rossum wrote: > On Wed, May 26, 2010 at 5:53 PM, Colin H wrote: >> ? Thanks for the possible workaround - unfortunately 'stuff' will >> contain a whole stack of things that are not in 'context', and were >> not defined in 'user_code' - things that python embeds - a (very >> small) selection - >> >> {..., 'NameError': , 'BytesWarning': >> , 'dict': , 'input': >> , 'oct': , >> 'bin': , ...} >> >> It makes sense why this happens of course, but upon return, the >> globals dict is very large, and finding the stuff you defined in your >> user_code amongst it is a very difficult task. ?Avoiding this problem >> is the 'locals' use-case for me. ?Cheers, > > No, if taken literally that doesn't make sense. Those are builtins. I > think you are mistaken that each of those (e.g. NameError) is in stuff > -- they are in stuff['__builtins__'] which represents the built-in > namespace. You should remove that key from stuff as well. > > --Guido > >> Colin >> >> On Thu, May 27, 2010 at 1:38 AM, Guido van Rossum wrote: >>> This is not easy to fix. The best short-term work-around is probably a >>> hack like this: >>> >>> def define_stuff(user_code): >>> ?context = {...} >>> ?stuff = {} >>> ?stuff.update(context) >>> ?exec(user_code, stuff) >>> ?for key in context: >>> ? ?if key in stuff and stuff[key] == context[key]: >>> ? ? ?del stuff[key] >>> ?return stuff >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> >> > > > > -- > --Guido van Rossum (python.org/~guido) > From greg.ewing at canterbury.ac.nz Thu May 27 03:46:07 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 27 May 2010 13:46:07 +1200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4BF672E8.1080001@canonical.com> <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <20100526120651.GA12751@laurie.devork.be> <8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com> Message-ID: <4BFDCEDF.1060303@canterbury.ac.nz> On 27/05/10 00:31, Brian Quinlan wrote: > You have two semantic choices here: > 1. let the interpreter exit with the future still running > 2. wait until the future finishes and then exit I'd go for (1). I don't think it's unreasonable to expect a program that wants all its tasks to finish to explicitly wait for that to happen. Also, automatically doing (2) would seem to make it difficult for a program to bail out if something unexpected happens. It would have to explicitly shut down the thread pool instead of just letting an exception propagate. -- Greg From yaniv at aknin.name Thu May 27 04:09:07 2010 From: yaniv at aknin.name (Yaniv Aknin) Date: Thu, 27 May 2010 12:09:07 +1000 Subject: [Python-Dev] Sumo In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> <4BFDB566.2060502@gmail.com> Message-ID: > > > Because scientists, financial analysts, web designers, etc all have > > different needs. > > My point is just that a web designer probably doesn't care if he's > got numpy, nor does a mathematician care if he has cherrypy > onboard. They only care when the tools they need aren't there, > which is where sumo can help. Why do we want "distributions" (whether 'sumo' or domain specific) in the first place? Obviously because we have some pains to alleviate, but I think Linux distributions, particularly Ubuntu, have been coping with similar pains for a while now with good results. I used to be rather informed on the state of Linux distributions circa the end of the 90's. Should I use this distribution or that one? Now I don't care, the answer is always 'Ubuntu, and apt-get it to fit your needs'. The pain list, as I see it: (a) Make it easy to find functionality that you'd typically be tempted to implement alone (b) Make it trivial, mind boggling easy, to add this functionality to your interpreter once you know of it (b.1) Make it easy to package a Python 'distribution' so end users don't need to muck around with (b) (c) Make it easy for software to specify its requirements in code and get them mostly automagically (d) Make it *legally* least painful to acquire and use endorsed functionality sets ... (pipe in if I missed something important) I'm wondering if expanding the efforts in Python packaging ("easy_install" and PEP376 comes to mind, only on steroids) isn't the correct solution for alleviating these pains in a much better fashion. I'm not saying it's easy, but I think that superb packaging technologies are a very well proven technology, that they're well within the community's capabilities/needs and that maximizing flexibility is a move in a better direction than creating one or two or seven 'distributions'. I hope you would intuitively grasp what I mean, even if I fail to articulate it well, if you ever used dpkg and apt-get (or equivalents), along with the ability to add sources and the simple distinctions between sources (core vs. universe vs. multiverse), along with eco systems like launchpad and PPAs, etc. To my 1999 self these features of Ubuntu seem like miracles, not so much because Debian and dpkg/apt-get weren't there back then (they were life saving back then too), but because of the whole system's huge support base and centrally-designed decentrally-implemented nature. 'apt-cache search' is usually how I find new software these days, and the same could be true for Python when I need futures or multiprocessing or whatever. I could write lots about what I think we should have in such a system, but maybe it's more befitting for python-ideas or a blog post than here; for this argument, 'mirror Ubuntu's winnage' is a good enough description. I think a good and lively packaging system beats 'distributions' any day. This leaves us only with the legal pain, which I felt harshly on my flesh following some acquisition by a corporation and it's painful and frustrating and mostly unreasonable. I suspect, though I'm a legal n00b (and proud of it), that packaging in multiple repositories like Ubuntu does could help a lot, along with taking some responsibility over the most recommended repo (candidates are the 'almost made it into stdlib' packages, maybe even the giants like Django, NumPy, TurboGears, Twisted, etc). This is not about development responsibility and surely not about taking legal liability, it's about actively helping facilitate these packages' use in hostile legal environments. While the hackers in the trenches working on trunk and py3k may not care much (and probably needn't care much), I humbly think the PSF may care more and is best equipped to help. Maybe by requiring certain licenses for 'recommended' repo inclusion, maybe by working with the likes of Google or IBM on publishing legal reviews of select Python distributions, I don't know, depends what would help a corporate lawyer make speedy approval. - Yaniv -------------- next part -------------- An HTML attachment was scrubbed... URL: From yaniv at aknin.name Thu May 27 04:15:55 2010 From: yaniv at aknin.name (Yaniv Aknin) Date: Thu, 27 May 2010 12:15:55 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFD13F6.1080300@voidspace.org.uk> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <4BFD13F6.1080300@voidspace.org.uk> Message-ID: > > Well... a middle ground certainly could exist; perhaps in the form of an > "Extended Standard Library" (community distribution), with simple > installation and management tools. > > It could be "blessed" by python-dev and maintain a high standard (only well > established best-of-breed modules with a commitment of ongoing maintenance > and more than one maintainer - something that the stdlib itself doesn't > stick to). A common license could even be chosen, potentially allowing > corporations to approve the extended package in a single pass. > I read the 'sumo' thread before I read this (and replied in depth there), but I think Michael and I mean similar things. - Yaniv -------------- next part -------------- An HTML attachment was scrubbed... URL: From debatem1 at gmail.com Thu May 27 04:29:18 2010 From: debatem1 at gmail.com (geremy condra) Date: Wed, 26 May 2010 19:29:18 -0700 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <4BFD13F6.1080300@voidspace.org.uk> Message-ID: On Wed, May 26, 2010 at 7:15 PM, Yaniv Aknin wrote: >> Well... a middle ground certainly could exist; perhaps in the form of an >> "Extended Standard Library" (community distribution), with simple >> installation and management tools. I'm not sure about the 'installation and management tools' part, but this is basically the idea I was trying to articulate: a middle ground between a 'fat' stdlib and a 'lean' one. >> It could be "blessed" by python-dev and maintain a high standard (only >> well established best-of-breed modules with a commitment of ongoing >> maintenance and more than one maintainer - something that the stdlib itself >> doesn't stick to). A common license could even be chosen, potentially >> allowing corporations to approve the extended package in a single pass. If we could do it that would be great, IMHO. > I read the 'sumo' thread before I read this?(and replied in depth there), > but I think Michael and I mean similar things. > ?- Yaniv I don't think I'm understanding you correctly in that thread then, ISTM that you're advocating better packaging systems as an alternative to this. Would you mind clarifying? Geremy Condra From greg.ewing at canterbury.ac.nz Thu May 27 04:29:28 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 27 May 2010 14:29:28 +1200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFD26A8.9050802@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4BF916EA.2090507@holdenweb.com> <20100523144754.1c6f1afc@pitrou.net> <4BFC6325.5050700@gmail.com> <20100526093820.4fbf2f87@pitrou.net> <4BFD14E1.6000305@gmail.com> <20100526145033.3b61d95b@pitrou.net> <4BFD26A8.9050802@gmail.com> Message-ID: <4BFDD908.90609@canterbury.ac.nz> On 27/05/10 01:48, Nick Coghlan wrote: > I would say it is precisely that extra configurability which separates > the executor pools in the PEP implementation from more flexible general > purpose pools. Wouldn't this be better addressed by adding the relevant options to the futures pools, rather than adding another module that provides almost exactly the same thing with different options? -- Greg From scott+python-dev at scottdial.com Thu May 27 04:48:04 2010 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 26 May 2010 22:48:04 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFDB6CB.7030104@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com> <4BFDB6CB.7030104@gmail.com> Message-ID: <4BFDDD64.9060701@scottdial.com> On 5/26/2010 8:03 PM, Nick Coghlan wrote: > On 27/05/10 02:27, Terry Reedy wrote: >> I am suggesting that if we add a package, we do it right, from the >> beginning. > > This is a reasonable point of view, but I wouldn't want to hold up PEP > 3148 over it (call it a +0 for the idea in general, but a -1 for linking > it to the acceptance of PEP 3148). That sounds backward. How can you justify accepting PEP 3148 into a "concurrent" namespace without also accepting the demand for such a namespace? What is the contingency if this TBD migration PEP is not accepted, what happens to PEP 3148? After all, there was some complaints about just calling it "futures", without putting it in a "concurrent" namespace. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From greg.ewing at canterbury.ac.nz Thu May 27 05:13:16 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 27 May 2010 15:13:16 +1200 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: <4BFDAFDA.7060404@voidspace.org.uk> References: <4BFDB10A.4080002@canterbury.ac.nz> <4BFDAFDA.7060404@voidspace.org.uk> Message-ID: <4BFDE34C.1080405@canterbury.ac.nz> On 27/05/10 11:33, Michael Foord wrote: > On 27/05/2010 00:38, Greg Ewing wrote: >> Maybe the second scope argument to exec() should >> be deprecated? >> > Sounds good to me, certainly ends the confusion over this undoubtedly > unintuitive behaviour. :-) Although it's a fair point that it can be useful to have a way of capturing definitions made by the execed code, while still making an environment of other stuff available to it. So, I'd be in favour of changing the behaviour of exec so that the local scope is made visible inside functions. However, it would be non-trivial to implement this the way things are currently structured, which is probably one of the reasons it hasn't already been done. I don't think that simply using LOAD_NAME inside the function would work, because that would only look in the function's own local namespace first, then in the global one. There is just no mechanism available at the moment for the function to know about the local namespace passed in to exec. The way that functions get access to names in enclosing local scopes is by having them passed in as cells, but that mechanism is only available for optimised local namespaces, not ones implemented as dicts. -- Greg From greg.ewing at canterbury.ac.nz Thu May 27 05:21:14 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 27 May 2010 15:21:14 +1200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> <4BFDB067.4060800@canterbury.ac.nz> Message-ID: <4BFDE52A.7040209@canterbury.ac.nz> On 27/05/10 12:04, Jesse Noller wrote: >> Namespaces are >> only a honking great idea if you actually let them do the job >> they're designed for. > > concurrent.* is the namespace, futures is the package within the > namespace - concurrent.futures is highly descriptive of the items > contained therein. I was referring to the issue of ThreadPool vs. ThreadPoolExecutor etc. By your own argument above, concurrent.futures.ThreadPool is quite descriptive enough of what it provides. It's not a problem if some other module also provides something called a ThreadPool. -- Greg From greg.ewing at canterbury.ac.nz Thu May 27 05:35:35 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 27 May 2010 15:35:35 +1200 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: Message-ID: <4BFDE887.80904@canterbury.ac.nz> On 27/05/10 12:37, Colin H wrote: > This is a major use > case for exec() - defining code from strings (e.g. enabling you to > store python code in the database), and using it at runtime. It seems > to me this must have been the point of locals in the first place. I suspect that originally it just fell out of the implementation. The function in the interpreter that executes code objects requires two namespaces as arguments, and they were both exposed via exec just in case anyone found a use for them. -- Greg From greg.ewing at canterbury.ac.nz Thu May 27 05:40:16 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 27 May 2010 15:40:16 +1200 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: <4BFDB8EE.4010307@gmail.com> Message-ID: <4BFDE9A0.8050602@canterbury.ac.nz> On 27/05/10 12:38, Guido van Rossum wrote: > the compiler normally uses syntactic clues to decide > whether to generate code using closures, in particular, the presence > of nested functions. Well, the compiler could be passed a flag indicating that the code is being compiled for an exec statement. -- Greg From greg.ewing at canterbury.ac.nz Thu May 27 05:47:26 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 27 May 2010 15:47:26 +1200 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: <4BFDB8EE.4010307@gmail.com> Message-ID: <4BFDEB4E.50100@canterbury.ac.nz> Another approach to all this might be to generalise the mechanism by which a lookup of the globals falls back to a lookup of __builtins__. If this were done recursively, then the "stuff" could be attached to the globals dict, e.g. stuff['__builtins__'] = __builtins__ g = dict(__builtins__ = stuff) exec(code, g) del g['__builtins__'] -- Greg From yaniv at aknin.name Thu May 27 06:31:03 2010 From: yaniv at aknin.name (Yaniv Aknin) Date: Thu, 27 May 2010 14:31:03 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <4BFD13F6.1080300@voidspace.org.uk> Message-ID: > > I don't think I'm understanding you correctly in that thread then, ISTM > that you're advocating better packaging systems as an alternative to > this. Would you mind clarifying? Gladly. In my mind, 'better packaging' is not "just" about something that will let you do 'pypkg install foo' and 'pypkg remove foo' along with dependancies, it's also about meta-packages (packages that have nothing but dependencies) but also about separate repositories with different natures, some endorsed by python-dev, some not. It's not so much the packaging system that does the trick - it's the eco system around it. When you install something from the 'recommended' repository (should find a better name), you know its a 'blessed' package. You know it has unittests, documentation, active development, good license, a bug tracker and the general-soundedness which I hope users have come to expect of python-dev. It is not stdlib, it's not made by python-dev, it doesn't come with python.org's default build by default, but you have a lot of important assurances with ease. You know 'it could have been in stdlib, but happens not to be'. To make this really work, the future tool I called "pypkg" will come with all future Python versions (so people can rely on it), and will arrive configured by default to use only the 'recommended' repository. From a developer's standpoint, relying on package 'foo' of the 'recommended' repository is safe: even if their end users don't happen to have 'foo', the developer is certain that 'recommended' is configured on their user's Python and is a quick (automagical?) download away. - Yaniv (p.s.: again, think Ubuntu core vs universe vs multiverse vs. non-Ubuntu repositories vs. PPAs; forgive me for waving the Ubuntu flag, from the packaging systems /and accompanying eco-systems/ I know, they did it best) -------------- next part -------------- An HTML attachment was scrubbed... URL: From morph at debian.org Thu May 27 07:40:48 2010 From: morph at debian.org (Sandro Tosi) Date: Thu, 27 May 2010 07:40:48 +0200 Subject: [Python-Dev] Sumo In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> <4BFDB566.2060502@gmail.com> Message-ID: Hello, sorry to interrupt your discussion but.. On Thu, May 27, 2010 at 04:09, Yaniv Aknin wrote: >> > Because scientists, financial analysts, web designers, etc all have >> > different needs. >> >> My point is just that a web designer probably doesn't care if he's >> got numpy, nor does a mathematician care if he has cherrypy >> onboard. They only care when the tools they need aren't there, >> which is where sumo can help. > > Why do we want "distributions" (whether 'sumo' or domain specific) in the > first place? Obviously because we have some pains to alleviate, but I think > Linux distributions, particularly Ubuntu, have been coping with similar > pains for a while now with good results. I used to be rather informed on the > state of Linux distributions circa the end of the 90's. Should I use this > distribution or that one? Now I don't care, the answer is always 'Ubuntu, > and apt-get it to fit your needs'. please note that the huge bunch of work for Python third-party modules is done in *Debian* and Ubuntu just takes those packages without advertise enough where they come from and who did the work (and not the merchandizing only). > I hope you would intuitively grasp what I mean, even if I fail to articulate > it well,?if you ever used dpkg and apt-get (or equivalents), along with the > ability to add sources and the simple distinctions between sources (core vs. > universe vs. multiverse), along with eco systems like launchpad and PPAs, > etc. To my 1999 self these features of Ubuntu seem like miracles, not so > much because Debian and dpkg/apt-get weren't there back then (they were life > saving back then too), but because of the whole system's huge support base > and centrally-designed decentrally-implemented nature. mh? Debian was not in present in 1999? Debian started in 1993 (dpkg in 1994 and apt-get in 1998) while ubuntu only in mid-2000 as a layer over Debian packages (and hiring several Debian Developers for its core positions). Also, let me remind you that transition to Python 2.6 was a huge mess for Ubuntu, where several packages were just left broken (f.e. failing unit tests were disabled to make the package build...) and only now that Debian started to migrate to 2.6 too, you can see a "flow" of packages that works for 2.6 too coming to Ubuntu. Debian can be slow, but we care about quality. End of "give credit where it's due" post :) Regards, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From brian at sweetapp.com Thu May 27 08:20:04 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Thu, 27 May 2010 16:20:04 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFDE52A.7040209@canterbury.ac.nz> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> <4BFDB067.4060800@canterbury.ac.nz> <4BFDE52A.7040209@canterbury.ac.nz> Message-ID: On May 27, 2010, at 1:21 PM, Greg Ewing wrote: > On 27/05/10 12:04, Jesse Noller wrote: > >>> Namespaces are >>> only a honking great idea if you actually let them do the job >>> they're designed for. >> >> concurrent.* is the namespace, futures is the package within the >> namespace - concurrent.futures is highly descriptive of the items >> contained therein. > > I was referring to the issue of ThreadPool vs. ThreadPoolExecutor > etc. By your own argument above, concurrent.futures.ThreadPool is > quite descriptive enough of what it provides. It's not a problem > if some other module also provides something called a ThreadPool. I think that the "Executor" suffix is a good indicator of the interface being provided. "Pool" is not because you can could have Executor implementations that don't involve pools. Cheers, Brian From regebro at gmail.com Thu May 27 09:37:00 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 27 May 2010 09:37:00 +0200 Subject: [Python-Dev] Sumo In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> <4BFDB566.2060502@gmail.com> Message-ID: One worry with an official sumo distribution is that it could become an excuse for *not* putting something in the stdlib. Otherwise it's an interesting idea. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From p.f.moore at gmail.com Thu May 27 09:46:34 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 27 May 2010 08:46:34 +0100 Subject: [Python-Dev] Sumo In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> Message-ID: On 27 May 2010 00:11, geremy condra wrote: > I'm not clear, you seem to be arguing that there's a market for many > augmented python distributions but not one. Why not just have one > that includes the best from each domain? Because that's "bloat". You later argue that a web designer wouldn't care if his "distribution" included numpy. OK, maybe, but if my needs are simply futures, cx_Oracle and pywin32, I *would* object to downloading many megabytes of other stuff just to get those three. It's a matter of degree. >> I'm genuinely struggling to see how a Sumo distribution ever comes >> into being under your proposal. There's no evidence that anyone wants >> it (otherwise it would have been created by now!!) > > Everything worth making has already been made? Not what I'm saying. But if/when it happens, something will trigger it. I see no sign of such a trigger. That's all I'm saying. >> and until it exists, it's not a plausible "place" to put modules that don't >> make it into the stdlib. > > Of course its implausible to put something somewhere that > doesn't exist... until it does. Hence my point - people are saying futures don't belong in the stdlib but they could go in a sumo distribution. The second half of that statement is (currently) content free if not self-contradictory. > I'd say rather that there are a large number of specialized tools which > aren't individually popular enough to be included in Python, but which > when taken together greatly increase its utility, and that sumo offers a > way to provide that additional utility to python's users without forcing > python core devs to shoulder the maintenance burden. I don't believe that there's evidence that aggregation (except in the context of specialist areas) does provide additional utility. (In the context of the discussion that sparked this debate, that contrasts with inclusion in the stdlib, which *does* offer additional utility - "batteries included", guaranteed and tested cross-platform functioning, a statement of best practice, etc etc). Paul. PS One thing I haven't seen made clear - in my view, they hypothetical "sumo" is a single aggregated distribution of Python modules/packages/extensions. It would NOT include core Python and the stdlib (in contrast to Enthought or ActivePython). I get the impression that other people may be thinking in terms of a full Python distribution, like those 2 cases. We probably ought to be clear which we're talking about. From floris.bruynooghe at gmail.com Thu May 27 09:53:49 2010 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Thu, 27 May 2010 08:53:49 +0100 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFDCEDF.1060303@canterbury.ac.nz> References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <20100526120651.GA12751@laurie.devork.be> <8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com> <4BFDCEDF.1060303@canterbury.ac.nz> Message-ID: <20100527075349.GA32291@laurie.devork.be> On Thu, May 27, 2010 at 01:46:07PM +1200, Greg Ewing wrote: > On 27/05/10 00:31, Brian Quinlan wrote: > > >You have two semantic choices here: > >1. let the interpreter exit with the future still running > >2. wait until the future finishes and then exit > > I'd go for (1). I don't think it's unreasonable to > expect a program that wants all its tasks to finish > to explicitly wait for that to happen. I'd got for (1) as well, it's no more then reasonable that if you want a result you wait for it. And I dislike libraries doing magic you can't see, I'd prefer if I explicitly had to shut a pool down. And yes, if you shut the interpreter down while threads are running they sometimes wake up at the wrong time to find the world around them destroyed. But that's part of programming with threads so it's not like the futures lib suddenly makes things behave differently. I'm glad I'm not alone in preferring (1) tough. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From brian at sweetapp.com Thu May 27 10:13:01 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Thu, 27 May 2010 18:13:01 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <20100527075349.GA32291@laurie.devork.be> References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <20100526120651.GA12751@laurie.devork.be> <8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com> <4BFDCEDF.1060303@canterbury.ac.nz> <20100527075349.GA32291@laurie.devork.be> Message-ID: <2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com> On 27 May 2010, at 17:53, Floris Bruynooghe wrote: > On Thu, May 27, 2010 at 01:46:07PM +1200, Greg Ewing wrote: >> On 27/05/10 00:31, Brian Quinlan wrote: >> >>> You have two semantic choices here: >>> 1. let the interpreter exit with the future still running >>> 2. wait until the future finishes and then exit >> >> I'd go for (1). I don't think it's unreasonable to >> expect a program that wants all its tasks to finish >> to explicitly wait for that to happen. > > I'd got for (1) as well, it's no more then reasonable that if you want > a result you wait for it. And I dislike libraries doing magic you > can't see, I'd prefer if I explicitly had to shut a pool down. And > yes, if you shut the interpreter down while threads are running they > sometimes wake up at the wrong time to find the world around them > destroyed. But that's part of programming with threads so it's not > like the futures lib suddenly makes things behave differently. > > I'm glad I'm not alone in preferring (1) tough. Keep in mind that this library magic is consistent with the library magic that the threading module does - unless the user sets Thread.daemon to True, the interpreter does *not* exit until the thread does. Cheers, Brian From regebro at gmail.com Thu May 27 11:01:17 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 27 May 2010 11:01:17 +0200 Subject: [Python-Dev] Sumo In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> Message-ID: OK, I had an idea here: How about that the people affected by difficulties in getting software approved got together to put together not a sumo-python, but a python-extras package? That package could include all the popular stuff, like SciPy, Numpy, twisted, distribute, buildout, virtualenv, pip, pytz, PIL, openid, docutils, simplejson, nose, genshi, and tons of others. That would be a big download. But here's the trick: You don't *have* to install them! Just bundle all of it. If licensing is a problem I guess you'd need to have permission to relicense them all to the Python license, which would be problematic. But otherwise having a team of people overseeing and bundling all this might not be that much work, and you'd avoid the bloat by not installing all of it. :-) Or would this not fool the company trolls? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 From solipsis at pitrou.net Thu May 27 12:25:37 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 27 May 2010 12:25:37 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4BF916EA.2090507@holdenweb.com> <20100523144754.1c6f1afc@pitrou.net> <4BFC6325.5050700@gmail.com> <20100526093820.4fbf2f87@pitrou.net> <4BFD14E1.6000305@gmail.com> <20100526145033.3b61d95b@pitrou.net> <4BFD26A8.9050802@gmail.com> <4BFDD908.90609@canterbury.ac.nz> Message-ID: <20100527122537.133899de@pitrou.net> On Thu, 27 May 2010 14:29:28 +1200 Greg Ewing wrote: > On 27/05/10 01:48, Nick Coghlan wrote: > > > I would say it is precisely that extra configurability which separates > > the executor pools in the PEP implementation from more flexible general > > purpose pools. > > Wouldn't this be better addressed by adding the relevant > options to the futures pools, rather than adding another > module that provides almost exactly the same thing with > different options? +1. From hawkett at gmail.com Thu May 27 13:14:05 2010 From: hawkett at gmail.com (Colin H) Date: Thu, 27 May 2010 12:14:05 +0100 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: <4BFDB8EE.4010307@gmail.com> Message-ID: I needed to make a small modification to the workaround - I wasn't able to delete from 'stuff', as the definitions in exec()'d code won't run - they're relying on that being present at runtime. In practice the overhead of doing this is quite noticeable if you run your code like this a lot, and build up a decent sized context (which I do). It will obviously depend on the usage scenario though. def define_stuff(user_code): ?context = {...} ?stuff = {} ?stuff.update(context) ?exec(user_code, stuff) return_stuff = {} return_stuff.update(stuff) del return_stuff['__builtins__'] ?for key in context: ? ?if key in return_stuff and return_stuff[key] == context[key]: ? ? ?del return_stuff[key] ?return return_stuff On Thu, May 27, 2010 at 2:13 AM, Colin H wrote: > Of course :) - I need to pay more attention. Your workaround should do > the trick. It would make sense if locals could be used for this > purpose, but the workaround doesn't add so much overhead in most > situations. ?Thanks for the help, much appreciated, > > Colin > > On Thu, May 27, 2010 at 2:05 AM, Guido van Rossum wrote: >> On Wed, May 26, 2010 at 5:53 PM, Colin H wrote: >>> ? Thanks for the possible workaround - unfortunately 'stuff' will >>> contain a whole stack of things that are not in 'context', and were >>> not defined in 'user_code' - things that python embeds - a (very >>> small) selection - >>> >>> {..., 'NameError': , 'BytesWarning': >>> , 'dict': , 'input': >>> , 'oct': , >>> 'bin': , ...} >>> >>> It makes sense why this happens of course, but upon return, the >>> globals dict is very large, and finding the stuff you defined in your >>> user_code amongst it is a very difficult task. ?Avoiding this problem >>> is the 'locals' use-case for me. ?Cheers, >> >> No, if taken literally that doesn't make sense. Those are builtins. I >> think you are mistaken that each of those (e.g. NameError) is in stuff >> -- they are in stuff['__builtins__'] which represents the built-in >> namespace. You should remove that key from stuff as well. >> >> --Guido >> >>> Colin >>> >>> On Thu, May 27, 2010 at 1:38 AM, Guido van Rossum wrote: >>>> This is not easy to fix. The best short-term work-around is probably a >>>> hack like this: >>>> >>>> def define_stuff(user_code): >>>> ?context = {...} >>>> ?stuff = {} >>>> ?stuff.update(context) >>>> ?exec(user_code, stuff) >>>> ?for key in context: >>>> ? ?if key in stuff and stuff[key] == context[key]: >>>> ? ? ?del stuff[key] >>>> ?return stuff >>>> >>>> -- >>>> --Guido van Rossum (python.org/~guido) >>>> >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > From scott+python-dev at scottdial.com Thu May 27 15:09:55 2010 From: scott+python-dev at scottdial.com (Scott Dial) Date: Thu, 27 May 2010 09:09:55 -0400 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: <4BFDB8EE.4010307@gmail.com> Message-ID: <4BFE6F23.8030504@scottdial.com> On 5/27/2010 7:14 AM, Colin H wrote: > def define_stuff(user_code): > context = {...} > stuff = {} > stuff.update(context) > > exec(user_code, stuff) > > return_stuff = {} > return_stuff.update(stuff) > > del return_stuff['__builtins__'] > for key in context: > if key in return_stuff and return_stuff[key] == context[key]: > del return_stuff[key] > > return return_stuff I'm not sure your application, but I suspect you would benefit from using an identity check instead of an __eq__ check. The equality check may be expensive (e.g., a large dictionary), and I don't think it actually is checking what you want -- if the user_code generates an __eq__-similar dictionary, wouldn't you still want that? The only reason I can see to use __eq__ is if you are trying to detect user_code modifying an object passed in, which is something that wouldn't be addressed by your original complaint about exec (as in, modifying a global data structure). Instead of: > if key in return_stuff and return_stuff[key] == context[key]: Use: > if key in return_stuff and return_stuff[key] is context[key]: -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From hawkett at gmail.com Thu May 27 15:39:28 2010 From: hawkett at gmail.com (Colin H) Date: Thu, 27 May 2010 14:39:28 +0100 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: <4BFE6F23.8030504@scottdial.com> References: <4BFDB8EE.4010307@gmail.com> <4BFE6F23.8030504@scottdial.com> Message-ID: Yep fair call - was primarily modifying Guido's example to make the point about not being able to delete from the globals returned from exec - cheers, Colin On Thu, May 27, 2010 at 2:09 PM, Scott Dial wrote: > On 5/27/2010 7:14 AM, Colin H wrote: >> def define_stuff(user_code): >> ? context = {...} >> ? stuff = {} >> ? stuff.update(context) >> >> ? exec(user_code, stuff) >> >> ? return_stuff = {} >> ? return_stuff.update(stuff) >> >> ? del return_stuff['__builtins__'] >> ? for key in context: >> ? ? if key in return_stuff and return_stuff[key] == context[key]: >> ? ? ? del return_stuff[key] >> >> ? return return_stuff > > I'm not sure your application, but I suspect you would benefit from > using an identity check instead of an __eq__ check. The equality check > may be expensive (e.g., a large dictionary), and I don't think it > actually is checking what you want -- if the user_code generates an > __eq__-similar dictionary, wouldn't you still want that? The only reason > I can see to use __eq__ is if you are trying to detect user_code > modifying an object passed in, which is something that wouldn't be > addressed by your original complaint about exec (as in, modifying a > global data structure). > > Instead of: >> ? ? if key in return_stuff and return_stuff[key] == context[key]: > > Use: >> ? ? if key in return_stuff and return_stuff[key] is context[key]: > > -- > Scott Dial > scott at scottdial.com > scodial at cs.indiana.edu > From hawkett at gmail.com Thu May 27 16:14:44 2010 From: hawkett at gmail.com (Colin H) Date: Thu, 27 May 2010 15:14:44 +0100 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: <4BFE6F23.8030504@scottdial.com> References: <4BFDB8EE.4010307@gmail.com> <4BFE6F23.8030504@scottdial.com> Message-ID: Just to put a couple of alternatives on the table that don't break existing code - not necessarily promoting them, or suggesting they would be easy to do - 1. modify exec() to take an optional third argument - 'scope_type' - if it is not supplied (but locals is), then it runs as class namespace - i.e. identical to existing behaviour. If it is supplied then it will run as whichever is specified, with function namespace being an option. The API already operates along these lines, with the second argument being optional and implying module namespace if it is not present. 2. a new API exec2() which uses function namespace, and deprecating the old exec() - assuming there is agreement that function namespace makes more sense than the class namespace, because there are real use cases, and developers would generally expect this behaviour when approaching the API for the first time. From ncoghlan at gmail.com Thu May 27 17:42:17 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 May 2010 01:42:17 +1000 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: <4BFDB8EE.4010307@gmail.com> Message-ID: <4BFE92D9.1060306@gmail.com> On 27/05/10 10:38, Guido van Rossum wrote: > On Wed, May 26, 2010 at 5:12 PM, Nick Coghlan wrote: >> Lexical scoping only works for code that is compiled as part of a single >> operation - the separation between the compilation of the individual string >> and the code defining that string means that the symbol table analysis >> needed for lexical scoping can't cross the boundary. > > Hi Nick, > > I don't think Colin was asking for such things. Yes, I realised some time after sending that message that I'd gone off on a tangent unrelated to the original question (as a result of earlier parts of the discussion I'd been pondering the scoping differences between exec with two namespaces and a class definition and ended up writing about that instead of the topic Colin originally brought up). I suspect Thomas is right that the current two namespace exec behaviour is mostly a legacy of the standard scoping before nested scopes were added. To state the problem as succinctly as I can, the basic issue is that a code object which includes a function definition that refers to top level variables will execute correctly when the same namespace is used for both locals and globals (i.e. like module level code) but will fail when these namespaces are different (i.e. like code in class definition). So long as the code being executed doesn't define any functions that refer to top level variables in the executed code the two argument form is currently perfectly usable, so deprecating it would be an overreaction. However, attaining the (sensible) behaviour Colin is requesting when such top level variable references exist would actually be somewhat tricky. Considering Guido's suggestion to treat two argument exec like a function rather than a class and generate a closure with full lexical scoping a little further, I don't believe this could be done in exec itself without breaking code that expects the current behaviour. However, something along these lines could probably be managed as a new compilation mode for compile() (e.g. compile(code_str, name, "closure")), which would then allow these code objects to be passed to exec to get the desired behaviour. Compare and contrast: >>> def f(): ... x = 1 ... def g(): ... print x ... g() ... >>> exec f.func_code in globals(), {} 1 >>> source = """\ ... x = 1 ... def g(): ... print x ... g() ... """ >>> exec source in globals(), {} Traceback (most recent call last): File "", line 1, in File "", line 4, in File "", line 3, in g NameError: global name 'x' is not defined Breaking out dis.dis on these examples is fairly enlightening, as they generate *very* different bytecode for the definition of g(). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu May 27 17:46:13 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 May 2010 01:46:13 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFDDD64.9060701@scottdial.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com> <4BFDB6CB.7030104@gmail.com> <4BFDDD64.9060701@scottdial.com> Message-ID: <4BFE93C5.6050905@gmail.com> On 27/05/10 12:48, Scott Dial wrote: > On 5/26/2010 8:03 PM, Nick Coghlan wrote: >> On 27/05/10 02:27, Terry Reedy wrote: >>> I am suggesting that if we add a package, we do it right, from the >>> beginning. >> >> This is a reasonable point of view, but I wouldn't want to hold up PEP >> 3148 over it (call it a +0 for the idea in general, but a -1 for linking >> it to the acceptance of PEP 3148). > > That sounds backward. How can you justify accepting PEP 3148 into a > "concurrent" namespace without also accepting the demand for such a > namespace? What is the contingency if this TBD migration PEP is not > accepted, what happens to PEP 3148? After all, there was some complaints > about just calling it "futures", without putting it in a "concurrent" > namespace. We can accept PEP 3148 by saying that we're happy to add the extra namespace level purely for disambiguation purposes, even if we never follow through on adding anything else to the package (although I consider such an outcome to be highly unlikely). Any future additions or renames to move things into the concurrent package would then be handled as their own PEPs. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu May 27 17:55:21 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 May 2010 01:55:21 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFDD908.90609@canterbury.ac.nz> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> <4BF916EA.2090507@holdenweb.com> <20100523144754.1c6f1afc@pitrou.net> <4BFC6325.5050700@gmail.com> <20100526093820.4fbf2f87@pitrou.net> <4BFD14E1.6000305@gmail.com> <20100526145033.3b61d95b@pitrou.net> <4BFD26A8.9050802@gmail.com> <4BFDD908.90609@canterbury.ac.nz> Message-ID: <4BFE95E9.9030504@gmail.com> On 27/05/10 12:29, Greg Ewing wrote: > On 27/05/10 01:48, Nick Coghlan wrote: > >> I would say it is precisely that extra configurability which separates >> the executor pools in the PEP implementation from more flexible general >> purpose pools. > > Wouldn't this be better addressed by adding the relevant > options to the futures pools, rather than adding another > module that provides almost exactly the same thing with > different options? It would depend on the details, but my instinct says no (instead, the futures pools would be refactored to be task specific tailorings of the general purpose pools). However, this is all very hypothetical at this point and not really relevant to the PEP review. We may never even bother creating these more general purpose threading pools - it was just an example of the kind of thing that may go alongside the futures module. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From stephen at xemacs.org Thu May 27 17:56:33 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 28 May 2010 00:56:33 +0900 Subject: [Python-Dev] Sumo In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> Message-ID: <87wrupjq32.fsf@uwakimon.sk.tsukuba.ac.jp> Paul Moore writes: > On 27 May 2010 00:11, geremy condra wrote: > > I'm not clear, you seem to be arguing that there's a market for many > > augmented python distributions but not one. Why not just have one > > that includes the best from each domain? > > Because that's "bloat". You later argue that a web designer wouldn't > care if his "distribution" included numpy. OK, maybe, but if my needs > are simply futures, cx_Oracle and pywin32, I *would* object to > downloading many megabytes of other stuff just to get those three. > It's a matter of degree. So don't do that. Go to PyPI and get just what you need. The point of the sumo is that there are people and organizations with more bandwidth/diskspace than brains (or to be more accurate, they have enough bandwidth that optimizing bandwidth is a poor use of their brains). XEmacs has used a separate distribution for packages for over a decade, and it's been a very popular feature. Since originally all packages were part of Emacs (and still are in GNU Emacs), the package distribution is a single source hierarchy (like the Python stdlib). So there are three ways of acquiring packages: check out the sources and build and install them, download individual pre-built packages, and download the sumo of all pre-built packages. The sumos are very popular. The reason is simple. A distribution of all Emacs packages ever made would still probably be under 1GB. This just isn't a lot of bandwidth or disk space when people are schlepping around DVD images, even BD images. A Python sumo would probably be much bigger (multiple GB) than XEmacs's (about 120MB, IIRC), but it would still be a negligible amount of resources *for some people/organizations*. And I have to support the "organizational constraints" argument here. Several people have told me that (strangely enough, given its rather random nature, both in what is provided and the quality) getting the sumo certified by their organization was less trouble than getting XEmacs itself certified, and neither was all that much more effort than getting a single package certified. Maintaining a sumo would be a significant effort. The XEmacs rule is that we allow someone to add a package to the distro if they promise to maintain it for a couple years, or if we think it matters enough that we'll accept the burden. We're permissive enough that there are at least 4 different MUAs in the distribution, several IRC clients, two TeX modes, etc, etc. Still, just maintaining contact with "external maintainers" (who do go AWOL regularly), and dealing with issues where somebody wants to upgrade (eg) "vcard" which is provided by "gnus" but doesn't want to deal with "gnus", etc takes time, thought, and sometimes improvement in the distribution infrastructure. It's not clear to me that Python users would benefit that much over and above the current stdlib, which provides a huge amount of functionality, of somewhat uneven but generally high quality. But I certainly think significant additional benefit would be gained, the question is is it worth the effort? It's worth discussing. > I don't believe that there's evidence that aggregation (except in the > context of specialist areas) does provide additional utility. We'll just have to agree to disagree, then. Plenty of evidence has been provided; it just doesn't happen to apply to you. Fine, but I wish you'd make the "to me" part explicit, because I know that it does apply to others, many of them, from their personal testimony, both related to XEmacs and to Python. > PS One thing I haven't seen made clear - in my view, they hypothetical > "sumo" is a single aggregated distribution of Python > modules/packages/extensions. It would NOT include core Python and the > stdlib (in contrast to Enthought or ActivePython). I get the > impression that other people may be thinking in terms of a full Python > distribution, like those 2 cases. We probably ought to be clear which > we're talking about. On the XEmacs model, it would not include core Python, but it would include much of the stdlib. The reason is that the stdlib makes commitments to compatibility that the sumo would not need to. So the sumo might include (a) recent, relatively experimental versions of stdlib packages (yes, this kind of duplication is a pain, but (some) users do want it) and (b) packages which are formally separate but duplicate functionality in the stdlib (eg, ipaddr and netaddr) -- in some cases the sumo distro would want to make adjustments so they can co-exist. I wouldn't recommend building a production system on top of a sumo in any case. But (given resources to maintain multiple Python development installations) it is a good environment for experimentation, because not only batteries but screwdrivers and duct tape are supplied. From ncoghlan at gmail.com Thu May 27 17:58:55 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 May 2010 01:58:55 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com> References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <20100526120651.GA12751@laurie.devork.be> <8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com> <4BFDCEDF.1060303@canterbury.ac.nz> <20100527075349.GA32291@laurie.devork.be> <2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com> Message-ID: <4BFE96BF.5010400@gmail.com> On 27/05/10 18:13, Brian Quinlan wrote: > > On 27 May 2010, at 17:53, Floris Bruynooghe wrote: >> I'm glad I'm not alone in preferring (1) tough. > > Keep in mind that this library magic is consistent with the library > magic that the threading module does - unless the user sets > Thread.daemon to True, the interpreter does *not* exit until the thread > does. Along those lines, an Executor.daemon option may be a good idea. That way the default behaviour is to wait until things are done (just like threading itself), but it is easy for someone to turn that behaviour off for a specific executor. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu May 27 18:05:14 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 May 2010 02:05:14 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <20100527022935.2cfd04c5@pitrou.net> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> <4BFDB067.4060800@canterbury.ac.nz> <4BFDBAA6.2010104@gmail.com> <20100527022935.2cfd04c5@pitrou.net> Message-ID: <4BFE983A.8080909@gmail.com> On 27/05/10 10:29, Antoine Pitrou wrote: > On Thu, 27 May 2010 10:19:50 +1000 > Nick Coghlan wrote: >> >> futures.ThreadPoolExecutor would likely be refactored to inherit from >> the mooted pool.ThreadPool. > > There still doesn't seem to be reason to have two different thread pool > APIs, though. Shouldn't there be one obvious way to do it? Executors and thread pools are not the same thing. I might create a thread pool for *anything*. An executor will always have a specific execution model associated with it (whether it be called futures, as in this case, or runnables or something else). This confusion is making me think that dropping the "Pool" from the names might even be beneficial (since, to my mind, it currently emphasises a largely irrelevant implementation detail). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu May 27 18:08:02 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 May 2010 02:08:02 +1000 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: <4BFDE34C.1080405@canterbury.ac.nz> References: <4BFDB10A.4080002@canterbury.ac.nz> <4BFDAFDA.7060404@voidspace.org.uk> <4BFDE34C.1080405@canterbury.ac.nz> Message-ID: <4BFE98E2.8020302@gmail.com> On 27/05/10 13:13, Greg Ewing wrote: > The way that functions get access to names in enclosing > local scopes is by having them passed in as cells, but that > mechanism is only available for optimised local namespaces, > not ones implemented as dicts. I believe exec already includes the tapdancing needed to make that work. As Guido pointed out, it's the ability to generate closures directly from a source string that is currently missing. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From stephen at xemacs.org Thu May 27 18:12:35 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 28 May 2010 01:12:35 +0900 Subject: [Python-Dev] Sumo In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> <4BFDB566.2060502@gmail.com> Message-ID: <87vda9jpcc.fsf@uwakimon.sk.tsukuba.ac.jp> Lennart Regebro writes: > One worry with an official sumo distribution is that it could become > an excuse for *not* putting something in the stdlib. > Otherwise it's an interesting idea. On the contrary, that is the meat of why it's an interesting idea. I really don't think the proponents of ipaddr and futures (to take two recent PEPs) would have been willing to stop with a hypothetical sumo. Both of those packages were designed with general use in mind. Substantial effort was put into making them TOOWTDI-able. Partly that's pride ("my stuff is good enough for the stdlib"), and partly there's a genuine need for it to be there (for your customers or just to pay back the community). Of course there was a lot of criticism of both that they don't really come up to that standard, but even opponents would credit the proponents for good intentions and making the necessary effort, I think. And it's the stdlib that (in a certain sense) puts the "OO" in "TOOWTDI". On the other hand, some ideas deserve widespread exposure, but they need real experience because the appropriate requirements and specs are unclear. It would be premature to put in the effort to make them TOOWTDI. However, to get the momentum to become BCP, and thus an obvious candidate for stdlib inclusion, it's helpful to be *already* available on *typical* installations. PyPI is great, but it's not quite there; it's not as discoverable and accessible as simply putting "import stuff" based on some snippet you found on the web. And the stdlib itself can't be the means, it's the end. At present, such ideas face the alternative "stdlib or die". The sumo would give them a place to be. From solipsis at pitrou.net Thu May 27 18:16:38 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 27 May 2010 18:16:38 +0200 Subject: [Python-Dev] PEP 3148 ready for pronouncement References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> <4BFDB067.4060800@canterbury.ac.nz> <4BFDBAA6.2010104@gmail.com> <20100527022935.2cfd04c5@pitrou.net> <4BFE983A.8080909@gmail.com> Message-ID: <20100527181638.2164a9e1@pitrou.net> On Fri, 28 May 2010 02:05:14 +1000 Nick Coghlan wrote: > > Executors and thread pools are not the same thing. > > I might create a thread pool for *anything*. An executor will always > have a specific execution model associated with it (whether it be called > futures, as in this case, or runnables or something else). I'm sorry, but what is the specific execution model you are talking about, and how is it different from what you usually do with a thread pool? Why would you do something other with a thread pool than running tasks and (optionally) collecting their results? Thanks Antoine. From fuzzyman at voidspace.org.uk Thu May 27 18:22:10 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 27 May 2010 17:22:10 +0100 Subject: [Python-Dev] Sumo In-Reply-To: <87wrupjq32.fsf@uwakimon.sk.tsukuba.ac.jp> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> <87wrupjq32.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4BFE9C32.8050209@voidspace.org.uk> On 27/05/2010 16:56, Stephen J. Turnbull wrote: > Paul Moore writes: > > On 27 May 2010 00:11, geremy condra wrote: > > > I'm not clear, you seem to be arguing that there's a market for many > > > augmented python distributions but not one. Why not just have one > > > that includes the best from each domain? > > > > Because that's "bloat". You later argue that a web designer wouldn't > > care if his "distribution" included numpy. OK, maybe, but if my needs > > are simply futures, cx_Oracle and pywin32, I *would* object to > > downloading many megabytes of other stuff just to get those three. > > It's a matter of degree. > > So don't do that. Go to PyPI and get just what you need. > > The point of the sumo is that there are people and organizations with > more bandwidth/diskspace than brains (or to be more accurate, they > have enough bandwidth that optimizing bandwidth is a poor use of their > brains). > To my mind one of the most important benefits of a "sumo" style distribution is not just that it easily provides a whole bunch of useful modules - but that it *highlights* which modules are the community blessed "best of breed". At the moment if a new user wants to work out how to achieve a particular task (work with images for example) they have to google around and try and work out what the right module to use is. For some problem domains there are a host of modules on PyPI many of which are unmaintained, immature or simply rubbish. A standardised solution makes choosing solutions for common problems *dramatically* easier, and may save people much heartache and frustration. For that to work though it needs to be well curated and genuinely have the substantial backing of the Python development community. All the best, Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From stephen at xemacs.org Thu May 27 18:31:56 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 28 May 2010 01:31:56 +0900 Subject: [Python-Dev] Sumo In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> Message-ID: <87typtjog3.fsf@uwakimon.sk.tsukuba.ac.jp> Lennart Regebro writes: > If licensing is a problem I guess you'd need to have permission to > relicense them all to the Python license, Licensing compatibility is only a problem for copyleft, but most copyleft licenses have "mere aggregation is not derivation" clauses. Corporate concern about *knowing* what the license is, is a problem. The XEmacs experience I've referred to elsewhere doesn't apply because all our stuff is GPL, and therefore all our stuff has to be GPL. :-( It's not obvious to me what the resolution is, although lots of distributions now have some way to find out what licenses are. GCC (and soon GNU Emacs) even have a way to check GPL-compatibility at runtime (inspired by the Linux kernel feature, maybe?) Perhaps the sumo infrastructure could provide a license-ok-or-fatal feature. Eg, the application would do something like sumo_ok_licenses = ['gplv2','bsd','microsoft_eula'] and the sumo version of the package's __init.py__ would do sumo_check_my_license('artistic') and raise LicenseError if it's not in sumo_ok_licenses. In theory it might be able to do more complex stuff like keep track of declared licenses and barf if they're incompatible. This scheme probably doesn't save lawyer time. The lawyers would still have to go over the actual packages to make sure the licenses are what they say etc. before going into distribution. Its selling point is that the developers would be warned of problems that need corporate legal's attention early in 90% of the cases, thus not wasting developer time on using packages that were non-starters because of license issues. > which would be problematic. But otherwise having a team of people > overseeing and bundling all this might not be that much work, and > you'd avoid the bloat by not installing all of it. :-) As I've argued elsewhere, bloat is good, for some purposes. > Or would this not fool the company trolls? It will satisfy some, and not others, in my experience, described elsewhere. From ncoghlan at gmail.com Thu May 27 19:08:40 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 May 2010 03:08:40 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <20100527181638.2164a9e1@pitrou.net> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> <4BFDB067.4060800@canterbury.ac.nz> <4BFDBAA6.2010104@gmail.com> <20100527022935.2cfd04c5@pitrou.net> <4BFE983A.8080909@gmail.com> <20100527181638.2164a9e1@pitrou.net> Message-ID: <4BFEA718.6070007@gmail.com> On 28/05/10 02:16, Antoine Pitrou wrote: > On Fri, 28 May 2010 02:05:14 +1000 > Nick Coghlan wrote: >> >> Executors and thread pools are not the same thing. >> >> I might create a thread pool for *anything*. An executor will always >> have a specific execution model associated with it (whether it be called >> futures, as in this case, or runnables or something else). > > I'm sorry, but what is the specific execution model you are talking > about, and how is it different from what you usually do with a thread > pool? Why would you do something other with a thread pool than running > tasks and (optionally) collecting their results? Both the execution and communications models may be different. The components may be long-lived state machines, they may be active objects, they may communicate by message passing or by manipulating shared state, who knows. Executors are designed around a model of "go do this and let me know when you're done". A general purpose pool needs to support other execution models, and hence will look different (and is harder to design and write, since it needs to be more flexible). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From stephen at xemacs.org Thu May 27 19:52:34 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 28 May 2010 02:52:34 +0900 Subject: [Python-Dev] Sumo In-Reply-To: <4BFE9C32.8050209@voidspace.org.uk> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> <87wrupjq32.fsf@uwakimon.sk.tsukuba.ac.jp> <4BFE9C32.8050209@voidspace.org.uk> Message-ID: <87r5kxjkpp.fsf@uwakimon.sk.tsukuba.ac.jp> Michael Foord writes: > To my mind one of the most important benefits of a "sumo" style > distribution is not just that it easily provides a whole bunch of useful > modules - but that it *highlights* which modules are the community > blessed "best of breed". That has several problems. (1) There is a lot of overlap with the mission of the stdlib, and I think confusion over roles would be quite costly. (2) As the stdlib demonstrates, picking winners is expensive. I greatly doubt that running *two* such processes is worthwhile. (3) Very often there is no best of breed. From p.f.moore at gmail.com Thu May 27 21:44:30 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 27 May 2010 20:44:30 +0100 Subject: [Python-Dev] Sumo In-Reply-To: <87wrupjq32.fsf@uwakimon.sk.tsukuba.ac.jp> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <20100526113915.5bf46665@pitrou.net> <201005262042.12825.steve@pearwood.info> <20100526125614.0a7ae470@pitrou.net> <1274877963.3162.20.camel@localhost.localdomain> <87wrupjq32.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 27 May 2010 16:56, Stephen J. Turnbull wrote: > We'll just have to agree to disagree, then. ?Plenty of evidence has > been provided; it just doesn't happen to apply to you. ?Fine, but I > wish you'd make the "to me" part explicit, because I know that it does > apply to others, many of them, from their personal testimony, both > related to XEmacs and to Python. Sorry, you're right. There's a very strong "to me" in all of this, but I more or less assumed it was obvious, as I was originally responding to comments implying that a sumo distribution was a solution to a problem I stated that I have. In trying to trim things, and keep things concise, I completely lost the context. My apologies. > I wouldn't recommend building a production system on top of a sumo in > any case. ?But (given resources to maintain multiple Python development > installations) it is a good environment for experimentation, because > not only batteries but screwdrivers and duct tape are supplied. That's an interesting perspective that I hadn't seen mentioned before. For experimentation, I'd *love* a sumo distribution as you describe. But I thought this whole discussion focussed around building production systems. For that, the stdlib's quality guarantees are a major benefit, and the costs of locating and validating appropriately high-quality external packages are (sometimes prohibitively) high. But I think I'm getting to the point where I'm adding more confusion than information, so I'll bow out of this discussion at this point. Paul. From rnk at mit.edu Thu May 27 22:05:26 2010 From: rnk at mit.edu (Reid Kleckner) Date: Thu, 27 May 2010 16:05:26 -0400 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: <4BFE92D9.1060306@gmail.com> References: <4BFDB8EE.4010307@gmail.com> <4BFE92D9.1060306@gmail.com> Message-ID: On Thu, May 27, 2010 at 11:42 AM, Nick Coghlan wrote: > However, attaining the (sensible) behaviour Colin is requesting when such > top level variable references exist would actually be somewhat tricky. > Considering Guido's suggestion to treat two argument exec like a function > rather than a class and generate a closure with full lexical scoping a > little further, I don't believe this could be done in exec itself without > breaking code that expects the current behaviour. Just to give a concrete example, here is code that would break if exec were to execute code in a function scope instead of a class scope: exec """ def len(xs): return -1 def foo(): return len([]) print foo() """ in globals(), {} Currently, the call to 'len' inside 'foo' skips the outer scope (because it's a class scope) and goes straight to globals and builtins. If it were switched to a local scope, a cell would be created for the broken definition of 'len', and the call would resolve to it. Honestly, to me, the fact that the above code ever worked (ie prints "0", not "-1") seems like a bug, so I wouldn't worry about backwards compatibility. Reid From greg.ewing at canterbury.ac.nz Fri May 28 01:18:47 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 28 May 2010 11:18:47 +1200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> <4BFDB067.4060800@canterbury.ac.nz> <4BFDE52A.7040209@canterbury.ac.nz> Message-ID: <4BFEFDD7.70001@canterbury.ac.nz> Brian Quinlan wrote: > I think that the "Executor" suffix is a good indicator of the interface > being provided. It's not usually considered necessary for the name of a type to indicate its interface. We don't have 'listsequence' and 'dictmapping' for example. I think what bothers me most about these names is their longwindedness. Two parts to a name is okay, but three or more starts to sound pedantic. And for me, "Pool" is a more important piece of information than "Executor". The fact that it manages a pool is the main reason I'd use such a module rather than just spawning a thread myself for each task. -- Greg From hawkett at gmail.com Fri May 28 01:32:18 2010 From: hawkett at gmail.com (Colin H) Date: Fri, 28 May 2010 00:32:18 +0100 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: <4BFE92D9.1060306@gmail.com> References: <4BFDB8EE.4010307@gmail.com> <4BFE92D9.1060306@gmail.com> Message-ID: This option sounds very promising - seems right to do it at the compile stage - i.e. compile(code_str, name, "closure") as you have suggested. If there were any argument against, it would be that the most obvious behaviour (function namespace) is the hardest to induce, but the value in knowing you're not breaking anything is pretty high. Cheers, Colin On Thu, May 27, 2010 at 4:42 PM, Nick Coghlan wrote: > On 27/05/10 10:38, Guido van Rossum wrote: >> >> On Wed, May 26, 2010 at 5:12 PM, Nick Coghlan ?wrote: >>> >>> Lexical scoping only works for code that is compiled as part of a single >>> operation - the separation between the compilation of the individual >>> string >>> and the code defining that string means that the symbol table analysis >>> needed for lexical scoping can't cross the boundary. >> >> Hi Nick, >> >> I don't think Colin was asking for such things. > > Yes, I realised some time after sending that message that I'd gone off on a > tangent unrelated to the original question (as a result of earlier parts of > the discussion I'd been pondering the scoping differences between exec with > two namespaces and a class definition and ended up writing about that > instead of the topic Colin originally brought up). > > I suspect Thomas is right that the current two namespace exec behaviour is > mostly a legacy of the standard scoping before nested scopes were added. > > To state the problem as succinctly as I can, the basic issue is that a code > object which includes a function definition that refers to top level > variables will execute correctly when the same namespace is used for both > locals and globals (i.e. like module level code) but will fail when these > namespaces are different (i.e. like code in class definition). > > So long as the code being executed doesn't define any functions that refer > to top level variables in the executed code the two argument form is > currently perfectly usable, so deprecating it would be an overreaction. > > However, attaining the (sensible) behaviour Colin is requesting when such > top level variable references exist would actually be somewhat tricky. > Considering Guido's suggestion to treat two argument exec like a function > rather than a class and generate a closure with full lexical scoping a > little further, I don't believe this could be done in exec itself without > breaking code that expects the current behaviour. However, something along > these lines could probably be managed as a new compilation mode for > compile() (e.g. compile(code_str, name, "closure")), which would then allow > these code objects to be passed to exec to get the desired behaviour. > > Compare and contrast: > >>>> def f(): > ... ? x = 1 > ... ? def g(): > ... ? ? print x > ... ? g() > ... >>>> exec f.func_code in globals(), {} > 1 > >>>> source = """\ > ... x = 1 > ... def g(): > ... ? print x > ... g() > ... """ >>>> exec source in globals(), {} > Traceback (most recent call last): > ?File "", line 1, in > ?File "", line 4, in > ?File "", line 3, in g > NameError: global name 'x' is not defined > > Breaking out dis.dis on these examples is fairly enlightening, as they > generate *very* different bytecode for the definition of g(). > > Cheers, > Nick. > > -- > Nick Coghlan ? | ? ncoghlan at gmail.com ? | ? Brisbane, Australia > --------------------------------------------------------------- > From hawkett at gmail.com Fri May 28 01:33:25 2010 From: hawkett at gmail.com (Colin H) Date: Fri, 28 May 2010 00:33:25 +0100 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: <4BFDB8EE.4010307@gmail.com> <4BFE92D9.1060306@gmail.com> Message-ID: By hardest to induce I mean the default compile exec(code_str, {}, {}) would still be class namespace, but it's pretty insignificant. On Fri, May 28, 2010 at 12:32 AM, Colin H wrote: > This option sounds very promising - seems right to do it at the > compile stage - i.e. compile(code_str, name, "closure") as you have > suggested. ?If there were any argument against, it would be that the > most obvious behaviour (function namespace) is the hardest to induce, > but the value in knowing you're not breaking anything is pretty high. > > Cheers, > Colin > > On Thu, May 27, 2010 at 4:42 PM, Nick Coghlan wrote: >> On 27/05/10 10:38, Guido van Rossum wrote: >>> >>> On Wed, May 26, 2010 at 5:12 PM, Nick Coghlan ?wrote: >>>> >>>> Lexical scoping only works for code that is compiled as part of a single >>>> operation - the separation between the compilation of the individual >>>> string >>>> and the code defining that string means that the symbol table analysis >>>> needed for lexical scoping can't cross the boundary. >>> >>> Hi Nick, >>> >>> I don't think Colin was asking for such things. >> >> Yes, I realised some time after sending that message that I'd gone off on a >> tangent unrelated to the original question (as a result of earlier parts of >> the discussion I'd been pondering the scoping differences between exec with >> two namespaces and a class definition and ended up writing about that >> instead of the topic Colin originally brought up). >> >> I suspect Thomas is right that the current two namespace exec behaviour is >> mostly a legacy of the standard scoping before nested scopes were added. >> >> To state the problem as succinctly as I can, the basic issue is that a code >> object which includes a function definition that refers to top level >> variables will execute correctly when the same namespace is used for both >> locals and globals (i.e. like module level code) but will fail when these >> namespaces are different (i.e. like code in class definition). >> >> So long as the code being executed doesn't define any functions that refer >> to top level variables in the executed code the two argument form is >> currently perfectly usable, so deprecating it would be an overreaction. >> >> However, attaining the (sensible) behaviour Colin is requesting when such >> top level variable references exist would actually be somewhat tricky. >> Considering Guido's suggestion to treat two argument exec like a function >> rather than a class and generate a closure with full lexical scoping a >> little further, I don't believe this could be done in exec itself without >> breaking code that expects the current behaviour. However, something along >> these lines could probably be managed as a new compilation mode for >> compile() (e.g. compile(code_str, name, "closure")), which would then allow >> these code objects to be passed to exec to get the desired behaviour. >> >> Compare and contrast: >> >>>>> def f(): >> ... ? x = 1 >> ... ? def g(): >> ... ? ? print x >> ... ? g() >> ... >>>>> exec f.func_code in globals(), {} >> 1 >> >>>>> source = """\ >> ... x = 1 >> ... def g(): >> ... ? print x >> ... g() >> ... """ >>>>> exec source in globals(), {} >> Traceback (most recent call last): >> ?File "", line 1, in >> ?File "", line 4, in >> ?File "", line 3, in g >> NameError: global name 'x' is not defined >> >> Breaking out dis.dis on these examples is fairly enlightening, as they >> generate *very* different bytecode for the definition of g(). >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan ? | ? ncoghlan at gmail.com ? | ? Brisbane, Australia >> --------------------------------------------------------------- >> > From brian at sweetapp.com Fri May 28 01:37:19 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Fri, 28 May 2010 09:37:19 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFEFDD7.70001@canterbury.ac.nz> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> <4BFDB067.4060800@canterbury.ac.nz> <4BFDE52A.7040209@canterbury.ac.nz> <4BFEFDD7.70001@canterbury.ac.nz> Message-ID: On 28 May 2010, at 09:18, Greg Ewing wrote: > Brian Quinlan wrote: > >> I think that the "Executor" suffix is a good indicator of the >> interface being provided. > > It's not usually considered necessary for the name of a > type to indicate its interface. We don't have 'listsequence' > and 'dictmapping' for example. > > I think what bothers me most about these names is their > longwindedness. Two parts to a name is okay, but three or > more starts to sound pedantic. And for me, "Pool" is a > more important piece of information than "Executor". > The fact that it manages a pool is the main reason I'd > use such a module rather than just spawning a thread myself > for each task. Actually, an executor implementation that created a new thread per task would still be useful - it would save you the hassle of developing a mechanism to wait for the thread to finish and to collect the results. We actually have such an implementation at Google and it is quite popular. Cheers, Brian From greg.ewing at canterbury.ac.nz Fri May 28 01:52:52 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 28 May 2010 11:52:52 +1200 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFE93C5.6050905@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com> <4BFDB6CB.7030104@gmail.com> <4BFDDD64.9060701@scottdial.com> <4BFE93C5.6050905@gmail.com> Message-ID: <4BFF05D4.4000202@canterbury.ac.nz> Nick Coghlan wrote: > We can accept PEP 3148 by saying that we're happy to add the extra > namespace level purely for disambiguation purposes, If that's the only rationale for the namespace, it makes it sound like a kludge to work around a poor choice of name. -- Greg From rnk at mit.edu Fri May 28 03:57:48 2010 From: rnk at mit.edu (Reid Kleckner) Date: Thu, 27 May 2010 21:57:48 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com> References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <20100526120651.GA12751@laurie.devork.be> <8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com> <4BFDCEDF.1060303@canterbury.ac.nz> <20100527075349.GA32291@laurie.devork.be> <2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com> Message-ID: On Thu, May 27, 2010 at 4:13 AM, Brian Quinlan wrote: > Keep in mind that this library magic is consistent with the library magic > that the threading module does - unless the user sets Thread.daemon to True, > the interpreter does *not* exit until the thread does. Is there a compelling to make the threads daemon threads? If not, perhaps they can just be normal threads, and you can rely on the threading module to wait for them to finish. Unrelatedly, I feel like this behavior of waiting for the thread to terminate usually manifests as deadlocks when the main thread throws an uncaught exception. The application then no longer responds properly to interrupts, since it's stuck waiting on a semaphore. I guess it's better than the alternative of random crashes when daemon threads wake up during interpreter shutdown, though. Reid From brian at sweetapp.com Fri May 28 05:06:02 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Fri, 28 May 2010 13:06:02 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <20100526120651.GA12751@laurie.devork.be> <8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com> <4BFDCEDF.1060303@canterbury.ac.nz> <20100527075349.GA32291@laurie.devork.be> <2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com> Message-ID: <31F8D1F3-13C9-4211-B808-388DB2CC705F@sweetapp.com> On May 28, 2010, at 11:57 AM, Reid Kleckner wrote: > On Thu, May 27, 2010 at 4:13 AM, Brian Quinlan > wrote: >> Keep in mind that this library magic is consistent with the library >> magic >> that the threading module does - unless the user sets Thread.daemon >> to True, >> the interpreter does *not* exit until the thread does. > > Is there a compelling to make the threads daemon threads? If not, > perhaps they can just be normal threads, and you can rely on the > threading module to wait for them to finish. Did you read my explanation of the reasoning behind my approach? Cheers, Brian > Unrelatedly, I feel like this behavior of waiting for the thread to > terminate usually manifests as deadlocks when the main thread throws > an uncaught exception. The application then no longer responds > properly to interrupts, since it's stuck waiting on a semaphore. I > guess it's better than the alternative of random crashes when daemon > threads wake up during interpreter shutdown, though. > > Reid From jyasskin at gmail.com Fri May 28 05:16:04 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Thu, 27 May 2010 20:16:04 -0700 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <31F8D1F3-13C9-4211-B808-388DB2CC705F@sweetapp.com> References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <20100526120651.GA12751@laurie.devork.be> <8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com> <4BFDCEDF.1060303@canterbury.ac.nz> <20100527075349.GA32291@laurie.devork.be> <2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com> <31F8D1F3-13C9-4211-B808-388DB2CC705F@sweetapp.com> Message-ID: On Thu, May 27, 2010 at 8:06 PM, Brian Quinlan wrote: > > On May 28, 2010, at 11:57 AM, Reid Kleckner wrote: > >> On Thu, May 27, 2010 at 4:13 AM, Brian Quinlan wrote: >>> >>> Keep in mind that this library magic is consistent with the library magic >>> that the threading module does - unless the user sets Thread.daemon to >>> True, >>> the interpreter does *not* exit until the thread does. >> >> Is there a compelling to make the threads daemon threads? ?If not, >> perhaps they can just be normal threads, and you can rely on the >> threading module to wait for them to finish. > > Did you read my explanation of the reasoning behind my approach? You should try to link to explanations. These have been long threads, and it's often hard to find the previous message where a subject was addressed. From scott+python-dev at scottdial.com Fri May 28 05:39:36 2010 From: scott+python-dev at scottdial.com (Scott Dial) Date: Thu, 27 May 2010 23:39:36 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com> References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <20100526120651.GA12751@laurie.devork.be> <8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com> <4BFDCEDF.1060303@canterbury.ac.nz> <20100527075349.GA32291@laurie.devork.be> <2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com> Message-ID: <4BFF3AF8.9040504@scottdial.com> On 5/27/2010 4:13 AM, Brian Quinlan wrote: > On 27 May 2010, at 17:53, Floris Bruynooghe wrote: >> On Thu, May 27, 2010 at 01:46:07PM +1200, Greg Ewing wrote: >>> On 27/05/10 00:31, Brian Quinlan wrote: >>>> You have two semantic choices here: >>>> 1. let the interpreter exit with the future still running >>>> 2. wait until the future finishes and then exit >>> >>> I'd go for (1). I don't think it's unreasonable to >>> expect a program that wants all its tasks to finish >>> to explicitly wait for that to happen. >> >> I'd got for (1) as well, it's no more then reasonable that if you want >> a result you wait for it. And I dislike libraries doing magic you >> can't see, I'd prefer if I explicitly had to shut a pool down. >> >> I'm glad I'm not alone in preferring (1) tough. > > Keep in mind that this library magic is consistent with the library > magic that the threading module does - unless the user sets > Thread.daemon to True, the interpreter does *not* exit until the thread > does. Given your rationale, I don't understand from the PEP: > shutdown(wait=True) > > Signal the executor that it should free any resources that it is > using when the currently pending futures are done executing. Calls to > Executor.submit and Executor.map and made after shutdown will raise > RuntimeError. > > If wait is True then the executor will not return until all the > pending futures are done executing and the resources associated with > the executor have been freed. Can you tell me what is the expected execution time of the following: >>> executor = ThreadPoolExecutor(max_workers=1) >>> executor.submit(lambda: time.sleep(1000)) >>> executor.shutdown(wait=False) >>> sys.exit(0) I believe it's 1000 seconds, which seems to defy my request of shutdown(wait=False) because "secretly" the Python exit is going to wait anyways. ISTM, it is much easier to get behavior #2 if you have behavior #1, and it would also seem rather trivial to make ThreadPoolExecutor take an optional argument specifying which behavior you want. Your reference implementation does not actually implement the specification given in the PEP, so it's quite impossible to check this myself. There is no wait=True option for shutdown() in the reference implementation, so I can only guess what that implementation might look like. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From ncoghlan at gmail.com Fri May 28 06:35:19 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 May 2010 14:35:19 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFF05D4.4000202@canterbury.ac.nz> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com> <4BFDB6CB.7030104@gmail.com> <4BFDDD64.9060701@scottdial.com> <4BFE93C5.6050905@gmail.com> <4BFF05D4.4000202@canterbury.ac.nz> Message-ID: <4BFF4807.1050903@gmail.com> On 28/05/10 09:52, Greg Ewing wrote: > Nick Coghlan wrote: > >> We can accept PEP 3148 by saying that we're happy to add the extra >> namespace level purely for disambiguation purposes, > > If that's the only rationale for the namespace, it makes it > sound like a kludge to work around a poor choice of name. It's the right name though (it really is a futures implementation - I don't know what else you would even consider calling it). The problem is that the same word is used to mean different things in other programming domains (most obviously finance). Resolving that kind of ambiguity is an *excellent* use of a package namespace - you remove the ambiguity without imposing any significant long term cognitive overhead. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From brian at sweetapp.com Fri May 28 08:52:39 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Fri, 28 May 2010 16:52:39 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4BFF3AF8.9040504@scottdial.com> References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <20100526120651.GA12751@laurie.devork.be> <8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com> <4BFDCEDF.1060303@canterbury.ac.nz> <20100527075349.GA32291@laurie.devork.be> <2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com> <4BFF3AF8.9040504@scottdial.com> Message-ID: <2861971A-8E80-490C-8C50-7FC9C67F2B26@sweetapp.com> On May 28, 2010, at 1:39 PM, Scott Dial wrote: > On 5/27/2010 4:13 AM, Brian Quinlan wrote: >> On 27 May 2010, at 17:53, Floris Bruynooghe wrote: >>> On Thu, May 27, 2010 at 01:46:07PM +1200, Greg Ewing wrote: >>>> On 27/05/10 00:31, Brian Quinlan wrote: >>>>> You have two semantic choices here: >>>>> 1. let the interpreter exit with the future still running >>>>> 2. wait until the future finishes and then exit >>>> >>>> I'd go for (1). I don't think it's unreasonable to >>>> expect a program that wants all its tasks to finish >>>> to explicitly wait for that to happen. >>> >>> I'd got for (1) as well, it's no more then reasonable that if you >>> want >>> a result you wait for it. And I dislike libraries doing magic you >>> can't see, I'd prefer if I explicitly had to shut a pool down. >>> >>> I'm glad I'm not alone in preferring (1) tough. >> >> Keep in mind that this library magic is consistent with the library >> magic that the threading module does - unless the user sets >> Thread.daemon to True, the interpreter does *not* exit until the >> thread >> does. > > Given your rationale, I don't understand from the PEP: > >> shutdown(wait=True) >> >> Signal the executor that it should free any resources that it is >> using when the currently pending futures are done executing. Calls to >> Executor.submit and Executor.map and made after shutdown will raise >> RuntimeError. >> >> If wait is True then the executor will not return until all the >> pending futures are done executing and the resources associated with >> the executor have been freed. > > Can you tell me what is the expected execution time of the following: > >>>> executor = ThreadPoolExecutor(max_workers=1) >>>> executor.submit(lambda: time.sleep(1000)) >>>> executor.shutdown(wait=False) >>>> sys.exit(0) > > I believe it's 1000 seconds, which seems to defy my request of > shutdown(wait=False) because "secretly" the Python exit is going to > wait > anyways. It would take 1000 seconds. "...then the executor will not return..." should read "...then the method will not return...". > ISTM, it is much easier to get behavior #2 if you have behavior > #1, and it would also seem rather trivial to make ThreadPoolExecutor > take an optional argument specifying which behavior you want. Adding a daemon option would be reasonable. If you don't shutdown your executors you are pretty much guaranteed to get random traceback output on exit through. > Your reference implementation does not actually implement the > specification given in the PEP, so it's quite impossible to check this > myself. There is no wait=True option for shutdown() in the reference > implementation, so I can only guess what that implementation might > look > like. Look at around line 129 in: http://code.google.com/p/pythonfutures/source/browse/branches/feedback/python3/futures/thread.py Cheers, Brian From solipsis at pitrou.net Fri May 28 12:58:02 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 28 May 2010 12:58:02 +0200 Subject: [Python-Dev] Locks and signals References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com> <20100522135946.8F5451F960D@kimball.webabinitio.net> <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com> <20100526120651.GA12751@laurie.devork.be> <8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com> <4BFDCEDF.1060303@canterbury.ac.nz> <20100527075349.GA32291@laurie.devork.be> <2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com> Message-ID: <20100528125802.75f388b4@pitrou.net> On Thu, 27 May 2010 21:57:48 -0400 Reid Kleckner wrote: > > Unrelatedly, I feel like this behavior of waiting for the thread to > terminate usually manifests as deadlocks when the main thread throws > an uncaught exception. The application then no longer responds > properly to interrupts, since it's stuck waiting on a semaphore. I think the internal low-level lock implementation should be fixed so that it runs PyErr_CheckSignals() and is able to signal an error on function return (rather than the current binary "lock succeeded" / "lock timed out" status). Actually, it would be nice if you could open a bug entry for that :) Regards Antoine. From status at bugs.python.org Fri May 28 18:07:42 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 28 May 2010 18:07:42 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20100528160742.3EEED7813C@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-05-21 - 2010-05-28) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2705 open (+40) / 17957 closed (+21) / 20662 total (+61) Open issues with patches: 1095 Average duration of open issues: 720 days. Median duration of open issues: 500 days. Open Issues Breakdown open 2683 (+40) languishing 12 ( +0) pending 9 ( +0) Issues Created Or Reopened (62) _______________________________ httplib fails with HEAD requests to pages with "transfer-encod 2010-05-26 http://bugs.python.org/issue6312 reopened orsenthil patch The external link to a "Hash Collision FAQ" points to some com 2010-05-21 CLOSED http://bugs.python.org/issue8783 created stutzbach tarfile/Windows: Don't use mbcs as the default encoding 2010-05-22 http://bugs.python.org/issue8784 created haypo patch findall() and finditer() docs misleading 2010-05-22 CLOSED http://bugs.python.org/issue8785 created hagen Add support for IEEE 754 contexts to decimal module. 2010-05-22 http://bugs.python.org/issue8786 created mark.dickinson warnings inside PyRun_SimpleString() display argv[1] 2010-05-22 http://bugs.python.org/issue8787 created Sebastian patch urllib.urlencode documentation unclear on doseq 2010-05-22 http://bugs.python.org/issue8788 created VeXocide spam 2010-05-22 CLOSED http://bugs.python.org/issue8789 created renben spam 2010-05-22 CLOSED http://bugs.python.org/issue8790 created renben spam 2010-05-22 CLOSED http://bugs.python.org/issue8791 created renben xmlrpclib compatibility issues with Apache XML-RPC library 2010-05-22 http://bugs.python.org/issue8792 created bra IDLE crashes on opening invalid file 2010-05-23 http://bugs.python.org/issue8793 created royf spam 2010-05-23 CLOSED http://bugs.python.org/issue8794 created renben Error sending utf8 strings over syslog udp protocol 2010-05-23 CLOSED http://bugs.python.org/issue8795 created skliarie Deprecate codecs.open() 2010-05-23 http://bugs.python.org/issue8796 created haypo urllib2 basicauth broken in 2.6.5: RuntimeError: maximum recur 2010-05-24 http://bugs.python.org/issue8797 created jonozzz tkinter build variants on OSX 2010-05-24 http://bugs.python.org/issue8798 created ronaldoussoren Hang in lib/test/test_threading.py 2010-05-24 http://bugs.python.org/issue8799 created krisvale patch, patch, easy add threading.RWLock 2010-05-24 http://bugs.python.org/issue8800 created krisvale patch, patch, needs review Inconsistency in behaviour of urllib and urllib2 with file:// 2010-05-24 http://bugs.python.org/issue8801 created vinay.sajip about oa module 2010-05-24 CLOSED http://bugs.python.org/issue8802 created sudanti about oa module 2010-05-24 CLOSED http://bugs.python.org/issue8803 created sudanti http.client should support SSL contexts 2010-05-24 http://bugs.python.org/issue8804 created pitrou urllib should support SSL contexts 2010-05-24 http://bugs.python.org/issue8805 created pitrou ftplib should support SSL contexts 2010-05-24 CLOSED http://bugs.python.org/issue8806 created pitrou patch poplib should support SSL contexts 2010-05-24 http://bugs.python.org/issue8807 created pitrou patch imaplib should support SSL contexts 2010-05-24 http://bugs.python.org/issue8808 created pitrou smptlib should support SSL contexts 2010-05-24 http://bugs.python.org/issue8809 created pitrou TZ offset description is unclear in docs 2010-05-24 http://bugs.python.org/issue8810 created belopolsky easy fixing sqlite3 docs for py3k 2010-05-24 http://bugs.python.org/issue8811 created l0nwlf patch Show package path in repr string for packages installed to use 2010-05-24 CLOSED http://bugs.python.org/issue8812 created srid SSLContext doesn't support loading a CRL 2010-05-24 http://bugs.python.org/issue8813 created haypo functools.WRAPPER_ASSIGNMENTS should include __annotations__ 2010-05-24 http://bugs.python.org/issue8814 created terrence patch 65001 2010-05-24 CLOSED http://bugs.python.org/issue8815 created haypo Add test cases for builtins 2010-05-25 CLOSED http://bugs.python.org/issue8816 created gnofi patch Expose round-to-nearest division algorithm in Objects/longobje 2010-05-25 CLOSED http://bugs.python.org/issue8817 created mark.dickinson patch urlsplit and urlparse add extra slash when using scheme 2010-05-25 http://bugs.python.org/issue8818 reopened adamnelson variable resolution within exec() incorrect 2010-05-25 CLOSED http://bugs.python.org/issue8819 created hawkett IDLE not launching correctly 2010-05-26 http://bugs.python.org/issue8820 created NightFalcon Range check on unicode repr 2010-05-26 http://bugs.python.org/issue8821 created mgiuca patch datetime naive and aware types should have a well-defined defi 2010-05-26 http://bugs.python.org/issue8822 created techtonik urllib2 does not catch httplib.BadStatusLine 2010-05-26 http://bugs.python.org/issue8823 created adamnelson Improve documentation of exec 2010-05-26 http://bugs.python.org/issue8824 created tjreedy int("0",0) throws exception 2010-05-26 CLOSED http://bugs.python.org/issue8825 created cmoss60 Invalid expires date in cookiejar 2010-05-26 http://bugs.python.org/issue8826 created tzulberti poplib.POP3_SSL doesn't have an actual test suite 2010-05-26 CLOSED http://bugs.python.org/issue8827 created giampaolo.rodola Atomic function to rename a file 2010-05-26 http://bugs.python.org/issue8828 created haypo IDLE editior not opening 2010-05-27 http://bugs.python.org/issue8829 created Spatch Add nondestructive selection to sets 2010-05-27 CLOSED http://bugs.python.org/issue8830 created ipatrol recv and recvfrom on UDP socket do not return/throw exception 2010-05-27 http://bugs.python.org/issue8831 created Alessandro.Roat automate minidom.unlink() with a context manager 2010-05-27 http://bugs.python.org/issue8832 created krisvale patch, patch, easy, needs review tarfile: broken hardlink handling and testcase. 2010-05-27 http://bugs.python.org/issue8833 created jsbronder patch Define order of Misc/ACKS entries 2010-05-27 http://bugs.python.org/issue8834 created belopolsky patch buildbot: support.transient_internet() doesn't catch DNS socke 2010-05-27 http://bugs.python.org/issue8835 created haypo patch Conflicting default values of parameter to __import__ 2010-05-27 CLOSED http://bugs.python.org/issue8836 created afoglia PyArg_ParseTuple(): remove old and unused "O?" format 2010-05-27 http://bugs.python.org/issue8837 created haypo patch Remove codecs.readbuffer_encode() and codecs.charbuffer_encode 2010-05-27 http://bugs.python.org/issue8838 created haypo PyArg_ParseTuple(): remove "t# format 2010-05-27 http://bugs.python.org/issue8839 created haypo patch io.StringIO: truncate+print disabled in 3.1.2 2010-05-28 CLOSED http://bugs.python.org/issue8840 created tjreedy GetoptError strings should be localized 2010-05-28 http://bugs.python.org/issue8841 created richlowe sqlite3 library outdated in Windows builds 2010-05-28 CLOSED http://bugs.python.org/issue8842 created Marko.Kohtala urllib2 Digest Authorization uri must match request URI 2010-05-28 http://bugs.python.org/issue8843 created anelis patch Issues Now Closed (57) ______________________ _winreg.EnumValue sometimes raises WindowsError ("More data is 746 days http://bugs.python.org/issue2810 brian.curtin patch, needs review int() lies about base parameter 744 days http://bugs.python.org/issue2844 mark.dickinson patch SystemExit incorrectly displays unicode message 626 days http://bugs.python.org/issue3798 haypo patch cookielib chokes on non-integer cookie version, should ignore 608 days http://bugs.python.org/issue3924 georg.brandl patch Implement PEP 376 501 days http://bugs.python.org/issue4908 alexis patch Wrong print() result when unicode error handler is not 'strict 416 days http://bugs.python.org/issue5640 haypo patch CVE-2008-5983 python: untrusted python modules search path 402 days http://bugs.python.org/issue5753 pitrou patch Seeking to the beginning of a text file a second time will ret 345 days http://bugs.python.org/issue6268 haypo 2to3 fails to fix test.test_support 305 days http://bugs.python.org/issue6583 benjamin.peterson HTMLParser.HTMLParser doesn't handle malformed charrefs 291 days http://bugs.python.org/issue6662 haypo patch Sub-optimal "Locate" button behaviour in Windows CHM file 252 days http://bugs.python.org/issue6900 georg.brandl datetime operations spanning MINYEAR give bad results 224 days http://bugs.python.org/issue7150 belopolsky patch Struct incorrectly compiles format strings 184 days http://bugs.python.org/issue7355 mark.dickinson patch, easy A number tests "crash" if python is compiled --without-threads 13 days http://bugs.python.org/issue7449 haypo patch, easy, needs review Support for cp858 89 days http://bugs.python.org/issue8016 georg.brandl patch libffi update to 3.0.9 68 days http://bugs.python.org/issue8142 flox patch, buildbot urlparse has a duplicate of urllib.unquote 72 days http://bugs.python.org/issue8143 r.david.murray patch Unified hash for numeric types. 64 days http://bugs.python.org/issue8188 mark.dickinson patch siginterrupt with flag=False is reset when signal received 44 days http://bugs.python.org/issue8354 vila patch, needs review Make Context._clamp public in decimal module 26 days http://bugs.python.org/issue8540 mark.dickinson patch test/support: don't use localhost as IPv6 host name 25 days http://bugs.python.org/issue8598 haypo patch enumerate() docstring doesn't cover optional start argument 16 days http://bugs.python.org/issue8635 georg.brandl patch, easy enumerate() test cases do not cover optional start argument 19 days http://bugs.python.org/issue8636 mark.dickinson patch python3 FAQ mentions unicode() 10 days http://bugs.python.org/issue8694 georg.brandl Duplicated document in telnetlib. 8 days http://bugs.python.org/issue8707 georg.brandl patch, easy mention explicitly Windows support for os.devnull 8 days http://bugs.python.org/issue8709 georg.brandl patch bind_and_activate parameter is missed from directive 6 days http://bugs.python.org/issue8724 georg.brandl The Mapping ABC's __eq__ method should return NotImplemented i 6 days http://bugs.python.org/issue8729 benjamin.peterson patch Cruft in object.c: PyObject_GenericGetAttr 4 days http://bugs.python.org/issue8749 mark.dickinson Make 'python -m sysconfig' print something useful 5 days http://bugs.python.org/issue8770 tarek py3k: child process don't inherit stdout / stderr on Windows 0 days http://bugs.python.org/issue8780 haypo patch inspect.getsource returns invalid source for non-newline-endin 1 days http://bugs.python.org/issue8782 benjamin.peterson patch The external link to a "Hash Collision FAQ" points to some com 0 days http://bugs.python.org/issue8783 georg.brandl findall() and finditer() docs misleading 0 days http://bugs.python.org/issue8785 georg.brandl spam 1 days http://bugs.python.org/issue8789 mark.dickinson spam 1 days http://bugs.python.org/issue8790 mark.dickinson spam 1 days http://bugs.python.org/issue8791 mark.dickinson spam 0 days http://bugs.python.org/issue8794 mark.dickinson Error sending utf8 strings over syslog udp protocol 1 days http://bugs.python.org/issue8795 vinay.sajip about oa module 0 days http://bugs.python.org/issue8802 mark.dickinson about oa module 0 days http://bugs.python.org/issue8803 mark.dickinson ftplib should support SSL contexts 2 days http://bugs.python.org/issue8806 giampaolo.rodola patch Show package path in repr string for packages installed to use 0 days http://bugs.python.org/issue8812 srid 65001 0 days http://bugs.python.org/issue8815 haypo Add test cases for builtins 1 days http://bugs.python.org/issue8816 mark.dickinson patch Expose round-to-nearest division algorithm in Objects/longobje 1 days http://bugs.python.org/issue8817 mark.dickinson patch variable resolution within exec() incorrect 1 days http://bugs.python.org/issue8819 mark.dickinson int("0",0) throws exception 2 days http://bugs.python.org/issue8825 mark.dickinson poplib.POP3_SSL doesn't have an actual test suite 0 days http://bugs.python.org/issue8827 giampaolo.rodola Add nondestructive selection to sets 0 days http://bugs.python.org/issue8830 eric.smith Conflicting default values of parameter to __import__ 0 days http://bugs.python.org/issue8836 benjamin.peterson io.StringIO: truncate+print disabled in 3.1.2 0 days http://bugs.python.org/issue8840 pitrou sqlite3 library outdated in Windows builds 0 days http://bugs.python.org/issue8842 brian.curtin An inconsistency with nested scopes 2142 days http://bugs.python.org/issue991196 mark.dickinson Possible windows+python bug 1889 days http://bugs.python.org/issue1168427 tjreedy yday in datetime module 1552 days http://bugs.python.org/issue1436346 georg.brandl patch, needs review clean up Solaris port and allow C99 extension modules 1040 days http://bugs.python.org/issue1759169 loewis patch Top Issues Most Discussed (10) ______________________________ 21 datetime lacks concrete tzinfo impl. for UTC 485 days open http://bugs.python.org/issue5094 18 Remove codecs.readbuffer_encode() and codecs.charbuffer_encode( 1 days open http://bugs.python.org/issue8838 11 urlsplit and urlparse add extra slash when using scheme 3 days open http://bugs.python.org/issue8818 11 timedelta multiply and divide by floating point 1719 days open http://bugs.python.org/issue1289118 10 int("0",0) throws exception 2 days closed http://bugs.python.org/issue8825 9 Atomic function to rename a file 2 days open http://bugs.python.org/issue8828 9 xz compressor support 284 days open http://bugs.python.org/issue6715 9 please support lzma compression as an extension and in the tarf 419 days open http://bugs.python.org/issue5689 8 Too narrow platform check in test_datetime 110 days open http://bugs.python.org/issue7879 7 ftplib should support SSL contexts 2 days closed http://bugs.python.org/issue8806 From vinay_sajip at yahoo.co.uk Sat May 29 00:28:46 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 28 May 2010 22:28:46 +0000 (UTC) Subject: [Python-Dev] PEP 3148 ready for pronouncement References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> Message-ID: Brian Quinlan sweetapp.com> writes: > "future" is a computing science term of art, like "thread". Anyway, > this has been discussed in the past and Guido was happy with the name. I've not seen this mentioned, but on such a long thread I might have missed it: we already have a "__future__" module, as in from __future__ import with_statement and to my mind, this is a potential point of confusion with the term "future". > > * It seems unnecessarily verbose to tack "Executor" > > onto the end of every Executor subclass. They could > > simply be called ThreadPool and ProcessPool without > > losing anything. > > You could have general thread pools that aren't related to executors > (actually, it would be great if Python had a good built-in thread pool > implementation) and I'd like to avoid using an overly generic name. > Aren't executors and pools both housekeepers around a bunch of threads which execute code as a service for other threads? A thread is useless unless it executes code, isn't it? I'm not sure I understand the fine distinction between an executor and a pool. Having Executor as a suffix will give a point of familiarity to those who already know java.util.concurrent. And ProcessPool might cause confusion with multiprocessing.Pool because at first glance they seem to be the same thing. > > * I don't see a strong reason to put this module > > inside a newly-created namespace. If there were a > > namespace called "concurrent", I would expect to find > > other existing concurrency-related modules there as > > well, such as threading and multiprocessing. But we > > can't move them there without breaking existing code. > > I think that Jesse was planning to add some functionality to this > namespace. I don't really have an opinion on this. > I'm not sure of the benefit of a "concurrent" namespace, since it wouldn't contain the existing concurrency stuff. I think it's something to consider only for a big reorg which would break backward compatibility. IMO it would make more sense to leave this module as a top-level module for now (a sibling to "threading", "multiprocessing"). Regards, Vinay Sajip From tjreedy at udel.edu Sat May 29 01:03:37 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 28 May 2010 19:03:37 -0400 Subject: [Python-Dev] Bugfix releases should not change APIs Message-ID: I had the strong impression that there was a policy that x.y.z bugfix releases should only fix bugs and not add new features and revise current ones. The rationale, as I understood it, is that x.y.z releases should be increasingly better implementations of a stable x.y feature set. Adding features is 'bad' because code using a new feature will not work in previous releases of the same x.y version. Changing features is even worse because it may also break working code going forward. Because of this policy, an x.y.z Windows installer (I do not know about others) deletes any earlier release of the same version. Also, there is no What's New in Python x.y.z (relative to x.y.(z-1) since such should be empty. Consequently, violations of the policy are pretty much silent and well hidden. Yesterday, I spent two hours puzzling over the failure of my previously 'green' test sequence that tested a custom test function. I finally realized that the change was not due to anything I did (or undid), but a change in 3.1.2 in the interaction of StringIO.truncate, StringIO.getvalue, and print(x, StringIO()). (I should note the it is the usual stability and quality of Python that made me slow to blame Python rather than myself.) After filing http://bugs.python.org/issue8840 I was rather shocked to be told that the code-breaking and policy-violating change was intentional because being 'more consistent with other file-handling APIs out there' was somehow more important than consistency within version releases. It also seems to me that discussion of code-breaking API changes like this should involve more than one user and one developer. See http://bugs.python.org/issue6939 I have fixed my tests so they works in 3.1.2 and should work in other 3.1 releases, but that would be a nuisance to test. Of course, I should not have to worry about API changes within a version series. I think issue 8840 illustrates another reason for the bugfix-only policy. New x.y features and docs are (nearly always) added before the first beta. They can then be tested, discussed, and improved. This 'cooking' does not occur for bugfix releases. For reasons I give in my response on the tracker, I think the new behavior is buggy and the doc deficient. So, I think 1) the supposed bugfix-only policy should really be the policy; 2) the violation in 3.1.2 should be reverted in 3.1.3, and the API change reviewed in the normal 3.2 alpha/beta process. I am curious as to what others think on both points. Terry Jan Reedy From steve at pearwood.info Sat May 29 02:12:26 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 29 May 2010 10:12:26 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> Message-ID: <201005291012.26842.steve@pearwood.info> On Sat, 29 May 2010 08:28:46 am Vinay Sajip wrote: > I've not seen this mentioned, but on such a long thread I might have > missed it: we already have a "__future__" module, as in > > from __future__ import with_statement > > and to my mind, this is a potential point of confusion with the term > "future". [...] > I'm not sure of the benefit of a "concurrent" namespace, since it > wouldn't contain the existing concurrency stuff. I think it's > something to consider only for a big reorg which would break backward > compatibility. IMO it would make more sense to leave this module as a > top-level module for now (a sibling to "threading", > "multiprocessing"). I have suggested a way to move the existing concurrency stuff without breaking backwards compatibility, and Terry Reedy asked if it would work. I haven't seen any responses, either positive or negative. For the record, my suggestion was: for each concurrency modules: move it into the concurrency package add a top level module with the same name containing: # e.g. for threading from concurrency.threading import * Then in some future Python version, each top level module gets a PendingDeprecation warning, followed by a Deprecation warning some time later, and eventually in the indefinite future removal of the top level module. Leaving the futures module in the top level of the std lib is far more likely to confuse users than adding it to its own namespace. Compare: import __future__ import futures with: import __future__ import concurrency.futures In my opinion, it is high time for the std lib to pay more attention to the final Zen: Namespaces are one honking great idea -- let's do more of those! -- Steven D'Aprano From steve at holdenweb.com Sat May 29 02:16:04 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 28 May 2010 20:16:04 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com> <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com> <4BFC66F1.3060107@gmail.com> <4BFCFE8B.1080205@canterbury.ac.nz> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> Message-ID: <4C005CC4.2040205@holdenweb.com> Vinay Sajip wrote: > Brian Quinlan sweetapp.com> writes: > >> "future" is a computing science term of art, like "thread". Anyway, >> this has been discussed in the past and Guido was happy with the name. > > I've not seen this mentioned, but on such a long thread I might have missed it: > we already have a "__future__" module, as in > > from __future__ import with_statement > > and to my mind, this is a potential point of confusion with the term "future". > >>> * It seems unnecessarily verbose to tack "Executor" >>> onto the end of every Executor subclass. They could >>> simply be called ThreadPool and ProcessPool without >>> losing anything. >> You could have general thread pools that aren't related to executors >> (actually, it would be great if Python had a good built-in thread pool >> implementation) and I'd like to avoid using an overly generic name. >> > > Aren't executors and pools both housekeepers around a bunch of threads which > execute code as a service for other threads? A thread is useless unless it > executes code, isn't it? I'm not sure I understand the fine distinction between > an executor and a pool. Having Executor as a suffix will give a point of > familiarity to those who already know java.util.concurrent. And ProcessPool > might cause confusion with multiprocessing.Pool because at first glance they > seem to be the same thing. > >>> * I don't see a strong reason to put this module >>> inside a newly-created namespace. If there were a >>> namespace called "concurrent", I would expect to find >>> other existing concurrency-related modules there as >>> well, such as threading and multiprocessing. But we >>> can't move them there without breaking existing code. >> I think that Jesse was planning to add some functionality to this >> namespace. I don't really have an opinion on this. >> > > I'm not sure of the benefit of a "concurrent" namespace, since it wouldn't > contain the existing concurrency stuff. I think it's something to consider only > for a big reorg which would break backward compatibility. IMO it would make more > sense to leave this module as a top-level module for now (a sibling to > "threading", "multiprocessing"). > Unless there's some way of having the two namespaces (existing and concurrent-oriented) simultaneously coexist. A single implementation with two separate namespace mappings? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ "All I want for my birthday is another birthday" - Ian Dury, 1942-2000 From jnoller at gmail.com Sat May 29 02:19:09 2010 From: jnoller at gmail.com (Jesse Noller) Date: Fri, 28 May 2010 20:19:09 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <201005291012.26842.steve@pearwood.info> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> <201005291012.26842.steve@pearwood.info> Message-ID: <2F64E36B-5849-4FEB-8ACD-F81AE2417910@gmail.com> On May 28, 2010, at 8:12 PM, Steven D'Aprano wrote: > On Sat, 29 May 2010 08:28:46 am Vinay Sajip wrote: > >> I've not seen this mentioned, but on such a long thread I might have >> missed it: we already have a "__future__" module, as in >> >> from __future__ import with_statement >> >> and to my mind, this is a potential point of confusion with the term >> "future". > [...] >> I'm not sure of the benefit of a "concurrent" namespace, since it >> wouldn't contain the existing concurrency stuff. I think it's >> something to consider only for a big reorg which would break backward >> compatibility. IMO it would make more sense to leave this module as a >> top-level module for now (a sibling to "threading", >> "multiprocessing"). > > I have suggested a way to move the existing concurrency stuff without > breaking backwards compatibility, and Terry Reedy asked if it would > work. I haven't seen any responses, either positive or negative. > > For the record, my suggestion was: > > for each concurrency modules: > move it into the concurrency package > add a top level module with the same name containing: > # e.g. for threading > from concurrency.threading import * > > Then in some future Python version, each top level module gets a > PendingDeprecation warning, followed by a Deprecation warning some > time > later, and eventually in the indefinite future removal of the top > level > module. > > > Leaving the futures module in the top level of the std lib is far more > likely to confuse users than adding it to its own namespace. Compare: > > import __future__ > import futures > > with: > > import __future__ > import concurrency.futures > > > In my opinion, it is high time for the std lib to pay more attention > to > the final Zen: > > Namespaces are one honking great idea -- let's do more of those! > > > Yes, your suggestion for how to move things is the way we would want to do it, and in the back of my head, what we should do long term - just not right now. From tseaver at palladion.com Sat May 29 02:31:05 2010 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 28 May 2010 20:31:05 -0400 Subject: [Python-Dev] Bugfix releases should not change APIs In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Terry Reedy wrote: > I had the strong impression that there was a policy that x.y.z bugfix > releases should only fix bugs and not add new features and revise > current ones. The rationale, as I understood it, is that x.y.z releases > should be increasingly better implementations of a stable x.y feature > set. Adding features is 'bad' because code using a new feature will not > work in previous releases of the same x.y version. Changing features is > even worse because it may also break working code going forward. > > Because of this policy, an x.y.z Windows installer (I do not know about > others) deletes any earlier release of the same version. Also, there is > no What's New in Python x.y.z (relative to x.y.(z-1) since such should > be empty. Consequently, violations of the policy are pretty much silent > and well hidden. > > Yesterday, I spent two hours puzzling over the failure of my previously > 'green' test sequence that tested a custom test function. I finally > realized that the change was not due to anything I did (or undid), but a > change in 3.1.2 in the interaction of StringIO.truncate, > StringIO.getvalue, and print(x, StringIO()). (I should note the it is > the usual stability and quality of Python that made me slow to blame > Python rather than myself.) > > After filing > http://bugs.python.org/issue8840 > I was rather shocked to be told that the code-breaking and > policy-violating change was intentional because being 'more consistent > with other file-handling APIs out there' was somehow more important than > consistency within version releases. It also seems to me that discussion > of code-breaking API changes like this should involve more than one user > and one developer. See > http://bugs.python.org/issue6939 > > I have fixed my tests so they works in 3.1.2 and should work in other > 3.1 releases, but that would be a nuisance to test. Of course, I should > not have to worry about API changes within a version series. > > I think issue 8840 illustrates another reason for the bugfix-only > policy. New x.y features and docs are (nearly always) added before the > first beta. They can then be tested, discussed, and improved. This > 'cooking' does not occur for bugfix releases. For reasons I give in my > response on the tracker, I think the new behavior is buggy and the doc > deficient. > > So, I think > 1) the supposed bugfix-only policy should really be the policy; > 2) the violation in 3.1.2 should be reverted in 3.1.3, and the API > change reviewed in the normal 3.2 alpha/beta process. > I am curious as to what others think on both points. +1 on #1 as the general policy. I don't have enough skin in the game of the 3.1.x world to have an opinion about this specific breakage, but I have certainly seen other examples in the 2.x releases, where such a resolution was the appropriate outcome. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkwAYEkACgkQ+gerLs4ltQ6AuQCfTk4mAl3ClpE1uu6nBRNrNjBc g54AoI2SObUNn/d0RvIbYj/vl7HDQbbj =U8z2 -----END PGP SIGNATURE----- From steve at holdenweb.com Sat May 29 02:31:02 2010 From: steve at holdenweb.com (Steve Holden) Date: Fri, 28 May 2010 20:31:02 -0400 Subject: [Python-Dev] Bugfix releases should not change APIs In-Reply-To: References: Message-ID: Terry Reedy wrote: > I had the strong impression that there was a policy that x.y.z bugfix > releases should only fix bugs and not add new features and revise > current ones. The rationale, as I understood it, is that x.y.z releases > should be increasingly better implementations of a stable x.y feature > set. Adding features is 'bad' because code using a new feature will not > work in previous releases of the same x.y version. Changing features is > even worse because it may also break working code going forward. > > Because of this policy, an x.y.z Windows installer (I do not know about > others) deletes any earlier release of the same version. Also, there is > no What's New in Python x.y.z (relative to x.y.(z-1) since such should > be empty. Consequently, violations of the policy are pretty much silent > and well hidden. > > Yesterday, I spent two hours puzzling over the failure of my previously > 'green' test sequence that tested a custom test function. I finally > realized that the change was not due to anything I did (or undid), but a > change in 3.1.2 in the interaction of StringIO.truncate, > StringIO.getvalue, and print(x, StringIO()). (I should note the it is > the usual stability and quality of Python that made me slow to blame > Python rather than myself.) > > After filing > http://bugs.python.org/issue8840 > I was rather shocked to be told that the code-breaking and > policy-violating change was intentional because being 'more consistent > with other file-handling APIs out there' was somehow more important than > consistency within version releases. It also seems to me that discussion > of code-breaking API changes like this should involve more than one user > and one developer. See > http://bugs.python.org/issue6939 > > I have fixed my tests so they works in 3.1.2 and should work in other > 3.1 releases, but that would be a nuisance to test. Of course, I should > not have to worry about API changes within a version series. > > I think issue 8840 illustrates another reason for the bugfix-only > policy. New x.y features and docs are (nearly always) added before the > first beta. They can then be tested, discussed, and improved. This > 'cooking' does not occur for bugfix releases. For reasons I give in my > response on the tracker, I think the new behavior is buggy and the doc > deficient. > > So, I think > 1) the supposed bugfix-only policy should really be the policy; > 2) the violation in 3.1.2 should be reverted in 3.1.3, and the API > change reviewed in the normal 3.2 alpha/beta process. > I am curious as to what others think on both points. > I think it shows how developers can "get worked over" if they are insufficiently vigilant. 1) I completely agree, and adduce as evidence the fact that something like this always seems to happen when the rule is broken; 2) But we may wish to release 3.1.2.1(?) which backports fixes from the 3.2 line but retains the file store semantics (which I am assured will be easy in the glorious reign of Hg). Surely some compatible "shim" layer could have been introduced? What is the process for ensuring that such API changes do *not* creep into maintenance releases? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS: http://holdenweb.eventbrite.com/ "All I want for my birthday is another birthday" - Ian Dury, 1942-2000 From benjamin at python.org Sat May 29 03:23:37 2010 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 28 May 2010 20:23:37 -0500 Subject: [Python-Dev] Bugfix releases should not change APIs In-Reply-To: References: Message-ID: 2010/5/28 Steve Holden : > Terry Reedy wrote: >> http://bugs.python.org/issue8840 >> I was rather shocked to be told that the code-breaking and >> policy-violating change was intentional because being 'more consistent >> with other file-handling APIs out there' was somehow more important than >> consistency within version releases. It also seems to me that discussion >> of code-breaking API changes like this should involve more than one user >> and one developer. This was discussed on the mailing list and this was produced: http://mail.python.org/pipermail/python-dev/2009-September/092247.html > What is the process for ensuring that such API changes do *not* creep > into maintenance releases? Generally developers are good about checking with the wider developer community about possibly incompatible changes. And given the link I posted above I don't consider this change to have "crept" in. -- Regards, Benjamin From ncoghlan at gmail.com Sat May 29 05:31:07 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 May 2010 13:31:07 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <2F64E36B-5849-4FEB-8ACD-F81AE2417910@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> <201005291012.26842.steve@pearwood.info> <2F64E36B-5849-4FEB-8ACD-F81AE2417910@gmail.com> Message-ID: <4C008A7B.1090907@gmail.com> On 29/05/10 10:19, Jesse Noller wrote: >> In my opinion, it is high time for the std lib to pay more attention to >> the final Zen: >> >> Namespaces are one honking great idea -- let's do more of those! >> >> >> > Yes, your suggestion for how to move things is the way we would want to > do it, and in the back of my head, what we should do long term - just > not right now. Yep, this is what I have been saying as well. 1. Using concurrency.futures rather than a top level futures module resolves the potential confusion with __future__ and stock market futures without inventing our own name for a well established computer science concept. 2. With the concurrency package in place following PEP 3148, we can separately consider the question of if/when/how to move other concurrency related modules (e.g. threading, multiprocessing, Queue) into that package at a later date. Since this topic keeps coming up, some reasoning along these lines should go into PEP 3148. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sat May 29 05:41:33 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 May 2010 13:41:33 +1000 Subject: [Python-Dev] Bugfix releases should not change APIs In-Reply-To: References: Message-ID: <4C008CED.8080406@gmail.com> On 29/05/10 09:03, Terry Reedy wrote: > After filing > http://bugs.python.org/issue8840 > I was rather shocked to be told that the code-breaking and > policy-violating change was intentional because being 'more consistent > with other file-handling APIs out there' was somehow more important than > consistency within version releases. It also seems to me that discussion > of code-breaking API changes like this should involve more than one user > and one developer. See > http://bugs.python.org/issue6939 As Benjamin noted, that change was discussed on python-dev and actually made at Guido's direction. > So, I think > 1) the supposed bugfix-only policy should really be the policy; > 2) the violation in 3.1.2 should be reverted in 3.1.3, and the API > change reviewed in the normal 3.2 alpha/beta process. > I am curious as to what others think on both points. The bugfix-only policy remains in place, but there are corner cases (such as this one) where the definition of what is and isn't a bug gets fuzzy and we need to make a judgment call as to whether to fix them or not. In particular, fixing bugs often runs the risk of breaking existing workarounds for those bugs. The decimal module, for example, has a standing policy to treat deviations from the standard as bugs, and this may lead to changes in the module if the standard is updated between two releases. This specific change was discussed on python-dev and the chance of breaking existing 3.1 code was deemed preferable to retaining semantics that were seen as broken, so I would be against reverting it. However, it may be worth modifying the policy to ensure that such exceptional bug fixes be mentioned prominently in the release notes and on the download page for that maintenance release. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sat May 29 05:47:30 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 May 2010 13:47:30 +1000 Subject: [Python-Dev] Bugfix releases should not change APIs In-Reply-To: References: Message-ID: <4C008E52.8070106@gmail.com> On 29/05/10 09:03, Terry Reedy wrote: > After filing > http://bugs.python.org/issue8840 > I was rather shocked to be told that the code-breaking and > policy-violating change was intentional because being 'more consistent > with other file-handling APIs out there' was somehow more important than > consistency within version releases. It also seems to me that discussion > of code-breaking API changes like this should involve more than one user > and one developer. See > http://bugs.python.org/issue6939 For the record, I have added links to Guido's python-dev post to both of these tracker issues. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From eric at trueblade.com Sat May 29 06:53:23 2010 From: eric at trueblade.com (Eric Smith) Date: Sat, 29 May 2010 00:53:23 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <201005291012.26842.steve@pearwood.info> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> <201005291012.26842.steve@pearwood.info> Message-ID: <4C009DC3.3060006@trueblade.com> Steven D'Aprano wrote: > I have suggested a way to move the existing concurrency stuff without > breaking backwards compatibility, and Terry Reedy asked if it would > work. I haven't seen any responses, either positive or negative. > > For the record, my suggestion was: > > for each concurrency modules: > move it into the concurrency package > add a top level module with the same name containing: > # e.g. for threading > from concurrency.threading import * In the past the problem identified with this approach has been that pickles produced with new pythons would not be readable by older pythons. I think this was the main reason that Brett's 3.0 library reorganization wasn't more radical. Theres a discussion if this here: http://mail.python.org/pipermail/python-dev/2008-May/079535.html http://mail.python.org/pipermail/stdlib-sig/2008-May/000303.html and a little more here: http://bugs.python.org/issue2775 Eric. From hawkett at gmail.com Sat May 29 12:20:29 2010 From: hawkett at gmail.com (Colin H) Date: Sat, 29 May 2010 11:20:29 +0100 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: <4BFDB8EE.4010307@gmail.com> <4BFE92D9.1060306@gmail.com> Message-ID: Perhaps the next step is to re-open the issue? If it is seen as a bug, it would be great to see a fix in 2.6+ - a number of options which will not break backward compatibility have been put forward - cheers, Colin On Thu, May 27, 2010 at 9:05 PM, Reid Kleckner wrote: > On Thu, May 27, 2010 at 11:42 AM, Nick Coghlan wrote: >> However, attaining the (sensible) behaviour Colin is requesting when such >> top level variable references exist would actually be somewhat tricky. >> Considering Guido's suggestion to treat two argument exec like a function >> rather than a class and generate a closure with full lexical scoping a >> little further, I don't believe this could be done in exec itself without >> breaking code that expects the current behaviour. > > Just to give a concrete example, here is code that would break if exec > were to execute code in a function scope instead of a class scope: > > exec """ > def len(xs): > ? ?return -1 > def foo(): > ? ?return len([]) > print foo() > """ in globals(), {} > > Currently, the call to 'len' inside 'foo' skips the outer scope > (because it's a class scope) and goes straight to globals and > builtins. ?If it were switched to a local scope, a cell would be > created for the broken definition of 'len', and the call would resolve > to it. > > Honestly, to me, the fact that the above code ever worked (ie prints > "0", not "-1") seems like a bug, so I wouldn't worry about backwards > compatibility. > > Reid > From solipsis at pitrou.net Sat May 29 12:39:37 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 29 May 2010 12:39:37 +0200 Subject: [Python-Dev] Bugfix releases should not change APIs References: Message-ID: <20100529123937.222a4c0d@pitrou.net> On Fri, 28 May 2010 20:31:02 -0400 Steve Holden wrote: > I think it shows how developers can "get worked over" if they are > insufficiently vigilant. > > 1) I completely agree, and adduce as evidence the fact that something > like this always seems to happen when the rule is broken; Well it's true that the change may have been unfortunate (this is the second bug report we get about it, the first one was from Holger Krekel IIRC), its exceptional nature had also been discussed on this mailing-list, and supported by Guido. It is not the product of oversight. What it does teach us is that Python 3.1 sees some real use, and we have entered a phase where backwards compatibility will become as important as it was in the 2.x line. > 2) But we may wish to release 3.1.2.1(?) which backports fixes from the > 3.2 line but retains the file store semantics (which I am assured will > be easy in the glorious reign of Hg). I think this would be worse, as in "even more confusing". We would have a 3.1.2 with changed behaviour, a 3.1.2.1 with reverted behaviour, and a 3.2 with changed behaviour again. Now that we have inflicted this pain on our users, let's not inflict more pain on them. Regards Antoine. From ncoghlan at gmail.com Sat May 29 14:13:24 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 May 2010 22:13:24 +1000 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: <4BFDB8EE.4010307@gmail.com> <4BFE92D9.1060306@gmail.com> Message-ID: <4C0104E4.3080802@gmail.com> On 29/05/10 20:20, Colin H wrote: > Perhaps the next step is to re-open the issue? If it is seen as a bug, > it would be great to see a fix in 2.6+ - a number of options which > will not break backward compatibility have been put forward - cheers, A new feature request requesting a "closure" mode for compile() in 3.2 would probably be the best way forward. Once that is done, then the question of if or when to change the default behaviour for auto-compiled code in exec and/or dis can be considered. It definitely isn't a bug fix though - it's worked this way for years, and while the existing semantics can certainly be surprising, they're far from being buggy (as Thomas said, prior to the introduction of lexical scoping all Python namespaces worked this way). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From jnoller at gmail.com Sat May 29 14:46:28 2010 From: jnoller at gmail.com (Jesse Noller) Date: Sat, 29 May 2010 08:46:28 -0400 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <4C008A7B.1090907@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> <201005291012.26842.steve@pearwood.info> <2F64E36B-5849-4FEB-8ACD-F81AE2417910@gmail.com> <4C008A7B.1090907@gmail.com> Message-ID: <80D222EA-EAFD-4EDB-BECF-0FB7F0567A95@gmail.com> On May 28, 2010, at 11:31 PM, Nick Coghlan wrote: > On 29/05/10 10:19, Jesse Noller wrote: >>> In my opinion, it is high time for the std lib to pay more >>> attention to >>> the final Zen: >>> >>> Namespaces are one honking great idea -- let's do more of those! >>> >>> >>> >> Yes, your suggestion for how to move things is the way we would >> want to >> do it, and in the back of my head, what we should do long term - just >> not right now. > > Yep, this is what I have been saying as well. > > 1. Using concurrency.futures rather than a top level futures module > resolves the potential confusion with __future__ and stock market > futures without inventing our own name for a well established > computer science concept. > > 2. With the concurrency package in place following PEP 3148, we can > separately consider the question of if/when/how to move other > concurrency related modules (e.g. threading, multiprocessing, Queue) > into that package at a later date. > > Since this topic keeps coming up, some reasoning along these lines > should go into PEP 3148. > I'll type something up this weekend and shoot it to Brian for inclusion. I was hoping to be able to keep it out of the futures pep itself, but it seems that won't work :) Jesse From ncoghlan at gmail.com Sat May 29 17:51:52 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 May 2010 01:51:52 +1000 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <80D222EA-EAFD-4EDB-BECF-0FB7F0567A95@gmail.com> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> <201005291012.26842.steve@pearwood.info> <2F64E36B-5849-4FEB-8ACD-F81AE2417910@gmail.com> <4C008A7B.1090907@gmail.com> <80D222EA-EAFD-4EDB-BECF-0FB7F0567A95@gmail.com> Message-ID: <4C013818.3090000@gmail.com> On 29/05/10 22:46, Jesse Noller wrote: > On May 28, 2010, at 11:31 PM, Nick Coghlan wrote: >> Since this topic keeps coming up, some reasoning along these lines >> should go into PEP 3148. > > I'll type something up this weekend and shoot it to Brian for inclusion. > I was hoping to be able to keep it out of the futures pep itself, but it > seems that won't work :) Well, punting on whether or not we actually *do* part 2 is still fine. As Eric pointed out, there are issues with unpickling that make the wisdom of following through with renaming any existing modules fairly questionable. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From eric at trueblade.com Sat May 29 15:52:02 2010 From: eric at trueblade.com (Eric Smith) Date: Sat, 29 May 2010 09:52:02 -0400 Subject: [Python-Dev] Implementing PEP 382, Namespace Packages Message-ID: <4C011C02.4010605@trueblade.com> Last night Barry Warsaw, Jason Coombs, and I met to work on implementing PEP 382. As part of my research, I came across this email from Martin: http://mail.python.org/pipermail/python-dev/2009-May/089316.html In it he says that PEP 382 is being deferred until it can address PEP 302 loaders. I can't find any follow-up to this. I don't see any discussion in PEP 382 about PEP 302 loaders, so I assume this issue was never resolved. Does it need to be before PEP 382 is implemented? Are we wasting our time by designing and (eventually) coding before this issue is resolved? Eric. From martin at v.loewis.de Sat May 29 20:45:39 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 29 May 2010 20:45:39 +0200 Subject: [Python-Dev] Implementing PEP 382, Namespace Packages In-Reply-To: <4C011C02.4010605@trueblade.com> References: <4C011C02.4010605@trueblade.com> Message-ID: <4C0160D3.70304@v.loewis.de> > In it he says that PEP 382 is being deferred until it can address PEP > 302 loaders. I can't find any follow-up to this. I don't see any > discussion in PEP 382 about PEP 302 loaders, so I assume this issue was > never resolved. Does it need to be before PEP 382 is implemented? Are we > wasting our time by designing and (eventually) coding before this issue > is resolved? Yes, and yes. Regards, Martin From martin at v.loewis.de Sat May 29 21:29:33 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 29 May 2010 21:29:33 +0200 Subject: [Python-Dev] Implementing PEP 382, Namespace Packages In-Reply-To: <20100529190623.299C83A405F@sparrow.telecommunity.com> References: <4C011C02.4010605@trueblade.com> <4C0160D3.70304@v.loewis.de> <20100529190623.299C83A405F@sparrow.telecommunity.com> Message-ID: <4C016B1D.8010604@v.loewis.de> Am 29.05.2010 21:06, schrieb P.J. Eby: > At 08:45 PM 5/29/2010 +0200, Martin v. L?wis wrote: >>> In it he says that PEP 382 is being deferred until it can address PEP >>> 302 loaders. I can't find any follow-up to this. I don't see any >>> discussion in PEP 382 about PEP 302 loaders, so I assume this issue was >>> never resolved. Does it need to be before PEP 382 is implemented? Are we >>> wasting our time by designing and (eventually) coding before this issue >>> is resolved? >> >> Yes, and yes. > > Is there anything we can do to help regarding that? You could comment on the proposal I made back then, or propose a different solution. Regards, Martin From brett at python.org Sat May 29 22:46:25 2010 From: brett at python.org (Brett Cannon) Date: Sat, 29 May 2010 13:46:25 -0700 Subject: [Python-Dev] PEP 3148 ready for pronouncement In-Reply-To: <201005291012.26842.steve@pearwood.info> References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com> <201005291012.26842.steve@pearwood.info> Message-ID: On Fri, May 28, 2010 at 17:12, Steven D'Aprano wrote: > On Sat, 29 May 2010 08:28:46 am Vinay Sajip wrote: > >> I've not seen this mentioned, but on such a long thread I might have >> missed it: we already have a "__future__" module, as in >> >> from __future__ import with_statement >> >> and to my mind, this is a potential point of confusion with the term >> "future". > [...] >> I'm not sure of the benefit of a "concurrent" namespace, since it >> wouldn't contain the existing concurrency stuff. I think it's >> something to consider only for a big reorg which would break backward >> compatibility. IMO it would make more sense to leave this module as a >> top-level module for now (a sibling to "threading", >> "multiprocessing"). > > I have suggested a way to move the existing concurrency stuff without > breaking backwards compatibility, and Terry Reedy asked if it would > work. I haven't seen any responses, either positive or negative. > > For the record, my suggestion was: > > for each concurrency modules: > ?move it into the concurrency package > ?add a top level module with the same name containing: > ? ?# e.g. for threading > ? ?from concurrency.threading import * > > Then in some future Python version, each top level module gets a > PendingDeprecation warning, followed by a Deprecation warning some time > later, and eventually in the indefinite future removal of the top level > module. This was the procedure we used for about a month for Python 2.6 in order to help renamed modules migrate to their new names in Python 3. The issue that came up (and forced use to revert this approach and fully rely on 2to3) was anything pickled by the older interpreters is not going to be happy with that shift. Luckily the stuff being moved most likely does not contain things that have been pickled and stored to disk for ages and thus would break in a transition. From brett at python.org Sun May 30 00:44:28 2010 From: brett at python.org (Brett Cannon) Date: Sat, 29 May 2010 15:44:28 -0700 Subject: [Python-Dev] Implementing PEP 382, Namespace Packages In-Reply-To: <4C016B1D.8010604@v.loewis.de> References: <4C011C02.4010605@trueblade.com> <4C0160D3.70304@v.loewis.de> <20100529190623.299C83A405F@sparrow.telecommunity.com> <4C016B1D.8010604@v.loewis.de> Message-ID: On Sat, May 29, 2010 at 12:29, "Martin v. L?wis" wrote: > Am 29.05.2010 21:06, schrieb P.J. Eby: >> >> At 08:45 PM 5/29/2010 +0200, Martin v. L?wis wrote: >>>> >>>> In it he says that PEP 382 is being deferred until it can address PEP >>>> 302 loaders. I can't find any follow-up to this. I don't see any >>>> discussion in PEP 382 about PEP 302 loaders, so I assume this issue was >>>> never resolved. Does it need to be before PEP 382 is implemented? Are we >>>> wasting our time by designing and (eventually) coding before this issue >>>> is resolved? >>> >>> Yes, and yes. >> >> Is there anything we can do to help regarding that? > > You could comment on the proposal I made back then, or propose a different > solution. [sorry for the fundamental PEP questions, but I think PEP 382 came about while I was on my python-dev sabbatical last year] I have some questions about the PEP which might help clarify how to handle the API changes. For finders, their search algorithm is changed in a couple of ways. One is that modules are given priority over packages (is that intentional, Martin, or just an oversight?). Two, the package search requires checking for a .pth file on top of an __init__.py. This will change finders that could before simply do an existence check on an __init__ "file" (or whatever the storage back-end happened to be) and make it into a list-and-search which one would hope wasn't costly, but in same cases might be if the paths to files is not stored in a hierarchical fashion (e.g. zip files list entire files paths in their TOC or a sqlite3 DB which uses a path for keys will have to list **all** keys, sort them to just the relevant directory, and then look for .pth or some such approach). Are we worried about possible performance implications of this search? I say no, but I just want to make sure people we are not and people are aware about the design shift required in finders. This entire worry would be alleviated if only .pth files named after the package were supported, much like *.pkg files in pkgutil. And then the search for the __init__.py begins on the newly modified __path__, which I assume ends with the first __init__ found on __path__, but if no file is found it's okay and essentially an empty module with just module-specific attributes is used? In other words, can a .pth file replace an __init__ file in delineating a package? Or is it purely additive? I assume the latter for compatibility reasons, but the PEP says "a directory is considered a package if it **either** contains a file named __init__.py, **or** a file whose name ends with ".pth"" (emphasis mine). Otherwise I assume that the search will be done simply with ``os.path.isdir(os.path.join(sys_path_entry, top_level_package_name)`` and all existing paths will be added to __path__. Will they come before or after the directory where the *.pth was found? And will any subsequent *.pth files found in other directories also be executed? As for how "*" works, is this limited to top-level packages, or will sub-packages participate as well? I assume the former, but it is not directly stated in the PEP. If the latter, is a dotted package name changed to ``os.sep.join(sy_path_entry, package_name.replace('".", os.sep)``? For sys.path_hooks, I am assuming import will simply skip over passing that as it is a marker that __path__ represents a namsepace package and not in any way functional. Although with sys.namespace_packages, is leaving the "*" in __path__ truly necessary? For the search of paths to use to extend, are we limiting ourselves to actual file system entries on sys.path (as pkgutil does), or do we want to support other storage back-ends? To do the latter I would suggest having a successful path discovery be when a finder can be created for the hypothetical directory from sys.path_hooks. OK, I *think* that's all of my clarification questions when it comes to the PEP. =) Now, on to API discussion. The PEP (seems to) ask finders to look for a .pth file(s), calculate __path__, and then get a loader for the __init__. You could have finders grow a find_namespace method which returns the contents of the requisite .pth file(s). Import could then take that, calculate __path__, and then use that new search path to find a loader for the __init__ (I am assuming there is an __init__ file somewhere). That's straight-forward and makes supporting .pth files additive for finders. The trick then becomes how the heck you get the new __path__ value into the module through the loader as up to this point it has calculated __path__ on its own. You could slightly abuse load_module's semantics for reloading and stick the namespace module into sys.modules before calling the loader for __init__ and change the semantics definition such that if __path__ is already defined you don't change it. Unfortunately that seems rather messy in the face of reloads that want a fresh __path__. Another possibility is to have the loader add the new paths, but to provide the calculated value of __path__ be stored on sys.namespace_packages. That way the loader can simply calculate its own version and extend it with what the dictionary provides. This allows loaders get what import thinks __path__ should be and they still have a chance to tweak things. If you want even more abstraction I would change is_package to return what __path__ should be when it is a package and provide an ABC that does the proper calculation of the extended __path__ value for is_package() so they can do ``return [extras].extend(super().is_package())`` for packages. But unfortunately, because load_module is overloaded with responsibilities, there is no way to dynamically add support for any of this to existing loaders like there is with finders (unless we factor out the responsibilities of load_module so it isn't so overworked and is entirely optional to implement, but that goes beyond this PEP's scope). There is also the issue of reloading with this delineation of work since finders are not necessarily called by imp.reload. Otherwise the loaders will have to recalculate everything import calculates in order to find the __init__ module to begin with. The only other option I can think of is to tweak find_module to always take a path argument, not just meta path finders. Then the calculated __path__ value can be passed in through find_module and thus passed on to the loader through a constructor or some such. That doesn't duplicate the work of calculating the extended __path__ value in both the finder or loader, nor having to cache it somewhere outside of the importer's reach where it might go stale. The finder simply passed the __path__ value on to the loader however it wants (most likely through a constructor call or internal caching). This also acts as a performance perk when searching for the __init__ module as having the 'path' argument set can act as a flag to not look for a module but only a package. This would make it no longer an additive feature to finders, but wouldn't require anything to change in loaders directly. I'll shut up now and stop causing trouble. =) From pje at telecommunity.com Sun May 30 00:56:10 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sat, 29 May 2010 18:56:10 -0400 Subject: [Python-Dev] Implementing PEP 382, Namespace Packages In-Reply-To: <4C016B1D.8010604@v.loewis.de> References: <4C011C02.4010605@trueblade.com> <4C0160D3.70304@v.loewis.de> <20100529190623.299C83A405F@sparrow.telecommunity.com> <4C016B1D.8010604@v.loewis.de> Message-ID: <20100529225613.2DD0A3A405F@sparrow.telecommunity.com> At 09:29 PM 5/29/2010 +0200, Martin v. L?wis wrote: >Am 29.05.2010 21:06, schrieb P.J. Eby: >>At 08:45 PM 5/29/2010 +0200, Martin v. L?wis wrote: >>>>In it he says that PEP 382 is being deferred until it can address PEP >>>>302 loaders. I can't find any follow-up to this. I don't see any >>>>discussion in PEP 382 about PEP 302 loaders, so I assume this issue was >>>>never resolved. Does it need to be before PEP 382 is implemented? Are we >>>>wasting our time by designing and (eventually) coding before this issue >>>>is resolved? >>> >>>Yes, and yes. >> >>Is there anything we can do to help regarding that? > >You could comment on the proposal I made back then, or propose a >different solution. Looking at that proposal, I don't follow how changing *loaders* (vs. importers) would help. If an importer's find_module doesn't natively support PEP 382, then there's no way to get a loader for the package in the first place. Today, namespace packages work fine with PEP 302 loaders, because the namespace-ness is really only about setting up the __path__, and detecting that you need to do this in the first place. In the PEP 302 scheme, then, it's either importers that have to change, or the process that invokes them. Being able to ask an importer the equivalents of os.path.join, listdir, and get_data would suffice to make an import process that could do the trick. Essentially, you'd ask each importer to first attempt to find the module, and then asking it (or the loader, if the find worked) whether packagename/*.pth exists, and then processing their contents. I don't think there's a need to have a special method for executing a package __init__, since what you'd do in the case where there are .pth but no __init__, is to simply continue the search to the end of sys.path (or the parent package __path__), and *then* create the module with an appropriate __path__. If at any point the find_module() call succeeds, then subsequent importers will just be asked for .pth files, which can then be processed into the __path__ of the now-loaded module. IOW, something like this (very rough draft): pth_contents = [] module = None for pathitem in syspath_or_parent__path__: importer = pkgutil.get_importer(pathitem) if importer is None: continue if module is None: try: loader = importer.find_module(fullname) except ImportError: pass else: # errors here should propagate module = loader.load_module(fullname) if not hasattr(module, '__path__'): # found, but not a package return module pc = get_pth_contents(importer) if pc is not None: subpath = os.path.join(pathitem, modulebasename) pth_contents.append(subpath) pth_contents.extend(pc) if '*' not in pth_contents: # got a package, but not a namespace break if pth_contents: if module is None: # No __init__, but we have paths, so make an empty package module = # new module object w/empty __path__ modify__path__(module, pth_contents) return module Obviously, the details are all in the 'get_pth_contents()', and 'modify__path__()' functions, and the above process would do extra work in the case where an individual importer implements PEP 382 on its own (although why would it?). It's also the case that this algorithm will be slow to fail imports when implemented as a meta_path hook, since it will be doing an extra pass over sys.path or the parent __path__, in addition to the one that's done by the normal __import__ machinery. (Though that's not an issue for Python 3.x, since this can be built into the core __import__). (Technically, the 3.x version should probably ask meta_path hooks for their .pth files as well, but I'm not entirely sure that that's a meaningful thing to ask.) The PEP 302 questions all boil down to how get_pth_contents() is implemented, and whether 'subpath' really should be created with os.path.join. Simply adding a get_pth_contents() method to the importer protocol (that returns None or a list of lines), and maybe a get_subpath(modulename) method that returns the path string that should be used for a subdirectory importer (i.e. __path__ entry), or None if no such subpath exists. Adding finer-grained methods is probably a waste of time, as there aren't likely to be many use cases for asking an *importer* to fetch files (vs. a loader). (In my case, of course, I'd use the pkgutil-style approach of augmenting importers or loaders that don't natively implement a needed method, that still allows third parties to register their own support for a fourth party's loader or importer type.) From pje at telecommunity.com Sun May 30 09:40:37 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 30 May 2010 03:40:37 -0400 Subject: [Python-Dev] Implementing PEP 382, Namespace Packages Message-ID: <20100530074041.2279D3A405F@sparrow.telecommunity.com> At 03:44 PM 5/29/2010 -0700, Brett Cannon wrote: >On Sat, May 29, 2010 at 12:29, "Martin v. L??wis" wrote: > > Am 29.05.2010 21:06, schrieb P.J. Eby: > >> > >> At 08:45 PM 5/29/2010 +0200, Martin v. L??wis wrote: > >>>> > >>>> In it he says that PEP 382 is being deferred until it can address PEP > >>>> 302 loaders. I can't find any follow-up to this. I don't see any > >>>> discussion in PEP 382 about PEP 302 loaders, so I assume this issue was > >>>> never resolved. Does it need to be before PEP 382 is implemented? Are we > >>>> wasting our time by designing and (eventually) coding before this issue > >>>> is resolved? > >>> > >>> Yes, and yes. > >> > >> Is there anything we can do to help regarding that? > > > > You could comment on the proposal I made back then, or propose a different > > solution. > >[sorry for the fundamental PEP questions, but I think PEP 382 came >about while I was on my python-dev sabbatical last year] > >I have some questions about the PEP which might help clarify how to >handle the API changes. > >For finders, their search algorithm is changed in a couple of ways. >One is that modules are given priority over packages (is that >intentional, Martin, or just an oversight?). Two, the package search >requires checking for a .pth file on top of an __init__.py. This will >change finders that could before simply do an existence check on an >__init__ "file" (or whatever the storage back-end happened to be) and >make it into a list-and-search which one would hope wasn't costly, but >in same cases might be if the paths to files is not stored in a >hierarchical fashion (e.g. zip files list entire files paths in their >TOC or a sqlite3 DB which uses a path for keys will have to list >**all** keys, sort them to just the relevant directory, and then look >for .pth or some such approach). Are we worried about possible >performance implications of this search? No. First, an importer would not be required to implement it in a precisely analagous way; you could have database entries or a special consolidated index in a zipfile, if you wanted to do it like that. (In practice, Python's zipimporter has a memory cache of the TOC, and a simple database index on paths would make a search for .pth's in a subdirectory trivial for the database case.) > I say no, but I just want to >make sure people we are not and people are aware about the design >shift required in finders. This entire worry would be alleviated if >only .pth files named after the package were supported, much like >*.pkg files in pkgutil. Which would completely break one of the major use cases of the PEP, which is precisely to ensure that you can install two pieces of code to the same namespace without either one overwriting the other's files. >And then the search for the __init__.py begins on the newly modified >__path__, which I assume ends with the first __init__ found on >__path__, but if no file is found it's okay and essentially an empty >module with just module-specific attributes is used? In other words, >can a .pth file replace an __init__ file in delineating a package? Yes. >Or is it purely additive? I assume the latter for compatibility reasons, Nope. The idea is specifically to allow separately installed projects to create a package without overwriting any files (causing conflicts for system installers). >but the PEP says "a directory is considered a package if it **either** >contains a file named __init__.py, **or** a file whose name ends with >".pth"" (emphasis mine). Otherwise I assume that the search will be >done simply with ``os.path.isdir(os.path.join(sys_path_entry, >top_level_package_name)`` and all existing paths will be added to >__path__. Will they come before or after the directory where the *.pth >was found? And will any subsequent *.pth files found in other >directories also be executed? > >As for how "*" works, is this limited to top-level packages, or will >sub-packages participate as well? Sub-packages as well. > I assume the former, but it is not >directly stated in the PEP. If the latter, is a dotted package name >changed to ``os.sep.join(sy_path_entry, package_name.replace('".", >os.sep)``? > >For sys.path_hooks, I am assuming import will simply skip over passing >that as it is a marker that __path__ represents a namsepace package >and not in any way functional. Although with sys.namespace_packages, >is leaving the "*" in __path__ truly necessary? I'm going to leave these to Martin to answer. >For the search of paths to use to extend, are we limiting ourselves to >actual file system entries on sys.path (as pkgutil does), pkgutil doesn't have such a limitation, except in the case extend_path, and that limitation is one that PEP 382 intends to remove. >or do we >want to support other storage back-ends? To do the latter I would >suggest having a successful path discovery be when a finder can be >created for the hypothetical directory from sys.path_hooks. The downside to that is that NullImporter is the default importer, so you'd still have to special case it. It would make more sense to add to the PEP 302 protocols directly. >I'll shut up now and stop causing trouble. =) May I suggest you take a look at the implementation draft in my other email? I realize in retrospect it doesn't handle __init__ searching in precisely the order proposed by the PEP, but I'm not sure it would be that difficult to add. (It also needs to split the operation into find/load pieces, but that's also a straightforward mod: just defer the module loading until the end, and return a wrapper around the loader that finishes the process.) From tjreedy at udel.edu Mon May 31 01:22:11 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 30 May 2010 19:22:11 -0400 Subject: [Python-Dev] Bugfix releases should not change APIs In-Reply-To: <20100529123937.222a4c0d@pitrou.net> References: <20100529123937.222a4c0d@pitrou.net> Message-ID: On 5/29/2010 6:39 AM, Antoine Pitrou wrote: > It is not the product of oversight. I am actually glad, in a sense, that it was not casual whim. ;-) I do not like the change, since it moves streams back further away from Python's sequence model, but I withdraw the request for reversion in 3.1.3. I will add further comments on the docs to the issue. > What it does teach us is that Python 3.1 sees some real use, It is an odd 'coincidence' that the method changed was one of the only two stdlib methods I have used so far used directly. But with enough users, such happens. What it teaches *me* is that before I install another release, I should, as planned, automate the running of all module tests together so I can easily test everything before and after a new installation. When I do release sample chapters and code, I will try to remember to specify the version and platform I tested with. > we have entered a phase where backwards compatibility will become as > important as it was in the 2.x line. I have assumed that there might be a few stdlib API tweeks in 3.2 -- and that they would be well announced. Terry Jan Reedy From tjreedy at udel.edu Mon May 31 01:22:55 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 30 May 2010 19:22:55 -0400 Subject: [Python-Dev] Bugfix releases should not change APIs In-Reply-To: <4C008CED.8080406@gmail.com> References: <4C008CED.8080406@gmail.com> Message-ID: On 5/28/2010 11:41 PM, Nick Coghlan wrote: > However, it may be worth modifying the policy to ensure that such > exceptional bug fixes be mentioned prominently in the release notes and > on the download page for that maintenance release. A sentence like "The behavior of it.X.truncate has been intentionally changed from ... to ... .", if I read and cognized it, would have helped me, in this case, to the problem and fix much more quickly. Is it possible with svn or hg to get a list of the commits that changed version x to version y? Would is not be possible to get a diff between at least the .rst versions of the docs for version x and version y? Terry Jan Reedy From tjreedy at udel.edu Mon May 31 01:55:31 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 30 May 2010 19:55:31 -0400 Subject: [Python-Dev] variable name resolution in exec is incorrect In-Reply-To: References: <4BFDB8EE.4010307@gmail.com> <4BFE92D9.1060306@gmail.com> Message-ID: On 5/29/2010 6:20 AM, Colin H wrote: > Perhaps the next step is to re-open the issue? If it is seen as a bug, > it would be great to see a fix in 2.6+ - For the purpose of bugfix releases, a 'bug' is a discrepancy between doc and behavior. Every new feature is seen as a 'design bug' by someone. > a number of options which > will not break backward compatibility have been put forward - cheers, Code that uses a new x.y.z feature does not work in previous x.y versions. Problems with such micro-release additions lead to the current policy. The 3.2 feature addition deadline is about 5 months away. It will probably take 1 or more people at least a couple of months to write a PEP listing the rationale for a change, the options and possible pros and cons, possibly test one or more patches, solicit opinions on which is best, select one, write new test cases and docs, and get the final patch committed. Terry Jan Reedy From brett at python.org Mon May 31 02:59:57 2010 From: brett at python.org (Brett Cannon) Date: Sun, 30 May 2010 17:59:57 -0700 Subject: [Python-Dev] Implementing PEP 382, Namespace Packages In-Reply-To: <20100529225613.2DD0A3A405F@sparrow.telecommunity.com> References: <4C011C02.4010605@trueblade.com> <4C0160D3.70304@v.loewis.de> <20100529190623.299C83A405F@sparrow.telecommunity.com> <4C016B1D.8010604@v.loewis.de> <20100529225613.2DD0A3A405F@sparrow.telecommunity.com> Message-ID: On Sat, May 29, 2010 at 15:56, P.J. Eby wrote: > At 09:29 PM 5/29/2010 +0200, Martin v. L?wis wrote: >> >> Am 29.05.2010 21:06, schrieb P.J. Eby: >>> >>> At 08:45 PM 5/29/2010 +0200, Martin v. L?wis wrote: >>>>> >>>>> In it he says that PEP 382 is being deferred until it can address PEP >>>>> 302 loaders. I can't find any follow-up to this. I don't see any >>>>> discussion in PEP 382 about PEP 302 loaders, so I assume this issue was >>>>> never resolved. Does it need to be before PEP 382 is implemented? Are >>>>> we >>>>> wasting our time by designing and (eventually) coding before this issue >>>>> is resolved? >>>> >>>> Yes, and yes. >>> >>> Is there anything we can do to help regarding that? >> >> You could comment on the proposal I made back then, or propose a different >> solution. > > Looking at that proposal, I don't follow how changing *loaders* (vs. > importers) would help. ?If an importer's find_module doesn't natively > support PEP 382, then there's no way to get a loader for the package in the > first place. ?Today, namespace packages work fine with PEP 302 loaders, > because the namespace-ness is really only about setting up the __path__, and > detecting that you need to do this in the first place. > > In the PEP 302 scheme, then, it's either importers that have to change, or > the process that invokes them. ?Being able to ask an importer the > equivalents of os.path.join, listdir, and get_data would suffice to make an > import process that could do the trick. > > Essentially, you'd ask each importer to first attempt to find the module, > and then asking it (or the loader, if the find worked) whether > packagename/*.pth exists, and then processing their contents. > > I don't think there's a need to have a special method for executing a > package __init__, since what you'd do in the case where there are .pth but > no __init__, is to simply continue the search to the end of sys.path (or the > parent package __path__), and *then* create the module with an appropriate > __path__. > > If at any point the find_module() call succeeds, then subsequent importers > will just be asked for .pth files, which can then be processed into the > __path__ of the now-loaded module. > > IOW, something like this (very rough draft): > > ? ?pth_contents = [] > ? ?module = None > > ? ?for pathitem in syspath_or_parent__path__: > > ? ? ? ?importer = pkgutil.get_importer(pathitem) > ? ? ? ?if importer is None: > ? ? ? ? ? ?continue > > ? ? ? ?if module is None: > ? ? ? ? ? ?try: > ? ? ? ? ? ? ? ?loader = importer.find_module(fullname) > ? ? ? ? ? ?except ImportError: > ? ? ? ? ? ? ? ?pass > ? ? ? ? ? ?else: > ? ? ? ? ? ? ? ?# errors here should propagate > ? ? ? ? ? ? ? ?module = loader.load_module(fullname) > ? ? ? ? ? ? ? ?if not hasattr(module, '__path__'): > ? ? ? ? ? ? ? ? ? ?# found, but not a package > ? ? ? ? ? ? ? ? ? ?return module > > ? ? ? ?pc = get_pth_contents(importer) > ? ? ? ?if pc is not None: > ? ? ? ? ? ?subpath = os.path.join(pathitem, modulebasename) > ? ? ? ? ? ?pth_contents.append(subpath) > ? ? ? ? ? ?pth_contents.extend(pc) > ? ? ? ? ? ?if '*' not in pth_contents: > ? ? ? ? ? ? ? ?# got a package, but not a namespace > ? ? ? ? ? ? ? ?break > > ? ?if pth_contents: > ? ? ? ?if module is None: > ? ? ? ? ? ?# No __init__, but we have paths, so make an empty package > ? ? ? ? ? ?module = # new module object w/empty __path__ > ? ? ? ?modify__path__(module, pth_contents) > > ? ?return module > Is it wise to modify __path__ post-import? Today people can make sure that __path__ is set to what they want before potentially reading it in their __init__ module by making the pkgutil.extend_path() call first. This would actually defer to after the import and thus not allow any __init__ code to rely on what __path__ eventually becomes. > Obviously, the details are all in the 'get_pth_contents()', and > 'modify__path__()' functions, and the above process would do extra work in > the case where an individual importer implements PEP 382 on its own > (although why would it?). > > It's also the case that this algorithm will be slow to fail imports when > implemented as a meta_path hook, since it will be doing an extra pass over > sys.path or the parent __path__, in addition to the one that's done by the > normal __import__ machinery. ?(Though that's not an issue for Python 3.x, > since this can be built into the core __import__). > > (Technically, the 3.x version should probably ask meta_path hooks for their > .pth files as well, but I'm not entirely sure that that's a meaningful thing > to ask.) > > The PEP 302 questions all boil down to how get_pth_contents() is > implemented, and whether 'subpath' really should be created with > os.path.join. ?Simply adding a get_pth_contents() method to the importer > protocol (that returns None or a list of lines), and maybe a > get_subpath(modulename) method that returns the path string that should be > used for a subdirectory importer (i.e. __path__ entry), or None if no such > subpath exists. Code already out there uses os.path.join() to extend __path__ (e.g. Django), so I would stick with that unless we want to start transitioning to '/' only. From brett at python.org Mon May 31 03:18:07 2010 From: brett at python.org (Brett Cannon) Date: Sun, 30 May 2010 18:18:07 -0700 Subject: [Python-Dev] Implementing PEP 382, Namespace Packages In-Reply-To: <20100530074041.2279D3A405F@sparrow.telecommunity.com> References: <20100530074041.2279D3A405F@sparrow.telecommunity.com> Message-ID: On Sun, May 30, 2010 at 00:40, P.J. Eby wrote: > At 03:44 PM 5/29/2010 -0700, Brett Cannon wrote: >> >> On Sat, May 29, 2010 at 12:29, "Martin v. L?wis" >> wrote: >> > Am 29.05.2010 21:06, schrieb P.J. Eby: >> >> >> >> At 08:45 PM 5/29/2010 +0200, Martin v. L?wis wrote: >> >>>> >> >>>> In it he says that PEP 382 is being deferred until it can address PEP >> >>>> 302 loaders. I can't find any follow-up to this. I don't see any >> >>>> discussion in PEP 382 about PEP 302 loaders, so I assume this issue >> >>>> was >> >>>> never resolved. Does it need to be before PEP 382 is implemented? Are >> >>>> we >> >>>> wasting our time by designing and (eventually) coding before this >> >>>> issue >> >>>> is resolved? >> >>> >> >>> Yes, and yes. >> >> >> >> Is there anything we can do to help regarding that? >> > >> > You could comment on the proposal I made back then, or propose a >> > different >> > solution. >> >> [sorry for the fundamental PEP questions, but I think PEP 382 came >> about while I was on my python-dev sabbatical last year] >> >> I have some questions about the PEP which might help clarify how to >> handle the API changes. >> >> For finders, their search algorithm is changed in a couple of ways. >> One is that modules are given priority over packages (is that >> intentional, Martin, or just an oversight?). Two, the package search >> requires checking for a .pth file on top of an __init__.py. This will >> change finders that could before simply do an existence check on an >> __init__ "file" (or whatever the storage back-end happened to be) and >> make it into a list-and-search which one would hope wasn't costly, but >> in same cases might be if the paths to files is not stored in a >> hierarchical fashion (e.g. zip files list entire files paths in their >> TOC or a sqlite3 DB which uses a path for keys will have to list >> **all** keys, sort them to just the relevant directory, and then look >> for .pth or some such approach). Are we worried about possible >> performance implications of this search? > > No. ?First, an importer would not be required to implement it in a precisely > analagous way; you could have database entries or a special consolidated > index in a zipfile, if you wanted to do it like that. ?(In practice, > Python's zipimporter has a memory cache of the TOC, and a simple database > index on paths would make a search for .pth's in a subdirectory trivial for > the database case.) Sure, for the two examples this works, but who knows about other odd back-ends people might be using. Granted, this is all hypothetical and why I figured we wouldn't worry about it. > > >> ?I say no, but I just want to >> make sure people we are not and people are aware about the design >> shift required in finders. This entire worry would be alleviated if >> only .pth files named after the package were supported, much like >> *.pkg files in pkgutil. > > Which would completely break one of the major use cases of the PEP, which is > precisely to ensure that you can install two pieces of code to the same > namespace without either one overwriting the other's files. The PEP says the goal is to span packages across directories. If you split something like zope into multiple directories, does having a separate zope.pth file in each of those directories really cause a problem here? You are not importing them so it isn't like you are worrying about precedence. If you specify that all .pth files found are run then using the same file name in all package directories isn't an issue. But I guess packages that do this want to keep unique files per directory separation that they support and not have to fix the file names at distribution time. > > >> And then the search for the __init__.py begins on the newly modified >> __path__, which I assume ends with the first __init__ found on >> __path__, but if no file is found it's okay and essentially an empty >> module with just module-specific attributes is used? In other words, >> can a .pth file replace an __init__ file in delineating a package? > > Yes. > > >> Or is it purely additive? I assume the latter for compatibility reasons, > > Nope. ?The idea is specifically to allow separately installed projects to > create a package without overwriting any files (causing conflicts for system > installers). > > >> but the PEP says "a directory is considered a package if it **either** >> contains a file named __init__.py, **or** a file whose name ends with >> ".pth"" (emphasis mine). Otherwise I assume that the search will be >> done simply with ``os.path.isdir(os.path.join(sys_path_entry, >> top_level_package_name)`` and all existing paths will be added to >> __path__. Will they come before or after the directory where the *.pth >> was found? And will any subsequent *.pth files found in other >> directories also be executed? >> >> As for how "*" works, is this limited to top-level packages, or will >> sub-packages participate as well? > > Sub-packages as well. > > >> ?I assume the former, but it is not >> directly stated in the PEP. If the latter, is a dotted package name >> changed to ``os.sep.join(sy_path_entry, package_name.replace('".", >> os.sep)``? >> >> For sys.path_hooks, I am assuming import will simply skip over passing >> that as it is a marker that __path__ represents a namsepace package >> and not in any way functional. Although with sys.namespace_packages, >> is leaving the "*" in __path__ truly necessary? > > I'm going to leave these to Martin to answer. > > >> For the search of paths to use to extend, are we limiting ourselves to >> actual file system entries on sys.path (as pkgutil does), > > pkgutil doesn't have such a limitation, except in the case extend_path, and > that limitation is one that PEP 382 intends to remove. It's because pkgutil.extend_path has that limitation I am asking as that's what the PEP refers to. If the PEP wants to remove the limitation it should clearly state how it is going to do that. > > >> or do we >> want to support other storage back-ends? To do the latter I would >> suggest having a successful path discovery be when a finder can be >> created for the hypothetical directory from sys.path_hooks. > > The downside to that is that NullImporter is the default importer, so you'd > still have to special case it. ?It would make more sense to add to the PEP > 302 protocols directly. But import is the one that adds NullImporter. And if import is the one figuring out what paths do and don't work, then it doesn't matter about NullImporter as it will know what does and does not fail. And as you said, you can special-case it. As for adding to the PEP 302 protocols, it's a question of how much we want importer implementors to have control over this versus us. I personally would rather keep any protocol extensions simple and have import handle as many of the details as possible. I think the PEP 3147 has shown the benefits of letting import details be under our control as much as possible can be beneficial as it doesn't put pre-existing importers at a disadvantage. > > >> I'll shut up now and stop causing trouble. =) > > May I suggest you take a look at the implementation draft in my other email? > ?I realize in retrospect it doesn't handle __init__ searching in precisely > the order proposed by the PEP, but I'm not sure it would be that difficult > to add. ?(It also needs to split the operation into find/load pieces, but > that's also a straightforward mod: just defer the module loading until the > end, and return a wrapper around the loader that finishes the process.) I replied separately to that email. From pje at telecommunity.com Mon May 31 06:50:39 2010 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 31 May 2010 00:50:39 -0400 Subject: [Python-Dev] Implementing PEP 382, Namespace Packages In-Reply-To: References: <4C011C02.4010605@trueblade.com> <4C0160D3.70304@v.loewis.de> <20100529190623.299C83A405F@sparrow.telecommunity.com> <4C016B1D.8010604@v.loewis.de> <20100529225613.2DD0A3A405F@sparrow.telecommunity.com> Message-ID: <20100531045042.2A3DC3A402D@sparrow.telecommunity.com> At 05:59 PM 5/30/2010 -0700, Brett Cannon wrote: >Is it wise to modify __path__ post-import? Today people can make sure >that __path__ is set to what they want before potentially reading it >in their __init__ module by making the pkgutil.extend_path() call >first. This would actually defer to after the import and thus not >allow any __init__ code to rely on what __path__ eventually becomes. Well, that's what the other lines in the .pth files are for. Keep in mind that only *one* project can contain the namespace package's __init__ module, so it's only sane for that __init__ to import things that are bundled with the __init__ module. AFAIK, most uses of namespace packages today are via setuptools' API, which doesn't support having a non-empty __init__.py at all (apart from the namespace declaration), so this limitation is unlikely to cause problems in practice. When the code I gave is refactored into a proper importer/loader pair, it can actually be structured such that the full __path__ is set *before* the low-level loader is called; however, if the loader itself chooses to overwrite __path__ at that point, there's little that can be done about it. In the Python 3.x case, the loader protocol could be revised to require only *adding* a non-duplicate entry to __path__ if it's present, and the stdlib loaders updated accordingly. For my backport, OTOH, I'd have to do some sort of workaround to wrap the regular importers, so I'd just as soon leave it undefined by PEP 382 what an __init__ module sees in __path__ during its execution. (And for a backport whose sole purpose is to cut down on setuptools' funky .pth manipulations, that might suffice anyway.) From pje at telecommunity.com Mon May 31 07:03:26 2010 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 31 May 2010 01:03:26 -0400 Subject: [Python-Dev] Implementing PEP 382, Namespace Packages In-Reply-To: References: <20100530074041.2279D3A405F@sparrow.telecommunity.com> Message-ID: <20100531050328.40B073A402D@sparrow.telecommunity.com> At 06:18 PM 5/30/2010 -0700, Brett Cannon wrote: >On Sun, May 30, 2010 at 00:40, P.J. Eby wrote: > > > > Which would completely break one of the major use cases of the > PEP, which is > > precisely to ensure that you can install two pieces of code to the same > > namespace without either one overwriting the other's files. > >The PEP says the goal is to span packages across directories. The goal of namespace packages is to allow separately-distributed pieces of code to live in the same package namespace. That this is sometimes achieved by installing them to different paths is an implementation detail. In the case of e.g. Linux distributions and other system packaging scenarios, the code will all be installed to the *same* directory -- so there cannot be any filename collisions among the separately-distributed modules. That's why we want to get rid of the need for an __init__.py to mark the directory as a package: it's a collision point for system package management tools. > > pkgutil doesn't have such a limitation, except in the case extend_path, and > > that limitation is one that PEP 382 intends to remove. > >It's because pkgutil.extend_path has that limitation I am asking as >that's what the PEP refers to. If the PEP wants to remove the >limitation it should clearly state how it is going to do that. I'm flexible on it either way. The only other importer I know of that does anything else is one that actually allows (unsafely) importing from URLs. If we allow for other things, then we need to extend the PEP 302 protocol to have a way to ask an importer for a subpath string. >As for adding to the PEP 302 protocols, it's a question of how much we >want importer implementors to have control over this versus us. I >personally would rather keep any protocol extensions simple and have >import handle as many of the details as possible. I lean the other way a bit, in that the more of the importer internals you expose, the harder you make it for an importer to be anything other than a mere virtual file system. (As it is, I think there is too much "file-ness" coupling in the protocol already, what with file extensions and the like.) Indeed, now that I'm thinking about it, it actually seems to make more sense to just require the importers to implement PEP 382, and provide some common machinery in imp or pkgutil for reading .pth strings, setting up __path__, and hunting down all the other directories. From martin at v.loewis.de Mon May 31 09:24:22 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 31 May 2010 09:24:22 +0200 Subject: [Python-Dev] Implementing PEP 382, Namespace Packages In-Reply-To: <20100529225613.2DD0A3A405F@sparrow.telecommunity.com> References: <4C011C02.4010605@trueblade.com> <4C0160D3.70304@v.loewis.de> <20100529190623.299C83A405F@sparrow.telecommunity.com> <4C016B1D.8010604@v.loewis.de> <20100529225613.2DD0A3A405F@sparrow.telecommunity.com> Message-ID: <4C036426.5050907@v.loewis.de> > Looking at that proposal, I don't follow how changing *loaders* (vs. > importers) would help. If an importer's find_module doesn't natively > support PEP 382, then there's no way to get a loader for the package in > the first place. True. However, this requires no changes to the API, AFAICT. The *finder* (there are no importers, right?) will need to accept a folder with just a pth file as a package, but then still return a loader. The issue is then how to make the loader scan the folder for .pth files. One option would be to ask it "give me the contents of all the pth files", the other would be to have a method "load all the pth files". > In the PEP 302 scheme, then, it's either importers that have to change, > or the process that invokes them. Being able to ask an importer the > equivalents of os.path.join, listdir, and get_data would suffice to make > an import process that could do the trick. That would also work, but it would make a fairly wide interface. IIRC, MAL complained that doing so would break importers which can't do listdir. > Essentially, you'd ask each importer to first attempt to find the > module, and then asking it (or the loader, if the find worked) whether > packagename/*.pth exists, and then processing their contents. The latter is my proposal, yes: ask the loader to process all pth files. > If at any point the find_module() call succeeds, then subsequent > importers will just be asked for .pth files, which can then be processed > into the __path__ of the now-loaded module. > > IOW, something like this (very rough draft): > pth_contents = [] > module = None > > for pathitem in syspath_or_parent__path__: > > importer = pkgutil.get_importer(pathitem) > if importer is None: > continue > > if module is None: > try: > loader = importer.find_module(fullname) > except ImportError: > pass Is this really how it works today? Shouldn't it abort here if there is an ImportError? > else: > # errors here should propagate > module = loader.load_module(fullname) > if not hasattr(module, '__path__'): > # found, but not a package > return module > > pc = get_pth_contents(importer) Assuming we always get here with a loader, I'd rather call this on the loader. Regards, Martin From martin at v.loewis.de Mon May 31 09:53:19 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 31 May 2010 09:53:19 +0200 Subject: [Python-Dev] Implementing PEP 382, Namespace Packages In-Reply-To: References: <4C011C02.4010605@trueblade.com> <4C0160D3.70304@v.loewis.de> <20100529190623.299C83A405F@sparrow.telecommunity.com> <4C016B1D.8010604@v.loewis.de> Message-ID: <4C036AEF.6040504@v.loewis.de> > For finders, their search algorithm is changed in a couple of ways. > One is that modules are given priority over packages (is that > intentional, Martin, or just an oversight?). That's an oversight. Notice, however, that it's really not the case that currently directories have precedence over modules, either: if a directory is later on the __path__ than a module, it's still the module that gets imported. So the precedence takes place only when a module and a directory exist in the same directory. In any case, I have now fixed it. > Two, the package search > requires checking for a .pth file on top of an __init__.py. This will > change finders that could before simply do an existence check on an > __init__ "file" You are reading something into the PEP that isn't there yet. PEP 302 currently isn't considered, and the question of this discussion is precisely how the loaders API should be changed. > (or whatever the storage back-end happened to be) and > make it into a list-and-search which one would hope wasn't costly, but > in same cases might be if the paths to files is not stored in a > hierarchical fashion (e.g. zip files list entire files paths in their > TOC or a sqlite3 DB which uses a path for keys will have to list > **all** keys, sort them to just the relevant directory, and then look > for .pth or some such approach). First, I think it's up to the specific loader mechanism whether PEP 382 should be supported at all. It should be possible to implement it if desired, but if it's not feasible (e.g. for URL loaders), pth files just may not get considered. The loader may well provide a different mechanism to support namespace packages. > Are we worried about possible > performance implications of this search? For the specific case of zip files, I'm not. I don't think performance will suffer at all. > And then the search for the __init__.py begins on the newly modified > __path__, which I assume ends with the first __init__ found on > __path__, but if no file is found it's okay and essentially an empty > module with just module-specific attributes is used? Correct. > In other words, > can a .pth file replace an __init__ file in delineating a package? That's what it means by '''a directory is considered a package if it either contains a file named __init__.py, or a file whose name ends with ".pth".''' > Or > is it purely additive? I assume the latter for compatibility reasons, > but the PEP says "a directory is considered a package if it **either** > contains a file named __init__.py, **or** a file whose name ends with > ".pth"" (emphasis mine). Why do you think this causes an incompatibility? > Otherwise I assume that the search will be > done simply with ``os.path.isdir(os.path.join(sys_path_entry, > top_level_package_name)`` and all existing paths will be added to > __path__. Will they come before or after the directory where the *.pth > was found? And will any subsequent *.pth files found in other > directories also be executed? I may misremember, but from reading the text, it seems to say "no". It should work like the current pth mechanism (plus *, minus import). > As for how "*" works, is this limited to top-level packages, or will > sub-packages participate as well? I assume the former, but it is not > directly stated in the PEP. And indeed, the latter is intended. You should be able to create namespace packages on all levels. > If the latter, is a dotted package name > changed to ``os.sep.join(sy_path_entry, package_name.replace('".", > os.sep)``? No. Instead, the parent package's __path__ is being searched for directories; sys.path is not considered anymore. I have fixed the text. > For sys.path_hooks, I am assuming import will simply skip over passing > that as it is a marker that __path__ represents a namsepace package > and not in any way functional. Although with sys.namespace_packages, > is leaving the "*" in __path__ truly necessary? It would help with efficiency, no? > For the search of paths to use to extend, are we limiting ourselves to > actual file system entries on sys.path (as pkgutil does), or do we > want to support other storage back-ends? To do the latter I would > suggest having a successful path discovery be when a finder can be > created for the hypothetical directory from sys.path_hooks. Again: PEP 302 isn't really considered yet. Proposals are welcome. > The PEP (seems to) ask finders to look for a .pth file(s), calculate > __path__, and then get a loader for the __init__. You could have > finders grow a find_namespace method which returns the contents of the > requisite .pth file(s). I must be misunderstanding the concept of finders. Why is it that it would be their function to process the pth files, and not the function of the loader? Regards, Martin From martin at v.loewis.de Mon May 31 09:59:40 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 31 May 2010 09:59:40 +0200 Subject: [Python-Dev] Implementing PEP 382, Namespace Packages In-Reply-To: References: <20100530074041.2279D3A405F@sparrow.telecommunity.com> Message-ID: <4C036C6C.2080606@v.loewis.de> > The PEP says the goal is to span packages across directories. If you > split something like zope into multiple directories, does having a > separate zope.pth file in each of those directories really cause a > problem here? I think pje already answered this: yes, you do split zope into multiple packages. But then, you may install all of them into the same folder. This has caused pain for Linux package management tools in particular, because they dislike if to packages install the same file. So if you can arrange the pth files to have non-overlapping names, you can still install them into the same directory, and dpkg is happy. Regards, Martin From martin at v.loewis.de Mon May 31 10:03:00 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 31 May 2010 10:03:00 +0200 Subject: [Python-Dev] Bugfix releases should not change APIs In-Reply-To: References: <4C008CED.8080406@gmail.com> Message-ID: <4C036D34.7080401@v.loewis.de> > Is it possible with svn or hg to get a list of the commits that changed > version x to version y? A regular "svn log" on the maintenance branch will give you all the changes. You'll recognize from the checkin messages when the previous release was. > Would is not be possible to get a diff between at least the .rst > versions of the docs for version x and version y? That's most certainly possible, using "svn diff -rX:Y". Regards, Martin From pje at telecommunity.com Mon May 31 16:51:45 2010 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 31 May 2010 10:51:45 -0400 Subject: [Python-Dev] Implementing PEP 382, Namespace Packages Message-ID: <20100531145147.9BEF83A402D@sparrow.telecommunity.com> At 09:24 AM 5/31/2010 +0200, Martin v. L?wis wrote: >Is this really how it works today? Shouldn't it abort here >if there is an ImportError? Oops. You're right - I was confusing find_module with the path_hooks protoocol. > > else: > > # errors here should propagate > > module = loader.load_module(fullname) > > if not hasattr(module, '__path__'): > > # found, but not a package > > return module > > > > pc = get_pth_contents(importer) > >Assuming we always get here with a loader, I'd rather call this >on the loader. We don't, unless the finder natively supports PEP 382. From smarv at gmx.net Mon May 31 20:45:22 2010 From: smarv at gmx.net (smarv at gmx.net) Date: Mon, 31 May 2010 20:45:22 +0200 Subject: [Python-Dev] tp_dealloc Message-ID: <20100531184522.173170@gmx.net> Hi, when embedding python 3.1, I have my own free-method in tp_dealloc. The allocated memory is in host-memory, not in python (dll). Now, the problem is, Python appears to read-access the deallocated memory still after tp_dealloc. After tp_dealloc, I get an access violation if the pyobject-header fields have been modified inside tp_dealloc. If I leave the header unmodified, then no access violation occurs. Accessing deallocated memory sounds like a bug, or is this intended design? Thank you Marvin -- GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 From brett at python.org Mon May 31 22:10:45 2010 From: brett at python.org (Brett Cannon) Date: Mon, 31 May 2010 13:10:45 -0700 Subject: [Python-Dev] Implementing PEP 382, Namespace Packages In-Reply-To: <4C036AEF.6040504@v.loewis.de> References: <4C011C02.4010605@trueblade.com> <4C0160D3.70304@v.loewis.de> <20100529190623.299C83A405F@sparrow.telecommunity.com> <4C016B1D.8010604@v.loewis.de> <4C036AEF.6040504@v.loewis.de> Message-ID: On Mon, May 31, 2010 at 00:53, "Martin v. L?wis" wrote: >> For finders, their search algorithm is changed in a couple of ways. >> One is that modules are given priority over packages (is that >> intentional, Martin, or just an oversight?). > > That's an oversight. Notice, however, that it's really not the case that > currently directories have precedence over modules, either: if a directory > is later on the __path__ than a module, it's still the module that gets > imported. So the precedence takes place only when a module and a directory > exist in the same directory. > > In any case, I have now fixed it. > >> Two, the package search >> requires checking for a .pth file on top of an __init__.py. This will >> change finders that could before simply do an existence check on an >> __init__ "file" > > You are reading something into the PEP that isn't there yet. PEP 302 > currently isn't considered, and the question of this discussion is precisely > how the loaders API should be changed. > >> (or whatever the storage back-end happened to be) and >> make it into a list-and-search which one would hope wasn't costly, but >> in same cases might be if the paths to files is not stored in a >> hierarchical fashion (e.g. zip files list entire files paths in their >> TOC or a sqlite3 DB which uses a path for keys will have to list >> **all** keys, sort them to just the relevant directory, and then look >> for .pth or some such approach). > > First, I think it's up to the specific loader mechanism whether > PEP 382 should be supported at all. It should be possible to implement > it if desired, but if it's not feasible (e.g. for URL loaders), pth > files just may not get considered. The loader may well provide a > different mechanism to support namespace packages. > >> Are we worried about possible >> >> performance implications of this search? > > For the specific case of zip files, I'm not. I don't think performance will > suffer at all. > >> And then the search for the __init__.py begins on the newly modified >> __path__, which I assume ends with the first __init__ found on >> __path__, but if no file is found it's okay and essentially an empty >> module with just module-specific attributes is used? > > Correct. > >> In other words, >> can a .pth file replace an __init__ file in delineating a package? > > That's what it means by '''a directory is considered a package if it either > contains a file named __init__.py, or a file whose name ends with ".pth".''' > >> Or >> is it purely additive? I assume the latter for compatibility reasons, >> but the PEP says "a directory is considered a package if it **either** >> contains a file named __init__.py, **or** a file whose name ends with >> ".pth"" (emphasis mine). > > Why do you think this causes an incompatibility? It's just if a project has no __init__ older finders won't process it, that's all. But it looks like they are going to have to change somewhat anyway so that's not an issue. > >> Otherwise I assume that the search will be >> done simply with ``os.path.isdir(os.path.join(sys_path_entry, >> top_level_package_name)`` and all existing paths will be added to >> __path__. Will they come before or after the directory where the *.pth >> was found? And will any subsequent *.pth files found in other >> directories also be executed? > > I may misremember, but from reading the text, it seems to say "no". > It should work like the current pth mechanism (plus *, minus import). > >> As for how "*" works, is this limited to top-level packages, or will >> sub-packages participate as well? I assume the former, but it is not >> directly stated in the PEP. > > And indeed, the latter is intended. You should be able to create namespace > packages on all levels. > >> If the latter, is a dotted package name >> changed to ``os.sep.join(sy_path_entry, package_name.replace('".", >> os.sep)``? > > No. Instead, the parent package's __path__ is being searched for > directories; sys.path is not considered anymore. I have fixed the text. > >> For sys.path_hooks, I am assuming import will simply skip over passing >> that as it is a marker that __path__ represents a namsepace package >> and not in any way functional. Although with sys.namespace_packages, >> is leaving the "*" in __path__ truly necessary? > > It would help with efficiency, no? Not sure how having "*" in __path__ helps with efficiency. What are you thinking it will help with specifically? > >> For the search of paths to use to extend, are we limiting ourselves to >> actual file system entries on sys.path (as pkgutil does), or do we >> want to support other storage back-ends? To do the latter I would >> suggest having a successful path discovery be when a finder can be >> created for the hypothetical directory from sys.path_hooks. > > Again: PEP 302 isn't really considered yet. Proposals are welcome. > >> The PEP (seems to) ask finders to look for a .pth file(s), calculate >> __path__, and then get a loader for the __init__. You could have >> finders grow a find_namespace method which returns the contents of the >> requisite .pth file(s). > > I must be misunderstanding the concept of finders. Why is it that it would > be their function to process the pth files, and not the function of the > loader? I'm thinking from the perspective of finding an __init__ module that exists somewhere else than where the .pth file was discovered. Someone has to find the .pth files and provide their contents. Someone else needs to find the __init__ module for the package (if it exists). Then someone needs to load a namespace package, potentially from the __init__ module. It's that second step -- find the __init__ module -- that makes me think the finder is more involved. It doesn't have to be by definition, but seeing the word "find" just makes me think "finder". From brett at python.org Mon May 31 22:19:11 2010 From: brett at python.org (Brett Cannon) Date: Mon, 31 May 2010 13:19:11 -0700 Subject: [Python-Dev] Implementing PEP 382, Namespace Packages In-Reply-To: <20100531050328.40B073A402D@sparrow.telecommunity.com> References: <20100530074041.2279D3A405F@sparrow.telecommunity.com> <20100531050328.40B073A402D@sparrow.telecommunity.com> Message-ID: On Sun, May 30, 2010 at 22:03, P.J. Eby wrote: > At 06:18 PM 5/30/2010 -0700, Brett Cannon wrote: >> >> On Sun, May 30, 2010 at 00:40, P.J. Eby wrote: >> > >> > Which would completely break one of the major use cases of the PEP, >> > which is >> > precisely to ensure that you can install two pieces of code to the same >> > namespace without either one overwriting the other's files. >> >> The PEP says the goal is to span packages across directories. > > The goal of namespace packages is to allow separately-distributed pieces of > code to live in the same package namespace. ?That this is sometimes achieved > by installing them to different paths is an implementation detail. > > In the case of e.g. Linux distributions and other system packaging > scenarios, the code will all be installed to the *same* directory -- so > there cannot be any filename collisions among the separately-distributed > modules. ?That's why we want to get rid of the need for an __init__.py to > mark the directory as a package: it's a collision point for system package > management tools. > > >> > pkgutil doesn't have such a limitation, except in the case extend_path, >> > and >> > that limitation is one that PEP 382 intends to remove. >> >> It's because pkgutil.extend_path has that limitation I am asking as >> that's what the PEP refers to. If the PEP wants to remove the >> limitation it should clearly state how it is going to do that. > > I'm flexible on it either way. ?The only other importer I know of that does > anything else is one that actually allows (unsafely) importing from URLs. > > If we allow for other things, then we need to extend the PEP 302 protocol to > have a way to ask an importer for a subpath string. > > > >> As for adding to the PEP 302 protocols, it's a question of how much we >> want importer implementors to have control over this versus us. I >> personally would rather keep any protocol extensions simple and have >> import handle as many of the details as possible. > > I lean the other way a bit, in that the more of the importer internals you > expose, the harder you make it for an importer to be anything other than a > mere virtual file system. ?(As it is, I think there is too much "file-ness" > coupling in the protocol already, what with file extensions and the like.) > Yes, there is a VERY strong file path connection in loaders and they have essentially become simple virtual file systems. But that does make implementation of alternative back-ends easier which seems to be the common case. Plus pre-existing code already mutates __path__, __file__, etc. as if they are file paths using os.path.join. And imp's new source_from_cache and cache_from_source are not weakening the tie to file paths either. But as long as whatever mechanism gets exposed allows people to work from a module name that will be enough. The path connection is not required as load_module is the end-all-be-all method. If we have a similar API added for .pth files that works off of module names then those loaders that don't want to work from file paths don't have to.