From pje at telecommunity.com Fri Dec 1 00:34:38 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 30 Nov 2006 18:34:38 -0500 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <6e9196d20611301446y629c04cdn6c8215dfe065d006@mail.gmail.co m> References: <20061130032036.11053.1356768333.divmod.xquotient.888@joule.divmod.com> <20061130032036.11053.1356768333.divmod.xquotient.888@joule.divmod.com> Message-ID: <5.1.1.6.0.20061130182946.028b4850@sparrow.telecommunity.com> At 02:46 PM 11/30/2006 -0800, Mike Orr wrote: >Speaking of Virtual Python [1], I've heard some people recommending it >as a general solution to the "this library breaks that other >application" problem and "this app needs a different version of X >library than that other app does". It was actually created to help people who don't have root access (e.g. in shared web hosting accounts) get their "own" Python. It does work for other things, but I haven't been especially recommending it for anything besides that, since eggs take care of multi-version app/library support quite nicely. Indeeed, the virtual-python approach is itself unnecessary now, as easy_install has much better PYTHONPATH support now, than when virtual-python was created. From lists at janc.be Fri Dec 1 00:42:42 2006 From: lists at janc.be (Jan Claeys) Date: Fri, 01 Dec 2006 00:42:42 +0100 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <456F51C5.8080500@holdenweb.com> References: <5.1.1.6.0.20061126143124.027f3e68@sparrow.telecommunity.com> <5.1.1.6.0.20061127090652.04341cf8@sparrow.telecommunity.com> <200611290053.17199.anthony@interlink.com.au> <20061129112354.GA30665@code0.codespeak.net> <1164917149.31269.344.camel@localhost> <456F51C5.8080500@holdenweb.com> Message-ID: <1164930163.31269.407.camel@localhost> Op donderdag 30-11-2006 om 21:48 uur [tijdzone +0000], schreef Steve Holden: > I think the point is that some distros (Debian is the one that springs > to mind most readily, but I'm not a distro archivist) require a separate > install for distutils even though it's been a part of the standard > *Python* distro since 2.3 (2.2?) > > So, it isn't that you can't get distutils, it's that you have to take an > extra step over and above installing Python. No, it just means that several parts of the python.org source package are spread over several binary packages, just like happens with hundreds or thousands of other packages, and any Debian (or Ubuntu or other distro doing this) administrator worth his or her money should be aware of that, and be able to find those packages. E.g. on a debian "sarge" system: $ apt-cache showsrc python2.4 | grep Binary Binary: python2.4-doc, python2.4, python2.4-examples, python2.4-tk, python2.4-dev, idle-python2.4, python2.4-dbg, python2.4-gdbm Or on an Ubuntu "edgy" system: $ apt-cache showsrc python2.4 | grep Binary Binary: python2.4-dbg, python2.4-dev, python2.4, python2.4-doc, idle-python2.4, python2.4-minimal, python2.4-examples Probably the Debian maintainers could have named packages differently to make things less confusing for newbies (e.g. by having the 'pythonX.Y' packages being meta-packages that depend on all binary packages built from the upstream source package), but that doesn't mean splitting "python" (or other projects) up in several packages is wrong. E.g. when installing on an flash drive, people are probably quite happy to leave the 20 MiB of Python documentation out... Maybe python.org can include several logical "divisions" in the python.org distribution and make it easy for OS distro packagers to make separate packages if they want to, as most of them are quite happy to have less work to do, provided the upstream "divisions" do more or less what they want. ;-) (Oh, and such a division should IMHO also include a "minimal python" for embedded/low-resource hardware use, where things like distutils, GUI toolkits, a colelction of 20 XML libraries and documentation are most likely not needed.) -- Jan Claeys From greg.ewing at canterbury.ac.nz Fri Dec 1 00:44:26 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 01 Dec 2006 12:44:26 +1300 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <671B99D2-3174-4A5E-B12B-D2241EA22959@python.org> References: <20061129101826.11053.667681482.divmod.xquotient.770@joule.divmod.com> <78EBB5BE-1F47-4BA3-B22A-1966C949E25D@python.org> <456E26FB.2020502@canterbury.ac.nz> <456EED77.5060009@acm.org> <671B99D2-3174-4A5E-B12B-D2241EA22959@python.org> Message-ID: <456F6CDA.8070608@canterbury.ac.nz> Barry Warsaw wrote: > When I switched to OS X for most of my desktops, I had several > collisions in this namespace. I think on MacOSX you have to consider that it's really ~/Documents and the like that are *your* namespaces, rather than the top level of your home directory. Also, I think MacOSX has a somewhat different philosophy about hiding things. The Finder hides the internals of things like application bundles, which is fine, but the application itself is visible, so you can move it around or delete it if you want. With ~/.local, you're hiding the fact that the applications or libraries or whatever are even there in the first place. You've got all this disk space being used up, but no way of seeing where or by what, and no obvious way of freeing it up. I think that's bad HCI. -- Greg From lists at janc.be Fri Dec 1 01:00:15 2006 From: lists at janc.be (Jan Claeys) Date: Fri, 01 Dec 2006 01:00:15 +0100 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <456F6CDA.8070608@canterbury.ac.nz> References: <20061129101826.11053.667681482.divmod.xquotient.770@joule.divmod.com> <78EBB5BE-1F47-4BA3-B22A-1966C949E25D@python.org> <456E26FB.2020502@canterbury.ac.nz> <456EED77.5060009@acm.org> <671B99D2-3174-4A5E-B12B-D2241EA22959@python.org> <456F6CDA.8070608@canterbury.ac.nz> Message-ID: <1164931215.31269.414.camel@localhost> Op vrijdag 01-12-2006 om 12:44 uur [tijdzone +1300], schreef Greg Ewing: > With ~/.local, you're hiding the fact that the applications > or libraries or whatever are even there in the first > place. You've got all this disk space being used up, > but no way of seeing where or by what, and no > obvious way of freeing it up. I think that's bad HCI. AFAIK fd.o's use of ~/.local is mainly for things like *.desktop files (e.g. menu items added or changed by the user). -- Jan Claeys From steve at holdenweb.com Fri Dec 1 01:16:59 2006 From: steve at holdenweb.com (Steve Holden) Date: Fri, 01 Dec 2006 00:16:59 +0000 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <1164930163.31269.407.camel@localhost> References: <5.1.1.6.0.20061126143124.027f3e68@sparrow.telecommunity.com> <5.1.1.6.0.20061127090652.04341cf8@sparrow.telecommunity.com> <200611290053.17199.anthony@interlink.com.au> <20061129112354.GA30665@code0.codespeak.net> <1164917149.31269.344.camel@localhost> <456F51C5.8080500@holdenweb.com> <1164930163.31269.407.camel@localhost> Message-ID: <456F747B.9060107@holdenweb.com> Jan Claeys wrote: [...] > Probably the Debian maintainers could have named packages differently to > make things less confusing for newbies (e.g. by having the 'pythonX.Y' > packages being meta-packages that depend on all binary packages built > from the upstream source package), but that doesn't mean splitting > "python" (or other projects) up in several packages is wrong. E.g. when > installing on an flash drive, people are probably quite happy to leave > the 20 MiB of Python documentation out... > Right, who cares about newbies, they're only the future of the language, after all. I take your point that some flexibility is advantageous once you get past the newbie stage, but I think that here we are talking about trying to avoid mis-steps that will potentially put people off making that transition. > > Maybe python.org can include several logical "divisions" in the > python.org distribution and make it easy for OS distro packagers to make > separate packages if they want to, as most of them are quite happy to > have less work to do, provided the upstream "divisions" do more or less > what they want. ;-) (Oh, and such a division should IMHO also include > a "minimal python" for embedded/low-resource hardware use, where things > like distutils, GUI toolkits, a colelction of 20 XML libraries and > documentation are most likely not needed.) > > If only there were some guarantee that the distros would respect any project partitioning imposed by python-deb we might stand a chance of resolving these issues. By and large they do tend to go their own way, though. I suppose the only alternative is prominently-posted materials on python.org about "Python on Debian", "Python on Ubuntu", ... and various addition to the FAQs. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden From barry at python.org Fri Dec 1 01:39:17 2006 From: barry at python.org (Barry Warsaw) Date: Thu, 30 Nov 2006 19:39:17 -0500 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <456F6CDA.8070608@canterbury.ac.nz> References: <20061129101826.11053.667681482.divmod.xquotient.770@joule.divmod.com> <78EBB5BE-1F47-4BA3-B22A-1966C949E25D@python.org> <456E26FB.2020502@canterbury.ac.nz> <456EED77.5060009@acm.org> <671B99D2-3174-4A5E-B12B-D2241EA22959@python.org> <456F6CDA.8070608@canterbury.ac.nz> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Nov 30, 2006, at 6:44 PM, Greg Ewing wrote: > Barry Warsaw wrote: > >> When I switched to OS X for most of my desktops, I had several >> collisions in this namespace. > > I think on MacOSX you have to consider that it's really > ~/Documents and the like that are *your* namespaces, > rather than the top level of your home directory. That's not how a *nix developer is going to think, but OSX does straddle the line between my grandma's OS and a Real Development Platform. It usually does a pretty good job of letting the *nix geeks pop the hood and play in the sandbox they're used to (to mix some metaphors :), but letting you ignore all that stuff if you want, like when I'm just using it to write some music. I actually think it does a pretty good job here too, or at least as good as can be expected. But it does create collisions where I've generally not had them on e.g. Linux or Solaris. Maybe I'd see more of that if e.g. I actually tried to use a Linux desktop in grandma- mode too. > Also, I think MacOSX has a somewhat different philosophy > about hiding things. The Finder hides the internals of > things like application bundles, which is fine, but > the application itself is visible, so you can move it > around or delete it if you want. The Finder is for my grandma, the terminal is for me. Sometimes I / want/ to put on my grandma's clothes and pretend I'm just a desktop consumer. When I do, I'm usually fine with letting Finder protect me from all the hungry-wolf dot-files. > With ~/.local, you're hiding the fact that the applications > or libraries or whatever are even there in the first > place. You've got all this disk space being used up, > but no way of seeing where or by what, and no > obvious way of freeing it up. I think that's bad HCI. Yes, but who are they hidden from? Dot-files aren't for my grandma, they're for me (in my normal clothes). I'd expect that a geek who installed stuff into ~/.local would know enough to know how to find it, delete it, etc. To me that's good adaptive HCI -- novice users don't care and won't use it or see it, advanced users know how to take advantage of it because it fits in, or at least relates to, their world view. OTOH, the fact that most OSX applications don't store their data in dot-files but instead in ~/Library is fine too, because their data needs to be visible to grandma occasionally. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRW95u3EjvBPtnXfVAQK7fgP/ZfmTIg+x2QqrmDcRiomcYzioS6J8YXEY K0Iiy/uYjux/XfDYbCHUatl2hO/pIGH6NOPHOWZ3/eub+9BfKaIkIn3sIvxVOzd/ cN/AN0OVsHUAPqWiotrALFqAUNj6uP+UUs6/cmlx9LtkNiIIFs6eg16993vNu2eZ pWSK6CKeBnQ= =vzU5 -----END PGP SIGNATURE----- From rasky at develer.com Fri Dec 1 02:19:17 2006 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 01 Dec 2006 02:19:17 +0100 Subject: [Python-Dev] fpectl: does a better implementation make sense? Message-ID: Hello, I spent my last couple of hourse reading several past threads about fpectl. If I'm correct 1) fpectl is scheduled for deletion in 2.6. 2) The biggest problem is that the C standard says that it's undefined to return from a SIGFPE handler. Thus, it's impossible to traps floating point exceptions and convert them to Python exceptions in a way that really works. 3) Moreover, the current implementation of PyFPE_* macros (turning on/off the traps + setjmp) are pretty slow, so they're off by default. Now, I would like Python to rause exceptions (FloatingPointError) whenever a Inf or NaN is computed or used in calculations (which to the best of my little understanding of 754 basically means that I want all FPU errors to be detected and handled). I am not arguing that this should be the default behaviour, I'm just saying that I would like if there was a way to enable this (even if only at Python compile time, in fact). I read that Tim Peters suggested several times to rewrite fpectl so that it does not use traps/signals at all, but just checks the FPU bits to see if there was a FPU error. Basically, the PyFPE BEGIN macro would clear the FPU bits, and the STOP macro would check for FPU errors and raise an appropriate exception if needed. Is this suggestion still valid or people changed their mind meanwhile? Would such a rewrite of fpectl (or a new module with a different name) be accepted? -- Giovanni Bajo From kbk at shore.net Fri Dec 1 03:55:20 2006 From: kbk at shore.net (Kurt B. Kaiser) Date: Thu, 30 Nov 2006 21:55:20 -0500 (EST) Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200612010255.kB12tK59022240@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 407 open ( +1) / 3484 closed ( +5) / 3891 total ( +6) Bugs : 936 open ( +5) / 6363 closed (+14) / 7299 total (+19) RFE : 246 open ( +1) / 244 closed ( +0) / 490 total ( +1) New / Reopened Patches ______________________ sys.id() and sys.intern() (2006-11-23) http://python.org/sf/1601678 opened by Georg Brandl clarify comparison return values (2006-11-24) http://python.org/sf/1602128 opened by Jim Jewett non-framework built python fails to define environ properly (2006-11-23) http://python.org/sf/1602133 opened by paul Suggest a textlist() method for ElementTree (2006-11-24) http://python.org/sf/1602189 opened by Raymond Hettinger subprocess: error redirecting i/o from non-console process (2006-11-27) http://python.org/sf/1603907 opened by Oren Tirosh Performance boost for array repeat (2006-11-28) CLOSED http://python.org/sf/1605020 opened by Mike Klaas Make Imap Error more helpful (2006-11-29) http://python.org/sf/1605192 opened by Thomas Guettler vendor-packages directory. (2005-09-22) http://python.org/sf/1298835 reopened by loewis Patches Closed ______________ __bool__ instead of __nonzero__ (2006-11-21) http://python.org/sf/1600346 closed by jackdied SRE engine do not release the GIL (2005-11-25) http://python.org/sf/1366311 closed by eric_noyau Implement lazy read for sockets (2003-12-04) http://python.org/sf/853963 closed by loewis Disable POSIX for certain/older NetBSD versions (2003-12-05) http://python.org/sf/854796 closed by loewis Performance boost for array repeat (2006-11-28) http://python.org/sf/1605020 closed by mklaas New / Reopened Bugs ___________________ using python extension(wxPython) in c (2006-11-23) CLOSED http://python.org/sf/1601607 opened by jolleydtan getopt Documentation improvement (2006-11-23) CLOSED http://python.org/sf/1601630 opened by Thomas Guettler Incorrect docs for bisect_left (2006-11-24) http://python.org/sf/1602378 opened by Daniel Eloff itemconfigure returns incorrect text property of text items (2006-11-25) http://python.org/sf/1602742 opened by Wojciech Mula wave module forgets to close file on exception (2006-11-26) http://python.org/sf/1603150 opened by amerinese wave module forgets to close file on exception (2006-11-26) CLOSED http://python.org/sf/1603246 opened by amerinese pstats module (profiler) doens't support unicode paths (2006-11-26) CLOSED http://python.org/sf/1603321 opened by Adal Chiriliuc def foo(((x))) segfault (2006-11-26) CLOSED http://python.org/sf/1603332 opened by Tony Lownds f=open fails with TypeError (2006-11-26) CLOSED http://python.org/sf/1603336 opened by Gibholio f=open fails with TypeError (2006-11-26) CLOSED http://python.org/sf/1603412 opened by Gibholio subprocess.py (py2.5) wrongly claims py2.2 compatibility (2006-11-27) http://python.org/sf/1603424 opened by Tim Wegener Python socket library confused by IPV6 notation in /etc/host (2006-11-27) http://python.org/sf/1603527 opened by Eric S. Raymond SaveConfigParser.write() doesn't quote %-Sign (2006-11-27) http://python.org/sf/1603688 opened by Rebecca Breu grammatical error in Python Library Reference::Tkinter (2006-11-27) CLOSED http://python.org/sf/1603789 opened by Gabriel M. Elder subprocess.Popen closes fds for sys.stdout or sys.stderr (2006-11-28) http://python.org/sf/1604851 opened by Nishkar Grover _CRT_SECURE_NO_DEPRECATE macro redefinition with VC++ 8 (2006-11-28) CLOSED http://python.org/sf/1604862 opened by William Fulton logging %(module)s reporting wrong modules (2006-11-29) http://python.org/sf/1605110 opened by Mad-Marty csv module broken for unicode (2006-11-30) http://python.org/sf/1606092 opened by JettLogic readline on popen3 file returns empty string before end (2006-11-30) http://python.org/sf/1606233 opened by Bill Wallace Bugs Closed ___________ utf_8_sig decode fails with buffer input (2006-11-23) http://python.org/sf/1601501 closed by doerwalter using python extension(wxPython) in c (2006-11-23) http://python.org/sf/1601607 closed by loewis getopt Documentation improvement (2006-11-23) http://python.org/sf/1601630 closed by gbrandl ctypes Structure allows recursive definition (2006-11-17) http://python.org/sf/1598620 closed by theller 2.4 & 2.5 can't create win installer on linux (2006-10-04) http://python.org/sf/1570417 closed by loewis wave module forgets to close file on exception (2006-11-26) http://python.org/sf/1603246 closed by gbrandl pstats module (profiler) doens't support unicode paths (2006-11-26) http://python.org/sf/1603321 closed by gbrandl def foo(((x))) segfault (2006-11-26) http://python.org/sf/1603332 closed by gbrandl f=open fails with TypeError (2006-11-26) http://python.org/sf/1603336 closed by gbrandl f=open fails with TypeError (2006-11-26) http://python.org/sf/1603412 closed by nnorwitz grammatical error in Python Library Reference::Tkinter (2006-11-27) http://python.org/sf/1603789 closed by gbrandl read() in windows stops on chr(26) (2006-11-13) http://python.org/sf/1595822 closed by sf-robot _ctypes built with GCC on AIX 5.3 fails with ld ffi error (2006-09-23) http://python.org/sf/1563807 closed by theller _CRT_SECURE_NO_DEPRECATE macro redefinition with VC++ 8 (2006-11-28) http://python.org/sf/1604862 closed by loewis "make install" for Python 2.4.4 not working properly (2006-10-19) http://python.org/sf/1580563 closed by sf-robot From andrew-pythondev at puzzling.org Fri Dec 1 05:27:09 2006 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Fri, 1 Dec 2006 15:27:09 +1100 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <1164930163.31269.407.camel@localhost> References: <5.1.1.6.0.20061126143124.027f3e68@sparrow.telecommunity.com> <5.1.1.6.0.20061127090652.04341cf8@sparrow.telecommunity.com> <200611290053.17199.anthony@interlink.com.au> <20061129112354.GA30665@code0.codespeak.net> <1164917149.31269.344.camel@localhost> <456F51C5.8080500@holdenweb.com> <1164930163.31269.407.camel@localhost> Message-ID: <20061201042709.GE10614@steerpike.home.puzzling.org> On Fri, Dec 01, 2006 at 12:42:42AM +0100, Jan Claeys wrote: > Op donderdag 30-11-2006 om 21:48 uur [tijdzone +0000], schreef Steve > Holden: > > I think the point is that some distros (Debian is the one that springs > > to mind most readily, but I'm not a distro archivist) require a separate > > install for distutils even though it's been a part of the standard > > *Python* distro since 2.3 (2.2?) > > > > So, it isn't that you can't get distutils, it's that you have to take an > > extra step over and above installing Python. > > No, it just means that several parts of the python.org source package > are spread over several binary packages, just like happens with hundreds > or thousands of other packages, and any Debian (or Ubuntu or other > distro doing this) administrator worth his or her money should be aware > of that, and be able to find those packages. In both the current Debian and Ubuntu releases, the "python2.4" binary package includes distutils. See for yourself at http://packages.debian.org/cgi-bin/search_contents.pl?searchmode=filelist&word=python2.4&version=stable&arch=i386&page=1&number=all if you like. So I'm not sure what the fuss is about. > Maybe python.org can include several logical "divisions" in the > python.org distribution and make it easy for OS distro packagers to make > separate packages if they want to, as most of them are quite happy to > have less work to do, provided the upstream "divisions" do more or less > what they want. ;-) (Oh, and such a division should IMHO also include > a "minimal python" for embedded/low-resource hardware use, where things > like distutils, GUI toolkits, a colelction of 20 XML libraries and > documentation are most likely not needed.) There's already a "python2.4-minimal" package in Debian/Ubuntu that would probably be a good starting point for an embedded distribution that cares about space more than providing a complete Python. -Andrew. From mwh at python.net Fri Dec 1 10:21:50 2006 From: mwh at python.net (Michael Hudson) Date: Fri, 01 Dec 2006 10:21:50 +0100 Subject: [Python-Dev] fpectl: does a better implementation make sense? In-Reply-To: (Giovanni Bajo's message of "Fri, 01 Dec 2006 02:19:17 +0100") References: Message-ID: <87fyc0gedt.fsf@starship.python.net> Giovanni Bajo writes: > Hello, > > I spent my last couple of hourse reading several past threads about fpectl. If > I'm correct > > 1) fpectl is scheduled for deletion in 2.6. > 2) The biggest problem is that the C standard says that it's undefined to > return from a SIGFPE handler. Well, I'm not sure that's the "biggest" problem in any sense: C also doesn't say anything about how to set things up so that 1.0/0.0 gets you a SIGFPE. > Thus, it's impossible to traps floating point > exceptions and convert them to Python exceptions in a way that really works. "Impossible" is a strong word :-) But yeah. > 3) Moreover, the current implementation of PyFPE_* macros (turning on/off the > traps + setjmp) are pretty slow, so they're off by default. > > Now, I would like Python to rause exceptions (FloatingPointError) whenever a > Inf or NaN is computed or used in calculations (which to the best of my little > understanding of 754 basically means that I want all FPU errors to be > detected and handled). I suggest starting from here and forgetting about fpectl. > I am not arguing that this should be the default behaviour, Good :-) > I'm just saying that I would like if there was a way to enable this > (even if only at Python compile time, in fact). I think I would too. > I read that Tim Peters suggested several times to rewrite fpectl so that it > does not use traps/signals at all, but just checks the FPU bits to see if > there was a FPU error. Basically, the PyFPE BEGIN macro would clear the FPU > bits, and the STOP macro would check for FPU errors and raise an appropriate > exception if needed. This is part of the reason I want to rip out fpectl: I think a new set of macros is called for, if only so they can have better names. But you'll end up fighting a series of aggravating battles with compiler optimizations and platform support and so on. Did you read this post: http://mail.python.org/pipermail/python-list/2005-July/331509.html and the thread it came from? > Is this suggestion still valid or people changed their mind meanwhile? I haven't changed my mind appreciably. > Would such a rewrite of fpectl (or a new module with a different > name) be accepted? I would at least try to review it and press for its inclusion. Cheers, mwh -- The bottom tier is what a certain class of wanker would call "business objects" ... -- Greg Ward, 9 Dec 1999 From robinbryce at gmail.com Fri Dec 1 12:18:44 2006 From: robinbryce at gmail.com (Robin Bryce) Date: Fri, 1 Dec 2006 11:18:44 +0000 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <456E7651.7070100@v.loewis.de> References: <5.1.1.6.0.20061126143124.027f3e68@sparrow.telecommunity.com> <456AEA45.7060209@suse.cz> <5.1.1.6.0.20061127090652.04341cf8@sparrow.telecommunity.com> <456C7D20.1080604@v.loewis.de> <456E7651.7070100@v.loewis.de> Message-ID: Fair enough. Robin On 30/11/06, "Martin v. L?wis" wrote: > Robin Bryce schrieb: > > Yes, especially with the regard to the level you pitch for LSB. I > > would go as far as to say that if this "contract in spirit" is broken > > by vendor repackaging they should: > > * Call the binaries something else because it is NOT python any more. > > * Setup the installation layout so that it does NOT conflict or > > overlap with the standard layout. > > * Call the whole package something else. > > I think that would be counter-productive. If applied in a strict > sense, you couldn't call it Python anymore if it isn't in /usr/local. > I see no point to that. > > It shouldn't be called Python anymore if it doesn't implement > the Python language specification. No vendor is modifying it > in such a way that > > print "Hello" > > stops working. > > > Is it a bad idea to suggest that: Python grows a vendor_variant > > attribute somewhere in the standard lib; That its content is > > completely dictated by a new ./configure argument which is the empty > > string by default; And, request that it is left empty by re-packagers > > if the installation is 'reasonably standard' ? > > I'm not sure in what applications that would be useful. > > > I would strongly prefer _not_ write code that is conditional on such > > an attribute. However if there was a clear way for a vendor to > > communicate "This is not a standard python runtime" to the python run > > time, early failure (in the application) with informative error > > messages becomes much more viable. > > Again: none of the vendors modifies Python in a way that what > you get is "not a standard Python runtime". They *all* > are "standard Python runtimes". > > Regards, > Martin > From phd at phd.pp.ru Fri Dec 1 23:07:45 2006 From: phd at phd.pp.ru (Oleg Broytmann) Date: Sat, 2 Dec 2006 01:07:45 +0300 Subject: [Python-Dev] OT: World's oldest ritual discovered. Worshipped the python 70, 000 years ago Message-ID: <20061201220745.GA18461@phd.pp.ru> http://www.apollon.uio.no/vis/art/2006_4/Artikler/python_english (-: Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From talin at acm.org Sat Dec 2 09:34:22 2006 From: talin at acm.org (Talin) Date: Sat, 02 Dec 2006 00:34:22 -0800 Subject: [Python-Dev] OT: World's oldest ritual discovered. Worshipped the python 70, 000 years ago In-Reply-To: <20061201220745.GA18461@phd.pp.ru> References: <20061201220745.GA18461@phd.pp.ru> Message-ID: <45713A8E.2080506@acm.org> Oleg Broytmann wrote: > http://www.apollon.uio.no/vis/art/2006_4/Artikler/python_english > > (-: > > Oleg. I noticed the other day that the word "Pythonic" means "Prophetic", according to Webster's Revised Unabridged Dictionary, 1913 edition: Py*thon"ic (?), a. [L. pythonicus, Gr. . See Pythian.] Prophetic; oracular; pretending to foretell events. So, in the future, when someone says that a particular feature isn't "pythonic", what they are really saying is that the feature isn't a good indicator of things to come, which implies that such statements are self-fulfilling prophesies. Which means that statements about whether a particular language feature is pythonic are themselves pythonic. -- Talin From phd at phd.pp.ru Sat Dec 2 10:14:52 2006 From: phd at phd.pp.ru (Oleg Broytmann) Date: Sat, 2 Dec 2006 12:14:52 +0300 Subject: [Python-Dev] OT: World's oldest ritual discovered. Worshipped the python 70, 000 years ago In-Reply-To: <45713A8E.2080506@acm.org> References: <20061201220745.GA18461@phd.pp.ru> <45713A8E.2080506@acm.org> Message-ID: <20061202091452.GA22632@phd.pp.ru> On Sat, Dec 02, 2006 at 12:34:22AM -0800, Talin wrote: > I noticed the other day that the word "Pythonic" means "Prophetic", This is, of course, due to the Greek mythology and the oracle at Delphi! http://en.wikipedia.org/wiki/Python_%28mythology%29 Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From g.brandl at gmx.net Sat Dec 2 14:02:13 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 02 Dec 2006 14:02:13 +0100 Subject: [Python-Dev] OT: World's oldest ritual discovered. Worshipped the python 70, 000 years ago In-Reply-To: <45713A8E.2080506@acm.org> References: <20061201220745.GA18461@phd.pp.ru> <45713A8E.2080506@acm.org> Message-ID: Talin wrote: > Oleg Broytmann wrote: >> http://www.apollon.uio.no/vis/art/2006_4/Artikler/python_english >> >> (-: >> >> Oleg. > > I noticed the other day that the word "Pythonic" means "Prophetic", > according to Webster's Revised Unabridged Dictionary, 1913 edition: > > Py*thon"ic (?), a. [L. pythonicus, Gr. . See Pythian.] Prophetic; > oracular; pretending to foretell events. Surely it is. The PSU once used the time machine to travel to Ancient Greece and gave the Delphi priestess her name, along with a schoolbook about ancient histo From fredrik at pythonware.com Sat Dec 2 14:24:35 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat, 02 Dec 2006 14:24:35 +0100 Subject: [Python-Dev] Small tweak to tokenize.py? In-Reply-To: References: <5.1.1.6.0.20061130131522.033863a0@sparrow.telecommunity.com> Message-ID: Guido van Rossum wrote: >> it would be a good thing if it could, optionally, be made to report >> horizontal whitespace as well. > > It's remarkably easy to get this out of the existing API sure, but it would be even easier if I didn't have to write that code myself (last time I did that, I needed a couple of tries before the parser handled all cases correctly...). but maybe this could simply be handled by a helper generator in the tokenizer module, that simply wraps the standard tokenizer generator and inserts whitespace tokens where necessary? > keep track > of the end position returned by the previous call, and if it's > different from the start position returned by the next call, slice the > line text from the column positions, assuming the line numbers are the > same.If the line numbers differ, something has been eating \n tokens; > this shouldn't happen any more with my patch. you'll still have to deal with multiline strings, right? From arigo at tunes.org Sat Dec 2 15:19:56 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat, 2 Dec 2006 15:19:56 +0100 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <20061201042709.GE10614@steerpike.home.puzzling.org> References: <5.1.1.6.0.20061126143124.027f3e68@sparrow.telecommunity.com> <5.1.1.6.0.20061127090652.04341cf8@sparrow.telecommunity.com> <200611290053.17199.anthony@interlink.com.au> <20061129112354.GA30665@code0.codespeak.net> <1164917149.31269.344.camel@localhost> <456F51C5.8080500@holdenweb.com> <1164930163.31269.407.camel@localhost> <20061201042709.GE10614@steerpike.home.puzzling.org> Message-ID: <20061202141955.GA15771@code0.codespeak.net> Hi Andrew, On Fri, Dec 01, 2006 at 03:27:09PM +1100, Andrew Bennetts wrote: > In both the current Debian and Ubuntu releases, the "python2.4" binary package > includes distutils. Ah, good. This must be a relatively recent change. I'm not a Debian user, but merely a user that happens to have to use a Debian machine occasionally. None of these machines had distutils installed by default so far, but it looks like things are going to improve in the future then. Now I only have to hope that 2.4.4 makes its way out of 'unstable' soon. As far as I can tell sysadmins installing the current 'testing' would still be getting a Python 2.4.3, not modern enough to cope with the arithmetic overflow issues introduced by the cutting-edge GCC of 'testing'. But it looks like any year now I'll be able to assume that Linux machines all provide a Python that works as nicely as a freshly compiled one. A bientot, Armin From lomax at pumpichank.com Sat Dec 2 16:54:14 2006 From: lomax at pumpichank.com (Frank Lomax) Date: Sat, 2 Dec 2006 10:54:14 -0500 Subject: [Python-Dev] OT: World's oldest ritual discovered. Worshipped the python 70, 000 years ago In-Reply-To: References: <20061201220745.GA18461@phd.pp.ru> <45713A8E.2080506@acm.org> Message-ID: <7A2E660A-CB25-4437-BF0D-92F5BF535CDB@pumpichank.com> On Dec 2, 2006, at 8:02 AM, Georg Brandl wrote: > Surely it is. The PSU once used the time machine to travel to Ancient > Greece and gave the Delphi priestess her name, along with a schoolbook > about ancient histo The PSU does not, nor ever has existed. Any statement implying otherwise is false and subversive. There is no PSU and even if there is, it has no influence whatsoev From lists at janc.be Sat Dec 2 17:07:18 2006 From: lists at janc.be (Jan Claeys) Date: Sat, 02 Dec 2006 17:07:18 +0100 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <456F747B.9060107@holdenweb.com> References: <5.1.1.6.0.20061126143124.027f3e68@sparrow.telecommunity.com> <5.1.1.6.0.20061127090652.04341cf8@sparrow.telecommunity.com> <200611290053.17199.anthony@interlink.com.au> <20061129112354.GA30665@code0.codespeak.net> <1164917149.31269.344.camel@localhost> <456F51C5.8080500@holdenweb.com> <1164930163.31269.407.camel@localhost> <456F747B.9060107@holdenweb.com> Message-ID: <1165075639.31269.480.camel@localhost> Op vrijdag 01-12-2006 om 00:16 uur [tijdzone +0000], schreef Steve Holden: > Jan Claeys wrote: > [...] > > Probably the Debian maintainers could have named packages differently to > > make things less confusing for newbies (e.g. by having the 'pythonX.Y' > > packages being meta-packages that depend on all binary packages built > > from the upstream source package), but that doesn't mean splitting > > "python" (or other projects) up in several packages is wrong. E.g. when > > installing on an flash drive, people are probably quite happy to leave > > the 20 MiB of Python documentation out... > > > Right, who cares about newbies, they're only the future of the language, > after all. I take your point that some flexibility is advantageous once > you get past the newbie stage, but I think that here we are talking > about trying to avoid mis-steps that will potentially put people off > making that transition. Like I said, it's possible to split Python without making things complicated for newbies. The fact that Debian didn't do so in the past might be a considered a packaging bug, but the problem isn't in the practice of splitting upstream projects in several binary packages itself. > > Maybe python.org can include several logical "divisions" in the > > python.org distribution and make it easy for OS distro packagers to make > > separate packages if they want to, as most of them are quite happy to > > have less work to do, provided the upstream "divisions" do more or less > > what they want. ;-) (Oh, and such a division should IMHO also include > > a "minimal python" for embedded/low-resource hardware use, where things > > like distutils, GUI toolkits, a colelction of 20 XML libraries and > > documentation are most likely not needed.) > If only there were some guarantee that the distros would respect any > project partitioning imposed by python-deb we might stand a chance of > resolving these issues. There will never be a guarantee, as some distros might have very special targets, but I'm pretty sure that most distros would follow any _sensible_ proposition (and looking at current practice might give a good clue about what they want). -- Jan Claeys From fredrik at pythonware.com Sat Dec 2 17:17:21 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat, 02 Dec 2006 17:17:21 +0100 Subject: [Python-Dev] OT: World's oldest ritual discovered. Worshipped the python 70, 000 years ago In-Reply-To: <7A2E660A-CB25-4437-BF0D-92F5BF535CDB@pumpichank.com> References: <20061201220745.GA18461@phd.pp.ru> <45713A8E.2080506@acm.org> <7A2E660A-CB25-4437-BF0D-92F5BF535CDB@pumpichank.com> Message-ID: Frank Lomax wrote: > The PSU does not, nor ever has existed. Any statement implying > otherwise is false and subversive. There is no PSU and even if there > is, it has no influence whatsoev it's a bit interesting that every time someone writes something along those lines, their computer's Power Supply Unit always gives up halfway through the last sentence From baptiste13 at altern.org Sat Dec 2 18:32:06 2006 From: baptiste13 at altern.org (Baptiste Carvello) Date: Sat, 02 Dec 2006 18:32:06 +0100 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <20061202141955.GA15771@code0.codespeak.net> References: <5.1.1.6.0.20061126143124.027f3e68@sparrow.telecommunity.com> <5.1.1.6.0.20061127090652.04341cf8@sparrow.telecommunity.com> <200611290053.17199.anthony@interlink.com.au> <20061129112354.GA30665@code0.codespeak.net> <1164917149.31269.344.camel@localhost> <456F51C5.8080500@holdenweb.com> <1164930163.31269.407.camel@localhost> <20061201042709.GE10614@steerpike.home.puzzling.org> <20061202141955.GA15771@code0.codespeak.net> Message-ID: Armin Rigo a ?crit : > Now I only have to hope that 2.4.4 makes its way out of 'unstable' soon. > As far as I can tell sysadmins installing the current 'testing' would > still be getting a Python 2.4.3, not modern enough to cope with the > arithmetic overflow issues introduced by the cutting-edge GCC of > 'testing'. well, the usual way to express your "hope" would be to file a bug on the python2.4 package, to make sure the maintainers are aware of the problem. However, it looks like 2.4.4 made it to testing today, so it might not be necessary. Cheers, BC From guido at python.org Sat Dec 2 19:06:50 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 2 Dec 2006 10:06:50 -0800 Subject: [Python-Dev] Small tweak to tokenize.py? In-Reply-To: References: <5.1.1.6.0.20061130131522.033863a0@sparrow.telecommunity.com> Message-ID: On 12/2/06, Fredrik Lundh wrote: > Guido van Rossum wrote: > > >> it would be a good thing if it could, optionally, be made to report > >> horizontal whitespace as well. > > > > It's remarkably easy to get this out of the existing API > > sure, but it would be even easier if I didn't have to write that code > myself (last time I did that, I needed a couple of tries before the > parser handled all cases correctly...). > > but maybe this could simply be handled by a helper generator in the > tokenizer module, that simply wraps the standard tokenizer generator > and inserts whitespace tokens where necessary? A helper sounds like a promising idea. Anyone interested in volunteering a patch? > > keep track > > of the end position returned by the previous call, and if it's > > different from the start position returned by the next call, slice the > > line text from the column positions, assuming the line numbers are the > > same.If the line numbers differ, something has been eating \n tokens; > > this shouldn't happen any more with my patch. > > you'll still have to deal with multiline strings, right? No, they are returned as a single token whose start and stop correctly reflect line/col of the begin and end of the token. My current code (based on the second patch I gave in this thread and the algorithm described above) doesn't have to special-case anything except the ENDMARKER token (to break out of its loop :-). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Sat Dec 2 20:42:32 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 02 Dec 2006 20:42:32 +0100 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <1165075639.31269.480.camel@localhost> References: <5.1.1.6.0.20061126143124.027f3e68@sparrow.telecommunity.com> <5.1.1.6.0.20061127090652.04341cf8@sparrow.telecommunity.com> <200611290053.17199.anthony@interlink.com.au> <20061129112354.GA30665@code0.codespeak.net> <1164917149.31269.344.camel@localhost> <456F51C5.8080500@holdenweb.com> <1164930163.31269.407.camel@localhost> <456F747B.9060107@holdenweb.com> <1165075639.31269.480.camel@localhost> Message-ID: <4571D728.9070506@v.loewis.de> Jan Claeys schrieb: > Like I said, it's possible to split Python without making things > complicated for newbies. You may have that said, but I don't believe its truth. For example, most distributions won't include Tkinter in the "standard" Python installation: Tkinter depends on _tkinter depends on Tk depends on X11 client libraries. Since distributors want to make X11 client libraries optional, they exclude Tkinter. So people wonder why they can't run Tkinter applications (search comp.lang.python for proof that people wonder about precisely that). I don't think the current packaging tools can solve this newbie problem. It might be solvable if installation of X11 libraries would imply installation of Tcl, Tk, and Tkinter: people running X (i.e. most desktop users) would see Tkinter installed, yet it would be possible to omit Tkinter. Regards, Martin From fredrik at pythonware.com Sat Dec 2 21:18:29 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat, 02 Dec 2006 21:18:29 +0100 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <4571D728.9070506@v.loewis.de> References: <5.1.1.6.0.20061126143124.027f3e68@sparrow.telecommunity.com> <5.1.1.6.0.20061127090652.04341cf8@sparrow.telecommunity.com> <200611290053.17199.anthony@interlink.com.au> <20061129112354.GA30665@code0.codespeak.net> <1164917149.31269.344.camel@localhost> <456F51C5.8080500@holdenweb.com> <1164930163.31269.407.camel@localhost> <456F747B.9060107@holdenweb.com> <1165075639.31269.480.camel@localhost> <4571D728.9070506@v.loewis.de> Message-ID: Martin v. L?wis wrote: >> Like I said, it's possible to split Python without making things >> complicated for newbies. > > You may have that said, but I don't believe its truth. For example, > most distributions won't include Tkinter in the "standard" Python > installation: Tkinter depends on _tkinter depends on Tk depends on > X11 client libraries. Since distributors want to make X11 client > libraries optional, they exclude Tkinter. So people wonder why > they can't run Tkinter applications (search comp.lang.python for > proof that people wonder about precisely that). comp.lang.python only represents a *very* tiny fraction of the python universe, though. I'd be much more interested in hearing from people tracking distribution-specific forums. > I don't think the current packaging tools can solve this newbie > problem. It might be solvable if installation of X11 libraries > would imply installation of Tcl, Tk, and Tkinter: people running > X (i.e. most desktop users) would see Tkinter installed, yet > it would be possible to omit Tkinter. I think this is a way too python-centric view of things. maybe we could just ask distributors to prepare a page that describes what portions of the standard distribution they do include, and in what packages they've put the various components, and link to those from the library reference and/or the wiki or FAQ? is there perhaps some machine-readable format that could be used for this? From martin at v.loewis.de Sat Dec 2 23:10:27 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 02 Dec 2006 23:10:27 +0100 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: References: <5.1.1.6.0.20061126143124.027f3e68@sparrow.telecommunity.com> <5.1.1.6.0.20061127090652.04341cf8@sparrow.telecommunity.com> <200611290053.17199.anthony@interlink.com.au> <20061129112354.GA30665@code0.codespeak.net> <1164917149.31269.344.camel@localhost> <456F51C5.8080500@holdenweb.com> <1164930163.31269.407.camel@localhost> <456F747B.9060107@holdenweb.com> <1165075639.31269.480.camel@localhost> <4571D728.9070506@v.loewis.de> Message-ID: <4571F9D3.60204@v.loewis.de> Fredrik Lundh schrieb: > maybe we could just ask distributors to prepare a page that describes > what portions of the standard distribution they do include, and in what > packages they've put the various components, and link to those from the > library reference and/or the wiki or FAQ? is there perhaps some > machine-readable format that could be used for this? Debian has its machine-readable (in some sense) package list at packages.debian.org. Through several links, you get to http://packages.qa.debian.org/p/python2.4.html which gives you the list of binary packages produced from the source package: idle-python2.4 python2.4 python2.4-dbg python2.4-dev python2.4-doc python2.4-examples python2.4-minimal Interestingly enough, _tkinter.so isn't included in any of these; instead, you have to look at http://packages.qa.debian.org/p/python-stdlib-extensions.html Depending on what precisely "this" is you want to use it for, there are other lists as well. Regards, Martin From ben at 666.com Sun Dec 3 13:24:17 2006 From: ben at 666.com (Ben Wing) Date: Sun, 03 Dec 2006 06:24:17 -0600 Subject: [Python-Dev] a feature i'd like to see in python #1: better iteration control Message-ID: <4572C1F1.1050600@666.com> many times writing somewhat complex loops over lists i've found the need to sometimes delete an item from the list. currently there's no easy way to do so; basically, you have to write something like i = 0 while i < len(list): el = list[i] ...do something... if el should be deleted: del list[i] else: i += 1 note that you can't write for x in list: or even for i in xrange(len(list)): note also that you need to do some trickiness to adjust the index appropriately when deleting. i'd much rather see something like: for x:iter in list: ...do something... if x should be deleted: iter.delete() the idea is that you have a way of retrieving both the element itself and the iterator for the element, so that you can then call methods on the iterator. it shouldn't be too hard to implement iter.delete(), as well as iter.insert() and similar functions. (the recent changes to the generator protocol in 2.5 might help.) the only question then is how to access the iterator. the syntax i've proposed, with `x:iter', seems fairly logical (note that parallels with slice notation, which also uses a colon) and doesn't introduce any new operators. (comma is impossible since `for x,iter in list:' already has a meaning) btw someone is probably going to come out and say "why don't you just use a list comprehension with an `if' clause? the problems are [1] i'd like this to be destructive; [2] i'd like this to work over non-lists as well, e.g. hash-tables; [3] list comprehensions work well in simple cases, but in more complicated cases when you may be doing various things on each step, and might not know whether you need to delete or insert an element until after you've done various other things, a list comprehension would not fit naturally; [4] this mechanism is extendible to insertions, replacements and other such changes as well as just filterings. ben From ben at 666.com Sun Dec 3 13:24:31 2006 From: ben at 666.com (Ben Wing) Date: Sun, 03 Dec 2006 06:24:31 -0600 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects Message-ID: <4572C1FF.2000900@666.com> this one is fairly simple. if `m' is a match object, i'd like to be able to write m[1] instead of m.group(1). (similarly, m[:] should return the same as list(m.groups()).) this would remove some of the verbosity of regexp code, with probably a net gain in readability; certainly no loss. ben From ferringb at gmail.com Sun Dec 3 13:39:20 2006 From: ferringb at gmail.com (Brian Harring) Date: Sun, 3 Dec 2006 04:39:20 -0800 Subject: [Python-Dev] a feature i'd like to see in python #1: better iteration control In-Reply-To: <4572C1F1.1050600@666.com> References: <4572C1F1.1050600@666.com> Message-ID: <20061203123920.GA11625@seldon> On Sun, Dec 03, 2006 at 06:24:17AM -0600, Ben Wing wrote: > many times writing somewhat complex loops over lists i've found the need > to sometimes delete an item from the list. currently there's no easy > way to do so; basically, you have to write something like > > i = 0 > while i < len(list): > el = list[i] > ...do something... > if el should be deleted: > del list[i] > else: > i += 1 > > note that you can't write > > for x in list: > > or even > > for i in xrange(len(list)): > > note also that you need to do some trickiness to adjust the index > appropriately when deleting. > > i'd much rather see something like: > > for x:iter in list: > ...do something... > if x should be deleted: > iter.delete() > > the idea is that you have a way of retrieving both the element itself > and the iterator for the element, so that you can then call methods on > the iterator. it shouldn't be too hard to implement iter.delete(), as > well as iter.insert() and similar functions. (the recent changes to the > generator protocol in 2.5 might help.) > > the only question then is how to access the iterator. the syntax i've > proposed, with `x:iter', seems fairly logical (note that parallels with > slice notation, which also uses a colon) and doesn't introduce any new > operators. (comma is impossible since `for x,iter in list:' already has > a meaning) > > btw someone is probably going to come out and say "why don't you just > use a list comprehension with an `if' clause? the problems are [1] i'd > like this to be destructive; Just use slice assignment. l = list(xrange(100)) l2 = [x for x in l if x > 50] l[:] = l2[:] > [2] i'd like this to work over non-lists as well, e.g. hash-tables; There in is the sucky part; iterator protocol is simple; what you're proposing is extending iterators so that they recall the last value (else iter.delete() would not do anything), which... eh. Think it sucks, to say the least ;) Simple example of where this gets ugly is in iterating over a file. >[3] list comprehensions work well in simple > cases, but in more complicated cases when you may be doing various > things on each step, and might not know whether you need to delete or > insert an element until after you've done various other things, a list > comprehension would not fit naturally; Slice assignments work fine here. > [4] this mechanism is extendible > to insertions, replacements and other such changes as well as just > filterings. Yes it is, except the new 'iterator' isn't as extendable to arbitrary sequences after. Also is ignoring the fact that doing in place deletion of lists as you walk can get massively costly quick; worst case, quad for removal (hence collections.deque existing). ~harring -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20061203/dd0ead0f/attachment.pgp From martin at v.loewis.de Sun Dec 3 15:16:45 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 03 Dec 2006 15:16:45 +0100 Subject: [Python-Dev] a feature i'd like to see in python #1: better iteration control In-Reply-To: <4572C1F1.1050600@666.com> References: <4572C1F1.1050600@666.com> Message-ID: <4572DC4D.6070709@v.loewis.de> Ben Wing schrieb: > i'd much rather see something like: > > for x:iter in list: > ...do something... > if x should be deleted: > iter.delete() You can easily implement that feature yourself if you need it, at least for lists (or sequences that support integer indexing): class deletable_iter: def __init__(self, sequence): self.sequence = sequence self.last = -1 def __iter__(self): return self def next(self): self.last += 1 try: return self.sequence[self.last] except IndexError: raise StopIteration def delete_last(self): del self.sequence[self.last] self.last -= 1 You use this class like this: x = [1,2,3,4,5,6,7,8] y = deletable_iter(x) for i in y: print i if i%2 == 0: y.delete_last() print x This cannot be generalized for the iteration protocol, because it might not be possible for the iterator to delete an element after it has returned it. Notice that this version "supports" multiple invocations of delete_last in a single step; it may be useful to allow at most one deletion, and raise an exception if a second deletion is attempted. Even if the deletion was added to the iterator protocol (which would require a PEP, IMO), I don't think the syntax should be changed. If you want access to the iterator in the loop, you should explicitly assign it to a variable before entering the loop. Regards, Martin From martin at v.loewis.de Sun Dec 3 15:22:56 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 03 Dec 2006 15:22:56 +0100 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <4572C1FF.2000900@666.com> References: <4572C1FF.2000900@666.com> Message-ID: <4572DDC0.6040203@v.loewis.de> Ben Wing schrieb: > this one is fairly simple. if `m' is a match object, i'd like to be > able to write m[1] instead of m.group(1). (similarly, m[:] should return > the same as list(m.groups()).) this would remove some of the verbosity > of regexp code, with probably a net gain in readability; certainly no loss. Please post a patch to sf.net/projects/python (or its successor). Several issues need to be taken into account: - documentation and test cases must be updated to integrate the new API - for slicing, you need to consider not only omitted indices, but also "true" slices (e.g. m[1:5]) - how should you deal with negative indices? - should len(m) be supported? Regards, Martin From fredrik at pythonware.com Sun Dec 3 15:34:11 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sun, 03 Dec 2006 15:34:11 +0100 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <4572DDC0.6040203@v.loewis.de> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> Message-ID: Martin v. L?wis wrote: > Several issues need to be taken into account: the most important issue is that if you want an object to behave as a sequence of something, you need to decide what that something is before you start tinkering with the syntax. under Ben's simple proposal, m[:][1] and m[1] would be two different things. I'm not sure that's a good idea, really. From rhamph at gmail.com Sun Dec 3 16:41:55 2006 From: rhamph at gmail.com (Adam Olsen) Date: Sun, 3 Dec 2006 08:41:55 -0700 Subject: [Python-Dev] a feature i'd like to see in python #1: better iteration control In-Reply-To: <4572C1F1.1050600@666.com> References: <4572C1F1.1050600@666.com> Message-ID: On 12/3/06, Ben Wing wrote: > many times writing somewhat complex loops over lists i've found the need > to sometimes delete an item from the list. currently there's no easy > way to do so; basically, you have to write something like As I don't believe there's any need for a language extension, I've posted my approach to this on comp.lang.python: http://groups.google.com/group/comp.lang.python/browse_thread/thread/724aa6bcf021cfad/c4c629bd1bacc12b#c4c629bd1bacc12b Note that deleting from the list as you iterate over it tends to have very poor performance for larger lists. My post includes some timings, demonstrating this. -- Adam Olsen, aka Rhamphoryncus From martin at v.loewis.de Sun Dec 3 16:53:35 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 03 Dec 2006 16:53:35 +0100 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> Message-ID: <4572F2FF.3030206@v.loewis.de> Fredrik Lundh schrieb: > the most important issue is that if you want an object to behave as a > sequence of something, you need to decide what that something is before > you start tinkering with the syntax. > > under Ben's simple proposal, m[:][1] and m[1] would be two different > things. I'm not sure that's a good idea, really. Ah, right; I misread his proposal as saying that m[:] should return [m[0]] + list(m.groups()) (rather, I expected that m.groups() would include m.group(0)). To answer your first question: it is clearly groups that you want to index, just as the .group() method indexes groups. The typical equivalences should hold, of course, e.g. m[1:5][1] == m[2] etc. Regards, Martin From barry at python.org Sun Dec 3 17:07:05 2006 From: barry at python.org (Barry Warsaw) Date: Sun, 3 Dec 2006 11:07:05 -0500 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <4572DDC0.6040203@v.loewis.de> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> Message-ID: <9EB57908-7EFC-42D6-B27B-C313A7F9B1F5@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Dec 3, 2006, at 9:22 AM, Martin v. L?wis wrote: > Ben Wing schrieb: >> this one is fairly simple. if `m' is a match object, i'd like to be >> able to write m[1] instead of m.group(1). (similarly, m[:] should >> return >> the same as list(m.groups()).) this would remove some of the >> verbosity >> of regexp code, with probably a net gain in readability; certainly >> no loss. > > Please post a patch to sf.net/projects/python (or its successor). > > Several issues need to be taken into account: > - documentation and test cases must be updated to integrate the new > API > - for slicing, you need to consider not only omitted indices, but also > "true" slices (e.g. m[1:5]) > - how should you deal with negative indices? > - should len(m) be supported? what about m['named_group_1'] etc? - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRXL2LnEjvBPtnXfVAQILTwP/SRfvOXXhUXIBK52ByqwuhCwF+K/HfEYu 0+j/L3WQXFE4sZ1CHT6TaMT/K6tbhE7zuGamKmk1+CtSPQAKluwJ8d2/y6Ubp4KE S24sP8NOzKgDg/aTn5zFS/Up7HDfhMIWGCLbg5rY+/Bl48skEkqeo4w07XKwJzky CvxsrJb4wQY= =OPjI -----END PGP SIGNATURE----- From fredrik at pythonware.com Sun Dec 3 17:07:08 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sun, 03 Dec 2006 17:07:08 +0100 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <4572F2FF.3030206@v.loewis.de> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> Message-ID: Martin v. L?wis wrote: > Ah, right; I misread his proposal as saying that m[:] should return > [m[0]] + list(m.groups()) (rather, I expected that m.groups() would > include m.group(0)). match groups are numbered 1..N, not 0..(N-1), in both the API and in the RE syntax (and we don't have much control over the latter). > To answer your first question: it is clearly groups that you want > to index, just as the .group() method indexes groups. so what should len(m) do? From martin at v.loewis.de Sun Dec 3 17:10:08 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 03 Dec 2006 17:10:08 +0100 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <9EB57908-7EFC-42D6-B27B-C313A7F9B1F5@python.org> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <9EB57908-7EFC-42D6-B27B-C313A7F9B1F5@python.org> Message-ID: <4572F6E0.1060400@v.loewis.de> Barry Warsaw schrieb: >> Several issues need to be taken into account: >> - documentation and test cases must be updated to integrate the new API >> - for slicing, you need to consider not only omitted indices, but also >> "true" slices (e.g. m[1:5]) >> - how should you deal with negative indices? >> - should len(m) be supported? > > what about m['named_group_1'] etc? That should also be taken into consideration; I suggest to support it. Regards, Martin From aahz at pythoncraft.com Sun Dec 3 18:15:00 2006 From: aahz at pythoncraft.com (Aahz) Date: Sun, 3 Dec 2006 09:15:00 -0800 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <4572DDC0.6040203@v.loewis.de> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> Message-ID: <20061203171500.GA9392@panix.com> On Sun, Dec 03, 2006, "Martin v. L?wis" wrote: > Ben Wing schrieb: >> >> this one is fairly simple. if `m' is a match object, i'd like to be >> able to write m[1] instead of m.group(1). (similarly, m[:] should return >> the same as list(m.groups()).) this would remove some of the verbosity >> of regexp code, with probably a net gain in readability; certainly no loss. > > Please post a patch to sf.net/projects/python (or its successor). Given the list of issues and subsequent discussion so far, I think a PEP will be required. This needs more documentation than the typical patch. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ Member of the Groucho Marx Fan Club From arigo at tunes.org Sun Dec 3 19:28:41 2006 From: arigo at tunes.org (Armin Rigo) Date: Sun, 3 Dec 2006 19:28:41 +0100 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <20061201042709.GE10614@steerpike.home.puzzling.org> References: <5.1.1.6.0.20061126143124.027f3e68@sparrow.telecommunity.com> <5.1.1.6.0.20061127090652.04341cf8@sparrow.telecommunity.com> <200611290053.17199.anthony@interlink.com.au> <20061129112354.GA30665@code0.codespeak.net> <1164917149.31269.344.camel@localhost> <456F51C5.8080500@holdenweb.com> <1164930163.31269.407.camel@localhost> <20061201042709.GE10614@steerpike.home.puzzling.org> Message-ID: <20061203182841.GA21598@code0.codespeak.net> Hi Andrew, On Fri, Dec 01, 2006 at 03:27:09PM +1100, Andrew Bennetts wrote: > In both the current Debian and Ubuntu releases, the "python2.4" binary package > includes distutils. Ah, distutils are distributed in the base package now, but not the 'config' subdirectory of a standard library's normal installation, which makes distutils a bit useless. I should go and file a bug, I guess. The reason why I did not do it so far, is that the fact that no one did tends to show that they really think distributing 'config' is not the "right" thing to do. A reasoning along the lines of: "this subdir contains include files, so it should go in the python-dev package". I'm not too interested in fighting this kind of preconception. A bientot, Armin. From martin at v.loewis.de Sun Dec 3 19:38:21 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 03 Dec 2006 19:38:21 +0100 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <20061203171500.GA9392@panix.com> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <20061203171500.GA9392@panix.com> Message-ID: <4573199D.7050907@v.loewis.de> Aahz schrieb: >>> this one is fairly simple. if `m' is a match object, i'd like to be >>> able to write m[1] instead of m.group(1). (similarly, m[:] should return >>> the same as list(m.groups()).) this would remove some of the verbosity >>> of regexp code, with probably a net gain in readability; certainly no loss. >> Please post a patch to sf.net/projects/python (or its successor). > > Given the list of issues and subsequent discussion so far, I think a PEP > will be required. This needs more documentation than the typical patch. I disagree. So far, nobody has spoken against the proposed feature. It's really a small addition of a new method to an existing type. Entire classes have been added to the standard library without a PEP. People can still criticize the patch when its posted (and it's not clear that the OP is even willing to produce a patch). Regards, Martin From martin at v.loewis.de Sun Dec 3 19:56:34 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 03 Dec 2006 19:56:34 +0100 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <20061203182841.GA21598@code0.codespeak.net> References: <5.1.1.6.0.20061126143124.027f3e68@sparrow.telecommunity.com> <5.1.1.6.0.20061127090652.04341cf8@sparrow.telecommunity.com> <200611290053.17199.anthony@interlink.com.au> <20061129112354.GA30665@code0.codespeak.net> <1164917149.31269.344.camel@localhost> <456F51C5.8080500@holdenweb.com> <1164930163.31269.407.camel@localhost> <20061201042709.GE10614@steerpike.home.puzzling.org> <20061203182841.GA21598@code0.codespeak.net> Message-ID: <45731DE2.3020005@v.loewis.de> Armin Rigo schrieb: > Ah, distutils are distributed in the base package now, but not the > 'config' subdirectory of a standard library's normal installation, which > makes distutils a bit useless. > > I should go and file a bug, I guess. The reason why I did not do it so > far, is that the fact that no one did tends to show that they really > think distributing 'config' is not the "right" thing to do. A reasoning > along the lines of: "this subdir contains include files, so it should go > in the python-dev package". I'm not too interested in fighting this > kind of preconception. It really depends on what you are trying to do with distutils. If you want to package or install a distutils module, you need much more than distutils: you may also need a compiler, you may need the complete set of Python header files, and you may need header files for the libraries your package depends on. So if you use distutils to install a package, it's IMO reasonable to require that the -dev package is installed. People use distutils for other purposes today as well, and these purposes might be supported now. Regards, Martin From martin at v.loewis.de Sun Dec 3 19:59:00 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 03 Dec 2006 19:59:00 +0100 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> Message-ID: <45731E74.7010101@v.loewis.de> Fredrik Lundh schrieb: >> Ah, right; I misread his proposal as saying that m[:] should return >> [m[0]] + list(m.groups()) (rather, I expected that m.groups() would >> include m.group(0)). > > match groups are numbered 1..N, not 0..(N-1), in both the API and in the > RE syntax (and we don't have much control over the latter). py> m = re.match("a(b)","ab") py> m.group(0) 'ab' py> m.group(1) 'b' >> To answer your first question: it is clearly groups that you want >> to index, just as the .group() method indexes groups. > > so what should len(m) do? That's a question: should len be supported at all? If so, it's clear that len(m) == len(m[:]). Regards, Martin From fredrik at pythonware.com Sun Dec 3 20:14:17 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sun, 03 Dec 2006 20:14:17 +0100 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <45731E74.7010101@v.loewis.de> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> Message-ID: Martin v. L?wis wrote: >> match groups are numbered 1..N, not 0..(N-1), in both the API and in the >> RE syntax (and we don't have much control over the latter). > > py> m = re.match("a(b)","ab") > py> m.group(0) > 'ab' > py> m.group(1) > 'b' 0 isn't a group, it's an alias for the full match. From tjreedy at udel.edu Sun Dec 3 20:52:24 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 3 Dec 2006 14:52:24 -0500 Subject: [Python-Dev] a feature i'd like to see in python #1: betteriteration control References: <4572C1F1.1050600@666.com> Message-ID: "Ben Wing" wrote in message news:4572C1F1.1050600 at 666.com... > many times writing somewhat complex loops over lists i've found the need > to sometimes delete an item from the list. currently there's no easy > way to do so; basically, you have to write something like > > i = 0 > while i < len(list): > el = list[i] > ...do something... > if el should be deleted: > del list[i] > else: > i += 1 My view: while loops are the standard construct for iteration. They give you complete control over the control variable(s). Hence no need for 'better iteration control'. The above is easy enough and quite clear. > note that you can't write > > for x in list: > > or even > > for i in xrange(len(list)): For loops are for the common case of straightforwardly iterating over a sequence. They are efficient both to write and execute. Let's not mess with them. -1 > note also that you need to do some trickiness to adjust the index > appropriately when deleting. Iterate in reverse and no index adjustment is needed Terry Jan Reedy From martin at v.loewis.de Sun Dec 3 21:16:39 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 03 Dec 2006 21:16:39 +0100 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> Message-ID: <457330A7.6040401@v.loewis.de> Fredrik Lundh schrieb: >>> match groups are numbered 1..N, not 0..(N-1), in both the API and in the >>> RE syntax (and we don't have much control over the latter). >> py> m = re.match("a(b)","ab") >> py> m.group(0) >> 'ab' >> py> m.group(1) >> 'b' > > 0 isn't a group, it's an alias for the full match. So what is the proper term for the things that the .group() method returns? According to http://docs.python.org/lib/match-objects.html it returns "subgroups of the match". So the things to be indexed in this proposal are subgroups of the match. Regards, Martin From steve at holdenweb.com Sun Dec 3 21:41:04 2006 From: steve at holdenweb.com (Steve Holden) Date: Sun, 03 Dec 2006 20:41:04 +0000 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <457330A7.6040401@v.loewis.de> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> Message-ID: <45733660.7040202@holdenweb.com> Martin v. L?wis wrote: > Fredrik Lundh schrieb: >>>> match groups are numbered 1..N, not 0..(N-1), in both the API and in the >>>> RE syntax (and we don't have much control over the latter). >>> py> m = re.match("a(b)","ab") >>> py> m.group(0) >>> 'ab' >>> py> m.group(1) >>> 'b' >> 0 isn't a group, it's an alias for the full match. > > So what is the proper term for the things that the .group() method > returns? According to > > http://docs.python.org/lib/match-objects.html > > it returns "subgroups of the match". > > So the things to be indexed in this proposal are subgroups of the > match. > Precisely. But your example had only one group "(b)" in it, which is retrieved using m.group(1). So the subgroups are numbered starting from 1 and subgroup 0 is a special case which returns the whole match. I know what the Zen says about special cases, but in this case the rules were apparently broken with impunity. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden From steve at holdenweb.com Sun Dec 3 21:41:04 2006 From: steve at holdenweb.com (Steve Holden) Date: Sun, 03 Dec 2006 20:41:04 +0000 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <457330A7.6040401@v.loewis.de> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> Message-ID: <45733660.7040202@holdenweb.com> Martin v. L?wis wrote: > Fredrik Lundh schrieb: >>>> match groups are numbered 1..N, not 0..(N-1), in both the API and in the >>>> RE syntax (and we don't have much control over the latter). >>> py> m = re.match("a(b)","ab") >>> py> m.group(0) >>> 'ab' >>> py> m.group(1) >>> 'b' >> 0 isn't a group, it's an alias for the full match. > > So what is the proper term for the things that the .group() method > returns? According to > > http://docs.python.org/lib/match-objects.html > > it returns "subgroups of the match". > > So the things to be indexed in this proposal are subgroups of the > match. > Precisely. But your example had only one group "(b)" in it, which is retrieved using m.group(1). So the subgroups are numbered starting from 1 and subgroup 0 is a special case which returns the whole match. I know what the Zen says about special cases, but in this case the rules were apparently broken with impunity. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden From martin at v.loewis.de Sun Dec 3 22:01:04 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 03 Dec 2006 22:01:04 +0100 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <45733660.7040202@holdenweb.com> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> Message-ID: <45733B10.9090300@v.loewis.de> Steve Holden schrieb: > Precisely. But your example had only one group "(b)" in it, which is > retrieved using m.group(1). So the subgroups are numbered starting from > 1 and subgroup 0 is a special case which returns the whole match. > > I know what the Zen says about special cases, but in this case the rules > were apparently broken with impunity. Well, the proposal was to interpret m[i] as m.group(i), for all values of i. I can't see anything confusing with that. Regards, Martin From fredrik at pythonware.com Sun Dec 3 22:07:07 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sun, 03 Dec 2006 22:07:07 +0100 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <45733B10.9090300@v.loewis.de> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> <45733B10.9090300@v.loewis.de> Message-ID: Martin v. L?wis wrote: >> I know what the Zen says about special cases, but in this case the rules >> were apparently broken with impunity. > > Well, the proposal was to interpret m[i] as m.group(i), for all values > of i. I can't see anything confusing with that. it can quickly become rather confusing if you also interpret m[:] as m.groups(), not to mention if you add len() and arbitrary slicing to the mix. what about m[] and m[i,j,k], btw? From steve at holdenweb.com Sun Dec 3 22:12:43 2006 From: steve at holdenweb.com (Steve Holden) Date: Sun, 03 Dec 2006 21:12:43 +0000 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <45733B10.9090300@v.loewis.de> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> <45733B10.9090300@v.loewis.de> Message-ID: <45733DCB.4070309@holdenweb.com> Martin v. L?wis wrote: > Steve Holden schrieb: >> Precisely. But your example had only one group "(b)" in it, which is >> retrieved using m.group(1). So the subgroups are numbered starting from >> 1 and subgroup 0 is a special case which returns the whole match. >> >> I know what the Zen says about special cases, but in this case the rules >> were apparently broken with impunity. > > Well, the proposal was to interpret m[i] as m.group(i), for all values > of i. I can't see anything confusing with that. > I don't suppose that would be any more confusing than the present case. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden From steve at holdenweb.com Sun Dec 3 22:12:43 2006 From: steve at holdenweb.com (Steve Holden) Date: Sun, 03 Dec 2006 21:12:43 +0000 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <45733B10.9090300@v.loewis.de> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> <45733B10.9090300@v.loewis.de> Message-ID: <45733DCB.4070309@holdenweb.com> Martin v. L?wis wrote: > Steve Holden schrieb: >> Precisely. But your example had only one group "(b)" in it, which is >> retrieved using m.group(1). So the subgroups are numbered starting from >> 1 and subgroup 0 is a special case which returns the whole match. >> >> I know what the Zen says about special cases, but in this case the rules >> were apparently broken with impunity. > > Well, the proposal was to interpret m[i] as m.group(i), for all values > of i. I can't see anything confusing with that. > I don't suppose that would be any more confusing than the present case. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden From g.brandl at gmx.net Sun Dec 3 22:42:26 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 03 Dec 2006 22:42:26 +0100 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> <45733B10.9090300@v.loewis.de> Message-ID: Fredrik Lundh wrote: > Martin v. L?wis wrote: > >>> I know what the Zen says about special cases, but in this case the rules >>> were apparently broken with impunity. >> >> Well, the proposal was to interpret m[i] as m.group(i), for all values >> of i. I can't see anything confusing with that. > > it can quickly become rather confusing if you also interpret m[:] as > m.groups(), not to mention if you add len() and arbitrary slicing to > the mix. what about m[] and m[i,j,k], btw? What about them? They aren't supposed to be supported by every object that allows subscript, are they? And why not just not implement len()? As for the [:] <-> groups() issue, [:] would have to be consistent with indexing and return the whole match and the subgroups. (Or, the API could be overhauled completely of course, remember it's Py3k.) Georg From g.brandl at gmx.net Sun Dec 3 22:44:54 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 03 Dec 2006 22:44:54 +0100 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> <45733B10.9090300@v.loewis.de> Message-ID: Georg Brandl wrote: > (Or, the API could be overhauled completely of course, remember it's Py3k.) Erm, no, it's not. Strike that remark. Georg From martin at v.loewis.de Sun Dec 3 23:17:30 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 03 Dec 2006 23:17:30 +0100 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> <45733B10.9090300@v.loewis.de> Message-ID: <45734CFA.9090806@v.loewis.de> Fredrik Lundh schrieb: > it can quickly become rather confusing if you also interpret m[:] as > m.groups(), not to mention if you add len() and arbitrary slicing to > the mix. what about m[] and m[i,j,k], btw? I take it that you are objecting to that feature, then? Regards, Martin From greg.ewing at canterbury.ac.nz Mon Dec 4 02:11:07 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 04 Dec 2006 14:11:07 +1300 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <45733660.7040202@holdenweb.com> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> Message-ID: <457375AB.3070702@canterbury.ac.nz> Steve Holden wrote: > So the subgroups are numbered starting from > 1 and subgroup 0 is a special case which returns the whole match. But the subgroups can be nested too, so it's not really as special as all that. -- Greg From ben at 666.com Mon Dec 4 03:26:54 2006 From: ben at 666.com (Ben Wing) Date: Sun, 03 Dec 2006 20:26:54 -0600 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <4573199D.7050907@v.loewis.de> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <20061203171500.GA9392@panix.com> <4573199D.7050907@v.loewis.de> Message-ID: <4573876E.3040706@666.com> Martin v. L?wis wrote: > Aahz schrieb: > >>>> this one is fairly simple. if `m' is a match object, i'd like to be >>>> able to write m[1] instead of m.group(1). (similarly, m[:] should return >>>> the same as list(m.groups()).) this would remove some of the verbosity >>>> of regexp code, with probably a net gain in readability; certainly no loss. >>>> >>> Please post a patch to sf.net/projects/python (or its successor). >>> >> Given the list of issues and subsequent discussion so far, I think a PEP >> will be required. This needs more documentation than the typical patch. >> > > I disagree. So far, nobody has spoken against the proposed feature. It's > really a small addition of a new method to an existing type. Entire > classes have been added to the standard library without a PEP. People > can still criticize the patch when its posted (and it's not clear that > the OP is even willing to produce a patch). > > i've never worked up a python patch before, but i imagine this wouldn't be too hard. it seems that m[1] should be m.group(1), and everything else should follow. i forgot about m[0] when making my slice proposal; i suppose then that m[:] should just do what we expect, and m[1:] = m.groups(). len(m) = 1 + number of groups, m['name'] = m.group('name'). the only strangeness here is the numbering of groups starting at 1, and making 0 be a special case. this isn't any more (or less) of a problem for the indexing form than it is for m.group(), and it's well known from various other languages. we could always consider making groups start at 0 for python 3000, but this seems to me like a gratuitous incompatibility with the rest of the world. ben From ben at 666.com Mon Dec 4 03:35:58 2006 From: ben at 666.com (Ben Wing) Date: Sun, 03 Dec 2006 20:35:58 -0600 Subject: [Python-Dev] a feature i'd like to see in python #1: better iteration control In-Reply-To: <20061203123920.GA11625@seldon> References: <4572C1F1.1050600@666.com> <20061203123920.GA11625@seldon> Message-ID: <4573898E.9050508@666.com> Brian Harring wrote: > On Sun, Dec 03, 2006 at 06:24:17AM -0600, Ben Wing wrote: > >> many times writing somewhat complex loops over lists i've found the need >> to sometimes delete an item from the list. currently there's no easy >> way to do so; basically, you have to write something like >> >> i = 0 >> while i < len(list): >> el = list[i] >> ...do something... >> if el should be deleted: >> del list[i] >> else: >> i += 1 >> >> note that you can't write >> >> for x in list: >> >> or even >> >> for i in xrange(len(list)): >> >> note also that you need to do some trickiness to adjust the index >> appropriately when deleting. >> >> i'd much rather see something like: >> >> for x:iter in list: >> ...do something... >> if x should be deleted: >> iter.delete() >> >> the idea is that you have a way of retrieving both the element itself >> and the iterator for the element, so that you can then call methods on >> the iterator. it shouldn't be too hard to implement iter.delete(), as >> well as iter.insert() and similar functions. (the recent changes to the >> generator protocol in 2.5 might help.) >> >> the only question then is how to access the iterator. the syntax i've >> proposed, with `x:iter', seems fairly logical (note that parallels with >> slice notation, which also uses a colon) and doesn't introduce any new >> operators. (comma is impossible since `for x,iter in list:' already has >> a meaning) >> >> btw someone is probably going to come out and say "why don't you just >> use a list comprehension with an `if' clause? the problems are [1] i'd >> like this to be destructive; >> > > Just use slice assignment. > > l = list(xrange(100)) > l2 = [x for x in l if x > 50] > l[:] = l2[:] > > >> [2] i'd like this to work over non-lists as well, e.g. hash-tables; >> > > There in is the sucky part; iterator protocol is simple; what you're > proposing is extending iterators so that they recall the last value > (else iter.delete() would not do anything), which... eh. > > Think it sucks, to say the least ;) > > Simple example of where this gets ugly is in iterating over a file. > > with some more thought i see that a language extension to `for' isn't needed, as you can write something like it = iter(foo) for x in it: ... but i still don't see why supporting iter.delete() is so wrong. clearly it doesn't need to work on files or other such things where it doesn't make sense. before you diss this completely, note that java supports exactly the same thing: http://java.sun.com/j2se/1.4.2/docs/api/java/util/Iterator.html remove public void *remove*() Removes from the underlying collection the last element returned by the iterator (optional operation). This method can be called only once per call to next. The behavior of an iterator is unspecified if the underlying collection is modified while the iteration is in progress in any way other than by calling this method. *Throws:* |UnsupportedOperationException <../../java/lang/UnsupportedOperationException.html>| - if the remove operation is not supported by this Iterator. |IllegalStateException <../../java/lang/IllegalStateException.html>| - if the next method has not yet been called, or the remove method has already been called after the last call to the next method. From ben at 666.com Mon Dec 4 04:06:07 2006 From: ben at 666.com (Ben Wing) Date: Sun, 03 Dec 2006 21:06:07 -0600 Subject: [Python-Dev] features i'd like [Python 3000] ... #3: fix super() Message-ID: <4573909F.5050405@666.com> i don't like the current super() at all. having to type super(Foo, self).__init__(...) is messy, hard to remember, and error-prone. it also introduces an unfortunate dependency in that the name of the class (Foo) has to be hard-coded in the call, and if you change the class name you also have to go and change all super() calls -- more chances for errors. as a result, i imagine there's a strong urge to just hardcode the name of the parent -- a bad idea, and the reason why super() was introduced in the first place. instead, `super' should work like in other languages. a couple of ideas: -- super.meth(args) calls the superclass method `meth', in the same way that super(Foo, self).meth(args) currently works. (this is the same syntax as in java. this probably requires making `super' be a keyword, although it might be possible to hack this by examining the current call stack.) -- as an alternative or in addition, super(args) would work like super.meth(args) if we're currently inside of `meth' in a subclass. i suspect that 90% of the time or more, `super' is used to chain to the superclass version of an overridden method, and this optimizes for the common case. it makes for one less possibility of misspelling `__init__' and one less thing that requires renaming if a method is renamed or code copied to another method (granted, this isn't such a big deal as in the case of class renaming). if the form `super.meth(args)' isn't also supported, then a call of this sort would have to be made through the introspection API. (i think it would be a good idea to have this functionality available through the introspection API, in any case. currently i don't see any obvious way in module `inspect' to find out which class a call to `super(...).meth(...)' is invoked on, although maybe a loop with inspect.getmro() and hasattr() would work. it would be nice to have a function like inspect.getsuper(class, 'meth'), i think.) ben From ferringb at gmail.com Mon Dec 4 04:06:46 2006 From: ferringb at gmail.com (Brian Harring) Date: Sun, 3 Dec 2006 19:06:46 -0800 Subject: [Python-Dev] a feature i'd like to see in python #1: better iteration control In-Reply-To: <4573898E.9050508@666.com> References: <4572C1F1.1050600@666.com> <20061203123920.GA11625@seldon> <4573898E.9050508@666.com> Message-ID: <20061204030645.GB11625@seldon> On Sun, Dec 03, 2006 at 08:35:58PM -0600, Ben Wing wrote: > but i still don't see why supporting iter.delete() is so wrong. clearly > it doesn't need to work on files or other such things where it doesn't > make sense. > > before you diss this completely, note that java supports exactly the > same thing: > > http://java.sun.com/j2se/1.4.2/docs/api/java/util/Iterator.html Not all iterators would support remove; that right there is a bit of an issue since right now, the only exception you need to expect for iterator protocol is StopIteration being thrown when the iterator has nothing more to yield. So, it's no longer simpler, which is a bit of a con in my opinion. Question is, where _would_ it work? Doesn't really make much sense for generators (doable with 2.5, but most generators are just that, generators, not modifiable views), doesn't make sense for itertools.* for the most part, since it's combination of iterators. For dict; it actually *cannot* work. You can't remove keys from a dict as you're iterating over it (can change the val of a key, but not remove the key). So iter.delete would require fair bit of changes internally to dict, either tracking what it's yielded already, or forcing iterkeys to actually be iter(keys()) (creating an intermediate list), which is worse for memory usage and general performance. Set's suffer the same thing; can't change what it contains while iterating, have to restart the iteration after a removal/addition. Tuples are immutable, so end of discusion there. Leaves lists... which personally, I view as a mostly bad thing to be doing anyways. Trying to pop an item out of the middle of a list results in shifting everything right of it one spot to the left; this sucks from a performance standpoint, again, worst case, quad. Now... occasionally, have to do it admittedly. But it's not something you actaully want to be doing in your code all that much- admittedly generating a new list to avoid that hit also sucks somewhat, but the worst case there is far more behaved, a temp trade of space vs runtime. What I'm trying to get at is that what iter(list).next().delete() would do isn't a good thing to paper over, it makes the op look like it costs nothing when it can cost a _lot_. Unless I'm missing something, the only real usage of this is a list (could do it on files also, although that would suffer the same issue as a list, just worse via truncate calls). Would work better on collections.deque, but deque is a linked list and used rather selectively. So... why add it, if it's basically for one major type, and it's not a good idea to blindly do such an action on that type in the first place? ~harring -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20061203/fcae3584/attachment.pgp From ben at 666.com Mon Dec 4 04:27:51 2006 From: ben at 666.com (Ben Wing) Date: Sun, 03 Dec 2006 21:27:51 -0600 Subject: [Python-Dev] a feature i'd like to see in python #1: better iteration control In-Reply-To: <20061204030645.GB11625@seldon> References: <4572C1F1.1050600@666.com> <20061203123920.GA11625@seldon> <4573898E.9050508@666.com> <20061204030645.GB11625@seldon> Message-ID: <457395B7.8040701@666.com> Brian Harring wrote: > On Sun, Dec 03, 2006 at 08:35:58PM -0600, Ben Wing wrote: > >> but i still don't see why supporting iter.delete() is so wrong. clearly >> it doesn't need to work on files or other such things where it doesn't >> make sense. >> >> before you diss this completely, note that java supports exactly the >> same thing: >> >> http://java.sun.com/j2se/1.4.2/docs/api/java/util/Iterator.html >> > > Not all iterators would support remove; that right there is a bit of > an issue since right now, the only exception you need to expect for > iterator protocol is StopIteration being thrown when the iterator has > nothing more to yield. > > So, it's no longer simpler, which is a bit of a con in my opinion. > well, it's not like adding a new exception to any existing iterator method. it would only occur if you call iter.delete() in the wrong context. > For dict; it actually *cannot* work. You can't remove keys from a > dict as you're iterating over it (can change the val of a key, but not > remove the key). So iter.delete would require fair bit of changes > internally to dict, either tracking what it's yielded already, or > forcing iterkeys to actually be iter(keys()) (creating an intermediate > list), which is worse for memory usage and general performance. > > Set's suffer the same thing; can't change what it contains while > iterating, have to restart the iteration after a removal/addition. > > Leaves lists... which personally, I view as a mostly bad thing to be > doing anyways. Trying to pop an item out of the middle of a list > results in shifting everything right of it one spot to the left; this > sucks from a performance standpoint, again, worst case, quad. > > Now... occasionally, have to do it admittedly. But it's not something > you actaully want to be doing in your code all that much- admittedly > generating a new list to avoid that hit also sucks somewhat, but the > worst case there is far more behaved, a temp trade of space vs > runtime. > > What I'm trying to get at is that what iter(list).next().delete() > would do isn't a good thing to paper over, it makes the op look like > it costs nothing when it can cost a _lot_. > i do see your point. i was trying to remember what i ended up doing when i ran into this issue before, and in fact i ended up just making a new list and appending all the elements i didn't want to delete. i see you'd get N^2 behavior if you deleted lots of elements from a list, but you get the same thing currently if you call `del list[i]' a lot; it's not obvious that iter.delete() actually "papers over" the cost any more than `del list[i]' does. > Unless I'm missing something, the only real usage of this is a list > (could do it on files also, although that would suffer the same > issue as a list, just worse via truncate calls). Would work better on > collections.deque, but deque is a linked list and used rather > selectively. > > i actually think now this would be best for hash tables and such. copying a large hash table is a real drag, and using the iterator protocol is the *only* way to iterate over the elements of a hash table. in fact, i imagine this is exactly why java added this functionality. certainly in lisp, the `maphash' function explicitly allows you to delete the current element being iterated over, for the same reason. however, for hash tables there's no reason to use iter.delete(). you should just be able to write `del hash[x]'. is this disallowed currently? if so, it seems like something that should be fixed. > So... why add it, if it's basically for one major type, and it's > not a good idea to blindly do such an action on that type in the first > place? > ok, i see your point and i retract the suggestion. ben From ben at 666.com Mon Dec 4 05:32:54 2006 From: ben at 666.com (Ben Wing) Date: Sun, 03 Dec 2006 22:32:54 -0600 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #1: betteriteration control In-Reply-To: References: <4572C1F1.1050600@666.com> Message-ID: <4573A4F6.3030106@666.com> Terry Reedy wrote: > Iterate in reverse and no index adjustment is needed > > > a good idea, thanks. From hasan.diwan at gmail.com Mon Dec 4 04:13:43 2006 From: hasan.diwan at gmail.com (Hasan Diwan) Date: Sun, 3 Dec 2006 22:13:43 -0500 Subject: [Python-Dev] Makefile.pre.in Patch Message-ID: <2cda2fc90612031913k6241e0e8i9cf56f083eee193b@mail.gmail.com> The attached patch ensures that the $(DESTDIR) exists before installing the built binaries. It has been tested on Mac OS X. The patch is attached. -- Cheers, Hasan Diwan -------------- next part -------------- A non-text attachment was scrubbed... Name: Makefile.patch Type: application/octet-stream Size: 3638 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20061203/cc3e18b2/attachment.obj From ben at 666.com Mon Dec 4 06:12:16 2006 From: ben at 666.com (Ben Wing) Date: Sun, 03 Dec 2006 23:12:16 -0600 Subject: [Python-Dev] features i'd like [Python 3000?] ... #4: interpolated strings ala perl Message-ID: <4573AE30.9080305@666.com> sorry to be casting multiple ideas at once to the list. i've been looking into other languages recently and reading the recent PEP's and such and it's helped crystallize ideas about what could be better about python. i see in PEP 3101 that there's some work going on to fix up the string formatting capabilities of python. it looks good to me but it still doesn't really address the lack of a simple interpolated string mechanism, as in perl or ruby. i find myself constantly writing stuff like text="Family: %s" % self.name maybe_errout("%s, line %s: %s\n" % (title, lineno, errstr)) def __str__(self): return "CCGFeatval(%s, parents=%s, licensing=%s)" % ( (self.name, self.parents, self.licensing)) and lots of similar examples that are just crying out for perl-style variable interpolation. the proposals in PEP 3101 don't help much; i'd get instead something like maybe_errout("{0}, line {1}: {2}\n".format(title, lineno, errstr)) which isn't any better than the current % notation, or something like maybe_errout("{title}, line {lineno}: {errstr}\n".format(title=title, lineno=lineno, errstr=errstr)) where i have to repeat each interpolated variable three times. yuck yuck yuck. how about something nice like maybe_errout(i"[title], line [lineno]: [errstr]\n") or (with the first example above) text=i"Family: [self.name]" or (third example above) def __str__(self): return i"CCGFeatval([self.name], parents=[self.parents], licensing=[self.licensing])" the advantage of these is the same as for all such interpolations: the interpolated variable is logically placed exactly where it will be substituted, rather than somewhere else, with the brain needing to do some additional cross-referencing. `i' in front of a string indicates an "interpolated" string just like `r' in the same position means "raw string". if you think this is too invisible, you could maybe use `in' or something easier to see. however, i could see a case being made for combining both `i' and `r' on the same string, and so using a single letter seems to make the most sense. formatting params can follow, e.g. print i"The value of [stock[x]] is [stockval[x]:%6.2f]" some comments: 1. i suggest brackets here so as to parallel but not interfere with PEP 3101, which uses braces; PEP 3101 is somewhat orthogonal to this proposal and the two might want to coexist. i think using something like brackets or braces is better than perl's $ convention (it explicitly indicates the boundaries of the interpolation) and simpler than ruby's #{...} or make's ${...}; since strings are only interpolated if you specifically indicate this, there's less need to use funny characters to avoid accidental interpolation. 2. as shown in the last example, format specifiers can be added to an interpolation, as in PEP 3101. maybe the % is unnecessary. 3. as shown in various examples, things other than just straight variables can be interpolated. the above examples include array/hash references and object attributes. it's not exactly clear what should and shouldn't be allowed. one possibility is just to allow an arbitrary expression. PEP 3101 isn't going to do this because it's working on existing strings (including ones that may come from the user), and so allowing arbitrary expressions could lead to security holes. but here, we're talking about immediate strings, so security holes of this sort should not a concern. 4. the semantics of an interpolated string are exactly that of a series of string concatenations, e.g. return i"CCGFeatval([self.name], parents=[self.parents], licensing=[self.licensing])" is equivalent to return "CCGFeatval(" + self.name + ", parents=" + self.parents + ", licensing=" + self.licensing + ")" ben From ben at 666.com Mon Dec 4 06:19:06 2006 From: ben at 666.com (Ben Wing) Date: Sun, 03 Dec 2006 23:19:06 -0600 Subject: [Python-Dev] features i'd like [Python 3000?] ... #4: interpolated strings ala perl In-Reply-To: <4573AE30.9080305@666.com> References: <4573AE30.9080305@666.com> Message-ID: <4573AFCA.6020202@666.com> Ben Wing wrote: > [interpolated strings] btw if people think this idea is good, i can certainly write it up in PEP form. ben From fredrik at pythonware.com Mon Dec 4 07:38:36 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Mon, 04 Dec 2006 07:38:36 +0100 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <45734CFA.9090806@v.loewis.de> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> <45733B10.9090300@v.loewis.de> <45734CFA.9090806@v.loewis.de> Message-ID: Martin v. L?wis wrote: >> it can quickly become rather confusing if you also interpret m[:] as >> m.groups(), not to mention if you add len() and arbitrary slicing to >> the mix. what about m[] and m[i,j,k], btw? > > I take it that you are objecting to that feature, then? I haven't seen a complete and self-consistent proposal yet, so that's not easy to say. From ben at 666.com Mon Dec 4 09:04:39 2006 From: ben at 666.com (Ben Wing) Date: Mon, 04 Dec 2006 02:04:39 -0600 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> <45733B10.9090300@v.loewis.de> <45734CFA.9090806@v.loewis.de> Message-ID: <4573D697.6060802@666.com> Fredrik Lundh wrote: > Martin v. L?wis wrote: > > >>> it can quickly become rather confusing if you also interpret m[:] as >>> m.groups(), not to mention if you add len() and arbitrary slicing to >>> the mix. what about m[] and m[i,j,k], btw? >>> >> I take it that you are objecting to that feature, then? >> > > I haven't seen a complete and self-consistent proposal yet, so that's > not easy to say. > > > > my current proposal can be summarized: 1. m[x] == m.group(x) for x an integer >= 0. 2. all other sequence properties should be consistent with this numbering and with the view of `m' as basically an array. 3. m[name] == m.group(name) for name a string; names are aliases for group numbers. this implies, for example, that negative indices count from the end, that len(m) == 1 + m.lastindex, that the expression `m[1:]' should be the same as `m.groups()', that `foo in m' is true if `foo' is equal to any group in m or to the whole string, etc. property 3 should also probably imply that names should be allowed as slice indices -- a name is just an alias for a group number, and should behave the same way. an alternative would be to view a match object as a hash table. then, slices would presumably be disallowed, and `foo in m' would be true if `foo' is a group number in range, or a name of a group. but i don't like this as much; for example, it's not clear what len(m) should return in the case of a named group -- does it count the group once (since a name is just an alias), or twice? (btw i never really thought until now about the inconsistency in the 'in' operator between arrays and hash tables.) ben From jcarlson at uci.edu Mon Dec 4 09:32:46 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 04 Dec 2006 00:32:46 -0800 Subject: [Python-Dev] features i'd like [Python 3000?] ... #4: interpolated strings ala perl In-Reply-To: <4573AE30.9080305@666.com> References: <4573AE30.9080305@666.com> Message-ID: <20061204001736.BBC1.JCARLSON@uci.edu> Ben Wing wrote: > > sorry to be casting multiple ideas at once to the list. i've been > looking into other languages recently and reading the recent PEP's and > such and it's helped crystallize ideas about what could be better about > python. > > i see in PEP 3101 that there's some work going on to fix up the string > formatting capabilities of python. it looks good to me but it still > doesn't really address the lack of a simple interpolated string > mechanism, as in perl or ruby. i find myself constantly writing stuff like > > text="Family: %s" % self.name [snip] > maybe_errout(i"[title], line [lineno]: [errstr]\n") This can be done now via: maybe_errout("%(title)s, line %(lineno)i: %(errstr)s\n"%locals()) > def __str__(self): > return i"CCGFeatval([self.name], parents=[self.parents], > licensing=[self.licensing])" With the proper mapping, this is trivial... class namelookup: def __init__(self, namespace): self.ns = namespace def __getitem__(self, name): ns = self.ns names = name.split('.') try: if names[0] in ns: ns = ns[names[0]] names = names[1:] except IndexError: raise KeyError("bad namespace") for n in names: if hasattr(ns, n): ns = getattr(ns, n) else: return "" return ns >>> class foo: ... a = 1 ... b = 2 ... >>> foo = foo() >>> print "%(foo.b)i + %(foo.a)i"%namelookup(locals()) 2 + 1 >>> While I understand the desire for better string interpolation, many of your needs can be covered with pre-existing string formatting. You can even write a handler for "{2} and {1} or {3}" as "%(2)s and %(1)s or % (3)s"%positional(a, b, c). Toss it into site.py, and you can use it whenever. If you want this to be fully in regards to Py3k, post on the py3k mailing list: python-3000 at python.org - Josiah From ncoghlan at gmail.com Mon Dec 4 11:58:17 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 04 Dec 2006 20:58:17 +1000 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <4573876E.3040706@666.com> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <20061203171500.GA9392@panix.com> <4573199D.7050907@v.loewis.de> <4573876E.3040706@666.com> Message-ID: <4573FF49.9000406@gmail.com> Ben Wing wrote: > the only strangeness here is the numbering of groups starting at 1, and > making 0 be a special case. this isn't any more (or less) of a problem > for the indexing form than it is for m.group(), and it's well known from > various other languages. we could always consider making groups start > at 0 for python 3000, but this seems to me like a gratuitous > incompatibility with the rest of the world. As Greg pointed out, this is just a special case of the fact that subgroups can be nested with the ordering governed by the location of the left parenthesis: .>>> import re .>>> m = re.match("a(b(c))", "abc") .>>> m.group(0) 'abc' .>>> m.group(1) 'bc' .>>> m.group(2) 'c' That said, I like the definitions in your last message: len(m) == 1 + len(m.groups()) m[:] == [m.group(0)] + m.groups() all(m[i] == m.group(i) for i in range(len(m))) all(m[k] == m.group(k) for k in m.groupdict().keys()) The internally inconsistent* m.group() and m.groups() methods could even be slated for removal in Py3k (replaced by the subscript operations). Cheers, Nick. *The inconsistency being that group() considers the whole match to be group 0, while groups() does not. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Mon Dec 4 12:13:37 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 04 Dec 2006 21:13:37 +1000 Subject: [Python-Dev] features i'd like [Python 3000] ... #3: fix super() In-Reply-To: <4573909F.5050405@666.com> References: <4573909F.5050405@666.com> Message-ID: <457402E1.6060705@gmail.com> Ben Wing wrote: > i don't like the current super() at all. having to type super(Foo, > self).__init__(...) is messy, hard to remember, and error-prone. Yup. > it > also introduces an unfortunate dependency in that the name of the class > (Foo) has to be hard-coded in the call, and if you change the class name > you also have to go and change all super() calls -- more chances for > errors. as a result, i imagine there's a strong urge to just hardcode > the name of the parent -- a bad idea, and the reason why super() was > introduced in the first place. Also yup. The fact that these drawbacks are fairly obvious, yet nothing has been done about them should suggest something to you about the difficulty of actually solving this problem in a reliable fashion ;) The Python Cookbook has a few interesting approaches to making cooperative super calls easier to write, but they all tend to have some kind of drawback that makes them inappropriate for making part of the language core. > instead, `super' should work like in other languages. a couple of ideas: > > -- super.meth(args) calls the superclass method `meth', in the same way > that super(Foo, self).meth(args) currently works. (this is the same > syntax as in java. this probably requires making `super' be a keyword, > although it might be possible to hack this by examining the current call > stack.) It's the implementation that's the real sticking point here, so without code this idea isn't likely to go anywhere. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From arigo at tunes.org Mon Dec 4 12:16:47 2006 From: arigo at tunes.org (Armin Rigo) Date: Mon, 4 Dec 2006 12:16:47 +0100 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <45731DE2.3020005@v.loewis.de> References: <5.1.1.6.0.20061127090652.04341cf8@sparrow.telecommunity.com> <200611290053.17199.anthony@interlink.com.au> <20061129112354.GA30665@code0.codespeak.net> <1164917149.31269.344.camel@localhost> <456F51C5.8080500@holdenweb.com> <1164930163.31269.407.camel@localhost> <20061201042709.GE10614@steerpike.home.puzzling.org> <20061203182841.GA21598@code0.codespeak.net> <45731DE2.3020005@v.loewis.de> Message-ID: <20061204111647.GA30709@code0.codespeak.net> Hi Martin, On Sun, Dec 03, 2006 at 07:56:34PM +0100, "Martin v. L?wis" wrote: > People use distutils for other purposes today as well, and these > purposes might be supported now. OK, makes some kind of sense. I suppose (as you point out in another thread) that the issue is that distros generally don't have a way to say that if a machine has both, say, tcl/tk and python installed, then _tkinter.so would be a natural thing to have as well. Similarly, if it has got both python and gcc (which is quite common nowadays) then 'config' is a minor addition that would be quite handy -- even more so than _tkinter.so, because unless you go and hack, it's not obvious to me how you can install a 'config' in your home dir and have the system python pick it up. A bientot, Armin From tonynelson at georgeanelson.com Mon Dec 4 18:48:32 2006 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Mon, 4 Dec 2006 12:48:32 -0500 Subject: [Python-Dev] Polling with Pending Calls? Message-ID: I think I have a need to handle *nix signals through polling in a library. It looks like chaining Pending Calls is almost the way to do it, but I see that doing so would make the interpreter edgy. The RPM library takes (steals) the signal handling away from its client application. It has good reason to be so paranoid, but it breaks the handling keyboard interrupts, especially if rpmlib is used in the normal way: opened at the beginning, various things are done by the app, closed at the end. If there is an extended period in the middle where no calls are made to rpmlib (say, in yum during the downloading of packages or package headers), then responst to a keyboard interrupt can be delayed for /minutes/! Yum is presently doing something awful to work around that issue. It is possible to poll rpmlib to find if there is a pending keyboard interrupt. Client applications could have such polls sprinkled throughout them. I think getting yum, for example, to do that might be politically difficult. I'm hoping to propose a patch to rpmlib's Python bindings to do the polling automagically. Looking at Python's normal signal handling, I see that Py_AddPendingCall() and Py_MakePendingCalls(), and PyEvel_EvalFrameEx()'s ticker check are how signals and other async events are done. I could imagine making rpmlib's Python bindings add a Pending Call when the library is loaded (or some such), and that Pending Call would make a quick check of rpmlib's caught signals flags and then call Py_AddPendingCall() on itself. It appears that this would work, and is almost the expected thing to do, but unfortunately it would cause the ticker check to think that Py_MakePendingCalls() had failed and needed to be called again ASAP, which would drastically slow down the interpreter. Is there a right way to get the Python interpreter to poll something, or should I look for another approach? [I hope this message doesn't spend too many days in the grey list limbo.] -- ____________________________________________________________________ TonyN.:' The Great Writ ' is no more. From gjcarneiro at gmail.com Mon Dec 4 19:07:01 2006 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Mon, 4 Dec 2006 18:07:01 +0000 Subject: [Python-Dev] Polling with Pending Calls? In-Reply-To: References: Message-ID: This patch may interest you: http://www.python.org/sf/1564547 Not sure it completely solves your case, but it's at least close to your problem. On 12/4/06, Tony Nelson wrote: > > I think I have a need to handle *nix signals through polling in a library. > It looks like chaining Pending Calls is almost the way to do it, but I see > that doing so would make the interpreter edgy. > > The RPM library takes (steals) the signal handling away from its client > application. It has good reason to be so paranoid, but it breaks the > handling keyboard interrupts, especially if rpmlib is used in the normal > way: opened at the beginning, various things are done by the app, closed > at the end. If there is an extended period in the middle where no calls > are made to rpmlib (say, in yum during the downloading of packages or > package headers), then responst to a keyboard interrupt can be delayed for > /minutes/! Yum is presently doing something awful to work around that > issue. > > It is possible to poll rpmlib to find if there is a pending keyboard > interrupt. Client applications could have such polls sprinkled throughout > them. I think getting yum, for example, to do that might be politically > difficult. I'm hoping to propose a patch to rpmlib's Python bindings to > do > the polling automagically. > > Looking at Python's normal signal handling, I see that Py_AddPendingCall() > and Py_MakePendingCalls(), and PyEvel_EvalFrameEx()'s ticker check are > how > signals and other async events are done. I could imagine making rpmlib's > Python bindings add a Pending Call when the library is loaded (or some > such), and that Pending Call would make a quick check of rpmlib's caught > signals flags and then call Py_AddPendingCall() on itself. It appears > that > this would work, and is almost the expected thing to do, but unfortunately > it would cause the ticker check to think that Py_MakePendingCalls() had > failed and needed to be called again ASAP, which would drastically slow > down the interpreter. > > Is there a right way to get the Python interpreter to poll something, or > should I look for another approach? > > [I hope this message doesn't spend too many days in the grey list limbo.] > -- > ____________________________________________________________________ > TonyN.:' The Great Writ > ' is no more. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com > -- Gustavo J. A. M. Carneiro "The universe is always one step beyond logic." -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20061204/029bf450/attachment.htm From martin at v.loewis.de Mon Dec 4 19:15:35 2006 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 04 Dec 2006 19:15:35 +0100 Subject: [Python-Dev] a feature i'd like to see in python #1: better iteration control In-Reply-To: <20061204030645.GB11625@seldon> References: <4572C1F1.1050600@666.com> <20061203123920.GA11625@seldon> <4573898E.9050508@666.com> <20061204030645.GB11625@seldon> Message-ID: <457465C7.1040305@v.loewis.de> Brian Harring schrieb: > For dict; it actually *cannot* work. You can't remove keys from a > dict as you're iterating over it (can change the val of a key, but not > remove the key). I think this is incorrect. The implementation could well support it, putting a dummy object into the deleted key (which deletion needs to do, anyway). > So iter.delete would require fair bit of changes > internally to dict, either tracking what it's yielded already, or > forcing iterkeys to actually be iter(keys()) (creating an intermediate > list), which is worse for memory usage and general performance. I don't think either is necessary; deletion could occur "directly". > Set's suffer the same thing; can't change what it contains while > iterating, have to restart the iteration after a removal/addition. Again, I think that's incorrect. > Tuples are immutable, so end of discusion there. True. > Now... occasionally, have to do it admittedly. But it's not something > you actaully want to be doing in your code all that much- admittedly > generating a new list to avoid that hit also sucks somewhat, but the > worst case there is far more behaved, a temp trade of space vs > runtime. Also true. Regards, Martin From martin at v.loewis.de Mon Dec 4 19:21:33 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 04 Dec 2006 19:21:33 +0100 Subject: [Python-Dev] a feature i'd like to see in python #1: better iteration control In-Reply-To: <457395B7.8040701@666.com> References: <4572C1F1.1050600@666.com> <20061203123920.GA11625@seldon> <4573898E.9050508@666.com> <20061204030645.GB11625@seldon> <457395B7.8040701@666.com> Message-ID: <4574672D.8030804@v.loewis.de> Ben Wing schrieb: > i do see your point. i was trying to remember what i ended up doing > when i ran into this issue before, and in fact i ended up just making a > new list and appending all the elements i didn't want to delete. i see > you'd get N^2 behavior if you deleted lots of elements from a list, but > you get the same thing currently if you call `del list[i]' a lot; it's > not obvious that iter.delete() actually "papers over" the cost any more > than `del list[i]' does. No, it wouldn't. OTOH, copying into a new list and then performing slice-assignment is O(N). So performance-wise, it is (surprisingly) asymptotically better to create a new list. > however, for hash tables there's no reason to use iter.delete(). you > should just be able to write `del hash[x]'. is this disallowed > currently? Try for yourself: py> x={1:2,3:4} py> for k in x: ... if k < 2: ... del x[k] ... Traceback (most recent call last): File "", line 1, in ? RuntimeError: dictionary changed size during iteration > if so, it seems like something that should be fixed. It's difficult to fix: deleting an item may cause a resizing/rehashing of the entire dictionary, hence the dictionary is looked while being iterated over. I *think* that one could support deletion (but not addition); this would need further investigation. Regards, Martin From martin at v.loewis.de Mon Dec 4 19:22:32 2006 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 04 Dec 2006 19:22:32 +0100 Subject: [Python-Dev] Makefile.pre.in Patch In-Reply-To: <2cda2fc90612031913k6241e0e8i9cf56f083eee193b@mail.gmail.com> References: <2cda2fc90612031913k6241e0e8i9cf56f083eee193b@mail.gmail.com> Message-ID: <45746768.6000502@v.loewis.de> Hasan Diwan schrieb: > The attached patch ensures that the $(DESTDIR) exists before > installing the built binaries. It has been tested on Mac OS X. The > patch is attached. Please post patches to sf.net/projects/python Thanks, Martin From martin at v.loewis.de Mon Dec 4 19:37:42 2006 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 04 Dec 2006 19:37:42 +0100 Subject: [Python-Dev] Python and LSB: Meeting Report Message-ID: <45746AF6.5000506@v.loewis.de> We just had (the first day of) an LSB face-to-face meeting today [1], where Python was discussed. I started with presenting my slides, and stated what I see as main goals: - Allow Python scripts to run unmodified across Linux distributions - Optional: Allow extension modules to be used in binary form across distributions - Optional: Allow extension modules to be used in binary form across distributions During the discussion, Robert Schweikert proposed another goal, from the view of a proprietary software vendor: - Allow .pyc files to run unmodified across Linux distributions The discussion evolved around backwards compatibility. I'll start separate threads for these; here is the summary: - LSB attempts to guarantee backwards compatibility with previous versions (starting with 4.0; currently, 3.2 is targeted). Linux distributions implementing future LSB versions should be backwards-compatible with previous LSB versions. - Removal of features can only occur after three major LSB releases; this means all features will stay for about 6 years - If LSB integrated a specific version now, that version will have to be supported by vendors for those 6 years - given that the Python ABI changes frequently over time, this makes standardization of the ABI difficult - even for API (i.e. library interfaces), backwards-compatibility would be difficult: for those packages/classes/modules/functions that become part of the LSB API, stability would be desirable. Regards, Martin [1] http://www.freestandards.org/en/LSB_face-to-face_(December_2006) [2]http://www.freestandards.org/images/6/66/Lsb-f2f-200612-python.pdf From martin at v.loewis.de Mon Dec 4 19:40:34 2006 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 04 Dec 2006 19:40:34 +0100 Subject: [Python-Dev] LSB: pyc stability Message-ID: <45746BA2.2020601@v.loewis.de> Some people (Robert Schweikert) requested byte-code stability at the LSB meeting: LSB should standardize a certain version of the byte code, and then future versions of Python should continue to support this byte code version. I explained that this is currently not supported, but would be technically possible; somebody would have to implement it. What do you think? Regards, Martin From tonynelson at georgeanelson.com Mon Dec 4 19:44:44 2006 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Mon, 4 Dec 2006 13:44:44 -0500 Subject: [Python-Dev] Polling with Pending Calls? In-Reply-To: References: Message-ID: At 6:07 PM +0000 12/4/06, Gustavo Carneiro wrote: >This patch may interest you: >http://www.python.org/sf/1564547 > >Not sure it completely solves your case, but it's at least close to >your problem. I don't think that patch is useful in this case. This case is not stuck in some extension module's poll() call. The signal handler is not Python's nor is it under my control (so no chance that it would look at some new pipe), though the rpmlib Python bindings can look at the state bits it sets. The Python interpreter is running full-bore when the secret rpmlib SIGINT state is needed. I think the patch is for the exact /opposite/ of my problem. -- ____________________________________________________________________ TonyN.:' The Great Writ ' is no more. From martin at v.loewis.de Mon Dec 4 19:49:29 2006 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 04 Dec 2006 19:49:29 +0100 Subject: [Python-Dev] LSB: Selection of a Python version Message-ID: <45746DB9.9050501@v.loewis.de> At the LSB meeting, there was a brief discussion of what Python version should be incorporated into LSB. This is more an issue if ABI compatibility for the C ABI is desired, but even if only script portability is a goal, application developers will need to know what APIs they have available in LSB 3.2 and which they don't. LSB codifies existing practice, rather than defining "anticipating" standards, so things get only into LSB if they are already commonplace in distributions. Currently, Python 2.4 is widely available in Linux distributions; Python 2.5 is not and (apparently) will not be included in the next RHEL release (which apparently is coming soon). So it looks like LSB will standardize on Python 2.4 (which will also be the default version in the next Debian release). Regards, Martin From brett at python.org Mon Dec 4 19:54:06 2006 From: brett at python.org (Brett Cannon) Date: Mon, 4 Dec 2006 10:54:06 -0800 Subject: [Python-Dev] LSB: pyc stability In-Reply-To: <45746BA2.2020601@v.loewis.de> References: <45746BA2.2020601@v.loewis.de> Message-ID: On 12/4/06, "Martin v. L?wis" wrote: > > Some people (Robert Schweikert) requested byte-code stability at the > LSB meeting: LSB should standardize a certain version of the byte code, > and then future versions of Python should continue to support this > byte code version. Did they say why they wanted to distribute bytecode files? I assume it is either for space considerations or they think it will help to protect their IP somehow. I explained that this is currently not supported, but would be > technically possible; somebody would have to implement it. I guess we just don't remove opcodes and have import allow bytecode files within a range of magic numbers instead of having to match a specific one. What do you think? I am personally -0. I would prefer not to start supporting people pushing bytecode files around, but I am not going to throw a fit we do end up going down that road. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20061204/02d2bed8/attachment.htm From martin at v.loewis.de Mon Dec 4 20:05:13 2006 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 04 Dec 2006 20:05:13 +0100 Subject: [Python-Dev] LSB: Binary compatibility Message-ID: <45747169.4050003@v.loewis.de> At the LSB meeting, Jeff Licquia asked whether Python could provide binary compatibility with previous versions by use of ELF symbol versioning. In ELF symbol versioning, you can have multiple definitions for the same symbol; clients (e.g. extension modules) would refer to a specific version. During static linking, the most recent (?) version of the symbol is coded into the client object. With symbol versioning, you can change the implementation of a function and even its interface, and it will be compatible as long as you keep the original version around. My first reaction is that this is difficult due to the usage of function-like macros. However, if we replaced those with C functions (perhaps has a compile-time choice), and if we would also hide the layout of structures, I think providing binary compatibility (with a certain baseline version, or multiple of these) would be possible. Of course, several things need to be considered, e.g. - making Py_INCREF/Py_DECREF functions is likely a bad idea for performance reasons. OTOH, it seems safe that Py_INCREF/Py_DECREF can remain as-is for the rest of 2.x. - hiding PyTypeObject is a bad idea for source compatibility. OTOH, we already try to make only compatible changes to it (except when we don't :-), so exposing this as stable ABI might actually work. - certain kinds of modules likely need to be ruled out, e.g. modules that extend the layout of existing types (it would fail to compile because the base struct becomes an incomplete type). All in all, I think providing binary compatibility would be feasible, and should be attempted. What do you think? Regards, Martin From martin at v.loewis.de Mon Dec 4 20:08:27 2006 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 04 Dec 2006 20:08:27 +0100 Subject: [Python-Dev] LSB: pyc stability In-Reply-To: References: <45746BA2.2020601@v.loewis.de> Message-ID: <4574722B.6050903@v.loewis.de> Brett Cannon schrieb: > Some people (Robert Schweikert) requested byte-code stability at the > LSB meeting: LSB should standardize a certain version of the byte code, > and then future versions of Python should continue to support this > byte code version. > > > Did they say why they wanted to distribute bytecode files? I assume it > is either for space considerations or they think it will help to protect > their IP somehow. It's to protect the IP (i.e. for proprietary software). They are aware of decompyle, but still consider byte-code only distribution of their code necessary for protection. Regards, Martin From tonynelson at georgeanelson.com Mon Dec 4 20:37:12 2006 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Mon, 4 Dec 2006 14:37:12 -0500 Subject: [Python-Dev] Polling with Pending Calls? In-Reply-To: References: Message-ID: At 12:48 PM -0500 12/4/06, Tony Nelson wrote: >I think I have a need to handle *nix signals through polling in a library. >It looks like chaining Pending Calls is almost the way to do it, but I see >that doing so would make the interpreter edgy. ... Bah. Sorry to have put noise on the list. I'm obviously too close to the problem to see the simple solution of threading.Timer. Checking once or twice a second should be good enough. Sorry to bother you all. -- ____________________________________________________________________ TonyN.:' The Great Writ ' is no more. From sluggoster at gmail.com Mon Dec 4 20:38:20 2006 From: sluggoster at gmail.com (Mike Orr) Date: Mon, 4 Dec 2006 11:38:20 -0800 Subject: [Python-Dev] Virtual Python (was Re: Python and the Linux Standard Base (LSB)) Message-ID: <6e9196d20612041138n383b51a1yf99ea27da21229eb@mail.gmail.com> This may be a bit too FAQ-ish for some people but I'll post it in case anybody benefits from the answer. On 11/30/06, Phillip J. Eby wrote: > At 02:46 PM 11/30/2006 -0800, Mike Orr wrote: > >Speaking of Virtual Python [1], I've heard some people recommending it > >as a general solution to the "this library breaks that other > >application" problem and "this app needs a different version of X > >library than that other app does". > > It was actually created to help people who don't have root access (e.g. in > shared web hosting accounts) get their "own" Python. It does work for > other things, but I haven't been especially recommending it for anything > besides that, since eggs take care of multi-version app/library support > quite nicely. > > Indeeed, the virtual-python approach is itself unnecessary now, as > easy_install has much better PYTHONPATH support now, than when > virtual-python was created. What does this last part mean? Virtual Python is still one of the four recommended ways to set up a custom install location in the EasyInstall docs. http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations The other approaches work fine for giving each user a private install dir, but don't address the case of the same user wanting different install dirs for different projects. For instance, when exploring Pylons or TurboGears which install a lot of packages I may not otherwise want, I create a Virtual Python for each of them. If I'm developing an application under Virtual Python, I can see at a glance which packages my project needs installed. I can't think of any other way except Virtual Python to do this. Another point. Setuptools seems to have Two Ways To Do Things regarding package activiation. easy_install puts the latest-installed egg version in its .pth file so it's automatically available via a naive "import". This happens to clutters sys.path with a more entries than some people desire. Meanwhile, programs can use pkg_resources to activate a package or version that may not already be in sys.path. Is this the Way Of The Future? Should people start using pkg_resources for all packages they import? (That would also allow one to remove little-used packages from easy-install.pth.) Finally, one can use ~/.pydistutils.cfg to specify an install location, but that again allows only one location per user. And in the case of root, it can't distinguish between manually-installed packages and OS-installed packages, which the admin might want in different directories (/usr/local/lib/python vs /usr/lib/pythonVERSION). This hasn't been an issue because OSes haven't been using easy_install or eggs, but it will be when they start doing so. Do the PYTHONPATH improvements make it possible to just put a directory on your PYTHONPATH and have Python process .pth files in it without using the site.addsitedir() hack? That would probably be my biggest wishlist item. -- Mike Orr From ntoronto at cs.byu.edu Mon Dec 4 20:21:27 2006 From: ntoronto at cs.byu.edu (Neil Toronto) Date: Mon, 04 Dec 2006 12:21:27 -0700 Subject: [Python-Dev] LSB: pyc stability In-Reply-To: <4574722B.6050903@v.loewis.de> References: <45746BA2.2020601@v.loewis.de> <4574722B.6050903@v.loewis.de> Message-ID: <45747537.30602@cs.byu.edu> Martin v. L?wis wrote: > Brett Cannon schrieb: > >> Did they say why they wanted to distribute bytecode files? I assume it >> is either for space considerations or they think it will help to protect >> their IP somehow. >> > > It's to protect the IP (i.e. for proprietary software). They are aware > of decompyle, but still consider byte-code only distribution of their > code necessary for protection. It sounds like a trade secret issue. You have to take reasonable measures to protect trade secrets in order for them to be legally recognized as such. I wouldn't be surprised if compiling to bytecode counts. There are similar provisions in copyright due to the DMCA (which seems to require nothing stronger than, say, ROT26), but I don't think this is the right context for that. Space considerations shouldn't be much of an issue, since you can (and should in many cases) distribute your code in a ZIP file, and code compresses quite well. Can Python import modules from encrypted ZIP files? That'd be an interesting way to protect a trade secret, and probably safer (in the courts) than distributing bytecode. Neil From tjreedy at udel.edu Mon Dec 4 21:12:41 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 4 Dec 2006 15:12:41 -0500 Subject: [Python-Dev] Polling with Pending Calls? References: Message-ID: "Tony Nelson" wrote in message news:p04330105c19a0bbda4a4@[192.168.123.162]... > Is there a right way to get the Python interpreter to poll something, or > should I look for another approach? Usage questions belong on the general python list or comp.lang.python. The development list is mostly about developing the next Python releases. From brett at python.org Mon Dec 4 21:37:06 2006 From: brett at python.org (Brett Cannon) Date: Mon, 4 Dec 2006 12:37:06 -0800 Subject: [Python-Dev] LSB: pyc stability In-Reply-To: <45747537.30602@cs.byu.edu> References: <45746BA2.2020601@v.loewis.de> <4574722B.6050903@v.loewis.de> <45747537.30602@cs.byu.edu> Message-ID: On 12/4/06, Neil Toronto wrote: > > Martin v. L?wis wrote: > > Brett Cannon schrieb: > > > >> Did they say why they wanted to distribute bytecode files? I assume > it > >> is either for space considerations or they think it will help to > protect > >> their IP somehow. > >> > > > > It's to protect the IP (i.e. for proprietary software). They are aware > > of decompyle, but still consider byte-code only distribution of their > > code necessary for protection. > > It sounds like a trade secret issue. You have to take reasonable > measures to protect trade secrets in order for them to be legally > recognized as such. I wouldn't be surprised if compiling to bytecode > counts. There are similar provisions in copyright due to the DMCA (which > seems to require nothing stronger than, say, ROT26), but I don't think > this is the right context for that. > > Space considerations shouldn't be much of an issue, since you can (and > should in many cases) distribute your code in a ZIP file, and code > compresses quite well. Can Python import modules from encrypted ZIP > files? That'd be an interesting way to protect a trade secret, and > probably safer (in the courts) than distributing bytecode. No. Part of the issue is there is no way to syntactically support entering the password to decrypt the zip file. One solution, though, would be to write an importer/loader object for zip files that was instantiated with a password and that could then handle the decryption. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20061204/aa82d7fa/attachment.htm From jcarlson at uci.edu Mon Dec 4 21:46:45 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 04 Dec 2006 12:46:45 -0800 Subject: [Python-Dev] LSB: pyc stability In-Reply-To: <45747537.30602@cs.byu.edu> References: <4574722B.6050903@v.loewis.de> <45747537.30602@cs.byu.edu> Message-ID: <20061204124344.BBCD.JCARLSON@uci.edu> Neil Toronto wrote: > compresses quite well. Can Python import modules from encrypted ZIP > files? That'd be an interesting way to protect a trade secret, and > probably safer (in the courts) than distributing bytecode. An import hook would do it. - Josiah From pje at telecommunity.com Mon Dec 4 22:49:26 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 04 Dec 2006 16:49:26 -0500 Subject: [Python-Dev] Virtual Python (was Re: Python and the Linux Standard Base (LSB)) In-Reply-To: <6e9196d20612041138n383b51a1yf99ea27da21229eb@mail.gmail.co m> Message-ID: <5.1.1.6.0.20061204164056.028b27a0@sparrow.telecommunity.com> At 11:38 AM 12/4/2006 -0800, Mike Orr wrote: >The other approaches work fine for giving each user a private install >dir, but don't address the case of the same user wanting different >install dirs for different projects. For instance, when exploring >Pylons or TurboGears which install a lot of packages I may not >otherwise want, I create a Virtual Python for each of them. If I'm >developing an application under Virtual Python, I can see at a glance >which packages my project needs installed. I can't think of any other >way except Virtual Python to do this. Simply install the packages to an arbitrary directory using -m (--multi-version), and allow the scripts to be installed to the same directory. When the scripts are run, they'll find their eggs in the script directory. Neither PYTHONPATH manipulation nor virtual Pythons are needed to achieve this - it's a self-contained single-application directory. (It will still have to import pkg_resources from a copy of setuptools installed somewhere else, e.g. on PYTHONPATH or in site-packages, or someday perhaps the stdlib.) >Another point. Setuptools seems to have Two Ways To Do Things >regarding package activiation. easy_install puts the latest-installed >egg version in its .pth file so it's automatically available via a >naive "import". This happens to clutters sys.path with a more >entries than some people desire. Using -m (--multi-version) suppresses this behavior. The "just works" behavior of automatically adding to sys.path is just the default. > Meanwhile, programs can use >pkg_resources to activate a package or version that may not already be >in sys.path. Is this the Way Of The Future? Should people start >using pkg_resources for all packages they import? No. Setuptools automatically wraps generated scripts with pkg_resources-based activation, so there's almost never a need to request packages. So, if you need dynamic access for a project, just create yourself a setup.py for it and put the requirements in there. Then run "setup.py develop" to generate wrappers for your scripts. The wrappers will do all the activation for you when they're run. >Finally, one can use ~/.pydistutils.cfg to specify an install >location, but that again allows only one location per user. One *default* location per user... per current directory. Since it's based on distutils, easy_install always reads setup.cfg from the current directory, which can set the defaults. And of course, using -d (--install-dir) on the command line lets you specify a one-off target. >Do the PYTHONPATH improvements make it possible to just put a >directory on your PYTHONPATH and have Python process .pth files in it >without using the site.addsitedir() hack? That would probably be my >biggest wishlist item. Yes, it does -- but it only works for packages installed by easy_install; a special 'site.py' is added to the directory that adds the necessary hooks around the "real" 'site' module's processing. From greg at electricrain.com Mon Dec 4 23:10:59 2006 From: greg at electricrain.com (Gregory P. Smith) Date: Mon, 4 Dec 2006 14:10:59 -0800 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <4573199D.7050907@v.loewis.de> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <20061203171500.GA9392@panix.com> <4573199D.7050907@v.loewis.de> Message-ID: <20061204221058.GC10487@electricrain.com> On Sun, Dec 03, 2006 at 07:38:21PM +0100, "Martin v. L?wis" wrote: > Aahz schrieb: > >>> this one is fairly simple. if `m' is a match object, i'd like to be > >>> able to write m[1] instead of m.group(1). (similarly, m[:] should return > >>> the same as list(m.groups()).) this would remove some of the verbosity > >>> of regexp code, with probably a net gain in readability; certainly no loss. > >> Please post a patch to sf.net/projects/python (or its successor). > > > > Given the list of issues and subsequent discussion so far, I think a PEP > > will be required. This needs more documentation than the typical patch. > > I disagree. So far, nobody has spoken against the proposed feature. It's > really a small addition of a new method to an existing type. Entire > classes have been added to the standard library without a PEP. People > can still criticize the patch when its posted (and it's not clear that > the OP is even willing to produce a patch). Agreed. Just implement it including test cases testing and demoing the corner cases. Making match objects have sequence and dict behaviour for groups is imnsho just common sense. -greg From greg.ewing at canterbury.ac.nz Tue Dec 5 01:15:58 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 05 Dec 2006 13:15:58 +1300 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <4573FF49.9000406@gmail.com> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <20061203171500.GA9392@panix.com> <4573199D.7050907@v.loewis.de> <4573876E.3040706@666.com> <4573FF49.9000406@gmail.com> Message-ID: <4574BA3E.2060609@canterbury.ac.nz> Nick Coghlan wrote: > *The inconsistency being that group() considers the whole match to be group 0, > while groups() does not. The real inconsistency seems to be that the groups are being treated as an array when they're really a tree. Maybe a different API altogether would be better, e.g. m.range --> the whole match m.subgroups[i] --> another match object with its own range and subgroups attributes -- Greg From ferringb at gmail.com Tue Dec 5 01:51:59 2006 From: ferringb at gmail.com (Brian Harring) Date: Mon, 4 Dec 2006 16:51:59 -0800 Subject: [Python-Dev] a feature i'd like to see in python #1: better iteration control In-Reply-To: <457465C7.1040305@v.loewis.de> References: <4572C1F1.1050600@666.com> <20061203123920.GA11625@seldon> <4573898E.9050508@666.com> <20061204030645.GB11625@seldon> <457465C7.1040305@v.loewis.de> Message-ID: <20061205005159.GA24833@seldon> On Mon, Dec 04, 2006 at 07:15:35PM +0100, "Martin v. L??wis" wrote: > Brian Harring schrieb: > > For dict; it actually *cannot* work. You can't remove keys from a > > dict as you're iterating over it (can change the val of a key, but not > > remove the key). > > I think this is incorrect. The implementation could well support it, > putting a dummy object into the deleted key (which deletion needs > to do, anyway). The implementation already uses a sentinel (NULL)- point was that it does not support iteration over a dict that's being deleted from *currently* though. One thing to note; delitem is the easy case. Allowing for mutating the mapping as you're iterating via delitem implies that setitem should work also; setitem however can trigger a resize. Finally, if dicts were ever modified to shrink based on load, the resize there would be an issue. Mind you I've not seen proposals of that sort, just pointing out the potential. > > So iter.delete would require fair bit of changes > > internally to dict, either tracking what it's yielded already, or > > forcing iterkeys to actually be iter(keys()) (creating an intermediate > > list), which is worse for memory usage and general performance. > > I don't think either is necessary; deletion could occur "directly". > > > Set's suffer the same thing; can't change what it contains while > > iterating, have to restart the iteration after a removal/addition. > > Again, I think that's incorrect. Again, was refering to existing implementation (and long standing rules/conventions regarding it). Set suffers the same issue with setitem meanwhile. In my opinion, no point in doing the deltitem modification without a matching setitem. Not saying I think the modification is worth it mind you, just that there should be symmetry ;) ~harring -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20061204/22bc439d/attachment.pgp From python-dev at zesty.ca Tue Dec 5 06:14:40 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Mon, 4 Dec 2006 23:14:40 -0600 (CST) Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> <45733B10.9090300@v.loewis.de> Message-ID: On Sun, 3 Dec 2006, Fredrik Lundh wrote: > Martin v. L?wis wrote: > > Well, the proposal was to interpret m[i] as m.group(i), for all values > > of i. I can't see anything confusing with that. > > it can quickly become rather confusing if you also interpret m[:] as > m.groups(), not to mention if you add len() and arbitrary slicing to > the mix. what about m[] and m[i,j,k], btw? I'd say, don't pretend m is a sequence. Pretend it's a mapping. Then the conceptual issues go away. -- ?!ng From martin at v.loewis.de Tue Dec 5 07:02:55 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Dec 2006 07:02:55 +0100 Subject: [Python-Dev] Python and LSB: Meeting Report In-Reply-To: <20061204224223.GA25682@panix.com> References: <45746AF6.5000506@v.loewis.de> <20061204224223.GA25682@panix.com> Message-ID: <45750B8F.3010705@v.loewis.de> > [private, but feel free to respond on-list] >> - Allow Python scripts to run unmodified across Linux distributions >> - Optional: Allow extension modules to be used in binary form across >> distributions >> - Optional: Allow extension modules to be used in binary form across >> distributions > > Was this duplication of last two points cut'n'paste error or what? Oops, yes, it should have read - Optional: Allow installation of binary foreign Python add-on packages Thanks, Martin From nnorwitz at gmail.com Tue Dec 5 07:16:10 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 4 Dec 2006 22:16:10 -0800 Subject: [Python-Dev] LSB: Selection of a Python version In-Reply-To: <45746DB9.9050501@v.loewis.de> References: <45746DB9.9050501@v.loewis.de> Message-ID: What, if any, impact do you think the LSB should have wrt maintaining 2.4? n On 12/4/06, "Martin v. L?wis" wrote: > At the LSB meeting, there was a brief discussion of what Python > version should be incorporated into LSB. This is more an issue > if ABI compatibility for the C ABI is desired, but even if only > script portability is a goal, application developers will need > to know what APIs they have available in LSB 3.2 and which they > don't. > > LSB codifies existing practice, rather than defining "anticipating" > standards, so things get only into LSB if they are already > commonplace in distributions. Currently, Python 2.4 is widely > available in Linux distributions; Python 2.5 is not and (apparently) > will not be included in the next RHEL release (which apparently > is coming soon). > > So it looks like LSB will standardize on Python 2.4 (which will also > be the default version in the next Debian release). > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com > From martin at v.loewis.de Tue Dec 5 07:21:01 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Dec 2006 07:21:01 +0100 Subject: [Python-Dev] a feature i'd like to see in python #1: better iteration control In-Reply-To: <20061205005159.GA24833@seldon> References: <4572C1F1.1050600@666.com> <20061203123920.GA11625@seldon> <4573898E.9050508@666.com> <20061204030645.GB11625@seldon> <457465C7.1040305@v.loewis.de> <20061205005159.GA24833@seldon> Message-ID: <45750FCD.6060608@v.loewis.de> Brian Harring schrieb: >> I think this is incorrect. The implementation could well support it, >> putting a dummy object into the deleted key (which deletion needs >> to do, anyway). > > The implementation already uses a sentinel (NULL)- That' not the full truth. The implementation has a separate object (called the dummy object, see dictobject.c:140 in the trunk) to fill put in for deleted objects. You can't use NULL here because then the open addressing would break. NULL is used as a sentinel for the open addressing; the dummy object indicates that you have to continue searching. > point was that it > does not support iteration over a dict that's being deleted from > *currently* though. Yes, but I believe that's out of fear that you have to do resizing; iteration cannot be supported in the presence of resizing. Deletion does not involve resizing, instead, ma_fill won't decrease during deletion, so the next addition may trigger resizing if the fill level goes above 2/3. > One thing to note; delitem is the easy case. Allowing for mutating > the mapping as you're iterating via delitem implies that setitem > should work also; setitem however can trigger a resize. So that implication is wrong, no? Of course, setitem for an existing key could be allowed; that's also a meaningful operation while you are iterating over a dictionary. > Finally, if dicts were ever modified to shrink based on load, the > resize there would be an issue. Mind you I've not seen proposals of > that sort, just pointing out the potential. You'd have to defer the resizing while the dictionary is locked. > In my opinion, no point in doing the deltitem modification without a > matching setitem. Not saying I think the modification is worth it > mind you, just that there should be symmetry ;) IMO, it is clear that this symmetry is *not* necessary, for the reason that we are discussing: if iterators where to support a delete operation, then it would be possible to provide that for dict iterators. It wouldn't be necessary to support any other updates; it wouldn't even be necessary to support del d[k], even if k is the key returned from the iterator's .next(). Regards, Martin From martin at v.loewis.de Tue Dec 5 07:30:56 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Dec 2006 07:30:56 +0100 Subject: [Python-Dev] LSB: Selection of a Python version In-Reply-To: References: <45746DB9.9050501@v.loewis.de> Message-ID: <45751220.6020301@v.loewis.de> Neal Norwitz schrieb: > What, if any, impact do you think the LSB should have wrt maintaining 2.4? People at the meeting specifically said whether security patches would still be applied to older releases, and for how many older releases. Linux distributors are hesitant to make commitments to maintain a software package if they know that their upstream source doesn't provide security patches anymore. I think we should come up with a policy for dealing with security patches (there haven't been that many in the past, anyway); I believe users (i.e. vendors in this case) would be happy with the procedure we followed for 2.3: just produce a source release integrating the security patches; no need for binary releases (as they will produce binaries themselves). So I think a public statement that we will support 2.4 with security patches for a while longer (and perhaps with security patches *only*) would be a good thing - independent of the LSB, actually. Regards, Martin From nnorwitz at gmail.com Tue Dec 5 07:44:31 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 4 Dec 2006 22:44:31 -0800 Subject: [Python-Dev] LSB: Binary compatibility In-Reply-To: <45747169.4050003@v.loewis.de> References: <45747169.4050003@v.loewis.de> Message-ID: On 12/4/06, "Martin v. L?wis" wrote: > At the LSB meeting, Jeff Licquia asked whether Python could provide > binary compatibility with previous versions by use of ELF symbol > versioning. In ELF symbol versioning, you can have multiple > definitions for the same symbol; clients (e.g. extension modules) > would refer to a specific version. During static linking, the most > recent (?) version of the symbol is coded into the client object. > > With symbol versioning, you can change the implementation of a > function and even its interface, and it will be compatible as > long as you keep the original version around. [in a separate message that Python 2.4 is the first targetted version] > All in all, I think providing binary compatibility would > be feasible, and should be attempted. What do you think? Let's assume that 2.4 is the first LSB version. The ABI is different for 2.4 and 2.5. We can't change the ABI for 2.5 since it's already released and our policy is to keep it constant. Where does that leave us for moving forward? Would we have to modify 2.5 to make it entirely compatible with 2.4? Can we skip 2.5 altogether and worry about making 2.6 compatible with 2.4? Even if so, that means no matter what we still need to make 2.4 compatible APIs going forward, right? That will require some archeaology to determine preciesly what we changed from 2.4. Is our current practice of the following change acceptable? Original: PyAPI_FUNC(xxx) Py_Foo(xxx); New version: PyAPI_FUNC(xxx) Py_FooEx(xxx, yyy); #define Py_Foo(x) Py_FooEx(x, NULL) And similarly in the C file, but keeping the original PyFoo with something like: xxx Py_FooEx(...) { ... } #undef Py_Foo xxx Py_Foo(xxx) { return Py_FooEx(xxx, NULL); } I'm not arguing against the LSB at all. I think it's a good thing, but I'm trying to understand what work needs to be done and how we can acheive it in the face of apparent competing requirements. I also realize this is a one time cost and we don't need to address the details right now. n From fredrik at pythonware.com Tue Dec 5 07:46:39 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 05 Dec 2006 07:46:39 +0100 Subject: [Python-Dev] a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> <45733B10.9090300@v.loewis.de> Message-ID: Ka-Ping Yee wrote: > I'd say, don't pretend m is a sequence. Pretend it's a mapping. > Then the conceptual issues go away. almost; that would mean returning KeyError instead of IndexError for groups that don't exist, which means that the common pattern a, b, c = m.groups() cannot be rewritten as _, a, b, c = m which would, perhaps, be a bit unfortunate. taking everything into account, I think we should simply map __getitem__ to group, and stop there. no len(), no slicing, no sequence or mapping semantics. if people want full sequence behaviour with len and slicing and iterators and whatnot, they can do list(m) first. From ben at 666.com Tue Dec 5 10:02:39 2006 From: ben at 666.com (Ben Wing) Date: Tue, 05 Dec 2006 03:02:39 -0600 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> <45733B10.9090300@v.loewis.de> Message-ID: <457535AF.6050106@666.com> Fredrik Lundh wrote: > Ka-Ping Yee wrote: > > >> I'd say, don't pretend m is a sequence. Pretend it's a mapping. >> Then the conceptual issues go away. >> > > almost; that would mean returning KeyError instead of IndexError for > groups that don't exist, which means that the common pattern > > a, b, c = m.groups() > > cannot be rewritten as > > _, a, b, c = m > > which would, perhaps, be a bit unfortunate. > > taking everything into account, I think we should simply map __getitem__ > to group, and stop there. no len(), no slicing, no sequence or mapping > semantics. if people want full sequence behaviour with len and slicing > and iterators and whatnot, they can do list(m) first. > > i'm ok either way -- that is, either with the proposal i previously published, or with this restricted idea. ben From fredrik at pythonware.com Tue Dec 5 10:08:23 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 05 Dec 2006 10:08:23 +0100 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <457535AF.6050106@666.com> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> <45733B10.9090300@v.loewis.de> <457535AF.6050106@666.com> Message-ID: Ben Wing wrote: > i'm ok either way -- that is, either with the proposal i previously > published, or with this restricted idea. ok, I'll whip up a patch for the minimal version of the proposal, if nobody beats me to it (all that's needed is a as_sequence struct with a item slot that basically just calls match_getslice_by_index). From anthony at interlink.com.au Tue Dec 5 10:35:35 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue, 5 Dec 2006 20:35:35 +1100 Subject: [Python-Dev] LSB: Selection of a Python version In-Reply-To: <45751220.6020301@v.loewis.de> References: <45746DB9.9050501@v.loewis.de> <45751220.6020301@v.loewis.de> Message-ID: <200612052035.35984.anthony@interlink.com.au> On Tuesday 05 December 2006 17:30, Martin v. L?wis wrote: > People at the meeting specifically said whether security patches > would still be applied to older releases, and for how many older > releases. Linux distributors are hesitant to make commitments to > maintain a software package if they know that their upstream > source doesn't provide security patches anymore. I agree we should have a written policy. At the moment, my policy is this: normal bugfixes for 2.5 critical crasher bugfix releases for 2.5 and 2.4 security bugfix releases for 2.5, 2.4, and 2.3. I'm planning on dropping 2.3 from this list sometime next year. After that, I guess we can produce officially blessed patches or something. > I think we should come up with a policy for dealing with security > patches (there haven't been that many in the past, anyway); I > believe users (i.e. vendors in this case) would be happy with the > procedure we followed for 2.3: just produce a source release > integrating the security patches; no need for binary releases (as > they will produce binaries themselves). Depends - while 2.4 is officially "retired" now, if a security bugfix that affects windows/OS X comes up, I think we should still cut binary releases. > So I think a public statement that we will support 2.4 with > security patches for a while longer (and perhaps with security > patches *only*) would be a good thing - independent of the LSB, > actually. Well, I don't know what sort of public statement you want to issue, but will this do? (Wearing my release manager hat) Anthony -- Anthony Baxter It's never too late to have a happy childhood. From skip at pobox.com Tue Dec 5 15:09:55 2006 From: skip at pobox.com (skip at pobox.com) Date: Tue, 5 Dec 2006 08:09:55 -0600 Subject: [Python-Dev] LSB: Binary compatibility In-Reply-To: References: <45747169.4050003@v.loewis.de> Message-ID: <17781.32179.14808.124857@montanaro.dyndns.org> >> All in all, I think providing binary compatibility would be feasible, >> and should be attempted. What do you think? Neal> Let's assume that 2.4 is the first LSB version. The ABI is Neal> different for 2.4 and 2.5. We can't change the ABI for 2.5 since Neal> it's already released and our policy is to keep it constant. It seems that adhering to LSB's constraints is going to create a new set of problems for Python development. It's unclear to me what LSB brings to Python other than a bunch of new headaches. Skip From alastair at alastairs-place.net Tue Dec 5 15:30:38 2006 From: alastair at alastairs-place.net (Alastair Houghton) Date: Tue, 5 Dec 2006 14:30:38 +0000 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <457535AF.6050106@666.com> References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> <45733B10.9090300@v.loewis.de> <457535AF.6050106@666.com> Message-ID: On 5 Dec 2006, at 09:02, Ben Wing wrote: > Fredrik Lundh wrote: >> Ka-Ping Yee wrote: >> taking everything into account, I think we should simply map >> __getitem__ >> to group, and stop there. no len(), no slicing, no sequence or >> mapping >> semantics. if people want full sequence behaviour with len and >> slicing >> and iterators and whatnot, they can do list(m) first. >> > i'm ok either way -- that is, either with the proposal i previously > published, or with this restricted idea. I prefer your previous version. It matches my expectations as a user of regular expression matching and as someone with experience of other regexp implementations. (The current groups() method *doesn't* match those expectations, incidentally. I know I've been tripped up in the past because it didn't include the full match as element 0.) Basically, I don't see the advantage in the restrictions Frederik is proposing (other than possibly being simpler to implement, though not actually all that much, I think). Yes, it's a little unusual in that you'd be able to index the match "array" with either integer indices or using names, but I don't view that as a problem, and I don't see how not supporting len() or other list features like slicing and iterators helps. What's more, I think it will be confusing for Python newbies because they'll see someone doing m[3] and assume that m is a list-like object, then complain when things like for match in m: print match or m[3:4] fail to do what they expect. Yes, you might say "it's a match object, not a list". But, it seems to me, that's really in the same vein as "don't type quit or exit, press Ctrl-D". Kind regards, Alastair. -- http://alastairs-place.net From fredrik at pythonware.com Tue Dec 5 16:51:35 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 05 Dec 2006 16:51:35 +0100 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> <45733B10.9090300@v.loewis.de> <457535AF.6050106@666.com> Message-ID: Alastair Houghton wrote: > (The current groups() method *doesn't* match those expectations, > incidentally. I know I've been tripped up in the past because it > didn't include the full match as element 0.) that's because there is no "group 0" in a regular expression; that's just a historical API convenience thing. groups are numbered from 1 and upwards, and "groups()" returns all the actual groups. > What's more, I think it will be confusing for Python newbies because > they'll see someone doing > > m[3] > > and assume that m is a list-like object, then complain when things like > > for match in m: > print match that'll work, of course, which might be confusing for people who think they understand how for-in works but don't ;) > or > > m[3:4] > > fail to do what they expect. the problem with slicing is that people may 1) expect a slice to return a new object *of the same type* (which opens up a *gigantic* can of worms, both on the implementation level and on the wtf-is-this-thing- really level), and 2) expect things like [::-1] to work, which opens up another can of worms. I prefer the "If the implementation is easy to explain, it may be a good idea." design principle over "can of worms" design principle. From hanwen at xs4all.nl Tue Dec 5 17:43:11 2006 From: hanwen at xs4all.nl (Han-Wen Nienhuys) Date: Tue, 05 Dec 2006 17:43:11 +0100 Subject: [Python-Dev] cross-compile patches Message-ID: <4575A19F.2020609@xs4all.nl> Hello all, A couple of weeks ago, I posted a collection of patches to Python 2.5 on SF.net (#1597850) that enable python to be cross-compiled using standard tools (ie. configure and environment settings), among others on Linux/Darwin to Mingw32/Linux/Freebsd. In response to Martin v Loewis' initial reply we have sent in contributor agreements, but nothing else has happened so far. I was wondering whether there is any way for me to speed up the review process. For example, I could post each patch separately and/or prepare patches to Python SVN. -- Han-Wen Nienhuys - hanwen at xs4all.nl - http://www.xs4all.nl/~hanwen From arigo at tunes.org Tue Dec 5 18:16:56 2006 From: arigo at tunes.org (Armin Rigo) Date: Tue, 5 Dec 2006 18:16:56 +0100 Subject: [Python-Dev] Next PyPy sprint: Leysin 8-14th January 2007 Message-ID: <20061205171656.GA11821@code0.codespeak.net> Hi all, ===================================================================== PyPy Leysin Winter Sports Sprint (8-14th January 2007) ===================================================================== .. image:: http://www.ermina.ch/002.JPG The next PyPy sprint will be in Leysin, Switzerland, for the fourth time. This sprint will be the final public sprint of our EU-funded period, and a kick-off for the final work on the upcoming PyPy 1.0 release (scheduled for mid-February). The sprint is the last chance for students looking for a "summer" job with PyPy this winter! If you have a proposal and would like to work with us in the mountains please send it in before 15th December to "pypy-tb at codespeak.net" and cross-read this page: http://codespeak.net/pypy/dist/pypy/doc/summer-of-pypy.html ------------------------------ Goals and topics of the sprint ------------------------------ * Like previous winter, the main side goal is to have fun in winter sports :-) We can take a couple of days off for ski; at this time of year, ski days end before 4pm, which still leaves plenty of time to recover (er, I mean hack). * Progress on the JIT compiler, which we are just starting to scale to the whole of PyPy. `[1]`_ * Polish the code and documentation of the py lib, and eventually release it. * Work on prototypes that use the new features that PyPy enables: distribution `[2]`_ (based on transparent proxying `[3]`_), security `[4]`_, persistence `[5]`_, etc. `[6]`_. .. _[1]: http://codespeak.net/pypy/dist/pypy/doc/jit.html .. _[2]: http://codespeak.net/svn/pypy/dist/pypy/lib/distributed/ .. _[3]: http://codespeak.net/pypy/dist/pypy/doc/proxy.html .. _[4]: http://codespeak.net/pypy/dist/pypy/doc/project-ideas.html#security .. _[5]: http://codespeak.net/pypy/dist/pypy/doc/project-ideas.html#persistence .. _[6]: http://codespeak.net/pypy/dist/pypy/doc/project-ideas.html ----------------------- Location & Accomodation ----------------------- Leysin, Switzerland, "same place as before". Let me refresh your memory: both the sprint venue and the lodging will be in a very spacious pair of chalets built specifically for bed & breakfast: http://www.ermina.ch/. The place has a baseline ADSL Internet connexion (600Kb) with wireless installed. You can of course arrange your own lodging anywhere (so long as you are in Leysin, you cannot be more than a 15 minute walk away from the sprint venue), but I definitely recommend lodging there too -- you won't find a better view anywhere else (though you probably won't get much worse ones easily, either :-) I made pre-reservations in the Chalet, so please *confirm* quickly that you are coming so that we can adjust the reservations as appropriate. The rate so far has been 60 CHF a night all included in 2-person rooms, with breakfast. There are larger rooms too (less expensive, possibly more available too) and the possibility to get a single room if you really want to. Please register by svn: http://codespeak.net/svn/pypy/extradoc/sprintinfo/leysin-winter-2007/people.txt or on the pypy-sprint mailing list if you do not yet have check-in rights: http://codespeak.net/mailman/listinfo/pypy-sprint You need a Swiss-to-(insert country here) power adapter. There will be some Swiss-to-EU adapters around - bring a EU-format power strip if you have one. ----------- Exact times ----------- Officially, 8th-14th January 2007. Both dates are flexible, you can arrive or leave earlier or later, though it is recommended to arrive on the 7th (if many people arrive on the 6th we need to check for accomodation availability first). We will give introductions and tutorials on the 8th, in the morning if everybody is there, or in the afternoon otherwise. -------------------------------------- Armin From python at rcn.com Tue Dec 5 18:30:46 2006 From: python at rcn.com (Raymond Hettinger) Date: Tue, 5 Dec 2006 09:30:46 -0800 Subject: [Python-Dev] a feature i'd like to see in python #1: betteriteration control References: <4572C1F1.1050600@666.com> Message-ID: <009a01c71893$1a573350$ea146b0a@RaymondLaptop1> [Ben Wing ] >> many times writing somewhat complex loops over lists i've found the need >> to sometimes delete an item from the list. currently there's no easy >> way to do so; basically, you have to write something like [Adam Olsen] > As I don't believe there's any need for a language extension, I've > posted my approach to this on comp.lang.python: > > http://groups.google.com/group/comp.lang.python/browse_thread/thread/724aa6bcf021cfad/c4c629bd1bacc12b#c4c629bd1bacc12b > > Note that deleting from the list as you iterate over it tends to have > very poor performance for larger lists. My post includes some > timings, demonstrating this. Well said. And even if performance considerations could be addressed, it would be a crummy idea , leading people into knotty code. When looping over a structure, the act of actively inserting and deleting entries, can introduce unexpected feedback between loop control and the loop body. This is a recipe for bugs and hard-to-maintain code. Overall, feature #1 is a non-starter and hopefully this thread will die quickly or move off-list. Raymond From martin at v.loewis.de Tue Dec 5 19:28:37 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Dec 2006 19:28:37 +0100 Subject: [Python-Dev] LSB: Binary compatibility In-Reply-To: References: <45747169.4050003@v.loewis.de> Message-ID: <4575BA55.7070207@v.loewis.de> Neal Norwitz schrieb: >> All in all, I think providing binary compatibility would >> be feasible, and should be attempted. What do you think? > > Let's assume that 2.4 is the first LSB version. The ABI is different > for 2.4 and 2.5. We can't change the ABI for 2.5 since it's already > released and our policy is to keep it constant. I'm not absolutely sure it's true, because it may be possible to integrate the 2.4 ABI into future 2.5 releases by means of symbol versioning, but, more likely, that's not possible (as you likely can't represent the difference in symbol versions). > Where does that leave us for moving forward? Would we have to modify > 2.5 to make it entirely compatible with 2.4? Can we skip 2.5 > altogether and worry about making 2.6 compatible with 2.4? The most likely outcome is that LSB 3.2 won't provide any Python ABI, but just a Python API (for a scripting language, that's enough to achieve portability of applications). With that, extension modules would not be portable across releases. Then, with LSB 4.0, we could try to specify an ABI as well. Whether that would be the 2.4 ABI, the 2.5 ABI, or the 2.6 ABI must be decided. > Even if > so, that means no matter what we still need to make 2.4 compatible > APIs going forward, right? That will require some archeaology to > determine preciesly what we changed from 2.4. The LSB people are happy if we declare a subset of the ABI as stable. So if PyFrame_New wasn't in the ABI, that would probably be just fine. I think most ABI breakages are due to changes to structure layout, and we should remove all reliance on structure layout out of the ABI - requiring that ABI-compliant extensions use accessor functions for everything. > Is our current practice of the following change acceptable? > > Original: > PyAPI_FUNC(xxx) Py_Foo(xxx); > > New version: > PyAPI_FUNC(xxx) Py_FooEx(xxx, yyy); > #define Py_Foo(x) Py_FooEx(x, NULL) > > And similarly in the C file, but keeping the original PyFoo with > something like: > > xxx Py_FooEx(...) { ... } > #undef Py_Foo > xxx Py_Foo(xxx) { return Py_FooEx(xxx, NULL); } This would be perfectly fine for ABI stability. They could even accept if we just changed the signature of Py_Foo, and use symbol versioning to provide the old definition (*) - but for API evolution, that wouldn't work, of course. It would mean that extension writers need the old header files to build extensions. The LSB people would be fine with that - they provide a complete set of header files to build LSB-compatible binaries, anyway (so they can tell people: use these headers, and your binaries will be ABI-compliant). Regards, Martin (*) symbol versioning would help in cases where we change a parameter to be of type Py_ssize_t now: we'd provide another definition (with an old version number) that has the original signature, and do the same forwarding. From martin at v.loewis.de Tue Dec 5 19:32:04 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Dec 2006 19:32:04 +0100 Subject: [Python-Dev] LSB: Selection of a Python version In-Reply-To: <200612052035.35984.anthony@interlink.com.au> References: <45746DB9.9050501@v.loewis.de> <45751220.6020301@v.loewis.de> <200612052035.35984.anthony@interlink.com.au> Message-ID: <4575BB24.5000203@v.loewis.de> Anthony Baxter schrieb: >> So I think a public statement that we will support 2.4 with >> security patches for a while longer (and perhaps with security >> patches *only*) would be a good thing - independent of the LSB, >> actually. > > Well, I don't know what sort of public statement you want to issue, > but will this do? (Wearing my release manager hat) This would be a good statement, but I have also a set of policies in mind that clarify what a "security bugfix" is, so people know what kind of changes they get when they install a new version that only has security bugfixes. I believe this guarantee works in two ways: we should guarantee that we will accept security bugfixes, but we should also guarantee that we *only* accept security bugfixes (and perhaps critical crashers - although they overlap in the denial-of-service field). Regards, Martin From martin at v.loewis.de Tue Dec 5 19:42:00 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Dec 2006 19:42:00 +0100 Subject: [Python-Dev] LSB: Binary compatibility In-Reply-To: <17781.32179.14808.124857@montanaro.dyndns.org> References: <45747169.4050003@v.loewis.de> <17781.32179.14808.124857@montanaro.dyndns.org> Message-ID: <4575BD78.4000308@v.loewis.de> skip at pobox.com schrieb: > >> All in all, I think providing binary compatibility would be feasible, > >> and should be attempted. What do you think? > > Neal> Let's assume that 2.4 is the first LSB version. The ABI is > Neal> different for 2.4 and 2.5. We can't change the ABI for 2.5 since > Neal> it's already released and our policy is to keep it constant. > > It seems that adhering to LSB's constraints is going to create a new set of > problems for Python development. It's unclear to me what LSB brings to > Python other than a bunch of new headaches. I won't try to defend it, but would suggest that an evaluation is deferred until it is clear what the actual problems are, and then to judge whether they are additional problems (or perhaps just a tightening of procedures which we had been following all along). In any case, having Python in the LSB means that ISVs (software vendors) who target LSB (rather than targetting specific Linux distributions) could develop their applications also in Python (whereas now they have to use C or C++). With not even Java being included, that would put Python into a unique position in that field (perhaps along with Perl, which is also targetted for LSB 3.2 - this wasn't discussed yesterday, basically because nobody from the Perl community was present). Regards, Martin From fredrik at pythonware.com Tue Dec 5 19:44:56 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 05 Dec 2006 19:44:56 +0100 Subject: [Python-Dev] LSB: Binary compatibility In-Reply-To: <4575BD78.4000308@v.loewis.de> References: <45747169.4050003@v.loewis.de> <17781.32179.14808.124857@montanaro.dyndns.org> <4575BD78.4000308@v.loewis.de> Message-ID: Martin v. L?wis wrote: > In any case, having Python in the LSB means that ISVs (software > vendors) who target LSB (rather than targetting specific Linux > distributions) could develop their applications also in Python > (whereas now they have to use C or C++). ... without having to include a Python runtime. From martin at v.loewis.de Tue Dec 5 20:12:47 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 05 Dec 2006 20:12:47 +0100 Subject: [Python-Dev] cross-compile patches In-Reply-To: <4575A19F.2020609@xs4all.nl> References: <4575A19F.2020609@xs4all.nl> Message-ID: <4575C4AF.7000803@v.loewis.de> Han-Wen Nienhuys schrieb: > In response to Martin v Loewis' initial reply we have sent in > contributor agreements, but nothing else has happened so far. > > I was wondering whether there is any way for me to speed up the review > process. It may not be what you expect to hear, but there actually is: If you review 5 patches, I'll review yours quickly in return. Regards, Martin From skip at pobox.com Tue Dec 5 21:16:18 2006 From: skip at pobox.com (skip at pobox.com) Date: Tue, 5 Dec 2006 14:16:18 -0600 Subject: [Python-Dev] LSB: Binary compatibility In-Reply-To: <4575BD78.4000308@v.loewis.de> References: <45747169.4050003@v.loewis.de> <17781.32179.14808.124857@montanaro.dyndns.org> <4575BD78.4000308@v.loewis.de> Message-ID: <17781.54162.612270.573749@montanaro.dyndns.org> >>>>> "Martin" == Martin v L?wis writes: Martin> skip at pobox.com schrieb: >> >> All in all, I think providing binary compatibility would be feasible, >> >> and should be attempted. What do you think? >> Neal> Let's assume that 2.4 is the first LSB version. The ABI is Neal> different for 2.4 and 2.5. We can't change the ABI for 2.5 since Neal> it's already released and our policy is to keep it constant. >> >> It seems that adhering to LSB's constraints is going to create a new set of >> problems for Python development. It's unclear to me what LSB brings to >> Python other than a bunch of new headaches. Martin> I won't try to defend it, but would suggest that an evaluation Martin> is deferred until it is clear what the actual problems are, and Martin> then to judge whether they are additional problems (or perhaps Martin> just a tightening of procedures which we had been following all Martin> along). Taking one example from this thread, Python's bytecode has always been an internal implementation detail. If I read the thread correctly there is at least a request (if not a requirement) to make it part of an external ABI if Python is to become part of the ABI. That may or may not be a large technical challenge, but I think it would be a significant philosophical change. Martin> In any case, having Python in the LSB means that ISVs (software Martin> vendors) who target LSB (rather than targetting specific Linux Martin> distributions) could develop their applications also in Python Martin> (whereas now they have to use C or C++). Why? Lots of people write portable Python programs today. Skip From brett at python.org Tue Dec 5 22:09:11 2006 From: brett at python.org (Brett Cannon) Date: Tue, 5 Dec 2006 13:09:11 -0800 Subject: [Python-Dev] LSB: Binary compatibility In-Reply-To: <17781.54162.612270.573749@montanaro.dyndns.org> References: <45747169.4050003@v.loewis.de> <17781.32179.14808.124857@montanaro.dyndns.org> <4575BD78.4000308@v.loewis.de> <17781.54162.612270.573749@montanaro.dyndns.org> Message-ID: On 12/5/06, skip at pobox.com wrote: > > >>>>> "Martin" == Martin v L?wis writes: > > Martin> skip at pobox.com schrieb: > >> >> All in all, I think providing binary compatibility would be > feasible, > >> >> and should be attempted. What do you think? > >> > Neal> Let's assume that 2.4 is the first LSB version. The ABI is > Neal> different for 2.4 and 2.5. We can't change the ABI for 2.5since > Neal> it's already released and our policy is to keep it constant. > >> > >> It seems that adhering to LSB's constraints is going to create a > new set of > >> problems for Python development. It's unclear to me what LSB > brings to > >> Python other than a bunch of new headaches. > > Martin> I won't try to defend it, but would suggest that an evaluation > Martin> is deferred until it is clear what the actual problems are, > and > Martin> then to judge whether they are additional problems (or perhaps > Martin> just a tightening of procedures which we had been following > all > Martin> along). > > Taking one example from this thread, Python's bytecode has always been an > internal implementation detail. If I read the thread correctly there is > at > least a request (if not a requirement) to make it part of an external ABI > if > Python is to become part of the ABI. That may or may not be a large > technical challenge, but I think it would be a significant philosophical > change. I don't think we are being asked to standardize the bytecode, but that we will accept .pyc files as generated by 2.4 in future interpreters as legitimate. That seems to implicitly require us to standardize the bytecode *and* they .pyc file format. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20061205/5f2a9f05/attachment.html From hanwen at xs4all.nl Wed Dec 6 00:59:21 2006 From: hanwen at xs4all.nl (Han-Wen Nienhuys) Date: Wed, 06 Dec 2006 00:59:21 +0100 Subject: [Python-Dev] cross-compile patches In-Reply-To: <4575C4AF.7000803@v.loewis.de> References: <4575A19F.2020609@xs4all.nl> <4575C4AF.7000803@v.loewis.de> Message-ID: <457607D9.70107@xs4all.nl> Martin v. L?wis escreveu: > Han-Wen Nienhuys schrieb: >> In response to Martin v Loewis' initial reply we have sent in >> contributor agreements, but nothing else has happened so far. >> >> I was wondering whether there is any way for me to speed up the review >> process. > > It may not be what you expect to hear, but there actually is: > > If you review 5 patches, I'll review yours quickly in return. > Fair enough. I'm not familiar with the process, python internals, and couldn't find much of guidelines. I also don't have permission to fiddle with the tracker, so here are 5 small ones that I could understand 1608267 - added comment, asking for explanation. looks bogus. 1608579 - problem analysis + solution looks good 1507247 - I can reproduce the problem it tries to fix. Fix looks ok. 1520879 - trivial fix; should go in. disregard the comment about relative --prefix 1339673 - superseeded by #1597850 -- Han-Wen Nienhuys - hanwen at xs4all.nl - http://www.xs4all.nl/~hanwen From greg.ewing at canterbury.ac.nz Wed Dec 6 01:03:46 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 06 Dec 2006 13:03:46 +1300 Subject: [Python-Dev] LSB: Binary compatibility In-Reply-To: <17781.54162.612270.573749@montanaro.dyndns.org> References: <45747169.4050003@v.loewis.de> <17781.32179.14808.124857@montanaro.dyndns.org> <4575BD78.4000308@v.loewis.de> <17781.54162.612270.573749@montanaro.dyndns.org> Message-ID: <457608E2.2050009@canterbury.ac.nz> Could backwards compatibility concerns be addressed by including more than one version of Python in the LSB? Python already allows multiple versions to coexist, so applications targeting the LSB would just need to be explicit about which version of the interpreter to launch. -- Greg From nnorwitz at gmail.com Wed Dec 6 06:02:15 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 5 Dec 2006 21:02:15 -0800 Subject: [Python-Dev] LSB: Selection of a Python version In-Reply-To: <200612052035.35984.anthony@interlink.com.au> References: <45746DB9.9050501@v.loewis.de> <45751220.6020301@v.loewis.de> <200612052035.35984.anthony@interlink.com.au> Message-ID: On 12/5/06, Anthony Baxter wrote: > > > So I think a public statement that we will support 2.4 with > > security patches for a while longer (and perhaps with security > > patches *only*) would be a good thing - independent of the LSB, > > actually. > > Well, I don't know what sort of public statement you want to issue, > but will this do? (Wearing my release manager hat) I think we should document our intentions in the release PEP (or some other PEP possibly). Should we state that we will support version x.y for N years after initial release or based on version numbers such as x.(y-N) where y is the current version and N is a small number? I'd prefer the time based approach because it seems much more common. For reference here are some release dates: 2.2: 2001-12-21 2.3: 2003-07-29 2.4: 2004-11-30 If we stopped supporting 2.3, that would mean we support for 2-3 years. n From steven.bethard at gmail.com Wed Dec 6 06:44:34 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Tue, 5 Dec 2006 22:44:34 -0700 Subject: [Python-Dev] DRAFT: python-dev summary for 2006-11-16 through 2006-11-30 Message-ID: Here's the summary for the second half of November. As always, corrections and comments are greatly appreciated. If you were involved in the November portions of the LSB discussions, I'd particularly appreciate your reviews of that section. ============= Announcements ============= -------------------------- Python 2.5 malloc families -------------------------- Remember that if you find your extension module is crashing with Python 2.5 in malloc/free, there is a high chance that you have a mismatch in malloc "families". Fredrik Lundh's FAQ has more: http://effbot.org/pyfaq/why-does-my-c-extension-suddenly-crash-under-2.5.htm Contributing thread: - `2.5 portability problems `__ --------------------------------- Roundup tracker schema discussion --------------------------------- If you'd like to be involved in the discussion of the setup for the `new tracker`_, you can now file issues on the `meta tracker`_ or post to the `tracker-discuss mailing list`_. Be sure to sign up for an account so your comments don't show up as anonymous! .. _new tracker: http://psf.upfronthosting.co.za/roundup/tracker/ .. _meta tracker: http://psf.upfronthosting.co.za/roundup/meta/ .. _tracker-discuss mailing list: http://mail.python.org/mailman/listinfo/tracker-discuss Contributing thread: - `discussion of schema for new issue tracker starting `__ ========= Summaries ========= ---------------------------------------- Python and the Linux Standard Base (LSB) ---------------------------------------- Ian Murdock, the chair of the Linux Standard Base (LSB), explained that they wanted to add Python to `LSB 3.2`_. Martin v. L?wis promised to go to their meeting in Berlin and report back to python-dev. The discussion then turned to the various ways in which the different Linux variants package Python. A number of people had been troubled by Debian's handling of distutils. At one point, Debian had excluded distutils completely, requiring users to install the "python-dev" package to get distutils functionality. While current versions of Debian had put distutils back in the stdlib, they had excluded the ``config`` directory, meaning that distutils worked only for pure Python modules, not extension modules. And because Debian had no way of knowing that a computer with both gcc and Python installed would likely benefit from having the ``config`` directory installed, the user still had to install "python-dev" separately. There was also some discussion about how to handle third party modules so that updating a module didn't break some application which was expecting a different version. These kinds of problems were particularly dangerous on distributions like Gentoo and Ubuntu which relied heavily on their own system Python for the OS to work properly. Guido suggested introducing a vendor-packages directory for the third party modules required by the OS and Martin v. L?wis reopened an `earlier patch`_ suggesting this. A number of folks also thought that adding a ~/.local/lib/pythonX.X/site-packages directory for user specific (not site wide) packages could be useful. Phillip J. Eby pointed out that distutils and setuptools already allow you to install packages this way by putting:: [install] prefix = ~/.local into ./setup.cfg, ~/.pydistutils.cfg, or /usr/lib/python2.x/distutils/distutils.cfg. He also explained that setuptools could address some of the application-level problems: setuptools-generated scripts adjust their sys.path to include the specific eggs they need, and can specify these eggs with an exact version if necessary. Thus OS-level scripts would likely specify exact versions and then users could feel free to install newer eggs without worrying that the OS would try to use them instead. .. _LSB 3.2: http://www.freestandards.org/en/LSB_Roadmap .. _earlier patch: http://bugs.python.org/1298835 Contributing thread: - `Python and the Linux Standard Base (LSB) `__ ---------------------- Thread-safe operations ---------------------- Fredrik Lundh has been working on a `new Python FAQ`_ and asked about what kinds of operations could be considered "atomic" for the purposes of thread-safety. While almost any statement in Python can invoke an arbitrary special method (e.g. ``a = b`` can invoke ``a.__del__()``), Fredrik was interested in situations where the objects involved were either builtins or objects that didn't override special methods. In situations like these, you can be guaranteed things like:: * If two threads execute ``L.append(x)``, two items will be added to the list (though the order is unspecified) * If two threads execute ``x.y = z``, the field ``y`` on the ``x`` object will exist and contain one of the values assigned by one of the threads You get these guarantees mainly because the core operation in these examples involves only a single Python bytecode. However, Martin v. L?wis pointed out that even the above examples are not truly atomic in the strictest sense because they invoke bytecodes to load the values of the variables in addition to the bytecode to perform the operation. For example, if one thread does ``x = y`` while another thread does ``y = x``, at the end of the code in an "atomic" system, both ``x`` and ``y`` would have the same value. However, in Python, the values could get swapped if a context switch occurred between the loading of the values and the assignment operations. Much of this discussion was also posted to `the FAQ item`_. .. _new Python FAQ: http://effbot.org/pyfaq/ .. _the FAQ item: http://effbot.org/pyfaq/what-kinds-of-global-value-mutation-are-thread-safe.htm Contributing thread: - `PyFAQ: thread-safe interpreter operations `__ -------------------------------------------- From an empty directory to a package on PyPI -------------------------------------------- Talin suggested that distutils/setuptools and their documentation should be updated so that new users could more easily answer the question: "What is the smoothest path from empty directory to a finished package on PyPI?" In particular, Talin thought that having to cross-reference between distutils/setuptools/unittest/etc. was confusing, and that a more stand-alone version of the documentation was necessary. A number of people agreed that the documentation could use some reorganization and the addition of some more tutorial-like sections. Mike Orr promised to put together an initial "Table of Contents" that would have links to the most important information for package distribution, and Talin promised to make his notes available on the "baby steps" necessary to prepare a module for setuptools (e.g. create the directory structure, write a setup.py file, create source files in the appropriate directories, etc.) Contributing thread: - `Distribution tools: What I would like to see `__ -------------------------------------------- Monitoring progress with urllib's reporthook -------------------------------------------- Martin v. L?wis looked at a `patch to urllib's reporthook`_ aimed at more accurate progress reporting. The original code in urllib was passing the ``read()`` block size as the second argument to the reporthook. The patch would have instead passed as the second argument the actual count of bytes read. Guido pointed out that the block size and the actual count would always be identical except for the last block because of how Python's ``file.read(n)`` works. Thus urllib was already giving the reporthook as accurate a progress report as possible given the implementation, and so the patch was rejected. .. _patch to urllib's reporthook: http://bugs.python.org/849407 Contributing thread: - `Passing actual read size to urllib reporthook `__ --------------------------- Infinity and NaN singletons --------------------------- Tomer Filiba asked about making the positive-infinity, negative-infinity and not-a-number (NaN) singletons available as attributes of the ``float`` type, e.g. ``float.posinf``, ``float.neginf`` and ``float.nan``. Bob Ippolito pointed him to `PEP 754`_ and the fpconst_ module which addressed some of these issues though in a separate module instead of the builtin ``float`` type. When Tomer asked why `PEP 754`_ had not been accepted, Martin v. L?wis explained that while people were interested in the feature, it was difficult to implement in general, e.g. on platforms where the double type was not IEEE-754. .. _PEP 754: http://www.python.org/dev/peps/pep-0754/ .. _fpconst: http://www.python.org/pypi/fpconst/ Contributing thread: - `infinities `__ ------------------------------------------ Line continuations and the tokenize module ------------------------------------------ Guido asked about modifying the tokenize module to allow a better round-tripping of code with line continuations. While the tokenize module was generating pseudo-tokens for things like comments and "ignored" newlines, it was not generating anything for line continuation backslashes. Adding the appropriate yield would have been trivial, but would have been a (minor) backwards incompatible change. Phillip J. Eby pointed Guido to `scale.dsl`_ which dealt with similar issues, and suggested that even though the change was small, it might cause problems for some existing tools. Guido proposed a somewhat more backwards compatible version, where a NL pseudo-token was generated with '\\\n' as its text value, and asked folks to try it out and see if it gave them any trouble. .. _scale.dsl: http://peak.telecommunity.com/DevCenter/scale.dsl#converting-tokens-back-to-text Contributing thread: - `Small tweak to tokenize.py? `__ ----------------------- Summer of Code projects ----------------------- Georg Brandl asked about the status of the Google Summer of Code projects and got a number of responses: * Nilton Volpato reported the completion of the new ziparchive_ module, which includes file-like access to zip members, support for BZIP2 compression, support for member file removal and support for encryption. He explained that he was still doing a little work to clean up the API, and that he would appreciate any feedback people had on the module. * Facundo Batista reported that the decimal Python-to-C transliteration was completed successfully, but that they learned in the process that a simple transliteration was not going to suffice and the decimal module was going to have to undergo a structural redesign to perform well in C. * Jim Jewett reported that the work to make more stdlib modules use the logging module was incomplete, and not ready for stdlib inclusion yet. .. _ziparchive: http://ziparchive.sourceforge.net/ Contributing threads: - `Summer of Code: zipfile? `__ - `Results of the SOC projects `__ - `Summer of Code: zipfile? `__ - `Results of the SOC projects `__ =============== Skipped Threads =============== - `Weekly Python Patch/Bug Summary `__ - `Python in first-year MIT core curriculum `__ - `POSIX Capabilities `__ - `[1593035] Re: readline problem with python-2.5 `__ - `DRAFT: python-dev summary for 2006-10-01 to 2006-10-15 `__ - `Suggestion/ feature request `__ - `DRAFT: python-dev summary for 2006-10-16 to 2006-10-31 `__ - `DRAFT: python-dev summary for 2006-11-01 to 2006-11-15 `__ - `ctypes and powerpc `__ - `(no subject) `__ - `Cloning threading.py using processes `__ - `Objecttype of 'locals' argument in PyEval_EvalCode `__ From martin at v.loewis.de Wed Dec 6 07:03:18 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 06 Dec 2006 07:03:18 +0100 Subject: [Python-Dev] LSB: Binary compatibility In-Reply-To: <17781.54162.612270.573749@montanaro.dyndns.org> References: <45747169.4050003@v.loewis.de> <17781.32179.14808.124857@montanaro.dyndns.org> <4575BD78.4000308@v.loewis.de> <17781.54162.612270.573749@montanaro.dyndns.org> Message-ID: <45765D26.5010205@v.loewis.de> skip at pobox.com schrieb: > Taking one example from this thread, Python's bytecode has always been an > internal implementation detail. I think that has stopped being true at least since I wrote 1997, or perhaps even since dis.py was written (i.e. right from the beginning of the language). With dis.py, the bytecode instructions are documented, and their numeric values accessible through opcode.h. What *hasn't* been documented is the marshal format, as it may change across releases. That hasn't stopped people from using the format as if it were specified, e.g. there is a Perl reader for marshal data: http://search.cpan.org/~simonw/Python-Serialise-Marshal-0.02/lib/Python/Serialise/Marshal.pm > If I read the thread correctly there is at > least a request (if not a requirement) to make it part of an external ABI if > Python is to become part of the ABI. That may or may not be a large > technical challenge, but I think it would be a significant philosophical > change. The philosophical change would be that a specific version of that would be standardized. The "current" version could still evolve in a less documented way. > Martin> In any case, having Python in the LSB means that ISVs (software > Martin> vendors) who target LSB (rather than targetting specific Linux > Martin> distributions) could develop their applications also in Python > Martin> (whereas now they have to use C or C++). > > Why? Lots of people write portable Python programs today. But if they distribute them, they either have to request that a Python interpreter is installed on the target system, or they have to ship a Python interpreter with their installation DVD (as they can't rely on Python being available on the system, in a certain version). With LSB specifying that Python must be present, and must have a certain (minimal) version, they can include Python scripts in their application, and they will work out of the box without the administrator of the target machine having to do anything. Regards, Martin From fredrik at pythonware.com Wed Dec 6 07:07:04 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 06 Dec 2006 07:07:04 +0100 Subject: [Python-Dev] DRAFT: python-dev summary for 2006-11-16 through 2006-11-30 In-Reply-To: References: Message-ID: Steven Bethard wrote: > Fredrik Lundh has been working on a `new Python FAQ`_ footnote: working on cleaning up the old FAQ, to be precise. it'll end up on python.org, as soon as Andrew Kuchling gets around to complete his FAQ renderer (which may take a while, since he's busy with PyCon 2007). From steven.bethard at gmail.com Wed Dec 6 07:10:47 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Tue, 5 Dec 2006 23:10:47 -0700 Subject: [Python-Dev] Looking for new Python-Dev Summarizers Message-ID: It's been my pleasure to write the Python-Dev Summaries for the last year and a half -- 40 summaries all told, 8 of those with Tim Lesher and 23 with Tony Meyer. It's really been an incredible learning experience, both in how the Python development process works, and in how Python as a community interacts. As I'm now coming up on the last semester of so of my Ph.D., I need to make some more time to get research done, and so I'm looking to retire from the summaries. Hence, I'm soliciting for a good replacement (or a few) who would be willing to pick up the job. The job takes a few hours about once every two weeks. I have scripts to automate most of the grunt work (e.g. collecting the threads from mail.python.org and laying out the summary as necessary for the website), so the job is pretty much just writing a ReST-formatted paragraph or two on each topic. If you've been reading Python-Dev for a while, and you'd like to get involved, this is a great place to start - I can guarantee you'll learn a lot from it. Please drop me an email if you're interested! Thanks, Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From martin at v.loewis.de Wed Dec 6 08:37:12 2006 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 06 Dec 2006 08:37:12 +0100 Subject: [Python-Dev] Threading, atexit, and logging Message-ID: <45767328.5010701@v.loewis.de> In bug #1566280 somebody reported that he gets an exception where the logging module tries to write to closed file descriptor. Upon investigation, it turns out that the file descriptor is closed because the logging atexit handler is invoked. This is surprising, as the program is far from exiting at this point. Investigating further, I found that the logging atexit handler is registered *after* the threading atexit handler, so it gets invoked *before* the threading's atexit. Now, threading's atexit is the one that keeps the application running, by waiting for all non-daemon threads to shut down. As this application does all its work in non-daemon threads, it keeps running for quite a while - except that the logging module gives errors. The real problem here is that atexit handlers are invoked even though the program logically doesn't exit, yet (it's not just that the threading atexit is invoked after logging atexit - this could happen to any atexit handler that gets registered). I added a patch to this report which makes the MainThread __exitfunc a sys.exitfunc, chaining what is there already. This will work fine for atexit (as atexit is still imported explicitly to register its sys.exitfunc), but it might break if other applications still insist on installing a sys.exitfunc. What do you think about this approach? Regards, Martin P.S. There is another issue reported about the interpreter crashing; I haven't been able to reproduce this, yet. From talin at acm.org Wed Dec 6 08:31:23 2006 From: talin at acm.org (Talin) Date: Tue, 05 Dec 2006 23:31:23 -0800 Subject: [Python-Dev] Distribution tools: What I would like to see In-Reply-To: <456BE0D0.9010507@acm.org> References: <4569F7E3.9040004@acm.org> <4569FD85.4010006@acm.org> <456B7604.6050809@v.loewis.de> <6e9196d20611271744o2e4a8795od83828e280bfeeb9@mail.gmail.com> <456BE0D0.9010507@acm.org> Message-ID: <457671CB.9020309@acm.org> Talin wrote: > What I am doing right now is creating a new extension project using > setuputils, and keeping notes on what I do. So for example, I start by > creating the directory structure: > > mkdir myproject > cd myproject > mkdir src > mkdir test I'd forgotten about this until I was reminded in the python-dev summary (dang those summaries are useful.) Anyway, I've put my notes on the Wiki; you can find them here at: http://wiki.python.org/moin/ExtensionTutorial This is an extremely minimalist guide for people who want to write an extension module, starting from nothing but a bare interpreter prompt. If I made any mistakes, well - it's a wiki, you know what to do :) -- Talin From martin at v.loewis.de Wed Dec 6 08:43:57 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 06 Dec 2006 08:43:57 +0100 Subject: [Python-Dev] LSB: Selection of a Python version In-Reply-To: References: <45746DB9.9050501@v.loewis.de> <45751220.6020301@v.loewis.de> <200612052035.35984.anthony@interlink.com.au> Message-ID: <457674BD.4070904@v.loewis.de> Neal Norwitz schrieb: > I think we should document our intentions in the release PEP (or some > other PEP possibly). Should we state that we will support version x.y > for N years after initial release or based on version numbers such as > x.(y-N) where y is the current version and N is a small number? > > I'd prefer the time based approach because it seems much more common. > For reference here are some release dates: > > 2.2: 2001-12-21 > 2.3: 2003-07-29 > 2.4: 2004-11-30 > > If we stopped supporting 2.3, that would mean we support for 2-3 years. I like Anthony's view of having three modes of maintenance: general bug fixes, serious crashers, and security fixes. We should continue to have general bug fixes only for the most recent feature release (i.e. 2.5 at the moment). At the rate at which security fixes are reported, it should be possible to keep 2.3 supported (for security patches only) for quite some time longer. I'm not sure about "serious crashers", as I don't have a feeling how often they are reported and how difficult they are to backport. For many of these, it's often the case that it is bad coding on the application's side, in which case I think the application should just work-around when it wants to support old Python versions. Those are typically not security-relevant, at least not if you can't force the crash by passing bad input data (people accepting arbitrary code from untrusted users in their programs are on their own, anyway - since we removed the rexec module). Regards, Martin From tim.peters at gmail.com Wed Dec 6 09:47:26 2006 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 6 Dec 2006 03:47:26 -0500 Subject: [Python-Dev] Threading, atexit, and logging In-Reply-To: <45767328.5010701@v.loewis.de> References: <45767328.5010701@v.loewis.de> Message-ID: <1f7befae0612060047n4b6fc89eq4f192dd1e50ff51@mail.gmail.com> [Martin v. L?wis] > In bug #1566280 somebody reported that he gets an > exception where the logging module tries to write > to closed file descriptor. > > Upon investigation, it turns out that the file descriptor > is closed because the logging atexit handler is invoked. > This is surprising, as the program is far from exiting at > this point. But the main thread is done, right? None of this appears to make sense unless we got into Py_Finalize(), and that doesn't happen until the main thread has nothing left to do. > Investigating further, I found that the logging atexit > handler is registered *after* the threading atexit handler, > so it gets invoked *before* the threading's atexit. Ya, and that sucks. Can't recall details now, but it's not the first time the vagaries of atexit ordering bit a threaded program. IMO, `threading` shouldn't use atexit at all. > Now, threading's atexit is the one that keeps the > application running, by waiting for all non-daemon threads > to shut down. As this application does all its work in > non-daemon threads, it keeps running for quite a while - > except that the logging module gives errors. > > The real problem here is that atexit handlers are > invoked even though the program logically doesn't exit, > yet (it's not just that the threading atexit is invoked > after logging atexit - this could happen to any atexit > handler that gets registered). > > I added a patch to this report which makes the MainThread > __exitfunc a sys.exitfunc, chaining what is there already. > This will work fine for atexit (as atexit is still imported > explicitly to register its sys.exitfunc), but it might break > if other applications still insist on installing a > sys.exitfunc. Well, that's been officially deprecated since 2.4, but who knows? > What do you think about this approach? It's expedient :-) So was using atexit for this to begin with. Probably "good enough". I'd rather, e.g., that `threading` stuff an exit function into a module global, and change Py_Finalize() to look for that and run it (if present) before invoking call_sys_exitfunc(). That is, break all connection between the core's implementation of threading and the user-visible `atexit` machinery. `atexit` is a hack specific to "don't care about order" finalization functions, and it gets increasingly obscure to try to force it to respect a specific ordering sometimes (e.g., now you have a patch to try to fix it by relying on an obscure deprecated feature and hoping users don't screw with that too -- probably "good enough", but still sucky). From fredrik at pythonware.com Wed Dec 6 10:26:32 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 6 Dec 2006 10:26:32 +0100 Subject: [Python-Dev] __str__ and unicode Message-ID: over at my work copy of the python language reference, Adrian Holovaty asked about the exact semantics of the __str__ hook: http://effbot.org/pyref/__str__ "The return value must be a string object." Does this mean it can be a *Unicode* string object? This distinction is ambiguous to me because unicode objects and string objects are both subclasses of basestring. May a __str__() return a Unicode object? I seem to remember earlier discussions on this topic, but don't recall when and what. From what I can tell, __str__ may return a Unicode object, but only if can be converted to an 8-bit string using the default encoding. Is this on purpose or by accident? Do we have a plan for improving the situation in future 2.X releases ? From mal at egenix.com Wed Dec 6 10:46:34 2006 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 06 Dec 2006 10:46:34 +0100 Subject: [Python-Dev] __str__ and unicode In-Reply-To: References: Message-ID: <4576917A.8040809@egenix.com> On 2006-12-06 10:26, Fredrik Lundh wrote: > over at my work copy of the python language reference, Adrian Holovaty > asked about the exact semantics of the __str__ hook: > > http://effbot.org/pyref/__str__ > > "The return value must be a string object." Does this mean it can be a > *Unicode* string object? This distinction is ambiguous to me because > unicode objects and string objects are both subclasses of basestring. > May a __str__() return a Unicode object? > > I seem to remember earlier discussions on this topic, but don't recall when > and what. From what I can tell, __str__ may return a Unicode object, but > only if can be converted to an 8-bit string using the default encoding. Is this > on purpose or by accident? Do we have a plan for improving the situation > in future 2.X releases ? This was added to make the transition to all Unicode in 3k easier: .__str__() may return a string or Unicode object. .__unicode__() must return a Unicode object. There is no restriction on the content of the Unicode string for .__str__(). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 06 2006) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at egenix.com Wed Dec 6 10:51:27 2006 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 06 Dec 2006 10:51:27 +0100 Subject: [Python-Dev] __str__ and unicode In-Reply-To: <4576917A.8040809@egenix.com> References: <4576917A.8040809@egenix.com> Message-ID: <4576929F.40207@egenix.com> On 2006-12-06 10:46, M.-A. Lemburg wrote: > On 2006-12-06 10:26, Fredrik Lundh wrote: >> over at my work copy of the python language reference, Adrian Holovaty >> asked about the exact semantics of the __str__ hook: >> >> http://effbot.org/pyref/__str__ >> >> "The return value must be a string object." Does this mean it can be a >> *Unicode* string object? This distinction is ambiguous to me because >> unicode objects and string objects are both subclasses of basestring. >> May a __str__() return a Unicode object? >> >> I seem to remember earlier discussions on this topic, but don't recall when >> and what. From what I can tell, __str__ may return a Unicode object, but >> only if can be converted to an 8-bit string using the default encoding. Is this >> on purpose or by accident? Do we have a plan for improving the situation >> in future 2.X releases ? > > This was added to make the transition to all Unicode in 3k easier: > > .__str__() may return a string or Unicode object. > > .__unicode__() must return a Unicode object. > > There is no restriction on the content of the Unicode string > for .__str__(). One more thing, since these two hooks are commonly used with str() and unicode(): * unicode(obj) will first try .__unicode() and then revert to .__str__() (possibly converting the string return value to Unicode) * str(obj) will try .__str__() only (possibly converting the Unicode return value to a string using the default encoding) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 06 2006) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From fredrik at pythonware.com Wed Dec 6 10:56:00 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 6 Dec 2006 10:56:00 +0100 Subject: [Python-Dev] __str__ and unicode References: <4576917A.8040809@egenix.com> Message-ID: M.-A. Lemburg wrote: > This was added to make the transition to all Unicode in 3k easier: thanks for the clarification. do you recall when this was added? 2.5? From mal at egenix.com Wed Dec 6 11:04:52 2006 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 06 Dec 2006 11:04:52 +0100 Subject: [Python-Dev] __str__ and unicode In-Reply-To: References: <4576917A.8040809@egenix.com> Message-ID: <457695C4.3070908@egenix.com> On 2006-12-06 10:56, Fredrik Lundh wrote: > M.-A. Lemburg wrote: > >> This was added to make the transition to all Unicode in 3k easier: > > thanks for the clarification. > > do you recall when this was added? 2.5? Not really, only that it was definitely before 2.5. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 06 2006) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From ncoghlan at gmail.com Wed Dec 6 13:07:43 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 06 Dec 2006 22:07:43 +1000 Subject: [Python-Dev] __str__ and unicode In-Reply-To: <4576917A.8040809@egenix.com> References: <4576917A.8040809@egenix.com> Message-ID: <4576B28F.90507@gmail.com> M.-A. Lemburg wrote: > On 2006-12-06 10:26, Fredrik Lundh wrote: >> From what I can tell, __str__ may return a Unicode object, but >> only if can be converted to an 8-bit string using the default encoding. Is this >> on purpose or by accident? Do we have a plan for improving the situation >> in future 2.X releases ? It has worked that way since at Python least 2.4 (I just tried returning unicode from __str__ in 2.4.1 and it worked fine). That's the oldest version I have handy, so I don't know if it was possible in earlier versions. > This was added to make the transition to all Unicode in 3k easier: > > .__str__() may return a string or Unicode object. > > .__unicode__() must return a Unicode object. > > There is no restriction on the content of the Unicode string > for .__str__(). It's also the basis for a tweak that was made in 2.5 to permit conversion to a builtin string in a way that is idempotent for both str and unicode instances via: as_builtin_string = '%s' % original To use the terms from the deferred PEP 349, that conversion mechanism is both Unicode-safe (unicode doesn't get coerced to str) and str-stable (str doesn't get coerced to unicode). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Wed Dec 6 13:26:05 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 06 Dec 2006 22:26:05 +1000 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> <45733B10.9090300@v.loewis.de> <457535AF.6050106@666.com> Message-ID: <4576B6DD.9040808@gmail.com> Fredrik Lundh wrote: > the problem with slicing is that people may 1) expect a slice to return > a new object *of the same type* (which opens up a *gigantic* can of > worms, both on the implementation level and on the wtf-is-this-thing- > really level), and 2) expect things like [::-1] to work, which opens up > another can of worms. I prefer the "If the implementation is easy to > explain, it may be a good idea." design principle over "can of worms" > design principle. This is a good point - I know I consider "m[0:0] == type(m)()" to be a property a well-behaved sequence should preserve. Since match objects can't really do that, better not to pretend to be a sequence at all. With slicing out of the equation, that only leaves the question of whether or not len(m) should work. I believe it would be nice for len(m) to be supported, so that reversed(m) works along with iter(m). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From barry at python.org Wed Dec 6 14:09:52 2006 From: barry at python.org (Barry Warsaw) Date: Wed, 6 Dec 2006 08:09:52 -0500 Subject: [Python-Dev] __str__ and unicode In-Reply-To: <4576B28F.90507@gmail.com> References: <4576917A.8040809@egenix.com> <4576B28F.90507@gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Dec 6, 2006, at 7:07 AM, Nick Coghlan wrote: > M.-A. Lemburg wrote: >> On 2006-12-06 10:26, Fredrik Lundh wrote: >>> From what I can tell, __str__ may return a Unicode object, but >>> only if can be converted to an 8-bit string using the default >>> encoding. Is this >>> on purpose or by accident? Do we have a plan for improving the >>> situation >>> in future 2.X releases ? > > It has worked that way since at Python least 2.4 (I just tried > returning > unicode from __str__ in 2.4.1 and it worked fine). That's the > oldest version I > have handy, so I don't know if it was possible in earlier versions. I don't have anything older than 2.4 laying around either, but IIRC in 2.3 unicode() did not call __unicode__(). - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRXbBIHEjvBPtnXfVAQJC6AQAhPDrd451PYhQHTuFZqFX7oJpuadEONxb UaBEpWs6yzJjLAxC2tfRVT8vOc1bTmF3Wzf1y5HZsXcbklOFm3USl0YJ8206oDBN 2MGGf2e/JuC5oajo5RJqQ/oqaLDSHb8cD6GP2y/+FFaAhwDnlgnOlV0TxAggKv4K a9nCnFRwJ8c= =+mj8 -----END PGP SIGNATURE----- From murman at gmail.com Wed Dec 6 15:15:29 2006 From: murman at gmail.com (Michael Urman) Date: Wed, 6 Dec 2006 08:15:29 -0600 Subject: [Python-Dev] __str__ and unicode In-Reply-To: References: <4576917A.8040809@egenix.com> <4576B28F.90507@gmail.com> Message-ID: > I don't have anything older than 2.4 laying around either, but IIRC > in 2.3 unicode() did not call __unicode__(). It turns out __unicode__() is called on Python 2.3.5. % python2.3 Python 2.3.5 (#2, Oct 18 2006, 23:04:45) [GCC 4.1.2 20061015 (prerelease) (Debian 4.1.1-16.1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> class Foo(object): ... def __unicode__(self): ... print "unicode" ... return u"hi" ... def __str__(self): ... print "str" ... return "hello" ... >>> unicode(Foo()) unicode u'hi' -- Michael Urman http://www.tortall.net/mu/blog From barry at python.org Wed Dec 6 16:06:30 2006 From: barry at python.org (Barry Warsaw) Date: Wed, 6 Dec 2006 10:06:30 -0500 Subject: [Python-Dev] __str__ and unicode In-Reply-To: References: <4576917A.8040809@egenix.com> <4576B28F.90507@gmail.com> Message-ID: <7AAA2524-B49C-4475-BD52-8811F3E3C84C@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Dec 6, 2006, at 9:15 AM, Michael Urman wrote: >> I don't have anything older than 2.4 laying around either, but IIRC >> in 2.3 unicode() did not call __unicode__(). > > It turns out __unicode__() is called on Python 2.3.5. Ah cool, thanks. I must be misremembering that about some earlier Python. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRXbcenEjvBPtnXfVAQIZWAP/aRXJ1Rq0eMVWhYOYvP6Wdy2v5DCj0Arl yS3n0RURWJj5i+zYtqQUBIJEOcFSLJ69cb1SWl/KTvedI4y0SBQknX0o8EJaYhSU h1Y2gL2X+QnvJxlCf7PCdm2C1jYQgwAmKuebjCwaMPJYBqW9Z27+oSTsyFFM/mPR 2qx++VsRw68= =ABvs -----END PGP SIGNATURE----- From martin at v.loewis.de Wed Dec 6 19:37:11 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 06 Dec 2006 19:37:11 +0100 Subject: [Python-Dev] Threading, atexit, and logging In-Reply-To: <1f7befae0612060047n4b6fc89eq4f192dd1e50ff51@mail.gmail.com> References: <45767328.5010701@v.loewis.de> <1f7befae0612060047n4b6fc89eq4f192dd1e50ff51@mail.gmail.com> Message-ID: <45770DD7.50803@v.loewis.de> Tim Peters schrieb: >> Upon investigation, it turns out that the file descriptor >> is closed because the logging atexit handler is invoked. >> This is surprising, as the program is far from exiting at >> this point. > > But the main thread is done, right? Wrong. main.py (which is the __main__ script in the demo code) is done, yes. However, threading.py has machinery to not terminate the main thread as long as there are non-daemon threads. In that sense, the main thread is not done: it still has to .join() all the other threads (rather, they join the main thread). > Ya, and that sucks. Can't recall details now, but it's not the first > time the vagaries of atexit ordering bit a threaded program. In this case, logging/__init__.py imports threading, so that will register its atexit first - even if the application imports logging before threading. > IMO, `threading` shouldn't use atexit at all. That is (in a way) my proposal (although I suggest to use sys.exitfunc instead). > It's expedient :-) So was using atexit for this to begin with. > Probably "good enough". I'd rather, e.g., that `threading` stuff an > exit function into a module global, and change Py_Finalize() to look > for that and run it (if present) before invoking call_sys_exitfunc(). Ok, that's what I'll do then. Yet another alternative would be to have the "daemonic" thread feature in the thread module itself (along with keeping track of a list of all running non-daemonic thread). Regards, Martin From martin at v.loewis.de Wed Dec 6 20:07:34 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 06 Dec 2006 20:07:34 +0100 Subject: [Python-Dev] LSB: Binary compatibility In-Reply-To: <457608E2.2050009@canterbury.ac.nz> References: <45747169.4050003@v.loewis.de> <17781.32179.14808.124857@montanaro.dyndns.org> <4575BD78.4000308@v.loewis.de> <17781.54162.612270.573749@montanaro.dyndns.org> <457608E2.2050009@canterbury.ac.nz> Message-ID: <457714F6.8090207@v.loewis.de> Greg Ewing schrieb: > Could backwards compatibility concerns be addressed > by including more than one version of Python in > the LSB? Python already allows multiple versions > to coexist, so applications targeting the LSB > would just need to be explicit about which version > of the interpreter to launch. Yes. However, for that to work, the Linux distributors would not want to maintain the old Python releases themselves. Given the LSB release cycle, we would have still to support Python 2.0 today if that would have been included at the time of its release. Of course, for those vendors, *only* security patches matter. They can justify carrying outdated software around for compatibility only, but they cannot justify including it if it has known security bugs. Regards, Martin From alastair at alastairs-place.net Wed Dec 6 21:05:33 2006 From: alastair at alastairs-place.net (Alastair Houghton) Date: Wed, 6 Dec 2006 20:05:33 +0000 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <4572C1FF.2000900@666.com> <4572DDC0.6040203@v.loewis.de> <4572F2FF.3030206@v.loewis.de> <45731E74.7010101@v.loewis.de> <457330A7.6040401@v.loewis.de> <45733660.7040202@holdenweb.com> <45733B10.9090300@v.loewis.de> <457535AF.6050106@666.com> Message-ID: <1F552DFA-45F2-445A-A5A5-9CE6738E4875@alastairs-place.net> On 5 Dec 2006, at 15:51, Fredrik Lundh wrote: > Alastair Houghton wrote: > >> What's more, I think it will be confusing for Python newbies because >> they'll see someone doing >> >> m[3] >> >> and assume that m is a list-like object, then complain when things >> like >> >> for match in m: >> print match > > that'll work, of course, which might be confusing for people who think > they understand how for-in works but don't ;) Or (as in my case) guessed at how it works because they can't be bothered to check the code and can't remember from the last time they looked. I don't spend a great deal of time in the guts of Python. But I do use it and have a couple of extensions that I've written for it (one of which I was contemplating releasing publicly and that is impacted by this change---it provides, amongst other things, an alternate implementation of the "re" API, so I'm going to want to implement this too). >> or >> >> m[3:4] >> >> fail to do what they expect. > > the problem with slicing is that people may 1) expect a slice to > return > a new object *of the same type* What I would have expected is that it supported a similar set of sequence methods---that is, that it returned something with a similar signature. I don't see why code would care about it being the exact same type. Anyway, clearly what people will expect here (talking about the match object API) is that m[3:4] would give them a list (or some equivalent sequence object) containing groups 3 and 4. Why do you think someone would expect a match object? > 2) expect things like [::-1] to work, which opens up another can of > worms. As long as they aren't expecting it to return the same type of object, is there a can of worms here? > I prefer the "If the implementation is easy to explain, it may be a > good idea." design principle over "can of worms" design principle. As someone who is primarily a *user* of Python, I prefer the idea that sequence objects should operate consistently to the idea that there might be some that don't. By which I mean that anything that supports indexing using integer values should ideally support slicing (including things like [::-1]). Kind regards, Alastair. -- http://alastairs-place.net From jcarlson at uci.edu Wed Dec 6 21:29:56 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 06 Dec 2006 12:29:56 -0800 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <1F552DFA-45F2-445A-A5A5-9CE6738E4875@alastairs-place.net> References: <1F552DFA-45F2-445A-A5A5-9CE6738E4875@alastairs-place.net> Message-ID: <20061206122217.9766.JCARLSON@uci.edu> Alastair Houghton wrote: > On 5 Dec 2006, at 15:51, Fredrik Lundh wrote: > > Alastair Houghton wrote: > >> or > >> > >> m[3:4] > >> > >> fail to do what they expect. > > > > the problem with slicing is that people may 1) expect a slice to > > return > > a new object *of the same type* > > What I would have expected is that it supported a similar set of > sequence methods---that is, that it returned something with a similar > signature. I don't see why code would care about it being the exact > same type. The problem is that either we return a list (easy), or we return something that is basically another match object (not quite so easy). Either way, we would be confusing one set of users or another. By not including slicing functionality by default, we sidestep the confusion. > Anyway, clearly what people will expect here (talking about the match > object API) is that m[3:4] would give them a list (or some equivalent > sequence object) containing groups 3 and 4. Why do you think someone > would expect a match object? Because that is what all other slicing operations in base Python do. List, tuple, string, unicode, array, buffer, ... Even extension writers preserve the functionality with Numeric, etc. When you slice a sequence, you get back a slice of that sequence, of the same type you started out with. > > I prefer the "If the implementation is easy to explain, it may be a > > good idea." design principle over "can of worms" design principle. > > As someone who is primarily a *user* of Python, I prefer the idea > that sequence objects should operate consistently to the idea that > there might be some that don't. By which I mean that anything that > supports indexing using integer values should ideally support slicing > (including things like [::-1]). You are being inconsistant. You want list, tuple, etc. to be consistant, but you don't want match objects to be consistant. Sorry, but that is silly. Better to not support slices than to confuse the hell out of people by returning a tuple or list from a match slicing. If people want slicing, they can do list(m)[x:y]. If their matches are of sufficient size where that is a "slow" operation, then they can do [m[i] for i in xrange(x,y)] . - Josiah From alastair at alastairs-place.net Thu Dec 7 00:07:26 2006 From: alastair at alastairs-place.net (Alastair Houghton) Date: Wed, 6 Dec 2006 23:07:26 +0000 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <20061206122217.9766.JCARLSON@uci.edu> References: <1F552DFA-45F2-445A-A5A5-9CE6738E4875@alastairs-place.net> <20061206122217.9766.JCARLSON@uci.edu> Message-ID: <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> On 6 Dec 2006, at 20:29, Josiah Carlson wrote: > The problem is that either we return a list (easy), or we return > something that is basically another match object (not quite so easy). > Either way, we would be confusing one set of users or another. By not > including slicing functionality by default, we sidestep the confusion. But I don't believe that *anyone* will find it confusing that it returns a list. It's much more likely to be confusing to people that they have to write list(m)[x:y] or [m[i] for i in xrange(x,y)] when m[x] and m[y] work just fine. >> As someone who is primarily a *user* of Python, I prefer the idea >> that sequence objects should operate consistently to the idea that >> there might be some that don't. By which I mean that anything that >> supports indexing using integer values should ideally support slicing >> (including things like [::-1]). > > You are being inconsistant. You want list, tuple, etc. to be > consistant, > but you don't want match objects to be consistant. Sorry, but that is > silly. Better to not support slices than to confuse the hell out of > people by returning a tuple or list from a match slicing. That's not true *and* I object to your characterisation of the idea as "silly". What I'm saying is that the idea of slicing always returning the same exact type of object is pointless consistency, because nobody will care *provided* the thing that is returned supports a sensible set of operations given the original type. Look, I give in. There's no point trying to convince any of you further, and I don't have the time or energy to press the point. Implement it as you will. If necessary it can be an extension of my "re" replacement that slicing is supported on match objects. Kind regards, Alastair. -- http://alastairs-place.net From mike.klaas at gmail.com Thu Dec 7 01:39:30 2006 From: mike.klaas at gmail.com (Mike Klaas) Date: Wed, 6 Dec 2006 16:39:30 -0800 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> References: <1F552DFA-45F2-445A-A5A5-9CE6738E4875@alastairs-place.net> <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> Message-ID: <3d2ce8cb0612061639h42c8115m9b4ad0c9530a8aae@mail.gmail.com> On 12/6/06, Alastair Houghton wrote: [from previous message]: >> Anyway, clearly what people will expect here (talking about the match >> object API) is that m[3:4] would give them a list (or some equivalent >> sequence object) containing groups 3 and 4. Why do you think someone >> would expect a match object? > It's much more likely to be confusing to people that they have to write > > list(m)[x:y] > or > [m[i] for i in xrange(x,y)] > when m[x] and m[y] work just fine. <> > Look, I give in. There's no point trying to convince any of you > further, and I don't have the time or energy to press the point. > Implement it as you will. If necessary it can be an extension of my > "re" replacement that slicing is supported on match objects. Keep in mind when implementing that m[3:4] should contain only the element at index 3, not both 3 and 4, as you've seemed to imply twice. cheers, -Mike From jcarlson at uci.edu Thu Dec 7 02:01:48 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 06 Dec 2006 17:01:48 -0800 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> References: <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> Message-ID: <20061206162923.9772.JCARLSON@uci.edu> Alastair Houghton wrote: > On 6 Dec 2006, at 20:29, Josiah Carlson wrote: > > > The problem is that either we return a list (easy), or we return > > something that is basically another match object (not quite so easy). > > Either way, we would be confusing one set of users or another. By not > > including slicing functionality by default, we sidestep the confusion. > > But I don't believe that *anyone* will find it confusing that it > returns a list. We'll have to agree to disagree. > >> As someone who is primarily a *user* of Python, I prefer the idea > >> that sequence objects should operate consistently to the idea that > >> there might be some that don't. By which I mean that anything that > >> supports indexing using integer values should ideally support slicing > >> (including things like [::-1]). > > > > You are being inconsistant. You want list, tuple, etc. to be > > consistant, > > but you don't want match objects to be consistant. Sorry, but that is > > silly. Better to not support slices than to confuse the hell out of > > people by returning a tuple or list from a match slicing. > > That's not true *and* I object to your characterisation of the idea > as "silly". What I'm saying is that the idea of slicing always > returning the same exact type of object is pointless consistency, > because nobody will care *provided* the thing that is returned > supports a sensible set of operations given the original type. In Python 2.5: >>> a <_sre.SRE_Match object at 0x008F6020> >>> dir(a) ['__copy__', '__deepcopy__', 'end', 'expand', 'group', 'groupdict', 'groups', 'span', 'start'] Not including end, expand, group, groupdict, groups, span, and start may be confusing to some number of users. Why? Because of the historical invariant already present in the standard library (with the exception of buffer, I was wrong about that one). *We* may not be confused, but it's not about us (I'm personally happy to use the .group() interface); it's about relative newbies who, generally speaking, desire/need consistency (see [1] for a paper showing that certain kinds of inconsistancies are bad - at least in terms of grading - for new computer science students). Being inconsistant because it's *easy*, is what I consider silly. We've got the brains, we've got the time, if we want slicing, lets produce a match object. If we don't want slicing, or if prodicing a slice would produce a semantically questionable state, then lets not do it. I honestly don't care about whether slicing should go in or not, I use (?=) when I don't want a group. What I really don't want is someone coming in days after 2.6 is released complaining about match slicing not supporting things they think they need. Better to just tell them: use list(m)[x:y] or islice(iterable, [start,] stop [, step]) (both of which should work on arbitrary iterables, the latter of which works on *infinite* iterables) or produce a match object. All or nothing. Half-assing it is a waste. > Look, I give in. There's no point trying to convince any of you > further, and I don't have the time or energy to press the point. > Implement it as you will. If necessary it can be an extension of my > "re" replacement that slicing is supported on match objects. I'm sorry to see you give up so easily. One thing to realize/remember is that basically everyone who frequents python-dev has their own "make life easier" function/class library for those things that have been rejected for general inclusion in Python. - Josiah [1] http://www.cs.mdx.ac.uk/research/PhDArea/saeed/paper1.pdf From alastair at alastairs-place.net Thu Dec 7 01:43:13 2006 From: alastair at alastairs-place.net (Alastair Houghton) Date: Thu, 7 Dec 2006 00:43:13 +0000 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <3d2ce8cb0612061639h42c8115m9b4ad0c9530a8aae@mail.gmail.com> References: <1F552DFA-45F2-445A-A5A5-9CE6738E4875@alastairs-place.net> <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> <3d2ce8cb0612061639h42c8115m9b4ad0c9530a8aae@mail.gmail.com> Message-ID: <4EF34499-F8E7-4472-B99A-9F59391D99F0@alastairs-place.net> On 7 Dec 2006, at 00:39, Mike Klaas wrote: > Keep in mind when implementing that m[3:4] should contain only the > element at index 3, not both 3 and 4, as you've seemed to imply twice. Yes, you're quite right. I was writing off the top of my head and I'm still a relative newbie to Python coding. Kind regards, Alastair. -- http://alastairs-place.net From alastair at alastairs-place.net Thu Dec 7 02:28:16 2006 From: alastair at alastairs-place.net (Alastair Houghton) Date: Thu, 7 Dec 2006 01:28:16 +0000 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <20061206162923.9772.JCARLSON@uci.edu> References: <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> <20061206162923.9772.JCARLSON@uci.edu> Message-ID: On 7 Dec 2006, at 01:01, Josiah Carlson wrote: > *We* may not be confused, but it's not about us (I'm personally > happy to > use the .group() interface); it's about relative newbies who, > generally > speaking, desire/need consistency (see [1] for a paper showing that > certain kinds of inconsistancies are bad - at least in terms of > grading > - for new computer science students). Being inconsistant because it's > *easy*, is what I consider silly. We've got the brains, we've got the > time, if we want slicing, lets produce a match object. Oh, it isn't that I don't want to produce a match object; I think you've mistaken my intention in that respect. I'd be equally happy for it to be a match object, *but*... > If we don't want > slicing, or if prodicing a slice would produce a semantically > questionable state, then lets not do it. ...if you return match objects from slicing, you have problems like m [::-1].groups(). *I* don't know what that should return. What led me to think that a tuple or list would be appropriate is the idea that slicing was a useful operation and that I felt it was unlikely that anyone would want to call the match object methods on a slice, coupled with the fact that slices clearly have problems with some of the match object methods. A match object, plus sequence functionality, minus match object methods, is basically just a sequence. If you're worried about types, you could do something like this: generic match object | +--------------+-------------+ | | real match object match object slice where the "generic match object" perhaps doesn't have all the methods that a "real match object" would have. (In the extreme case, generic match object might basically just be a sequence type.) Then slicing something that was a "generic match object" always gives you a "generic match object", but it might not support all the methods that the original match object supported. > Half-assing it is a waste. Sure. We're agreed there :-) >> Look, I give in. There's no point trying to convince any of you >> further, and I don't have the time or energy to press the point. >> Implement it as you will. If necessary it can be an extension of my >> "re" replacement that slicing is supported on match objects. > > I'm sorry to see you give up so easily. One thing to realize/remember > is that basically everyone who frequents python-dev has their own > "make > life easier" function/class library for those things that have been > rejected for general inclusion in Python. It's just that I'm tired and have lots of other things that need doing as well. Maybe I do have a bit more time to talk about it, we'll see. Kind regards, Alastair. -- http://alastairs-place.net From murman at gmail.com Thu Dec 7 02:52:24 2006 From: murman at gmail.com (Michael Urman) Date: Wed, 6 Dec 2006 19:52:24 -0600 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <20061206162923.9772.JCARLSON@uci.edu> References: <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> <20061206162923.9772.JCARLSON@uci.edu> Message-ID: On 12/6/06, Josiah Carlson wrote: > *We* may not be confused, but it's not about us (I'm personally happy to > use the .group() interface); it's about relative newbies who, generally > speaking, desire/need consistency (see [1] for a paper showing that > certain kinds of inconsistancies are bad - at least in terms of grading > - for new computer science students). Being inconsistant because it's > *easy*, is what I consider silly. We've got the brains, we've got the > time, if we want slicing, lets produce a match object. If we don't want > slicing, or if prodicing a slice would produce a semantically > questionable state, then lets not do it. The idea that slicing a match object should produce a match object sounds like a foolish consistency to me. It's a useful invariant of lists that slicing them returns lists. It's not a useful invariant of sequences in general. This is similar to how it's a useful invariant that indexing a string returns a string; indexing a list generally does not return a list. I only found a couple __getslice__ definitions in a quick perusal of stdlib. ElementTree.py's _ElementInterface class returns a slice from a contained list; whereas sre_parse.py's SubPattern returns another SubPattern. UserList and UserString also define __getslice__ but I don't consider them representative of the standards of non-string/list classes. As an aside, if you're trying to show that inconsistencies in a language are bad by referencing a paper showing that people who used consistent (if incorrect) mental models scored better than those who did not, you may have to explain further; I don't see the connection. -- Michael Urman http://www.tortall.net/mu/blog From jcarlson at uci.edu Thu Dec 7 03:01:10 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 06 Dec 2006 18:01:10 -0800 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <20061206162923.9772.JCARLSON@uci.edu> Message-ID: <20061206174849.9779.JCARLSON@uci.edu> Alastair Houghton wrote: > On 7 Dec 2006, at 01:01, Josiah Carlson wrote: > > *We* may not be confused, but it's not about us (I'm personally > > happy to > > use the .group() interface); it's about relative newbies who, > > generally > > speaking, desire/need consistency (see [1] for a paper showing that > > certain kinds of inconsistancies are bad - at least in terms of > > grading > > - for new computer science students). Being inconsistant because it's > > *easy*, is what I consider silly. We've got the brains, we've got the > > time, if we want slicing, lets produce a match object. > > Oh, it isn't that I don't want to produce a match object; I think > you've mistaken my intention in that respect. I'd be equally happy > for it to be a match object, *but*... > > > If we don't want > > slicing, or if prodicing a slice would produce a semantically > > questionable state, then lets not do it. > > ...if you return match objects from slicing, you have problems like m > [::-1].groups(). *I* don't know what that should return. I would argue that any 'step' != 1 has no semantically correct result for slicing on a match object, so we shouldn't support it. In that sense, buffer also doesn't support step != 1, but that's because it's __getitem__ method doesn't accept slice objects, and uses the (I believe deprecated or removed in Py3k) __getslice__ method. We can easily check for such things in the __getitem__ method (to also support the removal? of __getslice__) and raise an exception. For those who want reversed slices, they can use reversed(m[x:y]), etc. > What led me to think that a tuple or list would be appropriate is the > idea that slicing was a useful operation and that I felt it was > unlikely that anyone would want to call the match object methods on a > slice, coupled with the fact that slices clearly have problems with > some of the match object methods. A match object, plus sequence > functionality, minus match object methods, is basically just a sequence. Explicit is better than implicit. Refuse the temptation to guess. Let us give them the full match object. If they want to do something silly with slicing (reversed, skipping some, etc.), make them use list() or islice(). > If you're worried about types, you could do something like this: > > generic match object > | > +--------------+-------------+ > | | > real match object match object slice I believe the above is unnecessary. Slicing a match could produce another match. It's all internal data semantics. - Josiah From jcarlson at uci.edu Thu Dec 7 03:49:04 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 06 Dec 2006 18:49:04 -0800 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <20061206162923.9772.JCARLSON@uci.edu> Message-ID: <20061206182848.977C.JCARLSON@uci.edu> "Michael Urman" wrote: > On 12/6/06, Josiah Carlson wrote: > > *We* may not be confused, but it's not about us (I'm personally happy to > > use the .group() interface); it's about relative newbies who, generally > > speaking, desire/need consistency (see [1] for a paper showing that > > certain kinds of inconsistancies are bad - at least in terms of grading > > - for new computer science students). Being inconsistant because it's > > *easy*, is what I consider silly. We've got the brains, we've got the > > time, if we want slicing, lets produce a match object. If we don't want > > slicing, or if prodicing a slice would produce a semantically > > questionable state, then lets not do it. > > The idea that slicing a match object should produce a match object > sounds like a foolish consistency to me. It's a useful invariant of > lists that slicing them returns lists. It's not a useful invariant of > sequences in general. This is similar to how it's a useful invariant > that indexing a string returns a string; indexing a list generally > does not return a list. The string and unicode case for S[i] is special. Such has already been discussed ad-nauseum. As for seq[i:j] returning an object of the same type, if it was "foolish consistency", then why is it consistent across literally the entire standard library (except for buffer), and (in my experience) many 3rd party libraries? > I only found a couple __getslice__ definitions in a quick perusal of > stdlib. ElementTree.py's _ElementInterface class returns a slice from > a contained list; whereas sre_parse.py's SubPattern returns another > SubPattern. UserList and UserString also define __getslice__ but I > don't consider them representative of the standards of non-string/list > classes. > > As an aside, if you're trying to show that inconsistencies in a > language are bad by referencing a paper showing that people who used > consistent (if incorrect) mental models scored better than those who > did not, you may have to explain further; I don't see the connection. The idea is that those who were consistant in their behavior, regardless of whether they were incorrect, can be trained to do things the correct way. That is to say, people who understand that X = Y will behave consistently regardless of context tend to do better than those who believe that it will do different things. Introducing inconsistencies because it is *easy* for the writer of an API, makes it more difficult to learn said API. In this context, the assumption that one makes when slicing in Python (as stated by someone else whom I can't remember in this thread): X[0:0] == type(X)(). That works _everywhere_ in Python where slices are allowed (except for buffers, which are generally rarely used except by certain crazies (like myself)). By not making it true here, we would be adding an exception to the rule. Special cases aren't special enough to break the rules. I'm not going to go all gloom and doom on you; maybe no one will ever have a situation where it is necessary. But implementing "slice of match returns a slice" isn't impossible, whether it is done via subclass, or by direct manipulation of the match struct. And not implementing the functionality because we are *lazy* isn't a terribly good excuse to give someone if/when they run into this. - Josiah From kbk at shore.net Thu Dec 7 06:14:15 2006 From: kbk at shore.net (Kurt B. Kaiser) Date: Thu, 7 Dec 2006 00:14:15 -0500 (EST) Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200612070514.kB75EF59004829@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 413 open ( +6) / 3489 closed ( +5) / 3902 total (+11) Bugs : 943 open ( +7) / 6364 closed ( +1) / 7307 total ( +8) RFE : 246 open ( +0) / 244 closed ( +0) / 490 total ( +0) New / Reopened Patches ______________________ popen() slow on AIX due to large FOPEN_MAX value (2006-12-01) http://python.org/sf/1607087 opened by John L. Allen bug# 1607041: Condition.wait timeout fails on clock change (2006-12-01) http://python.org/sf/1607149 opened by BC Optional Argument Syntax (2006-12-02) http://python.org/sf/1607548 opened by Tony Lownds Makefile fix (2006-12-03) http://python.org/sf/1608267 opened by Hasan Diwan Race condition in os.makedirs (2006-12-04) http://python.org/sf/1608579 opened by James Bowes Fix pickle doc typo "date", should be "data" (2006-12-04) CLOSED http://python.org/sf/1608758 opened by Graham Horler Py_FileSystemDefaultEncoding can be non-canonical (2006-12-04) http://python.org/sf/1608805 opened by Stephan R.A. Deibel Docstring support for ctypesgen (2006-12-05) CLOSED http://python.org/sf/1609108 opened by David Remahl #1603424 subprocess.py wrongly claims 2.2 compatibility. (2006-12-05) http://python.org/sf/1609282 opened by Robert Carr traceback on exit if syslog handler fails to initialize (2006-12-05) http://python.org/sf/1609407 opened by Jeremy Katz tarfile.py: Fix for bug #1609958 (2006-12-06) CLOSED http://python.org/sf/1610437 opened by Lars Gust?bel Patches Closed ______________ Socket module is not thread-safe (2006-08-22) http://python.org/sf/1544279 closed by loewis ConfigParser to accept a custom dict to allow ordering (2005-12-01) http://python.org/sf/1371075 closed by loewis Fix pickle doc typo "date", should be "data" (2006-12-05) http://python.org/sf/1608758 closed by quiver Docstring support for ctypesgen (2006-12-05) http://python.org/sf/1609108 closed by theller tarfile.py: Fix for bug #1609958 (2006-12-06) http://python.org/sf/1610437 closed by gbrandl New / Reopened Bugs ___________________ Condition.wait timeout fails when system clock changes (2006-12-01) http://python.org/sf/1607041 opened by BC Incorrect use of 'int' and 'long' descriptions (2006-12-01) CLOSED http://python.org/sf/1607061 opened by John logging %(module)s reporting wrong modules (2006-11-29) http://python.org/sf/1605110 reopened by mad-marty segfault in python24.dll (2006-12-03) http://python.org/sf/1607759 opened by joe mailbox.Maildir re-reads directory too often (2006-12-03) http://python.org/sf/1607951 opened by Matthias Klose Sloppy error checking in listdir() for Posix (2006-12-04) http://python.org/sf/1608818 opened by Stephan R.A. Deibel PyThread_release_lock with pthreads munges errno (2006-12-04) http://python.org/sf/1608921 opened by Stephan R.A. Deibel tarfile archive paths limited to less than 100 chars (2006-12-06) CLOSED http://python.org/sf/1609958 opened by Frank Rehberger GUI for Python 2.3, 2.4, and 2.5 is very sluggish (2006-12-06) http://python.org/sf/1610485 opened by g4rlik Bugs Closed ___________ Incorrect use of 'int' and 'long' descriptions (2006-12-01) http://python.org/sf/1607061 closed by gbrandl not configured for tk (2006-10-18) http://python.org/sf/1579931 closed by sf-robot tarfile archive paths limited to less than 100 chars (2006-12-06) http://python.org/sf/1609958 closed by gbrandl From tim.peters at gmail.com Thu Dec 7 06:14:51 2006 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 7 Dec 2006 00:14:51 -0500 Subject: [Python-Dev] Threading, atexit, and logging In-Reply-To: <45770DD7.50803@v.loewis.de> References: <45767328.5010701@v.loewis.de> <1f7befae0612060047n4b6fc89eq4f192dd1e50ff51@mail.gmail.com> <45770DD7.50803@v.loewis.de> Message-ID: <1f7befae0612062114j6c51e12ci3952e3602edceceb@mail.gmail.com> [Martin v. L?wis] >>> Upon investigation, it turns out that the file descriptor >>> is closed because the logging atexit handler is invoked. >>> This is surprising, as the program is far from exiting at >>> this point. [Tim Peters] >> But the main thread is done, right? [Martin] > Wrong. main.py (which is the __main__ script in the demo > code) is done, yes. Fine, but the main thread /has/ entered Py_Finalize(). That's key here, and wasn't clear originally. > However, threading.py has machinery to not terminate the main > thread as long as there are non-daemon threads. Right. ... >> IMO, `threading` shouldn't use atexit at all. > That is (in a way) my proposal (although I suggest to use > sys.exitfunc instead). Same thing to me. I'd rather thread cleanup, which is part of the Python core, not rely on any of the user-visible (hence also user-screwable) "do something at shutdown" gimmicks. Thread cleanup is only vaguely related to that concept because "cleanup" here implies waiting for an arbitrarily long time until all thread threads decide on their own to quit. That's not something to be cleaned up /at/ shutdown time, it's waiting (potentially forever!) /for/ shutdown time, and that mismatch is really the source of the problem. >> It's expedient :-) So was using atexit for this to begin with. >> Probably "good enough". I'd rather, e.g., that `threading` stuff an >> exit function into a module global, and change Py_Finalize() to look >> for that and run it (if present) before invoking call_sys_exitfunc(). > Ok, that's what I'll do then. > > Yet another alternative would be to have the "daemonic" thread feature > in the thread module itself (along with keeping track of a list of > all running non-daemonic thread). Sorry, I couldn't follow the intent there. Not obvious to me how moving this stuff from `threading` into `thread` would make it easier(?) for the implementation to wait for non-daemon threads to finish. From martin at v.loewis.de Thu Dec 7 06:24:23 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Dec 2006 06:24:23 +0100 Subject: [Python-Dev] Threading, atexit, and logging In-Reply-To: <1f7befae0612062114j6c51e12ci3952e3602edceceb@mail.gmail.com> References: <45767328.5010701@v.loewis.de> <1f7befae0612060047n4b6fc89eq4f192dd1e50ff51@mail.gmail.com> <45770DD7.50803@v.loewis.de> <1f7befae0612062114j6c51e12ci3952e3602edceceb@mail.gmail.com> Message-ID: <4577A587.6030401@v.loewis.de> Tim Peters schrieb: > Sorry, I couldn't follow the intent there. Not obvious to me how > moving this stuff from `threading` into `thread` would make it > easier(?) for the implementation to wait for non-daemon threads to > finish. Currently, if you create a thread through the thread module (rather than threading), it won't participate in the "wait until all threads are done" algorithm - you have to use the threading module. Moving it into the thread module would allow to cover all threads. Also, if the interpreter invokes, say, threading._shutdown(): that's also "user-screwable", as a user may put something else into threading._shutdown. To make it non-visible, it has to be in C, not Python (and even then it might be still visible to extension modules). Regards, Martin From tim.peters at gmail.com Thu Dec 7 06:54:42 2006 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 7 Dec 2006 00:54:42 -0500 Subject: [Python-Dev] Threading, atexit, and logging In-Reply-To: <4577A587.6030401@v.loewis.de> References: <45767328.5010701@v.loewis.de> <1f7befae0612060047n4b6fc89eq4f192dd1e50ff51@mail.gmail.com> <45770DD7.50803@v.loewis.de> <1f7befae0612062114j6c51e12ci3952e3602edceceb@mail.gmail.com> <4577A587.6030401@v.loewis.de> Message-ID: <1f7befae0612062154k76077fccl961fe3df90f3bdc9@mail.gmail.com> [Tim Peters] >> Sorry, I couldn't follow the intent there. Not obvious to me how >> moving this stuff from `threading` into `thread` would make it >> easier(?) for the implementation to wait for non-daemon threads to >> finish. [Martin v. L?wis] > Currently, if you create a thread through the thread module > (rather than threading), it won't participate in the "wait until > all threads are done" algorithm - you have to use the threading > module. Moving it into the thread module would allow to cover all > threads. True, but that doesn't appear to have any bearing on the bug originally discussed. You introduced this as "yet another alternative" in the context of how to address the original complaint, but if that was the intent, I still don't get it. Regardless, I personally view the `thread` module as being "de facto" deprecated. If someone /wants/ the ability to create a non-daemon thread, that the ability is only available via `threading` is an incentive to move to the newer, saner module. Besides, if the daemon distinction were grafted on to `thread` threads too, it would have to default to daemonic (a different default than `threading` threads) else be incompatible with current `thread` thread behavior. I personally don't want to add new features to `thread` threads in any case. > Also, if the interpreter invokes, say, threading._shutdown(): > that's also "user-screwable", as a user may put something else > into threading._shutdown. To make it non-visible, it has to be > in C, not Python (and even then it might be still visible to > extension modules). The leading underscore makes it officially invisible <0.7 wink>, and users would have to both seek it out and go out of their way to screw it. If some user believes they have a need to mess wtih threading._shutdown(), that's fine by me too. The problem with atexit and sys.exitfunc is that users can get in trouble now simply by using them in documented ways, because `threading` also uses them (but inappropriately so, IMO). Indeed, that's all the `logging` module did here. While this next is also irrelevant to the original complaint, I think it was also a minor mistake to build the newer atexit gimmicks on top of sys.exitfunc (same reason: a user can destroy the atexit functionality quite innocently if they happen to use sys.exitfunc after they (or code they build on) happens to import `atexit`. From fredrik at pythonware.com Thu Dec 7 08:15:23 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 07 Dec 2006 08:15:23 +0100 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> <20061206162923.9772.JCARLSON@uci.edu> Message-ID: Michael Urman wrote: > The idea that slicing a match object should produce a match object > sounds like a foolish consistency to me. well, the idea that adding m[x] as a convenience alias for m.group(x) automatically turns m into a list-style sequence that also has to support full slicing sounds like an utterly foolish consistency to me. the OP's original idea was to make a common use case slightly easier to use. if anyone wants to argue for other additions to the match object API, they should at least come up with use cases based on real existing code. (and while you guys are waiting, I suggest you start a new thread where you discuss some other inconsistency that would be easy to solve with more code in the interpreter, like why "-", "/", and "**" doesn't work for strings, lists don't have a "copy" method, sets and lists have different API:s for adding things, we have hex() and oct() but no bin(), str.translate and unicode.translate take different arguments, etc. get to work!) From alastair at alastairs-place.net Thu Dec 7 13:28:56 2006 From: alastair at alastairs-place.net (Alastair Houghton) Date: Thu, 7 Dec 2006 12:28:56 +0000 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <20061206174849.9779.JCARLSON@uci.edu> References: <20061206162923.9772.JCARLSON@uci.edu> <20061206174849.9779.JCARLSON@uci.edu> Message-ID: <12D4D0D4-C4F7-47E1-9356-A5862A5400A3@alastairs-place.net> On 7 Dec 2006, at 02:01, Josiah Carlson wrote: > Alastair Houghton wrote: >> On 7 Dec 2006, at 01:01, Josiah Carlson wrote: >>> If we don't want >>> slicing, or if prodicing a slice would produce a semantically >>> questionable state, then lets not do it. >> >> ...if you return match objects from slicing, you have problems like m >> [::-1].groups(). *I* don't know what that should return. > > I would argue that any 'step' != 1 has no semantically correct result > for slicing on a match object, so we shouldn't support it. OK, but even then, if you're returning a match object, how about the following: >>> m = re.match('(A)(B)(C)(D)(E)', 'ABCDE') >>> print m[0] ABCDE >>> n = m[2:5] >>> print list(n) ['B', 'C', 'D'] >>> print n[0] B >>> print n.group(0) B The problem I have with it is that it's violating the invariant that match objects should return the whole match in group(0). It's these kinds of things that make me think that slices shouldn't have all of the methods of a match object. I think that's probably why various others have suggested not supporting slicing, but I don't think it's necessary to avoid it as long as it has clearly specified behaviour. >> If you're worried about types, you could do something like this: >> >> generic match object >> | >> +--------------+-------------+ >> | | >> real match object match object slice > > I believe the above is unnecessary. Slicing a match could produce > another match. It's all internal data semantics. Sure. My point, though, was that you could view (from an external perspective) all results as instances of "generic match object", which might not have as many methods. Interestingly, at present, the match object type itself is an implementation detail; e.g. for SRE, it's an _sre.SRE_Match object. It's only the API that's documented, not the type. Kind regards, Alastair. -- http://alastairs-place.net From alastair at alastairs-place.net Thu Dec 7 14:00:26 2006 From: alastair at alastairs-place.net (Alastair Houghton) Date: Thu, 7 Dec 2006 13:00:26 +0000 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> <20061206162923.9772.JCARLSON@uci.edu> Message-ID: <66857090-9586-4C32-AC57-F18B19A5D576@alastairs-place.net> On 7 Dec 2006, at 07:15, Fredrik Lundh wrote: > Michael Urman wrote: > >> The idea that slicing a match object should produce a match object >> sounds like a foolish consistency to me. > > well, the idea that adding m[x] as a convenience alias for m.group(x) > automatically turns m into a list-style sequence that also has to > support full slicing sounds like an utterly foolish consistency to me. How about we remove the word "foolish" from the debate? > the OP's original idea was to make a common use case slightly > easier to > use. if anyone wants to argue for other additions to the match object > API, they should at least come up with use cases based on real > existing > code. An example where it might be useful: m = re.match('(?:([0-9]+) ([0-9]+) ([0-9]+) ([0-9]+) (?Prect)' '|([0-9]+) ([0-9]+) ([0-9]+) (?Pcircle))', lineFromFile) if m['rect']: drawRectangle(m[1:5]) elif m['circle']: drawCircle(m[1:3], m[3]) Is that really so outlandish? I'm not saying that this is necessarily the best way, but why force people to write list(m)[1:5] or [m[i] for i in xrange(1,5)] ?? If the only reason is that some of the match object APIs, which I maintain are very unlikely to be wanted on a slice anyway, can't possibly produce consistent results, then why not just do away with the APIs and return a tuple or something instead? That way you can treat the match object as if it were just a tuple (which it could easily have been). > (and while you guys are waiting, I suggest you start a new thread > where > you discuss some other inconsistency that would be easy to solve with > more code in the interpreter, like why "-", "/", and "**" doesn't work > for strings, lists don't have a "copy" method, sets and lists have > different API:s for adding things, we have hex() and oct() but no > bin(), > str.translate and unicode.translate take different arguments, etc. > get > to work!) Oh come on! Comparing this with exponentiating strings is just not helpful. Kind regards, Alastair. -- http://alastairs-place.net From murman at gmail.com Thu Dec 7 15:28:30 2006 From: murman at gmail.com (Michael Urman) Date: Thu, 7 Dec 2006 08:28:30 -0600 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> <20061206162923.9772.JCARLSON@uci.edu> Message-ID: On 12/7/06, Fredrik Lundh wrote: > (and while you guys are waiting, I suggest you start a new thread where > you discuss some other inconsistency that would be easy to solve with > more code in the interpreter, like why "-", "/", and "**" doesn't work > for strings, lists don't have a "copy" method, sets and lists have > different API:s for adding things, we have hex() and oct() but no bin(), > str.translate and unicode.translate take different arguments, etc. get > to work!) Personally I'd love a way to get an unbound method that handles either str or unicode instances. Perhaps py3k's unicode realignment will effectively give me that. (And agreed on there being no reason that supporting indexing requires supporting slicing. But also agreed that match slicing could be as useful as indexing. Really I don't use regexps enough in Python to have a position; I was more interested in figuring out where the type(m) == type(m[:]) idea had come from, as I had never formed it.) -- Michael Urman http://www.tortall.net/mu/blog From murman at gmail.com Thu Dec 7 15:47:01 2006 From: murman at gmail.com (Michael Urman) Date: Thu, 7 Dec 2006 08:47:01 -0600 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <20061206182848.977C.JCARLSON@uci.edu> References: <20061206162923.9772.JCARLSON@uci.edu> <20061206182848.977C.JCARLSON@uci.edu> Message-ID: On 12/6/06, Josiah Carlson wrote: > Special cases aren't special enough to break the rules. Sure, but where is this rule that would be broken? I've seen it invoked, but I've never felt it myself. I seriously thought of slicing as returning a list of elements per range(start,stop,skip), with the special case being str (and then unicode followed)'s type preservation. This is done because a list of characters is a pain to work with in most contexts, so there's an implicit ''.join on the list. And because assuming that the joined string is the desired result, it's much faster to have just built it in the first place. A pure practicality beats purity argument. We both arrive at the same place in that we have a model describing the behavior for list/str/unicode, but they're different models when extended outside. Now that I see the connection you're drawing between your argument and the paper, I don't believe it's directly inspired by the paper. I read the paper to say those who could create and work with a set of rules, could learn to work with the correct rules. Consistency in Python makes things easier on everyone because there's less to remember, not because it makes us better learners of the skills necessary for programming well. The arguments I saw in the paper only addressed the second point. -- Michael Urman http://www.tortall.net/mu/blog From jcarlson at uci.edu Thu Dec 7 17:58:20 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 07 Dec 2006 08:58:20 -0800 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: Message-ID: <20061207084925.978F.JCARLSON@uci.edu> "Michael Urman" wrote: > On 12/7/06, Fredrik Lundh wrote: > > (and while you guys are waiting, I suggest you start a new thread where > > you discuss some other inconsistency that would be easy to solve with > > more code in the interpreter, like why "-", "/", and "**" doesn't work > > for strings, lists don't have a "copy" method, sets and lists have > > different API:s for adding things, we have hex() and oct() but no bin(), > > str.translate and unicode.translate take different arguments, etc. get > > to work!) > > Personally I'd love a way to get an unbound method that handles either > str or unicode instances. Perhaps py3k's unicode realignment will > effectively give me that. Immutable byte strings won't exist in Py3k, and the mutable byte strings (bytes) won't support very many, if any current string/unicode methods. No bytes.replace, bytes.split, bytes.partition, etc. So no, Py3k's unicode change won't get you that. All it will get you is that every string you interact with; literals, file.read, etc., will all be text (equivalent to Python 2.x unicode). - Josiah From talin at acm.org Thu Dec 7 18:21:22 2006 From: talin at acm.org (Talin) Date: Thu, 07 Dec 2006 09:21:22 -0800 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> <20061206162923.9772.JCARLSON@uci.edu> Message-ID: <45784D92.4000100@acm.org> Fredrik Lundh wrote: > Michael Urman wrote: > >> The idea that slicing a match object should produce a match object >> sounds like a foolish consistency to me. > > well, the idea that adding m[x] as a convenience alias for m.group(x) > automatically turns m into a list-style sequence that also has to > support full slicing sounds like an utterly foolish consistency to me. Maybe instead of considering a match object to be a sequence, a match object should be considered a map? After all, we do have named, as well as numbered, groups...? -- Talin From fredrik at pythonware.com Thu Dec 7 18:43:36 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 07 Dec 2006 18:43:36 +0100 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <45784D92.4000100@acm.org> References: <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> <20061206162923.9772.JCARLSON@uci.edu> <45784D92.4000100@acm.org> Message-ID: Talin wrote: > Maybe instead of considering a match object to be a sequence, a match > object should be considered a map? sure, except for one small thing. from earlier in this thread: > Ka-Ping Yee wrote: > >> I'd say, don't pretend m is a sequence. Pretend it's a mapping. >> Then the conceptual issues go away. to which I replied: > almost; that would mean returning KeyError instead of IndexError for > groups that don't exist, which means that the common pattern > > a, b, c = m.groups() > > cannot be rewritten as > > _, a, b, c = m > > which would, perhaps, be a bit unfortunate. From janssen at parc.com Thu Dec 7 18:48:52 2006 From: janssen at parc.com (Bill Janssen) Date: Thu, 7 Dec 2006 09:48:52 PST Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <45784D92.4000100@acm.org> References: <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> <20061206162923.9772.JCARLSON@uci.edu> <45784D92.4000100@acm.org> Message-ID: <06Dec7.094857pst."58648"@synergy1.parc.xerox.com> > Maybe instead of considering a match object to be a sequence, a match > object should be considered a map? After all, we do have named, as well > as numbered, groups...? To me, that makes a lot more sense. Bill From talin at acm.org Thu Dec 7 19:24:02 2006 From: talin at acm.org (Talin) Date: Thu, 07 Dec 2006 10:24:02 -0800 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> <20061206162923.9772.JCARLSON@uci.edu> <45784D92.4000100@acm.org> Message-ID: <45785C42.8040608@acm.org> Fredrik Lundh wrote: > Talin wrote: > >> Maybe instead of considering a match object to be a sequence, a match >> object should be considered a map? > > sure, except for one small thing. from earlier in this thread: > > > Ka-Ping Yee wrote: > > > >> I'd say, don't pretend m is a sequence. Pretend it's a mapping. > >> Then the conceptual issues go away. > > to which I replied: > > > almost; that would mean returning KeyError instead of IndexError for > > groups that don't exist, which means that the common pattern > > > > a, b, c = m.groups() > > > > cannot be rewritten as > > > > _, a, b, c = m > > > > which would, perhaps, be a bit unfortunate. I think the confusion lies between the difference between 'group' (which takes either an integer or string argument, and behaves like a map), and 'groups' (which returns a tuple of the numbered arguments, and behaves like a sequence.) The original proposal was to make m[n] a synonym for m.group(n). "group()" is clearly map-like in its behavior. It seems to me that there's exactly three choices: -- Match objects behave like 'group' -- Match objects behave like 'groups' -- Match objects behave like 'group' some of the time, and like 'groups' some of the time, depending on how you refer to it. In case 1, a match object is clearly a map; In case 2, it's clearly a sequence; In case 3, it's neither, and all talk of consistency with either map or sequence is irrelevant. -- Talin From martin at v.loewis.de Thu Dec 7 19:54:52 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Dec 2006 19:54:52 +0100 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <66857090-9586-4C32-AC57-F18B19A5D576@alastairs-place.net> References: <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> <20061206162923.9772.JCARLSON@uci.edu> <66857090-9586-4C32-AC57-F18B19A5D576@alastairs-place.net> Message-ID: <4578637C.2050404@v.loewis.de> Alastair Houghton schrieb: > How about we remove the word "foolish" from the debate? We should table the debate. If you really want that feature, write a PEP. You want it, some people are opposed; a PEP is the procedure to settle the difference. Regards, Martin From aahz at pythoncraft.com Thu Dec 7 20:16:38 2006 From: aahz at pythoncraft.com (Aahz) Date: Thu, 7 Dec 2006 11:16:38 -0800 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <66857090-9586-4C32-AC57-F18B19A5D576@alastairs-place.net> References: <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> <20061206162923.9772.JCARLSON@uci.edu> <66857090-9586-4C32-AC57-F18B19A5D576@alastairs-place.net> Message-ID: <20061207191638.GB18494@panix.com> On Thu, Dec 07, 2006, Alastair Houghton wrote: > > An example where it might be useful: > > m = re.match('(?:([0-9]+) ([0-9]+) ([0-9]+) ([0-9]+) (?Prect)' > '|([0-9]+) ([0-9]+) ([0-9]+) (?Pcircle))', > lineFromFile) > > if m['rect']: > drawRectangle(m[1:5]) > elif m['circle']: > drawCircle(m[1:3], m[3]) > > Is that really so outlandish? Likely; normally I would expect that drawRectangle would break on string arguments instead of ints. I think that the amount of usefulness compared to problems doesn't really make this worth it. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ Member of the Groucho Marx Fan Club From alastair at alastairs-place.net Thu Dec 7 20:27:46 2006 From: alastair at alastairs-place.net (Alastair Houghton) Date: Thu, 7 Dec 2006 19:27:46 +0000 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <4578637C.2050404@v.loewis.de> References: <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> <20061206162923.9772.JCARLSON@uci.edu> <66857090-9586-4C32-AC57-F18B19A5D576@alastairs-place.net> <4578637C.2050404@v.loewis.de> Message-ID: <35D0BFE4-DF0D-4286-99EF-E711DA1DEA93@alastairs-place.net> On 7 Dec 2006, at 18:54, Martin v. L?wis wrote: > Alastair Houghton schrieb: >> How about we remove the word "foolish" from the debate? > > We should table the debate. If you really want that feature, > write a PEP. You want it, some people are opposed; a PEP is > the procedure to settle the difference. As I said a couple of e-mails back, I don't really have the time (I have lots of other things to do, most of them more important [to me, anyway]). If someone else agrees and wants to do it, great. If not, as I said before, I'm happy to let whoever do whatever. I might not agree, but that's my problem. Kind regards, Alastair. -- http://alastairs-place.net From Martin.vonLoewis at hpi.uni-potsdam.de Thu Dec 7 20:23:27 2006 From: Martin.vonLoewis at hpi.uni-potsdam.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 07 Dec 2006 20:23:27 +0100 Subject: [Python-Dev] Adding C99 bool to struct module Message-ID: <45786A2F.9070508@hpi.uni-potsdam.de> #1610575 suggests to introduce the 't' code to support the _Bool type where available, and uses char if it isn't available. Any objections to adding it? Regards, Martin From snaury at gmail.com Thu Dec 7 21:01:46 2006 From: snaury at gmail.com (Alexey Borzenkov) Date: Thu, 7 Dec 2006 23:01:46 +0300 Subject: [Python-Dev] features i'd like [Python 3000?] ... #4: interpolated strings ala perl In-Reply-To: <20061204001736.BBC1.JCARLSON@uci.edu> References: <4573AE30.9080305@666.com> <20061204001736.BBC1.JCARLSON@uci.edu> Message-ID: On 12/4/06, Josiah Carlson wrote: > With the proper mapping, this is trivial... > > class namelookup: > [...snip...] >>> foo = foo() > >>> print "%(foo.b)i + %(foo.a)i"%namelookup(locals()) > 2 + 1 > >>> > It can even be simpler and more powerful: class evallookup: def __init__(self, nsg, nsl): self.nsg = nsg self.nsl = nsl def __getitem__(self, name): return eval(name, self.nsg, self.nsl) class foo: a = 1 b = 2 >>> print "%(foo.a)i + %(foo.b)i = %(foo.a + foo.b)i" % evallookup(globals(), locals()) 1 + 2 = 3 >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20061207/372b3526/attachment.html From theller at ctypes.org Thu Dec 7 21:11:58 2006 From: theller at ctypes.org (Thomas Heller) Date: Thu, 07 Dec 2006 21:11:58 +0100 Subject: [Python-Dev] Adding C99 bool to struct module In-Reply-To: <45786A2F.9070508@hpi.uni-potsdam.de> References: <45786A2F.9070508@hpi.uni-potsdam.de> Message-ID: Martin v. L?wis schrieb: > #1610575 suggests to introduce the 't' code to support the _Bool > type where available, and uses char if it isn't available. > > Any objections to adding it? Not from me, although the patch should probably be extended to add a ctypes.c99_bool (or how it would be named) type - see also the comments that I posted to the tracker. Thomas From jcarlson at uci.edu Thu Dec 7 22:47:41 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 07 Dec 2006 13:47:41 -0800 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <12D4D0D4-C4F7-47E1-9356-A5862A5400A3@alastairs-place.net> References: <20061206174849.9779.JCARLSON@uci.edu> <12D4D0D4-C4F7-47E1-9356-A5862A5400A3@alastairs-place.net> Message-ID: <20061207134126.9798.JCARLSON@uci.edu> Alastair Houghton wrote: > On 7 Dec 2006, at 02:01, Josiah Carlson wrote: > > Alastair Houghton wrote: > >> On 7 Dec 2006, at 01:01, Josiah Carlson wrote: > >>> If we don't want > >>> slicing, or if prodicing a slice would produce a semantically > >>> questionable state, then lets not do it. > >> > >> ...if you return match objects from slicing, you have problems like m > >> [::-1].groups(). *I* don't know what that should return. > > > > I would argue that any 'step' != 1 has no semantically correct result > > for slicing on a match object, so we shouldn't support it. > > OK, but even then, if you're returning a match object, how about the > following: > > >>> m = re.match('(A)(B)(C)(D)(E)', 'ABCDE') > >>> print m[0] > ABCDE > >>> n = m[2:5] > >>> print list(n) > ['B', 'C', 'D'] > >>> print n[0] > B > >>> print n.group(0) > B > > The problem I have with it is that it's violating the invariant that > match objects should return the whole match in group(0). If we were going to go with slicing, then it would be fairly trivial to include the whole match range. Some portion of the underlying structure knows where the start of group 2 is, and knows where the end of group 5 is, so we can slice or otherwise use that for subsequent sliced groups. > Interestingly, at present, the match object type itself is an > implementation detail; e.g. for SRE, it's an _sre.SRE_Match object. > It's only the API that's documented, not the type. I believe that is the case with all built in cPython structures. - Josiah From jcarlson at uci.edu Thu Dec 7 22:57:49 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 07 Dec 2006 13:57:49 -0800 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <20061206182848.977C.JCARLSON@uci.edu> Message-ID: <20061207131552.9795.JCARLSON@uci.edu> "Michael Urman" wrote: > On 12/6/06, Josiah Carlson wrote: > > Special cases aren't special enough to break the rules. > > Sure, but where is this rule that would be broken? I've seen it > invoked, but I've never felt it myself. I seriously thought of slicing > as returning a list of elements per range(start,stop,skip), with the > special case being str (and then unicode followed)'s type > preservation. Tuple slicing doesn't return lists. Array slicing doesn't return lists. None of Numarray, Numeric, or Numpy array slicing returns lists. Only list slicing returns lists in current stdlib and major array package Python. Someone please correct me if I am wrong. > This is done because a list of characters is a pain to work with in > most contexts, so there's an implicit ''.join on the list. And because > assuming that the joined string is the desired result, it's much > faster to have just built it in the first place. A pure practicality > beats purity argument. Python returns strings from string slices not because of a "practicality beats purity" argument, but because not returning a string from string slicing is *insane*. The operations on strings tend to not be of the kind that is done with lists (insertion, sorting, etc.), they are typically linguistic, parsing, or data related (chop, append, prepend, scan for X, etc.). Also, typically, each item in a list is a singular item. Whereas in a string, typically _blocks of characters_ represent a single item: spaces, words, short, long, float, double, etc. (the latter being related to packed representations of data structures, as is the case in some socket protocols). Semantically, lists differ from strings in various *substantial* ways, which is why you will never see a typical user of Python asking for list.partition(lst). Strings have the sequence interface out of *convenience*, not because strings are like a list. Don't try to combine the two ideas. Also, when I said that strings/unicode were special and "has already been discussed ad-nauseum", I wasn't kidding. Take string and unicode out of the discussion and search google for the thousands of other threads that talk about why string and unicode are the way they are. They don't belong in this conversation. > We both arrive at the same place in that we have a model describing > the behavior for list/str/unicode, but they're different models when > extended outside. But this isn't about str/unicode (or buffer). For all other types available in Python with a slice operation, slicing them returns the same type as the original sequence. Expand that to 3rd party modules, and the case still holds for every package (at least those that I have seen and used). If you can point out an example (not str/unicode/buffer) for which this rule is broken in the stdlib or even any *major* 3rd party library, I'll buy you a cookie if we ever meet. > Now that I see the connection you're drawing between your argument and > the paper, I don't believe it's directly inspired by the paper. I read > the paper to say those who could create and work with a set of rules, > could learn to work with the correct rules. Consistency in Python > makes things easier on everyone because there's less to remember, not > because it makes us better learners of the skills necessary for > programming well. The arguments I saw in the paper only addressed the > second point. Right, but if you have a set of rules: 1. RULE 1 2. RULE 2 3. RULE 3 The above will be easier to understand than: 1. RULE 1 2. a. RULE 2 if X b. RULE 2a otherwise 3. RULE 3 This is the case even ignoring the paper. Special cases are more difficult to learn than no special cases. Want a real world example? English. The english language is so fraught with special cases that it is the only language for which dyslexia is known to exist (or was known to exist for many years, I haven't kept up on it). In the context of the paper, their findings suggested that those who could work with a *consistent* set of rules could be taught the right consistent rules. Toss in an inconsistency? Who knows if in this case it will make *any* difference in Python; regular expressions are already confusing for many people. This is a special case. There's a zen. Does practicality beat purity zen apply? I don't know. At this point I've just about stopped caring. Make the slice return a list, don't allow slicing, or make it a full on group variant. I don't really care at this point. Someone write a patch and lets go with it. Adding slicing producing a list should be easy after the main patch is done, and we can emulate it in Python if necessary. - Josiah From jcarlson at uci.edu Thu Dec 7 23:19:58 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 07 Dec 2006 14:19:58 -0800 Subject: [Python-Dev] features i'd like [Python 3000?] ... #4: interpolated strings ala perl In-Reply-To: References: <20061204001736.BBC1.JCARLSON@uci.edu> Message-ID: <20061207140927.979E.JCARLSON@uci.edu> "Alexey Borzenkov" wrote: > It can even be simpler and more powerful: > > class evallookup: > def __init__(self, nsg, nsl): > self.nsg = nsg > self.nsl = nsl > def __getitem__(self, name): > return eval(name, self.nsg, self.nsl) Never use eval in any code where you don't have 100% control of the content of what is being evaluated. And even then, never use eval. It doesn't make a difference in the earlier example, but I'm sure there are ways of breaking the above in nasty ways. - Josiah From greg.ewing at canterbury.ac.nz Fri Dec 8 00:35:30 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 08 Dec 2006 12:35:30 +1300 Subject: [Python-Dev] Threading, atexit, and logging In-Reply-To: <4577A587.6030401@v.loewis.de> References: <45767328.5010701@v.loewis.de> <1f7befae0612060047n4b6fc89eq4f192dd1e50ff51@mail.gmail.com> <45770DD7.50803@v.loewis.de> <1f7befae0612062114j6c51e12ci3952e3602edceceb@mail.gmail.com> <4577A587.6030401@v.loewis.de> Message-ID: <4578A542.4000007@canterbury.ac.nz> Martin v. L?wis wrote: > Also, if the interpreter invokes, say, threading._shutdown(): > that's also "user-screwable", as a user may put something else > into threading._shutdown. Although it would require being somewhat more deliberate, since threading._shutdown clearly has something to do with threading, whereas the atexit mechanism could get screwed up by someone who wasn't even thinking about what effect it might have on threading. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From kbk at shore.net Fri Dec 8 01:55:24 2006 From: kbk at shore.net (Kurt B. Kaiser) Date: Thu, 07 Dec 2006 19:55:24 -0500 Subject: [Python-Dev] RealClearPolitics - Articles - Exit Rumsfeld, Smiling In-Reply-To: <000001c71a4d$77a4c700$0201010a@karen> (Karen Treanor's message of "Fri, 8 Dec 2006 06:17:22 +0800") References: <000001c71a4d$77a4c700$0201010a@karen> Message-ID: <87irgn6war.fsf@hydra.bayview.thirdcreek.com> "Karen Treanor" writes: > Unfortunately this won't work for me, not by clicking on it or by > cutting and pasting it. The url below is wrapped, I assume you fixed the wrap? > > -----Original Message----- From: Kurt B. Kaiser [mailto:kbk at shore.net] >Sent: Friday, 8 December 2006 4:57 AM To: cwkaiser at fastmail.fm; Karen >Treanor Subject: RealClearPolitics - Articles - Exit Rumsfeld, Smiling > > http://www.realclearpolitics.com/articles/2006/12/exit_rumsfeld_smilin > g.html >> -- KBK From kbk at shore.net Fri Dec 8 02:10:58 2006 From: kbk at shore.net (Kurt B. Kaiser) Date: Thu, 07 Dec 2006 20:10:58 -0500 Subject: [Python-Dev] Spam and semi-spam and neo-spam In-Reply-To: <000001c71a52$7755f170$0201010a@karen> (Karen Treanor's message of "Fri, 8 Dec 2006 06:53:09 +0800") References: <000001c71a52$7755f170$0201010a@karen> Message-ID: <87ejrb6vkt.fsf@hydra.bayview.thirdcreek.com> "Karen Treanor" writes: > More and more people seem to be having a selection of email addresses, > some for general contact, some for business, and one private one for > only a select few contacts. This isn't possible in business, as your > email address is by necessity public, it has to be. > > I recently tried replying to a piece of spam and clicked on the opt > out button, but the sent item just bounced back as undeliverable. > Yahoo has a Spam tag facility but that only works for 'fixed' sites, > these random rotating ones each come in as discreet items and there's > no way to block them at source. Don't ever do that! 1. The majority of these links are bogus. 2. The majority of the rest use your click to verify you are a working email address. Then they sell the 'verified' list and you get even more spam. 3. Some of the links are fraudulent: they are phishing links which don't go where you expect, i.e. they use special characters to fool you. The result is you are directed to a site which infects your computer. -- KBK From kbk at shore.net Fri Dec 8 02:14:22 2006 From: kbk at shore.net (Kurt B. Kaiser) Date: Thu, 07 Dec 2006 20:14:22 -0500 Subject: [Python-Dev] Spam and semi-spam and neo-spam In-Reply-To: <87ejrb6vkt.fsf@hydra.bayview.thirdcreek.com> (Kurt B. Kaiser's message of "Thu, 07 Dec 2006 20:10:58 -0500") References: <000001c71a52$7755f170$0201010a@karen> <87ejrb6vkt.fsf@hydra.bayview.thirdcreek.com> Message-ID: <878xhj6vf5.fsf@hydra.bayview.thirdcreek.com> "Kurt B. Kaiser" writes: > Don't ever do that! Ahhh... sorry about that!! -- KBK From fghaibach at comcast.net Wed Dec 6 22:02:26 2006 From: fghaibach at comcast.net (a Fred) Date: Wed, 6 Dec 2006 21:02:26 +0000 (UTC) Subject: [Python-Dev] help for a noob - version for a sharp ARM Message-ID: I'm looking for advice on stripping down Python for an SBC to run Numpy and Scipy. I have the following notes on the system We have code that requires recent versions of Numpy and Scipy. The processor is a 32 bit Sharp ARM Sharp LH7A404 32 bit ARM922TDMI with 32MB of SDRAM. 512 MB flash disk - but 400 MB is reserved for data The OS is Linux only a tty interface is used. Does anyone have a feel as to whether Python for Arm is the correct starting point? What modules can I safely add/strip out? Will Python take more than 5-6 seconds to load? Will old Python releases, like 1.5.x, work with newer Numpy and Scipy? Thanks! a Fred From fredrik at pythonware.com Fri Dec 8 07:09:30 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 08 Dec 2006 07:09:30 +0100 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <45785C42.8040608@acm.org> References: <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> <20061206162923.9772.JCARLSON@uci.edu> <45784D92.4000100@acm.org> <45785C42.8040608@acm.org> Message-ID: Talin wrote: > The original proposal was to make m[n] a synonym for m.group(n). > "group()" is clearly map-like in its behavior. so have you checked what exception m.group(n) raises when you try to access a group that doesn't exist ? frankly, speaking as the original SRE author, I will now flip the bikeshed bit on all contributors to this thread, and consider it closed. I'll post a patch shortly. From jcarlson at uci.edu Fri Dec 8 08:07:16 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 07 Dec 2006 23:07:16 -0800 Subject: [Python-Dev] help for a noob - version for a sharp ARM In-Reply-To: References: Message-ID: <20061207220005.97A6.JCARLSON@uci.edu> a Fred wrote: > > I'm looking for advice on stripping down Python for an SBC to run Numpy > and Scipy. I have the following notes on the system > > We have code that requires recent versions of Numpy and Scipy. > The processor is a 32 bit Sharp ARM Sharp LH7A404 32 bit ARM922TDMI with > 32MB of SDRAM. > 512 MB flash disk - but 400 MB is reserved for data > The OS is Linux > only a tty interface is used. > > Does anyone have a feel as to whether Python for Arm is the correct > starting point? Sounds like plenty of muscle to handle Python certain domain number crunching problems. > What modules can I safely add/strip out? Most every C module you don't need. You can also compile without unicode. I believe you can more or less toss everything except for the base python binary and the pythonxy.so (or pythonxy.dll) for your platform, but I am most likely wrong. My (awful) suggestion: start with a Python installation in some user path, like ~/python . Toss everything and start adding things that it complains about until it starts working again. > Will Python take more than 5-6 seconds to load? I have previously gotten dos versions of Python (I think 1.5 or 1.4) to load on a 486 sx 33 with 8 megs of ram in a couple seconds. You shouldn't have issues with startup on an ARM. > Will old Python releases, like 1.5.x, work with newer Numpy and Scipy? I don't know. But you should be able to remove unused portions of Python for your platform. There are somewhat recent posts in this mailing lists about embedded systems and Python. Others may be able to help you more. - Josiah From ncoghlan at gmail.com Fri Dec 8 10:41:01 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 08 Dec 2006 19:41:01 +1000 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: References: <20061206122217.9766.JCARLSON@uci.edu> <427A1D94-E853-450B-9985-B73B6831AE8D@alastairs-place.net> <20061206162923.9772.JCARLSON@uci.edu> <45784D92.4000100@acm.org> <45785C42.8040608@acm.org> Message-ID: <4579332D.6010608@gmail.com> Fredrik Lundh wrote: > Talin wrote: > >> The original proposal was to make m[n] a synonym for m.group(n). >> "group()" is clearly map-like in its behavior. > > so have you checked what exception m.group(n) raises when you try to > access a group that doesn't exist ? The KeyError vs IndexError distinction is unreliable enough that I'll typically just catch LookupError if I don't know exactly what type I'm dealing with. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From alastair at alastairs-place.net Fri Dec 8 12:11:00 2006 From: alastair at alastairs-place.net (Alastair Houghton) Date: Fri, 8 Dec 2006 11:11:00 +0000 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <20061207134126.9798.JCARLSON@uci.edu> References: <20061206174849.9779.JCARLSON@uci.edu> <12D4D0D4-C4F7-47E1-9356-A5862A5400A3@alastairs-place.net> <20061207134126.9798.JCARLSON@uci.edu> Message-ID: <59CC94E3-7A46-40D4-A58C-6A4D15FB7E77@alastairs-place.net> On 7 Dec 2006, at 21:47, Josiah Carlson wrote: > Alastair Houghton wrote: >> On 7 Dec 2006, at 02:01, Josiah Carlson wrote: >>> Alastair Houghton wrote: >>>> On 7 Dec 2006, at 01:01, Josiah Carlson wrote: >>>>> If we don't want >>>>> slicing, or if prodicing a slice would produce a semantically >>>>> questionable state, then lets not do it. >>>> >>>> ...if you return match objects from slicing, you have problems >>>> like m >>>> [::-1].groups(). *I* don't know what that should return. >>> >>> I would argue that any 'step' != 1 has no semantically correct >>> result >>> for slicing on a match object, so we shouldn't support it. >> >> OK, but even then, if you're returning a match object, how about the >> following: >> >>>>> m = re.match('(A)(B)(C)(D)(E)', 'ABCDE') >>>>> print m[0] >> ABCDE >>>>> n = m[2:5] >>>>> print list(n) >> ['B', 'C', 'D'] >>>>> print n[0] >> B >>>>> print n.group(0) >> B >> >> The problem I have with it is that it's violating the invariant that >> match objects should return the whole match in group(0). > > If we were going to go with slicing, then it would be fairly > trivial to > include the whole match range. Some portion of the underlying > structure > knows where the start of group 2 is, and knows where the end of > group 5 > is, so we can slice or otherwise use that for subsequent sliced > groups. But then you're proposing that this thing (which looks like a tuple, when you're indexing it) should slice in a funny way. i.e. m = re.match('(A)(B)(C)(D)(E)', 'ABCDE') print m[0] ABCDE print list(m) ['ABCDE', 'A', 'B', 'C', 'D', 'E'] n = m[2:5] print list(n) ['BCD', 'B', 'C', 'D'] print len(n) 4 p = list(m)[2:5] print p ['B', 'C', 'D'] print len(p) Or are you saying that m[2:5][0] != m[2:5].group(0) but m[0] == m.group(0) ?? Either way I think that's *really* counter-intuitive. Honestly, I don't think that slicing should be supported if it's going to have to result in match objects, because I can't see a way to make them make sense. I think that's Frederik's objection also, but unlike me he doesn't feel that the slice operation should return something different (e.g. a tuple). Kind regards, Alastair. -- http://alastairs-place.net From guido at python.org Fri Dec 8 16:01:05 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Dec 2006 09:01:05 -0600 Subject: [Python-Dev] MSI being downloaded 10x more than all other files?! Message-ID: I looked through the python.org web stats (as I usually do when preparing for a keynote) and discovered that /ftp/python/2.5/python-2.5.msi is by far the top download -- 271,971 hits, more than 5x the next one, /ftp/python/2.5/Python-2.5.tgz (47,898 hits). Are these numbers real? (The byte counts suggest they are.) What could cause this dramatic popularity of Python on Windows? Does some vendor have an auto-install hook that installs Python whenever someone opens up their new computer? Or have we just hit the jackpot? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From scott+python-dev at scottdial.com Fri Dec 8 16:13:31 2006 From: scott+python-dev at scottdial.com (Scott Dial) Date: Fri, 08 Dec 2006 10:13:31 -0500 Subject: [Python-Dev] MSI being downloaded 10x more than all other files?! In-Reply-To: References: Message-ID: <4579811B.40400@scottdial.com> Guido van Rossum wrote: > What could cause this dramatic popularity of Python on Windows? You should ask yourself: 1) Where else can people grab Python on Windows? 2) Where else can people grab Python for [every other operating system]? Most distros are kind enough to provide their own mirror, I would say that easily accounts for the discrepancy since I am not aware of any other place to grab a windows install. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From theller at ctypes.org Fri Dec 8 16:18:03 2006 From: theller at ctypes.org (Thomas Heller) Date: Fri, 08 Dec 2006 16:18:03 +0100 Subject: [Python-Dev] MSI being downloaded 10x more than all other files?! In-Reply-To: References: Message-ID: Guido van Rossum schrieb: > I looked through the python.org web stats (as I usually do when > preparing for a keynote) and discovered that > /ftp/python/2.5/python-2.5.msi is by far the top download -- 271,971 > hits, more than 5x the next one, /ftp/python/2.5/Python-2.5.tgz > (47,898 hits). Are these numbers real? (The byte counts suggest they > are.) You probably have to add the downloads for /ftp/python/2.5/Python-2.5.tar.bz2 (31,492 hits) to the last number. Also interesting are the hits for 64-bit windows Pythons: /ftp/python/2.5/python-2.5.amd64.msi (23192 hits) /ftp/python/2.5/python-2.5.ia64.msi (22523 hits) > What could cause this dramatic popularity of Python on Windows? Have the ratios changed against past figures (too lazy to look them up now)? > Does some vendor have an auto-install hook that installs Python > whenever someone opens up their new computer? Or have we just hit the > jackpot? > Thomas From guido at python.org Fri Dec 8 16:46:42 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Dec 2006 09:46:42 -0600 Subject: [Python-Dev] [Python-checkins] MSI being downloaded 10x more than all otherfiles?! In-Reply-To: References: Message-ID: That was the month of October. If people believe these numbers are real, we're doing great!!! On 12/8/06, Fredrik Lundh wrote: > Guido van Rossum wrote: > > >I looked through the python.org web stats (as I usually do when > > preparing for a keynote) and discovered that > > /ftp/python/2.5/python-2.5.msi is by far the top download -- 271,971 > > hits, more than 5x the next one, /ftp/python/2.5/Python-2.5.tgz > > (47,898 hits). > > over what period ? > > fwiw, I've seen a 2.5x ratio for my stuff over the last year; I've always assumed > that most people on other platforms are simply getting their stuff from the vendor's > standard repository. > > > > > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Fri Dec 8 17:38:29 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri, 08 Dec 2006 08:38:29 -0800 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <59CC94E3-7A46-40D4-A58C-6A4D15FB7E77@alastairs-place.net> References: <20061207134126.9798.JCARLSON@uci.edu> <59CC94E3-7A46-40D4-A58C-6A4D15FB7E77@alastairs-place.net> Message-ID: <20061208082450.97B5.JCARLSON@uci.edu> Alastair Houghton wrote: > On 7 Dec 2006, at 21:47, Josiah Carlson wrote: > > If we were going to go with slicing, then it would be fairly > > trivial to > > include the whole match range. Some portion of the underlying > > structure > > knows where the start of group 2 is, and knows where the end of > > group 5 > > is, so we can slice or otherwise use that for subsequent sliced > > groups. > > But then you're proposing that this thing (which looks like a tuple, > when you're indexing it) should slice in a funny way. i.e. Let me be clear: I'm not proposing anything. I have little to no interest in seeing slices available to match objects, and as said in a message 20 minutes prior to the message you are replying to, "Make the slice return a list, don't allow slicing, or make it a full on group variant. I don't really care at this point." My statement in the email you replied to above was to say that if we wanted it to return a group, then we could include subsequent .group(0) with the same semantics as the original match object. At this point it doesn't matter, Frederik will produce what he wants to produce, and I'm sure most of us will be happy with the outcome. Those that are unhappy will need to write their own patch or deal with being unhappy. - Josiah From alastair at alastairs-place.net Fri Dec 8 17:37:44 2006 From: alastair at alastairs-place.net (Alastair Houghton) Date: Fri, 8 Dec 2006 16:37:44 +0000 Subject: [Python-Dev] [NPERS] Re: a feature i'd like to see in python #2: indexing of match objects In-Reply-To: <20061208082450.97B5.JCARLSON@uci.edu> References: <20061207134126.9798.JCARLSON@uci.edu> <59CC94E3-7A46-40D4-A58C-6A4D15FB7E77@alastairs-place.net> <20061208082450.97B5.JCARLSON@uci.edu> Message-ID: On 8 Dec 2006, at 16:38, Josiah Carlson wrote: > My statement in the email you replied to above was to say that if we > wanted it to return a group, then we could include subsequent .group > (0) > with the same semantics as the original match object. And my reply was simply to point out that that's not workable. > At this point it doesn't matter, Frederik will produce what he > wants to > produce, and I'm sure most of us will be happy with the outcome. > Those > that are unhappy will need to write their own patch or deal with being > unhappy. I believe I've already conceded that twice. Kind regards, Alastair. -- http://alastairs-place.net From fredrik at pythonware.com Fri Dec 8 17:40:25 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 08 Dec 2006 17:40:25 +0100 Subject: [Python-Dev] MSI being downloaded 10x more than all otherfiles?! In-Reply-To: References: Message-ID: Guido van Rossum wrote: > That was the month of October. > > If people believe these numbers are real, we're doing great!!! 2.5 was of course released in september, so I assume that part of what you're seeing is simply tinkerers upgrading their existing installations. plotting weekly figures for 2.4 and 2.5 over the last few months might give you a better idea of what the "sustained" download rate is, these days. (but we're definitely doing great. let there be no doubt about that ;-) From jcarlson at uci.edu Fri Dec 8 17:46:11 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri, 08 Dec 2006 08:46:11 -0800 Subject: [Python-Dev] [Python-checkins] MSI being downloaded 10x more than all otherfiles?! In-Reply-To: References: Message-ID: <20061208083951.97B8.JCARLSON@uci.edu> > On 12/8/06, Fredrik Lundh wrote: > > fwiw, I've seen a 2.5x ratio for my stuff over the last year; I've always assumed > > that most people on other platforms are simply getting their stuff from the vendor's > > standard repository. I've noticed a similar thing, but I've also noticed that some distributions seem to like to offer old (sometimes by a year or more) packages that are known to not work with those platforms. - Josiah From kbk at shore.net Fri Dec 8 18:46:21 2006 From: kbk at shore.net (Kurt B. Kaiser) Date: Fri, 08 Dec 2006 12:46:21 -0500 Subject: [Python-Dev] MSI being downloaded 10x more than all other files?! In-Reply-To: <4579811B.40400@scottdial.com> (Scott Dial's message of "Fri, 08 Dec 2006 10:13:31 -0500") References: <4579811B.40400@scottdial.com> Message-ID: <87zm9yi8lu.fsf@hydra.bayview.thirdcreek.com> Scott Dial writes: > You should ask yourself: > 1) Where else can people grab Python on Windows? > 2) Where else can people grab Python for [every other operating system]? > > Most distros are kind enough to provide their own mirror, I would say > that easily accounts for the discrepancy since I am not aware of any > other place to grab a windows install. I think that's right, Linux and BSD distros (and no doubt OS X) provide their own packaging of Python, while hackers sync with svn. But it's also fair to say that most users run Windows, and as Python goes mainstream, which seems to be the case lately, we should see an even greater proportion of Windows/everthing_else Python installations. -- KBK From martin at v.loewis.de Fri Dec 8 19:59:23 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Dec 2006 19:59:23 +0100 Subject: [Python-Dev] [Python-checkins] MSI being downloaded 10x more than all other files?! In-Reply-To: References: Message-ID: <4579B60B.3020907@v.loewis.de> Guido van Rossum schrieb: > I looked through the python.org web stats (as I usually do when > preparing for a keynote) and discovered that > /ftp/python/2.5/python-2.5.msi is by far the top download -- 271,971 > hits, more than 5x the next one, /ftp/python/2.5/Python-2.5.tgz > (47,898 hits). Are these numbers real? They should be. Thomas and I fixed the counting during the Google sprint. The log files are now processed once a day; before, they were rotated only once a week, and the web server stopped logging when the partition storing the log files were full (which frequently happened). So if you look at the statistics before August, you see that some days don't have any accesses counted - the stats can't be trusted in the period between the move away from creosote and the Google sprint. > (The byte counts suggest they > are.) What could cause this dramatic popularity of Python on Windows? > Does some vendor have an auto-install hook that installs Python > whenever someone opens up their new computer? Or have we just hit the > jackpot? I think this is Python's popularity. One factor is ready availability: "normal" users don't build Python from source. So Windows users download it from python.org, everybody else gets the binaries from the OS vendor. They come with the Linux distribution, they come with OS X, and they come with Solaris. Many of these people don't build Python themselves (and they can't, because they'll also have to rebuild mod_python, the Python subversion modules, etc). So the 50,000 source downloads are just from people who want to see the cutting egde, when their "regular" sources only provide 2.4 binaries (say). Also, Python 2.5 is relatively new. After some time, people will have local copies of the 2.5 MSI file, and don't redownload if they install on a new machine. Regards, Martin From martin at v.loewis.de Fri Dec 8 20:08:01 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Dec 2006 20:08:01 +0100 Subject: [Python-Dev] [Python-checkins] MSI being downloaded 10x more than all otherfiles?! In-Reply-To: References: Message-ID: <4579B811.1040403@v.loewis.de> Terry Reedy schrieb: > How do the first two months downloads of 2.5.msi compare to 2.4.msi? It's actually publicly available: http://www.python.org/webstats/ Regards, Martin From martin at v.loewis.de Fri Dec 8 20:11:07 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Dec 2006 20:11:07 +0100 Subject: [Python-Dev] MSI being downloaded 10x more than all other files?! In-Reply-To: References: Message-ID: <4579B8CB.8060803@v.loewis.de> Thomas Heller schrieb: > Also interesting are the hits for 64-bit windows Pythons: > > /ftp/python/2.5/python-2.5.amd64.msi (23192 hits) > /ftp/python/2.5/python-2.5.ia64.msi (22523 hits) But maybe misleading, as well. People just don't understand Win64: the hardware guy told them they have "64-bits", and now download everything with "64" in its name - just to see the message that this file isn't for their operating system. > Have the ratios changed against past figures (too lazy to look them up now)? They did, but also because the statistics weren't updated correctly until recently. Regards, Martin From martin at v.loewis.de Fri Dec 8 20:24:30 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 08 Dec 2006 20:24:30 +0100 Subject: [Python-Dev] MSI being downloaded 10x more than all other files?! In-Reply-To: <4579B8CB.8060803@v.loewis.de> References: <4579B8CB.8060803@v.loewis.de> Message-ID: <4579BBEE.3070102@v.loewis.de> Martin v. L?wis schrieb: >> Have the ratios changed against past figures (too lazy to look them up now)? > > They did, but also because the statistics weren't updated correctly > until recently. Qualifying a bit further: the last month that apparently had nearly correct statistics before September was February. We had these numbers for the MSI file: Jan: 2.4.2.msi 202046 Feb: 2.4.2.msi 209234 Sep: 2.4.3.msi 155689 2.5.msi 120817 Oct: 2.4.3.msi 36479 2.4.4.msi 38526 2.5.msi 271971 Nov: 2.4.4.msi 45696 2.5.msi 224247 Dec: 2.5.msi 55521 (Dec 8, 6:36 CET) Regards, Martin From mbk.lists at gmail.com Fri Dec 8 20:27:12 2006 From: mbk.lists at gmail.com (Mike Krell) Date: Fri, 8 Dec 2006 12:27:12 -0700 Subject: [Python-Dev] [Python-checkins] MSI being downloaded 10x more than all other files?! In-Reply-To: <4579B60B.3020907@v.loewis.de> References: <4579B60B.3020907@v.loewis.de> Message-ID: > I think this is Python's popularity. One factor is ready availability: > "normal" users don't build Python from source. So Windows users download > it from python.org, everybody else gets the binaries from the OS vendor. Another factor is that the ActiveState ActivePython distribution for Windows isn't available for 2.5 yet. Early adopters have even fewer places to go than normal. Mike From jordangreenberg at gmail.com Fri Dec 8 20:52:11 2006 From: jordangreenberg at gmail.com (Jordan Greenberg) Date: Fri, 08 Dec 2006 14:52:11 -0500 Subject: [Python-Dev] MSI being downloaded 10x more than all other files?! In-Reply-To: References: Message-ID: <4579C26B.8050004@gmail.com> Guido van Rossum wrote: > Or have we just hit the jackpot? Hi all, I've been lurking the list for a while (fun to know what you py-devs are up too!) but for this I have to speak up and give congratulations. You're all doing amazing work, and I think thats being reflected in these numbers. I know I recommend it to all of my friends, who are bogged down trying to learn Java or Perl, and they've all been amazed by how much of a joy it is to work in Python. So, if you have hit the jackpot, it's truly deserved. Congratulations again, and keep up the good work! -Jordan Greenberg From david at boddie.org.uk Sat Dec 9 01:41:40 2006 From: david at boddie.org.uk (David Boddie) Date: Sat, 9 Dec 2006 01:41:40 +0100 Subject: [Python-Dev] help for a noob - version for a sharp ARM Message-ID: <200612090141.41417.david@boddie.org.uk> On Fri Dec 8 08:07:16 CET 2006, Josiah Carlson wrote: > a Fred wrote: > > > > I'm looking for advice on stripping down Python for an SBC to run Numpy > > and Scipy. I have the following notes on the system > > > > We have code that requires recent versions of Numpy and Scipy. > > The processor is a 32 bit Sharp ARM Sharp LH7A404 32 bit ARM922TDMI with > > 32MB of SDRAM. > > 512 MB flash disk - but 400 MB is reserved for data > > The OS is Linux > > only a tty interface is used. > > > > Does anyone have a feel as to whether Python for Arm is the correct > > starting point? > > Sounds like plenty of muscle to handle Python certain domain number > crunching problems. It appears that this ARM variant doesn't have a floating point coprocessor. http://www.sharpsma.com/Page.aspx/americas/en/part/LH7A404/ I hope we're dealing with those problem domains here. ;-) > > What modules can I safely add/strip out? > > Most every C module you don't need. You can also compile without > unicode. I believe you can more or less toss everything except for the > base python binary and the pythonxy.so (or pythonxy.dll) for your > platform, but I am most likely wrong. I found that Python tries to import various modules when it starts up, so you might want to be a bit conservative with the ones you discard. However, if the target platform doesn't have libraries that various extension modules require then there's clearly no point in deploying those extensions. There are lots of irrelevant modules (for your application, at least) in the standard library. Anything that's platform-specific can go, apart from any Linux-specific modules, obviously, and even those might be unnecessary. I'm not sure if it's possible to automatically strip out Tkinter (lib-tk) and IDLE (idlelib), but it's probably worth doing so in your case. I should return to my own experiments and do all that again in a systematic way, just to see what can be removed. > My (awful) suggestion: start with > a Python installation in some user path, like ~/python . Toss > everything and start adding things that it complains about until it > starts working again. Or there's that approach. A module dependency graph would be useful to have, or even just a list of modules that are required to get to the interactive prompt. It might also be worthwhile compiling all the pure Python modules to bytecode and discarding the sources, but only if the compiled code is smaller, of course. > > Will Python take more than 5-6 seconds to load? > > I have previously gotten dos versions of Python (I think 1.5 or 1.4) to > load on a 486 sx 33 with 8 megs of ram in a couple seconds. You > shouldn't have issues with startup on an ARM. Less than a second, I would imagine. Extension modules will also increase the loading time, especially if they need to be perform some tasks when they start. > > Will old Python releases, like 1.5.x, work with newer Numpy and Scipy? > > I don't know. But you should be able to remove unused portions of > Python for your platform. There are somewhat recent posts in this > mailing lists about embedded systems and Python. Others may be able to > help you more. It should be possible to compile a version of Python for the ARM922TDMI that uses Thumb instructions. These are 16 bit instructions that are expanded on the fly so that they can be processed by the CPU as normal 32 bit wide instructions. This doesn't mean that the resulting executable and libraries will be half the size that they would have been when compiled normally, but you may be able to save some storage space that way. I have no idea whether this trick will help you save RAM as well. David From news at antroy.co.uk Fri Dec 8 23:03:08 2006 From: news at antroy.co.uk (Anthony Roy) Date: Fri, 08 Dec 2006 22:03:08 +0000 Subject: [Python-Dev] fileinput module tests Message-ID: <4579E11C.7020804@antroy.co.uk> Hi all, I have a patch for the fileinput.FileInput class, adding a parameter to the __init__ method called write_mode in order to specify the write mode when using the class with the inplace parameter set to True. Before I submit the patch, I've added a test to the test module, and noticed that the module is pretty messy, with half of the tests being run in the module body, and the rest in a large function body. I propose to refactor the module, moving the tests into a unittest.TestCase subclass (essentially unchanged, bar changing verify calls to self.assert_ calls, raise TestFailed(...) to self.fail(...) etc). I think this will add clarity and modularity to the tests, and bring them into line with the unittest based test suite used by the test_file module amongst others (which I'm guessing are substantially more up to date than test_fileinput). Any thought? Cheers, -- Ant... From fullung at gmail.com Wed Dec 6 00:51:57 2006 From: fullung at gmail.com (Albert Strasheim) Date: Wed, 6 Dec 2006 01:51:57 +0200 Subject: [Python-Dev] Windows SDK Message-ID: <003801c718c8$591f2e50$0a83a8c0@ratbert> Hello all (A quick google of the python-dev archive didn't turn up any discussions on this topic. If this has already been discussed, please accept my humble apologies.) As part of the Windows Vista release, Microsoft have created the "Windows SDK" that looks like Platform SDK on steroids. It includes 32-bit and 64-bit libraries and compilers, debugging tools, etc. and supports Windows XP, Windows Server 2003 and Windows Vista. Check the Microsoft Windows SDK Blog for more info: http://blogs.msdn.com/windowssdk/ Go here for links to ISO and web install: http://blogs.msdn.com/windowssdk/archive/2006/11/08/windows-sdk-for-windows-vista-is-live.aspx I'm only guessing here, but I think the Windows SDK is probably going to become the de facto standard for building software on Windows in the absence of Visual Studio. Has anybody else looked at the Windows SDK yet? Any thoughts on what needs to be done with distutils so that the Windows SDK can be supported in Python 2.6? Cheers, Albert Strasheim From brett at python.org Sat Dec 9 19:32:57 2006 From: brett at python.org (Brett Cannon) Date: Sat, 9 Dec 2006 10:32:57 -0800 Subject: [Python-Dev] fileinput module tests In-Reply-To: <4579E11C.7020804@antroy.co.uk> References: <4579E11C.7020804@antroy.co.uk> Message-ID: On 12/8/06, Anthony Roy wrote: > > Hi all, > > I have a patch for the fileinput.FileInput class, adding a parameter > to the __init__ method called write_mode in order to specify the write > mode when using the class with the inplace parameter set to True. > > Before I submit the patch, I've added a test to the test module, and > noticed that the module is pretty messy, with half of the tests being > run in the module body, and the rest in a large function body. > > I propose to refactor the module, moving the tests into a > unittest.TestCase subclass (essentially unchanged, bar changing verify > calls to self.assert_ calls, raise TestFailed(...) to self.fail(...) > etc). I think this will add clarity and modularity to the tests, and > bring them into line with the unittest based test suite used by the > test_file module amongst others (which I'm guessing are substantially > more up to date than test_fileinput). > > Any thought? Please do! It's always nice to have once less test not using unittest or doctest. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20061209/1d99d111/attachment.html From jcarlson at uci.edu Sat Dec 9 20:28:08 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat, 09 Dec 2006 11:28:08 -0800 Subject: [Python-Dev] Windows SDK In-Reply-To: <003801c718c8$591f2e50$0a83a8c0@ratbert> References: <003801c718c8$591f2e50$0a83a8c0@ratbert> Message-ID: <20061209112256.97E2.JCARLSON@uci.edu> "Albert Strasheim" wrote: > As part of the Windows Vista release, Microsoft have created the "Windows > SDK" that looks like Platform SDK on steroids. It includes 32-bit and 64-bit > libraries and compilers, debugging tools, etc. and supports Windows XP, > Windows Server 2003 and Windows Vista. Possibly not being able to run the compiler on Windows 2000 (or ME/98/95) in a "supported" mode may be a deal killer for people to not switch (and stick with the Platform SDK or Visual Studio 2003). I also wonder if the results of the compilations are usable on Windows 2000 (or ME/98/95). > I'm only guessing here, but I think the Windows SDK is probably going to > become the de facto standard for building software on Windows in the absence > of Visual Studio. Has anybody else looked at the Windows SDK yet? Any > thoughts on what needs to be done with distutils so that the Windows SDK can > be supported in Python 2.6? Someone will have to add/update a visual studio project file equivalent in the PCBuild directory. I think it would be nice if the person who updates the PCBuild stuff also tried to find someone with other Windows platforms (64 bit, win2k, possibly the 95/98/ME family). - Josiah From martin at v.loewis.de Sat Dec 9 21:10:11 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 09 Dec 2006 21:10:11 +0100 Subject: [Python-Dev] Windows SDK In-Reply-To: <003801c718c8$591f2e50$0a83a8c0@ratbert> References: <003801c718c8$591f2e50$0a83a8c0@ratbert> Message-ID: <457B1823.2030404@v.loewis.de> Albert Strasheim schrieb: > I'm only guessing here, but I think the Windows SDK is probably going to > become the de facto standard for building software on Windows in the absence > of Visual Studio. Has anybody else looked at the Windows SDK yet? Any > thoughts on what needs to be done with distutils so that the Windows SDK can > be supported in Python 2.6? As always, the question is what msvcrt version the compiler links with. For Itanium and AMD64, the SDK (compiler) links with mscvrt.dll, which is the CRT version distributed with the system. Python 2.5 is already build with the SDK compiler on these targets, so you can use the SDK to build extensions. To use the SDK compiler, open an SDK build environment, and set the environment variable DISTUTILS_USE_SDK. For x86, the situation is more difficult. I'm uncertain what CRT the x86 SDK compiler links with. Mixing CRTs may or may not work, depending on what precisely the extension module does. For 2.6, I hope we will switch to VS 2007 (assuming that's released at that point). Then, the question is whether the Windows SDK will support VS 2007. Regards, Martin From martin at v.loewis.de Sat Dec 9 22:19:42 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 09 Dec 2006 22:19:42 +0100 Subject: [Python-Dev] fileinput module tests In-Reply-To: <4579E11C.7020804@antroy.co.uk> References: <4579E11C.7020804@antroy.co.uk> Message-ID: <457B286E.3010106@v.loewis.de> Anthony Roy schrieb: > Any thought? In addition to what Brett said: please make functionality changes and refactoring separate patches (specify the order of application if one depends on the other). When reading commit message, or performing "svn blame", it's important to know whether a change is just a structural one, or a substantial one. Regards, Martin From nnorwitz at gmail.com Mon Dec 11 02:09:06 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sun, 10 Dec 2006 17:09:06 -0800 Subject: [Python-Dev] Looking for new Python-Dev Summarizers In-Reply-To: References: Message-ID: On 12/5/06, Steven Bethard wrote: > > As I'm now coming up on the last semester of so of my Ph.D., I need to > make some more time to get research done, and so I'm looking to retire > from the summaries. Hence, I'm soliciting for a good replacement (or a > few) who would be willing to pick up the job. Steve, Thanks for all the work you've put into the summaries. It helps many people keep up with python-dev even if they can't spend as much time as they'd like. As you say, it's a great way to learn a lot, have fun, and help out the community. Sometimes you even get a thank you. :-) Cheers, n From jimjjewett at gmail.com Mon Dec 11 23:57:20 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 11 Dec 2006 17:57:20 -0500 Subject: [Python-Dev] [Python-checkins] MSI being downloaded 10x more than all other files?! In-Reply-To: References: Message-ID: On 12/8/06, Guido van Rossum wrote: > /ftp/python/2.5/python-2.5.msi is by far the top download -- 271,971 > hits, more than 5x the next one, /ftp/python/2.5/Python-2.5.tgz > (47,898 hits). Are these numbers real? Why wouldn't it be? Most computers are running windows. Most people running some other operating system can get python from that someplace else -- and have to if they don't want to compile it themselves. (But recompiling isn't usually a barrier, as compilers usually come with the system.) Most people running windows cannot compile python. I'm not sure there even *is* a free way to do it; mingw may well compile additional extensions, but it can't compile the main python tree. There have been recipes for using free MSVC versions, but I'm not sure whether any work today. -jJ From guido at python.org Tue Dec 12 00:17:28 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Dec 2006 15:17:28 -0800 Subject: [Python-Dev] [Python-checkins] MSI being downloaded 10x more than all other files?! In-Reply-To: References: Message-ID: On 12/11/06, Jim Jewett wrote: > On 12/8/06, Guido van Rossum wrote: > > /ftp/python/2.5/python-2.5.msi is by far the top download -- 271,971 > > hits, more than 5x the next one, /ftp/python/2.5/Python-2.5.tgz > > (47,898 hits). Are these numbers real? > > Why wouldn't it be? Just because in the past the ratio of downloads for a particular version was always about 70% Windows vs. 30% source. Now it seems closer to 90/10. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From glyph at divmod.com Tue Dec 12 00:59:25 2006 From: glyph at divmod.com (glyph at divmod.com) Date: Mon, 11 Dec 2006 23:59:25 -0000 Subject: [Python-Dev] [Python-checkins] MSI being downloaded 10x more than all other files?! Message-ID: <20061211235925.14681.1576768442.divmod.xquotient.89@joule.divmod.com> On 11:17 pm, guido at python.org wrote: >On 12/11/06, Jim Jewett wrote: >> On 12/8/06, Guido van Rossum wrote: >> > /ftp/python/2.5/python-2.5.msi is by far the top download -- 271,971 >> > hits, more than 5x the next one, /ftp/python/2.5/Python-2.5.tgz >> > (47,898 hits). Are these numbers real? >> >> Why wouldn't it be? > >Just because in the past the ratio of downloads for a particular >version was always about 70% Windows vs. 30% source. Now it seems >closer to 90/10. Personally speaking, since switching to Ubuntu, I've been so happy with the speed of releases and the quality of packaged Python that I haven't downloaded a source release from python.org in over a year. If I need packages, they're already installed. If I need source from a release, I 'apt-get source' to conveniently install it from a (very fast) ubuntu mirror. When I need something outside the Ubuntu release structure, it's typically an SVN trunk checkout, not a release tarball. I don't know what Ubuntu's impact in the general user community has been, but it seems that the vast majority of python developers I interact with on a regular basis have switched. I wouldn't be surprised if this were a major part of the impact. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20061211/24057246/attachment.html From kbk at shore.net Tue Dec 12 06:47:05 2006 From: kbk at shore.net (Kurt B. Kaiser) Date: Tue, 12 Dec 2006 00:47:05 -0500 Subject: [Python-Dev] [Python-checkins] MSI being downloaded 10x more than all other files?! In-Reply-To: (Guido van Rossum's message of "Mon, 11 Dec 2006 15:17:28 -0800") References: Message-ID: <87d56pwtra.fsf@hydra.bayview.thirdcreek.com> "Guido van Rossum" writes: > Just because in the past the ratio of downloads for a particular > version was always about 70% Windows vs. 30% source. Now it seems > closer to 90/10. Going mainstream :-)) The Rails buzz seems to be jumping to Python lately. -- KBK From fredrik at pythonware.com Tue Dec 12 14:54:11 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 12 Dec 2006 14:54:11 +0100 Subject: [Python-Dev] [Python-checkins] MSI being downloaded 10x morethan all other files?! References: <87d56pwtra.fsf@hydra.bayview.thirdcreek.com> Message-ID: Kurt B. Kaiser wrote: > Going mainstream :-)) indeed. from what I can tell on my local market, we've "crossed the chasm" [1], and are seeing wider range of "pragmatists" adding Python to the tool chain. > The Rails buzz seems to be jumping to Python lately. fwiw, the people I see pick up Python haven't even heard of Ruby or Rails (not every- one is doing web 2.0 stuff, after all). 1) http://www.testing.com/writings/reviews/moore-chasm.html From steve at holdenweb.com Tue Dec 12 14:53:43 2006 From: steve at holdenweb.com (Steve Holden) Date: Tue, 12 Dec 2006 13:53:43 +0000 Subject: [Python-Dev] [Python-checkins] MSI being downloaded 10x more than all other files?! In-Reply-To: <87d56pwtra.fsf@hydra.bayview.thirdcreek.com> References: <87d56pwtra.fsf@hydra.bayview.thirdcreek.com> Message-ID: <457EB467.1060702@holdenweb.com> Kurt B. Kaiser wrote: > "Guido van Rossum" writes: > >> Just because in the past the ratio of downloads for a particular >> version was always about 70% Windows vs. 30% source. Now it seems >> closer to 90/10. > > Going mainstream :-)) > > The Rails buzz seems to be jumping to Python lately. > Though of course it would be interesting to know the growth in absolute download count: if there have only been half the number of downloads then we aren't winning at all ;-) regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden From daniel at thewatkins.org.uk Tue Dec 12 15:05:08 2006 From: daniel at thewatkins.org.uk (Daniel Watkins) Date: Tue, 12 Dec 2006 14:05:08 +0000 (UTC) Subject: [Python-Dev] Adding resume (206) support to urllib(2) Message-ID: Hi guys, I've just been putting together a podcasting doodad and have included resuming support in it. Is this something that's already in the pipeline or should I abstract it out to urllib and submit a patch? Dan From skip at pobox.com Tue Dec 12 15:52:51 2006 From: skip at pobox.com (skip at pobox.com) Date: Tue, 12 Dec 2006 08:52:51 -0600 Subject: [Python-Dev] Adding resume (206) support to urllib(2) In-Reply-To: References: Message-ID: <17790.49731.131545.176423@montanaro.dyndns.org> Dan> I've just been putting together a podcasting doodad and have Dan> included resuming support in it. Is this something that's already Dan> in the pipeline or should I abstract it out to urllib and submit a Dan> patch? Check urllib2 before putting any effort into extending urllib. If a patch is needed it should go in urllib2. Skip From kbk at shore.net Tue Dec 12 17:03:50 2006 From: kbk at shore.net (Kurt B. Kaiser) Date: Tue, 12 Dec 2006 11:03:50 -0500 Subject: [Python-Dev] [Python-checkins] MSI being downloaded 10x morethan all other files?! In-Reply-To: (Fredrik Lundh's message of "Tue, 12 Dec 2006 14:54:11 +0100") References: <87d56pwtra.fsf@hydra.bayview.thirdcreek.com> Message-ID: <87vekhummx.fsf@hydra.bayview.thirdcreek.com> "Fredrik Lundh" writes: >> The Rails buzz seems to be jumping to Python lately. > > fwiw, the people I see pick up Python haven't even heard of Ruby or > Rails (not every- one is doing web 2.0 stuff, after all). Yes, separate but related groups and issues. MIT's adopting Python in their introductory programs will drive other schools in our direction, I think. -- KBK From guido at python.org Tue Dec 12 17:37:50 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 12 Dec 2006 08:37:50 -0800 Subject: [Python-Dev] [Python-checkins] MSI being downloaded 10x morethan all other files?! In-Reply-To: <87vekhummx.fsf@hydra.bayview.thirdcreek.com> References: <87d56pwtra.fsf@hydra.bayview.thirdcreek.com> <87vekhummx.fsf@hydra.bayview.thirdcreek.com> Message-ID: And I just found out (after everyone else probably :-) that YouTube is almost entirely written in Python. (And now I can rub shoulders with the developers since they're all Googlers now... :-) On 12/12/06, Kurt B. Kaiser wrote: > "Fredrik Lundh" writes: > > >> The Rails buzz seems to be jumping to Python lately. > > > > fwiw, the people I see pick up Python haven't even heard of Ruby or > > Rails (not every- one is doing web 2.0 stuff, after all). > > Yes, separate but related groups and issues. > > MIT's adopting Python in their introductory programs will drive > other schools in our direction, I think. > > -- > KBK > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From simon at brunningonline.net Tue Dec 12 18:24:15 2006 From: simon at brunningonline.net (Simon Brunning) Date: Tue, 12 Dec 2006 17:24:15 +0000 Subject: [Python-Dev] [Python-checkins] MSI being downloaded 10x morethan all other files?! In-Reply-To: References: <87d56pwtra.fsf@hydra.bayview.thirdcreek.com> <87vekhummx.fsf@hydra.bayview.thirdcreek.com> Message-ID: <8c7f10c60612120924y34e62957s1a4f3d638ff78150@mail.gmail.com> On 12/12/06, Guido van Rossum wrote: > And I just found out (after everyone else probably :-) that YouTube is > almost entirely written in Python. (And now I can rub shoulders with > the developers since they're all Googlers now... :-) That'll put to bed any "Does Python scale" discussions. Cheers, Simon B From martin at v.loewis.de Wed Dec 13 01:39:04 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Dec 2006 01:39:04 +0100 Subject: [Python-Dev] Adding resume (206) support to urllib(2) In-Reply-To: References: Message-ID: <457F4BA8.5020507@v.loewis.de> Daniel Watkins schrieb: > I've just been putting together a podcasting doodad and have included resuming > support in it. Is this something that's already in the pipeline or should I > abstract it out to urllib and submit a patch? Not sure where you got the impression that 206 is "resume"; in my copy of the spec it's "partial content", and you must have put a Range: header into the request to get that in the first place. If I had to use that, I'd implement it right on top of httplib, and wouldn't bother with urllib*: this is really specific to http, and adding it to urllib would break the abstraction. In any case, there is no "pipeline" it may be in (except for changes that have already been committed to the trunk). Something may have been submitted as a patch or feature request, but a quick search reveals no relevant open issues. Regards, Martin From fredrik at pythonware.com Wed Dec 13 07:17:10 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 13 Dec 2006 07:17:10 +0100 Subject: [Python-Dev] Adding resume (206) support to urllib(2) In-Reply-To: <457F4BA8.5020507@v.loewis.de> References: <457F4BA8.5020507@v.loewis.de> Message-ID: Martin v. L?wis wrote: >> I've just been putting together a podcasting doodad and have included resuming >> support in it. Is this something that's already in the pipeline or should I >> abstract it out to urllib and submit a patch? > > Not sure where you got the impression that 206 is "resume"; in my copy > of the spec it's "partial content", and you must have put a Range: > header into the request to get that in the first place. > > If I had to use that, I'd implement it right on top of httplib, and > wouldn't bother with urllib*: this is really specific to http, and > adding it to urllib would break the abstraction. given that urllib2 already supports partial requests, I'm not sure I see the point of reimplementing this on top of httplib. an example: import urllib2 request = urllib2.Request("http://www.pythonware.com/daily/index.htm") request.add_header("range", "bytes=0-999") http_file = urllib2.urlopen(request) print http_file.headers["content-range"] print len(http_file.read()) this prints: bytes 0-999/245105 1000 From martin at v.loewis.de Wed Dec 13 08:30:00 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Dec 2006 08:30:00 +0100 Subject: [Python-Dev] Adding resume (206) support to urllib(2) In-Reply-To: References: <457F4BA8.5020507@v.loewis.de> Message-ID: <457FABF8.9050908@v.loewis.de> Fredrik Lundh schrieb: > given that urllib2 already supports partial requests, I'm not sure I see > the point of reimplementing this on top of httplib. an example: > > import urllib2 > > request = urllib2.Request("http://www.pythonware.com/daily/index.htm") > request.add_header("range", "bytes=0-999") But what does this do if the URL was a file URL, or an ftp URL? You have to know the syntax of the range header, and you have to know the syntax of the content-range header, to process it. With that, you can just as easily use httplib: py> import httplib py> h=httplib.HTTPConnection("www.pythonware.com") py> h.putrequest("GET","/daily/index.htm") py> h.putheader("range","bytes=0-999") py> h.endheaders() py> r=h.getresponse() py> r.getheader("content-range") 'bytes 0-999/245105' py> len(r.read()) 1000 If you add protocol-specifics to urllib, the abstraction that urllib provides goes away, and you are better off (IMO) to use the underlying protocol library in the first place. I'm not sure what the OP wanted to contribute in the first place (given that it "works" already either way), but it might have been a convenience API for the range header, and a parser for the content-range header. That should go IMO into httplib, so that all users of httplib get access to it, not just urllib*. Regards, Martin From kay.schluehr at gmx.net Wed Dec 13 08:32:45 2006 From: kay.schluehr at gmx.net (Kay Schluehr) Date: Wed, 13 Dec 2006 08:32:45 +0100 Subject: [Python-Dev] Open CPython VM for type information Message-ID: <457FAC9D.1030801@gmx.net> Dear Python developers, while the JVM is opened to support dynamically typed languages [1] I wonder if the CPyVM could not show more openness to statically typed languages? Hereby I don't think that much about arbitrary languages for the CPyVM but sublanguages like RPython which are "static enough" to be compilable into extension modules [2]. However creating C extension modules from Python code is not really the standard way I do think Python source should be handled. Instead I think about a more direct VM support like the JVM does, that provides opcodes like iload, fstore etc. Besides this I don't expect one canonical approach for these type information to be feeded into the CPyVM. There are type inferencer like rtyper for PyPy, Psycos continous bytecode monitoring and also other approaches are possible that use type feedback. Finally optional type declarations are also an issue; however I would like to see language design discussions being separated ( -> Python 3000 ) and focus on general APIs for compiler extensions / plugs that deal with type information ( the architecture ) and VM implementation issues. Thanks, Kay [1] http://jcp.org/en/jsr/detail?id=292 [2] http://codespeak.net/pypy/dist/pypy/doc/extcompiler.html From phd at phd.pp.ru Wed Dec 13 08:49:12 2006 From: phd at phd.pp.ru (Oleg Broytmann) Date: Wed, 13 Dec 2006 10:49:12 +0300 Subject: [Python-Dev] Adding resume (206) support to urllib(2) In-Reply-To: <457FABF8.9050908@v.loewis.de> References: <457F4BA8.5020507@v.loewis.de> <457FABF8.9050908@v.loewis.de> Message-ID: <20061213074912.GA3502@phd.pp.ru> On Wed, Dec 13, 2006 at 08:30:00AM +0100, "Martin v. L?wis" wrote: > If you add protocol-specifics to urllib, the abstraction that urllib > provides goes away, and you are better off (IMO) to use the underlying > protocol library in the first place. IMO you better don't because urllib2 provides not only an abstraction, but a lot of services (authenticated proxies, cached FTP files)... Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From fredrik at pythonware.com Wed Dec 13 08:58:46 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 13 Dec 2006 08:58:46 +0100 Subject: [Python-Dev] Adding resume (206) support to urllib(2) In-Reply-To: <457FABF8.9050908@v.loewis.de> References: <457F4BA8.5020507@v.loewis.de> <457FABF8.9050908@v.loewis.de> Message-ID: Martin v. L?wis wrote: >> given that urllib2 already supports partial requests, I'm not sure I see >> the point of reimplementing this on top of httplib. an example: >> >> import urllib2 >> >> request = urllib2.Request("http://www.pythonware.com/daily/index.htm") >> request.add_header("range", "bytes=0-999") > > But what does this do if the URL was a file URL, or an ftp URL? same thing as if you use range on a HTTP server that doesn't support ranges. you get all the data, and there's no content-range field in the response header. > You have to know the syntax of the range header, and you have to > know the syntax of the content-range header, to process it. With > that, you can just as easily use httplib: I'm not sure "as easily" is the right way to describe something that requires more code but yet leaves out practical things as redirection support, host and user-agent headers, etc. From martin at v.loewis.de Wed Dec 13 09:05:49 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Dec 2006 09:05:49 +0100 Subject: [Python-Dev] Adding resume (206) support to urllib(2) In-Reply-To: <20061213074912.GA3502@phd.pp.ru> References: <457F4BA8.5020507@v.loewis.de> <457FABF8.9050908@v.loewis.de> <20061213074912.GA3502@phd.pp.ru> Message-ID: <457FB45D.8080802@v.loewis.de> Oleg Broytmann schrieb: > IMO you better don't because urllib2 provides not only an abstraction, > but a lot of services (authenticated proxies, cached FTP files)... If you are using http ranges, cached FTP files won't do any good. As for authenticated proxies: I think they ought to be implemented in httplib as well. If everybody wants to become urllib just a better library to access http servers, I probably can't do much about it, though. Regards, Martin From phd at phd.pp.ru Wed Dec 13 09:18:36 2006 From: phd at phd.pp.ru (Oleg Broytmann) Date: Wed, 13 Dec 2006 11:18:36 +0300 Subject: [Python-Dev] Adding resume (206) support to urllib(2) In-Reply-To: <457FB45D.8080802@v.loewis.de> References: <457F4BA8.5020507@v.loewis.de> <457FABF8.9050908@v.loewis.de> <20061213074912.GA3502@phd.pp.ru> <457FB45D.8080802@v.loewis.de> Message-ID: <20061213081836.GA9589@phd.pp.ru> On Wed, Dec 13, 2006 at 09:05:49AM +0100, "Martin v. L?wis" wrote: > As for authenticated proxies: I think they ought to be implemented > in httplib as well. Agree. > If everybody wants to become urllib just a better library to access > http servers, I probably can't do much about it, though. HTTP is one of the most widely known and used protocol. Would you better have big httplib and small abstract urllib? so abstract it doesn't allow a user to change protocol-specific handling? Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From terry at jon.es Tue Dec 12 21:32:10 2006 From: terry at jon.es (Terry Jones) Date: Tue, 12 Dec 2006 21:32:10 +0100 Subject: [Python-Dev] [Python-checkins] MSI being downloaded 10x morethan all other files?! In-Reply-To: Your message at 08:37:50 on Tuesday, 12 December 2006 References: <87d56pwtra.fsf@hydra.bayview.thirdcreek.com> <87vekhummx.fsf@hydra.bayview.thirdcreek.com> Message-ID: <17791.4554.712959.78566@terry-jones-computer.local> >>>>> "Guido" == Guido van Rossum writes: Guido> And I just found out (after everyone else probably :-) that YouTube Guido> is almost entirely written in Python. (And now I can rub shoulders Guido> with the developers since they're all Googlers now... :-) Are any other details available? I'm mainly curious to know if they were using Twisted. Terry From walter at livinglogic.de Wed Dec 13 17:57:16 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 13 Dec 2006 17:57:16 +0100 Subject: [Python-Dev] Creating dicts from dict subclasses Message-ID: <458030EC.8050102@livinglogic.de> I tried to reimplement weakref.WeakValueDictionary as a subclass of dict. The test passes except for one problem: To compare results test_weakref.py converts a weakdict to a real dict via dict(weakdict). This no longer works because PyDict_Merge() does a PyDict_Check() on the argument and then ignores all overwritten methods. (The old version worked because UserDict.UserDict was used). The simplest solution is to replace the PyDict_Check() call with PyDict_CheckExact(), but this might slow things down too much, because the fallback code basically does: for key in iter(arg.keys()): self[key] = arg.__getitem__(key) Why can't we use: for key in iter(arg): self[key] = arg.__getitem__(key) instead? Servus, Walter From guido at python.org Wed Dec 13 19:52:34 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Dec 2006 10:52:34 -0800 Subject: [Python-Dev] [Python-checkins] MSI being downloaded 10x morethan all other files?! In-Reply-To: <17791.4554.712959.78566@terry-jones-computer.local> References: <87d56pwtra.fsf@hydra.bayview.thirdcreek.com> <87vekhummx.fsf@hydra.bayview.thirdcreek.com> <17791.4554.712959.78566@terry-jones-computer.local> Message-ID: No they aren't using Twisted. On 12/12/06, Terry Jones wrote: > >>>>> "Guido" == Guido van Rossum writes: > Guido> And I just found out (after everyone else probably :-) that YouTube > Guido> is almost entirely written in Python. (And now I can rub shoulders > Guido> with the developers since they're all Googlers now... :-) > > Are any other details available? I'm mainly curious to know if they were > using Twisted. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Dec 13 19:58:55 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Dec 2006 10:58:55 -0800 Subject: [Python-Dev] Creating dicts from dict subclasses In-Reply-To: <458030EC.8050102@livinglogic.de> References: <458030EC.8050102@livinglogic.de> Message-ID: On 12/13/06, Walter D?rwald wrote: > I tried to reimplement weakref.WeakValueDictionary as a subclass of > dict. The test passes except for one problem: To compare results > test_weakref.py converts a weakdict to a real dict via dict(weakdict). > This no longer works because PyDict_Merge() does a PyDict_Check() on the > argument and then ignores all overwritten methods. (The old version > worked because UserDict.UserDict was used). > > The simplest solution is to replace the PyDict_Check() call with > PyDict_CheckExact(), but this might slow things down too much, because > the fallback code basically does: > > for key in iter(arg.keys()): > self[key] = arg.__getitem__(key) > > Why can't we use: > > for key in iter(arg): > self[key] = arg.__getitem__(key) > > instead? The only reason I can think of is backwards compatibility: not all "mappings" created pre-2.2 would support iteration. Maybe you could check for a tp_iter slot and if non-NULL use the latter otherwise use the original fallback? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From brett at python.org Wed Dec 13 20:11:27 2006 From: brett at python.org (Brett Cannon) Date: Wed, 13 Dec 2006 11:11:27 -0800 Subject: [Python-Dev] Open CPython VM for type information In-Reply-To: <457FAC9D.1030801@gmx.net> References: <457FAC9D.1030801@gmx.net> Message-ID: On 12/12/06, Kay Schluehr wrote: > > Dear Python developers, > > while the JVM is opened to support dynamically typed languages [1] I > wonder if the CPyVM could not show more openness to statically typed > languages? Hereby I don't think that much about arbitrary languages for > the CPyVM but sublanguages like RPython which are "static enough" to be > compilable into extension modules [2]. > > However creating C extension modules from Python code is not really the > standard way I do think Python source should be handled. Instead I think > about a more direct VM support like the JVM does, that provides opcodes > like iload, fstore etc. Besides this I don't expect one canonical > approach for these type information to be feeded into the CPyVM. There > are type inferencer like rtyper for PyPy, Psycos continous bytecode > monitoring and also other approaches are possible that use type > feedback. Finally optional type declarations are also an issue; however > I would like to see language design discussions being separated ( -> > Python 3000 ) and focus on general APIs for compiler extensions / plugs > that deal with type information ( the architecture ) and VM > implementation issues. > > Thanks, Kay > > [1] http://jcp.org/en/jsr/detail?id=292 > [2] http://codespeak.net/pypy/dist/pypy/doc/extcompiler.html The Python VM is not designed to be general-purpose, period. Nor does anyone on python-dev, to my knowledge, have any interest in putting the time and energy into making it general. I am quite happy with letting the Parrot folk handle this goal. This means that the maintenance cost of adding more opcodes is not worth it. But a bigger issue with me is the performance hit. Every opcode will make the eval loop larger. And it is known that the eval loop is rather finicky in terms of size and performance. While no hard testing has been done, it has been assumed we have accidentally hit the sweet spot in terms of CPU cache size. An important thing to keep in mind when comparing Python's VM to Java's is that Java works (somewhat) for dynamic languages because people are motivated enough to shoehorn languages into it by just assuming everything is just an object and tossing out the type checking inherent to Java. But Java has not, to my knowledge, grown any great features yet to make dynamic languages work better. I know about JSR 292, but that is not in yet and adding just one opcode is not that expensive to the JVM. But compare the cost of adding a slew of typed opcodes to Python compared to the JVM adding just one is not really equal. In other words, I wouldn't count on this ever happening. =) The best you could do is try to propose some direct mechanism for adding support to the eval loop to dispatch to some function for unrecognized opcodes, but I wouldn't count on that being accepted either. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20061213/2e16ea85/attachment.htm From martin at v.loewis.de Wed Dec 13 22:00:48 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 13 Dec 2006 22:00:48 +0100 Subject: [Python-Dev] Adding resume (206) support to urllib(2) In-Reply-To: <20061213081836.GA9589@phd.pp.ru> References: <457F4BA8.5020507@v.loewis.de> <457FABF8.9050908@v.loewis.de> <20061213074912.GA3502@phd.pp.ru> <457FB45D.8080802@v.loewis.de> <20061213081836.GA9589@phd.pp.ru> Message-ID: <45806A00.7030909@v.loewis.de> Oleg Broytmann schrieb: > HTTP is one of the most widely known and used protocol. Would you better > have big httplib and small abstract urllib? so abstract it doesn't allow a > user to change protocol-specific handling? Personally, I think very elaborate support for HTTP in httplib, and very few generalizations and abstractions in urllib* would be the "right" way to do it, IMO. For example, there might be the notion of an "http session" object where a single application request can resolve to multiple http requests (with redirection, authentication negotiation, cookies, 100 continue, implicit headers, etc). For compatibility, urllib* can't drop features, and we'd need contributors who contribute such a refactoring, but IMO that would be the right way. If applications use urllib *only* for http, and *only* because it has these multi-request, implicit headers features, something is wrong with the abstractions. Regards, Martin From phd at phd.pp.ru Wed Dec 13 22:38:32 2006 From: phd at phd.pp.ru (Oleg Broytmann) Date: Thu, 14 Dec 2006 00:38:32 +0300 Subject: [Python-Dev] Adding resume (206) support to urllib(2) In-Reply-To: <45806A00.7030909@v.loewis.de> References: <457F4BA8.5020507@v.loewis.de> <457FABF8.9050908@v.loewis.de> <20061213074912.GA3502@phd.pp.ru> <457FB45D.8080802@v.loewis.de> <20061213081836.GA9589@phd.pp.ru> <45806A00.7030909@v.loewis.de> Message-ID: <20061213213832.GB24636@phd.pp.ru> On Wed, Dec 13, 2006 at 10:00:48PM +0100, "Martin v. L?wis" wrote: > Personally, I think very elaborate support for HTTP in httplib, and very > few generalizations and abstractions in urllib* would be the "right" > way to do it, IMO. For example, there might be the notion of an > "http session" object where a single application request can resolve > to multiple http requests (with redirection, authentication negotiation, > cookies, 100 continue, implicit headers, etc). I see. > For compatibility, urllib* can't drop features Leave it for py3k, then. > and we'd need > contributors who contribute such a refactoring That's the hardest part. > If applications use urllib *only* for http, and > *only* because it has these multi-request, implicit headers > features, something is wrong with the abstractions. I think I agree. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From walter at livinglogic.de Wed Dec 13 23:18:53 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 13 Dec 2006 23:18:53 +0100 Subject: [Python-Dev] Creating dicts from dict subclasses In-Reply-To: References: <458030EC.8050102@livinglogic.de> Message-ID: <45807C4D.6000207@livinglogic.de> Guido van Rossum wrote: > On 12/13/06, Walter D?rwald wrote: >> I tried to reimplement weakref.WeakValueDictionary as a subclass of >> dict. The test passes except for one problem: To compare results >> test_weakref.py converts a weakdict to a real dict via dict(weakdict). >> This no longer works because PyDict_Merge() does a PyDict_Check() on the >> argument and then ignores all overwritten methods. (The old version >> worked because UserDict.UserDict was used). >> >> The simplest solution is to replace the PyDict_Check() call with >> PyDict_CheckExact(), but this might slow things down too much, because >> the fallback code basically does: >> >> for key in iter(arg.keys()): >> self[key] = arg.__getitem__(key) >> >> Why can't we use: >> >> for key in iter(arg): >> self[key] = arg.__getitem__(key) >> >> instead? > > The only reason I can think of is backwards compatibility: not all > "mappings" created pre-2.2 would support iteration. Maybe you could > check for a tp_iter slot and if non-NULL use the latter otherwise use > the original fallback? This doesn't seem to work. It breaks test_update() in test_dict.py which does this: d = {} class SimpleUserDict: def __init__(self): self.d = {1:1, 2:2, 3:3} def keys(self): return self.d.keys() def __getitem__(self, i): return self.d[i] d.update(SimpleUserDict()) self.assertEqual(d, {1:1, 2:2, 3:3}) This fails with KeyError: 0 because SimpleUserDict doesn't implement __iter__, so it gets an iterator implementation via __getitem__. So maybe this only makes sense for Python 3.0 where we can demand that dict-like classes implement __iter__? Servus, Walter From guido at python.org Wed Dec 13 23:26:17 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 13 Dec 2006 14:26:17 -0800 Subject: [Python-Dev] Creating dicts from dict subclasses In-Reply-To: <45807C4D.6000207@livinglogic.de> References: <458030EC.8050102@livinglogic.de> <45807C4D.6000207@livinglogic.de> Message-ID: On 12/13/06, Walter D?rwald wrote: > Guido van Rossum wrote: > > On 12/13/06, Walter D?rwald wrote: > >> I tried to reimplement weakref.WeakValueDictionary as a subclass of > >> dict. The test passes except for one problem: To compare results > >> test_weakref.py converts a weakdict to a real dict via dict(weakdict). > >> This no longer works because PyDict_Merge() does a PyDict_Check() on the > >> argument and then ignores all overwritten methods. (The old version > >> worked because UserDict.UserDict was used). > >> > >> The simplest solution is to replace the PyDict_Check() call with > >> PyDict_CheckExact(), but this might slow things down too much, because > >> the fallback code basically does: > >> > >> for key in iter(arg.keys()): > >> self[key] = arg.__getitem__(key) > >> > >> Why can't we use: > >> > >> for key in iter(arg): > >> self[key] = arg.__getitem__(key) > >> > >> instead? > > > > The only reason I can think of is backwards compatibility: not all > > "mappings" created pre-2.2 would support iteration. Maybe you could > > check for a tp_iter slot and if non-NULL use the latter otherwise use > > the original fallback? > > This doesn't seem to work. It breaks test_update() in test_dict.py which > does this: > > d = {} > class SimpleUserDict: > def __init__(self): > self.d = {1:1, 2:2, 3:3} > def keys(self): > return self.d.keys() > def __getitem__(self, i): > return self.d[i] > d.update(SimpleUserDict()) > self.assertEqual(d, {1:1, 2:2, 3:3}) > > This fails with > > KeyError: 0 > > because SimpleUserDict doesn't implement __iter__, so it gets an > iterator implementation via __getitem__. > > So maybe this only makes sense for Python 3.0 where we can demand that > dict-like classes implement __iter__? Ah, right. But I think you should still use PyDict_CheckExact, and slow fallbacks be damned. (I guess you could look for iterkeys first.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From kay.schluehr at gmx.net Thu Dec 14 08:38:21 2006 From: kay.schluehr at gmx.net (Kay Schluehr) Date: Thu, 14 Dec 2006 08:38:21 +0100 Subject: [Python-Dev] Open CPython VM for type information In-Reply-To: References: <457FAC9D.1030801@gmx.net> Message-ID: <4580FF6D.2090004@gmx.net> Brett Cannon schrieb: > > > On 12/12/06, *Kay Schluehr* > wrote: > > Dear Python developers, > > while the JVM is opened to support dynamically typed languages [1] I > wonder if the CPyVM could not show more openness to statically typed > languages? Hereby I don't think that much about arbitrary > languages for > the CPyVM but sublanguages like RPython which are "static enough" > to be > compilable into extension modules [2]. > > However creating C extension modules from Python code is not > really the > standard way I do think Python source should be handled. Instead I > think > about a more direct VM support like the JVM does, that provides > opcodes > like iload, fstore etc. Besides this I don't expect one canonical > approach for these type information to be feeded into the CPyVM. There > are type inferencer like rtyper for PyPy, Psycos continous bytecode > monitoring and also other approaches are possible that use type > feedback. Finally optional type declarations are also an issue; > however > I would like to see language design discussions being separated ( -> > Python 3000 ) and focus on general APIs for compiler extensions / > plugs > that deal with type information ( the architecture ) and VM > implementation issues. > > Thanks, Kay > > [1] http://jcp.org/en/jsr/detail?id=292 > [2] http://codespeak.net/pypy/dist/pypy/doc/extcompiler.html > > > The Python VM is not designed to be general-purpose, period. Nor does > anyone on python-dev, to my knowledge, have any interest in putting > the time and energy into making it general. I am quite happy with > letting the Parrot folk handle this goal. This means that the > maintenance cost of adding more opcodes is not worth it. > > But a bigger issue with me is the performance hit. Every opcode will > make the eval loop larger. And it is known that the eval loop is > rather finicky in terms of size and performance. While no hard > testing has been done, it has been assumed we have accidentally hit > the sweet spot in terms of CPU cache size. > > An important thing to keep in mind when comparing Python's VM to > Java's is that Java works (somewhat) for dynamic languages because > people are motivated enough to shoehorn languages into it by just > assuming everything is just an object and tossing out the type > checking inherent to Java. But Java has not, to my knowledge, grown > any great features yet to make dynamic languages work better. I know > about JSR 292, but that is not in yet and adding just one opcode is > not that expensive to the JVM. But compare the cost of adding a slew > of typed opcodes to Python compared to the JVM adding just one is not > really equal. > > In other words, I wouldn't count on this ever happening. =) The best > you could do is try to propose some direct mechanism for adding > support to the eval loop to dispatch to some function for unrecognized > opcodes, but I wouldn't count on that being accepted either. > > -Brett Brett, thanks for the short answer. Although it is negative regarding my attempt, it is also important to know which ideas are outlawed and where one steps into a no-go area concerning the evolving language standard and its main implementation. Regards From arigo at tunes.org Thu Dec 14 08:41:34 2006 From: arigo at tunes.org (Armin Rigo) Date: Thu, 14 Dec 2006 08:41:34 +0100 Subject: [Python-Dev] Creating dicts from dict subclasses In-Reply-To: <458030EC.8050102@livinglogic.de> References: <458030EC.8050102@livinglogic.de> Message-ID: <20061214074134.GA15823@code0.codespeak.net> Hi Walter, On Wed, Dec 13, 2006 at 05:57:16PM +0100, Walter D?rwald wrote: > I tried to reimplement weakref.WeakValueDictionary as a subclass of > dict. The test passes except for one problem: To compare results > test_weakref.py converts a weakdict to a real dict via dict(weakdict). > This no longer works because PyDict_Merge() does a PyDict_Check() on the > argument and then ignores all overwritten methods. (The old version > worked because UserDict.UserDict was used). This is an instance of a general problem in Python: if you subclass a built-in type, then your overridden methods may or may not be used in various situations. In this case you might have subtle problems with built-in functions and statements that expect a dict and manipulate it directly, because they will see the underlying dict structure. It is also quite fragile: e.g. if a future version of CPython adds a new method to dicts, then your existing code will also grow the new method automatically - but as inherited from 'dict', which produces quite surprizing results for the user. > for key in iter(arg.keys()): > self[key] = arg.__getitem__(key) > > Why can't we use: > > for key in iter(arg): > self[key] = arg.__getitem__(key) The latter would allow 'arg' to be a sequence instead of a mapping. It may even not crash but produce nonsense instead, e.g. if 'arg' is a list of small integers. Moreover there are multiple places in the code base that assume that mappings are "something with a 'keys' and a '__getitem__'", so I suppose any change in that should be done carefully. A bientot, Armin From walter at livinglogic.de Thu Dec 14 14:11:38 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Thu, 14 Dec 2006 14:11:38 +0100 Subject: [Python-Dev] Creating dicts from dict subclasses In-Reply-To: References: <458030EC.8050102@livinglogic.de> <45807C4D.6000207@livinglogic.de> Message-ID: <45814D8A.1080907@livinglogic.de> Guido van Rossum wrote: > On 12/13/06, Walter D?rwald wrote: >> Guido van Rossum wrote: >> > On 12/13/06, Walter D?rwald wrote: >> >> I tried to reimplement weakref.WeakValueDictionary as a subclass of >> >> dict. The test passes except for one problem: To compare results >> >> test_weakref.py converts a weakdict to a real dict via dict(weakdict). >> >> This no longer works because PyDict_Merge() does a PyDict_Check() >> on the >> >> argument and then ignores all overwritten methods. (The old version >> >> worked because UserDict.UserDict was used). >> >> >> >> The simplest solution is to replace the PyDict_Check() call with >> >> PyDict_CheckExact(), but this might slow things down too much, because >> >> the fallback code basically does: >> >> >> >> for key in iter(arg.keys()): >> >> self[key] = arg.__getitem__(key) >> >> >> >> Why can't we use: >> >> >> >> for key in iter(arg): >> >> self[key] = arg.__getitem__(key) >> >> >> >> instead? >> > >> > The only reason I can think of is backwards compatibility: not all >> > "mappings" created pre-2.2 would support iteration. Maybe you could >> > check for a tp_iter slot and if non-NULL use the latter otherwise use >> > the original fallback? >> >> This doesn't seem to work. It breaks test_update() in test_dict.py which >> does this: >> >> d = {} >> class SimpleUserDict: >> def __init__(self): >> self.d = {1:1, 2:2, 3:3} >> def keys(self): >> return self.d.keys() >> def __getitem__(self, i): >> return self.d[i] >> d.update(SimpleUserDict()) >> self.assertEqual(d, {1:1, 2:2, 3:3}) >> >> This fails with >> >> KeyError: 0 >> >> because SimpleUserDict doesn't implement __iter__, so it gets an >> iterator implementation via __getitem__. >> >> So maybe this only makes sense for Python 3.0 where we can demand that >> dict-like classes implement __iter__? > > Ah, right. But I think you should still use PyDict_CheckExact, and > slow fallbacks be damned. (I guess you could look for iterkeys first.) OK, here's a patch that tries iterkeys() before keys(): http://bugs.python.org/1615701 Both versions seem to be faster than Python 2.5: class dictik: def __init__(self, n): self.d = dict((i, i) for i in xrange(n)) def iterkeys(self): return iter(self.d) def __getitem__(self, i): return self.d[i] class dictk: def __init__(self, n): self.d = dict((i, i) for i in xrange(n)) def keys(self): return self.d.keys() def __getitem__(self, i): return self.d[i] $ python2.5 -mtimeit -s'from foo import dictik, dictk; d=dictk(100000)' 'dict(d)' 10 loops, best of 3: 179 msec per loop $ ./python -mtimeit -s'from foo import dictik, dictk; d=dictk(100000)' 'dict(d)' 10 loops, best of 3: 138 msec per loop $ ./python -mtimeit -s'from foo import dictik, dictk; d=dictik(100000)' 'dict(d)' 10 loops, best of 3: 123 msec per loop Servus, Walter From walter at livinglogic.de Thu Dec 14 14:35:10 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Thu, 14 Dec 2006 14:35:10 +0100 Subject: [Python-Dev] Creating dicts from dict subclasses In-Reply-To: <20061214074134.GA15823@code0.codespeak.net> References: <458030EC.8050102@livinglogic.de> <20061214074134.GA15823@code0.codespeak.net> Message-ID: <4581530E.6050304@livinglogic.de> Armin Rigo wrote: > Hi Walter, > > On Wed, Dec 13, 2006 at 05:57:16PM +0100, Walter D?rwald wrote: >> I tried to reimplement weakref.WeakValueDictionary as a subclass of >> dict. The test passes except for one problem: To compare results >> test_weakref.py converts a weakdict to a real dict via dict(weakdict). >> This no longer works because PyDict_Merge() does a PyDict_Check() on the >> argument and then ignores all overwritten methods. (The old version >> worked because UserDict.UserDict was used). > > This is an instance of a general problem in Python: if you subclass a > built-in type, then your overridden methods may or may not be used in > various situations. In this case you might have subtle problems with > built-in functions and statements that expect a dict and manipulate it > directly, because they will see the underlying dict structure. It is > also quite fragile: e.g. if a future version of CPython adds a new > method to dicts, then your existing code will also grow the new method > automatically - but as inherited from 'dict', which produces quite > surprizing results for the user. > >> for key in iter(arg.keys()): >> self[key] = arg.__getitem__(key) >> >> Why can't we use: >> >> for key in iter(arg): >> self[key] = arg.__getitem__(key) > > The latter would allow 'arg' to be a sequence instead of a mapping. It > may even not crash but produce nonsense instead, e.g. if 'arg' is a list > of small integers. Of course I meant: use the alternate code inside PyDict_Merge() where dict_update_common() already has decided that the argument is a mapping (which is done via PyObject_HasAttrString(arg, "keys")). > Moreover there are multiple places in the code base > that assume that mappings are "something with a 'keys' and a > '__getitem__'", so I suppose any change in that should be done > carefully. Doing a grep PyMapping_Keys `find -name '*.[ch]'` reveals the following: ./Python/ceval.c: all = PyMapping_Keys(dict); This is used for "import *" and simply iterates over the keys, so it could use iterkeys()/iter() ./Objects/object.c: result = PyMapping_Keys(locals); This is in PyObject_Dir(). It does return the keylist, so no iterkeys()/iter() here. ./Objects/descrobject.c: return PyMapping_Keys(pp->dict); This too must return a list of keys. ./Objects/dictobject.c: PyObject *keys = PyMapping_Keys(b); This is the dict constructor. ./PC/_subprocess.c: keys = PyMapping_Keys(environment); This iterates over keys() and values() to format the complete environment, so it probably could be switched to iterkeys()/iter(). ./Modules/_sre.c: keys = PyMapping_Keys(self->pattern->groupindex); This again does iteration, so could be switched. ./Modules/posixmodule.c: keys = PyMapping_Keys(env); ./Modules/posixmodule.c: keys = PyMapping_Keys(env); ./Modules/posixmodule.c: keys = PyMapping_Keys(env); Those three are for execve/spawnve/spawnvpe and do basically the same as PC/_subprocess.c, so could be switched too. Servus, Walter From kbk at shore.net Fri Dec 15 07:42:26 2006 From: kbk at shore.net (Kurt B. Kaiser) Date: Fri, 15 Dec 2006 01:42:26 -0500 (EST) Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200612150642.kBF6gQ59028713@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 414 open ( +1) / 3498 closed ( +9) / 3912 total (+10) Bugs : 949 open ( +6) / 6376 closed (+12) / 7325 total (+18) RFE : 247 open ( +1) / 245 closed ( +1) / 492 total ( +2) New / Reopened Patches ______________________ C99 _Bool support for struct (2006-12-07) http://python.org/sf/1610575 opened by David Remahl BSD version of ctypes.util.find_library (2006-12-07) http://python.org/sf/1610795 opened by Martin Kammerhofer whitespace in `svnversion .` (2006-10-15) CLOSED http://python.org/sf/1577756 reopened by schmaller Enhanced tabbed pane widget (2006-12-10) http://python.org/sf/1612746 opened by Tal Einat encoding directive -- more examples (2006-12-11) http://python.org/sf/1613352 opened by Jim Jewett Write mode option for fileinput module. (2006-12-11) http://python.org/sf/1613500 opened by Anthony Roy POSIX capabilities support (2006-12-13) http://python.org/sf/1615158 opened by Matt Kern Creating dicts for dict subclasses (2006-12-14) http://python.org/sf/1615701 opened by Walter D?rwald BZ2File.seek() fails for large files (2006-12-14) http://python.org/sf/1615868 opened by Lars Gust?bel Cached globals+builtins lookup optimization (2006-12-15) http://python.org/sf/1616125 opened by Andrea Griffini Patches Closed ______________ whitespace in `svnversion .` (2006-10-15) http://python.org/sf/1577756 closed by gbrandl traceback on exit if syslog handler fails to initialize (2006-12-05) http://python.org/sf/1609407 closed by vsajip Race condition in os.makedirs (2006-12-04) http://python.org/sf/1608579 closed by gbrandl Prevent race condition in os.makedirs (2005-07-17) http://python.org/sf/1239890 closed by gbrandl os.makedirs - robust against partial path (2005-10-05) http://python.org/sf/1314067 closed by gbrandl modsupport does not use va_copy when available (2003-12-11) http://python.org/sf/858318 closed by sf-robot acknowledge signals in non-main threads (2004-12-21) http://python.org/sf/1088832 closed by loewis Patch for bug 999042. (2004-12-23) http://python.org/sf/1090482 closed by loewis socket leak in SocketServer (2004-12-30) http://python.org/sf/1093468 closed by loewis mailbox.py: check that os.fsync is available before using it (2006-11-19) http://python.org/sf/1599256 closed by akuchling New / Reopened Bugs ___________________ GUI for Python 2.3, 2.4, and 2.5 is very sluggish (2006-12-06) CLOSED http://python.org/sf/1610485 reopened by g4rlik cgi.py multipart/form-data (2006-12-07) http://python.org/sf/1610654 opened by Chui Tey \b in unicode regex gives strange results (2006-12-07) CLOSED http://python.org/sf/1611131 opened by akaihola os.path.exists("file/") failure on Solaris 9 (2006-12-07) http://python.org/sf/1611154 opened by Paul Eggert can't pickle NAN's in binary mode (2006-12-08) CLOSED http://python.org/sf/1611753 opened by Wayne Christopher sndhdr.what() does not recognize wav file (2006-12-09) http://python.org/sf/1611944 reopened by klankschap sndhdr.what() does not recognize wav file (2006-12-09) http://python.org/sf/1611944 opened by Floris van Manen builtin compile() doc needs PyCF_DONT_IMPLY_DEDENT (2006-12-09) http://python.org/sf/1612012 opened by Anthony Baxter Dictionary ordering docs are too unclear of dangers (2006-12-09) http://python.org/sf/1612113 opened by Calvin Spealman webchecker/urllib chokes on 404 pages (2006-12-10) CLOSED http://python.org/sf/1612729 opened by Fredrik Lundh lambda tuple parameter bus error (2006-12-11) CLOSED http://python.org/sf/1613059 opened by Bruce Cropley str.split creates new string even if pattern not found (2006-12-11) http://python.org/sf/1613130 opened by Antoine Pitrou pydoc info for a package doesn't list all package contents (2006-12-11) http://python.org/sf/1613479 opened by Nishkar Grover xmlrpclib ServerProxy uses old httplib interface (2006-12-12) http://python.org/sf/1613573 opened by Matt Brown recv_into not documented (2006-12-11) http://python.org/sf/1613651 opened by Eric Huss AttributesImpl does not implement __contains__ on Linux (2006-12-13) http://python.org/sf/1614387 opened by Jason Briggs dict throwing inaccurate KeyError on small tuple keys (2006-12-13) CLOSED http://python.org/sf/1614429 opened by toidinamai python-logging compatability with Zope. (2006-12-13) http://python.org/sf/1614460 opened by Simon Hookway tempile.TemporaryFile differences between linux and windows (2006-12-13) http://python.org/sf/1615275 opened by hirzel subprocess doesn't handle SIGPIPE (2006-12-14) http://python.org/sf/1615376 opened by Mark Diekhans IA64/AMD64/x64 confusion (2006-12-14) http://python.org/sf/1616109 opened by Sidnei da Silva Bugs Closed ___________ GUI for Python 2.3, 2.4, and 2.5 is very sluggish (2006-12-06) http://python.org/sf/1610485 closed by kbk \b in unicode regex gives strange results (2006-12-07) http://python.org/sf/1611131 deleted by akaihola can't pickle NAN's in binary mode (2006-12-08) http://python.org/sf/1611753 closed by mwh race in os.makedirs() (2005-06-18) http://python.org/sf/1223238 closed by gbrandl simple moves freeze IDLE (2006-10-05) http://python.org/sf/1571112 closed by kbk webchecker/urllib chokes on 404 pages (2006-12-10) http://python.org/sf/1612729 closed by gbrandl lambda tuple parameter bus error (2006-12-11) http://python.org/sf/1613059 closed by gbrandl logger.config problems with comma separated lists (2006-06-09) http://python.org/sf/1503765 closed by vsajip mailbox: Maildir.get_folder does not inherit factory (2006-11-20) http://python.org/sf/1600152 closed by akuchling dict throwing inaccurate KeyError on small tuple keys (2006-12-12) http://python.org/sf/1614429 closed by rhettinger New / Reopened RFE __________________ Py_DEBUG (2006-12-09) http://python.org/sf/1612190 opened by nitro Class Browser doesn't show internal classes (2006-12-09) http://python.org/sf/1612262 opened by Tal Einat RFE Closed __________ "".translate() docs should mention string.maketrans() (2006-11-08) http://python.org/sf/1592899 closed by gbrandl From ncoghlan at gmail.com Sun Dec 17 01:54:57 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Dec 2006 10:54:57 +1000 Subject: [Python-Dev] Open CPython VM for type information In-Reply-To: <4580FF6D.2090004@gmx.net> References: <457FAC9D.1030801@gmx.net> <4580FF6D.2090004@gmx.net> Message-ID: <45849561.3080508@gmail.com> Kay Schluehr wrote: > Brett Cannon schrieb: >> In other words, I wouldn't count on this ever happening. =) The best >> you could do is try to propose some direct mechanism for adding >> support to the eval loop to dispatch to some function for unrecognized >> opcodes, but I wouldn't count on that being accepted either. >> >> -Brett > > Brett, thanks for the short answer. Although it is negative regarding my > attempt, it is also important to know which ideas are outlawed and where > one steps into a no-go area concerning the evolving language standard > and its main implementation. To add one thing to Brett's response: making it easier to experiment with the VM (e.g. to explore static typing options) was one of the motivations behind the PyPy project [1]. Although given the mention of RPython in your original message I suspect you already know about that ;) Cheers, Nick. [1] http://codespeak.net/pypy/dist/pypy/doc/architecture.html#mission-statement -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From agriff at tin.it Tue Dec 19 09:03:39 2006 From: agriff at tin.it (Andrea Griffini) Date: Tue, 19 Dec 2006 09:03:39 +0100 Subject: [Python-Dev] About dictionary lookup caching Message-ID: <45879CDB.2070705@tin.it> I'm experimenting with a patch for dictionary lookup caching, the main idea being avoiding the lookup of a constant (co_names) in a dictionary if the dictionary didn't change since last lookup. Currently the cache is held in a structure that is parallel to co_names (the LOAD_GLOBAL opcode uses oparg to access the cache slot) and this is wasteful if there are co_names entries that are not used for global lookups. Probably I could act at the code object creation time to sort names so that ones used by LOAD_GLOBAL are at the beginning of co_names but this would require inspecting and hacking the already generated opcode. Another case that would IMO be worth optimizing is the LOAD_ATTR lookup when it's done after a LOAD_GLOBAL (I suspect that in many cases the element being searched will be a module so caching would be simple to implement; for general objects things are way more complex and I see no simple solution for caching dictionary lookups). My opinion is that it would be by far better to do this ordering of co_names at compile time but I've no idea where to look for trying to make such a change. Can someone please point me in the right direction ? Andrea From brett at python.org Tue Dec 19 19:53:35 2006 From: brett at python.org (Brett Cannon) Date: Tue, 19 Dec 2006 10:53:35 -0800 Subject: [Python-Dev] About dictionary lookup caching In-Reply-To: <45879CDB.2070705@tin.it> References: <45879CDB.2070705@tin.it> Message-ID: On 12/19/06, Andrea Griffini wrote: > > I'm experimenting with a patch for dictionary lookup caching, the > main idea being avoiding the lookup of a constant (co_names) in > a dictionary if the dictionary didn't change since last lookup. > > Currently the cache is held in a structure that is parallel to > co_names (the LOAD_GLOBAL opcode uses oparg to access > the cache slot) and this is wasteful if there are co_names entries > that are not used for global lookups. > > Probably I could act at the code object creation time to > sort names so that ones used by LOAD_GLOBAL are > at the beginning of co_names but this would require inspecting > and hacking the already generated opcode. > Another case that would IMO be worth optimizing is the > LOAD_ATTR lookup when it's done after a LOAD_GLOBAL > (I suspect that in many cases the element being searched will > be a module so caching would be simple to implement; for > general objects things are way more complex and I see no > simple solution for caching dictionary lookups). > > My opinion is that it would be by far better to do this ordering > of co_names at compile time but I've no idea where to look > for trying to make such a change. > > Can someone please point me in the right direction ? For guidance on how the compiler works, see PEP 339 ( http://www.python.org/dev/peps/pep-0339). -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20061219/54c0ceb2/attachment.htm From martin at v.loewis.de Tue Dec 19 22:15:42 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 19 Dec 2006 22:15:42 +0100 Subject: [Python-Dev] About dictionary lookup caching In-Reply-To: <45879CDB.2070705@tin.it> References: <45879CDB.2070705@tin.it> Message-ID: <4588567E.8050105@v.loewis.de> Andrea Griffini schrieb: > My opinion is that it would be by far better to do this ordering > of co_names at compile time but I've no idea where to look > for trying to make such a change. > > Can someone please point me in the right direction ? It's all in Python/compiler.c. The names list is created in dict_keys_inorder, where the dictionary keys are supposed to be tuples (name, str), and the values are the variable number. The dictionaries are filled through compiler_add_o, which looks up whether there is already a name in the table (and if so returns its number), or else adds a new entry, using len(dict) as the new number. As you can see, this inherently adds the names in the order in which they are encountered. I can see a number of choices: 1. make a separate list for names loaded through LOAD_GLOBAL. This would be an API change (and affect reflection and whatnot), but it would likely be the algorithmically easiest way 2. make a pass through the AST before compilation starts, to already add all globals to the names dictionary (while you are at it, you might as well fill all symbol tables in that pass). You can then renumber them as you please, and only invoke byte code generation afterwards. 3. try fitting this into the peephole optimizer. You'd make a peephole pass which reindexes co_names, and then run through the byte code to update all opargs. Be prepared to have this pass fail, in case you'd need an extended arg, but don't have space for it. Regards, Martin From tomerfiliba at gmail.com Tue Dec 19 22:59:37 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Tue, 19 Dec 2006 23:59:37 +0200 Subject: [Python-Dev] classes and cell variables question Message-ID: <1d85506f0612191359i7831b097s9a1dcbe1cdaa1e49@mail.gmail.com> to my understanding of the object model, the code of snippet 1 and snippet 2 should be equivalent. a class is just a "special function" that returns its locals automatically and passes them to the metaclass constructor: --- snippet 1 --- class foo(object): x = 5 def f(self): print "f", x --- snippet 2 --- def bar(): x = 5 def g(self): print "g", x return locals() barcls = type("barcls", (object,), bar()) but there's one big difference. classes don't create cell variables to hold bound external variables. the "bar" version works, because "x" is a bound cell variable, but the "foo" version fails, as it attempts to access "x" as a global. .>>> barcls().g() g 5 .>>> foo().f() f Traceback (most recent call last): File "", line 1, in File "", line 4, in f NameError: global name 'x' is not defined for reference, i attached the code of all four functions below. my question is, how come classes don't create cell variables, like normal functions? was this done on purpose? does it have to do with inheritance? if so, what's wrong with my "bar" version? [1] # code of class foo ############################################################ # 2 0 LOAD_NAME 0 (__name__) # 3 STORE_NAME 1 (__module__) # 3 6 LOAD_CONST 0 (5) # 9 STORE_NAME 2 (x) # # 4 12 LOAD_CONST 1 (", line 4>) # 15 MAKE_FUNCTION 0 # 18 STORE_NAME 3 (f) # 21 LOAD_LOCALS # 22 RETURN_VALUE [2] # code of foo.f: ############################################################ # 5 0 LOAD_CONST 1 ('f') # 3 PRINT_ITEM # 4 LOAD_GLOBAL 0 (x) # 7 PRINT_ITEM # 8 PRINT_NEWLINE # 9 LOAD_CONST 0 (None) # 12 RETURN_VALUE [3] # code of bar: ############################################################ # 2 0 LOAD_CONST 1 (5) # 3 STORE_DEREF 0 (x) # # 3 6 LOAD_CLOSURE 0 (x) # 9 BUILD_TUPLE 1 # 12 LOAD_CONST 2 (", line 3>) # 15 MAKE_CLOSURE 0 # 18 STORE_FAST 0 (g) # # 5 21 LOAD_GLOBAL 0 (locals) # 24 CALL_FUNCTION 0 # 27 RETURN_VALUE [4] # code of bar.g: ############################################################ # 4 0 LOAD_CONST 1 ('g') # 3 PRINT_ITEM # 4 LOAD_DEREF 0 (x) # 7 PRINT_ITEM # 8 PRINT_NEWLINE # 9 LOAD_CONST 0 (None) # 12 RETURN_VALUE From python at rcn.com Tue Dec 19 23:09:04 2006 From: python at rcn.com (Raymond Hettinger) Date: Tue, 19 Dec 2006 17:09:04 -0500 Subject: [Python-Dev] About dictionary lookup caching References: <45879CDB.2070705@tin.it> Message-ID: <00fc01c723ba$4bf057c0$ac02a8c0@RaymondLaptop1> [Andrea Griffini] > I'm experimenting with a patch for dictionary lookup caching, the > main idea being avoiding the lookup of a constant (co_names) in > a dictionary if the dictionary didn't change since last lookup. . . . > My opinion is that it would be by far better to do this ordering > of co_names at compile time It would be nice if you could avoid introducing a new invariant that would be have to be maintained by other code generation tools or bytecode hacks. Raymond From martin at v.loewis.de Tue Dec 19 23:19:22 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 19 Dec 2006 23:19:22 +0100 Subject: [Python-Dev] About dictionary lookup caching In-Reply-To: <009f01c723b8$91a4f4d0$ac02a8c0@RaymondLaptop1> References: <45879CDB.2070705@tin.it> <4588567E.8050105@v.loewis.de> <009f01c723b8$91a4f4d0$ac02a8c0@RaymondLaptop1> Message-ID: <4588656A.20209@v.loewis.de> Raymond Hettinger schrieb: > If you can find some other design that doesn't depend on the ordering of > co_names, that would be nice; otherwise, we're adding a permanent > complication to the compiler and establishing a new invariant that would > have to be maintained by everyone hoping to generate or hack bytecodes. It wouldn't be a strict invariant, but instead, breaking it would mean that the memory consumption goes up somewhat. Regards, Martin From guido at python.org Tue Dec 19 23:24:52 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 19 Dec 2006 14:24:52 -0800 Subject: [Python-Dev] classes and cell variables question In-Reply-To: <1d85506f0612191359i7831b097s9a1dcbe1cdaa1e49@mail.gmail.com> References: <1d85506f0612191359i7831b097s9a1dcbe1cdaa1e49@mail.gmail.com> Message-ID: On 12/19/06, tomer filiba wrote: > to my understanding of the object model, the code of snippet 1 > and snippet 2 should be equivalent. a class is just a "special function" > that returns its locals automatically and passes them to the metaclass > constructor: > > --- snippet 1 --- > class foo(object): > x = 5 > def f(self): > print "f", x > > --- snippet 2 --- > def bar(): > x = 5 > def g(self): > print "g", x > return locals() > barcls = type("barcls", (object,), bar()) > > but there's one big difference. classes don't create cell variables > to hold bound external variables. the "bar" version works, because > "x" is a bound cell variable, but the "foo" version fails, as it attempts > to access "x" as a global. > > .>>> barcls().g() > g 5 > > .>>> foo().f() > f > Traceback (most recent call last): > File "", line 1, in > File "", line 4, in f > NameError: global name 'x' is not defined > > for reference, i attached the code of all four functions below. > > my question is, how come classes don't create cell variables, like > normal functions? was this done on purpose? does it have to > do with inheritance? if so, what's wrong with my "bar" version? I did this very much on purpose, although I have to think a bit to remember the reason. I think I didn't want "accidental Java code" where one would define a class variable and reference it from a method without prefixing it with self or the class name and it would accidentally work, but it would be very unpythonic. So I rigged it so that it wouldn't work. > [1] > # code of class foo > ############################################################ > # 2 0 LOAD_NAME 0 (__name__) > # 3 STORE_NAME 1 (__module__) > # 3 6 LOAD_CONST 0 (5) > # 9 STORE_NAME 2 (x) > # > # 4 12 LOAD_CONST 1 ( 009E5608, file "", line 4>) > # 15 MAKE_FUNCTION 0 > # 18 STORE_NAME 3 (f) > # 21 LOAD_LOCALS > # 22 RETURN_VALUE > > [2] > # code of foo.f: > ############################################################ > # 5 0 LOAD_CONST 1 ('f') > # 3 PRINT_ITEM > # 4 LOAD_GLOBAL 0 (x) > # 7 PRINT_ITEM > # 8 PRINT_NEWLINE > # 9 LOAD_CONST 0 (None) > # 12 RETURN_VALUE > > [3] > # code of bar: > ############################################################ > # 2 0 LOAD_CONST 1 (5) > # 3 STORE_DEREF 0 (x) > # > # 3 6 LOAD_CLOSURE 0 (x) > # 9 BUILD_TUPLE 1 > # 12 LOAD_CONST 2 ( 009F6698, file "", line 3>) > # 15 MAKE_CLOSURE 0 > # 18 STORE_FAST 0 (g) > # > # 5 21 LOAD_GLOBAL 0 (locals) > # 24 CALL_FUNCTION 0 > # 27 RETURN_VALUE > > [4] > # code of bar.g: > ############################################################ > # 4 0 LOAD_CONST 1 ('g') > # 3 PRINT_ITEM > # 4 LOAD_DEREF 0 (x) > # 7 PRINT_ITEM > # 8 PRINT_NEWLINE > # 9 LOAD_CONST 0 (None) > # 12 RETURN_VALUE > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Tue Dec 19 23:29:30 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 19 Dec 2006 23:29:30 +0100 Subject: [Python-Dev] classes and cell variables question In-Reply-To: <1d85506f0612191359i7831b097s9a1dcbe1cdaa1e49@mail.gmail.com> References: <1d85506f0612191359i7831b097s9a1dcbe1cdaa1e49@mail.gmail.com> Message-ID: <458867CA.7060809@v.loewis.de> tomer filiba schrieb: > my question is, how come classes don't create cell variables, like > normal functions? Not sure what you mean by "how come"? Why is the implementation reacting as it is? Because the body of class is compiled as a global code fragment, not as a nested one. Or why is the implementation the way it is? For several reasons, one being that classes predate nested functions. > was this done on purpose? Yes. Attributes defined inside a class are assumed to be accessed through attribute access *only*. So you write self.foo() to invoke a method, instead of invoking foo() Python treats methods and data attributes uniformly, so the same applies to data variables. > does it have to > do with inheritance? if so, what's wrong with my "bar" version? It also has to do with inheritance. If you do self.foo, it looks 1. in the object itself 2. in the class of the object 3. in the bases of the class of the object (recursively) It would be counter-intuitive if some things (i.e. things defined in the class itself) could be accessed directly, whereas other things (ie. attributes of the object itself, and things in the base classes) would need to be accessed through qualification. It would also be counter-intuitive if you find methods in an unqualified manner, but then can't call them because you didn't give a self argument. If you don't follow this reasoning, please write a counter-proposal so that people have something to shoot down. Regards, Martin From tomerfiliba at gmail.com Tue Dec 19 23:34:58 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Wed, 20 Dec 2006 00:34:58 +0200 Subject: [Python-Dev] classes and cell variables question In-Reply-To: <458867CA.7060809@v.loewis.de> References: <1d85506f0612191359i7831b097s9a1dcbe1cdaa1e49@mail.gmail.com> <458867CA.7060809@v.loewis.de> Message-ID: <1d85506f0612191434p686165fi987515f57fe9ec71@mail.gmail.com> > If you don't follow this reasoning, please write a counter-proposal > so that people have something to shoot down. ? i just wanted to be sure it was done on purpose, and what were the reasons for that. -tomer On 12/20/06, "Martin v. L?wis" wrote: > tomer filiba schrieb: > > my question is, how come classes don't create cell variables, like > > normal functions? > > Not sure what you mean by "how come"? Why is the implementation > reacting as it is? Because the body of class is compiled as > a global code fragment, not as a nested one. > Or why is the implementation the way it is? For several reasons, > one being that classes predate nested functions. > > > was this done on purpose? > > Yes. Attributes defined inside a class are assumed to be accessed > through attribute access *only*. So you write > > self.foo() > > to invoke a method, instead of invoking > > foo() > > Python treats methods and data attributes uniformly, so the > same applies to data variables. > > > does it have to > > do with inheritance? if so, what's wrong with my "bar" version? > > It also has to do with inheritance. If you do self.foo, it looks > 1. in the object itself > 2. in the class of the object > 3. in the bases of the class of the object (recursively) > > It would be counter-intuitive if some things (i.e. things defined > in the class itself) could be accessed directly, whereas other things > (ie. attributes of the object itself, and things in the base classes) > would need to be accessed through qualification. It would also be > counter-intuitive if you find methods in an unqualified manner, > but then can't call them because you didn't give a self argument. > > If you don't follow this reasoning, please write a counter-proposal > so that people have something to shoot down. > > Regards, > Martin > From lars at gustaebel.de Wed Dec 20 16:41:15 2006 From: lars at gustaebel.de (Lars =?iso-8859-15?Q?Gust=E4bel?=) Date: Wed, 20 Dec 2006 16:41:15 +0100 Subject: [Python-Dev] Request for developer privileges. Message-ID: <20061220154115.GA32287@axis.g33x.de> Hello, my name is Lars Gust?bel (SF gustaebel). I contributed tarfile.py to the Python standard library in January 2003 and have been the maintainer since then. I have provided about 25 patches over the years, most of them fixes, some of them new features and improvements. As a result, I am pretty familiar with the Python development process. If possible I would like to get developer privileges to be able to work more actively on tarfile.py for a certain time. I am currently implementing read-write POSIX.1-2001 pax format support. Development is still in progress, but it is already clear at this point, that it will be a complex change, which will definitely require some maintenance once it is finished and in day-to-day use. I would like to clean up the tarfile test suite during this process as well. The introduction of the pax format is important because it is the first tar specification that puts an end to those annoying limitations of the "original" tar format. It will become the default format for GNU tar some day. Thank you, Lars. -- Lars Gust?bel lars at gustaebel.de From brett at python.org Wed Dec 20 19:23:16 2006 From: brett at python.org (Brett Cannon) Date: Wed, 20 Dec 2006 10:23:16 -0800 Subject: [Python-Dev] New Python-Ideas mailing list Message-ID: At Guido's suggestion, a new mailing list has been created named Python-Ideas (http://mail.python.org/mailman/listinfo/python-ideas). This list is meant as a place for speculative, pie-in-the-sky language design ideas to be discussed and honed to the point of practically being a PEP before being presented to python-dev or python-3000. This allows both python-dev and python-3000 to focus more on implementation work or final approval/denial of ideas instead of being flooded with long threads where people discuss ideas that are too nebulous to be considered for inclusion into Python. Like python-dev and python-3000, Python-Ideas requires you subscribe before you can post, but there is no moderator approval required. If you are interested in helping me out by being an administrator or moderator for the list, please let me know. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20061220/c8919f70/attachment.htm From brett at python.org Wed Dec 20 20:11:11 2006 From: brett at python.org (Brett Cannon) Date: Wed, 20 Dec 2006 11:11:11 -0800 Subject: [Python-Dev] Listing of PEP 328 status and possible minor oversight Message-ID: I just noticed that PEP 328 (relative imports) is listed as an accepted PEP, but not completed. Is this because there is still things to do for 2.6 and 2.7? Or did someone just forget to move it to the completed PEPs section of the PEP index? If it is the former then PEP 362 will need to be moved. I also noticed that there is no mention of removing import redirection to a top-level module by having None in sys.modules. Was this for backwards-compatibility issues, or was it just not thought of? -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20061220/d6c7ff30/attachment.htm From thomas at python.org Wed Dec 20 21:17:32 2006 From: thomas at python.org (Thomas Wouters) Date: Wed, 20 Dec 2006 12:17:32 -0800 Subject: [Python-Dev] Listing of PEP 328 status and possible minor oversight In-Reply-To: References: Message-ID: <9e804ac0612201217t78325fedn9e798ff014e5d4f@mail.gmail.com> On 12/20/06, Brett Cannon wrote: > > I just noticed that PEP 328 (relative imports) is listed as an accepted > PEP, but not completed. Is this because there is still things to do for > 2.6 and 2.7? Or did someone just forget to move it to the completed PEPs > section of the PEP index? If it is the former then PEP 362 will need to be > moved. There was some minor finagling about the wording of the PEP, I think, that I was still supposed to do. Some explanation of common practices, maybe; the python-dev archives might have it. I could never get it into the PEP the right way, the end result was never that big a change (and a bit awkward with the rest.) I also noticed that there is no mention of removing import redirection to a > top-level module by having None in sys.modules. Was this for > backwards-compatibility issues, or was it just not thought of? I don't think I've ever heard of that idea. I don't know what 'import redirection' is either. But maybe it was one of those things that were still supposed to go into PEP 328. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20061220/e6560b8c/attachment.html From brett at python.org Wed Dec 20 22:45:42 2006 From: brett at python.org (Brett Cannon) Date: Wed, 20 Dec 2006 13:45:42 -0800 Subject: [Python-Dev] Listing of PEP 328 status and possible minor oversight In-Reply-To: <9e804ac0612201217t78325fedn9e798ff014e5d4f@mail.gmail.com> References: <9e804ac0612201217t78325fedn9e798ff014e5d4f@mail.gmail.com> Message-ID: On 12/20/06, Thomas Wouters wrote: > > > > On 12/20/06, Brett Cannon wrote: > > > > I just noticed that PEP 328 (relative imports) is listed as an accepted > > PEP, but not completed. Is this because there is still things to do for > > 2.6 and 2.7? Or did someone just forget to move it to the completed > > PEPs section of the PEP index? If it is the former then PEP 362 will need > > to be moved. > > > There was some minor finagling about the wording of the PEP, I think, that > I was still supposed to do. Some explanation of common practices, maybe; the > python-dev archives might have it. I could never get it into the PEP the > right way, the end result was never that big a change (and a bit awkward > with the rest.) > > I also noticed that there is no mention of removing import redirection to > > a top-level module by having None in sys.modules. Was this for > > backwards-compatibility issues, or was it just not thought of? > > > I don't think I've ever heard of that idea. I don't know what 'import > redirection' is either. But maybe it was one of those things that were still > supposed to go into PEP 328. > So if sys.modules has an entry of None for a dotted name then import is supposed to stop trying to import that name and instead import a top-level name. This happens if you do ``import string`` from within a package and there is no 'string' module there. This leads to sys.modules['pkg.string'] being None and thus redirecting to sys.modules['string']. This is only useful for classic relative import semantics and thus can probably go once absolute imports are the only way to do imports. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20061220/6ddaf348/attachment.htm From nnorwitz at gmail.com Thu Dec 21 05:43:15 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 20 Dec 2006 20:43:15 -0800 Subject: [Python-Dev] Request for developer privileges. In-Reply-To: <20061220154115.GA32287@axis.g33x.de> References: <20061220154115.GA32287@axis.g33x.de> Message-ID: Hi Lars. Thanks for all your work on tarfile! Please send your ssh2 key to pydotorg at python.org as an attachment. One of us will add your key. Hopefully I can remember how to do it. Here's some other info: http://www.python.org/dev/faq/#subversion-svn I can't add you to SF to be assigned bugs since I'm not an admin. So hopefully one of the admins can either add me as an admin or you as a developer. I already updated Misc/developers.txt. n -- On 12/20/06, Lars Gust?bel wrote: > Hello, > > my name is Lars Gust?bel (SF gustaebel). I contributed > tarfile.py to the Python standard library in January 2003 and > have been the maintainer since then. I have provided about 25 > patches over the years, most of them fixes, some of them new > features and improvements. As a result, I am pretty familiar > with the Python development process. > > If possible I would like to get developer privileges to be able > to work more actively on tarfile.py for a certain time. > > I am currently implementing read-write POSIX.1-2001 pax format > support. Development is still in progress, but it is already > clear at this point, that it will be a complex change, which > will definitely require some maintenance once it is finished and > in day-to-day use. I would like to clean up the tarfile test > suite during this process as well. The introduction of the pax > format is important because it is the first tar specification > that puts an end to those annoying limitations of the "original" > tar format. It will become the default format for GNU tar some > day. > > Thank you, > Lars. > > -- > Lars Gust?bel > lars at gustaebel.de > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com > From g.brandl at gmx.net Thu Dec 21 09:11:20 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 21 Dec 2006 09:11:20 +0100 Subject: [Python-Dev] Request for developer privileges. In-Reply-To: References: <20061220154115.GA32287@axis.g33x.de> Message-ID: Neal Norwitz schrieb: > Hi Lars. > > Thanks for all your work on tarfile! > > Please send your ssh2 key to pydotorg at python.org as an attachment. > One of us will add your key. Hopefully I can remember how to do it. > Here's some other info: http://www.python.org/dev/faq/#subversion-svn > > I can't add you to SF to be assigned bugs since I'm not an admin. So > hopefully one of the admins can either add me as an admin or you as a > developer. Okay, I added both you as a project admin and Lars as a developer and gave him tracker permissions (hopefully). Georg From ncoghlan at gmail.com Thu Dec 21 10:48:04 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 21 Dec 2006 19:48:04 +1000 Subject: [Python-Dev] classes and cell variables question In-Reply-To: <1d85506f0612191359i7831b097s9a1dcbe1cdaa1e49@mail.gmail.com> References: <1d85506f0612191359i7831b097s9a1dcbe1cdaa1e49@mail.gmail.com> Message-ID: <458A5854.8090208@gmail.com> tomer filiba wrote: > to my understanding of the object model, the code of snippet 1 > and snippet 2 should be equivalent. a class is just a "special function" > that returns its locals automatically and passes them to the metaclass > constructor: PEP 227 (Statically Nested Scopes) covers this in detail (and it echoes Guido's earlier message - the lexical scoping skips class statements because class attributes are intended to be accessed via attribute reference on the class or instance, not as a bare name). You'll probably find the resulting behaviour less surprising if you think of class creation in terms of the exec statement rather than as involving any kind of function definition: d = dict() exec in globals(), d mcl = _determine_metaclass(d, ) # Not magic, but close ;) cls = mcl(, , d) Class bodies actually have more in common with module level code than they do with function bodies (e.g. see how far you get trying to pass a function's co_code object to the exec statement - it breaks for reasons somewhat related to what we're discussing here). To go back to your original example, the second snippet should actually look more like this: --- snippet 2a --- body = """ x = 5 def g(self): print "g", x """ d = dict() exec body in globals(), d barcls = type("barcls", (object,), d) ------ .>>> barcls().g() g Traceback (most recent call last): File "", line 1, in File "", line 4, in g NameError: global name 'x' is not defined Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From amk at amk.ca Thu Dec 21 14:53:43 2006 From: amk at amk.ca (A.M. Kuchling) Date: Thu, 21 Dec 2006 08:53:43 -0500 Subject: [Python-Dev] [Python-checkins] r53110 - python/trunk/Lib/mailbox.py In-Reply-To: <200612210214.55585.fdrake@acm.org> References: <20061220194820.ED8DD1E4008@bag.python.org> <200612210214.55585.fdrake@acm.org> Message-ID: <20061221135343.GA5855@localhost.localdomain> On Thu, Dec 21, 2006 at 02:14:55AM -0500, Fred L. Drake, Jr. wrote: > I (only vaguely) recall Guido telling people that that's how it's documented, > and that's what they should use. Don't know if he still feels that way; if > he's recanted the obscure path, we can fix both the calls and the > documentation. I'd lend a hand! The constants were added in 2.5. I've gone ahead and made the documentation change to trunk and 25-maint, because why were the constants added if they weren't intended to be used? While looking at the code, I noticed a minor potential bug. Python defines SEEK_SET and friends to be 0,1,2 and then os.lseek() maps them to SEEK_SET, SEEK_CUR, SEEK_END. file.seek() does *not* do this; it just passes the 0,1,2 through to the stdio fseek(), and therefore would be broken if there's some weird platform where 0,1,2 aren't the correct values. This doesn't seem worth fixing, given that no one has ever run into it, and might be an argument for removing the conversion from os.lseek(); clearly it isn't needed. --amk From martin at v.loewis.de Thu Dec 21 21:23:10 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 21 Dec 2006 21:23:10 +0100 Subject: [Python-Dev] [Python-checkins] r53110 - python/trunk/Lib/mailbox.py In-Reply-To: <20061221135343.GA5855@localhost.localdomain> References: <20061220194820.ED8DD1E4008@bag.python.org> <200612210214.55585.fdrake@acm.org> <20061221135343.GA5855@localhost.localdomain> Message-ID: <458AED2E.5080709@v.loewis.de> A.M. Kuchling schrieb: > The constants were added in 2.5. I've gone ahead and made the > documentation change to trunk and 25-maint, because why were the > constants added if they weren't intended to be used? See #711830. They were added for compatibility (with earlier Python releases) primarily. Some people believe in these constants, so just let them have them. Python (the API) doesn't really rely on them, but instead guarantees that the numeric values are always the same across platforms. Now it seem that introducing them has the unfortunate side effect that people think they *have* to use them, and that doing so gives better code... Regards, Martin From kbk at shore.net Fri Dec 22 06:14:17 2006 From: kbk at shore.net (Kurt B. Kaiser) Date: Fri, 22 Dec 2006 00:14:17 -0500 (EST) Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200612220514.kBM5EH59004389@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 420 open ( +6) / 3510 closed (+12) / 3930 total (+18) Bugs : 944 open ( -5) / 6391 closed (+15) / 7335 total (+10) RFE : 249 open ( +2) / 245 closed ( +0) / 494 total ( +2) New / Reopened Patches ______________________ cp720 encoding map (2006-12-16) http://python.org/sf/1616979 opened by Alexander Belchenko urllib HTTPS Basic authentication fix (2006-12-17) CLOSED http://python.org/sf/1617413 opened by Dug Song The ability to keep configuration files intact (2006-12-17) CLOSED http://python.org/sf/1617496 opened by Roman Kurakin extended slicing for bytes objects (2006-12-18) CLOSED http://python.org/sf/1617678 opened by Thomas Wouters slice-object support for sre_parse.SubPattern (2006-12-18) CLOSED http://python.org/sf/1617680 opened by Thomas Wouters specialcase simple sliceobj in tuple/str/unicode (2006-12-18) http://python.org/sf/1617682 opened by Thomas Wouters specialcase simple sliceobj in list (and bugfixes) (2006-12-18) http://python.org/sf/1617687 opened by Thomas Wouters Extended slicing for UserString (2006-12-18) http://python.org/sf/1617691 opened by Thomas Wouters Extended slicing for array objects (2006-12-18) http://python.org/sf/1617698 opened by Thomas Wouters slice-object support for ctypes Pointer/Array (2006-12-18) http://python.org/sf/1617699 opened by Thomas Wouters slice-object support for mmap (2006-12-18) http://python.org/sf/1617700 opened by Thomas Wouters extended slicing for structseq (2006-12-18) http://python.org/sf/1617701 opened by Thomas Wouters extended slicing for buffer objects (2006-12-18) http://python.org/sf/1617702 opened by Thomas Wouters add None values in SimpleXMLRPCServer (2006-12-19) http://python.org/sf/1618485 opened by Maximiliano Curia sys.intern() 2to3 fixer (2006-12-19) CLOSED http://python.org/sf/1619049 opened by Georg Brandl fix urllib to raise IOError correctly (2006-12-20) CLOSED http://python.org/sf/1619247 opened by Dug Song Bug fixes for int unification branch (2006-12-20) http://python.org/sf/1619846 opened by Adam Olsen Improve platform.py usability on Windows (2006-12-21) http://python.org/sf/1620174 opened by Luke Dunstan Patches Closed ______________ urllib HTTPS Basic authentication fix (2006-12-17) http://python.org/sf/1617413 closed by akuchling The ability to keep configuration files intact (2006-12-17) http://python.org/sf/1617496 closed by loewis extended slicing for bytes objects (2006-12-18) http://python.org/sf/1617678 closed by twouters slice-object support for sre_parse.SubPattern (2006-12-18) http://python.org/sf/1617680 closed by twouters BZ2File.seek() fails for large files (2006-12-14) http://python.org/sf/1615868 closed by akuchling 1572210 doc patch (2006-11-21) http://python.org/sf/1600491 closed by akuchling cookielib: lock acquire/release try..finally protected (2006-10-30) http://python.org/sf/1587139 closed by akuchling sys.intern() (2006-11-23) http://python.org/sf/1601678 closed by gbrandl sys.intern() 2to3 fixer (2006-12-19) http://python.org/sf/1619049 closed by gbrandl tarfile.py fix for #1471427 and updates (2006-05-09) http://python.org/sf/1484695 closed by gbrandl encoding directive -- more examples (2006-12-11) http://python.org/sf/1613352 closed by gbrandl fix urllib to raise IOError correctly (2006-12-20) http://python.org/sf/1619247 closed by gbrandl New / Reopened Bugs ___________________ Wrong pathname value in logging output (2006-12-15) http://python.org/sf/1616422 reopened by simleo Wrong pathname value in logging output (2006-12-15) http://python.org/sf/1616422 reopened by simleo Wrong pathname value in logging output (2006-12-15) http://python.org/sf/1616422 opened by Tekkaman Vague description of generator close() method (2006-12-15) CLOSED http://python.org/sf/1616726 opened by Lenard Lindstrom Instance methods compare equal when their self's are equal (2006-12-16) http://python.org/sf/1617161 opened by Frank Niessink test_logging hangs on cygwin (2006-04-06) http://python.org/sf/1465643 reopened by tebeka missing word in "sqlite3-Module-Contents"-documentation? (2006-12-18) CLOSED http://python.org/sf/1618083 opened by lgvienna HMAC can get a 6x performance increase easily (2006-12-18) CLOSED http://python.org/sf/1618455 opened by Ben Maurer bisect on presorted list (2006-12-19) http://python.org/sf/1619060 opened by Jeffrey C. Jacobs 64-bit Universal Binary build broken (2006-12-19) http://python.org/sf/1619130 opened by Thomas Treadway format error in 2.5 ref.pdf in section 5.3.4 Calls (2006-12-20) http://python.org/sf/1619641 opened by cwm42 htonl, ntohl don't handle negative longs (2006-12-20) http://python.org/sf/1619659 opened by Adam Olsen Minor error in sum() docs (2006-12-20) CLOSED http://python.org/sf/1619674 opened by Kent Johnson ctypes in_dll documented with wrong argument order (2006-12-20) CLOSED http://python.org/sf/1619680 opened by Lenard Lindstrom Bugs Closed ___________ Wrong pathname value in logging output (2006-12-15) http://python.org/sf/1616422 closed by vsajip Vague description of generator close() method (2006-12-15) http://python.org/sf/1616726 closed by akuchling AttributesImpl does not implement __contains__ on Linux (2006-12-13) http://python.org/sf/1614387 closed by gbrandl IA64/AMD64/x64 confusion (2006-12-15) http://python.org/sf/1616109 closed by loewis Python socket library confused by IPV6 notation in /etc/host (2006-11-27) http://python.org/sf/1603527 closed by loewis missing word in "sqlite3-Module-Contents"-documentation? (2006-12-18) http://python.org/sf/1618083 closed by akuchling HMAC can get a 6x performance increase easily (2006-12-18) http://python.org/sf/1618455 closed by akuchling recv_into not documented (2006-12-11) http://python.org/sf/1613651 closed by akuchling random.randrange don't return correct value for big number (2006-11-05) http://python.org/sf/1590891 closed by rhettinger Minor error in sum() docs (2006-12-20) http://python.org/sf/1619674 closed by akuchling ctypes in_dll documented with wrong argument order (2006-12-20) http://python.org/sf/1619680 closed by akuchling Problem with tapedevices and the tarfile module (2005-11-21) http://python.org/sf/1362587 closed by esshenrik resolver not thread safe (2002-04-04) http://python.org/sf/539175 closed by dustin New / Reopened RFE __________________ Improve list comprehensions to Language Integrated Query? (2006-12-19) http://python.org/sf/1618676 opened by JettLogic From amk at amk.ca Fri Dec 22 14:05:47 2006 From: amk at amk.ca (A.M. Kuchling) Date: Fri, 22 Dec 2006 08:05:47 -0500 Subject: [Python-Dev] PEP 8: mention bare 'except:' Message-ID: <20061222130547.GA5678@localhost.localdomain> http://www.python.org/sf/411881 is a bug about removing use of 'except:' in stdlib code. In many cases the intent is to catch one particular exception such as ImportError or AttributeError, but catching all exceptions can disguise other problems. Should PEP 8 mention this issue? Here's some proposed text for discussion: - When catching exceptions, mention specific exceptions whenever possible instead of using a bare 'except:' clause. For example, use:: try: import platform_specific_module except ImportError: platform_specific_module = None A bare 'except:' clause will catch SystemExit and KeyboardInterrupt exceptions, making it harder to interrupt a program with Control-C, and can disguise other problems. If you want to catch all exceptions that signal program errors, use 'except StandardError:'. A good rule of thumb is that you should only use 'except:' if the exception handler will be printing out or logging the traceback; at least the user will be aware that an error has occurred. --amk From barry at python.org Fri Dec 22 16:37:45 2006 From: barry at python.org (Barry Warsaw) Date: Fri, 22 Dec 2006 10:37:45 -0500 Subject: [Python-Dev] PEP 8: mention bare 'except:' In-Reply-To: <20061222130547.GA5678@localhost.localdomain> References: <20061222130547.GA5678@localhost.localdomain> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Dec 22, 2006, at 8:05 AM, A.M. Kuchling wrote: > A good rule of thumb is that you should only use 'except:' if the > exception handler will be printing out or logging the traceback; at > least the user will be aware that an error has occurred. Another aspect of bare except (IME) is that they are used when you want to do some work, but continue to propagate the exception upwards via a bare raise. E.g. rolling back a transaction, or as you mention, logging the error and continuing. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRYv7znEjvBPtnXfVAQKjYAP/fxLKXLYQvA/CAfYKOVhBkeIUdqJ7qyxH fv6PXuOM7RwS4FuwtmmbHOw9omyO0Kv+iddx/MD+2hJN9TeeLveGSyr4kEDG5cYD APhy8KW18bKhtwAiZbxzV4VQ3Q22IQah/nHEmzUfaRYAl8Dms9FjwIPIrlEY2yaS TY+MNrUum2Y= =CZuk -----END PGP SIGNATURE----- From jeremy.kloth at 4suite.org Fri Dec 22 20:42:11 2006 From: jeremy.kloth at 4suite.org (Jeremy Kloth) Date: Fri, 22 Dec 2006 12:42:11 -0700 Subject: [Python-Dev] multiple interpreters and extension modules Message-ID: <200612221242.11576.jeremy.kloth@4suite.org> [[ This may be somewhat c.l.p.-ish but I feel that this crossed into CPython development enough to merit posting here ]] I have received a bug report for 4Suite that involves a PyObject_IsInstance check failing for what appears to be the correct class, that is, the class names match. With some investigating, I have found that the true problem is with multiple interpreters. The reason for this is that each sub-interpreter has a "new" copy of any pure Python module. The following issues are also true for modules that have been reloaded, but I think that is common knowledge. I mention it only for completeness. Here is where it crosses into CPython development. Now extension modules do not get reloaded into each sub-interpreter, but a copy of the module's dict from when it was first loaded. This particular extension, when initialized, loads the class in question from the pure Python module and stores in in a global variable for quick access. I know I can change this specific case to reload the class each time and for this exact case, it is most likely what I'll do. However, we have other extensions that make use of this same technique and their performance would suffer greatly by doing the lookup for each use of the class. Also performance isn't the only issue here as we also have C types that inherit from a Python class and those types will obviously fail an isinstance() check in Python code run in different interpreters as the base class is "frozen" at the first import. So I guess the questions are: 1) is subclassing Python classes in C as a static type supported? Even if they would be declared on the heap, they would be bound to the first loaded Python class. 2) would it be worthwhile to have an extension loading function similar to DllMain() for Windows' DLLs which is called multiple times with a flag stating the reason for the call? For reference, see: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/dllmain.asp I envision the reasons as: 'module init', 'module fini', 'interpreter init', 'interpreter fini'. I see no current need for special treatment of threads as they share the same interpreter space. As to question 2, I'm in the process of prototyping that approach to see if it is feasible in our code and if so, (and people find it useful) I'll attempt to write a PEP for that feature. My current thoughts on that are instead of the current init entry point in C modules, a new entry point is defined, and if found, used in its place. That way, this proposal would be optional for extension writers and existing modules would continue to work without change. Feedback appreciated. -- Jeremy Kloth http://4suite.org/ From brett at python.org Fri Dec 22 21:06:59 2006 From: brett at python.org (Brett Cannon) Date: Fri, 22 Dec 2006 12:06:59 -0800 Subject: [Python-Dev] Possible platforms to drop in 2.6 Message-ID: I originally posted this list to python-3000 since I figured we could be more aggressive with Py3K, but Guido said I should move it over here and just be aggressive in 2.6. So, here are the platforms I figured we should drop: * AtheOS * BeOS * FreeBSD 2 (maybe more?) * IRIX * NeXT * OS/2 * UnixWare I had SunoS 5 but Ronald Oussoren said that is actually Solaris through version 9, so I took that off. Several people have questioned AtheOS, but considering the site for the OS has not been updated since 2002 and it was a niche OS to begin with I doubt we really need the support. There is a fork of AtheOS called Syllable (http://www.syllable.org/) that is still being developed, but someone from that group should probably step forward and offer to help maintain the code if we are to keep it. Same goes with BeOS and Zeta (http://www.zeta-os.com/cms/news.php; there is a drop-down in the right-hand nav column to change the language to English for us non-German speakers). And I listed FreeBSD 2 as a drop since FreeBSD 3 seemed to have been released in 1999. But if most users have upgraded by now (release 6 is the most current) then we could consider dropping 3 as well. -Brett From brett at python.org Fri Dec 22 21:15:23 2006 From: brett at python.org (Brett Cannon) Date: Fri, 22 Dec 2006 12:15:23 -0800 Subject: [Python-Dev] PEP 8: mention bare 'except:' In-Reply-To: <20061222130547.GA5678@localhost.localdomain> References: <20061222130547.GA5678@localhost.localdomain> Message-ID: On 12/22/06, A.M. Kuchling wrote: > http://www.python.org/sf/411881 is a bug about removing use > of 'except:' in stdlib code. In many cases the intent is to catch > one particular exception such as ImportError or AttributeError, > but catching all exceptions can disguise othter problems. > > Should PEP 8 mention this issue? Yes. -Brett From skip at pobox.com Fri Dec 22 21:19:35 2006 From: skip at pobox.com (skip at pobox.com) Date: Fri, 22 Dec 2006 14:19:35 -0600 Subject: [Python-Dev] Possible platforms to drop in 2.6 In-Reply-To: References: Message-ID: <17804.15831.520751.95213@montanaro.dyndns.org> Brett> So, here are the platforms I figured we should drop: ... Brett> * OS/2 I'm pretty sure Andrew MacIntyre is still maintaining the OS/2+EMX port: http://members.pcug.org.au/~andymac/python.html Skip From jcarlson at uci.edu Sat Dec 23 01:02:32 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri, 22 Dec 2006 16:02:32 -0800 Subject: [Python-Dev] multiple interpreters and extension modules In-Reply-To: <200612221242.11576.jeremy.kloth@4suite.org> References: <200612221242.11576.jeremy.kloth@4suite.org> Message-ID: <20061222155542.BC31.JCARLSON@uci.edu> Jeremy Kloth wrote: > [[ This may be somewhat c.l.p.-ish but I feel that this crossed into CPython > development enough to merit posting here ]] > > I have received a bug report for 4Suite that involves a PyObject_IsInstance > check failing for what appears to be the correct class, that is, the class > names match. With some investigating, I have found that the true problem is > with multiple interpreters. The reason for this is that each sub-interpreter > has a "new" copy of any pure Python module. The following issues are also > true for modules that have been reloaded, but I think that is common > knowledge. I mention it only for completeness. If I remember correctly, Python allows you to use multiple interpreters in the same process, but it makes no guarantees as to their correctness when running. See this post for further discussion on the issue: http://mail.python.org/pipermail/python-list/2004-January/244343.html You can also search for 'multiple python interpreters single process' in google without quotes to hear people lament over the (generally broken) multiple Python interpreter support. - Josiah From rrr at ronadam.com Sat Dec 23 01:00:31 2006 From: rrr at ronadam.com (Ron Adam) Date: Fri, 22 Dec 2006 18:00:31 -0600 Subject: [Python-Dev] Pydoc Improvements / Rewrite Message-ID: <458C719F.3000904@ronadam.com> I started to discuss this on the new python-ideas list because I thought it may need a bit more completion before bringing it here. But It was suggested that it really does belong here as it is something that would be nice to have in python 2.6. So I'm reposting the introduction here. The still very rough source files can be downloaded from: http://ronadam.com/dl/_pydoc.zip There is still much to do, but I think having some experienced feed back on where it should go is important. Cheers, Ron Adam ps.. Please disregard the website for now, it's purpose was to share with family and friends, although I will probably share the python source to the ledger program there and other python toys and experiments (for fun) in the near future. [Introduction From python-ideas list] Improving pydoc has been suggested before by me and others. I've been working on a version that is probably 80% done and would like to get feed back at this point to determine if I'm approaching this in the best way. Basically pydoc.py consists of over 2,000 lines of python code and is not well organized inside which means (my thoughts) it will pretty much stay the way it is and not be as useful as it could be. Currently pydoc.py has the following uses. * It is imported and used as pythons console help function. * It can be used to generate help text files. * It can open a tkinter search index and from that launch a web server and a browser to veiw a html help page. * It can be used to generate static html help files. It looks (to me) like splitting it into two modules would be good. One module for just the text help and introspection functions, and the other for the html server and html output stuff. [It was suggested on python-ideas that making it a package may be good.] 1. pyhelp.py - Pythons help function from inside the console, and running it directly would open an interactive text help session. 2. _pydoc.py - Python html help browser. This starts an html server and opens a web page with a modules/package index. The html page headers would contain the current Python version info and the following inputs fields. * get field - directly bring up help on an object/module/or package. * Search field - returns a search results index. * Modules button - modules index page * Keywords button - keywords index page * Help button - pydoc Help instructions, version, and credits info. Note: The leading underscore "_pydoc.py" is to keep it from clashing with the current pydoc version. It probably should be something else. An additional feature is clicking on a filename opens up a numbered source listing. Which is nice for source code browsing and for referencing specific lines of code in python list discussions. ;-) The colors, fonts and general appearance can be changed by editing the style sheet. The output is readable as plain (outline form) text if the style sheet is ignored by the browser. _pydoc.py imports pyhelp and uses it to do the introspection work and extends classes in pyhelp to produce html output. I've tried to make pyhelp.py useful in a general way so that it can more easily be used as a base that other output formats can be built from. That's something that can't be done presently. These improvements to pydoc mean you can browse pythons library dynamically without ever leaving the web browser. Currently you switch back and forth between the browser and a tkinter index window. Something I found to be annoying enough to discourage me from using pydoc. The version I'm working on is split up into eight python files, each addressing a particular function of pydoc. That was to help me organize things better. These will be recombined into fewer files. Some parts of it could be moved to other modules if they seem more generally useful. For example, the console text pager could be used in many other console applications. Things that still need to be done are adding the object resolution order output back in. And adding inter-page html links back in. And a few other things I just haven't gotten to yet. I got a bit burned out on this project a while back, and then moved to a new house.. etc.. etc.. But I'm starting to have more time, and with the current discussion s turning on to refining pythons library this seems like it would be a useful tool in that effort. Any comments on this general approach? Any suggestions, questions, or comments? I'll post a link to the zipped files in the next day or two and announce it here. I need to look into a glitch on my computer first that's probably a windows path/name problem. I don't think it's anything that needs to be fixed in the files but I want to be sure. Cheers, Ron Adam From jeremy.kloth at 4suite.org Sat Dec 23 01:49:18 2006 From: jeremy.kloth at 4suite.org (Jeremy Kloth) Date: Fri, 22 Dec 2006 17:49:18 -0700 Subject: [Python-Dev] multiple interpreters and extension modules In-Reply-To: <20061222155542.BC31.JCARLSON@uci.edu> References: <200612221242.11576.jeremy.kloth@4suite.org> <20061222155542.BC31.JCARLSON@uci.edu> Message-ID: <200612221749.18493.jeremy.kloth@4suite.org> On Friday 22 December 2006 5:02 pm, Josiah Carlson wrote: > Jeremy Kloth wrote: > > [[ This may be somewhat c.l.p.-ish but I feel that this crossed into > > CPython development enough to merit posting here ]] > > > > I have received a bug report for 4Suite that involves a > > PyObject_IsInstance check failing for what appears to be the correct > > class, that is, the class names match. With some investigating, I have > > found that the true problem is with multiple interpreters. The reason > > for this is that each sub-interpreter has a "new" copy of any pure Python > > module. The following issues are also true for modules that have been > > reloaded, but I think that is common knowledge. I mention it only for > > completeness. > > If I remember correctly, Python allows you to use multiple interpreters > in the same process, but it makes no guarantees as to their correctness > when running. > > See this post for further discussion on the issue: > http://mail.python.org/pipermail/python-list/2004-January/244343.html > > You can also search for 'multiple python interpreters single process' in > google without quotes to hear people lament over the (generally broken) > multiple Python interpreter support. The problem here is that it is mod_python using the multiple interpreters. We have no control over that. What I'm proposing is fixing the extension module support for multiple interpreters with the bonus of adding extension module finalization which I've seen brought up here before. Fixing this does require support by the extension module author, but if that author doesn't feel the need to work in mod_python (if, of course, they load module level constants), that is their loss. Is 4Suite that different in its use of hybrid Python and C extensions? There is lots of back and forth between the two layers and performance is critical. I really don't feel like recoding thousands of lines of Python code into C just to get 4Suite to work in mod_python without error. -- Jeremy Kloth http://4suite.org/ From guido at python.org Sat Dec 23 03:16:37 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 22 Dec 2006 18:16:37 -0800 Subject: [Python-Dev] multiple interpreters and extension modules In-Reply-To: <200612221242.11576.jeremy.kloth@4suite.org> References: <200612221242.11576.jeremy.kloth@4suite.org> Message-ID: I think you understand exactly what is happening. It is happening for good reasons. Rather than asking for a change in semantics, I recommend that you deal with it, either in your Python code, or in your extension. It's not likely to change. --Guido On 12/22/06, Jeremy Kloth wrote: > [[ This may be somewhat c.l.p.-ish but I feel that this crossed into CPython > development enough to merit posting here ]] > > I have received a bug report for 4Suite that involves a PyObject_IsInstance > check failing for what appears to be the correct class, that is, the class > names match. With some investigating, I have found that the true problem is > with multiple interpreters. The reason for this is that each sub-interpreter > has a "new" copy of any pure Python module. The following issues are also > true for modules that have been reloaded, but I think that is common > knowledge. I mention it only for completeness. > > Here is where it crosses into CPython development. > > Now extension modules do not get reloaded into each sub-interpreter, but a > copy of the module's dict from when it was first loaded. This particular > extension, when initialized, loads the class in question from the pure Python > module and stores in in a global variable for quick access. I know I can > change this specific case to reload the class each time and for this exact > case, it is most likely what I'll do. However, we have other extensions that > make use of this same technique and their performance would suffer greatly by > doing the lookup for each use of the class. > > Also performance isn't the only issue here as we also have C types that > inherit from a Python class and those types will obviously fail an > isinstance() check in Python code run in different interpreters as the base > class is "frozen" at the first import. > > So I guess the questions are: > 1) is subclassing Python classes in C as a static type supported? Even if they > would be declared on the heap, they would be bound to the first loaded Python > class. > > 2) would it be worthwhile to have an extension loading function similar to > DllMain() for Windows' DLLs which is called multiple times with a flag > stating the reason for the call? For reference, see: > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/dllmain.asp > I envision the reasons as: 'module init', 'module fini', 'interpreter > init', 'interpreter fini'. I see no current need for special treatment of > threads as they share the same interpreter space. > > As to question 2, I'm in the process of prototyping that approach to see if it > is feasible in our code and if so, (and people find it useful) I'll attempt > to write a PEP for that feature. > > My current thoughts on that are instead of the current init entry > point in C modules, a new entry point is defined, and if found, used in its > place. That way, this proposal would be optional for extension writers and > existing modules would continue to work without change. > > Feedback appreciated. > > -- > Jeremy Kloth > http://4suite.org/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bob at redivi.com Sat Dec 23 03:54:39 2006 From: bob at redivi.com (Bob Ippolito) Date: Sat, 23 Dec 2006 10:54:39 +0800 Subject: [Python-Dev] multiple interpreters and extension modules In-Reply-To: <200612221749.18493.jeremy.kloth@4suite.org> References: <200612221242.11576.jeremy.kloth@4suite.org> <20061222155542.BC31.JCARLSON@uci.edu> <200612221749.18493.jeremy.kloth@4suite.org> Message-ID: <6a36e7290612221854j330bfdf9qb22c6cb2331930e9@mail.gmail.com> On 12/23/06, Jeremy Kloth wrote: > On Friday 22 December 2006 5:02 pm, Josiah Carlson wrote: > > Jeremy Kloth wrote: > > > [[ This may be somewhat c.l.p.-ish but I feel that this crossed into > > > CPython development enough to merit posting here ]] > > > > > > I have received a bug report for 4Suite that involves a > > > PyObject_IsInstance check failing for what appears to be the correct > > > class, that is, the class names match. With some investigating, I have > > > found that the true problem is with multiple interpreters. The reason > > > for this is that each sub-interpreter has a "new" copy of any pure Python > > > module. The following issues are also true for modules that have been > > > reloaded, but I think that is common knowledge. I mention it only for > > > completeness. > > > > If I remember correctly, Python allows you to use multiple interpreters > > in the same process, but it makes no guarantees as to their correctness > > when running. > > > > See this post for further discussion on the issue: > > http://mail.python.org/pipermail/python-list/2004-January/244343.html > > > > You can also search for 'multiple python interpreters single process' in > > google without quotes to hear people lament over the (generally broken) > > multiple Python interpreter support. > > The problem here is that it is mod_python using the multiple interpreters. We > have no control over that. What I'm proposing is fixing the extension module > support for multiple interpreters with the bonus of adding extension module > finalization which I've seen brought up here before. > > Fixing this does require support by the extension module author, but if that > author doesn't feel the need to work in mod_python (if, of course, they load > module level constants), that is their loss. > > Is 4Suite that different in its use of hybrid Python and C extensions? There > is lots of back and forth between the two layers and performance is critical. > I really don't feel like recoding thousands of lines of Python code into C > just to get 4Suite to work in mod_python without error. It's a whole lot more practical to just stop using mod_python and go for one of the other ways of exposing Python code to the internet. I bet you can get the same or better performance out of another solution anyway, and you'd save deployment headaches. -bob From jeremy.kloth at 4suite.org Sat Dec 23 04:04:25 2006 From: jeremy.kloth at 4suite.org (Jeremy Kloth) Date: Fri, 22 Dec 2006 20:04:25 -0700 Subject: [Python-Dev] multiple interpreters and extension modules In-Reply-To: <6a36e7290612221854j330bfdf9qb22c6cb2331930e9@mail.gmail.com> References: <200612221242.11576.jeremy.kloth@4suite.org> <200612221749.18493.jeremy.kloth@4suite.org> <6a36e7290612221854j330bfdf9qb22c6cb2331930e9@mail.gmail.com> Message-ID: <200612222004.26146.jeremy.kloth@4suite.org> On Friday 22 December 2006 7:54 pm, Bob Ippolito wrote: > It's a whole lot more practical to just stop using mod_python and go > for one of the other ways of exposing Python code to the internet. I > bet you can get the same or better performance out of another solution > anyway, and you'd save deployment headaches. I have no control over end-users choice of Python/webserver integration, I just end up making it possible to run our software in the environment of *their* choice. If it is the opinion that it is mod_python that is broken, I'd gladly point the users to the location stating that fact/belief. It would make my life easier. -- Jeremy Kloth http://4suite.org/ From jeremy.kloth at 4suite.org Sat Dec 23 04:09:56 2006 From: jeremy.kloth at 4suite.org (Jeremy Kloth) Date: Fri, 22 Dec 2006 20:09:56 -0700 Subject: [Python-Dev] multiple interpreters and extension modules In-Reply-To: References: <200612221242.11576.jeremy.kloth@4suite.org> Message-ID: <200612222009.56915.jeremy.kloth@4suite.org> On Friday 22 December 2006 7:16 pm, Guido van Rossum wrote: > I think you understand exactly what is happening. It is happening for > good reasons. Rather than asking for a change in semantics, I > recommend that you deal with it, either in your Python code, or in > your extension. It's not likely to change. I don't believe I was asking for a change in semantics, rather an additional, optional interface for extension module writers. > > 2) would it be worthwhile to have an extension loading function similar > > to DllMain() for Windows' DLLs which is called multiple times with a flag > > stating the reason for the call? For reference, see: > > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/ > >base/dllmain.asp I envision the reasons as: 'module init', 'module fini', > > 'interpreter init', 'interpreter fini'. I see no current need for special > > treatment of threads as they share the same interpreter space. > > > > As to question 2, I'm in the process of prototyping that approach to see > > if it is feasible in our code and if so, (and people find it useful) I'll > > attempt to write a PEP for that feature. I'll add here that it has been brought up here before that extension module finalization is a feature that would be appreciated. With that, it is not that far to add support for initialization/finalization for each interpreter. That is, of course, using a DllMain-like solution. > > My current thoughts on that are instead of the current init > > entry point in C modules, a new entry point is defined, and if found, > > used in its place. That way, this proposal would be optional for > > extension writers and existing modules would continue to work without > > change. With that approach in mind, I will be making changes so 4Suite will work in a production mod_python deployment (where the aforementioned error occurred). When that works, I'll come back with a proper PEP *and* patches against Python SVN to support its use. I hope no one was thinking I wanted someone else to do the work. -- Jeremy Kloth http://4suite.org/ From bob at redivi.com Sat Dec 23 05:11:27 2006 From: bob at redivi.com (Bob Ippolito) Date: Sat, 23 Dec 2006 12:11:27 +0800 Subject: [Python-Dev] multiple interpreters and extension modules In-Reply-To: <200612222004.26146.jeremy.kloth@4suite.org> References: <200612221242.11576.jeremy.kloth@4suite.org> <200612221749.18493.jeremy.kloth@4suite.org> <6a36e7290612221854j330bfdf9qb22c6cb2331930e9@mail.gmail.com> <200612222004.26146.jeremy.kloth@4suite.org> Message-ID: <6a36e7290612222011u5dfb1d78l5921d182f311da77@mail.gmail.com> On 12/23/06, Jeremy Kloth wrote: > On Friday 22 December 2006 7:54 pm, Bob Ippolito wrote: > > It's a whole lot more practical to just stop using mod_python and go > > for one of the other ways of exposing Python code to the internet. I > > bet you can get the same or better performance out of another solution > > anyway, and you'd save deployment headaches. > > I have no control over end-users choice of Python/webserver integration, I > just end up making it possible to run our software in the environment of > *their* choice. > > If it is the opinion that it is mod_python that is broken, I'd gladly point > the users to the location stating that fact/belief. It would make my life > easier. Well, it clearly is broken wrt pure python modules and objects that persist across requests. I believe that it's also broken with any extension that uses the PyGILState API due to the way it interacts with multiple interpreters. I stopped using mod_python years ago due to the sorts of issues that you're bringing up here (plus problems compiling, deploying, RAM bloat, etc.). I don't have any recent experience or references that I can point you to, but I can definitely say that I have had many good experiences with the WSGI based solutions (and Twisted, but that's a different game). I would at least advise your user that there are several perfectly good ways to make Python speak HTTP, and mod_python is the only one with this issue. -bob From xramtsov at gmail.com Sat Dec 23 06:03:28 2006 From: xramtsov at gmail.com (Evgeniy Khramtsov) Date: Sat, 23 Dec 2006 14:03:28 +0900 Subject: [Python-Dev] Non-blocking (asynchronous) timer without thread? Message-ID: <458CB8A0.9030907@gmail.com> The question to python core developers: Is there any plans to implement non-blocking timer like a threading.Timer() but without thread? Some interpreted languages (like Tcl or Erlang) have such functionality, so I think it would be a great feature in Python :) The main goal is to prevent threads overhead and problems with race conditions and deadlocks. Thanks in advance! From mike.klaas at gmail.com Sat Dec 23 06:50:09 2006 From: mike.klaas at gmail.com (Mike Klaas) Date: Fri, 22 Dec 2006 21:50:09 -0800 Subject: [Python-Dev] Non-blocking (asynchronous) timer without thread? In-Reply-To: <458CB8A0.9030907@gmail.com> References: <458CB8A0.9030907@gmail.com> Message-ID: <3d2ce8cb0612222150w6a0e4b7do3bd28ea294add85d@mail.gmail.com> On 12/22/06, Evgeniy Khramtsov wrote: > The question to python core developers: > Is there any plans to implement non-blocking timer like a > threading.Timer() but without thread? > Some interpreted languages (like Tcl or Erlang) have such functionality, > so I think it would be a great > feature in Python :) > > The main goal is to prevent threads overhead and problems with race > conditions and deadlocks. I'm not sure how having python execute code at an arbitrary time would _reduce_ race conditions and/or deadlocks. And if you want to make it safe by executing code that shares no variables or resources, then it is no less safe to use threads, due to the GIL. If you can write you application in an event-driven way, Twisted might be able to do what you are looking for. cheers, -Mike From jcarlson at uci.edu Sat Dec 23 07:01:40 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri, 22 Dec 2006 22:01:40 -0800 Subject: [Python-Dev] Non-blocking (asynchronous) timer without thread? In-Reply-To: <458CB8A0.9030907@gmail.com> References: <458CB8A0.9030907@gmail.com> Message-ID: <20061222215125.BC39.JCARLSON@uci.edu> Evgeniy Khramtsov wrote: > > The question to python core developers: > Is there any plans to implement non-blocking timer like a > threading.Timer() but without thread? > Some interpreted languages (like Tcl or Erlang) have such functionality, > so I think it would be a great > feature in Python :) > > The main goal is to prevent threads overhead and problems with race > conditions and deadlocks. How would you propose a non-threaded timer work? Would it involve the writing of a scheduler and the execution of scheduled tasks in a scheduler? If so, you can implement it today (I would suggest using the heapq module, and/or the pair heap I posted to this list a few weeks ago), or you can use a previously existing mainloop for handling events (wx.App().MainLoop() works well, you just need to write a wx.EvtHandler subclass). Heck, you could even use Twisted's event loop to handle it. If you were really crazy, you could even use exceptions to signal that an event was ready... import time import sys _scheduled = [] def hook(*args): if _scheduled and time.time() >= _scheduled[0][0]: raise Exception("Scheduled Event Pending!") sys.settrace(hook) Aside from using a library that does it for you, or hooking into native message hooks (I'm sure that Windows has a "notify this process in X seconds" message), I personally don't see a way of making what you want happen work in Python, or even *how* you would signal to the only thread in your application, "hey, there is this other task that should happen" that is substantially different from a periodic "if time.time() >= scheduled" check. - Josiah From xramtsov at gmail.com Sat Dec 23 07:53:25 2006 From: xramtsov at gmail.com (Evgeniy Khramtsov) Date: Sat, 23 Dec 2006 15:53:25 +0900 Subject: [Python-Dev] Non-blocking (asynchronous) timer without thread? In-Reply-To: <3d2ce8cb0612222150w6a0e4b7do3bd28ea294add85d@mail.gmail.com> References: <458CB8A0.9030907@gmail.com> <3d2ce8cb0612222150w6a0e4b7do3bd28ea294add85d@mail.gmail.com> Message-ID: <458CD265.9070008@gmail.com> Mike Klaas ?????: > I'm not sure how having python execute code at an arbitrary time would > _reduce_ race conditions and/or deadlocks. And if you want to make it > safe by executing code that shares no variables or resources, then it > is no less safe to use threads, due to the GIL. > Ok. And what about a huge thread overhead? Just try to start 10-50k threading timers :) > If you can write you application in an event-driven way, Twisted might > be able to do what you are looking for. I don't like an idea of Twisted: you want the banana, but get the whole gorilla as well :) From fdrake at acm.org Sat Dec 23 07:59:08 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Sat, 23 Dec 2006 01:59:08 -0500 Subject: [Python-Dev] =?iso-8859-1?q?=5BPython-checkins=5D__r53110_-=09pyt?= =?iso-8859-1?q?hon/trunk/Lib/mailbox=2Epy?= In-Reply-To: <458AED2E.5080709@v.loewis.de> References: <20061220194820.ED8DD1E4008@bag.python.org> <20061221135343.GA5855@localhost.localdomain> <458AED2E.5080709@v.loewis.de> Message-ID: <200612230159.08151.fdrake@acm.org> On Thursday 21 December 2006 15:23, Martin v. L?wis wrote: > Now it seem that introducing them has the unfortunate side effect > that people think they *have* to use them, and that doing so gives > better code... Speaking strictly for myself: I don't think I *have* to use them, but I do prefer to use them because I don't like magic constants that affect what a function does in code; I'd rather have a named constant for readability's sake. Maybe I just can't keep enough in my head, but I don't find I actually use seek() enough to remember what the numeric values mean with checking the reference documentation. -Fred -- Fred L. Drake, Jr. From xramtsov at gmail.com Sat Dec 23 08:03:23 2006 From: xramtsov at gmail.com (Evgeniy Khramtsov) Date: Sat, 23 Dec 2006 16:03:23 +0900 Subject: [Python-Dev] Non-blocking (asynchronous) timer without thread? In-Reply-To: <20061222215125.BC39.JCARLSON@uci.edu> References: <458CB8A0.9030907@gmail.com> <20061222215125.BC39.JCARLSON@uci.edu> Message-ID: <458CD4BB.20301@gmail.com> Josiah Carlson ?????: >writing of a scheduler and the execution of scheduled tasks in a >scheduler? If so, you can implement it today (I would suggest using the >heapq module, and/or the pair heap I posted to this list a few weeks ago), >or you can use a previously existing mainloop for handling events >(wx.App().MainLoop() works well, you just need to write a wx.EvtHandler >subclass). > I did a dirty hack in the asyncore.py module, and now my asyncore objects have methods for timers (self.start_timer, self.cancel_timer, etc.) But I think that timer scheduling is a job for a python interpreter and it must be hidden from an end programmer. Regards. From bob at redivi.com Sat Dec 23 08:13:19 2006 From: bob at redivi.com (Bob Ippolito) Date: Sat, 23 Dec 2006 15:13:19 +0800 Subject: [Python-Dev] Non-blocking (asynchronous) timer without thread? In-Reply-To: <458CD265.9070008@gmail.com> References: <458CB8A0.9030907@gmail.com> <3d2ce8cb0612222150w6a0e4b7do3bd28ea294add85d@mail.gmail.com> <458CD265.9070008@gmail.com> Message-ID: <6a36e7290612222313k31ac7f07y8b803e64e03302c0@mail.gmail.com> On 12/23/06, Evgeniy Khramtsov wrote: > Mike Klaas ?????: > > > I'm not sure how having python execute code at an arbitrary time would > > _reduce_ race conditions and/or deadlocks. And if you want to make it > > safe by executing code that shares no variables or resources, then it > > is no less safe to use threads, due to the GIL. > > > Ok. And what about a huge thread overhead? Just try to start 10-50k > threading timers :) > > > If you can write you application in an event-driven way, Twisted might > > be able to do what you are looking for. > > I don't like an idea of Twisted: you want the banana, but get the whole > gorilla as well :) Well you simply can't do what you propose without writing code in the style of Twisted or with interpreter modifications or evil stack slicing such as with stackless or greenlet. If you aren't willing to choose any of those then you'll have to live without that functionality or use another language (though I can't think of any usable ones that actually safely do what you're asking). It should be relatively efficient to do what you want with a thread pool (one thread that manages all of the timers, and worker threads to execute the timer callbacks). FWIW, Erlang doesn't have that functionality. You can wait on messages with a timeout, but there are no interrupts. You do have cheap and isolated processes instead of expensive shared state threads, though. Writing Erlang/OTP code is actually a lot closer to writing Twisted style code than it is to other styles of concurrency (that you'd find in Python). It's just that Erlang/OTP has better support for concurrency oriented programming than Python does (across the board; syntax, interpreter, convention and libraries). -bob From xramtsov at gmail.com Sat Dec 23 08:26:13 2006 From: xramtsov at gmail.com (Evgeniy Khramtsov) Date: Sat, 23 Dec 2006 16:26:13 +0900 Subject: [Python-Dev] Non-blocking (asynchronous) timer without thread? In-Reply-To: <6a36e7290612222313k31ac7f07y8b803e64e03302c0@mail.gmail.com> References: <458CB8A0.9030907@gmail.com> <3d2ce8cb0612222150w6a0e4b7do3bd28ea294add85d@mail.gmail.com> <458CD265.9070008@gmail.com> <6a36e7290612222313k31ac7f07y8b803e64e03302c0@mail.gmail.com> Message-ID: <458CDA15.5010602@gmail.com> Bob Ippolito ?????: > Well you simply can't do what you propose without writing code in the > style of Twisted or with interpreter modifications or evil stack > slicing such as with stackless or greenlet. If you aren't willing to > choose any of those then you'll have to live without that > functionality or use another language (though I can't think of any > usable ones that actually safely do what you're asking). It should be > relatively efficient to do what you want with a thread pool (one > thread that manages all of the timers, and worker threads to execute > the timer callbacks). > > FWIW, Erlang doesn't have that functionality. You can wait on messages > with a timeout, but there are no interrupts. You do have cheap and > isolated processes instead of expensive shared state threads, though. > Writing Erlang/OTP code is actually a lot closer to writing Twisted > style code than it is to other styles of concurrency (that you'd find > in Python). It's just that Erlang/OTP has better support for > concurrency oriented programming than Python does (across the board; > syntax, interpreter, convention and libraries). Ok. I see your point. It looks like Python was not the right choice for my task :( Thanks. From andymac at bullseye.apana.org.au Sat Dec 23 14:19:59 2006 From: andymac at bullseye.apana.org.au (Andrew MacIntyre) Date: Sat, 23 Dec 2006 23:19:59 +1000 Subject: [Python-Dev] Possible platforms to drop in 2.6 In-Reply-To: <17804.15831.520751.95213@montanaro.dyndns.org> References: <17804.15831.520751.95213@montanaro.dyndns.org> Message-ID: <458D2CFF.1050106@bullseye.andymac.org> skip at pobox.com wrote: > Brett> So, here are the platforms I figured we should drop: > > ... > Brett> * OS/2 > > I'm pretty sure Andrew MacIntyre is still maintaining the OS/2+EMX port: > > http://members.pcug.org.au/~andymac/python.html I am, although I haven't managed a binary release of 2.5 yet (should be in a few days). If its just the os2vacpp subdirectory in PC/ that is being discussed, that could go for 2.6. It will be a fair amount of work to disentangle the defines used for the 2 ports - a fair bit of the platform specific code is shared, but with compiler related differences. I would suggest more trouble than its worth for 2.6, but I could work on it. Of course, if the project management decide that even the EMX support should be removed from the official tree - so be it; I will just have to maintain the port outside the official tree. -- ------------------------------------------------------------------------- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac at bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac at pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From tomerfiliba at gmail.com Sat Dec 23 14:31:14 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Sat, 23 Dec 2006 15:31:14 +0200 Subject: [Python-Dev] Non-blocking (asynchronous) timer without thread? Message-ID: <1d85506f0612230531t471afc71o1522407267d2ba4c@mail.gmail.com> > The main goal is to prevent threads overhead and problems with race > conditions and deadlocks. check out stackless python -- http://www.stackless.com/ -tomer From guido at python.org Sat Dec 23 16:39:48 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 23 Dec 2006 07:39:48 -0800 Subject: [Python-Dev] Possible platforms to drop in 2.6 In-Reply-To: <458D2CFF.1050106@bullseye.andymac.org> References: <17804.15831.520751.95213@montanaro.dyndns.org> <458D2CFF.1050106@bullseye.andymac.org> Message-ID: On 12/23/06, Andrew MacIntyre wrote: > Of course, if the project management decide that even the EMX support > should be removed from the official tree - so be it; I will just have > to maintain the port outside the official tree. I don't think that's the objective. In general we don't mind if a resonable amount of platform-specific stuff sprinkled over the source tree, *as long as there is someone to maintain it*. Such code tends to break each time a major feature is added or modified, which is pretty much every release. If no-one is there to repair these, the platform code becomes useless, and *then* it becomes a matter of deciding to clean up unneeded hacks, to simplify future maintenance. We want to keep a reasonable balance where the code is reasonably clean and doesn't contain pockets of dead code that nobody understands, but if someone is actively maintaining a port we'd like to support them by minimizing the effort it takes for them to port each new Python version. It seems that you're on the ball, so we can keep your changes. We would appreciate it if you could point out things you no longer need (I understand the os2vacpp directory might be one of these?). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Sat Dec 23 18:10:25 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat, 23 Dec 2006 09:10:25 -0800 Subject: [Python-Dev] Non-blocking (asynchronous) timer without thread? In-Reply-To: <458CD265.9070008@gmail.com> References: <3d2ce8cb0612222150w6a0e4b7do3bd28ea294add85d@mail.gmail.com> <458CD265.9070008@gmail.com> Message-ID: <20061223085826.BC3C.JCARLSON@uci.edu> Evgeniy Khramtsov wrote: > Mike Klaas ??????????: > > > I'm not sure how having python execute code at an arbitrary time would > > _reduce_ race conditions and/or deadlocks. And if you want to make it > > safe by executing code that shares no variables or resources, then it > > is no less safe to use threads, due to the GIL. > > > Ok. And what about a huge thread overhead? Just try to start 10-50k > threading timers :) That is a foolish task in any language, and is why heap-based schedulers exist. You can use a Queue as synchronization between timer producers and the scheduler, and another Queue as synchronization between the scheduler and your threadpool (5-10 should be fine and have minimal overhead). Also 10-50k threads, in just about any language or runtime (except using C in solaris on SPARC, which can handle that pretty well), is way too much to handle. With thread stack sizes as they are, I wouldn't be surprised if you were running into the limit of your main memory size, and you would certianly be destroying the effectiveness of most caches. One thing that came to mind is that perhaps the "overhead" you are experiencing when using threads isn't context switch time, but is the fact that you have thousands of threads waiting on signals from the OS to continue, or even that Python switches threads after a certain number of bytecodes have been interpreted (if they are not waiting on wait syscalls). Python 2.5 sets the interval at 100 bytecodes (see sys.[get|set]checkinterval()), which is either 1 million or 5 million bytecodes with your load of 10k-50k. For most operations, this would be a palpable delay of up to a couple seconds before a thread gets back to doing what it was doing. Seriously, threadpool with a scheduler. - Josiah From tonynelson at georgeanelson.com Sat Dec 23 20:12:52 2006 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Sat, 23 Dec 2006 19:12:52 -0000 Subject: [Python-Dev] Python and the Linux Standard Base (LSB) In-Reply-To: <4571D728.9070506@v.loewis.de> References: <5.1.1.6.0.20061126143124.027f3e68@sparrow.telecommunity.com> <5.1.1.6.0.20061127090652.04341cf8@sparrow.telecommunity.com> <200611290053.17199.anthony@interlink.com.au> <20061129112354.GA30665@code0.codespeak.net> <1164917149.31269.344.camel@localhost> <456F51C5.8080500@holdenweb.com> <1164930163.31269.407.camel@localhost> <456F747B.9060107@holdenweb.com> <1165075639.31269.480.camel@localhost> <4571D728.9070506@v.loewis.de> Message-ID: At 8:42 PM +0100 12/2/06, Martin v. L?wis wrote: >Jan Claeys schrieb: >> Like I said, it's possible to split Python without making things >> complicated for newbies. > >You may have that said, but I don't believe its truth. For example, >most distributions won't include Tkinter in the "standard" Python >installation: Tkinter depends on _tkinter depends on Tk depends on >X11 client libraries. Since distributors want to make X11 client >libraries optional, they exclude Tkinter. So people wonder why >they can't run Tkinter applications (search comp.lang.python for >proof that people wonder about precisely that). > >I don't think the current packaging tools can solve this newbie >problem. It might be solvable if installation of X11 libraries >would imply installation of Tcl, Tk, and Tkinter: people running >X (i.e. most desktop users) would see Tkinter installed, yet >it would be possible to omit Tkinter. Given the current packaging tools, could Python have stub modules for such things that would just throw a useful exception giving the name of the required package? Perhaps if Python just had an example of such a stub (and Tkinter comes to mind), packagers would customize it and make any others they needed? -- ____________________________________________________________________ TonyN.:' The Great Writ ' is no more. From anthony at interlink.com.au Sun Dec 24 00:18:38 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Sun, 24 Dec 2006 10:18:38 +1100 Subject: [Python-Dev] Possible platforms to drop in 2.6 In-Reply-To: <458D2CFF.1050106@bullseye.andymac.org> References: <17804.15831.520751.95213@montanaro.dyndns.org> <458D2CFF.1050106@bullseye.andymac.org> Message-ID: <200612241018.41366.anthony@interlink.com.au> On Sunday 24 December 2006 00:19, Andrew MacIntyre wrote: > Of course, if the project management decide that even the EMX > support should be removed from the official tree - so be it; I > will just have to maintain the port outside the official tree. I feel that so long as there's an active maintainer for a port, it can and should stay. It's the older systems where we have no maintainer and no way to check if it's still working that are the problem. From guido at python.org Sun Dec 24 19:06:07 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 24 Dec 2006 10:06:07 -0800 Subject: [Python-Dev] Fwd: survey about the use of agile practices In-Reply-To: <7LAcRIZu.1166981065.5419330.s.uras@diee.unica.it> References: <7LAcRIZu.1166981065.5419330.s.uras@diee.unica.it> Message-ID: Anyone? ---------- Forwarded message ---------- From: Selene Uras Date: Dec 24, 2006 9:24 AM Subject: survey about the use of agile practices To: s.uras at diee.unica.it Dear project manager/developer I am conducting at the University of Cagliari, under the supervision of prof. Michele Marchesi, a survey about the use of agile practices in open source successful projects. I would really appreciate if you can devote 5 mins. of your time, answering the following questionnaire directly in the "Reply" mail to me. Thanking in advance, Selene Uras 1. Project management: - Do you use fixed (time-boxed) iterations? Y/N How many days do they last? ___ - Do you plan development by gathering a list of features, and then assign them to an iteration? Y/N Since when? Since the start of the project / Since ___ years. - Are users actively involved in requirements gathering and in project management? Y/N 2. Testing: - Has the project an automatic unit test suite? Y/N Which one? ___________ - If Yes, what is the percentage of code covered by tests? ___ % - Do you use automatic acceptance tests? Y/N - Since when have you been using automatic tests? Since the start of the project / Since ___ years. 3. Refactoring: - Do you practice explicit code refactoring? Y/N - What is the percentage of developing effort for refactoring? ___% - Since when have you been using refactoring? Since the start of the project / Since ___ years. 4. Integration and releases: - How long does it take to produce the executable code starting from source code? ____ - Do you have a "working version" of the code, which is often updated by developers who add their code and run tests? Y/N 5. General: - Can you mention some other agile practices you are using in the project? For each of them, since when has it been used? - Practice: __________________________ Used since: ______ - Practice: __________________________ Used since: ______ - .... -- --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Sun Dec 24 20:05:31 2006 From: aahz at pythoncraft.com (Aahz) Date: Sun, 24 Dec 2006 11:05:31 -0800 Subject: [Python-Dev] Fwd: survey about the use of agile practices In-Reply-To: References: <7LAcRIZu.1166981065.5419330.s.uras@diee.unica.it> Message-ID: <20061224190530.GA23928@panix.com> On Sun, Dec 24, 2006, Guido van Rossum wrote: > > Anyone? It was discarded as probable spam by me due to the lack of a valid "To:" line. Do you have any particular reason for believing that it's real? -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "I support family values -- Addams family values" --www.nancybuttons.com From pedronis at openendsystems.com Sun Dec 24 20:19:06 2006 From: pedronis at openendsystems.com (Samuele Pedroni) Date: Sun, 24 Dec 2006 20:19:06 +0100 Subject: [Python-Dev] Fwd: survey about the use of agile practices In-Reply-To: <20061224190530.GA23928@panix.com> References: <7LAcRIZu.1166981065.5419330.s.uras@diee.unica.it> <20061224190530.GA23928@panix.com> Message-ID: <458ED2AA.5020507@openendsystems.com> Aahz wrote: > On Sun, Dec 24, 2006, Guido van Rossum wrote: > >> Anyone? >> > > It was discarded as probable spam by me due to the lack of a valid "To:" > line. Do you have any particular reason for believing that it's real? > http://agile.diee.unica.it/uras.html (it's in italian) seems to make this credible. FWIW. From fuzzyman at voidspace.org.uk Mon Dec 25 01:31:42 2006 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 25 Dec 2006 00:31:42 +0000 Subject: [Python-Dev] PropertyType Message-ID: <458F1BEE.6040606@voidspace.org.uk> Hello all, Could / should 'PropertyType' be added to the types module ? Michael Foord http://www.voidspace.org.uk/python/articles.shtml -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.409 / Virus Database: 268.15.26/600 - Release Date: 23/12/2006 From skip at pobox.com Mon Dec 25 05:44:51 2006 From: skip at pobox.com (skip at pobox.com) Date: Sun, 24 Dec 2006 22:44:51 -0600 Subject: [Python-Dev] PropertyType In-Reply-To: <458F1BEE.6040606@voidspace.org.uk> References: <458F1BEE.6040606@voidspace.org.uk> Message-ID: <17807.22339.529977.721525@montanaro.dyndns.org> Michael> Could / should 'PropertyType' be added to the types module ? I thought the types module was deprecated. Skip From guido at python.org Mon Dec 25 05:51:52 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 24 Dec 2006 20:51:52 -0800 Subject: [Python-Dev] Fwd: survey about the use of agile practices In-Reply-To: <20061224190530.GA23928@panix.com> References: <7LAcRIZu.1166981065.5419330.s.uras@diee.unica.it> <20061224190530.GA23928@panix.com> Message-ID: On 12/24/06, Aahz wrote: > On Sun, Dec 24, 2006, Guido van Rossum wrote: > > > > Anyone? > > It was discarded as probable spam by me due to the lack of a valid "To:" > line. Do you have any particular reason for believing that it's real? Even apart from the website that Samuele found, the content sounded genuine. Just not something I feel like dealing with (too many surveys). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Mon Dec 25 06:14:19 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 25 Dec 2006 00:14:19 -0500 Subject: [Python-Dev] PropertyType In-Reply-To: <17807.22339.529977.721525@montanaro.dyndns.org> References: <458F1BEE.6040606@voidspace.org.uk> <17807.22339.529977.721525@montanaro.dyndns.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Dec 24, 2006, at 11:44 PM, skip at pobox.com wrote: > > Michael> Could / should 'PropertyType' be added to the types > module ? > > I thought the types module was deprecated. Except that there are some types you can't (easily) get any other way. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRY9eMXEjvBPtnXfVAQJjIQP9Ei/KayIeuU3CAkPKsSFKw43v1sscMOP4 Y7BaUWbRlwptj4k4T2spm0leFneUoOl513B+Vyz/hZps9Zd2ltOo7NtBTqmGQ02P jAK6wcRkjgXJDjvZo2YWbabZLy97vCX1/GHGKbnLGlB6w6t3bI7xF/bB9Zo9OeKA jTS5Z6m+AEk= =OB2f -----END PGP SIGNATURE----- From python at rcn.com Mon Dec 25 06:39:18 2006 From: python at rcn.com (Raymond Hettinger) Date: Mon, 25 Dec 2006 00:39:18 -0500 Subject: [Python-Dev] PropertyType References: <458F1BEE.6040606@voidspace.org.uk> Message-ID: <001001c727e7$06172fe0$f915fea9@RaymondLaptop1> [Michael Foord] > Could / should 'PropertyType' be added to the types module ? PropertyType = property Yawn, Raymond From fredrik at pythonware.com Mon Dec 25 13:28:54 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Mon, 25 Dec 2006 13:28:54 +0100 Subject: [Python-Dev] Fwd: survey about the use of agile practices In-Reply-To: References: <7LAcRIZu.1166981065.5419330.s.uras@diee.unica.it> <20061224190530.GA23928@panix.com> Message-ID: Guido van Rossum wrote: > Even apart from the website that Samuele found, the content sounded > genuine. Just not something I feel like dealing with (too many > surveys). from what I can tell, it was blasted to everyone with a sourceforge account, which indicates that they're don't really care what projects or persons they get information from. even if it's a genuine survey, it's crap science. From ncoghlan at gmail.com Mon Dec 25 14:56:52 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 25 Dec 2006 23:56:52 +1000 Subject: [Python-Dev] Fwd: survey about the use of agile practices In-Reply-To: References: <7LAcRIZu.1166981065.5419330.s.uras@diee.unica.it> <20061224190530.GA23928@panix.com> Message-ID: <458FD8A4.5050906@gmail.com> Fredrik Lundh wrote: > Guido van Rossum wrote: > >> Even apart from the website that Samuele found, the content sounded >> genuine. Just not something I feel like dealing with (too many >> surveys). > > from what I can tell, it was blasted to everyone with a sourceforge > account, which indicates that they're don't really care what projects > or persons they get information from. even if it's a genuine survey, > it's crap science. I was going to respond to the survey (I try to help out that kind of research when I can), but far too many of my responses would have been of the "it depends on what you mean by term X" variety (e.g. what's the dividing line between a unit test and an acceptance test? Does the fact that essentially all Python developers are also Python users count as users being involved in project management? Is refactoring still explicit when you're doing it specifically to make something else you want to do easier?). The biggest single flaw I found is that the survey assumes the reader is familiar with the Agile Manifesto's particular brand of KoolAid, without providing any context or links to additional information. Given the flaws in the survey itself, I don't see how anything meaningful could be extracted from the responses (and collating emailed survey responses manually is going to be one heckuva task). Cheers, Nick. P.S. The *real* title of the thesis: An Investigation into the Response of Open Source Developers to Flawed Research Surveys };> -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From nnorwitz at gmail.com Tue Dec 26 07:13:48 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 25 Dec 2006 22:13:48 -0800 Subject: [Python-Dev] 2.5.1 plans Message-ID: I don't have a schedule in mind for 2.5.1 yet, however we should start preparing for it. The release will probably happen sometime in January if everyone is available. The branch has been pretty quiet, so I'm not expecting too many problems. Once we figure out a date for release I'll follow up here. If you have any bugs or patches that should be addressed prior to release, please assign them to me and bump the priority to 9. I'm not sure that there are any bugs which absolutely must be fixed before release, though there are a few that would be nice. Any other discussion for 2.5.1 necessary? n From ironfroggy at gmail.com Fri Dec 29 08:40:05 2006 From: ironfroggy at gmail.com (Calvin Spealman) Date: Fri, 29 Dec 2006 02:40:05 -0500 Subject: [Python-Dev] Cached Property Pattern Message-ID: <76fd5acf0612282340v7fc02032odd524c56e222133f@mail.gmail.com> A very common pattern in using properties is as follows: class Foo(object): @property def c(self): if not hasattr(self, '_c'): self._c = some_operation(self.a, self.b) return self._c Basically, the common usage is to only calculate a properties value once, and to reuse the result for every lookup after that. This is so common, that it seems most of my uses of property come in this form. To this end, should a cachedproperty builtin be included to do this automatically? Optionally, it might automatically include a default fset (change the cached value) and fdel (remove the cached value, to be regenerated next time). I will contribute code, if such a patch would be accepted. PS - It seems I am taking over the python-dev summaries, but the holidays have interfered with those duties! I apologize, and promise they will come shortly. -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/ From phd at phd.pp.ru Fri Dec 29 09:12:54 2006 From: phd at phd.pp.ru (Oleg Broytmann) Date: Fri, 29 Dec 2006 11:12:54 +0300 Subject: [Python-Dev] Cached Property Pattern In-Reply-To: <76fd5acf0612282340v7fc02032odd524c56e222133f@mail.gmail.com> References: <76fd5acf0612282340v7fc02032odd524c56e222133f@mail.gmail.com> Message-ID: <20061229081254.GB24148@phd.pp.ru> http://mail.python.org/pipermail/python-dev/2005-September/056782.html On Fri, Dec 29, 2006 at 02:40:05AM -0500, Calvin Spealman wrote: > To this end, should a cachedproperty builtin be included to do this The issue was discussed a year ago: http://mail.python.org/pipermail/python-dev/2005-September/056782.html http://mail.python.org/pipermail/python-dev/2005-October/057120.html Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From zhzhao at cisco.com Wed Dec 27 02:15:41 2006 From: zhzhao at cisco.com (Jenny Zhao (zhzhao)) Date: Tue, 26 Dec 2006 17:15:41 -0800 Subject: [Python-Dev] Looking for python SIP and MGCP stacks Message-ID: <1ECA23B17E8AB54C859FA0449B729F0E03428081@xmb-sjc-235.amer.cisco.com> Hi Python developers, I am using python to write a testing tools, currently this tool only supports skinny protocol. I am planning to add SIP and MGCP support as well, wondering if you have written these protocol stacks before which can be leveraged from. thanks Jenny -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20061226/5e88b86d/attachment.html From anthony at interlink.com.au Fri Dec 29 15:04:06 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat, 30 Dec 2006 01:04:06 +1100 Subject: [Python-Dev] Looking for python SIP and MGCP stacks In-Reply-To: <1ECA23B17E8AB54C859FA0449B729F0E03428081@xmb-sjc-235.amer.cisco.com> References: <1ECA23B17E8AB54C859FA0449B729F0E03428081@xmb-sjc-235.amer.cisco.com> Message-ID: <200612300104.08675.anthony@interlink.com.au> On Wednesday 27 December 2006 12:15, Jenny Zhao (zhzhao) wrote: > Hi Python developers, > > I am using python to write a testing tools, currently this tool > only supports skinny protocol. I am planning to add SIP and MGCP > support as well, wondering if you have written these protocol > stacks before which can be leveraged from. This should go to python-list at python.org (aka comp.lang.python), not this list. This list is for development _of_ python, not development _in_ python. Thanks, Anthony From ironfroggy at gmail.com Fri Dec 29 15:55:46 2006 From: ironfroggy at gmail.com (Calvin Spealman) Date: Fri, 29 Dec 2006 09:55:46 -0500 Subject: [Python-Dev] Cached Property Pattern In-Reply-To: <20061229081254.GB24148@phd.pp.ru> References: <76fd5acf0612282340v7fc02032odd524c56e222133f@mail.gmail.com> <20061229081254.GB24148@phd.pp.ru> Message-ID: <76fd5acf0612290655y22968021t28d1147329e3e819@mail.gmail.com> It may have been discussed before, but there does not seem to have been any resolution on the issue. Am I missing something or did the discussion just kind of stop, with no solution or agreement ever reached? In which case, reviving the question is not a bad idea, is it? On 12/29/06, Oleg Broytmann wrote: > http://mail.python.org/pipermail/python-dev/2005-September/056782.html > > On Fri, Dec 29, 2006 at 02:40:05AM -0500, Calvin Spealman wrote: > > To this end, should a cachedproperty builtin be included to do this > > The issue was discussed a year ago: > > http://mail.python.org/pipermail/python-dev/2005-September/056782.html > http://mail.python.org/pipermail/python-dev/2005-October/057120.html > > Oleg. > -- > Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com > -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/ From phd at phd.pp.ru Fri Dec 29 16:50:50 2006 From: phd at phd.pp.ru (Oleg Broytmann) Date: Fri, 29 Dec 2006 18:50:50 +0300 Subject: [Python-Dev] Cached Property Pattern In-Reply-To: <76fd5acf0612290655y22968021t28d1147329e3e819@mail.gmail.com> References: <76fd5acf0612282340v7fc02032odd524c56e222133f@mail.gmail.com> <20061229081254.GB24148@phd.pp.ru> <76fd5acf0612290655y22968021t28d1147329e3e819@mail.gmail.com> Message-ID: <20061229155050.GA5939@phd.pp.ru> On Fri, Dec 29, 2006 at 09:55:46AM -0500, Calvin Spealman wrote: > It may have been discussed before, but there does not seem to have > been any resolution on the issue. Am I missing something or did the > discussion just kind of stop, with no solution or agreement ever > reached? In which case, reviving the question is not a bad idea, is > it? I don't remember any resolution. I think submitting a small module to the patch tracker would be the simplest way to revive the discussion. > On 12/29/06, Oleg Broytmann wrote: > > http://mail.python.org/pipermail/python-dev/2005-September/056782.html > > http://mail.python.org/pipermail/python-dev/2005-October/057120.html Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From fdrake at acm.org Fri Dec 29 20:04:50 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri, 29 Dec 2006 14:04:50 -0500 Subject: [Python-Dev] Cached Property Pattern In-Reply-To: <20061229155050.GA5939@phd.pp.ru> References: <76fd5acf0612282340v7fc02032odd524c56e222133f@mail.gmail.com> <76fd5acf0612290655y22968021t28d1147329e3e819@mail.gmail.com> <20061229155050.GA5939@phd.pp.ru> Message-ID: <200612291404.50752.fdrake@acm.org> On Friday 29 December 2006 10:50, Oleg Broytmann wrote: > I don't remember any resolution. I think submitting a small module to > the patch tracker would be the simplest way to revive the discussion. We have a handful of interesting descriptors we use for Zope 3 development: http://svn.zope.org/Zope3/trunk/src/zope/cachedescriptors/ I find I use the Lazy property quite a bit in short-lived contexts (a single web request); this sounds very much like what's being described here. The readproperty is probably better when the computation isn't so expensive and the value may need to be re-computed frequently anyway. -Fred -- Fred L. Drake, Jr. From jeremy at alum.mit.edu Fri Dec 29 20:29:58 2006 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Fri, 29 Dec 2006 14:29:58 -0500 Subject: [Python-Dev] classes and cell variables question In-Reply-To: <1d85506f0612191359i7831b097s9a1dcbe1cdaa1e49@mail.gmail.com> References: <1d85506f0612191359i7831b097s9a1dcbe1cdaa1e49@mail.gmail.com> Message-ID: On 12/19/06, tomer filiba wrote: > to my understanding of the object model, the code of snippet 1 > and snippet 2 should be equivalent. a class is just a "special function" > that returns its locals automatically and passes them to the metaclass > constructor: > > --- snippet 1 --- > class foo(object): > x = 5 > def f(self): > print "f", x > > --- snippet 2 --- > def bar(): > x = 5 > def g(self): > print "g", x > return locals() > barcls = type("barcls", (object,), bar()) > > but there's one big difference. classes don't create cell variables > to hold bound external variables. the "bar" version works, because > "x" is a bound cell variable, but the "foo" version fails, as it attempts > to access "x" as a global. Others have explained the rationale for this design choice, and the PEP is a good source for what we were thinking at the time. I thought it was worth adding that a) class variables can never be access as free variables, as your example foo() shows, and b) you could access a variable bound in a function scope in a method. Example of b): def spam(): x = 5 # provides the binding for the free variable x in methods of eggs class eggs(object): x = 6 # irrelevant for purposes of resolving the free variable in the method spam def spam(self): # can't have too much spam return x return eggs spam()().spam() should return 5. Jeremy > > .>>> barcls().g() > g 5 > > .>>> foo().f() > f > Traceback (most recent call last): > File "", line 1, in > File "", line 4, in f > NameError: global name 'x' is not defined > > for reference, i attached the code of all four functions below. > > my question is, how come classes don't create cell variables, like > normal functions? was this done on purpose? does it have to > do with inheritance? if so, what's wrong with my "bar" version? > > > [1] > # code of class foo > ############################################################ > # 2 0 LOAD_NAME 0 (__name__) > # 3 STORE_NAME 1 (__module__) > # 3 6 LOAD_CONST 0 (5) > # 9 STORE_NAME 2 (x) > # > # 4 12 LOAD_CONST 1 ( 009E5608, file "", line 4>) > # 15 MAKE_FUNCTION 0 > # 18 STORE_NAME 3 (f) > # 21 LOAD_LOCALS > # 22 RETURN_VALUE > > [2] > # code of foo.f: > ############################################################ > # 5 0 LOAD_CONST 1 ('f') > # 3 PRINT_ITEM > # 4 LOAD_GLOBAL 0 (x) > # 7 PRINT_ITEM > # 8 PRINT_NEWLINE > # 9 LOAD_CONST 0 (None) > # 12 RETURN_VALUE > > [3] > # code of bar: > ############################################################ > # 2 0 LOAD_CONST 1 (5) > # 3 STORE_DEREF 0 (x) > # > # 3 6 LOAD_CLOSURE 0 (x) > # 9 BUILD_TUPLE 1 > # 12 LOAD_CONST 2 ( 009F6698, file "", line 3>) > # 15 MAKE_CLOSURE 0 > # 18 STORE_FAST 0 (g) > # > # 5 21 LOAD_GLOBAL 0 (locals) > # 24 CALL_FUNCTION 0 > # 27 RETURN_VALUE > > [4] > # code of bar.g: > ############################################################ > # 4 0 LOAD_CONST 1 ('g') > # 3 PRINT_ITEM > # 4 LOAD_DEREF 0 (x) > # 7 PRINT_ITEM > # 8 PRINT_NEWLINE > # 9 LOAD_CONST 0 (None) > # 12 RETURN_VALUE > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu > From tomerfiliba at gmail.com Fri Dec 29 21:30:23 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Fri, 29 Dec 2006 22:30:23 +0200 Subject: [Python-Dev] classes and cell variables question In-Reply-To: References: <1d85506f0612191359i7831b097s9a1dcbe1cdaa1e49@mail.gmail.com> Message-ID: <1d85506f0612291230i6e1ed83fg55bd8a6ce5b9c5db@mail.gmail.com> On 12/29/06, Jeremy Hylton wrote: > def spam(): > x = 5 > class eggs(object): > x = 6 > def spam(self): > return x > return eggs > > spam()().spam() should return 5. > the question that arises is -- is this what we wanted? if i had to read such code, where i can't (easily) tell where "x" is bound to, i wouldn't be content, to say the least. -tomer From guido at python.org Fri Dec 29 22:23:14 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 29 Dec 2006 13:23:14 -0800 Subject: [Python-Dev] classes and cell variables question In-Reply-To: <1d85506f0612291230i6e1ed83fg55bd8a6ce5b9c5db@mail.gmail.com> References: <1d85506f0612191359i7831b097s9a1dcbe1cdaa1e49@mail.gmail.com> <1d85506f0612291230i6e1ed83fg55bd8a6ce5b9c5db@mail.gmail.com> Message-ID: On 12/29/06, tomer filiba wrote: > On 12/29/06, Jeremy Hylton wrote: > > def spam(): > > x = 5 > > class eggs(object): > > x = 6 > > def spam(self): > > return x > > return eggs > > > > spam()().spam() should return 5. > > > > the question that arises is -- is this what we wanted? > if i had to read such code, where i can't (easily) tell where "x" > is bound to, i wouldn't be content, to say the least. It's obfuscated code. But there is no doubt that these are the semantics that we wanted, and still want. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Fri Dec 29 22:41:24 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 29 Dec 2006 22:41:24 +0100 Subject: [Python-Dev] [Python-checkins] r53110 - python/trunk/Lib/mailbox.py In-Reply-To: <200612230159.08151.fdrake@acm.org> References: <20061220194820.ED8DD1E4008@bag.python.org> <20061221135343.GA5855@localhost.localdomain> <458AED2E.5080709@v.loewis.de> <200612230159.08151.fdrake@acm.org> Message-ID: <45958B84.60501@v.loewis.de> Fred L. Drake, Jr. schrieb: > Speaking strictly for myself: I don't think I *have* to use them, but I do > prefer to use them because I don't like magic constants that affect what a > function does in code; I'd rather have a named constant for readability's > sake. Maybe I just can't keep enough in my head, but I don't find I actually > use seek() enough to remember what the numeric values mean with checking the > reference documentation. But you *can* remember the symbolic names (plus you can remember what module they are in, in what Python version)? I'd have to consult the documentation for either; help(file.seek) gives the numeric values. Regards, Martin From guido at python.org Fri Dec 29 22:55:43 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 29 Dec 2006 13:55:43 -0800 Subject: [Python-Dev] [Python-checkins] r53110 - python/trunk/Lib/mailbox.py In-Reply-To: <45958B84.60501@v.loewis.de> References: <20061220194820.ED8DD1E4008@bag.python.org> <20061221135343.GA5855@localhost.localdomain> <458AED2E.5080709@v.loewis.de> <200612230159.08151.fdrake@acm.org> <45958B84.60501@v.loewis.de> Message-ID: On 12/29/06, "Martin v. L?wis" wrote: > Fred L. Drake, Jr. schrieb: > > Speaking strictly for myself: I don't think I *have* to use them, but I do > > prefer to use them because I don't like magic constants that affect what a > > function does in code; I'd rather have a named constant for readability's > > sake. Maybe I just can't keep enough in my head, but I don't find I actually > > use seek() enough to remember what the numeric values mean with checking the > > reference documentation. > > But you *can* remember the symbolic names (plus you can remember what > module they are in, in what Python version)? I'd have to consult the > documentation for either; help(file.seek) gives the numeric values. I'm with Martin here. But my main objection to suggesting that these constants ought to be used is that open() is a built-in but you would have to import os to be able to call the seek method on the object it returns. If we want to make the seek API more 21st century, why not use keyword arguments? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake at acm.org Fri Dec 29 23:03:22 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri, 29 Dec 2006 17:03:22 -0500 Subject: [Python-Dev] [Python-checkins] r53110 - python/trunk/Lib/mailbox.py In-Reply-To: References: <20061220194820.ED8DD1E4008@bag.python.org> <45958B84.60501@v.loewis.de> Message-ID: <200612291703.22535.fdrake@acm.org> On Friday 29 December 2006 16:55, Guido van Rossum wrote: > If we want to make the seek API more 21st century, why not use keyword > arguments? I'd prefer that myself. I'm not advocating the constants as a way to go forward, but was simply expressing a preference for the named constant over a magic number in code. -Fred -- Fred L. Drake, Jr. From barry at python.org Fri Dec 29 23:05:34 2006 From: barry at python.org (Barry Warsaw) Date: Fri, 29 Dec 2006 17:05:34 -0500 Subject: [Python-Dev] [Python-checkins] r53110 - python/trunk/Lib/mailbox.py In-Reply-To: References: <20061220194820.ED8DD1E4008@bag.python.org> <20061221135343.GA5855@localhost.localdomain> <458AED2E.5080709@v.loewis.de> <200612230159.08151.fdrake@acm.org> <45958B84.60501@v.loewis.de> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Dec 29, 2006, at 4:55 PM, Guido van Rossum wrote: > But my main objection to suggesting that these constants ought to be > used is that open() is a built-in but you would have to import os to > be able to call the seek method on the object it returns. > > If we want to make the seek API more 21st century, why not use keyword > arguments? Or put the constants on class attributes. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRZWRM3EjvBPtnXfVAQLWdwQAmrt5ncAhFIxCK2Tm72g90P9qckVrkdAt odvZ+fw8RFrKs3MUpsP2HtfHWHrouC6SHQCXoU63msfgy52vO4M9rG3NLN5Hi0KK tU4J9pkSqqi+Jfu4LvCs+ouURqxTbGicyMBUuw54hQWqg0bxvYvjEyuC1NFltcV6 HJ2qElSTOw0= =LAq1 -----END PGP SIGNATURE----- From martin at v.loewis.de Sat Dec 30 02:19:12 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 30 Dec 2006 02:19:12 +0100 Subject: [Python-Dev] multiple interpreters and extension modules In-Reply-To: <200612221242.11576.jeremy.kloth@4suite.org> References: <200612221242.11576.jeremy.kloth@4suite.org> Message-ID: <4595BE90.1030700@v.loewis.de> Jeremy Kloth schrieb: > 1) is subclassing Python classes in C as a static type supported? Even if they > would be declared on the heap, they would be bound to the first loaded Python > class. As you found out: no, this isn't supported. To work around, you can wrap the extension module with Python module which invokes an explicit module initialization function; you then need to lookup the subclasses dynamically (going through thread_state->interp->modules). > As to question 2, I'm in the process of prototyping that approach to see if it > is feasible in our code and if so, (and people find it useful) I'll attempt > to write a PEP for that feature. I'd like to see the module initialization/finalization be redone for Python 3, in particular with explicit finalization procedures. If Python 3 still supports multiple interpreters, this aspect should be taken into account. Whether there is any chance to add the PEP to 2.x, I don't know - it needs to be written first. Regards, Martin From martin at v.loewis.de Sat Dec 30 02:23:43 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 30 Dec 2006 02:23:43 +0100 Subject: [Python-Dev] multiple interpreters and extension modules In-Reply-To: <200612222009.56915.jeremy.kloth@4suite.org> References: <200612221242.11576.jeremy.kloth@4suite.org> <200612222009.56915.jeremy.kloth@4suite.org> Message-ID: <4595BF9F.80009@v.loewis.de> Jeremy Kloth schrieb: >> I think you understand exactly what is happening. It is happening for >> good reasons. Rather than asking for a change in semantics, I >> recommend that you deal with it, either in your Python code, or in >> your extension. It's not likely to change. > > I don't believe I was asking for a change in semantics, rather an additional, > optional interface for extension module writers. And that *is* a change in semantics. The algorithm run by the interpreter on module startup will be different from what it is now, i.e. it is a behavior (semantic) change. > I'll add here that it has been brought up here before that extension module > finalization is a feature that would be appreciated. With that, it is not > that far to add support for initialization/finalization for each interpreter. > That is, of course, using a DllMain-like solution. I wouldn't like to see a DllMain-like solution. Instead, there should be a function vector, and a specification which function is called in what cases. There should be an entry point that typically just returns a pointer to this module vtable. > With that approach in mind, I will be making changes so 4Suite will work in a > production mod_python deployment (where the aforementioned error occurred). > When that works, I'll come back with a proper PEP *and* patches against > Python SVN to support its use. I hope no one was thinking I wanted someone > else to do the work. I was actually going to, for several years now. Please do write the PEP before making the implementation. Regards, Martin From martin at v.loewis.de Sat Dec 30 02:33:15 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 30 Dec 2006 02:33:15 +0100 Subject: [Python-Dev] Possible platforms to drop in 2.6 In-Reply-To: References: Message-ID: <4595C1DB.1090304@v.loewis.de> Brett Cannon schrieb: > I originally posted this list to python-3000 since I figured we could > be more aggressive with Py3K, but Guido said I should move it over > here and just be aggressive in 2.6. Please follow PEP 11 in doing so. This means you cannot remove the code in Python 2.6, only break the build with an error message. Actual removal would be deferred to 2.7. > So, here are the platforms I figured we should drop: > > * AtheOS > * BeOS In both cases, the last maintainer should be contacted before the platform is unsupported. > I had SunoS 5 but Ronald Oussoren said that is actually Solaris > through version 9, so I took that off. It's actually *all* Solaris versions (up to 11). Dropping support for 5.6 (Solaris 2.6) and earlier may be an option; we have some special-cased code for 5.6. > Several people have questioned AtheOS, but considering the site for > the OS has not been updated since 2002 and it was a niche OS to begin > with I doubt we really need the support. IMO, that should really depend on active maintenance. Somebody should confirm that Python 2.5 still compiles out of the box on that system, and, if not, volunteer to fix it. If nobody does, we can remove the code in 2.7. > And I listed FreeBSD 2 as a drop since FreeBSD 3 seemed to have been > released in 1999. But if most users have upgraded by now (release 6 > is the most current) then we could consider dropping 3 as well. This really should use the PEP 11 procedure: let configure fail (early) on the system, and then remove support if nobody complains (in 2.7 and 3k). Regards, Martin From brett at python.org Sat Dec 30 04:37:01 2006 From: brett at python.org (Brett Cannon) Date: Fri, 29 Dec 2006 19:37:01 -0800 Subject: [Python-Dev] Possible platforms to drop in 2.6 In-Reply-To: <4595C1DB.1090304@v.loewis.de> References: <4595C1DB.1090304@v.loewis.de> Message-ID: On 12/29/06, "Martin v. L?wis" wrote: > > Brett Cannon schrieb: > > I originally posted this list to python-3000 since I figured we could > > be more aggressive with Py3K, but Guido said I should move it over > > here and just be aggressive in 2.6. > > Please follow PEP 11 in doing so. This means you cannot remove the code > in Python 2.6, only break the build with an error message. Actual > removal would be deferred to 2.7. I wasn't planning on skipping the procedures in PEP 11. I just wanted to get the list of possible platforms to eliminate out there for people to comment on. > So, here are the platforms I figured we should drop: > > > > * AtheOS > > * BeOS > > In both cases, the last maintainer should be contacted before the > platform is unsupported. I guess I can go off the emails listed in README and Misc/BeOS-NOTES, although I would hope that any maintainer would watch python-dev in some fashion. Is there an official list of maintainers? If not perhaps there should be a PEP listing who maintains what platforms. > I had SunoS 5 but Ronald Oussoren said that is actually Solaris > > through version 9, so I took that off. > > It's actually *all* Solaris versions (up to 11). > Dropping support for 5.6 (Solaris 2.6) and earlier may be > an option; we have some special-cased code for 5.6. OK. I don't have a Solaris box so someone else might need to help with that. > Several people have questioned AtheOS, but considering the site for > > the OS has not been updated since 2002 and it was a niche OS to begin > > with I doubt we really need the support. > > IMO, that should really depend on active maintenance. Somebody should > confirm that Python 2.5 still compiles out of the box on that system, > and, if not, volunteer to fix it. If nobody does, we can remove the > code in 2.7. Yep. Guess that goes along with contacting the maintainers. > And I listed FreeBSD 2 as a drop since FreeBSD 3 seemed to have been > > released in 1999. But if most users have upgraded by now (release 6 > > is the most current) then we could consider dropping 3 as well. > > This really should use the PEP 11 procedure: let configure fail > (early) on the system, and then remove support if nobody complains > (in 2.7 and 3k). Sounds reasonable. Hopefully it would catch people early in the alpha stage to deal with this. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20061229/f6b44b52/attachment.html From kbk at shore.net Sat Dec 30 06:41:03 2006 From: kbk at shore.net (Kurt B. Kaiser) Date: Sat, 30 Dec 2006 00:41:03 -0500 (EST) Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200612300541.kBU5f359027319@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 413 open ( -7) / 3521 closed (+11) / 3934 total ( +4) Bugs : 946 open ( +2) / 6400 closed ( +9) / 7346 total (+11) RFE : 248 open ( -1) / 246 closed ( +1) / 494 total ( +0) New / Reopened Patches ______________________ Auto-completion list placement (2006-12-23) http://python.org/sf/1621265 opened by Tal Einat normalize namespace from minidom (2006-12-23) http://python.org/sf/1621421 opened by Paul Pacheco Allow __class __ assignment for classes with __slots__ (2006-12-28) http://python.org/sf/1623563 opened by TH fast subclasses of builtin types (2006-12-28) http://python.org/sf/1624059 opened by Neal Norwitz Patches Closed ______________ add direct access to MD5 compression function to md5 module (2003-03-16) http://python.org/sf/704676 closed by akuchling pty.fork() compatibility code wrong? (2003-08-04) http://python.org/sf/783050 closed by akuchling for i in range(N) optimization (2003-05-15) http://python.org/sf/738094 closed by gvanrossum Add --remove-source option to setup.py (2003-08-22) http://python.org/sf/793070 closed by akuchling SimpleHTTPServer directory-indexing "bug" fix (2003-10-21) http://python.org/sf/827559 closed by akuchling Remove unncessary NLST from ftp transfers (2004-10-12) http://python.org/sf/1045783 closed by akuchling add Bunch type to collections module (2005-01-02) http://python.org/sf/1094542 closed by akuchling HMAC hexdigest and general review (2005-04-13) http://python.org/sf/1182394 closed by akuchling tarfile.py: ExFileObject\'s tell() is wrong after readline() (2005-06-30) http://python.org/sf/1230446 closed by gustaebel tarfile: fix for bug #1257255 (2005-08-17) http://python.org/sf/1262036 closed by gustaebel Patch for 1496501 tarfile opener order (2006-06-10) http://python.org/sf/1504073 closed by gustaebel New / Reopened Bugs ___________________ minor inconsistency in socket.close (2006-12-22) http://python.org/sf/1620945 opened by Jonathan Ellis IDLE crashes on OS X 10.4 when "Preferences" selected (2006-12-23) http://python.org/sf/1621111 opened by Mick L random import works? (2006-12-23) CLOSED http://python.org/sf/1621367 opened by Msword this module (Zen of Python) docs list broken URL (2006-12-24) http://python.org/sf/1621660 opened by Calvin Spealman Tcl/Tk auto-expanding window (2006-12-25) http://python.org/sf/1622010 opened by Fabian_M null bytes in docstrings (2006-12-26) http://python.org/sf/1622533 opened by Fredrik Lundh language reference index links are broken (2006-12-26) http://python.org/sf/1622664 opened by Drew Perttula Exception when compressing certain data with bz2 (2006-12-27) http://python.org/sf/1622896 opened by Alex Gontmakher preferred diff format should be mentioned as "unified". (2006-12-27) http://python.org/sf/1623153 opened by Raghuram Devarakonda module docstring for subprocess is wrong (2006-12-28) CLOSED http://python.org/sf/1623890 opened by Neal Norwitz updating a set in a list of sets will update all of them (2006-12-29) CLOSED http://python.org/sf/1624534 opened by Amir Reza Khosroshahi webbrowser.open_new() suggestion (2006-12-30) http://python.org/sf/1624674 opened by Imre P?ntek Bugs Closed ___________ gethostbyaddr lag (2002-07-19) http://python.org/sf/583975 closed by nnorwitz CGIHTTPServer does not handle scripts in sub-dirs (2003-05-13) http://python.org/sf/737202 closed by akuchling MacOS9: test_uu fails (2003-07-23) http://python.org/sf/776202 closed by akuchling Mode argument of dumbdbm does not work (2003-09-07) http://python.org/sf/802128 closed by akuchling random import works? (2006-12-23) http://python.org/sf/1621367 closed by rhettinger tarfile local name is local, should be abspath (2005-08-12) http://python.org/sf/1257255 closed by gustaebel Dictionary ordering docs are too unclear of dangers (2006-12-09) http://python.org/sf/1612113 closed by sf-robot logging %(module)s reporting wrong modules (2006-11-29) http://python.org/sf/1605110 closed by sf-robot array.array borks on deepcopy (2006-08-24) http://python.org/sf/1545837 closed by twouters tarfile.py: dict order dependency (2006-05-28) http://python.org/sf/1496501 closed by gustaebel module docstring for subprocess is wrong (2006-12-28) http://python.org/sf/1623890 closed by nnorwitz updating a set in a list of sets will update all of them (2006-12-29) http://python.org/sf/1624534 deleted by amir_reza Immediate Crash on Open (2006-11-20) http://python.org/sf/1599931 closed by sf-robot python-logging compatability with Zope. (2006-12-12) http://python.org/sf/1614460 closed by sf-robot RFE Closed __________ Promote list comprehensions to Language Integrated Query? (2006-12-19) http://python.org/sf/1618676 closed by jettlogic From martin at v.loewis.de Sat Dec 30 10:21:55 2006 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 30 Dec 2006 10:21:55 +0100 Subject: [Python-Dev] Possible platforms to drop in 2.6 In-Reply-To: References: <4595C1DB.1090304@v.loewis.de> Message-ID: <45962FB3.3030708@v.loewis.de> Brett Cannon schrieb: > > * AtheOS > > * BeOS > > In both cases, the last maintainer should be contacted before the > platform is unsupported. > > > I guess I can go off the emails listed in README and Misc/BeOS-NOTES, > although I would hope that any maintainer would watch python-dev in some > fashion. Is there an official list of maintainers? Not that I know of. The AtheOS port was contributed through patch #488073, by Octavian Cerna (tavyc). It seems that the primary contributor for the BeOS port was Chris Herborth (chrish at pobox.com). Regards, Martin From martin at v.loewis.de Sat Dec 30 16:17:47 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 30 Dec 2006 16:17:47 +0100 Subject: [Python-Dev] Non-blocking (asynchronous) timer without thread? In-Reply-To: <458CB8A0.9030907@gmail.com> References: <458CB8A0.9030907@gmail.com> Message-ID: <4596831B.3090200@v.loewis.de> Evgeniy Khramtsov schrieb: > Is there any plans to implement non-blocking timer like a > threading.Timer() but without thread? On Unix, you can use signal.alarm. Regards, Martin