From guido at python.org Tue Nov 1 00:36:09 2005 From: guido at python.org (Guido van Rossum) Date: Mon, 31 Oct 2005 16:36:09 -0700 Subject: [Python-Dev] Freezing the CVS on Oct 26 for SVN switchover In-Reply-To: <43654858.9020108@v.loewis.de> References: <435BC27C.1010503@v.loewis.de> <2mbr1g6loh.fsf@starship.python.net> <17248.52771.225830.484931@montanaro.dyndns.org> <43610C36.2030500@v.loewis.de> <1f7befae0510281829n20ae2936pbc9f923da807bf6a@mail.gmail.com> <17252.50390.256221.4882@montanaro.dyndns.org> <17252.59653.792906.582288@montanaro.dyndns.org> <43654858.9020108@v.loewis.de> Message-ID: Help! What's the magic to get $Revision$ and $Date$ to be expanded upon checkin? Comparing pep-0352.txt and pep-0343.txt, I noticed that the latter has the svn revision and date in the headers, while the former still has Brett's original revision 1.5 and a date somewhere in June. I tried to fix this by rewriting the fields as $Revision$ and $Date$ but that doesn't seem to make a difference. Googling for this is a bit tricky because Google collapses $Revision and Revision, which makes any query for svn and $Revision rather non-specific. :-( It's also not yet in our Wiki. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.peters at gmail.com Tue Nov 1 00:48:44 2005 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 31 Oct 2005 18:48:44 -0500 Subject: [Python-Dev] Freezing the CVS on Oct 26 for SVN switchover In-Reply-To: References: <435BC27C.1010503@v.loewis.de> <2mbr1g6loh.fsf@starship.python.net> <17248.52771.225830.484931@montanaro.dyndns.org> <43610C36.2030500@v.loewis.de> <1f7befae0510281829n20ae2936pbc9f923da807bf6a@mail.gmail.com> <17252.50390.256221.4882@montanaro.dyndns.org> <17252.59653.792906.582288@montanaro.dyndns.org> <43654858.9020108@v.loewis.de> Message-ID: <1f7befae0510311548v34da0695jc38e0a5c831256c8@mail.gmail.com> [Guido] > Help! > > What's the magic to get $Revision$ and $Date$ to be expanded upon > checkin? Comparing pep-0352.txt and pep-0343.txt, I noticed that the > latter has the svn revision and date in the headers, while the former > still has Brett's original revision 1.5 and a date somewhere in June. > I tried to fix this by rewriting the fields as $Revision$ and $Date$ > but that doesn't seem to make a difference. > > Googling for this is a bit tricky because Google collapses $Revision > and Revision, which makes any query for svn and $Revision rather > non-specific. :-( It's also not yet in our Wiki. You have to set the `svn:keywords` property on each file for which you want these kinds of expansions: http://svnbook.red-bean.com/en/1.0/ch07s02.html#svn-ch-7-sect-2.3.4 Use svn propedit svn:keywords path/to/file to set that property to what you want. Looking at your examples, C:\Code>svn proplist -v http://svn.python.org/projects/peps/trunk/pep-0343.txt Properties on 'http://svn.python.org/projects/peps/trunk/pep-0343.txt': svn:keywords : Author Date Id Revision svn:eol-style : native So that has svn:keywords set, and expansion occurs. OTOH, C:\Code>svn proplist -v http://svn.python.org/projects/peps/trunk/pep-0352.txt Nada -- that one doesn't even have svn:eol-style set. See http://wiki.python.org/moin/CvsToSvn section "File Modes" for how to convince SVN to automatically set the properties you want on new files you commit (unfortunately, each developer has to do this in their own SVN config file). From pinard at iro.umontreal.ca Tue Nov 1 00:50:00 2005 From: pinard at iro.umontreal.ca (=?iso-8859-1?Q?Fran=E7ois?= Pinard) Date: Mon, 31 Oct 2005 18:50:00 -0500 Subject: [Python-Dev] Freezing the CVS on Oct 26 for SVN switchover In-Reply-To: References: <435BC27C.1010503@v.loewis.de> <2mbr1g6loh.fsf@starship.python.net> <17248.52771.225830.484931@montanaro.dyndns.org> <43610C36.2030500@v.loewis.de> <1f7befae0510281829n20ae2936pbc9f923da807bf6a@mail.gmail.com> <17252.50390.256221.4882@montanaro.dyndns.org> <17252.59653.792906.582288@montanaro.dyndns.org> <43654858.9020108@v.loewis.de> Message-ID: <20051031235000.GA14812@alcyon.progiciels-bpi.ca> [Guido van Rossum] >What's the magic to get $Revision$ and $Date$ to be expanded upon >checkin? Expansion does not occur on checkin, but on checkout, and even then, only in your copy -- that one you see (the internal Subversion copy is untouched). You have to edit a property for the file where you want substitutions. That property is named "svn:keywords" and its value decides which kind of substitution you want to allow. This is all theory for me, I never used them. -- Fran?ois Pinard http://pinard.progiciels-bpi.ca From gherron at islandtraining.com Tue Nov 1 00:54:46 2005 From: gherron at islandtraining.com (Gary Herron) Date: Mon, 31 Oct 2005 15:54:46 -0800 Subject: [Python-Dev] Freezing the CVS on Oct 26 for SVN switchover In-Reply-To: References: <435BC27C.1010503@v.loewis.de> <2mbr1g6loh.fsf@starship.python.net> <17248.52771.225830.484931@montanaro.dyndns.org> <43610C36.2030500@v.loewis.de> <1f7befae0510281829n20ae2936pbc9f923da807bf6a@mail.gmail.com> <17252.50390.256221.4882@montanaro.dyndns.org> <17252.59653.792906.582288@montanaro.dyndns.org> <43654858.9020108@v.loewis.de> Message-ID: <4366AEC6.2000803@islandtraining.com> Guido van Rossum wrote: >Help! > >What's the magic to get $Revision$ and $Date$ to be expanded upon >checkin? Comparing pep-0352.txt and pep-0343.txt, I noticed that the >latter has the svn revision and date in the headers, while the former >still has Brett's original revision 1.5 and a date somewhere in June. >I tried to fix this by rewriting the fields as $Revision$ and $Date$ >but that doesn't seem to make a difference. > >Googling for this is a bit tricky because Google collapses $Revision >and Revision, which makes any query for svn and $Revision rather >non-specific. :-( It's also not yet in our Wiki. > > It's an svn property associated with the file. The property name is svn:keywords, and the value is a space separated list of keywords you'd like to have substituted. Like this: svn propset svn:keywords "Date Revision" ...file list... The list of keywords it will handle is LastChangedDate (or Date) LastChangedRevision (or Revision or Rev) LastChangedBy (or Author) HeadURL (or URL) Id Gary Herron From greg.ewing at canterbury.ac.nz Tue Nov 1 02:03:25 2005 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 01 Nov 2005 14:03:25 +1300 Subject: [Python-Dev] Divorcing str and unicode (no more implicit conversions). In-Reply-To: <20051031022554.GA20255@alcyon.progiciels-bpi.ca> References: <50862ebd0510271721i77c1ebb4x6bcb39a4756c3a99@mail.gmail.com> <4362A44F.9010506@v.loewis.de> <20051029110331.D5AA.ISHIMOTO@gembook.org> <4363395A.3040606@v.loewis.de> <1130589142.5945.11.camel@fsol> <43638BC0.40108@v.loewis.de> <20051031022554.GA20255@alcyon.progiciels-bpi.ca> Message-ID: <4366BEDD.9020100@canterbury.ac.nz> Fran?ois Pinard wrote: > All development is done in house by French people. All documentation, > external or internal, comments, identifier and function names, > everything is in French. There's nothing stopping you from creating your own Frenchified version of Python that lets you use all the characters you want, for your own in-house use. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing at canterbury.ac.nz +--------------------------------------+ From greg.ewing at canterbury.ac.nz Tue Nov 1 02:24:11 2005 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 01 Nov 2005 14:24:11 +1300 Subject: [Python-Dev] a different kind of reduce... In-Reply-To: <8393fff0510311113p63bc194ak88580f84a25b1a1a@mail.gmail.com> References: <8393fff0510311113p63bc194ak88580f84a25b1a1a@mail.gmail.com> Message-ID: <4366C3BB.3010407@canterbury.ac.nz> Martin Blais wrote: > I'm always--literally every time-- looking for a more functional form, > something that would be like this: > > # apply dirname() 3 times on its results, initializing with p > ... = repapply(dirname, 3, p) Maybe ** should be defined for functions so that you could do things like up3levels = dirname ** 3 -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing at canterbury.ac.nz +--------------------------------------+ From pinard at iro.umontreal.ca Tue Nov 1 03:51:15 2005 From: pinard at iro.umontreal.ca (=?iso-8859-1?Q?Fran=E7ois?= Pinard) Date: Mon, 31 Oct 2005 21:51:15 -0500 Subject: [Python-Dev] Divorcing str and unicode (no more implicit conversions). In-Reply-To: <4366BEDD.9020100@canterbury.ac.nz> References: <50862ebd0510271721i77c1ebb4x6bcb39a4756c3a99@mail.gmail.com> <4362A44F.9010506@v.loewis.de> <20051029110331.D5AA.ISHIMOTO@gembook.org> <4363395A.3040606@v.loewis.de> <1130589142.5945.11.camel@fsol> <43638BC0.40108@v.loewis.de> <20051031022554.GA20255@alcyon.progiciels-bpi.ca> <4366BEDD.9020100@canterbury.ac.nz> Message-ID: <20051101025115.GA18573@alcyon.progiciels-bpi.ca> [Greg Ewing] >> All development is done in house by French people. All documentation, >> external or internal, comments, identifier and function names, >> everything is in French. > There's nothing stopping you from creating your own Frenchified > version of Python that lets you use all the characters you want, for > your own in-house use. No doubt that we, you and me and everybody, could all have our own little version of Python. :-) To tell all the truth, the very topic of your suggestion has already been discussed in-house already, and the decision has been to stick to Python mainstream. We could not justify to our administration that we start modifying our sources, in such a way that we ought to invest maintainance each time a new Python version appears, forever. On the other hand, we may reasonably guess that many people in this world would love being as comfortable as possible using Python, while naming identifiers naturally. It is not so unreasonable that we keep some _hope_ that Guido will soon choose to help us all, not only me. -- Fran?ois Pinard http://pinard.progiciels-bpi.ca From amk at amk.ca Tue Nov 1 15:35:05 2005 From: amk at amk.ca (A.M. Kuchling) Date: Tue, 1 Nov 2005 09:35:05 -0500 Subject: [Python-Dev] python-dev sprint at PyCon Message-ID: <20051101143505.GE14719@rogue.amk.ca> Every PyCon has featured a python-dev sprint. For the past few years, hacking on the AST branch has been a tradition, but we'll have to come up with something new for this year's conference (in Dallas Texas; sprints will be Monday Feb. 27 through Thursday March 2). According to Anthony's release plan, a first alpha of 2.5 would be released in March, hence after PyCon and the sprints. We should discuss possible tasks for a python-dev sprint. What could we do? When the discussion is over, someone should update the wiki page with whatever tasks are suggested: . --amk From dave at boost-consulting.com Tue Nov 1 17:25:23 2005 From: dave at boost-consulting.com (David Abrahams) Date: Tue, 01 Nov 2005 11:25:23 -0500 Subject: [Python-Dev] [C++-sig] GCC version compatibility References: <42CDA654.2080106@v.loewis.de> <20050708072807.GC3581@lap200.cdc.informatik.tu-darmstadt.de> <42CEF948.3010908@v.loewis.de> <20050709102010.GA3836@lap200.cdc.informatik.tu-darmstadt.de> <42D0D215.9000708@v.loewis.de> <20050710125458.GA3587@lap200.cdc.informatik.tu-darmstadt.de> <42D15DB2.3020300@v.loewis.de> <20050716101357.GC3607@lap200.cdc.informatik.tu-darmstadt.de> <20051012120917.GA11058@lap200.cdc.informatik.tu-darmstadt.de> Message-ID: Christoph Ludwig writes: > Hi, > > this is to continue a discussion started back in July by a posting by > Dave Abrahams > regarding the compiler (C vs. C++) used to compile python's main() and to link > the executable. > > > On Sat, Jul 16, 2005 at 12:13:58PM +0200, Christoph Ludwig wrote: >> On Sun, Jul 10, 2005 at 07:41:06PM +0200, "Martin v. L?wis" wrote: >> > Maybe. For Python 2.4, feel free to contribute a more complex test. For >> > Python 2.5, I would prefer if the entire code around ccpython.cc was >> > removed. >> >> I submitted patch #1239112 that implements the test involving two TUs for >> Python 2.4. I plan to work on a more comprehensive patch for Python 2.5 but >> that will take some time. > > > I finally had the spare time to look into this problem again and submitted > patch #1324762. The proposed patch implements the following: I just wanted to write to encourage some Python developers to look at (and accept!) Christoph's patch. This is really crucial for smooth interoperability between C++ and Python. Thank you, Dave -- Dave Abrahams Boost Consulting www.boost-consulting.com From pje at telecommunity.com Tue Nov 1 18:16:52 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 01 Nov 2005 12:16:52 -0500 Subject: [Python-Dev] python-dev sprint at PyCon In-Reply-To: <20051101143505.GE14719@rogue.amk.ca> Message-ID: <5.1.1.6.0.20051101121245.020559e8@mail.telecommunity.com> At 09:35 AM 11/1/2005 -0500, A.M. Kuchling wrote: >Every PyCon has featured a python-dev sprint. For the past few years, >hacking on the AST branch has been a tradition, but we'll have to come >up with something new for this year's conference (in Dallas Texas; >sprints will be Monday Feb. 27 through Thursday March 2). > >According to Anthony's release plan, a first alpha of 2.5 would be >released in March, hence after PyCon and the sprints. We should >discuss possible tasks for a python-dev sprint. What could we do? * PEP 343 implementation ('with:') * PEP 308 implementation ('x if y else z') * A bytes type Or perhaps some of the things that have been waiting for the AST branch to be finished, i.e.: * One of the "global variable speedup" PEPs * Guido's instance variable speedup idea (LOAD_SELF_IVAR and STORE_SELF_IVAR, see http://mail.python.org/pipermail/python-dev/2002-February/019854.html) From guido at python.org Tue Nov 1 18:22:16 2005 From: guido at python.org (Guido van Rossum) Date: Tue, 1 Nov 2005 10:22:16 -0700 Subject: [Python-Dev] python-dev sprint at PyCon In-Reply-To: <5.1.1.6.0.20051101121245.020559e8@mail.telecommunity.com> References: <20051101143505.GE14719@rogue.amk.ca> <5.1.1.6.0.20051101121245.020559e8@mail.telecommunity.com> Message-ID: On 11/1/05, Phillip J. Eby wrote: > At 09:35 AM 11/1/2005 -0500, A.M. Kuchling wrote: > >Every PyCon has featured a python-dev sprint. For the past few years, > >hacking on the AST branch has been a tradition, but we'll have to come > >up with something new for this year's conference (in Dallas Texas; > >sprints will be Monday Feb. 27 through Thursday March 2). > > > >According to Anthony's release plan, a first alpha of 2.5 would be > >released in March, hence after PyCon and the sprints. We should > >discuss possible tasks for a python-dev sprint. What could we do? > > * PEP 343 implementation ('with:') > * PEP 308 implementation ('x if y else z') > * A bytes type * PEP 328 - absolute/relative import * PEP 341 - unifying try/except and try/finally (I believe this was accepted; it's still marked Open in PEP 0) > Or perhaps some of the things that have been waiting for the AST branch to > be finished, i.e.: > > * One of the "global variable speedup" PEPs > * Guido's instance variable speedup idea (LOAD_SELF_IVAR and > STORE_SELF_IVAR, see > http://mail.python.org/pipermail/python-dev/2002-February/019854.html) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From nnorwitz at gmail.com Tue Nov 1 18:59:26 2005 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 1 Nov 2005 09:59:26 -0800 Subject: [Python-Dev] python-dev sprint at PyCon In-Reply-To: References: <20051101143505.GE14719@rogue.amk.ca> <5.1.1.6.0.20051101121245.020559e8@mail.telecommunity.com> Message-ID: On 11/1/05, Guido van Rossum wrote: > On 11/1/05, Phillip J. Eby wrote: > > At 09:35 AM 11/1/2005 -0500, A.M. Kuchling wrote: > > >Every PyCon has featured a python-dev sprint. For the past few years, > > >hacking on the AST branch has been a tradition, but we'll have to come > > >up with something new for this year's conference (in Dallas Texas; > > >sprints will be Monday Feb. 27 through Thursday March 2). > > > > > >According to Anthony's release plan, a first alpha of 2.5 would be > > >released in March, hence after PyCon and the sprints. We should > > >discuss possible tasks for a python-dev sprint. What could we do? I added the 4 PEPs mentioned and a few more ideas here: http://wiki.python.org/moin/PyCon2006/Sprints/PythonCore n From pje at telecommunity.com Tue Nov 1 19:02:09 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 01 Nov 2005 13:02:09 -0500 Subject: [Python-Dev] python-dev sprint at PyCon Message-ID: <5.1.1.6.0.20051101130208.02047018@mail.telecommunity.com> At 10:22 AM 11/1/2005 -0700, Guido van Rossum wrote: >* PEP 328 - absolute/relative import I assume that references to 2.4 in that PEP should be changed to 2.5, and so on. It also appears to me that the PEP doesn't record the issue brought up by some people about the current absolute/relative ambiguity being useful for packaging purposes. i.e., being able to nest third-party packages such that they end up seeing their dependencies, even though they're not installed at the "root" package level. For example, I have a package that needs Python 2.4's version of pyexpat, and I need it to run in 2.3, but I can't really overwrite the 2.3 pyexpat, so I just build a backported pyexpat and drop it in the package, so that the code importing it just ends up with the right thing. Of course, that specific example is okay since 2.3 isn't going to somehow grow absolute importing. :) But I think people brought up other examples besides that, it's just the one that I personally know I've done. From guido at python.org Tue Nov 1 19:14:46 2005 From: guido at python.org (Guido van Rossum) Date: Tue, 1 Nov 2005 11:14:46 -0700 Subject: [Python-Dev] python-dev sprint at PyCon In-Reply-To: <5.1.1.6.0.20051101130208.02047018@mail.telecommunity.com> References: <5.1.1.6.0.20051101130208.02047018@mail.telecommunity.com> Message-ID: On 11/1/05, Phillip J. Eby wrote: > At 10:22 AM 11/1/2005 -0700, Guido van Rossum wrote: > >* PEP 328 - absolute/relative import > > I assume that references to 2.4 in that PEP should be changed to 2.5, and > so on. For the part that hasn't been implemented yet, yes. > It also appears to me that the PEP doesn't record the issue brought up by > some people about the current absolute/relative ambiguity being useful for > packaging purposes. i.e., being able to nest third-party packages such > that they end up seeing their dependencies, even though they're not > installed at the "root" package level. > > For example, I have a package that needs Python 2.4's version of pyexpat, > and I need it to run in 2.3, but I can't really overwrite the 2.3 pyexpat, > so I just build a backported pyexpat and drop it in the package, so that > the code importing it just ends up with the right thing. > > Of course, that specific example is okay since 2.3 isn't going to somehow > grow absolute importing. :) But I think people brought up other examples > besides that, it's just the one that I personally know I've done. I guess this ought to be recorded. :-( The issue has been beaten to death and my position remains firm: rather than playing namespace games, consistent renaming is the right thing to do here. This becomes a trivial source edit, which beats the problems of debugging things when it doesn't work out as expected (which is very common due to the endless subtleties of loading multiple versions of the same code). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Tue Nov 1 19:28:12 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 01 Nov 2005 13:28:12 -0500 Subject: [Python-Dev] python-dev sprint at PyCon In-Reply-To: References: <5.1.1.6.0.20051101130208.02047018@mail.telecommunity.com> <5.1.1.6.0.20051101130208.02047018@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051101132151.02fe9708@mail.telecommunity.com> At 11:14 AM 11/1/2005 -0700, Guido van Rossum wrote: >I guess this ought to be recorded. :-( > >The issue has been beaten to death and my position remains firm: >rather than playing namespace games, consistent renaming is the right >thing to do here. This becomes a trivial source edit, Well, it's not trivial if you're (in my case) trying to support 2.3 and 2.4 with the same code base. It'd be nice to have some other advice to offer people besides, "go edit your code". Of course, if the feature hadn't already existed, I suppose a PEP to add it would have been shot down, so it's a reasonable decision. >which beats the >problems of debugging things when it doesn't work out as expected >(which is very common due to the endless subtleties of loading >multiple versions of the same code). Yeah, Bob Ippolito and I batted around a few ideas about how to implement simultaneous multi-version imports for Python Eggs, some of which relied on the relative/absolute ambiguity, but I think the main subtleties have to do with dynamic imports (including pickling) and the use of __name__. Of course, since we never actually implemented it, I don't know what other subtleties could potentially exist. Python Eggs currently allow you to install multiple versions of a package, but at runtime you can only import one of them, and you get a runtime VersionConflict exception if two eggs' version criteria are incompatible. From nnorwitz at gmail.com Tue Nov 1 19:34:29 2005 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 1 Nov 2005 10:34:29 -0800 Subject: [Python-Dev] python-dev sprint at PyCon In-Reply-To: <5.1.1.6.0.20051101132151.02fe9708@mail.telecommunity.com> References: <5.1.1.6.0.20051101130208.02047018@mail.telecommunity.com> <5.1.1.6.0.20051101132151.02fe9708@mail.telecommunity.com> Message-ID: On 11/1/05, Phillip J. Eby wrote: > At 11:14 AM 11/1/2005 -0700, Guido van Rossum wrote: > >I guess this ought to be recorded. :-( > > > >The issue has been beaten to death and my position remains firm: > >rather than playing namespace games, consistent renaming is the right > >thing to do here. This becomes a trivial source edit, > > Well, it's not trivial if you're (in my case) trying to support 2.3 and 2.4 > with the same code base. > > It'd be nice to have some other advice to offer people besides, "go edit > your code". Of course, if the feature hadn't already existed, I suppose a > PEP to add it would have been shot down, so it's a reasonable decision. Why can't you add your version's directory to sys.path before importing pyexpat? n From jcarlson at uci.edu Tue Nov 1 19:48:46 2005 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 01 Nov 2005 10:48:46 -0800 Subject: [Python-Dev] apparent ruminations on mutable immutables (was: PEP 351, the freeze protocol) In-Reply-To: References: <20051031120205.3A0C.JCARLSON@uci.edu> Message-ID: <20051101104731.0389.JCARLSON@uci.edu> Noam Raphael wrote: > On 10/31/05, Josiah Carlson wrote: > > > About the users-changing-my-internal-data issue: > ... > > You can have a printout before it dies: > > "I'm crashing your program because something attempted to modify a data > > structure (here's the traceback), and you were told not to." > > > > Then again, you can even raise an exception when people try to change > > the object, as imdict does, as tuples do, etc. > > Both solutions would solve the problem, but would require me to wrap > the built-in set with something which doesn't allow changes. This is a > lot of work - but it's quite similiar to what my solution would > actually do, in a single built-in function. I am an advocate for PEP 351. However, I am against your proposed implementation/variant of PEP 351 because I don't believe it ads enough to warrant the additional complication and overhead necessary for every object (even tuples would need to get a .frozen_cache member). Give me a recursive freeze from PEP 351 (which handles objects that are duplicated, but errors out on circular references), and I'll be happy. > > > You suggest two ways for solving the problem. The first is by copying > > > my mutable objects to immutable copies: > > > > And by caching those results, then invalidating them when they are > > updated by your application. This is the same as what you would like to > > do, except that I do not rely on copy-on-write semantics, which aren't > > any faster than freeze+cache by your application. > > This isn't correct - freezing a set won't require a single copy to be > performed, as long as the frozen copy isn't saved after the original > is changed. Copy+cache always requires one copy. You are wrong, and you even say you are wrong..."freezing a set doesn't require a COPY, IF the frozen COPY isn't saved after the original is CHANGED". Creating an immutable set IS CREATING A COPY, so it ALSO copies, and you admit as much, but then say the equivalent of "copying isn't copying because I say so". > > In any case, whether you choose to use freeze, or use a different API, > > this particular problem is solvable without copy-on-write semantics. > > Right. But I think that a significant simplification of the API is a > nice bonus for my solution. And about those copy-on-write semantics - > it should be proven how complex they are. Remember that we are talking > about frozen-copy-on-write, which I think would simplify matters > considerably - for example, there are at most two instances sharing > the same data, since the frozen copy can be returned again and again. I think that adding an additional attribute to literally every single object to handle the caching of 'frozen' objects, as well as a list to every object to handle callbacks which should be called on object mutation, along with a _call_stuff_when_mutated() method that handles these callback calls, IN ADDITION TO the __freeze__ method which is necessary to support this, is a little much, AND IS CERTAINLY NOT A SIMPLIFICATION! Let us pause for a second and consider: Original PEP proposed 1 new method: __freeze__, which could be implemented as a subclass of the original object (now), and integrated into the original classes as time goes on. One could /register/ __freeze__ functions/methods a'la Pickle, at which point objects wouldn't even need a native freeze method. Your suggestion offers 2 new methods along with 2 new instance variables. Let's see, a callback handler, __freeze__, the cache, and the callback list. Doesn't that seem a little excessive to you to support freezing? It does to me. If Guido were to offer your implementation of freeze, or no freeze at all, I would opt for no freeze, as implementing your freeze on user-defined classes would be a pain in the ass, not to mention implementing them in C code would be more than I would care to do, and more than I would ask any of the core developers to work on. > > Even without validation, there are examples that force a high number of > > calls, which are not O(1), ammortized or otherwise. > > > [Snap - a very interesting example] > > > > Now, the actual time analysis on repeated freezings and such gets ugly. > > There are actually O(k) objects, which take up O(k**2) space. When you > > modify object b[i][j] (which has just been frozen), you get O(k) > > callbacks, and when you call freeze(b), it actually results in O(k**2) > > time to re-copy the O(k**2) pointers to the O(k) objects. It should be > > obvious that this IS NOT AMMORTIZABLE to original object creation time. > > > That's absolutely right. My ammortized analysis is correct only if you > limit yourself to cases in which the original object doesn't change > after a frozen() call was made. In that case, it's ok to count the > O(k**2) copy with the O(k**2) object creation, because it's made only > once. But here's the crucial observation which you are missing. You yourself have stated that in both your table and graph examples you want your application to continue to modify values while the user can't manipulate them. So even in your own use-cases, you are going to be modifying objects after they have been frozen, and even then it won't be fast! I believe that in general, people who are freezing things are going to want to be changing the original objects - hence the use of mutables to begin with - maybe for situations like yours where you don't want users mutating returns, whatever. If after they have frozen the object, they don't want to be changing the original objects, then they are probably going to be tossing out the original mutable and using the immutable created with freeze anyways (mutate your object until you get it right, then freeze it and use that so that no one can alter your data, not even yourself), so I think that caching is something that the /user/ should be doing, NOT Python. The simple implementation (not copy-on-write) leads us to a simple matter of documenting, "Freeze is 'stateless'; every call to freeze returns a new object, regardless of modifications (or lack thereof) between freeze calls." Remember: "Simple is better than complex." > Why it's ok to analyze only that limited case? I am suggesting a > change in Python: that every object you would like be mutable, and > would support the frozen() protocol. When you evaluate my suggestion, > you need to take a program, and measure its performance in the current > Python and in a Python which implements my suggestion. This means that > the program should work also on the current Python. In that case, my > assumption is true - you won't change objects after you have frozen > them, simply because these objects (strings which are used as dict > keys, for example) can't be changed at all in the current Python > implementation! Not everything can/should become mutable. Integers should never become mutable, as tuples should never become mutable, as strings/unicode should never become mutable...wait, aren't we getting to the point that everything which is currently immutable shouldn't become mutable? Indeed. I don't believe that any currently immutable object should be able to become mutable in order to satisfy /anyone's/ desire for mutable /anything/. In starting to bring up benchmarks you are falling into the trap of needing to /have/ a benchmark (I have too), for which there are very few, if any, current use-cases. Without having or needing a benchmark, I'll state quite clearly where your proposed copy-on-write would beat out the naive 'create a new copy on every call to freeze': 1. If objects after they are frozen are never modified, copy on write will be faster. 2. If original objects are modified after they are frozen, then the naive implementation will be as fast if not faster in general, due to far lower overhead, but may be slower in corner cases where some nested structure is unchanged, and some shallow bit has changed: x = [[], NEVER_CHANGED_MUTABLE_NESTED_STRUCTURE] y = freeze(x) x[0].append(1) z = freeze(x) Further, discussing benchmarks on use-cases, for which there are few (if any) previously existing uses, is like saying "let's race cars" back in 1850; it's a bit premature. Then there is this other example: x = [1,2,3] y = freeze(x) The flat version of freeze in the PEP right now handles this case. I can change x all I want, yet I have a frozen y which stays unchanged. This is what I would want, and I would imagine it is what others would want too. In fact, this is precisely the use-case you offered for your table and graph examples, so your expression of a sentiment of "users aren't going to be changing the object after it has been frozen" is, by definition, wrong: you do it yourself! > I will write it in another way: I am proposing a change that will make > Python objects, including strings, mutable, and gives you other > advantages as well. I claim that it won't make existing Python > programs run slower in O() terms. It would allow you to do many things > that you can't do today; some of them would be fast, like editing a > string, and some of them would be less fast - for example, repeatedly > changing an object and freezing it. Your claim on running time only works if the original isn't changed after it is frozen And I don't like making everything mutable, it's a "solution looking for a problem", or a "tail wagging the dog" idea. There is no good reason to make everything mutable, and I challenge you to come up with a valid one that isn't already covered by the existing standard library or extension modules. There is no need to bring strings into this conversation as there are string variants which are already mutable: array.array('c', ...), StringIO, mmap, take your pick! And some future Python (perhaps 2.5) will support a 'bytes' object, which is essentially an mmap which doesn't need to be backed by a file. > I think that the performance penalty may be rather small - remember > that in programs which do not change strings, there would never be a > need to copy the string data at all. And since I think that usually > most of the dict lookups are for method or function names, there would > almost never be a need to constuct a new object on dict lookup, > because you search for the same names again and again, and a new > object is created only on the first frozen() call. You might even gain > performance, because s += x would be faster. You really don't know how Python internals work. The slow part of s += x on strings in Python 2.4 is the memory reallocation and occasional data copy (it has been tuned like crazy by Raymond in 2.4, see _PyString_Resize in stringobject.c). Unless you severely over-allocated your strings, this WOULD NOT BE SPED UP BY MUTABLE STRINGS. Further, identifiers/names (obj, obj.attr, obj.attr1.attr2, ...) are already created during compile-time, and are 'interned'. That is, if you have an object that you call 'foo', there gets to be a single "foo" string, which is referenced by pointer by any code in that module which references the 'foo' object to the single, always unchanging "foo" string. And because the string has already been hashed, it has a cached hash value, and lookups in dictionaries are already fast due to a check for pointer equivalency before comparing contents. Mutable strings CANNOT be faster than this method. > > You have clarified it, but it is still wrong. I stand by 'it is not > > easy to get right', and would further claim, "I doubt it is possible to > > make it fast." > > It would not be very easy to implement, of course, but I hope that it > won't be very hard either, since the basic idea is quite simple. Do > you still doubt the possibility of making it fast, given my (correct) > definition of fast? I would claim that your definition is limited. Yours would be fast if objects never changed after they are frozen, which is counter to your own use-cases. This suggests that your definition is in fact incorrect, and you fail to see your own inconsistancy. > And if it's possible (which I think it is), it would allow us to get > rid of inconvinient immutable objects, and it would let us put > everything into a set. Isn't that nice? No, it sounds like a solution looking for a problem. I see no need to make strings, floats, ints, tuples, etc. mutable, and I think that you will have very little luck in getting core Python developer support for any attempt to make them mutable. If you make such a suggestion, I would offer that you create a new PEP, because this discussion has gone beyond PEP 351, and has wandered into the realm of "What other kinds of objects would be interesting to have in a Python-like system?" I'll summarize your claims: 1. copy-on-write is a simplification 2. everything being mutable would add to Python 3. copy-on-write is fast 4. people won't be mutating objects after they are frozen I'll counter your claims: 1. 2 methods and 2 instance variables on ALL OBJECTS is not a simplification. 2. a = b = 1; a += 1; If all objects were to become mutable, then a == b, despite what Python and every other sane language would tell you, and dct[a] would stop working (you would have to use c = freeze(a);dct[c], or dct[x] would need to automatically call freeze and only ever reference the result, significantly slowing down ALL dictionary references). 3. only if you NEVER MUTATE an object after it has been frozen 4. /you/ mutate original objects after they are frozen ALSO: 5. You fail to realize that if all objects were to become mutable, then one COULDN'T implement frozen, because the frozen objects THEMSELVES would be mutable. I'm going to bow out of this discussion for a few reasons, not the least of which being that I've spent too much time on this subject, and that I think it is quite clear that your proposal is dead, whether I had anything to do with it or not. - Josiah From pje at telecommunity.com Tue Nov 1 19:50:00 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 01 Nov 2005 13:50:00 -0500 Subject: [Python-Dev] python-dev sprint at PyCon In-Reply-To: References: <5.1.1.6.0.20051101132151.02fe9708@mail.telecommunity.com> <5.1.1.6.0.20051101130208.02047018@mail.telecommunity.com> <5.1.1.6.0.20051101132151.02fe9708@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051101134754.0380cf68@mail.telecommunity.com> At 10:34 AM 11/1/2005 -0800, Neal Norwitz wrote: >Why can't you add your version's directory to sys.path before importing >pyexpat? With library code that can be imported in any order, there is no such thing as "before". Anyway, Guido has pronounced on this already, so it's moot. From guido at python.org Tue Nov 1 20:39:39 2005 From: guido at python.org (Guido van Rossum) Date: Tue, 1 Nov 2005 12:39:39 -0700 Subject: [Python-Dev] python-dev sprint at PyCon In-Reply-To: <5.1.1.6.0.20051101132151.02fe9708@mail.telecommunity.com> References: <5.1.1.6.0.20051101130208.02047018@mail.telecommunity.com> <5.1.1.6.0.20051101132151.02fe9708@mail.telecommunity.com> Message-ID: On 11/1/05, Phillip J. Eby wrote: > At 11:14 AM 11/1/2005 -0700, Guido van Rossum wrote: > >I guess this ought to be recorded. :-( > > > >The issue has been beaten to death and my position remains firm: > >rather than playing namespace games, consistent renaming is the right > >thing to do here. This becomes a trivial source edit, > > Well, it's not trivial if you're (in my case) trying to support 2.3 and 2.4 > with the same code base. You should just bite the bullet and make a privatized copy of the package(s) on which you depend part of your own distributions. > It'd be nice to have some other advice to offer people besides, "go edit > your code". Of course, if the feature hadn't already existed, I suppose a > PEP to add it would have been shot down, so it's a reasonable decision. I agree it would be nice if we could do something about deep version issues. But it's hard, and using the absolute/relative ambiguity isn't a solution but a nasty hack. I don't have a solution either except copying code (which IMO is a *fine* solution in most cases as long as copyright issues don't prevent you). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From raymond.hettinger at verizon.net Tue Nov 1 21:14:32 2005 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue, 01 Nov 2005 15:14:32 -0500 Subject: [Python-Dev] a different kind of reduce... In-Reply-To: <4366C3BB.3010407@canterbury.ac.nz> Message-ID: <001301c5df20$df865f00$153dc797@oemcomputer> [Martin Blais] > > I'm always--literally every time-- looking for a more functional form, > > something that would be like this: > > > > # apply dirname() 3 times on its results, initializing with p > > ... = repapply(dirname, 3, p) [Greg Ewing] > Maybe ** should be defined for functions so that you > could do things like > > up3levels = dirname ** 3 Hmm, using the function's own namespace is an interesting idea. It might also be a good place to put other functionals: results = f.map(data) newf = f.partial(somearg) Raymond From dberlin at dberlin.org Tue Nov 1 21:15:08 2005 From: dberlin at dberlin.org (Daniel Berlin) Date: Tue, 01 Nov 2005 15:15:08 -0500 Subject: [Python-Dev] svn checksum error In-Reply-To: <17253.28294.538932.570903@montanaro.dyndns.org> References: <17252.59531.252751.768301@montanaro.dyndns.org> <43654CA7.8030200@v.loewis.de> <17253.28294.538932.570903@montanaro.dyndns.org> Message-ID: <1130876108.7280.35.camel@IBM-82ZWS052TEN.watson.ibm.com> On Sun, 2005-10-30 at 19:08 -0600, skip at pobox.com wrote: > Martin> The natural question then is: what operating system, what > Martin> subversion version are you using? > > Sorry, wasn't thinking in terms of svn bugs. I was anticipating some sort > of obvious pilot error. I am on Mac OSX 10.3.9, running svn 1.1.3 I built > from source back in the May timeframe. Should I upgrade to 1.2.3 as a > matter of course? > > Fredrik> "welcome to the wonderful world of subversion error messages" > ... > Fredrik> deleting the offending directory and doing "svn up" is the > Fredrik> easiest way to fix this. > > Thanks. I zapped Objects. The next svn up complained about Misc. The next > about Lib. After that, the next svn up ran to completion. > > Skip You didn't happen to try to update a checked out copy from a repo that had an older cvs2svn conversion to the one produced by the final conversion, did you? Cause that will cause these errors too. --Dan From jeremy at alum.mit.edu Tue Nov 1 21:23:05 2005 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Tue, 1 Nov 2005 15:23:05 -0500 Subject: [Python-Dev] python-dev sprint at PyCon In-Reply-To: <5.1.1.6.0.20051101121245.020559e8@mail.telecommunity.com> References: <20051101143505.GE14719@rogue.amk.ca> <5.1.1.6.0.20051101121245.020559e8@mail.telecommunity.com> Message-ID: On 11/1/05, Phillip J. Eby wrote: > At 09:35 AM 11/1/2005 -0500, A.M. Kuchling wrote: > >Every PyCon has featured a python-dev sprint. For the past few years, > >hacking on the AST branch has been a tradition, but we'll have to come > >up with something new for this year's conference (in Dallas Texas; > >sprints will be Monday Feb. 27 through Thursday March 2). > > > >According to Anthony's release plan, a first alpha of 2.5 would be > >released in March, hence after PyCon and the sprints. We should > >discuss possible tasks for a python-dev sprint. What could we do? > > * PEP 343 implementation ('with:') > * PEP 308 implementation ('x if y else z') > * A bytes type > > Or perhaps some of the things that have been waiting for the AST branch to > be finished, i.e.: > > * One of the "global variable speedup" PEPs > * Guido's instance variable speedup idea (LOAD_SELF_IVAR and > STORE_SELF_IVAR, see > http://mail.python.org/pipermail/python-dev/2002-February/019854.html) I hope to attend the sprints this year, so i'd be around to help people get started and answer questions. With luck, I'll also be giving a technical presentation on the work at the main conference. Jeremy From reinhold-birkenfeld-nospam at wolke7.net Tue Nov 1 21:27:23 2005 From: reinhold-birkenfeld-nospam at wolke7.net (Reinhold Birkenfeld) Date: Tue, 01 Nov 2005 21:27:23 +0100 Subject: [Python-Dev] a different kind of reduce... In-Reply-To: <001301c5df20$df865f00$153dc797@oemcomputer> References: <4366C3BB.3010407@canterbury.ac.nz> <001301c5df20$df865f00$153dc797@oemcomputer> Message-ID: Raymond Hettinger wrote: > [Martin Blais] >> > I'm always--literally every time-- looking for a more functional > form, >> > something that would be like this: >> > >> > # apply dirname() 3 times on its results, initializing with p >> > ... = repapply(dirname, 3, p) > > [Greg Ewing] >> Maybe ** should be defined for functions so that you >> could do things like >> >> up3levels = dirname ** 3 > > Hmm, using the function's own namespace is an interesting idea. It > might also be a good place to put other functionals: > > results = f.map(data) > newf = f.partial(somearg) And we have solved the "map, filter and reduce are going away! Let's all weep together" problem with one strike! Reinhold -- Mail address is perfectly valid! From noamraph at gmail.com Tue Nov 1 21:49:59 2005 From: noamraph at gmail.com (Noam Raphael) Date: Tue, 1 Nov 2005 22:49:59 +0200 Subject: [Python-Dev] apparent ruminations on mutable immutables (was: PEP 351, the freeze protocol) In-Reply-To: <20051101104731.0389.JCARLSON@uci.edu> References: <20051031120205.3A0C.JCARLSON@uci.edu> <20051101104731.0389.JCARLSON@uci.edu> Message-ID: On 11/1/05, Josiah Carlson wrote: ... > > I am an advocate for PEP 351. However, I am against your proposed > implementation/variant of PEP 351 because I don't believe it ads enough > to warrant the additional complication and overhead necessary for every > object (even tuples would need to get a .frozen_cache member). > > Give me a recursive freeze from PEP 351 (which handles objects that are > duplicated, but errors out on circular references), and I'll be happy. > That's fine - but it doesn't mean that I must be happy with it. > ... > > > > This isn't correct - freezing a set won't require a single copy to be > > performed, as long as the frozen copy isn't saved after the original > > is changed. Copy+cache always requires one copy. > > You are wrong, and you even say you are wrong..."freezing a set doesn't > require a COPY, IF the frozen COPY isn't saved after the original is > CHANGED". Creating an immutable set IS CREATING A COPY, so it ALSO > copies, and you admit as much, but then say the equivalent of "copying > isn't copying because I say so". No, I am not wrong. I am just using misleading terms. I will call a "frozen copy" a "frozen image". Here it goes: "freezing a set doesn't require a COPY, IF the frozen IMAGE isn't saved after the original is CHANGED". I suggest that there would be a way to create a frozenset without COPYING an O(n) amount of MEMORY. When a frozen set is created by a call frozen(x), it would not copy all the data, but would rather reference the existing data, which was created by the non-frozen set. Only if the original set changes, when there's a frozen set referencing the data, the MEMORY would be actually copied. I call it a "frozen copy" because it behaves as a frozen copy, even though not all the memory is being copied. When you call the COPY function in the COPY module with a string, it doesn't really copy memory - the same string is returned. When you copy a file inside subversion, it doesn't actually copy all the data associated with it, but does something smarter, which takes O(1). The point is, for the user, it's a copy. Whether or not memory is actually being copied, is an implementation detail. > ... > > I think that adding an additional attribute to literally every single > object to handle the caching of 'frozen' objects, as well as a list to > every object to handle callbacks which should be called on object > mutation, along with a _call_stuff_when_mutated() method that handles > these callback calls, IN ADDITION TO the __freeze__ method which is > necessary to support this, is a little much, AND IS CERTAINLY NOT A > SIMPLIFICATION! I don't agree. You don't need to add a list to every object, since you can store all those relations in one place, with a standard function for registering them. Anyway, code written in Python (which is the language we are discussing) WON'T BE COMPLICATED! The frozen mechanism, along with two new protocols (__frozen__ and __changed__), would be added automatically! The internal state of a class written in Python can be automatically frozen, since it's basically a dict. Now let's see if it's a simplification: 1. No Python code would have to be made more complicated because of the change. 2. There would be no need to find workarounds, like cStringIO, for the fact that strings and tuples are immutable. 3. You would be able to put any kind of object into a set, or use it as a dict key. 4. Classes (like the graph example) would be able to give users things without having to make a choice between risking their users with strange bugs, making a complicated interface, making very inefficient methods, and writing complicated wrapper classes. I will ask you: Is this a complication? The answer is: it requires a significent change of the CPython implementation. But about the Python language: it's definitely a simplification. > > Let us pause for a second and consider: > Original PEP proposed 1 new method: __freeze__, which could be > implemented as a subclass of the original object (now), and integrated > into the original classes as time goes on. One could /register/ > __freeze__ functions/methods a'la Pickle, at which point objects > wouldn't even need a native freeze method. > > Your suggestion offers 2 new methods along with 2 new instance variables. > Let's see, a callback handler, __freeze__, the cache, and the callback > list. Doesn't that seem a little excessive to you to support freezing? > It does to me. If Guido were to offer your implementation of freeze, or > no freeze at all, I would opt for no freeze, as implementing your freeze > on user-defined classes would be a pain in the ass, not to mention > implementing them in C code would be more than I would care to do, and > more than I would ask any of the core developers to work on. > As I said above: this suggestion would certainly require more change in the Python implementation than your suggestion. But the Python language would gain a lot more. Implementing my frozen on user-defined classes would not be a pain in the ass, because it will require no work at all - the Python implementation would provide it automatically. The fact that it can be done automatically for user-defined classes raises a hope in me that it can be made not too complicated for classes written in C. > ... > > But here's the crucial observation which you are missing. You yourself > have stated that in both your table and graph examples you want your > application to continue to modify values while the user can't manipulate > them. So even in your own use-cases, you are going to be modifying > objects after they have been frozen, and even then it won't be fast! No. In the table example, the table would never change the object themselves - it may only calculate new values, and drop the references to the old ones. This is definitely a case of not changing the value after it has been frozen. In the graph example, it is true that the set would be changed after it's frozen, but it is expected that the frozen copy would not exist by the time the change happens - think about the x is graph.neighbours(y) example. There is actually no reason for keeping them, besides for tracking the history of the graph - which would require a copy anyway. The frozen() implementation of objects which do not reference non-frozen objects, such as sets, really doesn't copy any memory when it's called, and will never cause a memory copy if there are no living frozen copies of the object while the object changes. > > I believe that in general, people who are freezing things are going to > want to be changing the original objects - hence the use of mutables to > begin with - maybe for situations like yours where you don't want users > mutating returns, whatever. If after they have frozen the object, they > don't want to be changing the original objects, then they are probably > going to be tossing out the original mutable and using the immutable > created with freeze anyways (mutate your object until you get it right, > then freeze it and use that so that no one can alter your data, not even > yourself), so I think that caching is something that the /user/ should > be doing, NOT Python. I don't agree. The table and the graph are examples. The common use patterns I see regarding frozen() are: 1. Don't use frozen() at all. Think about strings becoming mutable. Most strings which are changed would never be frozen. When you are using a list, how many times do you make a frozen copy of it? (The answer is zero, of course, you can't. You can't use it as a dict key, or as a member of a set. This is just to show you that not freezing mutable objects is a common thing.) 2. Create the object using more operations than constructing it, and then don't change it, possibly making a frozen copy of it. The table is an example: functions given by the user create objects, in whatever way they choose, and then the table doesn't need to change them, and needs to create a frozen copy. It's a very reasonable use case: I would say that the less common case is that you can create an object using only the constructor. Many times you make a tuple out of a list that you've created just for that purpose. It's not intuitive! > > The simple implementation (not copy-on-write) leads us to a simple > matter of documenting, "Freeze is 'stateless'; every call to freeze > returns a new object, regardless of modifications (or lack thereof) > between freeze calls." > > Remember: "Simple is better than complex." Firstly, you are talking about implementation. Secondly, sometimes things are too simple, and lead to complex workarounds. > ... > > Not everything can/should become mutable. Integers should never become > mutable, as tuples should never become mutable, as strings/unicode > should never become mutable...wait, aren't we getting to the point that > everything which is currently immutable shouldn't become mutable? > Indeed. I don't believe that any currently immutable object should be > able to become mutable in order to satisfy /anyone's/ desire for mutable > /anything/. Integers should never become mutable - right. There should be no mutable ints in Python. Tuples and strings should never become mutable - wrong. Strings created by the user should be mutable - those immutable strings are a Python anomaly. All I was saying was that sometimes, the Python implementation would want to use immutable strings. So will users, sometimes. There is a need for a mutable string, and a need for an immutable string, and a need for an efficient conversion between the two. That's all. > > > In starting to bring up benchmarks you are falling into the trap of > needing to /have/ a benchmark (I have too), for which there are very few, > if any, current use-cases. > No, I don't. There are a lot of use cases. As I said, I suggest a change to the Python language, which would give you many benefits. When suggesting such a change, it should be verified that the performance of existing Python programs won't be harmed, which I did. What might be done as well, is to compare my suggestion to yours: > Without having or needing a benchmark, I'll state quite clearly where > your proposed copy-on-write would beat out the naive 'create a new copy > on every call to freeze': > 1. If objects after they are frozen are never modified, copy on write > will be faster. > 2. If original objects are modified after they are frozen, then the > naive implementation will be as fast if not faster in general, due to > far lower overhead, but may be slower in corner cases where some nested > structure is unchanged, and some shallow bit has changed: As I said, many objects are never modified after they are frozen. ***This includes all the strings which are used in current Python programs as dict keys*** - I suggest that strings would become mutable by default. This means that whenever you use a string as a dict key, a call to frozen() is done by the dict. It's obvious that the string won't change after it is frozen. Now, my suggestion is faster in its order of complexity than yours. In some cases, yours is faster by a constant, which I claim that would be quite small in real use cases. > > x = [[], NEVER_CHANGED_MUTABLE_NESTED_STRUCTURE] > y = freeze(x) > x[0].append(1) > z = freeze(x) > This is one of the cases in which the change in order of complexity is significant. > Further, discussing benchmarks on use-cases, for which there are few (if > any) previously existing uses, is like saying "let's race cars" back in > 1850; it's a bit premature. I don't agree. That's why we're discussing it. > > > Then there is this other example: > > x = [1,2,3] > y = freeze(x) > > The flat version of freeze in the PEP right now handles this case. I > can change x all I want, yet I have a frozen y which stays unchanged. > This is what I would want, and I would imagine it is what others would > want too. In fact, this is precisely the use-case you offered for your > table and graph examples, so your expression of a sentiment of "users > aren't going to be changing the object after it has been frozen" is, by > definition, wrong: you do it yourself! Okay, so may I add another use case in which frozen() is fast: If an object which only holds references to frozen object is changed after a frozen copy of it has been made, and the frozen copy is discarded before the change is made, frozen() would still take O(1). This is the case with the graph. > > > > I will write it in another way: I am proposing a change that will make > > Python objects, including strings, mutable, and gives you other > > advantages as well. I claim that it won't make existing Python > > programs run slower in O() terms. It would allow you to do many things > > that you can't do today; some of them would be fast, like editing a > > string, and some of them would be less fast - for example, repeatedly > > changing an object and freezing it. > > Your claim on running time only works if the original isn't changed > after it is frozen But they won't, in existing Python programs, so my claim: "it won't make existing Python programs run slower in O() terms" is absolutely correct! > > And I don't like making everything mutable, it's a "solution looking for > a problem", or a "tail wagging the dog" idea. There is no good reason > to make everything mutable, and I challenge you to come up with a valid > one that isn't already covered by the existing standard library or > extension modules. > > There is no need to bring strings into this conversation as there are > string variants which are already mutable: array.array('c', ...), > StringIO, mmap, take your pick! And some future Python (perhaps 2.5) > will support a 'bytes' object, which is essentially an mmap which > doesn't need to be backed by a file. My two examples don't have a satisfactory solution currently. All this variety acutally proves my point: There is "more than one way to do it" because these are all workarounds! If strings were mutable, I won't have to learn about all these nice modules. And if that's not enough, here's a simple use case which isn't covered by all those modules. Say I store my byte arrays using array.array, or mmap. What if I want to make a set of those, in order to check if a certain byte sequence was already encountered? I CAN'T. I have to do another workaround, which will probably be complicated and unefficient, to convert my byte array into a string. Everything is possible, if you are willing to work hard enough. I am suggesting to simplify things. ... > > You really don't know how Python internals work. > > The slow part of s += x on strings in Python 2.4 is the memory > reallocation and occasional data copy (it has been tuned like crazy by > Raymond in 2.4, see _PyString_Resize in stringobject.c). Unless you > severely over-allocated your strings, this WOULD NOT BE SPED UP BY > MUTABLE STRINGS. The fact that it has been tuned like crazy, and that it had to wait for Python 2.4, is just showing us that we talking on *another* workaround. And please remember that this optimization was announced not to be counted on, in case you want your code to work efficiently on other Python implementations. In that case (which would just grow more common in the future), you would have to get back to the other workarounds, like cStringIO. > > Further, identifiers/names (obj, obj.attr, obj.attr1.attr2, ...) are > already created during compile-time, and are 'interned'. That is, if > you have an object that you call 'foo', there gets to be a single "foo" > string, which is referenced by pointer by any code in that module which > references the 'foo' object to the single, always unchanging "foo" > string. And because the string has already been hashed, it has a cached > hash value, and lookups in dictionaries are already fast due to a check > for pointer equivalency before comparing contents. Mutable strings > CANNOT be faster than this method. Right - they can't be faster than this method. But they can be virtually AS FAST. Store frozen strings as identifiers/names, and you can continue to use exactly the same method you described. > > ... > > I would claim that your definition is limited. Yours would be fast if > objects never changed after they are frozen, which is counter to your > own use-cases. This suggests that your definition is in fact incorrect, > and you fail to see your own inconsistancy. > I have answered this above: It is not counter to my use cases, and it's a very good assumption, as it is true in many examples, including all current Python programs. > > > And if it's possible (which I think it is), it would allow us to get > > rid of inconvinient immutable objects, and it would let us put > > everything into a set. Isn't that nice? > > No, it sounds like a solution looking for a problem. I see no need to > make strings, floats, ints, tuples, etc. mutable, and I think that you > will have very little luck in getting core Python developer support for > any attempt to make them mutable. Concerning ints, floats, complexes, and any other object with a constant memory use, I agree. Concerning other objects, I disagree. I think that it would simplify things considerably, and that many things that we are used to are actually workarounds. > > If you make such a suggestion, I would offer that you create a new PEP, > because this discussion has gone beyond PEP 351, and has wandered into > the realm of "What other kinds of objects would be interesting to have > in a Python-like system?" > That is a good suggestion, and I have already started to write one. It takes me a long time, but I hope I will manage. > > > I'll summarize your claims: > 1. copy-on-write is a simplification > 2. everything being mutable would add to Python > 3. copy-on-write is fast > 4. people won't be mutating objects after they are frozen > > I'll counter your claims: I'll counter-counter them: > 1. 2 methods and 2 instance variables on ALL OBJECTS is not a > simplification. It is. This is basically an implementation detail, Python code would never be complicated. > 2. a = b = 1; a += 1; If all objects were to become mutable, then a == > b, despite what Python and every other sane language would tell you, and > dct[a] would stop working (you would have to use c = freeze(a);dct[c], > or dct[x] would need to automatically call freeze and only ever > reference the result, significantly slowing down ALL dictionary > references). This might be the point that I didn't stress enough. Dict *would* call freeze, and this is why more work is needed to make sure it is a quick operation. I have proven that it is quick in O() terms, and I claimed that it can be made quick in actual terms. > 3. only if you NEVER MUTATE an object after it has been frozen ...or if the frozen copy is killed before the change, for many types of objects. > 4. /you/ mutate original objects after they are frozen Yes I do, but see 3. > > ALSO: > 5. You fail to realize that if all objects were to become mutable, then > one COULDN'T implement frozen, because the frozen objects THEMSELVES > would be mutable. Really, you take me by the word. All objects COULD become mutable, if we supply a frozen version of it. This doesn't include any object which you don't want, including ints, and including frozen objects. > > I'm going to bow out of this discussion for a few reasons, not the least > of which being that I've spent too much time on this subject, and that I > think it is quite clear that your proposal is dead, whether I had > anything to do with it or not. > > - Josiah That's fine. I wish that you read my answer, think about it a little, and just tell me in a yes or a no if you still consider it dead. I think that I have answered all your questions, and I hope that at least others would be convinced by them, and that at the end my suggestion would be accepted. Others who read this - please respond if you think there's something to my suggestion! Thanks for your involvement. I hope it would at least help me better explain my idea. Noam From noamraph at gmail.com Tue Nov 1 21:55:14 2005 From: noamraph at gmail.com (Noam Raphael) Date: Tue, 1 Nov 2005 22:55:14 +0200 Subject: [Python-Dev] a different kind of reduce... In-Reply-To: References: <4366C3BB.3010407@canterbury.ac.nz> <001301c5df20$df865f00$153dc797@oemcomputer> Message-ID: On 11/1/05, Reinhold Birkenfeld wrote: > > Hmm, using the function's own namespace is an interesting idea. It > > might also be a good place to put other functionals: > > > > results = f.map(data) > > newf = f.partial(somearg) > > And we have solved the "map, filter and reduce are going away! Let's all > weep together" problem with one strike! > > Reinhold I have no problems with map and filter goint away. About reduce - please remember that you need to add this method to any callable, including every type (I mean the constructor). I am not sure it is a good trade for throwing away one builting, which is a perfectly reasonable function. Noam From pedronis at strakt.com Tue Nov 1 22:00:20 2005 From: pedronis at strakt.com (Samuele Pedroni) Date: Tue, 01 Nov 2005 22:00:20 +0100 Subject: [Python-Dev] a different kind of reduce... In-Reply-To: References: <4366C3BB.3010407@canterbury.ac.nz> <001301c5df20$df865f00$153dc797@oemcomputer> Message-ID: <4367D764.2090609@strakt.com> Reinhold Birkenfeld wrote: > Raymond Hettinger wrote: > >>[Martin Blais] >> >>>>I'm always--literally every time-- looking for a more functional >> >>form, >> >>>>something that would be like this: >>>> >>>> # apply dirname() 3 times on its results, initializing with p >>>> ... = repapply(dirname, 3, p) >> >>[Greg Ewing] >> >>>Maybe ** should be defined for functions so that you >>>could do things like >>> >>> up3levels = dirname ** 3 >> >>Hmm, using the function's own namespace is an interesting idea. It >>might also be a good place to put other functionals: >> >> results = f.map(data) >> newf = f.partial(somearg) > > > And we have solved the "map, filter and reduce are going away! Let's all > weep together" problem with one strike! not really, those right now work with any callable, >>> class C: ... def __call__(self, x): ... return 2*x ... >>> map(C(), [1,2,3]) [2, 4, 6] that's why attaching functionaliy as methods is not always the best solution. regards. From skip at pobox.com Tue Nov 1 21:58:37 2005 From: skip at pobox.com (skip@pobox.com) Date: Tue, 1 Nov 2005 14:58:37 -0600 Subject: [Python-Dev] python-dev sprint at PyCon In-Reply-To: <20051101143505.GE14719@rogue.amk.ca> References: <20051101143505.GE14719@rogue.amk.ca> Message-ID: <17255.55037.609312.773649@montanaro.dyndns.org> amk> Every PyCon has featured a python-dev sprint. For the past few amk> years, hacking on the AST branch has been a tradition, but we'll amk> have to come up with something new for this year's conference... This is just a comment from the peanut gallery, as it's highly unlikely I'll be in attendance, but why not continue with the AST theme? Instead of working on the AST branch, you could start to propagate the AST representation around. For example, you could use the new AST code to improve/extend/rewrite the optimization steps the compiler currently performs. Another alternative would be to rewrite Pychecker (or Pychecker 2) to operate from the AST representation. Skip From tdelaney at avaya.com Tue Nov 1 21:59:07 2005 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Wed, 2 Nov 2005 07:59:07 +1100 Subject: [Python-Dev] apparent ruminations on mutable immutables (was:PEP 351, the freeze protocol) Message-ID: <2773CAC687FD5F4689F526998C7E4E5F4DB75B@au3010avexu1.global.avaya.com> Noam, There's a simple solution to all this - write a competing PEP. One of the two competing PEPs may be accepted. FWIW, I'm +1 on PEP 351 in general, and -1 on what you've proposed. PEP 351 is simple to explain, simple to implement and leaves things under the control of the developer. I think there are still some issues to be resolved, but the basic premise is exactly what I would want of a freeze protocol. Tim Delaney From jcarlson at uci.edu Tue Nov 1 22:03:56 2005 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 01 Nov 2005 13:03:56 -0800 Subject: [Python-Dev] apparent ruminations on mutable immutables (was: PEP 351, the freeze protocol) In-Reply-To: References: <20051101104731.0389.JCARLSON@uci.edu> Message-ID: <20051101125918.0396.JCARLSON@uci.edu> > That's fine. I wish that you read my answer, think about it a little, > and just tell me in a yes or a no if you still consider it dead. I > think that I have answered all your questions, and I hope that at > least others would be convinced by them, and that at the end my > suggestion would be accepted. I still consider it dead. "If the implementation is hard to explain, it's a bad idea." Also, not all user-defined classes have a __dict__, and not all user-defined classes can have arbitrary attributes added to them. c>>> class foo(object): ... __slots__ = ['lst'] ... def __init__(self): ... self.lst = [] ... >>> a = foo() >>> a.bar = 1 Traceback (most recent call last): File "", line 1, in ? AttributeError: 'foo' object has no attribute 'bar' >>> - Josiah From tdelaney at avaya.com Tue Nov 1 22:02:04 2005 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Wed, 2 Nov 2005 08:02:04 +1100 Subject: [Python-Dev] a different kind of reduce... Message-ID: <2773CAC687FD5F4689F526998C7E4E5F4DB75C@au3010avexu1.global.avaya.com> Reinhold Birkenfeld wrote: > And we have solved the "map, filter and reduce are going away! Let's > all weep together" problem with one strike! I'm not sure if you're wildly enthusiastic, or very sarcastic. I'm not sure which I should be either ... The thought does appeal to me - especially func.partial(args). I don't see any advantage to func.map(args) over func(*args), and it loses functionality in comparison with map(func, args) (passing the function as a separate reference). Tim Delaney From mal at egenix.com Tue Nov 1 22:11:52 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 01 Nov 2005 22:11:52 +0100 Subject: [Python-Dev] PEP 328 - absolute imports (python-dev sprint at PyCon) In-Reply-To: References: <5.1.1.6.0.20051101130208.02047018@mail.telecommunity.com> Message-ID: <4367DA18.6070502@egenix.com> Guido van Rossum wrote: > On 11/1/05, Phillip J. Eby wrote: > >>At 10:22 AM 11/1/2005 -0700, Guido van Rossum wrote: >> >>>* PEP 328 - absolute/relative import >> >>I assume that references to 2.4 in that PEP should be changed to 2.5, and >>so on. > > > For the part that hasn't been implemented yet, yes. > > >>It also appears to me that the PEP doesn't record the issue brought up by >>some people about the current absolute/relative ambiguity being useful for >>packaging purposes. i.e., being able to nest third-party packages such >>that they end up seeing their dependencies, even though they're not >>installed at the "root" package level. >> >>For example, I have a package that needs Python 2.4's version of pyexpat, >>and I need it to run in 2.3, but I can't really overwrite the 2.3 pyexpat, >>so I just build a backported pyexpat and drop it in the package, so that >>the code importing it just ends up with the right thing. >> >>Of course, that specific example is okay since 2.3 isn't going to somehow >>grow absolute importing. :) But I think people brought up other examples >>besides that, it's just the one that I personally know I've done. > > > I guess this ought to be recorded. :-( > > The issue has been beaten to death and my position remains firm: > rather than playing namespace games, consistent renaming is the right > thing to do here. This becomes a trivial source edit, which beats the > problems of debugging things when it doesn't work out as expected > (which is very common due to the endless subtleties of loading > multiple versions of the same code). Just for reference, may I remind you of this thread last year: http://mail.python.org/pipermail/python-dev/2004-September/048695.html The PEP's timeline should be updated accordingly. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 01 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From noamraph at gmail.com Tue Nov 1 22:20:34 2005 From: noamraph at gmail.com (Noam Raphael) Date: Tue, 1 Nov 2005 23:20:34 +0200 Subject: [Python-Dev] apparent ruminations on mutable immutables (was:PEP 351, the freeze protocol) In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5F4DB75B@au3010avexu1.global.avaya.com> References: <2773CAC687FD5F4689F526998C7E4E5F4DB75B@au3010avexu1.global.avaya.com> Message-ID: On 11/1/05, Delaney, Timothy (Tim) wrote: > Noam, > > There's a simple solution to all this - write a competing PEP. One of > the two competing PEPs may be accepted. I will. It may take some time, though. > > FWIW, I'm +1 on PEP 351 in general, and -1 on what you've proposed. > > PEP 351 is simple to explain, simple to implement and leaves things > under the control of the developer. I think there are still some issues > to be resolved, but the basic premise is exactly what I would want of a > freeze protocol. > > Tim Delaney It is true that PEP 351 is simpler. The problem is, that thanks to PEP 351 I have found a fundamental place in which the current Python design is not optimal. It is not easy to fix it, because 1) it would require a significant change to the current implementation, and 2) people are so used to the current design that it is hard to convince them that it's flawed. The fact that discussing the design is long doesn't mean that the result, for the Python programmer, would be complicated. They won't - my suggestion will cause almost no backward-compatibility problems. Think about it - it clearly means that my suggestion simply can't make Python programming *more* complicated. Please consider new-style classes. I'm sure they required a great deal of discussion, but they are simple to use -- and they are a good thing. And I think that my suggestion would make things easier, more than the new-style-classes change did. Features of new-style classes are an advanced topic. The questions, "why can't I change my strings?" "why do you need both a tuple and a list?" and maybe "why can't I add my list to a set", are fundamental ones, which would all not be asked at all if my suggestion is accepted. Noam From jcarlson at uci.edu Tue Nov 1 22:29:29 2005 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 01 Nov 2005 13:29:29 -0800 Subject: [Python-Dev] a different kind of reduce... In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5F4DB75C@au3010avexu1.global.avaya.com> References: <2773CAC687FD5F4689F526998C7E4E5F4DB75C@au3010avexu1.global.avaya.com> Message-ID: <20051101131830.0399.JCARLSON@uci.edu> "Delaney, Timothy (Tim)" wrote: > > Reinhold Birkenfeld wrote: > > > And we have solved the "map, filter and reduce are going away! Let's > > all weep together" problem with one strike! > > I'm not sure if you're wildly enthusiastic, or very sarcastic. > > I'm not sure which I should be either ... > > The thought does appeal to me - especially func.partial(args). I don't > see any advantage to func.map(args) over func(*args), and it loses > functionality in comparison with map(func, args) (passing the function > as a separate reference). I was under the impression that: fcn.(...) would perform equivalently as (fcn, ...) does now. So all the following would be equivalent... func.map(args) map(func, args) [func(i) for i in args] Me, I still use map, so seeing it as fcn.map(...) instead of map(fcn,...) sounds good to me...though it does have the ugly rub of suggesting that None.map/filter should exist, which I'm not really happy about. In regards to the instance __call__ method, it seems reasonable to require users to implement their own map/filter/reduce call. - Josiah From noamraph at gmail.com Tue Nov 1 22:30:48 2005 From: noamraph at gmail.com (Noam Raphael) Date: Tue, 1 Nov 2005 23:30:48 +0200 Subject: [Python-Dev] apparent ruminations on mutable immutables (was: PEP 351, the freeze protocol) In-Reply-To: <20051101125918.0396.JCARLSON@uci.edu> References: <20051101104731.0389.JCARLSON@uci.edu> <20051101125918.0396.JCARLSON@uci.edu> Message-ID: On 11/1/05, Josiah Carlson wrote: ... > > I still consider it dead. > "If the implementation is hard to explain, it's a bad idea." It is sometimes true, but not always. It may mean two other things: 1. The one trying to explain is not talented enough. 2. The implementation is really not very simple. A hash table, used so widely in Python, is really not a simple idea, and it's not that easy to explain. > > Also, not all user-defined classes have a __dict__, and not all > user-defined classes can have arbitrary attributes added to them. > > c>>> class foo(object): > ... __slots__ = ['lst'] > ... def __init__(self): > ... self.lst = [] > ... > >>> a = foo() > >>> a.bar = 1 > Traceback (most recent call last): > File "", line 1, in ? > AttributeError: 'foo' object has no attribute 'bar' > >>> It doesn't matter. It only means that the implementation would have to make frozen copies also of __slots__ items, when freezing a user-defined class. I am afraid that this question proves that I didn't convey my idea to you. If you like, please forgive my inability to explain it clearly, and try again to understand my idea, by going over what I wrote again, and thinking on it. You can also wait for the PEP that I intend to write. And you can also forget about it, if you don't want to bother with it - you've already helped a lot. Noam From guido at python.org Tue Nov 1 22:40:40 2005 From: guido at python.org (Guido van Rossum) Date: Tue, 1 Nov 2005 14:40:40 -0700 Subject: [Python-Dev] a different kind of reduce... In-Reply-To: <001301c5df20$df865f00$153dc797@oemcomputer> References: <4366C3BB.3010407@canterbury.ac.nz> <001301c5df20$df865f00$153dc797@oemcomputer> Message-ID: > [Greg Ewing] > > Maybe ** should be defined for functions so that you > > could do things like > > > > up3levels = dirname ** 3 [Raymond Hettinger] > Hmm, using the function's own namespace is an interesting idea. It > might also be a good place to put other functionals: > > results = f.map(data) > newf = f.partial(somearg) Sorry to rain on everybody's parade, but I don't think so. There are many different types of callables. This stuff would only work if they all implemented the same API. That's unlikely to happen. A module with functions to implement the various functional operations has much more potential. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From s.percivall at chello.se Wed Nov 2 00:14:20 2005 From: s.percivall at chello.se (Simon Percivall) Date: Wed, 2 Nov 2005 00:14:20 +0100 Subject: [Python-Dev] a different kind of reduce... In-Reply-To: References: <4366C3BB.3010407@canterbury.ac.nz> <001301c5df20$df865f00$153dc797@oemcomputer> Message-ID: <6FF0C116-C55B-43CC-AA21-2A7D72E09545@chello.se> On 1 nov 2005, at 22.40, Guido van Rossum wrote: >> [Greg Ewing] >>> Maybe ** should be defined for functions so that you >>> could do things like >>> >>> up3levels = dirname ** 3 > > [Raymond Hettinger] >> Hmm, using the function's own namespace is an interesting idea. It >> might also be a good place to put other functionals: >> >> results = f.map(data) >> newf = f.partial(somearg) > > Sorry to rain on everybody's parade, but I don't think so. There are > many different types of callables. This stuff would only work if they > all implemented the same API. That's unlikely to happen. A module with > functions to implement the various functional operations has much more > potential. Perhaps then a decorator that uses these functions? //Simon From noamraph at gmail.com Wed Nov 2 02:21:38 2005 From: noamraph at gmail.com (Noam Raphael) Date: Wed, 2 Nov 2005 03:21:38 +0200 Subject: [Python-Dev] Why should the default hash(x) == id(x)? Message-ID: Hello, While writing my PEP about unifying mutable and immutable, I came upon this: Is there a reason why the default __hash__ method returns the id of the objects? It is consistent with the default __eq__ behaviour, which is the same as "is", but: 1. It can easily become inconsistent, if someone implements __eq__ and doesn't implement __hash__. 2. It is confusing: even if someone doesn't implement __eq__, he may see that it is suitable as a key to a dict, and expect it to be found by other objects with the same "value". 3. If someone does want to associate values with objects, he can explicitly use id: dct[id(x)] = 3. This seems to better explain what he wants. Now, I just thought of a possible answer: "because he wants to store in his dict both normal objects and objects of his user-defined type, which turn out to be not equal to any other object." This leads me to another question: why should the default __eq__ method be the same as "is"? If someone wants to check if two objects are the same object, that's what the "is" operator is for. Why not make the default __eq__ really compare the objects, that is, their dicts and their slot-members? I would be happy to get answers. Noam From nnorwitz at gmail.com Wed Nov 2 03:23:23 2005 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 1 Nov 2005 18:23:23 -0800 Subject: [Python-Dev] python-dev sprint at PyCon In-Reply-To: <17255.55037.609312.773649@montanaro.dyndns.org> References: <20051101143505.GE14719@rogue.amk.ca> <17255.55037.609312.773649@montanaro.dyndns.org> Message-ID: On 11/1/05, skip at pobox.com wrote: > > This is just a comment from the peanut gallery, as it's highly unlikely I'll > be in attendance, but why not continue with the AST theme? Instead of > working on the AST branch, you could start to propagate the AST > representation around. For example, you could use the new AST code to > improve/extend/rewrite the optimization steps the compiler currently > performs. Another alternative would be to rewrite Pychecker (or Pychecker > 2) to operate from the AST representation. That's an excellent suggestion. I think I will borrow the time machine and add it to the wiki. :-) It's up on the wiki. Brett also added an item for the peephole optimizer. Everyone should add whatever they think are good ideas, even if they don't plan to attend the sprints. n From mcherm at mcherm.com Wed Nov 2 14:48:55 2005 From: mcherm at mcherm.com (Michael Chermside) Date: Wed, 02 Nov 2005 05:48:55 -0800 Subject: [Python-Dev] apparent ruminations on mutable immutables (was:PEP 351, the freeze protocol) Message-ID: <20051102054855.st4vtcvrorogggc8@login.werra.lunarpages.com> Josiah Carlson writes: > If you make such a suggestion, I would offer that you create a new PEP, > because this discussion has gone beyond PEP 351, and has wandered into > the realm of "What other kinds of objects would be interesting to have > in a Python-like system?" Noam Raphael replies: > That is a good suggestion, and I have already started to write one. It > takes me a long time, but I hope I will manage. My thanks to both of you... following this conversation has been an educational experience. Just for the record, I wanted to chime in with my own opinion formed after following the full interchange. I think Noam's propsal is very interesting. I like the idea of allowing both "frozen" (ie, immutable) and mutable treatments for the same object. I think that C++'s version of this concept (the "const" modifier) has, on balance, been only a very limited success. I find myself convinced by Noam's claims that many common use patterns either (1) only use mutables, or (2) only use immutables, or (3) only use immutable copies temporarily and avoid mutating while doing so. Any such use patterns (particularly use (3)) would benefit from the presence of an efficient method for creating an immutable copy of a mutable object which avoids the copy where possible. However... it seems to me that what is being described here is not Python. Python is a wonderful language, but it has certain characteristics, like extremely dynamic behavior and close integration with underlying system methods (C in CPython, Java in Jython, etc) that seem to me to make this particular feature a poor fit. That's OK... not all languages need to be Python! I would encourage you (Noam) to go ahead and explore this idea of yours. You might wind up building a new language from scratch (in which case I strongly encourage you to borrow _syntax_ from Python -- its syntax is more usable than that of any other language I know of). Or perhaps you will prefer to take CPython and make minor modifications. This kind of experimentation is allowed (open source) and even encouraged... consider Christian Tismer's Stackless -- a widely admired variant of CPython which is unlikely to ever become part of the core, but is nevertheless an important part of the vivrant Python community. You might even be interested in starting, instead, with PyPy -- an large project which has as its main goal producing an implementation of Python which is easy to modify so as to support just this kind of experimentation. You are also welcome to submit a PEP for modifying Python (presumably CPython, Jython, Iron Python, and all other implementations). However, I think such a PEP would be rejected. Building your own thing that works well with Python would NOT be rejected. The idea is interesting, and it _may_ be sound; only an actual implementation could prove this either way. -- Michael Chermside From mcherm at mcherm.com Wed Nov 2 18:39:44 2005 From: mcherm at mcherm.com (Michael Chermside) Date: Wed, 02 Nov 2005 09:39:44 -0800 Subject: [Python-Dev] Why should the default hash(x) == id(x)? Message-ID: <20051102093944.8jhktwq4e98g4444@login.werra.lunarpages.com> Noam Raphael writes: > Is there a reason why the default __hash__ method returns the id of the objects? > > It is consistent with the default __eq__ behaviour, which is the same > as "is", but: > > 1. It can easily become inconsistent, if someone implements __eq__ and > doesn't implement __hash__. > 2. It is confusing: even if someone doesn't implement __eq__, he may > see that it is suitable as a key to a dict, and expect it to be found > by other objects with the same "value". > 3. If someone does want to associate values with objects, he can > explicitly use id: > dct[id(x)] = 3. This seems to better explain what he wants. Your first criticism is valid... it's too bad that there isn't a magical __hash__ function that automatically derived its behavior from __eq__. To your second point, I would tell this user to read the requirements. And your third point isn't a criticism, just an alternative. But to answer your question, the reason that the default __hash__ returns the ID in CPython is just that this works. In Jython, I belive that the VM provides a native hash method, and __hash__ uses that instead of returning ID. Actually, it not only works, it's also FAST (which is important... many algorithms prefer that __hash__ being O(1)). I can't imagine what you would propose instead. Keep in mind that the requirements are that __hash__ must return a value which distinguishes the object. So, for instance, two mutable objects with identical values MUST (probably) return different __hash__ values as they are distinct objects. > This leads me to another question: why should the default __eq__ > method be the same as "is"? Another excellent question. The answer is that this is the desired behavior of the language. Two user-defined object references are considered equal if and only if (1) they are two references to the same object, or (2) the user who designed it has specified a way to compare objects (implemented __eq__) and it returns a True value. > Why not make the default __eq__ really compare the objects, that is, > their dicts and their slot-members? Short answer: not the desired behavior. Longer answer: there are three common patterns in object design. There are "value" objects, which should be considered equal if all fields are equal. There are "identity" objects which are considered equal only when they are the same object. And then there are (somewhat less common) "value" objects in which a few fields don't count -- they may be used for caching a pre-computed result for example. The default __eq__ behavior has to cater to one of these -- clearly either "value" objects or "identity" objects. Guido chose to cater to "identity" objects believing that they are actually more common in most situations. A beneficial side-effect is that the default behavior of __eq__ is QUITE simple to explain, and if the implementation is easy to explain then it may be a good idea. -- Michael Chermside From jcarlson at uci.edu Wed Nov 2 18:46:09 2005 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 02 Nov 2005 09:46:09 -0800 Subject: [Python-Dev] Why should the default hash(x) == id(x)? In-Reply-To: References: Message-ID: <20051102092422.F283.JCARLSON@uci.edu> Noam Raphael wrote: > > Hello, > > While writing my PEP about unifying mutable and immutable, I came upon this: > > Is there a reason why the default __hash__ method returns the id of the objects? A quick search in the list archives via google search "site:mail.python.org object __hash__" Says that Guido wanted to remove the default __hash__ method for object in Python 2.4, but that never actually happened. http://www.python.org/sf/660098 http://mail.python.org/pipermail/python-dev/2003-December/041375.html There may be more code which relies on the default behavior now, but fixing such things is easy. > Now, I just thought of a possible answer: "because he wants to store > in his dict both normal objects and objects of his user-defined type, > which turn out to be not equal to any other object." Which is a use-case, but a use-case which isn't always useful. Great for singleton/default arguments that no one should ever pass, not quite so good when you need the /original key/ (no copies) in order to get at a value in a dictionary - but that could be something that someone wants. > This leads me to another question: why should the default __eq__ > method be the same as "is"? If someone wants to check if two objects > are the same object, that's what the "is" operator is for. Why not > make the default __eq__ really compare the objects, that is, their > dicts and their slot-members? Using 'is' makes sense when the default hash is id (and actually in certain other cases as well). Actually comparing the contents of an object is certainly not desireable with the default hash, and probably not desireable in the general case because equality doesn't always depend on /all/ attributes of extension objects. Explicit is better than implicit. In the face of ambiguity, refuse the temptation to guess. I believe the current behavior of __eq__ is more desireable than comparing contents, as this may result in undesireable behavior (recursive compares on large nested objects are now slow, which used to be fast because default methods wouldn't cause a recursive comparison at all). As for removing the default __hash__ for objects, I'm actually hovering around a -0, if only because it is sometimes useful to generate unique keys for dictionaries (which can be done right now with object() ), and I acknowledge that it would be easy to subclass and use that instead. - Josiah From noamraph at gmail.com Wed Nov 2 20:04:50 2005 From: noamraph at gmail.com (Noam Raphael) Date: Wed, 2 Nov 2005 21:04:50 +0200 Subject: [Python-Dev] apparent ruminations on mutable immutables (was:PEP 351, the freeze protocol) In-Reply-To: <20051102054855.st4vtcvrorogggc8@login.werra.lunarpages.com> References: <20051102054855.st4vtcvrorogggc8@login.werra.lunarpages.com> Message-ID: Thank you for your encouraging words! I am currently working on a PEP. I am sure that writing it is a good idea, and that it would help with explaining this idea both to others and to myself. What I already wrote makes me think that it can be accomplished with no really large changes to the language - only six built-in types are affected, and there is no reason why existing code, both in C and in Python, would stop working. I hope others would be interested in the idea too, when I finish writing the PEP draft, so it would be discussed. Trying the idea with PyPy is a really nice idea - it seems that it would be much simpler to implement, and I'm sure that learning PyPy would be interesting. Thanks again, and I would really like to hear your comments when I post the PEP draft, Noam From runehol at ping.uio.no Wed Nov 2 20:18:52 2005 From: runehol at ping.uio.no (Rune Holm) Date: Wed, 02 Nov 2005 20:18:52 +0100 Subject: [Python-Dev] Optimizations on the AST representation Message-ID: <4369111C.5060508@ping.uio.no> Hi, I'm a norwegian applied mathematics student with an interest in compilers, and I've been a long-time python user and a python-dev lurker for some time. I'm very happy that you've integrated the AST branch into mainline, but I noticed that the AST compiler does not perform much optimization yet, so I though I'd take a crack at it. I just submitted the following patches: http://www.python.org/sf/1346214 http://www.python.org/sf/1346238 which adds better dead code elimination and constant folding of the AST representation to Python. The constant folding patch adds two new files, Include/optimize.h and Python/optimize.c, which includes a general visitor interface abstracted from exisiting visitor code for easy optimization pass creation. The code is in new files in order to make room for more AST optimization passes, and since Python/compile.c is already quite crowded with byte code generation and bytecode optimization. If desired, this patch could changed to add code to Python/compile.c instead. Further work: A limited form of type interference (e.g. as a synthesized attribute) could be very useful for further optimizations. Since python allows operator overloading, it isn't safe to perform strength reductions on expressions with operands of unknown type, as there is no way to know if algebraic identities will hold. However, if we can infer from the context that expressions have the type of int, float or long, many optimizations become possible, for instance: x**2 => x*x x*2 => x+x x*0 => 0 x*1 => x 4*x + 5*x => 9*x (this optimization actually requires common subexpression elimination for the general case, but simple cases can be performed without this) and so forth. Another interesting optimization that can potensially bring a lot of additional speed is hoisting of loop invariants, since calling python methods involves allocating and creating a method-wrapper object. An informal test shows that optimizing lst = [] for i in range(10): lst.append(i+1) into lst = [] tmp = lst.append for i in range(10): tmp(i+1) will yield a 10% speed increase. This operation is of course not safe with arbitrary types, but with the above type interference, we could perform this operation if the object is of a type that disallows attribute assignment, for instance lists, tuples, strings and unicode strings. Regards, Rune Holm From noamraph at gmail.com Wed Nov 2 20:26:52 2005 From: noamraph at gmail.com (Noam Raphael) Date: Wed, 2 Nov 2005 21:26:52 +0200 Subject: [Python-Dev] Why should the default hash(x) == id(x)? In-Reply-To: <20051102092422.F283.JCARLSON@uci.edu> References: <20051102092422.F283.JCARLSON@uci.edu> Message-ID: On 11/2/05, Josiah Carlson wrote: ... > > A quick search in the list archives via google search > "site:mail.python.org object __hash__" > Says that Guido wanted to remove the default __hash__ method for object > in Python 2.4, but that never actually happened. > > http://www.python.org/sf/660098 > http://mail.python.org/pipermail/python-dev/2003-December/041375.html > > There may be more code which relies on the default behavior now, but > fixing such things is easy. > Cool! If Guido also thinks that it should be gone, who am I to argue... (Seriously, I am in favor of removing it. I really think that it is confusing.) And if backwards-compatibility is a problem: You can, in Python 2.5, show a warning when the default __hash__ method is being called, saying that it is going to disappear in Python 2.6. [Snip - I will open a new thread about the equality operator] > As for removing the default __hash__ for objects, I'm actually hovering > around a -0, if only because it is sometimes useful to generate unique > keys for dictionaries (which can be done right now with object() ), and > I acknowledge that it would be easy to subclass and use that instead. > I can suggest a new class, that will help you in the cases that you do want a dict of identities: class ref(object): def __init__(self, obj): self._obj = obj def __call__(self): return self._obj def __eq__(self, other): return self._obj is other._obj def __hash__(self): return hash(id(self._obj)) It has the advantage over using ids as keys, that it saves a reference to the object, so it won't be killed. It lets you make a dict of object identities just as easily as before, in a more explicit and error-prone way. Perhaps it should become a builtin? Noam From rmunn at pobox.com Wed Nov 2 20:46:05 2005 From: rmunn at pobox.com (Robin Munn) Date: Wed, 02 Nov 2005 13:46:05 -0600 Subject: [Python-Dev] Problems with revision 4077 of new SVN repository Message-ID: <4369177D.3020000@pobox.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm trying to mirror the brand-new Python SVN repository with SVK, to better be able to track both the trunk and the various branches. Since I'm not a Python developer and don't have svn+ssh access, I'm doing so over http. The process fails when trying to fetch revision 4077, with the following error message: "RA layer request failed: REPORT request failed on 'projects/!svn/bc/41373/python': The REPORT request returned invalid XML in the response: XML parse error at line 7: not well-formed (invalid token) (/projects/!svn/bc/41373/python)" The thread at http://svn.haxx.se/dev/archive-2004-07/0793.shtml suggests that the problem may lie in the commit message for revision 4077: if it has a character in the 0x01-0x1f range (which are invalid XML), then Subversion methods like http: will fail to retrieve it, while methods like file: will succeed. I haven't tried svn+ssh: since I don't have an SSH key on the server. Trying "svn log -r 4077 http://svn.python.org/projects/python/" also fails: subversion/libsvn_ra_dav/util.c:780: (apr_err=175002) svn: REPORT request failed on '/projects/!svn/bc/4077/python' subversion/libsvn_ra_dav/util.c:760: (apr_err=175002) svn: The REPORT request returned invalid XML in the response: XML parse error at line 7: not well-formed (invalid token) (/projects/!svn/bc/4077/python) When I visit http://svn.python.org/view/python/?rev=4077, I can see the offending log message. Sure enough, there's a 0x1b character in it, between the space after "Added" and the "h" immediately before the word "Moved". This problem can be fixed by someone with root permissions on the SVN server logging in and running the following: echo "New commit message goes here" > new-message.txt svnadmin setlog --bypass-hooks -r 4077 /path/to/repos new-message.txt If there are other, similar problems later in the SVN repository, I was unable to find them because the SVK mirror process consistently halts at revision 4077. If revision 4077 is fixed and I turn up other log problems, I'll report them as well. - -- Robin Munn rmunn at pobox.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (Darwin) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDaRd46OLMk9ZJcBQRApjAAJ9K3Y5z1q4TulqwVjmZTZb9ZgY31ACcD8RI fNFmGL2U4XaIKa2n6UUyxEA= =tEbq -----END PGP SIGNATURE----- From noamraph at gmail.com Wed Nov 2 21:36:54 2005 From: noamraph at gmail.com (Noam Raphael) Date: Wed, 2 Nov 2005 22:36:54 +0200 Subject: [Python-Dev] Should the default equality operator compare values instead of identities? Message-ID: I think it should. (I copy here messages from the thread about the default hash method.) On 11/2/05, Michael Chermside wrote: > > Why not make the default __eq__ really compare the objects, that is, > > their dicts and their slot-members? > > Short answer: not the desired behavior. Longer answer: there are > three common patterns in object design. There are "value" objects, > which should be considered equal if all fields are equal. There are > "identity" objects which are considered equal only when they are > the same object. And then there are (somewhat less common) "value" > objects in which a few fields don't count -- they may be used for > caching a pre-computed result for example. The default __eq__ > behavior has to cater to one of these -- clearly either "value" > objects or "identity" objects. Guido chose to cater to "identity" > objects believing that they are actually more common in most > situations. A beneficial side-effect is that the default behavior > of __eq__ is QUITE simple to explain, and if the implementation is > easy to explain then it may be a good idea. > This is a very nice observation. I wish to explain why I think that the default __eq__ should compare values, not identities. 1. If you want to compare identities, you can always use "is". There is currently no easy way to compare your user-defined classes by value, in case they happen to be "value objects", in Michael's terminology - you have to compare every single member. (Comparing the __dict__ attributes is ugly, and will not always work). If the default were to compare the objects by value, and they happen to be "identity objects", you can always do: def __eq__(self, other): return self is other 2. I believe that counter to what Michael said, "value objects" are more common than "identity objects", at least when talking about user-defined classes, and especially when talking about simple user-defined classes, where the defaults are most important, since the writer wouldn't care to define all the appropriate protocols. (this was a long sentence) Can you give examples of common "identity objects"? I believe that they are usually dealing with some input/output, that is, with things that interact with the environment (files, for example). I believe almost all "algorithmic" classes are "value objects". And I think that usually, comparison based on value will give the correct result for "identity objects" too, since if they do I/O, they will usually hold a reference to an I/O object, like file, which is an "identity object" by itself. This means that the comparison will compare those objects, and return false, since the I/O objects they hold are not the same one. 3. I think that value-based comparison is also quite easy to explain: user-defined classes combine functions with a data structure. In Python, the "data structure" is simply member names which reference other objects. The default, value-based, comparison, checks if two objects have the same member names, and that they are referencing equal (by value) objects, and if so, returns True. I think that explaining this is not harder than explaining the current dict comparison. Now, for Josiah's reply: On 11/2/05, Josiah Carlson wrote: > > This leads me to another question: why should the default __eq__ > > method be the same as "is"? If someone wants to check if two objects > > are the same object, that's what the "is" operator is for. Why not > > make the default __eq__ really compare the objects, that is, their > > dicts and their slot-members? > > Using 'is' makes sense when the default hash is id (and actually in > certain other cases as well). Actually comparing the contents of an > object is certainly not desireable with the default hash, and probably > not desireable in the general case because equality doesn't always > depend on /all/ attributes of extension objects. > > Explicit is better than implicit. > In the face of ambiguity, refuse the temptation to guess. > I hope that the default hash would stop being id, as Josiah showed that Guido decided, so let's don't discuss it. Now, about the good point that sometimes the state doesn't depend on all the attributes. Right. But the current default doesn't compare them well too - you have no escape from writing an equality operator by yourself. And I think this is not the common case. I think that the meaning of "in the face of ambiguity, refuse the temptation to guess" is that you should not write code that changes its behaviour according to what the user will do, based on your guess as to what he meant. This is not the case - the value-based comparison is strictly defined. It may just not be what the user would want - and in most cases, I think it will. "Explicit is better than implicit" says only "better". identity-based comparison is just as implicit as value-based comparison. (I want to add that there is a simple way to support value-based comparison when some members don't count, by writing a metaclass that will check if your class has a member like __non_state_members__ = ["_calculated_hash", "debug_member"] and if so, would not compare them in the default equality-testing method. I would say that this can even be made the behavior of the default type.) > I believe the current behavior of __eq__ is more desireable than > comparing contents, as this may result in undesireable behavior > (recursive compares on large nested objects are now slow, which used to > be fast because default methods wouldn't cause a recursive comparison at > all). But if the default method doesn't do what you want, it doesn't matter how fast it is. Remember that it's very easy to make recursive comparisons, by comparing lists for example, and it hasn't disturbed anyone. To summarize, I think that value-based equality testing would usually be what you want, and currently implementing it is a bit of a pain. Concerning backwards-compatibility: show a warning in Python 2.5 when the default equality test is being made, and change it in Python 2.6. Comments, please! Thanks, Noam From noamraph at gmail.com Wed Nov 2 22:11:25 2005 From: noamraph at gmail.com (Noam Raphael) Date: Wed, 2 Nov 2005 23:11:25 +0200 Subject: [Python-Dev] Should the default equality operator compare values instead of identities? In-Reply-To: References: Message-ID: I've looked for classes in my /usr/lib/python2.4 directory. I won't go over all the 7346 classes that were found there, but let's see: "identity objects" that will continue to work because they contain other "identity objects" ======================== SocketServer, and everything which inherits from it (like HTTPServer) Queue csv (contains _csv objects) "value objects" that would probably gain a meaningful equality operator ============================================ StringIO ConfigParser markupbase, HTMLParser HexBin, BinHex cgi.FieldStorage AST Nodes others ====== Cookie - inherits from dict its __eq__ method. I'll stop here. I was not strictly scientific, because I chose classes that I thought that I might guess what they do easily, and perhaps discarded classes that didn't look interesting to me. But I didn't have any bad intention when choosing the classes. I have seen no class that the change would damage its equality operator. I have seen quite a lot of classes which didn't define an equality operator, and that a value-based comparison would be the right way to compare them. I'm getting more convinced in my opinion. Noam From noamraph at gmail.com Wed Nov 2 22:18:58 2005 From: noamraph at gmail.com (Noam Raphael) Date: Wed, 2 Nov 2005 23:18:58 +0200 Subject: [Python-Dev] Should the default equality operator compare valuesinstead of identities? In-Reply-To: <001101c5dff0$462fa5c0$153dc797@oemcomputer> References: <001101c5dff0$462fa5c0$153dc797@oemcomputer> Message-ID: On 11/2/05, Raymond Hettinger wrote: > > Should the default equality operator compare valuesinstead of > identities? > > No. Look back into last year's python-dev postings where we agreed that > identity would always imply equality. There were a number of practical > reasons. Also, there are a number of places in CPython where that > assumption is implicit. > Perhaps you've meant something else, or I didn't understand? Identity implying equality is true also in value-based comparison. If the default __eq__ operator compares by value, I would say that it would do something like: def __eq__(self, other): if self is other: return True if type(self) is not type(other): return False (compare the __dict__ and any __slots__, and if they are all ==, return True.) Noam From martin at v.loewis.de Wed Nov 2 23:10:12 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 02 Nov 2005 23:10:12 +0100 Subject: [Python-Dev] Problems with revision 4077 of new SVN repository In-Reply-To: <4369177D.3020000@pobox.com> References: <4369177D.3020000@pobox.com> Message-ID: <43693944.3090803@v.loewis.de> Robin Munn wrote: > echo "New commit message goes here" > new-message.txt > svnadmin setlog --bypass-hooks -r 4077 /path/to/repos new-message.txt Thanks for pointing that out, and for giving those instructions. I now corrected the log message. Regards, Martin From rmunn at pobox.com Thu Nov 3 00:14:50 2005 From: rmunn at pobox.com (Robin Munn) Date: Wed, 02 Nov 2005 17:14:50 -0600 Subject: [Python-Dev] Problems with revision 4077 of new SVN repository In-Reply-To: <43693944.3090803@v.loewis.de> References: <4369177D.3020000@pobox.com> <43693944.3090803@v.loewis.de> Message-ID: <4369486A.8090107@pobox.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: > Robin Munn wrote: > >> echo "New commit message goes here" > new-message.txt >> svnadmin setlog --bypass-hooks -r 4077 /path/to/repos new-message.txt > > > Thanks for pointing that out, and for giving those instructions. > I now corrected the log message. Revision 4077 is fine now. However, the same problem exists in revision 4284, which has a 0x01 character before the word "add". Same solution: echo "New commit message goes here" > new-message.txt svnadmin setlog --bypass-hooks -r 4284 /path/to/repos new-message.txt If there are two errors of the same type within about 200 revisions, there may be more. I'm currently running "svn log" on every revision in the Python SVN repository to see if I find any more errors of this type, so that I don't have to hunt them down one-by-one by rerunning SVK. I'll post my findings when I'm done. - -- Robin Munn rmunn at pobox.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (Darwin) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDaUho6OLMk9ZJcBQRAg5eAJ9cJTPKX69DhXJyoT/cDV5GmZlC3QCfRj/E wCix8IYU8xbh5/Ibnpa+kg4= =+jLR -----END PGP SIGNATURE----- From greg.ewing at canterbury.ac.nz Thu Nov 3 01:39:44 2005 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 03 Nov 2005 13:39:44 +1300 Subject: [Python-Dev] Why should the default hash(x) == id(x)? In-Reply-To: References: Message-ID: <43695C50.5070600@canterbury.ac.nz> Noam Raphael wrote: > 3. If someone does want to associate values with objects, he can > explicitly use id: > dct[id(x)] = 3. This is fragile. Once all references to x are dropped, it is possible for another object to be created having the same id that x used to have. The dict now unintentionally references the new object. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing at canterbury.ac.nz +--------------------------------------+ From jcarlson at uci.edu Thu Nov 3 02:16:40 2005 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 02 Nov 2005 17:16:40 -0800 Subject: [Python-Dev] Should the default equality operator compare values instead of identities? In-Reply-To: References: Message-ID: <20051102125437.F290.JCARLSON@uci.edu> Noam Raphael wrote: > On 11/2/05, Josiah Carlson wrote: > > I believe the current behavior of __eq__ is more desireable than > > comparing contents, as this may result in undesireable behavior > > (recursive compares on large nested objects are now slow, which used to > > be fast because default methods wouldn't cause a recursive comparison at > > all). > > But if the default method doesn't do what you want, it doesn't matter > how fast it is. Remember that it's very easy to make recursive > comparisons, by comparing lists for example, and it hasn't disturbed > anyone. Right, but lists (dicts, tuples, etc.) are defined as containers, and their comparison operation is defined on their contents. Objects are not defined as containers in the general case, so defining comparisons based on their contents (as opposed to identity) is just one of the two assumptions to be made. I personally like the current behavior, and I see no /compelling/ reason to change it. You obviously feel so compelled for the behavior to change that you are willing to express your desires. How about you do something more productive and produce a patch which implements the changes you want, verify that it passes tests in the standard library, then post it on sourceforge. If someone is similarly compelled and agrees with you (so far I've not seen any public support for your proposal by any of the core developers), the discussion will restart, and it will be decided (not by you or I). > To summarize, I think that value-based equality testing would usually > be what you want, and currently implementing it is a bit of a pain. Actually, implementing value-based equality testing, when you have a finite set of values you want to test, is quite easy. def __eq__(self, other): for i in self.__cmp_eq__: if getattr(self, i) != getattr(other, i): return False return True With a simple metaclass that discovers all of those values automatically, and/or your own protocol for exclusion, and you are done. Remember, not all 5-line functions should become builtin/default behavior, and this implementation shows that it is not a significant burdon for you (or anyone else) to implement this in your own custom library. - Josiah P.S. One thing that you should remember is that even if your patch is accepted, and even if this is desireable, Python 2.5 is supposed to be released sometime next year (spring/summer?), and because it is a backwards incompatible change, would need at least 2.6-2.7 before it becomes the default behavior without a __future__ import, which is another 3-4 years down the line. I understand you are passionate, really I do (you should see some of my proposals), but by the time these things get around to getting into mainline Python, there are high odds that you probably won't care about them much anymore (I've come to feel that way myself about many of my proposals), and I think it is a good idea to attempt to balance - when it comes to Python - "Now is better than never." and "Although never is often better than *right* now." Removing __hash__, changing __eq__, and trying to get in copy-on-write freezing (which is really copy-and-cache freezing), all read to me like "We gotta do this now!", which certainly isn't helping the proposal. From rmunn at pobox.com Thu Nov 3 03:13:33 2005 From: rmunn at pobox.com (Robin Munn) Date: Wed, 02 Nov 2005 20:13:33 -0600 Subject: [Python-Dev] Problems with revision 4077 of new SVN repository In-Reply-To: <4369486A.8090107@pobox.com> References: <4369177D.3020000@pobox.com> <43693944.3090803@v.loewis.de> <4369486A.8090107@pobox.com> Message-ID: <4369724D.8060001@pobox.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Robin Munn wrote: > Revision 4077 is fine now. However, the same problem exists in revision > 4284, which has a 0x01 character before the word "add". Same solution: > > echo "New commit message goes here" > new-message.txt > svnadmin setlog --bypass-hooks -r 4284 /path/to/repos new-message.txt > > If there are two errors of the same type within about 200 revisions, > there may be more. I'm currently running "svn log" on every revision in > the Python SVN repository to see if I find any more errors of this type, > so that I don't have to hunt them down one-by-one by rerunning SVK. I'll > post my findings when I'm done. My script is up to revision 17500 with no further problems found; I now believe that 4077 and 4284 were isolated cases. Once 4284 is fixed, it should now be possible to SVK-mirror the entire repository. - -- Robin Munn rmunn at pobox.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (Darwin) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDaXJF6OLMk9ZJcBQRAtZpAJ9iE1SlRJiQQOdIuBFuvjmQG3gshACgl9/A vbsGD0bX3NCirQC5qtxdLYo= =sgk/ -----END PGP SIGNATURE----- From martin at v.loewis.de Thu Nov 3 08:57:30 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 03 Nov 2005 08:57:30 +0100 Subject: [Python-Dev] Problems with revision 4077 of new SVN repository In-Reply-To: <4369724D.8060001@pobox.com> References: <4369177D.3020000@pobox.com> <43693944.3090803@v.loewis.de> <4369486A.8090107@pobox.com> <4369724D.8060001@pobox.com> Message-ID: <4369C2EA.6030407@v.loewis.de> Robin Munn wrote: >>Revision 4077 is fine now. However, the same problem exists in revision >>4284, which has a 0x01 character before the word "add". Same solution: I now have fixed that as well. Regards, Martin From rmunn at pobox.com Thu Nov 3 09:07:43 2005 From: rmunn at pobox.com (Robin Munn) Date: Thu, 03 Nov 2005 02:07:43 -0600 Subject: [Python-Dev] Problems with revision 4077 of new SVN repository In-Reply-To: <4369C2EA.6030407@v.loewis.de> References: <4369177D.3020000@pobox.com> <43693944.3090803@v.loewis.de> <4369486A.8090107@pobox.com> <4369724D.8060001@pobox.com> <4369C2EA.6030407@v.loewis.de> Message-ID: <4369C54F.3050803@pobox.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: > Robin Munn wrote: > >>> Revision 4077 is fine now. However, the same problem exists in revision >>> 4284, which has a 0x01 character before the word "add". Same solution: > > > I now have fixed that as well. > > Regards, > Martin And my script just finished running, with no further errors of this type found. So doing an SVK mirror of the repository should work now, barring any further surprises. I'm starting the SVK sync now; we'll see what happens. Thanks for fixing these! - -- Robin Munn rmunn at pobox.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (Darwin) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDacVN6OLMk9ZJcBQRApUbAJ9+Ly5vPr8HRmoRbwJ3po4IWe8PBwCePTdm XNx8HGqPvs7fwahHuJSogMw= =a6Nc -----END PGP SIGNATURE----- From mwh at python.net Thu Nov 3 13:48:06 2005 From: mwh at python.net (Michael Hudson) Date: Thu, 03 Nov 2005 12:48:06 +0000 Subject: [Python-Dev] PyPy 0.8.0 is released! Message-ID: <2mbr113njt.fsf@starship.python.net> pypy-0.8.0: Translatable compiler/parser and some more speed ============================================================== The PyPy development team has been busy working and we've now packaged our latest improvements, completed work and new experiments as version 0.8.0, our third public release. The highlights of this third release of PyPy are: - Translatable parser and AST compiler. PyPy now integrates its own compiler based on Python own 'compiler' package but with a number of fixes and code simplifications in order to get it translated with the rest of PyPy. This makes using the translated pypy interactively much more pleasant, as compilation is considerably faster than in 0.7.0. - Some Speed enhancements. Translated PyPy is now about 10 times faster than 0.7 but still 10-20 times slower than CPython on pystones and other benchmarks. At the same time, language compliancy has been slightly increased compared to 0.7 which had already reached major CPython compliancy goals. - Some experimental features are now translateable. Since 0.6.0, PyPy shipped with an experimental Object Space (the part of PyPy implementing Python object operations and manipulation) implementing lazily computed objects, the "Thunk" object space. With 0.8.0 this object space can also be translated preserving its feature additions. What is PyPy (about)? ------------------------------------------------ PyPy is a MIT-licensed research-oriented reimplementation of Python written in Python itself, flexible and easy to experiment with. It translates itself to lower level languages. Our goals are to target a large variety of platforms, small and large, by providing a compilation toolsuite that can produce custom Python versions. Platform, Memory and Threading models are to become aspects of the translation process - as opposed to encoding low level details into a language implementation itself. Eventually, dynamic optimization techniques - implemented as another translation aspect - should become robust against language changes. Note that PyPy is mainly a research and development project and does not by itself focus on getting a production-ready Python implementation although we do hope and expect it to become a viable contender in that area sometime next year. PyPy is partially funded as a research project under the European Union's IST programme. Where to start? ----------------------------- Getting started: http://codespeak.net/pypy/dist/pypy/doc/getting-started.html PyPy Documentation: http://codespeak.net/pypy/dist/pypy/doc/ PyPy Homepage: http://codespeak.net/pypy/ The interpreter and object model implementations shipped with the 0.8 version can run on their own and implement the core language features of Python as of CPython 2.4. However, we still do not recommend using PyPy for anything else than for education, playing or research purposes. Ongoing work and near term goals --------------------------------- At the last sprint in Paris we started exploring the new directions of our work, in terms of extending and optimising PyPy further. We started to scratch the surface of Just-In-Time compiler related work, which we still expect will be the major source of our future speed improvements and some successful amount of work has been done on the support needed for stackless-like features. This release also includes the snapshots in preliminary or embryonic form of the following interesting but yet not completed sub projects: - The OOtyper, a RTyper variation for higher-level backends (Squeak, ...) - A JavaScript backend - A limited (PPC) assembler backend (this related to the JIT) - some bits for a socket module PyPy has been developed during approximately 16 coding sprints across Europe and the US. It continues to be a very dynamically and incrementally evolving project with many of these one-week workshops to follow. PyPy has been a community effort from the start and it would not have got that far without the coding and feedback support from numerous people. Please feel free to give feedback and raise questions. contact points: http://codespeak.net/pypy/dist/pypy/doc/contact.html have fun, the pypy team, (Armin Rigo, Samuele Pedroni, Holger Krekel, Christian Tismer, Carl Friedrich Bolz, Michael Hudson, and many others: http://codespeak.net/pypy/dist/pypy/doc/contributor.html) PyPy development and activities happen as an open source project and with the support of a consortium partially funded by a two year European Union IST research grant. The full partners of that consortium are: Heinrich-Heine University (Germany), AB Strakt (Sweden) merlinux GmbH (Germany), tismerysoft GmbH (Germany) Logilab Paris (France), DFKI GmbH (Germany) ChangeMaker (Sweden), Impara (Germany) From theller at python.net Thu Nov 3 21:01:35 2005 From: theller at python.net (Thomas Heller) Date: Thu, 03 Nov 2005 21:01:35 +0100 Subject: [Python-Dev] PYTHOPN_API_VERSION Message-ID: Shouldn't PYTHON_API_VERSION be different between 2.3 and 2.4? It is 1012 in both versions. I tried to detect whether PyTuple_Pack is supported, which was added in 2.4. Or is this only to detect changed apis, and not added apis? Thomas From Jack.Jansen at cwi.nl Thu Nov 3 22:29:37 2005 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Thu, 3 Nov 2005 22:29:37 +0100 Subject: [Python-Dev] Proposal: can we have a python-dev-announce mailing list? Message-ID: <4407AF2E-9F9F-4D75-B890-052438D20468@cwi.nl> As people may have noticed (or possibly not:-) I've been rather inactive on python-dev the last year or so, due to being completely inundated with other work. Too bad that I've missed all the interesting discussions on Python 3000, but I'm bound to catch up some time later this year:-). BUT: what I also missed are all the important announcements, such as new releases, the switch to svn, and a couple more (I think). I know I would be much helped with a moderated python-dev-announce mailing list, which would be only low-volume, time-critical announcements for people developing Python. Even during times when I am actively following python-dev it would be handy to have important announcements coming in in a separate mailbox in stead of buried under design discussions and such... -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From phd at mail2.phd.pp.ru Thu Nov 3 22:36:59 2005 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Fri, 4 Nov 2005 00:36:59 +0300 Subject: [Python-Dev] Proposal: can we have a python-dev-announce mailing list? In-Reply-To: <4407AF2E-9F9F-4D75-B890-052438D20468@cwi.nl> References: <4407AF2E-9F9F-4D75-B890-052438D20468@cwi.nl> Message-ID: <20051103213659.GA26132@phd.pp.ru> On Thu, Nov 03, 2005 at 10:29:37PM +0100, Jack Jansen wrote: > I know I would be much helped with a moderated python-dev-announce > mailing list, which would be only low-volume http://www.google.com/search?q=python-dev+summary+site%3Amail.python.org Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From jcarlson at uci.edu Thu Nov 3 22:52:25 2005 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 03 Nov 2005 13:52:25 -0800 Subject: [Python-Dev] Proposal: can we have a python-dev-announce mailing list? In-Reply-To: <20051103213659.GA26132@phd.pp.ru> References: <4407AF2E-9F9F-4D75-B890-052438D20468@cwi.nl> <20051103213659.GA26132@phd.pp.ru> Message-ID: <20051103134856.BFB2.JCARLSON@uci.edu> Even when they are on the ball, the summaries generally occur one week after the discussion/execution happens. That's not so much in the 'time-critical' aspect which, I would imagine, is about as important as the 'low-volume' aspect. - Josiah Oleg Broytmann wrote: > > On Thu, Nov 03, 2005 at 10:29:37PM +0100, Jack Jansen wrote: > > I know I would be much helped with a moderated python-dev-announce > > mailing list, which would be only low-volume > > http://www.google.com/search?q=python-dev+summary+site%3Amail.python.org > > Oleg. > -- > Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jcarlson%40uci.edu From Jack.Jansen at cwi.nl Thu Nov 3 22:51:14 2005 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Thu, 3 Nov 2005 22:51:14 +0100 Subject: [Python-Dev] Proposal: can we have a python-dev-announce mailing list? In-Reply-To: <20051103213659.GA26132@phd.pp.ru> References: <4407AF2E-9F9F-4D75-B890-052438D20468@cwi.nl> <20051103213659.GA26132@phd.pp.ru> Message-ID: <5630A610-FB3B-4359-8E86-39CBF074CF0D@cwi.nl> On 3-nov-2005, at 22:36, Oleg Broytmann wrote: > On Thu, Nov 03, 2005 at 10:29:37PM +0100, Jack Jansen wrote: > >> I know I would be much helped with a moderated python-dev-announce >> mailing list, which would be only low-volume >> > > http://www.google.com/search?q=python-dev+summary+site% > 3Amail.python.org Hmm. I wouldn't mind if it was push in stead of pull, I wouldn't mind if it was in the right order, and I wouldn't mind if itwas more concise:-) But: I'll just wait to see whether more people chime in that they'd like this, or that I'm alone... -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From skip at pobox.com Thu Nov 3 22:55:23 2005 From: skip at pobox.com (skip@pobox.com) Date: Thu, 3 Nov 2005 15:55:23 -0600 Subject: [Python-Dev] Proposal: can we have a python-dev-announce mailing list? In-Reply-To: <20051103213659.GA26132@phd.pp.ru> References: <4407AF2E-9F9F-4D75-B890-052438D20468@cwi.nl> <20051103213659.GA26132@phd.pp.ru> Message-ID: <17258.34635.582411.34526@montanaro.dyndns.org> >> I know I would be much helped with a moderated python-dev-announce >> mailing list, which would be only low-volume Oleg> http://www.google.com/search?q=python-dev+summary+site%3Amail.python.org That works up to a point, however the python-dev summaries only come out once every couple of weeks, so probably aren't going to catch important stuff that comes and goes with less than a two-week lifespan. Alerts that machines are going down for maintenance fall into this category. Also, I think the cvs->svn switch probably didn't take more than a few days once the ball got rolling. I think Martin announced the demise of the SF repository around 20 October, with a cutover date of 26 October. Skip From martin at v.loewis.de Thu Nov 3 23:08:55 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 03 Nov 2005 23:08:55 +0100 Subject: [Python-Dev] PYTHOPN_API_VERSION In-Reply-To: References: Message-ID: <436A8A77.4040306@v.loewis.de> Thomas Heller wrote: > Shouldn't PYTHON_API_VERSION be different between 2.3 and 2.4? > It is 1012 in both versions. > > I tried to detect whether PyTuple_Pack is supported, which was added in > 2.4. Or is this only to detect changed apis, and not added apis? It's meant to detect changes that can break existing binary modules. In most cases, this would be changed structs. Whether such changes happened between 2.3 and 2.4, I don't know. If you want to ask whether a certain function is present, either use autoconf, or check for the Python (hex) version where it was first introduced. Regards, Martin From martin at v.loewis.de Thu Nov 3 23:16:42 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 03 Nov 2005 23:16:42 +0100 Subject: [Python-Dev] Proposal: can we have a python-dev-announce mailing list? In-Reply-To: <5630A610-FB3B-4359-8E86-39CBF074CF0D@cwi.nl> References: <4407AF2E-9F9F-4D75-B890-052438D20468@cwi.nl> <20051103213659.GA26132@phd.pp.ru> <5630A610-FB3B-4359-8E86-39CBF074CF0D@cwi.nl> Message-ID: <436A8C4A.8090908@v.loewis.de> Jack Jansen wrote: > Hmm. I wouldn't mind if it was push in stead of pull, I wouldn't mind > if it was in the right order, and I wouldn't mind if itwas more > concise:-) > > But: I'll just wait to see whether more people chime in that they'd > like this, or that I'm alone... I'm -1 on such a list. If it existed, people could complain "why wasn't this announced properly". So the "blame" would be on people who failed to give proper notice, instead of on the people who did not care enough to follow the entire discussion. More specifically, I'm sure I would have forgotten to post about the svn switchover to python-dev-announce, just as I failed to post to comp.lang.python.announce. This is all volunteer work. Regards, Martin From t-meyer at ihug.co.nz Fri Nov 4 00:41:12 2005 From: t-meyer at ihug.co.nz (Tony Meyer) Date: Fri, 4 Nov 2005 12:41:12 +1300 Subject: [Python-Dev] Proposal: can we have a python-dev-announce mailing list? In-Reply-To: <4407AF2E-9F9F-4D75-B890-052438D20468@cwi.nl> References: <4407AF2E-9F9F-4D75-B890-052438D20468@cwi.nl> Message-ID: <27F1C5EB-9459-4F1F-A43F-9898941D66CF@ihug.co.nz> > I know I would be much helped with a moderated python-dev-announce > mailing list, which would be only low-volume, time-critical > announcements for people developing Python. Even during times when I > am actively following python-dev it would be handy to have important > announcements coming in in a separate mailbox in stead of buried > under design discussions and such... Firstly, my apologies for the current delay in summaries, which exacerbates this problem (although others are right when they say that things sometimes happen too fast even for on-time summaries). A while back there was talk about a mailing list for PEP changes and the solution was instead to use the "topic" feature of mailman, essentially creating a subset-mailing-list. Would something like this be feasible for this? (I don't really know enough how how the topic feature can be used to know if it is workable or not). I presume that this would still need some sort of action from the poster (e.g. including a tag somewhere), but it would probably be easier for people to remember to do that than cross-post to another list entirely. =Tony.Meyer From rmunn at pobox.com Fri Nov 4 01:17:32 2005 From: rmunn at pobox.com (Robin Munn) Date: Thu, 03 Nov 2005 18:17:32 -0600 Subject: [Python-Dev] No more problems with new SVN repository In-Reply-To: <4369C54F.3050803@pobox.com> References: <4369177D.3020000@pobox.com> <43693944.3090803@v.loewis.de> <4369486A.8090107@pobox.com> <4369724D.8060001@pobox.com> <4369C2EA.6030407@v.loewis.de> <4369C54F.3050803@pobox.com> Message-ID: <436AA89C.6050401@pobox.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Robin Munn wrote: > So doing an SVK mirror of the repository should work now, barring > any further surprises. I'm starting the SVK sync now; we'll see what > happens. Confirmed; the SVK mirror took about 18 hours, but it completed successfully with no further problems. Again, thanks for fixing the issues so quickly. - -- Robin Munn rmunn at pobox.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (Darwin) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDaqiZ6OLMk9ZJcBQRAjGuAJwLmbrxBgrHYUb/7LOvjq89GfKrWACghGgn pvuMT5edAfMw3OAoZf5mJiw= =2i88 -----END PGP SIGNATURE----- From guido at python.org Fri Nov 4 01:21:15 2005 From: guido at python.org (Guido van Rossum) Date: Thu, 3 Nov 2005 16:21:15 -0800 Subject: [Python-Dev] No more problems with new SVN repository In-Reply-To: <436AA89C.6050401@pobox.com> References: <4369177D.3020000@pobox.com> <43693944.3090803@v.loewis.de> <4369486A.8090107@pobox.com> <4369724D.8060001@pobox.com> <4369C2EA.6030407@v.loewis.de> <4369C54F.3050803@pobox.com> <436AA89C.6050401@pobox.com> Message-ID: I have a question after this exhilarating exchange. Is there a way to prevent this kind of thing in the future, e.g. by removing or rejecting change log messages with characters that are considered invalid in XML? (Or should perhaps the fix be to suppress or quote these characters somehow in XML?) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Fri Nov 4 07:31:03 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 04 Nov 2005 07:31:03 +0100 Subject: [Python-Dev] No more problems with new SVN repository In-Reply-To: References: <4369177D.3020000@pobox.com> <43693944.3090803@v.loewis.de> <4369486A.8090107@pobox.com> <4369724D.8060001@pobox.com> <4369C2EA.6030407@v.loewis.de> <4369C54F.3050803@pobox.com> <436AA89C.6050401@pobox.com> Message-ID: <436B0027.6010808@v.loewis.de> Guido van Rossum wrote: > I have a question after this exhilarating exchange. > > Is there a way to prevent this kind of thing in the future, e.g. by > removing or rejecting change log messages with characters that are > considered invalid in XML? I don't think it can happen again. Without testing, I would hope subversion rejects log messages that contain "random" control characters (if it doesn't, I should report that as a bug). The characters are in there because of the CVS conversion (that might be a bug in cvs2svn, which should have replaced them perhaps). It only happened in very old log messages - so perhaps even CVS doesn't allow them anymore. In XML 1.0, there is a lot of confusion about including control characters in text. In XML 1.1, this was clarified that you can include them, but only through character references. So in the future, subversion might be able to transmit such log messages in well-formed webdav. Regards, Martin From fredrik at pythonware.com Fri Nov 4 10:05:40 2005 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 4 Nov 2005 10:05:40 +0100 Subject: [Python-Dev] Adding examples to PEP 263 Message-ID: the runtime warning you get when you use non-ascii characters in python source code points the poor user to this page: http://www.python.org/peps/pep-0263.html which tells the user to add a # -*- coding: -*- to the source, and then provides a more detailed syntax description as a RE pattern. to help people that didn't grow up with emacs, and don't speak fluent RE, and/or prefer to skim documentation, it would be a quite helpful if the page also contained a few examples; e.g. # -*- coding: utf-8 -*- # -*- coding: iso-8859-1 -*- can anyone with SVN write access perhaps add this? (I'd probably add a note to the top of the page for anyone who arrives there via a Python error message, which summarizes the pep and provides an example or two; abstracts and rationales are nice, but if you're just a plain user, a "do this; here's how it works; further discussion below" style is a bit more practical...) From mal at egenix.com Fri Nov 4 10:27:43 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 04 Nov 2005 10:27:43 +0100 Subject: [Python-Dev] Adding examples to PEP 263 In-Reply-To: References: Message-ID: <436B298F.3050803@egenix.com> Fredrik Lundh wrote: > the runtime warning you get when you use non-ascii characters in > python source code points the poor user to this page: > > http://www.python.org/peps/pep-0263.html > > which tells the user to add a > > # -*- coding: -*- > > to the source, and then provides a more detailed syntax description > as a RE pattern. to help people that didn't grow up with emacs, and > don't speak fluent RE, and/or prefer to skim documentation, it would > be a quite helpful if the page also contained a few examples; e.g. > > # -*- coding: utf-8 -*- > # -*- coding: iso-8859-1 -*- > > can anyone with SVN write access perhaps add this? Good point. I'll add some examples. > (I'd probably add a note to the top of the page for anyone who arrives > there via a Python error message, which summarizes the pep and provides > an example or two; abstracts and rationales are nice, but if you're just a > plain user, a "do this; here's how it works; further discussion below" style > is a bit more practical...) The PEP isn't all that long, so I don't think a summary would help. However, we might want to point the user to a different URL in the error message, e.g. a Wiki page with more user-friendly content. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 04 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2005-10-17: Released mxODBC.Zope.DA 1.0.9 http://zope.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From noamraph at gmail.com Fri Nov 4 13:02:31 2005 From: noamraph at gmail.com (Noam Raphael) Date: Fri, 4 Nov 2005 14:02:31 +0200 Subject: [Python-Dev] Why should the default hash(x) == id(x)? In-Reply-To: <43695C50.5070600@canterbury.ac.nz> References: <43695C50.5070600@canterbury.ac.nz> Message-ID: On 11/3/05, Greg Ewing wrote: > > 3. If someone does want to associate values with objects, he can > > explicitly use id: > > dct[id(x)] = 3. > > This is fragile. Once all references to x are dropped, > it is possible for another object to be created having > the same id that x used to have. The dict now > unintentionally references the new object. > You are right. Please see the simple "ref" class that I wrote in my previous post, which solves this problem. Noam From steve at holdenweb.com Fri Nov 4 13:03:44 2005 From: steve at holdenweb.com (Steve Holden) Date: Fri, 04 Nov 2005 12:03:44 +0000 Subject: [Python-Dev] Adding examples to PEP 263 In-Reply-To: <436B298F.3050803@egenix.com> References: <436B298F.3050803@egenix.com> Message-ID: M.-A. Lemburg wrote: > Fredrik Lundh wrote: > >>the runtime warning you get when you use non-ascii characters in >>python source code points the poor user to this page: >> >> http://www.python.org/peps/pep-0263.html >> >>which tells the user to add a >> >> # -*- coding: -*- >> >>to the source, and then provides a more detailed syntax description >>as a RE pattern. to help people that didn't grow up with emacs, and >>don't speak fluent RE, and/or prefer to skim documentation, it would >>be a quite helpful if the page also contained a few examples; e.g. >> >># -*- coding: utf-8 -*- >># -*- coding: iso-8859-1 -*- >> >>can anyone with SVN write access perhaps add this? > > > Good point. I'll add some examples. > > >>(I'd probably add a note to the top of the page for anyone who arrives >>there via a Python error message, which summarizes the pep and provides >>an example or two; abstracts and rationales are nice, but if you're just a >>plain user, a "do this; here's how it works; further discussion below" style >>is a bit more practical...) > > > The PEP isn't all that long, so I don't think a summary would > help. However, we might want to point the user to a different > URL in the error message, e.g. a Wiki page with more user-friendly > content. > Under NO circumstances should a Wiki page be used as the destination for a link in a runtime error message. If the page happens to be spammed when the user follows the link they'll wonder why the error message is pointing to a page full of links to hot babes, or whatever. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC www.holdenweb.com PyCon TX 2006 www.python.org/pycon/ From pinard at iro.umontreal.ca Fri Nov 4 16:32:24 2005 From: pinard at iro.umontreal.ca (=?iso-8859-1?Q?Fran=E7ois?= Pinard) Date: Fri, 4 Nov 2005 10:32:24 -0500 Subject: [Python-Dev] No more problems with new SVN repository In-Reply-To: References: <4369177D.3020000@pobox.com> <43693944.3090803@v.loewis.de> <4369486A.8090107@pobox.com> <4369724D.8060001@pobox.com> <4369C2EA.6030407@v.loewis.de> <4369C54F.3050803@pobox.com> <436AA89C.6050401@pobox.com> Message-ID: <20051104153224.GA22469@alcyon.progiciels-bpi.ca> [Guido van Rossum] > Is there a way to prevent this kind of thing in the future, e.g. by > removing or rejecting change log messages with characters that are > considered invalid in XML? Suppose TOP is the top of the Subversion repository. The easiest way is providing a TOP/hook/pre-commit script. If the script exits with non-zero status, usually with some clear diagnostic on stderr, the whole commit aborts, and the diagnostic is shown to the committing user. The tricky part is getting the tentative log message from within the script. This is done by popening "svnlook log -t ARG2 ARG1", where ARG1 and ARG2 are arguments given to the pre-commit script. -- Fran?ois Pinard http://pinard.progiciels-bpi.ca From dave at boost-consulting.com Fri Nov 4 21:09:39 2005 From: dave at boost-consulting.com (David Abrahams) Date: Fri, 04 Nov 2005 15:09:39 -0500 Subject: [Python-Dev] Plea to distribute debugging lib Message-ID: For years, Boost.Python has been doing some hacks to work around the fact that a Windows Python distro doesn't include the debug build of the library. http://www.boost.org/libs/python/doc/building.html#variants explains. We wanted to make it reasonably convenient for Windows developers (and our distributed testers) to work with a debug build of the Boost.Python library and of their own code. Having to download the Python source and build the debug DLL was deemed unacceptable. Well, those hacks have run out of road. VC++8 now detects that some of its headers have been #included with _DEBUG and some without, and it will refuse to build anything when it does. We have several new hacks to work around that detection, and I think we _might_ be able to get away with them for one more release. But it's really time to do it right. MS is recommending that we (Boost) start distributing a debug build of the Python DLL with Boost, but Boost really seems like the wrong place to host such a thing. Is there any way Python.org can make a debug build more accessible? Thanks, Dave -- Dave Abrahams Boost Consulting www.boost-consulting.com From python at discworld.dyndns.org Fri Nov 4 21:28:25 2005 From: python at discworld.dyndns.org (Charles Cazabon) Date: Fri, 4 Nov 2005 14:28:25 -0600 Subject: [Python-Dev] Plea to distribute debugging lib In-Reply-To: References: Message-ID: <20051104202824.GA19678@discworld.dyndns.org> David Abrahams wrote: > > For years, Boost.Python has been doing some hacks to work around the fact > that a Windows Python distro doesn't include the debug build of the library. [...] > Having to download the Python source and build the debug DLL was deemed > unacceptable. I'm curious: why was this "deemed unacceptable"? Python's license is about as liberal as it gets, and the code is almost startlingly easy to compile -- easier than any other similarly-sized codebase I've had to work with. Charles -- ----------------------------------------------------------------------- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ ----------------------------------------------------------------------- From guido at python.org Fri Nov 4 21:33:44 2005 From: guido at python.org (Guido van Rossum) Date: Fri, 4 Nov 2005 12:33:44 -0800 Subject: [Python-Dev] Plea to distribute debugging lib In-Reply-To: <20051104202824.GA19678@discworld.dyndns.org> References: <20051104202824.GA19678@discworld.dyndns.org> Message-ID: I vaguely recall that there were problems with distributing the debug version of the MS runtime. Anyway, why can't you do this yourself for all Boost users? It's all volunteer time, you know... --Guido On 11/4/05, Charles Cazabon wrote: > David Abrahams wrote: > > > > For years, Boost.Python has been doing some hacks to work around the fact > > that a Windows Python distro doesn't include the debug build of the library. > [...] > > Having to download the Python source and build the debug DLL was deemed > > unacceptable. > > I'm curious: why was this "deemed unacceptable"? Python's license is about as > liberal as it gets, and the code is almost startlingly easy to compile -- > easier than any other similarly-sized codebase I've had to work with. > > Charles > -- > ----------------------------------------------------------------------- > Charles Cazabon > GPL'ed software available at: http://pyropus.ca/software/ > ----------------------------------------------------------------------- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.peters at gmail.com Fri Nov 4 21:37:37 2005 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 4 Nov 2005 15:37:37 -0500 Subject: [Python-Dev] Plea to distribute debugging lib In-Reply-To: References: Message-ID: <1f7befae0511041237j2156306fhe5b90053a7e027f8@mail.gmail.com> [David Abrahams] > For years, Boost.Python has been doing some hacks to work around the > fact that a Windows Python distro doesn't include the debug build of > the library. > ... > MS is recommending that we (Boost) start distributing a debug build of the > Python DLL with Boost, but Boost really seems like the wrong place to host > such a thing. Is there any way Python.org can make a debug build more > accessible? Possibly. I don't do this anymore (this == build the Python Windows installers), but I used to. For some time I also made available a zip file containing various debug-build bits, captured at the time the official installer was built. We didn't (and I'm sure we still don't) want to include them in the main installer, because they bloat its size for something most users truly do not want. I got sick of building the debug zip file, and stopped doing that too. No two users wanted the same set of stuff in it, so it grew to contain the union of everything everyone wanted, and then people complained that it was "too big". This is one of the few times in your Uncle Timmy's life that he said "so screw it -- do it yourself, you whiny baby whiners with your incessant baby whining you " ;-) Based on that sure-to-be universal reaction from anyone who signs up for this, I'd say the best thing you could do to help it along is to define precisely (a) what an acceptable distribution format is; and, (b) what exactly it should contain. That, and being nice to Martin, would go a long way. From theller at python.net Fri Nov 4 21:47:40 2005 From: theller at python.net (Thomas Heller) Date: Fri, 04 Nov 2005 21:47:40 +0100 Subject: [Python-Dev] Plea to distribute debugging lib References: <20051104202824.GA19678@discworld.dyndns.org> Message-ID: Guido van Rossum writes: > I vaguely recall that there were problems with distributing the debug > version of the MS runtime. Right: the debug runtime dlls are not disributable. > Anyway, why can't you do this yourself for all Boost users? It's all > volunteer time, you know... Doesn't any boost user need a C compiler anyway, so it should not really be a problem to compile Python? Anyway, AFAIK, the activestate distribution contains Python debug dlls. Thomas From dave at boost-consulting.com Fri Nov 4 21:58:11 2005 From: dave at boost-consulting.com (David Abrahams) Date: Fri, 04 Nov 2005 15:58:11 -0500 Subject: [Python-Dev] Plea to distribute debugging lib In-Reply-To: <1f7befae0511041237j2156306fhe5b90053a7e027f8@mail.gmail.com> (Tim Peters's message of "Fri, 4 Nov 2005 15:37:37 -0500") References: <1f7befae0511041237j2156306fhe5b90053a7e027f8@mail.gmail.com> Message-ID: Tim Peters writes: > [David Abrahams] >> For years, Boost.Python has been doing some hacks to work around the >> fact that a Windows Python distro doesn't include the debug build of >> the library. >> ... >> MS is recommending that we (Boost) start distributing a debug build of the >> Python DLL with Boost, but Boost really seems like the wrong place to host >> such a thing. Is there any way Python.org can make a debug build more >> accessible? > > Possibly. I don't do this anymore (this == build the Python Windows > installers), but I used to. For some time I also made available a zip > file containing various debug-build bits, captured at the time the > official installer was built. We didn't (and I'm sure we still don't) > want to include them in the main installer, because they bloat its > size for something most users truly do not want. > > I got sick of building the debug zip file, and stopped doing that too. > No two users wanted the same set of stuff in it, so it grew to > contain the union of everything everyone wanted, and then people > complained that it was "too big". This is one of the few times in > your Uncle Timmy's life that he said "so screw it -- do it yourself, > you whiny baby whiners with your incessant baby whining you " ;-) > > Based on that sure-to-be universal reaction from anyone who signs up > for this, I'd say the best thing you could do to help it along is to > define precisely (a) what an acceptable distribution format is; and, > (b) what exactly it should contain. Who knows what the whiny babies will accept? That said, I think people would be happy with a .zip file containing whatever is built by selecting the debug build in the VS project and asking it to build everything. (**) > That, and being nice to Martin, I'm always as nice as Davidly possible to Martin! > would go a long way. My fingers and toes are crossed. Thanks! (**) If you could build the ability to download the debugging binaries into the regular installer, that would be the shiznit, but I don't dare ask for it. ;-) -- Dave Abrahams Boost Consulting www.boost-consulting.com From dave at boost-consulting.com Fri Nov 4 23:25:55 2005 From: dave at boost-consulting.com (David Abrahams) Date: Fri, 04 Nov 2005 17:25:55 -0500 Subject: [Python-Dev] Plea to distribute debugging lib In-Reply-To: <436BD111.5080808@rubikon.pl> (Bronek Kozicki's message of "Fri, 04 Nov 2005 21:22:25 +0000") References: <1f7befae0511041237j2156306fhe5b90053a7e027f8@mail.gmail.com> <436BD111.5080808@rubikon.pl> Message-ID: Bronek Kozicki writes: > David Abrahams wrote: >> Who knows what the whiny babies will accept? That said, I think >> people would be happy with a .zip file containing whatever is built by >> selecting the debug build in the VS project and asking it to build >> everything. (**) > > Just to clarify - what we are asking for is library built with _DEBUG > and no BOOST_DEBUG_PYTHON, that is the one compatible with default > Python distribution. Bronek, I know you're trying to help, but I'm sure that's not making anything clearer for these people. They don't know anything about BOOST_DEBUG_PYTHON and would never have cause to define it. -- Dave Abrahams Boost Consulting www.boost-consulting.com From martin at v.loewis.de Fri Nov 4 23:29:56 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 04 Nov 2005 23:29:56 +0100 Subject: [Python-Dev] Plea to distribute debugging lib In-Reply-To: References: <1f7befae0511041237j2156306fhe5b90053a7e027f8@mail.gmail.com> Message-ID: <436BE0E4.6@v.loewis.de> David Abrahams wrote: > Who knows what the whiny babies will accept? That said, I think > people would be happy with a .zip file containing whatever is built by > selecting the debug build in the VS project and asking it to build > everything. (**) I would go a step further than Tim: Send me (*) a patch to msi.py (which is used to build the distribution) that picks up the files and packages them in the desired way, and I will include the files it outputs in the official distribution. This is how the libpython24.a got in (and this is also the way in which it will get out again). In the patch, preferably state whom to contact for the specific feature, as I won't be able to answer questions about it. I don't have a personal need for the feature (I do have debug builds myself, and it takes only 10 minutes or so to create them), so I won't even have a way to test whether the feature works correctly. Regards, Martin (*) that is, sf.net/projects/python From eyal.lotem at gmail.com Fri Nov 4 23:33:29 2005 From: eyal.lotem at gmail.com (Eyal Lotem) Date: Sat, 5 Nov 2005 00:33:29 +0200 Subject: [Python-Dev] Class decorators vs metaclasses Message-ID: I have a few claims, some unrelated, and some built on top of each other. I would like to hear your responses as to which are convincing, which arne't, and why. I think that if these claims are true, Python 3000 should change quite a bit. A. Metaclass code is black magic and few understand how it works, while decorator code is mostly understandable, even by non-gurus. B. One of Decorators' most powerful features is that they can mixed-and-matched, which makes them very powerful for many purposes, while metaclasses are exclusive, and only one can be used. This is especially problematic as some classes may assume their subclasses must use their respective metaclasses. This means classdecorators are strictly more powerful than metaclasses, without cumbersome convertions between metaclass mechanisms and decorator mechanisms. C. Interesting uses of classdecorators are allowing super-calling without redundantly specifying the name of your class, or your superclass. D. Python seems to be incrementally adding power to the core language with these features, which is great, but it also causes significant overlapping of language features, which I believe is something to avoid when possible. If metaclasses are replaced with class decorators, then suddenly inheritence becomes a redundant feature. E. If inheritence is a redundant feature, it can be removed and an "inherit" class decorator can be used. This could also reduce all the __mro__ clutter from the language along with other complexities, into alternate implementations of the inherit classdecorator. From martin at v.loewis.de Fri Nov 4 23:44:53 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 04 Nov 2005 23:44:53 +0100 Subject: [Python-Dev] Plea to distribute debugging lib In-Reply-To: References: <1f7befae0511041237j2156306fhe5b90053a7e027f8@mail.gmail.com> <436BD111.5080808@rubikon.pl> Message-ID: <436BE465.4000100@v.loewis.de> David Abrahams wrote: >>Just to clarify - what we are asking for is library built with _DEBUG >>and no BOOST_DEBUG_PYTHON, that is the one compatible with default >>Python distribution. > > > I know you're trying to help, but I'm sure that's not making anything > clearer for these people. They don't know anything about > BOOST_DEBUG_PYTHON and would never have cause to define it. > Actually, I'm truly puzzled. Why would a library that has _DEBUG defined be compatible with the standard distribution? Doesn't _DEBUG cause linkage with msvcr71d.dll? In addition (more correctly: for that reason), the debug build causes python2x_d.dll to be build, instead of python2x.dll, which definitely is incompatible with the standard DLL. It not only uses a different C library; it also causes Py_DEBUG to be defined, which in turn creates a different memory layout for PyObject. So in the end, I would assume you are requesting what you call a debug-python, i.e. one that (in your system) *has* BOOST_DEBUG_PYTHON defined. Regards, Martin From aleaxit at gmail.com Sat Nov 5 00:02:42 2005 From: aleaxit at gmail.com (Alex Martelli) Date: Fri, 4 Nov 2005 15:02:42 -0800 Subject: [Python-Dev] Class decorators vs metaclasses In-Reply-To: References: Message-ID: On 11/4/05, Eyal Lotem wrote: > I have a few claims, some unrelated, and some built on top of each > other. I would like to hear your responses as to which are > convincing, which arne't, and why. I think that if these claims are > true, Python 3000 should change quite a bit. > > A. Metaclass code is black magic and few understand how it works, > while decorator code is mostly understandable, even by non-gurus. I disagree. I've held many presentations and classes on both subjects, and while people may INITIALLY feel like metaclasses are black magic, as soon as I've explained it the fear dissipates. It all boils down do understanding that: class Name(Ba,Ses): <> means Name = suitable_metaclass('Name', (Ba,Ses), <>) which isn't any harder than understanding that @foo(bar) def baz(args): ... means def baz(args): ... baz = foo(bar)(baz) > B. One of Decorators' most powerful features is that they can > mixed-and-matched, which makes them very powerful for many purposes, > while metaclasses are exclusive, and only one can be used. This is Wrong. You can mix as many metaclasses as you wish, as long as they're properly coded for multiple inheritance (using super, etc) -- just inherit from them all. This is reasonably easy to automate (see the last recipe in the 2nd ed of the Python Cookbook), too. > especially problematic as some classes may assume their subclasses > must use their respective metaclasses. This means classdecorators are > strictly more powerful than metaclasses, without cumbersome > convertions between metaclass mechanisms and decorator mechanisms. The assertion that classdecorators are strictly more powerful than custom metaclasses is simply false. How would you design classdecorator XXX so that @XXX class Foo: ... allows 'print Foo' to emit 'this is beautiful class Foo', for example? the str(Foo) implicit in print calls type(Foo).__str__(Foo), so you do need a custom type(Foo) -- which is all that is meant by "a custom metaclass"... a custom type whose instances are classes, that's all. > C. Interesting uses of classdecorators are allowing super-calling > without redundantly specifying the name of your class, or your > superclass. Can you give an example? > > D. Python seems to be incrementally adding power to the core language > with these features, which is great, but it also causes significant > overlapping of language features, which I believe is something to > avoid when possible. If metaclasses are replaced with class > decorators, then suddenly inheritence becomes a redundant feature. And how do you customize what "print Foo" emits, as above? > E. If inheritence is a redundant feature, it can be removed and an > "inherit" class decorator can be used. This could also reduce all the > __mro__ clutter from the language along with other complexities, into > alternate implementations of the inherit classdecorator. How do you propose to get exactly the same effects as inheritance (affect every attribute lookup on a class and its instances) without inheritance? Essentially, inheritance is automated delegation obtained by having getattr(foo, 'bar') look through a chain of objects (essentially the __mro__) until an attribute named 'bar' is found in one of those objects, plus a few minor but useful side effects, e.g. on isinstance and issubclass, and the catching of exceptions in try/except statements. How would any mechanism allowing all of these uses NOT be inheritance? Alex From dave at boost-consulting.com Sat Nov 5 00:04:29 2005 From: dave at boost-consulting.com (David Abrahams) Date: Fri, 04 Nov 2005 18:04:29 -0500 Subject: [Python-Dev] Plea to distribute debugging lib In-Reply-To: <436BE0E4.6@v.loewis.de> (Martin v. =?iso-8859-1?Q?L=F6wis's?= message of "Fri, 04 Nov 2005 23:29:56 +0100") References: <1f7befae0511041237j2156306fhe5b90053a7e027f8@mail.gmail.com> <436BE0E4.6@v.loewis.de> Message-ID: "Martin v. L?wis" writes: > David Abrahams wrote: >> Who knows what the whiny babies will accept? That said, I think >> people would be happy with a .zip file containing whatever is built by >> selecting the debug build in the VS project and asking it to build >> everything. (**) > > I would go a step further than Tim: Send me (*) a patch to msi.py (which > is used to build the distribution) that picks up the files and packages > them in the desired way, and I will include the files it outputs > in the official distribution. This is how the libpython24.a got in > (and this is also the way in which it will get out again). Not to look a gift horse in the mouth, but won't that cause the problem that Tim was worried about, i.e. a bloated Python installer? > In the patch, preferably state whom to contact for the specific feature, > as I won't be able to answer questions about it. > > I don't have a personal need for the feature (I do have debug builds > myself, and it takes only 10 minutes or so to create them), I know, me too. It's easy enough once you get started building Python. I just think it's too big a hump for many people. > so I won't even have a way to test whether the feature works > correctly. > > Regards, > Martin > > (*) that is, sf.net/projects/python I s'pose that means, "put it in the patches tracker." grateful-ly y'rs, -- Dave Abrahams Boost Consulting www.boost-consulting.com From dave at boost-consulting.com Sat Nov 5 00:13:39 2005 From: dave at boost-consulting.com (David Abrahams) Date: Fri, 04 Nov 2005 18:13:39 -0500 Subject: [Python-Dev] Plea to distribute debugging lib In-Reply-To: <436BE465.4000100@v.loewis.de> (Martin v. =?iso-8859-1?Q?L=F6?= =?iso-8859-1?Q?wis's?= message of "Fri, 04 Nov 2005 23:44:53 +0100") References: <1f7befae0511041237j2156306fhe5b90053a7e027f8@mail.gmail.com> <436BD111.5080808@rubikon.pl> <436BE465.4000100@v.loewis.de> Message-ID: "Martin v. L?wis" writes: > David Abrahams wrote: >>> Just to clarify - what we are asking for is library built with >>> _DEBUG and no BOOST_DEBUG_PYTHON, that is the one compatible with >>> default Python distribution. >> I know you're trying to help, but I'm sure that's not making >> anything >> clearer for these people. They don't know anything about >> BOOST_DEBUG_PYTHON and would never have cause to define it. >> > > Actually, I'm truly puzzled. I was afraid this would happen. Really, you're better off ignoring Bronek's message. > Why would a library that has _DEBUG defined > be compatible with the standard distribution? Doesn't _DEBUG cause > linkage with msvcr71d.dll? Unless you do the hacks that I mentioned in my opening message. Read http://www.boost.org/libs/python/doc/building.html#variants for details. > In addition (more correctly: for that reason), the debug build causes > python2x_d.dll to be build, instead of python2x.dll, which definitely > is incompatible with the standard DLL. It not only uses a different > C library; it also causes Py_DEBUG to be defined, which in turn creates > a different memory layout for PyObject. Exactly. > So in the end, I would assume you are requesting what you call a > debug-python, i.e. one that (in your system) *has* > BOOST_DEBUG_PYTHON defined. What I am requesting is the good old python2x_d.dll and any associated extension modules that get built as part of the Python distro, so I can stop doing the hack, drop BOOST_DEBUG_PYTHON, and tell people use _DEBUG in the usual way. -- Dave Abrahams Boost Consulting www.boost-consulting.com From martin at v.loewis.de Sat Nov 5 00:21:05 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 05 Nov 2005 00:21:05 +0100 Subject: [Python-Dev] Plea to distribute debugging lib In-Reply-To: References: <1f7befae0511041237j2156306fhe5b90053a7e027f8@mail.gmail.com> <436BE0E4.6@v.loewis.de> Message-ID: <436BECE1.3090202@v.loewis.de> David Abrahams wrote: >>I would go a step further than Tim: Send me (*) a patch to msi.py (which >>is used to build the distribution) that picks up the files and packages >>them in the desired way, and I will include the files it outputs >>in the official distribution. This is how the libpython24.a got in >>(and this is also the way in which it will get out again). > > > Not to look a gift horse in the mouth, but won't that cause the > problem that Tim was worried about, i.e. a bloated Python installer? Not if done properly: it would, of course, *not* add the desired files in to the msi file, but create a separate file. It is pure Python code, and called msi.py because that's it main function. It does several other things, though (such as creating a .cab file and a .a file); it could well create another zip file. As to how it would work: preferably by invoking the Python zip library, but invoking external programs to package up everything might be acceptable as well (assuming I'm told what these tools are, and assuming it falls back to doing nothing if the tools are not available). The separate file would have a name similar to the MSI file, so that the debug file has the same version number as the MSI file. > I s'pose that means, "put it in the patches tracker." > Exactly. Regards, Martin From eyal.lotem at gmail.com Sat Nov 5 12:27:55 2005 From: eyal.lotem at gmail.com (Eyal Lotem) Date: Sat, 5 Nov 2005 13:27:55 +0200 Subject: [Python-Dev] Class decorators vs metaclasses In-Reply-To: References: Message-ID: On 11/5/05, Alex Martelli wrote: > On 11/4/05, Eyal Lotem wrote: > > I have a few claims, some unrelated, and some built on top of each > > other. I would like to hear your responses as to which are > > convincing, which arne't, and why. I think that if these claims are > > true, Python 3000 should change quite a bit. > > > > A. Metaclass code is black magic and few understand how it works, > > while decorator code is mostly understandable, even by non-gurus. > > I disagree. I've held many presentations and classes on both > subjects, and while people may INITIALLY feel like metaclasses are > black magic, as soon as I've explained it the fear dissipates. It all > boils down do understanding that: > > class Name(Ba,Ses): <> > > means > > Name = suitable_metaclass('Name', (Ba,Ses), <>) > > which isn't any harder than understanding that > > @foo(bar) > def baz(args): ... > > means > > def baz(args): ... > baz = foo(bar)(baz) I disagree again. My experience is that metaclass code is very hard to understand. Especially when it starts doing non-trivial things, such as using a base metaclass class that is parametrized by metaclass attributes in its subclasses. Lookups of attributes in the base metaclass methods is mind boggling (is it searching them in the base metaclass, the subclass, the instance [which is the class]?). The same code would be much easier to understand with class decorators. > > B. One of Decorators' most powerful features is that they can > > mixed-and-matched, which makes them very powerful for many purposes, > > while metaclasses are exclusive, and only one can be used. This is > > Wrong. You can mix as many metaclasses as you wish, as long as > they're properly coded for multiple inheritance (using super, etc) -- > just inherit from them all. This is reasonably easy to automate (see > the last recipe in the 2nd ed of the Python Cookbook), too. Multiple inheritence is an awful way to mix class fucntionalities though. Lets take a simpler example. Most UT frameworks use a TestCase base class they inherit from to implement setup, tearDown, and then inherit from it again to implement the test itself. I argue this is a weak approach, because then mixing/matching setups is difficult. You would argue this is not the case, because of the ability to multiply-inherit from test cases, but how easy is the equivalent of: @with_setup('blah') @with_other_setup('bleh') def my_test(): # the blah setup and bleh other setup are up and usable here, # and will be "torn down" at the end of this test The equivalent of this requires a lot more work and violating DRY. Creating a specific function to multiply inherit from TestCases is a possible solution, but it is much more conceptually complex, and needs to be reimplemented in the next scenario (Metaclasses for example). > > especially problematic as some classes may assume their subclasses > > must use their respective metaclasses. This means classdecorators are > > strictly more powerful than metaclasses, without cumbersome > > convertions between metaclass mechanisms and decorator mechanisms. > > The assertion that classdecorators are strictly more powerful than > custom metaclasses is simply false. How would you design > classdecorator XXX so that > > @XXX > class Foo: ... > > allows 'print Foo' to emit 'this is beautiful class Foo', for example? > the str(Foo) implicit in print calls type(Foo).__str__(Foo), so you > do need a custom type(Foo) -- which is all that is meant by "a custom > metaclass"... a custom type whose instances are classes, that's all. I would argue that this is not such a useful feature, as in that case you can simply use a factory object instead of a class. If this feature remains, that's fine, but the fact it allows for a weak form of "decoration" of classes should not kill the concept of class decorators. The only reason of using metaclasses rather than factory objects, in my experience, was that references to class objects are considered different than references to factories (by pickle and deepcopy, and maybe others) and that can be a useful feature. This feature can be implemented in more readable means though. > > C. Interesting uses of classdecorators are allowing super-calling > > without redundantly specifying the name of your class, or your > > superclass. > > Can you give an example? @anotherclassdecorator @supercallerclass class MyClass(object): @supercaller def my_method(self, supcaller, x, y, z): ... result = supcaller.my_method(x, y, z) ... Could be nice to remove the need for decorating the class, and only decorating the methods, but the method decorators get a function object, not a method object, so its more difficult (perhaps portably impossible?) to do this. Note that "__metaclass__ = superclasscaller" could also work, but then combining "anotherclassdecorator" would require a lot more code at worst, or a complex mechanism to combine metaclasses via multiple inheritence at best. > > D. Python seems to be incrementally adding power to the core language > > with these features, which is great, but it also causes significant > > overlapping of language features, which I believe is something to > > avoid when possible. If metaclasses are replaced with class > > decorators, then suddenly inheritence becomes a redundant feature. > > And how do you customize what "print Foo" emits, as above? As I said, "Foo" can be a factory object rather than a class object. > > E. If inheritence is a redundant feature, it can be removed and an > > "inherit" class decorator can be used. This could also reduce all the > > __mro__ clutter from the language along with other complexities, into > > alternate implementations of the inherit classdecorator. > > How do you propose to get exactly the same effects as inheritance > (affect every attribute lookup on a class and its instances) without > inheritance? Essentially, inheritance is automated delegation > obtained by having getattr(foo, 'bar') look through a chain of objects > (essentially the __mro__) until an attribute named 'bar' is found in > one of those objects, plus a few minor but useful side effects, e.g. > on isinstance and issubclass, and the catching of exceptions in > try/except statements. How would any mechanism allowing all of these > uses NOT be inheritance? One possibility is to copy the superclass attributes into subclasses. Another is to allow the class decorator to specify getattr/setattr/delattr's implementation without modifying the metaclass [admittedly this is a difficult/problematic solution]. In any case, the inheritence class decorator could specify special attributes in the class (it can remain compatible with __bases__) for isinstance/try to work. I agree that implementing inheritence this way is problematic [I'm convinced], but don't let that determine the fate of class decorators in general, which are more useful than metaclasses in many (most?) scenarios. From pinard at iro.umontreal.ca Sat Nov 5 17:29:49 2005 From: pinard at iro.umontreal.ca (=?iso-8859-1?Q?Fran=E7ois?= Pinard) Date: Sat, 5 Nov 2005 11:29:49 -0500 Subject: [Python-Dev] PEP 352 Transition Plan In-Reply-To: References: <007d01c5dc00$738da2e0$b62dc797@oemcomputer> <4362DD15.4080606@gmail.com> Message-ID: <20051105162949.GA8992@phenix.sram.qc.ca> [Guido van Rossum] > I've made a final pass over PEP 352, mostly fixing the __str__, > __unicode__ and __repr__ methods to behave more reasonably. I'm all > for accepting it now. Does anybody see any last-minute show-stopping > problems with it? I did not follow the thread, so maybe I'm out in order, be kind with me. After having read PEP 352, it is not crystal clear whether in: try: ... except: ... the "except:" will mean "except BaseException:" or "except Exception:". I would except the first, but the text beginning the section titled "Exception Hierarchy Changes" suggests it could mean the second, without really stating it. Let me argue that "except BaseException:" is preferable. First, because there is no reason to load a bare "except:" by anything but a very simple and clean meaning, like the real base of the exception hierarchy. Second, as a bare "except:" is not considered good practice on average, it would be counter-productive trying to figure out ways to make it more frequently _usable_. -- Fran?ois Pinard http://pinard.progiciels-bpi.ca From guido at python.org Sat Nov 5 18:46:54 2005 From: guido at python.org (Guido van Rossum) Date: Sat, 5 Nov 2005 09:46:54 -0800 Subject: [Python-Dev] PEP 352 Transition Plan In-Reply-To: <20051105162949.GA8992@phenix.sram.qc.ca> References: <007d01c5dc00$738da2e0$b62dc797@oemcomputer> <4362DD15.4080606@gmail.com> <20051105162949.GA8992@phenix.sram.qc.ca> Message-ID: > [Guido van Rossum] > > > I've made a final pass over PEP 352, mostly fixing the __str__, > > __unicode__ and __repr__ methods to behave more reasonably. I'm all > > for accepting it now. Does anybody see any last-minute show-stopping > > problems with it? [Fran?ois] > I did not follow the thread, so maybe I'm out in order, be kind with me. > > After having read PEP 352, it is not crystal clear whether in: > > try: > ... > except: > ... > > the "except:" will mean "except BaseException:" or "except Exception:". > I would except the first, but the text beginning the section titled > "Exception Hierarchy Changes" suggests it could mean the second, without > really stating it. This is probably a leftover from PEP 348, which did have a change for bare 'except:' in mind. PEP 352 doesn't propose to change its meaning, and if there are words that suggest this, they should be removed. Until Python 3.0, it will not change its meaning from what it is now; this is because until then, it is still *possible* (though it will become deprecated behavior) to raise string exceptions or classes that don't inherit from BaseException. > Let me argue that "except BaseException:" is preferable. First, because > there is no reason to load a bare "except:" by anything but a very > simple and clean meaning, like the real base of the exception hierarchy. > Second, as a bare "except:" is not considered good practice on average, > it would be counter-productive trying to figure out ways to make it more > frequently _usable_. What bare 'except:' will mean in Python 3.0, and whether it is even allowed at all, is up for discussion -- it will have to be a new PEP. Personally, I think bare 'except:' should be removed from the language in Python 3.0, so that all except clauses are explicit in what they catch and there isn't any confusion over whether KeyboardInterrupt, SystemExit etc. are included or not. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From noamraph at gmail.com Sat Nov 5 20:05:28 2005 From: noamraph at gmail.com (Noam Raphael) Date: Sat, 5 Nov 2005 21:05:28 +0200 Subject: [Python-Dev] Should the default equality operator compare values instead of identities? In-Reply-To: <20051102125437.F290.JCARLSON@uci.edu> References: <20051102125437.F290.JCARLSON@uci.edu> Message-ID: On 11/3/05, Josiah Carlson wrote: ... > > Right, but lists (dicts, tuples, etc.) are defined as containers, and > their comparison operation is defined on their contents. Objects are > not defined as containers in the general case, so defining comparisons > based on their contents (as opposed to identity) is just one of the two > assumptions to be made. > > I personally like the current behavior, and I see no /compelling/ reason > to change it. You obviously feel so compelled for the behavior to > change that you are willing to express your desires. How about you do > something more productive and produce a patch which implements the > changes you want, verify that it passes tests in the standard library, > then post it on sourceforge. If someone is similarly compelled and > agrees with you (so far I've not seen any public support for your > proposal by any of the core developers), the discussion will restart, > and it will be decided (not by you or I). Thanks for the advice - I will try to do as you suggest. > > > > To summarize, I think that value-based equality testing would usually > > be what you want, and currently implementing it is a bit of a pain. > > Actually, implementing value-based equality testing, when you have a > finite set of values you want to test, is quite easy. > > def __eq__(self, other): > for i in self.__cmp_eq__: > if getattr(self, i) != getattr(other, i): > return False > return True > > With a simple metaclass that discovers all of those values automatically, > and/or your own protocol for exclusion, and you are done. Remember, not > all 5-line functions should become builtin/default behavior, and this > implementation shows that it is not a significant burdon for you (or > anyone else) to implement this in your own custom library. > You are right that not all 5-line functions should become builtin/default behaviour. However, I personally think that this one should, since: 1. It doesn't add complexity, or a new builtin. 2. Those five line doesn't include the metaclass code, which will probably take more than five lines and won't be trivial. 3. It will make other objects behave better, not only mine - other classes will get a meaningful comparison operator, for free. > > P.S. One thing that you should remember is that even if your patch is > accepted, and even if this is desireable, Python 2.5 is supposed to be > released sometime next year (spring/summer?), and because it is a > backwards incompatible change, would need at least 2.6-2.7 before it > becomes the default behavior without a __future__ import, which is > another 3-4 years down the line. I hope that the warning can go in by Python 2.5, so the change (which I think will cause relatively few backwards incompatibility problems) can go in by Python 2.6, which I think is less than 2 years down the line. > > I understand you are passionate, really I do (you should see some of my > proposals), but by the time these things get around to getting into > mainline Python, there are high odds that you probably won't care about > them much anymore (I've come to feel that way myself about many of my > proposals), and I think it is a good idea to attempt to balance - when > it comes to Python - "Now is better than never." and "Although never is > often better than *right* now." > > Removing __hash__, changing __eq__, and trying to get in copy-on-write > freezing (which is really copy-and-cache freezing), all read to me like > "We gotta do this now!", which certainly isn't helping the proposal. > Thanks - I should really calm down a bit. I will try to go "safe and slowly", and I hope that at the end I will succeed in making my own small contribution to Python. Noam From jcarlson at uci.edu Sat Nov 5 21:30:17 2005 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat, 05 Nov 2005 12:30:17 -0800 Subject: [Python-Dev] Should the default equality operator compare values instead of identities? In-Reply-To: References: <20051102125437.F290.JCARLSON@uci.edu> Message-ID: <20051105115816.BFE3.JCARLSON@uci.edu> Noam Raphael wrote: > On 11/3/05, Josiah Carlson wrote: > > > To summarize, I think that value-based equality testing would usually > > > be what you want, and currently implementing it is a bit of a pain. > > > > Actually, implementing value-based equality testing, when you have a > > finite set of values you want to test, is quite easy. > > > > def __eq__(self, other): > > for i in self.__cmp_eq__: > > if getattr(self, i) != getattr(other, i): > > return False > > return True > > > > With a simple metaclass that discovers all of those values automatically, > > and/or your own protocol for exclusion, and you are done. Remember, not > > all 5-line functions should become builtin/default behavior, and this > > implementation shows that it is not a significant burdon for you (or > > anyone else) to implement this in your own custom library. > > > You are right that not all 5-line functions should become > builtin/default behaviour. However, I personally think that this one > should, since: > 1. It doesn't add complexity, or a new builtin. It changes default behavior (which I specified as a portion of my statement, which you quote. And you are wrong, it adds complexity to the implementation of both class instantiation and the default comparison mechanism. The former, I believe, you will find more difficult to patch than the comparison, though if you have not yet had adventures in that which is writing C extension modules, modifying the default class instantiation may be the deal breaker for you (I personally would have no idea where to start). > 2. Those five line doesn't include the metaclass code, which will > probably take more than five lines and won't be trivial. class eqMetaclass(type): def __new__(cls, name, bases, dct): if '__cmp_include__' in dct: include = dict.fromkeys(dct['__cmp_include__']) else: include = dict.fromkeys(dct.keys) for i in dct.get('__cmp_exclude__'): _ = include.pop(i, None) dct['__cmp_eq__'] = include.keys() return type.__new__(cls, name, bases, dct) It took 10 lines of code, and was trivial (except for not-included multi-metaclass support code, which is being discussed in another thread). Oh, I suppose I should modify that __eq__ definition to be smarter about comparison... def __eq__(self, other): if not hasattr(other, '__cmp_eq__'): return False if dict.fromkeys(self.__cmp_eq__) != \ dict.fromkeys(other.__cmp_eq__): return False for i in self.__cmp_eq__: if getattr(self, i) != getattr(other, i): return False return True Wow, 20 lines of support code, how could one ever expect users to write that? ;) > 3. It will make other objects behave better, not only mine - other > classes will get a meaningful comparison operator, for free. You are that the comparison previously wasn't "meaningful". It has a meaning, though it may not be exactly what you wanted it to be, which is why Python allows users to define __eq__ operators to be exactly what they want, and which is why I don't find your uses compelling. > > P.S. One thing that you should remember is that even if your patch is > > accepted, and even if this is desireable, Python 2.5 is supposed to be > > released sometime next year (spring/summer?), and because it is a > > backwards incompatible change, would need at least 2.6-2.7 before it > > becomes the default behavior without a __future__ import, which is > > another 3-4 years down the line. > > I hope that the warning can go in by Python 2.5, so the change (which > I think will cause relatively few backwards incompatibility problems) > can go in by Python 2.6, which I think is less than 2 years down the > line. As per historical release schedules (available in PEP form at www.python.org/peps), alpha 1 to final generally takes 6 months. It then takes at least a year before the alpha 1 of the following version is to be released. Being that 2.4 final was released November 2004, and we've not seen an alpha for 2.5 yet, we are at least 6 months (according to history) from 2.5 final, and at least 2 years from 2.6 final. From what I have previously learned from others in python-dev, the warnings machinery is slow, so one is to be wary of using warnings unless absolutely necessary. Regardless of it being absolutely necessary, it would be 2 years at least before the feature would actually make it into Python and become default behavior, IF it were desireable default behavior. > Thanks - I should really calm down a bit. I will try to go "safe and > slowly", and I hope that at the end I will succeed in making my own > small contribution to Python. You should also realize that you can make contributions to Python without changing the language or the implementation of the langauge. Read and review patches, help with bug reports, hang out on python-list and attempt to help the hundreds (if not thousands) of users who are asking for help, try to help new users in python-tutor, etc. If you have an idea for a language change, offer it up on python-list first (I've forgotten to do this more often than I would like to admit), and if it generally has more "cool" than "ick", then bring it back here. - Josiah From martin at v.loewis.de Sat Nov 5 22:41:21 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 05 Nov 2005 22:41:21 +0100 Subject: [Python-Dev] Why should the default hash(x) == id(x)? In-Reply-To: References: Message-ID: <436D2701.6080400@v.loewis.de> Noam Raphael wrote: > Is there a reason why the default __hash__ method returns the id of the objects? You are asking "why" question of the kind which are best answered as "why not". IOW, you are saying that the current behaviour is bad, but you are not proposing any alternative behaviour. There are many alternatives possible, and they are presumably all worse than the current implementation. To give an example: "why does hash() return id()"? Answer: The alternative would be that hash() returns always 0 unless implemented otherwise. This would cause serious performance issues for people using the objects as dictionary keys. If they don't do that, it doesn't matter what hash() returns. > This leads me to another question: why should the default __eq__ > method be the same as "is"? Because the alternative would be to always return "False". This would be confusing, because it would cause "x == x" to give False. More generally, I claim that the current behaviour is better than *any* alternative. To refute this claim, you would have to come up with an alternative first. Regards, Martin From noamraph at gmail.com Sun Nov 6 00:00:16 2005 From: noamraph at gmail.com (Noam Raphael) Date: Sun, 6 Nov 2005 01:00:16 +0200 Subject: [Python-Dev] Should the default equality operator compare values instead of identities? In-Reply-To: <20051105115816.BFE3.JCARLSON@uci.edu> References: <20051102125437.F290.JCARLSON@uci.edu> <20051105115816.BFE3.JCARLSON@uci.edu> Message-ID: On 11/5/05, Josiah Carlson wrote: ... > > 1. It doesn't add complexity, or a new builtin. > > It changes default behavior (which I specified as a portion of my > statement, which you quote. > > And you are wrong, it adds complexity to the implementation of both > class instantiation and the default comparison mechanism. The former, I > believe, you will find more difficult to patch than the comparison, > though if you have not yet had adventures in that which is writing C > extension modules, modifying the default class instantiation may be > the deal breaker for you (I personally would have no idea where to start). Sorry, I meant complexity to the Python user - it won't require him to learn more in order to write programs in Python. > > class eqMetaclass(type): > def __new__(cls, name, bases, dct): > if '__cmp_include__' in dct: > include = dict.fromkeys(dct['__cmp_include__']) > else: > include = dict.fromkeys(dct.keys) > > for i in dct.get('__cmp_exclude__'): > _ = include.pop(i, None) > > dct['__cmp_eq__'] = include.keys() > return type.__new__(cls, name, bases, dct) > > It took 10 lines of code, and was trivial (except for not-included > multi-metaclass support code, which is being discussed in another thread). > > Oh, I suppose I should modify that __eq__ definition to be smarter about > comparison... > > def __eq__(self, other): > if not hasattr(other, '__cmp_eq__'): > return False > if dict.fromkeys(self.__cmp_eq__) != \ > dict.fromkeys(other.__cmp_eq__): > return False > for i in self.__cmp_eq__: > if getattr(self, i) != getattr(other, i): > return False > return True Thanks for the implementation. It would be very useful in order to explain my suggestion. It's nice that it compares only attributes, not types. It makes it possible for two people to write classes that can be equal to one another. > > Wow, 20 lines of support code, how could one ever expect users to write > that? ;) This might mean that implementing it in C, once I find the right place, won't be too difficult. And I think that for most users it will be harder than it was for you, and there are some subtleties in those lines. > > > > 3. It will make other objects behave better, not only mine - other > > classes will get a meaningful comparison operator, for free. > > You are that the comparison previously wasn't "meaningful". It has a > meaning, though it may not be exactly what you wanted it to be, which is > why Python allows users to define __eq__ operators to be exactly what > they want, and which is why I don't find your uses compelling. > I think that value-based equality testing is a better default, since in more cases it does what you want it to, and since in those cases they won't have to write those 20 lines, or download them from somewhere. > ... > > From what I have previously learned from others in python-dev, the > warnings machinery is slow, so one is to be wary of using warnings > unless absolutely necessary. Regardless of it being absolutely necessary, > it would be 2 years at least before the feature would actually make it > into Python and become default behavior, IF it were desireable default > behavior. All right. I hope that those warnings will be ok - it's yet to be seen. And about those 2 years - better later than never. ... > > You should also realize that you can make contributions to Python > without changing the language or the implementation of the langauge. > Read and review patches, help with bug reports, hang out on python-list > and attempt to help the hundreds (if not thousands) of users who are > asking for help, try to help new users in python-tutor, etc. I confess that I don't do these a lot. I can say that I from time to time teach beginners Python, and that where I work I help a lot of other people with Python. > If you > have an idea for a language change, offer it up on python-list first > (I've forgotten to do this more often than I would like to admit), and > if it generally has more "cool" than "ick", then bring it back here. > I will. Thanks again. Noam From jcarlson at uci.edu Sun Nov 6 00:40:34 2005 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat, 05 Nov 2005 15:40:34 -0800 Subject: [Python-Dev] Should the default equality operator compare values instead of identities? In-Reply-To: References: <20051105115816.BFE3.JCARLSON@uci.edu> Message-ID: <20051105151436.BFF7.JCARLSON@uci.edu> Noam Raphael wrote: > > On 11/5/05, Josiah Carlson wrote: > ... > > > 1. It doesn't add complexity, or a new builtin. > > > > It changes default behavior (which I specified as a portion of my > > statement, which you quote. > > > > And you are wrong, it adds complexity to the implementation of both > > class instantiation and the default comparison mechanism. The former, I > > believe, you will find more difficult to patch than the comparison, > > though if you have not yet had adventures in that which is writing C > > extension modules, modifying the default class instantiation may be > > the deal breaker for you (I personally would have no idea where to start). > > Sorry, I meant complexity to the Python user - it won't require him to > learn more in order to write programs in Python. Ahh, but it does add complexity. Along with knowing __doc__, __slots__, __metaclass__, __init__, __new__, __cmp__, __eq__, ..., __str__, __repr__, __getitem__, __setitem__, __delitem__, __getattr__, __setattr__, __delattr__, ... The user must also know what __cmp_include__ and __cmp_exclude__ means in order to understand code which uses them, and they must understand that exclude entries overwrite include entries. > > Wow, 20 lines of support code, how could one ever expect users to write > > that? ;) > > This might mean that implementing it in C, once I find the right > place, won't be too difficult. > > And I think that for most users it will be harder than it was for you, > and there are some subtleties in those lines. So put it in the Python Cookbook: http://aspn.activestate.com/ASPN/Cookbook/Python > > > 3. It will make other objects behave better, not only mine - other > > > classes will get a meaningful comparison operator, for free. > > > > You are that the comparison previously wasn't "meaningful". It has a > > meaning, though it may not be exactly what you wanted it to be, which is > > why Python allows users to define __eq__ operators to be exactly what > > they want, and which is why I don't find your uses compelling. > > > I think that value-based equality testing is a better default, since > in more cases it does what you want it to, and since in those cases > they won't have to write those 20 lines, or download them from > somewhere. You are making a value judgement on what people want to happen with default Python. Until others state that they want such an operation as a default, I'm going to consider this particular argument relatively unfounded. > > From what I have previously learned from others in python-dev, the > > warnings machinery is slow, so one is to be wary of using warnings > > unless absolutely necessary. Regardless of it being absolutely necessary, > > it would be 2 years at least before the feature would actually make it > > into Python and become default behavior, IF it were desireable default > > behavior. > > All right. I hope that those warnings will be ok - it's yet to be > seen. And about those 2 years - better later than never. It won't be OK. Every comparison using the default operator will incur a speed penalty while it checks the (pure Python) warning machinery to determine if the warning has been issued yet. This alone makes the transition require a __future__ import. - Josiah From noamraph at gmail.com Sun Nov 6 01:02:36 2005 From: noamraph at gmail.com (Noam Raphael) Date: Sun, 6 Nov 2005 02:02:36 +0200 Subject: [Python-Dev] Should the default equality operator compare values instead of identities? In-Reply-To: <20051105151436.BFF7.JCARLSON@uci.edu> References: <20051105115816.BFE3.JCARLSON@uci.edu> <20051105151436.BFF7.JCARLSON@uci.edu> Message-ID: On 11/6/05, Josiah Carlson wrote: ... > > > > Sorry, I meant complexity to the Python user - it won't require him to > > learn more in order to write programs in Python. > > Ahh, but it does add complexity. Along with knowing __doc__, __slots__, > __metaclass__, __init__, __new__, __cmp__, __eq__, ..., __str__, > __repr__, __getitem__, __setitem__, __delitem__, __getattr__, > __setattr__, __delattr__, ... > > > The user must also know what __cmp_include__ and __cmp_exclude__ means > in order to understand code which uses them, and they must understand > that exclude entries overwrite include entries. > You are right. But that's Python - I think that nobody knows all the exact details of what all these do. You look in the documentation. It is a compliation - but it's of the type that I can live with, if there's a reason. > > > > Wow, 20 lines of support code, how could one ever expect users to write > > > that? ;) > > > > This might mean that implementing it in C, once I find the right > > place, won't be too difficult. > > > > And I think that for most users it will be harder than it was for you, > > and there are some subtleties in those lines. > > So put it in the Python Cookbook: > http://aspn.activestate.com/ASPN/Cookbook/Python > A good idea. > > > > > 3. It will make other objects behave better, not only mine - other > > > > classes will get a meaningful comparison operator, for free. > > > > > > You are that the comparison previously wasn't "meaningful". It has a > > > meaning, though it may not be exactly what you wanted it to be, which is > > > why Python allows users to define __eq__ operators to be exactly what > > > they want, and which is why I don't find your uses compelling. > > > > > I think that value-based equality testing is a better default, since > > in more cases it does what you want it to, and since in those cases > > they won't have to write those 20 lines, or download them from > > somewhere. > > You are making a value judgement on what people want to happen with > default Python. Until others state that they want such an operation as a > default, I'm going to consider this particular argument relatively > unfounded. > All right. I will try to collect more examples for my proposal. > > > > From what I have previously learned from others in python-dev, the > > > warnings machinery is slow, so one is to be wary of using warnings > > > unless absolutely necessary. Regardless of it being absolutely necessary, > > > it would be 2 years at least before the feature would actually make it > > > into Python and become default behavior, IF it were desireable default > > > behavior. > > > > All right. I hope that those warnings will be ok - it's yet to be > > seen. And about those 2 years - better later than never. > > It won't be OK. Every comparison using the default operator will incur > a speed penalty while it checks the (pure Python) warning machinery to > determine if the warning has been issued yet. This alone makes the > transition require a __future__ import. > How will the __future__ statement help? I think that the warning is still needed, so that people using code that may stop working will know about it. I see that they can add a __future__ import and see if it still works, but it will catch much fewer problems, because usually code would be run without the __future__ import. If it really slows down things, it seems to me that the only solution is to optimize the warning module... Noam From noamraph at gmail.com Sun Nov 6 01:03:22 2005 From: noamraph at gmail.com (Noam Raphael) Date: Sun, 6 Nov 2005 02:03:22 +0200 Subject: [Python-Dev] Why should the default hash(x) == id(x)? In-Reply-To: <436D2701.6080400@v.loewis.de> References: <436D2701.6080400@v.loewis.de> Message-ID: On 11/5/05, "Martin v. L?wis" wrote: > More generally, I claim that the current behaviour is better than > *any* alternative. To refute this claim, you would have to come > up with an alternative first. > The alternative is to drop the __hash__ method of user-defined classes (as Guido already decided to do), and to make the default __eq__ method compare the two objects' __dict__ and slot members. See the thread about default equality operator - Josiah Carlson posted there a metaclass implementing this equality operator. Noam From jcarlson at uci.edu Sun Nov 6 01:19:49 2005 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat, 05 Nov 2005 16:19:49 -0800 Subject: [Python-Dev] Why should the default hash(x) == id(x)? In-Reply-To: References: <436D2701.6080400@v.loewis.de> Message-ID: <20051105161846.C004.JCARLSON@uci.edu> Noam Raphael wrote: > > On 11/5/05, "Martin v. L?wis" wrote: > > More generally, I claim that the current behaviour is better than > > *any* alternative. To refute this claim, you would have to come > > up with an alternative first. > > > The alternative is to drop the __hash__ method of user-defined classes > (as Guido already decided to do), and to make the default __eq__ > method compare the two objects' __dict__ and slot members. > > See the thread about default equality operator - Josiah Carlson posted > there a metaclass implementing this equality operator. The existance of a simple equality operator and metaclass is actually a strike against changing the default behavior for equality. - Josiah From pedronis at strakt.com Sun Nov 6 01:29:18 2005 From: pedronis at strakt.com (Samuele Pedroni) Date: Sun, 06 Nov 2005 01:29:18 +0100 Subject: [Python-Dev] Why should the default hash(x) == id(x)? In-Reply-To: References: <436D2701.6080400@v.loewis.de> Message-ID: <436D4E5E.7000301@strakt.com> Noam Raphael wrote: > On 11/5/05, "Martin v. L?wis" wrote: > >>More generally, I claim that the current behaviour is better than >>*any* alternative. To refute this claim, you would have to come >>up with an alternative first. >> > > The alternative is to drop the __hash__ method of user-defined classes > (as Guido already decided to do), and to make the default __eq__ > method compare the two objects' __dict__ and slot members. > no, whether object has an __hash__ and what is the default hashing are different issues. Also all this discussion should have started and lived on comp.lang.python and this is a good point as any to rectify this. > See the thread about default equality operator - Josiah Carlson posted > there a metaclass implementing this equality operator. > > Noam > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/pedronis%40strakt.com From jcarlson at uci.edu Sun Nov 6 01:30:52 2005 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat, 05 Nov 2005 16:30:52 -0800 Subject: [Python-Dev] Should the default equality operator compare values instead of identities? In-Reply-To: References: <20051105151436.BFF7.JCARLSON@uci.edu> Message-ID: <20051105162001.C007.JCARLSON@uci.edu> Noam Raphael wrote: > > On 11/6/05, Josiah Carlson wrote: > ... > > > > > > Sorry, I meant complexity to the Python user - it won't require him to > > > learn more in order to write programs in Python. > You are right. But that's Python - I think that nobody knows all the > exact details of what all these do. You look in the documentation. It > is a compliation - but it's of the type that I can live with, if > there's a reason. Regardless of whether people check the documentation, it does add complexity to Python. > > > All right. I hope that those warnings will be ok - it's yet to be > > > seen. And about those 2 years - better later than never. > > > > It won't be OK. Every comparison using the default operator will incur > > a speed penalty while it checks the (pure Python) warning machinery to > > determine if the warning has been issued yet. This alone makes the > > transition require a __future__ import. > > > How will the __future__ statement help? I think that the warning is > still needed, so that people using code that may stop working will > know about it. I see that they can add a __future__ import and see if > it still works, but it will catch much fewer problems, because usually > code would be run without the __future__ import. What has been common is to use __future__ along with a note in the release notes specifying the changes between 2.x and 2.x-1. The precise mechanisms when using __future__ vary from import to import, though this one could signal the change of a single variable as to which code path to use. > If it really slows down things, it seems to me that the only solution > is to optimize the warning module... Possible solutions to possible problem of default __eq__ behavior: 1. It is not a problem, leave it alone. 2. Use __future__. 3. Use warnings, and deal with it being slow. 4. Make warnings a C module and expose it to CPython internals. You are claiming that there is such a need to fix __eq__ that one would NEEDs to change the warnings module so that the __eq__ fix can be fast. Again, implement this, post it to sourceforge, and someone will decide. - Josiah From martin at v.loewis.de Sun Nov 6 11:08:22 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 06 Nov 2005 11:08:22 +0100 Subject: [Python-Dev] Why should the default hash(x) == id(x)? In-Reply-To: References: <436D2701.6080400@v.loewis.de> Message-ID: <436DD616.3080304@v.loewis.de> Noam Raphael wrote: > The alternative is to drop the __hash__ method of user-defined classes > (as Guido already decided to do), and to make the default __eq__ > method compare the two objects' __dict__ and slot members. The question then is what hash(x) would do. It seems that you expect it then somehow not to return a value. However, under this patch, the fallback implementation (use pointer as the hash) would be used, which would preserve hash(x)==id(x). > See the thread about default equality operator - Josiah Carlson posted > there a metaclass implementing this equality operator. This will likely cause a lot of breakage. Objects will compare equal even though they conceptually are not, and even though they did not compare equal in previous Python versions. Regards, Martin From jim at zope.com Sun Nov 6 17:15:58 2005 From: jim at zope.com (Jim Fulton) Date: Sun, 06 Nov 2005 11:15:58 -0500 Subject: [Python-Dev] For Python 3k, drop default/implicit hash, and comparison Message-ID: <436E2C3E.7060807@zope.com> The recent discussion about what the default hash and equality comparisons should do makes me want to chime in. IMO, the provision of defaults for hash, eq and other comparisons was a mistake. I'm especially sensitive to this because I do a lot of work with persistent data that outlives program execution. For such objects, memory address is meaningless. In particular, the default ordering of objects based in address has caused a great deal of pain to people who store data in persistent BTrees. Oddly, what I've read in these threads seems to be arguing about which implicit method is best. The answer, IMO, is to not do this implicitly at all. If programmers want their objects to be hashable, comparable, or orderable, then they should implement operators explicitly. There could even be a handy, but *optional*, base class that provides these operators based on ids. This would be too big a change for Python 2 but, IMO, should definately be made for Python 3k. I doubt any change in the default definition of these operations is practical for Python 2. Too many people rely on them, usually without really realizing it. Lets plan to stop guessing how to do hash and comparison. Explicit is better than implicit. :) Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From guido at python.org Sun Nov 6 20:47:20 2005 From: guido at python.org (Guido van Rossum) Date: Sun, 6 Nov 2005 11:47:20 -0800 Subject: [Python-Dev] For Python 3k, drop default/implicit hash, and comparison In-Reply-To: <436E2C3E.7060807@zope.com> References: <436E2C3E.7060807@zope.com> Message-ID: On 11/6/05, Jim Fulton wrote: > IMO, the provision of defaults for hash, eq and other comparisons > was a mistake. I agree with you for 66%. Default hash and inequalities were a mistake. But I wouldn't want to do without a default ==/!= implementation (and of course it should be defined so that an object is only equal to itself). In fact, the original hash() was clever enough to complain when __eq__ (or __cmp__) was overridden but __hash__ wasn't; but this got lost by accident for new-style classes when I added a default __hash__ to the new universal base class (object). But I think the original default hash() isn't particularly useful, so I think it's better to just not be hashable unless __hash__ is defined explicitly. > I'm especially sensitive to this because I do a lot > of work with persistent data that outlives program execution. For such > objects, memory address is meaningless. In particular, the default > ordering of objects based in address has caused a great deal of pain > to people who store data in persistent BTrees. This argues against the inequalities (<, <=, >, >=) and I agree. > Oddly, what I've read in these threads seems to be arguing about > which implicit method is best. The answer, IMO, is to not do this > implicitly at all. If programmers want their objects to be > hashable, comparable, or orderable, then they should implement operators > explicitly. There could even be a handy, but *optional*, base class that > provides these operators based on ids. I don't like that final suggestion. Before you know it, a meme develops telling newbies that all classes should inherit from that "optional" base class, and then later it's impossible to remove it because you can't tell whether it's actually needed or not. > This would be too big a change for Python 2 but, IMO, should definately > be made for Python 3k. I doubt any change in the default definition > of these operations is practical for Python 2. Too many people rely on > them, usually without really realizing it. Agreed. > Lets plan to stop guessing how to do hash and comparison. > > Explicit is better than implicit. :) Except that I really don't think that there's anything wrong with a default __eq__ that uses object identity. As Martin pointed out, it's just too weird that an object wouldn't be considered equal to itself. It's the default __hash__ and __cmp__ that mess things up. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jim at zope.com Sun Nov 6 21:13:23 2005 From: jim at zope.com (Jim Fulton) Date: Sun, 06 Nov 2005 15:13:23 -0500 Subject: [Python-Dev] For Python 3k, drop default/implicit hash, and comparison In-Reply-To: References: <436E2C3E.7060807@zope.com> Message-ID: <436E63E3.7040307@zope.com> Guido van Rossum wrote: > On 11/6/05, Jim Fulton wrote: > ... > Except that I really don't think that there's anything wrong with a > default __eq__ that uses object identity. As Martin pointed out, it's > just too weird that an object wouldn't be considered equal to itself. > It's the default __hash__ and __cmp__ that mess things up. Good point. I agree. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jrw at pobox.com Sun Nov 6 21:39:42 2005 From: jrw at pobox.com (John Williams) Date: Sun, 06 Nov 2005 14:39:42 -0600 Subject: [Python-Dev] For Python 3k, drop default/implicit hash, and comparison In-Reply-To: <436E2C3E.7060807@zope.com> References: <436E2C3E.7060807@zope.com> Message-ID: <436E6A0E.4070508@pobox.com> (This is kind of on a tangent to the original discussion, but I don't want to create yet another subject line about object comparisons.) Lately I've found that virtually all my implementations of __cmp__, __hash__, etc. can be factored into this form inspired by the "key" parameter to the built-in sorting functions: class MyClass: def __key(self): # Return a tuple of attributes to compare. return (self.foo, self.bar, ...) def __cmp__(self, that): return cmp(self.__key(), that.__key()) def __hash__(self): return hash(self.__key()) I wonder if it wouldn't make sense to formalize this pattern with a magic __key__ method such that a class with a __key__ method would behave as if it had interited the definitions of __cmp__ and __hash__ above. This scheme would eliminate the tedium of keeping the __hash__ method in sync with the __cmp__/__eq__ method, and writing a __key__ method would involve writing less code than a naive __eq__ method, since each attribute name only needs to be mentioned once instead of appearing on either side of a "==" expression. On the other hand, this idea doesn't work in all situations (for instance, I don't think you could define the default __cmp__/__hash__ semantics in terms of __key__), it would only eliminate two one-line methods for each class, and it would further complicate the "==" operator (__key__, falling back to __eq__, falling back to __cmp__, falling back to object identity--ouch!) If anyone thinks this is a good idea I'll investiate how many places in the standard library this pattern would apply. --jw From guido at python.org Sun Nov 6 21:58:57 2005 From: guido at python.org (Guido van Rossum) Date: Sun, 6 Nov 2005 12:58:57 -0800 Subject: [Python-Dev] For Python 3k, drop default/implicit hash, and comparison In-Reply-To: <436E6A0E.4070508@pobox.com> References: <436E2C3E.7060807@zope.com> <436E6A0E.4070508@pobox.com> Message-ID: On 11/6/05, John Williams wrote: > (This is kind of on a tangent to the original discussion, but I don't > want to create yet another subject line about object comparisons.) > > Lately I've found that virtually all my implementations of __cmp__, > __hash__, etc. can be factored into this form inspired by the "key" > parameter to the built-in sorting functions: > > class MyClass: > > def __key(self): > # Return a tuple of attributes to compare. > return (self.foo, self.bar, ...) > > def __cmp__(self, that): > return cmp(self.__key(), that.__key()) > > def __hash__(self): > return hash(self.__key()) The main way this breaks down is when comparing objects of different types. While most comparisons typically are defined in terms of comparisons on simpler or contained objects, two objects of different types that happen to have the same "key" shouldn't necessarily be considered equal. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Sun Nov 6 22:22:31 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 06 Nov 2005 16:22:31 -0500 Subject: [Python-Dev] For Python 3k, drop default/implicit hash, and comparison In-Reply-To: References: <436E6A0E.4070508@pobox.com> <436E2C3E.7060807@zope.com> <436E6A0E.4070508@pobox.com> Message-ID: <5.1.1.6.0.20051106162127.01ede358@mail.telecommunity.com> At 12:58 PM 11/6/2005 -0800, Guido van Rossum wrote: >The main way this breaks down is when comparing objects of different >types. While most comparisons typically are defined in terms of >comparisons on simpler or contained objects, two objects of different >types that happen to have the same "key" shouldn't necessarily be >considered equal. When I use this pattern, I often just include the object's type in the key. (I call it the 'hashcmp' value, but otherwise it's the same pattern.) From guido at python.org Sun Nov 6 22:29:27 2005 From: guido at python.org (Guido van Rossum) Date: Sun, 6 Nov 2005 13:29:27 -0800 Subject: [Python-Dev] For Python 3k, drop default/implicit hash, and comparison In-Reply-To: <5.1.1.6.0.20051106162127.01ede358@mail.telecommunity.com> References: <436E2C3E.7060807@zope.com> <436E6A0E.4070508@pobox.com> <5.1.1.6.0.20051106162127.01ede358@mail.telecommunity.com> Message-ID: On 11/6/05, Phillip J. Eby wrote: > At 12:58 PM 11/6/2005 -0800, Guido van Rossum wrote: > >The main way this breaks down is when comparing objects of different > >types. While most comparisons typically are defined in terms of > >comparisons on simpler or contained objects, two objects of different > >types that happen to have the same "key" shouldn't necessarily be > >considered equal. > > When I use this pattern, I often just include the object's type in the > key. (I call it the 'hashcmp' value, but otherwise it's the same pattern.) But how do you make that work with subclassing? (I'm guessing your answer is that you don't. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From josh at janrain.com Sun Nov 6 22:57:34 2005 From: josh at janrain.com (Josh Hoyt) Date: Sun, 6 Nov 2005 13:57:34 -0800 Subject: [Python-Dev] For Python 3k, drop default/implicit hash, and comparison In-Reply-To: References: <436E2C3E.7060807@zope.com> <436E6A0E.4070508@pobox.com> <5.1.1.6.0.20051106162127.01ede358@mail.telecommunity.com> Message-ID: <34714aad0511061357x2dc3765y2cdca412dec4e432@mail.gmail.com> On 11/6/05, Guido van Rossum wrote: > On 11/6/05, Phillip J. Eby wrote: > > When I use this pattern, I often just include the object's type in the > > key. (I call it the 'hashcmp' value, but otherwise it's the same pattern.) > > But how do you make that work with subclassing? (I'm guessing your > answer is that you don't. :-) If there is a well-defined desired behaviour for comparisons in the face of subclassing (which I'm not sure if there is) then that behaviour could become part of the definition of how __key__ works. Since __key__ would be for clarity of intent and convenience of implementation, adding default behaviour for the most common case seems like it would be a good idea. My initial thought was that all subclasses of the class where __key__ was defined would compare as equal if they return the same value. More precisely, if two objects have the same __key__ method, and it returns the same value, then they are equal. That does not solve the __cmp__ problem, unless the __key__ function is used as part of the ordering. For example: def getKey(obj): __key__ = getattr(obj.__class__, '__key__') return (id(key), key(obj)) An obvious drawback is that if __key__ is overridden, then the subclass where it is overridden and all further subclasses will no longer have equality to the superclass. I think that this is probably OK, except that it may be occasionally surprising. Josh From jcarlson at uci.edu Mon Nov 7 00:12:36 2005 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun, 06 Nov 2005 15:12:36 -0800 Subject: [Python-Dev] For Python 3k, drop default/implicit hash, and comparison In-Reply-To: <436E6A0E.4070508@pobox.com> References: <436E2C3E.7060807@zope.com> <436E6A0E.4070508@pobox.com> Message-ID: <20051106144700.C01C.JCARLSON@uci.edu> John Williams wrote: > > (This is kind of on a tangent to the original discussion, but I don't > want to create yet another subject line about object comparisons.) > > Lately I've found that virtually all my implementations of __cmp__, > __hash__, etc. can be factored into this form inspired by the "key" > parameter to the built-in sorting functions: > > class MyClass: > > def __key(self): > # Return a tuple of attributes to compare. > return (self.foo, self.bar, ...) > > def __cmp__(self, that): > return cmp(self.__key(), that.__key()) > > def __hash__(self): > return hash(self.__key()) > > I wonder if it wouldn't make sense to formalize this pattern with a > magic __key__ method such that a class with a __key__ method would > behave as if it had interited the definitions of __cmp__ and __hash__ above. You probably already realize this, but I thought I would point out the obvious. Given a suitably modified MyClass... >>> x = {} >>> a = MyClass() >>> a.a = 8 >>> x[a] = a >>> a.a = 9 >>> x[a] = a >>> >>> x {<__main__.MyClass instance at 0x007E0A08>: <__main__.MyClass instance at 0x007E 0A08>, <__main__.MyClass instance at 0x007E0A08>: <__main__.MyClass instance at 0x007E0A08>} Of course everyone is saying "Josiah, people shouldn't be doing that"; but they will. Given a mechanism to offer hash-by-value, a large number of users will think that it will work for what they want, regardless of the fact that in order for it to really work, those attributes must be read-only by semantics or access mechanisms. Not everyone who uses Python understands fully the concepts of mutability and immutability, and very few will realize that the attributes returned by __key() need to be immutable aspects of the instance of that class (you can perform at most one assignment to the attribute during its lifetime, and that assignment must occur before any hash calls). Call me a pessimist, but I don't believe that using magical key methods will be helpful for understanding or using Python. - Josiah From pje at telecommunity.com Mon Nov 7 01:12:21 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 06 Nov 2005 19:12:21 -0500 Subject: [Python-Dev] For Python 3k, drop default/implicit hash, and comparison In-Reply-To: References: <5.1.1.6.0.20051106162127.01ede358@mail.telecommunity.com> <436E2C3E.7060807@zope.com> <436E6A0E.4070508@pobox.com> <5.1.1.6.0.20051106162127.01ede358@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051106191059.01edcf78@mail.telecommunity.com> At 01:29 PM 11/6/2005 -0800, Guido van Rossum wrote: >On 11/6/05, Phillip J. Eby wrote: > > At 12:58 PM 11/6/2005 -0800, Guido van Rossum wrote: > > >The main way this breaks down is when comparing objects of different > > >types. While most comparisons typically are defined in terms of > > >comparisons on simpler or contained objects, two objects of different > > >types that happen to have the same "key" shouldn't necessarily be > > >considered equal. > > > > When I use this pattern, I often just include the object's type in the > > key. (I call it the 'hashcmp' value, but otherwise it's the same pattern.) > >But how do you make that work with subclassing? (I'm guessing your >answer is that you don't. :-) By either changing the subclass __init__ to initialize it with a different hashcmp value, or by redefining the method that computes it. From pje at telecommunity.com Mon Nov 7 01:15:01 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 06 Nov 2005 19:15:01 -0500 Subject: [Python-Dev] For Python 3k, drop default/implicit hash, and comparison In-Reply-To: <5.1.1.6.0.20051106191059.01edcf78@mail.telecommunity.com> References: <5.1.1.6.0.20051106162127.01ede358@mail.telecommunity.com> <436E2C3E.7060807@zope.com> <436E6A0E.4070508@pobox.com> <5.1.1.6.0.20051106162127.01ede358@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051106191251.01fa9818@mail.telecommunity.com> At 07:12 PM 11/6/2005 -0500, Phillip J. Eby wrote: >At 01:29 PM 11/6/2005 -0800, Guido van Rossum wrote: > >On 11/6/05, Phillip J. Eby wrote: > > > At 12:58 PM 11/6/2005 -0800, Guido van Rossum wrote: > > > >The main way this breaks down is when comparing objects of different > > > >types. While most comparisons typically are defined in terms of > > > >comparisons on simpler or contained objects, two objects of different > > > >types that happen to have the same "key" shouldn't necessarily be > > > >considered equal. > > > > > > When I use this pattern, I often just include the object's type in the > > > key. (I call it the 'hashcmp' value, but otherwise it's the same > pattern.) > > > >But how do you make that work with subclassing? (I'm guessing your > >answer is that you don't. :-) > >By either changing the subclass __init__ to initialize it with a different >hashcmp value, or by redefining the method that computes it. Scratch that. I realized 2 seconds after hitting "Send" that you meant the case where you want to compare instances with a common parent type. And the answer is, I can't recall having needed to. (Which is probably why it took me so long to realize what you meant.) From fakeaddress at nowhere.org Thu Nov 3 01:55:01 2005 From: fakeaddress at nowhere.org (Bryan Olson) Date: Wed, 02 Nov 2005 16:55:01 -0800 Subject: [Python-Dev] PEP submission broken? Message-ID: <43695FE5.1080803@nowhere.org> Though I tried to submit a (pre-) PEP in the proper form through the proper channels, it has disappeared into the ether. In building a class that supports Python's slicing interface, http://groups.google.com/group/comp.lang.python/msg/8f35464483aa7d7b I encountered a Python bug, which, upon further discussion, seemed to be a combination of a wart and a documentation error. http://groups.google.com/group/comp.lang.python/browse_frm/thread/402d770b6f503c27 I submitted the bug report via SourceForge; the resolution was to document the actual behavior. Next I worked out what behavior I think would eliminate the wart, wrote it up as a pre-PEP, and sent it peps at python.org on 27 Aug of this year. I promptly received an automated response from Barry Warsaw, saying, in part, "I get so much email that I can't promise a personal response." I gathered that he is a PEP editor. I did not infer from his reply that PEP's are simply ignored, but this automated reply was the only response I ever received. I subscribed to the Python-dev list, and watched, and waited; nothing on my concern appeared. One response on the comp.lang.python newsgroup noted that a popular extention module would have difficulty maintaining consistency with my proposed PEP. My proposal does not break how the extension currently works, but still, that's a valid point. There are variations which do not have that problem, and I think I can see a course that will serve the entire Python community. From what I can tell, We need to address fixing the PEP process before there is any point in working on PEP's, -- --Bryan From ncoghlan at gmail.com Mon Nov 7 10:32:57 2005 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 07 Nov 2005 19:32:57 +1000 Subject: [Python-Dev] PEP submission broken? In-Reply-To: <43695FE5.1080803@nowhere.org> References: <43695FE5.1080803@nowhere.org> Message-ID: <436F1F49.90606@gmail.com> Bryan Olson wrote: > From what I can tell, We need to address fixing the > PEP process before there is any point in working on PEP's, I think this is a somewhat fair point (although perhaps a bit overstated) - David and Barry can be busy IRL, which can certainly slow down the process of PEP submission. PEP 328 hung in limbo for a while on that basis (I'm going to have to look into if and how PEP 328 relates to Python eggs one of these days. . .). Would it be worth having a PEP category on the RFE tracker, rather than submitting pre-PEP's directly to the PEP editors? The process still wouldn't be perfect, but it would widen the pool of people that can bring a pre-PEP to the attention of python-dev. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com From bjourne at gmail.com Mon Nov 7 14:06:11 2005 From: bjourne at gmail.com (=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=) Date: Mon, 7 Nov 2005 14:06:11 +0100 Subject: [Python-Dev] Should the default equality operator compare values instead of identities? In-Reply-To: <20051105162001.C007.JCARLSON@uci.edu> References: <20051105151436.BFF7.JCARLSON@uci.edu> <20051105162001.C007.JCARLSON@uci.edu> Message-ID: <740c3aec0511070506v6833686ag4894270034e01559@mail.gmail.com> How would the value equality operator deal with recursive objects? class Foo: def __init__(self): self.foo = self Seems to me that it would take atleast some special-casing to get Foo() == Foo() to evalute to True in this case... -- mvh Bj?rn From guido at python.org Mon Nov 7 18:10:15 2005 From: guido at python.org (Guido van Rossum) Date: Mon, 7 Nov 2005 09:10:15 -0800 Subject: [Python-Dev] For Python 3k, drop default/implicit hash, and comparison In-Reply-To: <5.1.1.6.0.20051106191251.01fa9818@mail.telecommunity.com> References: <436E2C3E.7060807@zope.com> <436E6A0E.4070508@pobox.com> <5.1.1.6.0.20051106162127.01ede358@mail.telecommunity.com> <5.1.1.6.0.20051106191059.01edcf78@mail.telecommunity.com> <5.1.1.6.0.20051106191251.01fa9818@mail.telecommunity.com> Message-ID: Two more thoughts in this thread. (1) The "key" idiom is a great pattern but I don't think it would work well to make it a standard language API. (2) I'm changing my mind about the default hash(). The original default hash() (which would raise TypeError if __eq__ was overridden but __hash__ was not) is actually quite useful in some situations. Basically, simplifying a bit, there are two types of objects: those that represent *values* and those that do not. For value-ish objects, overriding __eq__ is common and then __hash__ needs to be overridden in order to get the proper dict and set behavior. In a sense, __eq__ defines an "equivalence class" in the mathematical sense. But in many applications I've used objects for which object identity is important. Let me construct a hypothetical example: suppose we represent a car and its parts as objects. Let's say each wheel is an object. Each wheel is unique and we don't have equivalency classes for them. However, it would be useful to construct sets of wheels (e.g. the set of wheels currently on my car that have never had a flat tire). Python sets use hashing just like dicts. The original hash() and __eq__ implementation would work exactly right for this purpose, and it seems silly to have to add it to every object type that could possibly be used as a set member (especially since this means that if a third party library creates objects for you that don't implement __hash__ you'd have a hard time of adding it). In short, I agree something's broken, but the fix should not be to remove the default __hash__ and __eq__ altogether. Instead, the default __hash__ should be made smarter (and perhaps the only way to do this is to build the smarts into hash() again). I do agree that __cmp__, __gt__ etc. should be left undefined by default. All of this is Python 3000 only. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From nnorwitz at gmail.com Mon Nov 7 21:49:21 2005 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 7 Nov 2005 12:49:21 -0800 Subject: [Python-Dev] cross-compiling Message-ID: We've been having some issues and discussions at work about cross compiling. There are various people that have tried (are) cross compiling python. Right now the support kinda sucks due to a couple of reasons. First, distutils is required to build all the modules. This means that python must be built twice. Once for the target machine and once for the host machine. The host machine is really not desired since it's only purpose is to run distutils. I don't know the history of why distutils is used. I haven't had much of an issue with it since I've never needed to cross compile. What are the issues with not requiring python to be built on the host machine (ie, not using distutils)? Second, in configure we try to run little programs (AC_TRY_RUN) to determine what to set. I don't know of any good alternative but to force those to be defined manually for cross-compiled environments. Any suggestions here? I'm thinking we can skip the the AC_TRY_RUNs if host != target and we pickup the answers to those from a user supplied file. I'm *not* suggesting that normal builds see any change in behaviour. Nothing will change for most developers. ie, ./configure ; make ; ./python will continue to work the same. I only want to make it possible to cross compile python by building it only on the target platform. n PS. I would be interested to hear from others who are doing cross compiling and know more about it than me. From guido at python.org Mon Nov 7 22:04:53 2005 From: guido at python.org (Guido van Rossum) Date: Mon, 7 Nov 2005 13:04:53 -0800 Subject: [Python-Dev] cross-compiling In-Reply-To: References: Message-ID: I know some folks have successfully used cross-compilation before. But this was in a distant past. There was some support for it in the configure script; surely you're using that? I believe it lets you specify defaults for the TRY_RUN macros. But it's probably very primitive. About using distutils to build the extensions, this is because some extensions require quite a bit of logic to determine the build commands (e.g. look at BSDDB or Tkinter). There was a pre-distutils way of building extensions using Modules/Setup* but this required extensive manual editing if tools weren't in the place where they were expected, and they never were. I don't have time to look into this further right now, but I hope I will in the future. Keep me posted! --Guido On 11/7/05, Neal Norwitz wrote: > We've been having some issues and discussions at work about cross > compiling. There are various people that have tried (are) cross > compiling python. Right now the support kinda sucks due to a couple > of reasons. > > First, distutils is required to build all the modules. This means > that python must be built twice. Once for the target machine and once > for the host machine. The host machine is really not desired since > it's only purpose is to run distutils. I don't know the history of > why distutils is used. I haven't had much of an issue with it since > I've never needed to cross compile. What are the issues with not > requiring python to be built on the host machine (ie, not using > distutils)? > > Second, in configure we try to run little programs (AC_TRY_RUN) to > determine what to set. I don't know of any good alternative but to > force those to be defined manually for cross-compiled environments. > Any suggestions here? I'm thinking we can skip the the AC_TRY_RUNs > if host != target and we pickup the answers to those from a user > supplied file. > > I'm *not* suggesting that normal builds see any change in behaviour. > Nothing will change for most developers. ie, ./configure ; make ; > ./python will continue to work the same. I only want to make it > possible to cross compile python by building it only on the target > platform. > > n > > PS. I would be interested to hear from others who are doing cross > compiling and know more about it than me. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From maillist at roomity.com Mon Nov 7 22:11:11 2005 From: maillist at roomity.com (shenanigans) Date: Mon, 7 Nov 2005 13:11:11 -0800 (PST) Subject: [Python-Dev] [OTAnn] Feedback Message-ID: <26578370.591131397871429.JavaMail.tomcat5@slave1.roomity.com> I was interested in getting feedback from current mail group users. We have mirrored your mail list in a new application that provides a more aggregated and safe environment which utilizes the power of broadband. Roomity.com v 1.5 is a web 2.01 community webapp. Our newest version adds broadcast video and social networking such as favorite authors and an html editor. It?s free to join and any feedback would be appreciated. S. -------------------------------------------------------------------------------------------------------------------------------------------------- Broadband interface (RIA) + mail box saftey = Python_Core_Developers_List.roomity.com *Your* clubs, no sign up to read, ad supported; try broadband internet. ~~1131397871425~~ -------------------------------------------------------------------------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20051107/ad884a6f/attachment.html From jeremy at alum.mit.edu Mon Nov 7 22:38:18 2005 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Mon, 7 Nov 2005 16:38:18 -0500 Subject: [Python-Dev] cross-compiling In-Reply-To: References: Message-ID: On 11/7/05, Guido van Rossum wrote: > About using distutils to build the extensions, this is because some > extensions require quite a bit of logic to determine the build > commands (e.g. look at BSDDB or Tkinter). There was a pre-distutils > way of building extensions using Modules/Setup* but this required > extensive manual editing if tools weren't in the place where they were > expected, and they never were. I think part of the problem is that setup.py has a bunch of heuristics that are intended to do the right thing without user intervention. If, on the other hand, the user wants to intervene, because "the right thing" is wrong for cross-compiling, you are kind of stuck. I don't think there is an obvious way to select the extension modules to build and the C libraries for them to link against. Jeremy From bcannon at gmail.com Mon Nov 7 22:41:33 2005 From: bcannon at gmail.com (Brett Cannon) Date: Mon, 7 Nov 2005 13:41:33 -0800 Subject: [Python-Dev] cross-compiling In-Reply-To: References: Message-ID: On 11/7/05, Neal Norwitz wrote: > We've been having some issues and discussions at work about cross > compiling. There are various people that have tried (are) cross > compiling python. Right now the support kinda sucks due to a couple > of reasons. This might make a good sprint topic. Maybe your employer might be willing to get some people to come to hack on this? I know I wouldn't mind seeing the whole build process cleaned up. It works well enough, but I think some things could stand to be updated (speaking from experience of adding EXTRA_CFLAGS to the build process), such as setup.py being made more modular. -Brett From barry at python.org Mon Nov 7 23:05:38 2005 From: barry at python.org (Barry Warsaw) Date: Mon, 07 Nov 2005 17:05:38 -0500 Subject: [Python-Dev] cross-compiling In-Reply-To: References: Message-ID: <1131401138.4926.38.camel@geddy.wooz.org> On Mon, 2005-11-07 at 16:38, Jeremy Hylton wrote: > I think part of the problem is that setup.py has a bunch of heuristics > that are intended to do the right thing without user intervention. > If, on the other hand, the user wants to intervene, because "the right > thing" is wrong for cross-compiling, you are kind of stuck. I don't > think there is an obvious way to select the extension modules to build > and the C libraries for them to link against. This relates to an issue we've had to workaround with the distutils based module builds in Python. For some of the modules, we want the auto-detection code to find versions of dependent libraries in locations other than the "standard" system locations. I don't think there's a good way to convince the various setup.py scripts to look elsewhere for things, short of modifying the code. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20051107/69363bd3/attachment.pgp From martin at v.loewis.de Mon Nov 7 23:34:26 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 07 Nov 2005 23:34:26 +0100 Subject: [Python-Dev] Should the default equality operator compare values instead of identities? In-Reply-To: <740c3aec0511070506v6833686ag4894270034e01559@mail.gmail.com> References: <20051105151436.BFF7.JCARLSON@uci.edu> <20051105162001.C007.JCARLSON@uci.edu> <740c3aec0511070506v6833686ag4894270034e01559@mail.gmail.com> Message-ID: <436FD672.5070807@v.loewis.de> BJ?rn Lindqvist wrote: > How would the value equality operator deal with recursive objects? > > class Foo: > def __init__(self): > self.foo = self > > Seems to me that it would take atleast some special-casing to get > Foo() == Foo() to evalute to True in this case... This is sort-of supported today: >>> a=[] >>> a.append(a) >>> b=[] >>> b.append(b) >>> a == b True Regards, Martin From martin at v.loewis.de Mon Nov 7 23:38:34 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 07 Nov 2005 23:38:34 +0100 Subject: [Python-Dev] cross-compiling In-Reply-To: References: Message-ID: <436FD76A.3020401@v.loewis.de> Neal Norwitz wrote: > First, distutils is required to build all the modules. As Guido already suggests, this assertion is false. In a cross-compilation environment, I would try to avoid distutils, and indeed, the build process to do so is still supported. > Second, in configure we try to run little programs (AC_TRY_RUN) to > determine what to set. I don't know of any good alternative but to > force those to be defined manually for cross-compiled environments. > Any suggestions here? I'm thinking we can skip the the AC_TRY_RUNs > if host != target and we pickup the answers to those from a user > supplied file. You shouldn't be required to do that. Instead, just edit pyconfig.h manually, to match the target. autoconf is designed to support that. It would help if Makefile was target-independent (only host-dependent). Not sure whether this is the case. Regards, Martin From martin at v.loewis.de Mon Nov 7 23:39:18 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 07 Nov 2005 23:39:18 +0100 Subject: [Python-Dev] cross-compiling In-Reply-To: References: Message-ID: <436FD796.3010208@v.loewis.de> Jeremy Hylton wrote: > I think part of the problem is that setup.py has a bunch of heuristics > that are intended to do the right thing without user intervention. > If, on the other hand, the user wants to intervene, because "the right > thing" is wrong for cross-compiling, you are kind of stuck. I don't > think there is an obvious way to select the extension modules to build > and the C libraries for them to link against. Of course there is: Modules/Setup. Regards, Martin From mwh at python.net Tue Nov 8 00:02:12 2005 From: mwh at python.net (Michael Hudson) Date: Mon, 07 Nov 2005 23:02:12 +0000 Subject: [Python-Dev] Should the default equality operator compare values instead of identities? In-Reply-To: <436FD672.5070807@v.loewis.de> ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "Mon, 07 Nov 2005 23:34:26 +0100") References: <20051105151436.BFF7.JCARLSON@uci.edu> <20051105162001.C007.JCARLSON@uci.edu> <740c3aec0511070506v6833686ag4894270034e01559@mail.gmail.com> <436FD672.5070807@v.loewis.de> Message-ID: <2mbr0w12q3.fsf@starship.python.net> "Martin v. L?wis" writes: > BJ?rn Lindqvist wrote: >> How would the value equality operator deal with recursive objects? >> >> class Foo: >> def __init__(self): >> self.foo = self >> >> Seems to me that it would take atleast some special-casing to get >> Foo() == Foo() to evalute to True in this case... > > This is sort-of supported today: > > >>> a=[] > >>> a.append(a) > >>> b=[] > >>> b.append(b) > >>> a == b > True Uh, I think this changed in Python 2.4: >>> a = [] >>> a.append(a) >>> b = [] >>> b.append(b) >>> a == b Traceback (most recent call last): File "", line 1, in ? RuntimeError: maximum recursion depth exceeded in cmp Cheers, mwh -- First of all, email me your AOL password as a security measure. You may find that won't be able to connect to the 'net for a while. This is normal. The next thing to do is turn your computer upside down and shake it to reboot it. -- Darren Tucker, asr From kbk at shore.net Tue Nov 8 03:29:49 2005 From: kbk at shore.net (Kurt B. Kaiser) Date: Mon, 7 Nov 2005 21:29:49 -0500 (EST) Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200511080229.jA82TnHG017341@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 365 open ( +5) / 2961 closed ( +5) / 3326 total (+10) Bugs : 904 open (+11) / 5367 closed (+14) / 6271 total (+25) RFE : 200 open ( +1) / 189 closed ( +0) / 389 total ( +1) New / Reopened Patches ______________________ new function: os.path.relpath (2005-10-27) http://python.org/sf/1339796 opened by Richard Barran commands.getstatusoutput() (2005-11-02) http://python.org/sf/1346211 opened by Dawa Lama Better dead code elimination for the AST compiler (2005-11-02) http://python.org/sf/1346214 opened by Rune Holm A constant folding optimization pass for the AST (2005-11-02) http://python.org/sf/1346238 opened by Rune Holm Remove inconsistent behavior between import and zipimport (2005-11-03) http://python.org/sf/1346572 opened by Osvaldo Santana Neto Patch f. bug 495682 cannot handle http_proxy with user:pass@ (2005-11-05) CLOSED http://python.org/sf/1349117 opened by Johannes Nicolai Patch f. bug 495682 cannot handle http_proxy with user:pass@ (2005-11-05) http://python.org/sf/1349118 opened by Johannes Nicolai [PATCH] 100x optimization for ngettext (2005-11-06) http://python.org/sf/1349274 opened by Joe Wreschnig Fix for signal related abort in Visual Studio 2005 (2005-11-07) http://python.org/sf/1350409 opened by Adal Chiriliuc Redundant connect() call in logging.handlers.SysLogHandler (2005-11-07) http://python.org/sf/1350658 opened by Ken Lalonde Patches Closed ______________ tarfile.py: fix for bug #1336623 (2005-10-26) http://python.org/sf/1338314 closed by nnorwitz Python 2.4.2 doesn't build with "--without-threads" (2005-10-22) http://python.org/sf/1335054 closed by nnorwitz Speedup PyUnicode_DecodeCharmap (2005-10-05) http://python.org/sf/1313939 closed by lemburg Allow use of non-latin1 chars in interactive shell (2005-10-21) http://python.org/sf/1333679 closed by loewis Patch f. bug 495682 cannot handle http_proxy with user:pass@ (2005-11-05) http://python.org/sf/1349117 closed by birkenfeld New / Reopened Bugs ___________________ CVS webbrowser.py (1.40) bugs (2005-10-27) CLOSED http://python.org/sf/1339806 opened by Greg Couch TAB SETTINGS DONT WORK (win) (2005-10-27) http://python.org/sf/1339883 opened by reson5 time.strptime() with bad % code throws bad exception (2005-10-27) CLOSED http://python.org/sf/1340337 opened by Steve R. Hastings mmap does not accept length as 0 (2005-10-28) http://python.org/sf/1341031 opened by liturgist "\n" is incorrectly represented (2005-10-30) CLOSED http://python.org/sf/1341934 opened by Pavel Tkinter.Menu.delete doesn't delete command of entry (2005-10-30) http://python.org/sf/1342811 opened by Sverker Nilsson Broken docs for os.removedirs (2005-10-31) http://python.org/sf/1343671 opened by David K?gedal UNIX mmap leaks file descriptors (2005-11-01) CLOSED http://python.org/sf/1344508 opened by Erwin S. Andreasen colorsys tests, bug in frange (2005-11-01) CLOSED http://python.org/sf/1345263 opened by Rune Holm Python 2.4 and 2.3.5 won't build on OpenBSD 3.7 (2005-11-01) http://python.org/sf/1345313 opened by Dan doc typo (2005-11-02) CLOSED http://python.org/sf/1346026 opened by Keith Briggs Segfaults from unaligned loads in floatobject.c (2005-11-02) http://python.org/sf/1346144 opened by Rune Holm Missing import in documentation (2005-11-03) CLOSED http://python.org/sf/1346395 opened by Aggelos Orfanakos selectmodule.c calls PyInt_AsLong without error checking (2005-11-03) CLOSED http://python.org/sf/1346533 opened by Luke _subprocess.c calls PyInt_AsLong without error checking (2005-11-03) http://python.org/sf/1346547 opened by Luke httplib simply ignores CONTINUE (2005-11-03) http://python.org/sf/1346874 opened by Mike Looijmans FeedParser does not comply with RFC2822 (2005-11-04) http://python.org/sf/1347874 opened by Julian Phillips pydoc seems to run some scripts! (2005-11-04) http://python.org/sf/1348477 opened by Olly Betts email.Generators does not separates headers with "\r\n" (2005-11-05) http://python.org/sf/1349106 opened by Manlio Perillo xmlrpclib does not use http proxy (2005-11-06) http://python.org/sf/1349316 opened by greatred urllib.urlencode provides two features in one param (2005-11-06) http://python.org/sf/1349732 opened by Ori Avtalion urllib2 blocked from news.google.com (2005-11-07) CLOSED http://python.org/sf/1349977 opened by Michael Hoisie built-in method .__cmp__ (2005-11-07) http://python.org/sf/1350060 opened by Armin Rigo "setdlopenflags" leads to crash upon "import" (2005-11-07) http://python.org/sf/1350188 opened by John Pye CVS migration not in www.python.org docs (2005-11-07) http://python.org/sf/1350498 opened by Jim Jewett zlib.crc32 doesn't handle 0xffffffff seed (2005-11-07) http://python.org/sf/1350573 opened by Danny Yoo Bugs Closed ___________ CVS webbrowser.py (1.40) bugs (2005-10-27) http://python.org/sf/1339806 deleted by gregcouch Memory keeping (2005-10-26) http://python.org/sf/1338264 closed by tim_one tarfile can't extract some tar archives.. (2005-10-24) http://python.org/sf/1336623 closed by nnorwitz time.strptime() with bad % code throws bad exception (2005-10-27) http://python.org/sf/1340337 closed by bcannon _socket module not build under cygwin (2005-09-22) http://python.org/sf/1298709 closed by jlt63 "\n" is incorrectly represented (2005-10-30) http://python.org/sf/1341934 closed by perky pydoc HTTP reload failure (2001-04-21) http://python.org/sf/417833 closed by ping UNIX mmap leaks file descriptors (2005-10-31) http://python.org/sf/1344508 closed by nnorwitz colorsys tests, bug in frange (2005-11-01) http://python.org/sf/1345263 closed by nnorwitz Python.h should include system headers properly [POSIX] (2005-10-25) http://python.org/sf/1337400 closed by loewis doc typo (2005-11-02) http://python.org/sf/1346026 closed by nnorwitz Missing import in documentation (2005-11-02) http://python.org/sf/1346395 closed by bcannon selectmodule.c calls PyInt_AsLong without error checking (2005-11-02) http://python.org/sf/1346533 closed by nnorwitz pydoc ignores $PAGER if TERM='dumb' (2002-12-09) http://python.org/sf/651124 closed by ping urllib2 blocked from news.google.com (2005-11-06) http://python.org/sf/1349977 closed by bcannon New / Reopened RFE __________________ please support the free visual studio sdk compiler (2005-11-04) http://python.org/sf/1348719 opened by David McNab From ronaldoussoren at mac.com Tue Nov 8 08:37:16 2005 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 8 Nov 2005 08:37:16 +0100 Subject: [Python-Dev] Should the default equality operator compare values instead of identities? In-Reply-To: <436FD672.5070807@v.loewis.de> References: <20051105151436.BFF7.JCARLSON@uci.edu> <20051105162001.C007.JCARLSON@uci.edu> <740c3aec0511070506v6833686ag4894270034e01559@mail.gmail.com> <436FD672.5070807@v.loewis.de> Message-ID: <994C0B10-7382-4492-9EDE-1AE47BF7FA32@mac.com> On 7-nov-2005, at 23:34, Martin v. L?wis wrote: > BJ?rn Lindqvist wrote: >> How would the value equality operator deal with recursive objects? >> >> class Foo: >> def __init__(self): >> self.foo = self >> >> Seems to me that it would take atleast some special-casing to get >> Foo() == Foo() to evalute to True in this case... > > This is sort-of supported today: But only for lists ;-) >>> a = {} >>> a[1] = a >>> >>> b = {} >>> b[1] = b >>> >>> a == b Traceback (most recent call last): File "", line 1, in ? RuntimeError: maximum recursion depth exceeded in cmp >>> > >>>> a=[] >>>> a.append(a) >>>> b=[] >>>> b.append(b) >>>> a == b > True > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ > ronaldoussoren%40mac.com From winlinchu at yahoo.it Tue Nov 8 10:16:06 2005 From: winlinchu at yahoo.it (winlinchu) Date: Tue, 8 Nov 2005 10:16:06 +0100 (CET) Subject: [Python-Dev] Unifying decimal numbers. Message-ID: <20051108091607.13272.qmail@web26010.mail.ukl.yahoo.com> Now, Python unified ints and long ints. For Python 3k, could be introduced a "Decimal" type (yes, Decimal, the library module!) in place of the actual float object. Of course, the Decimal type would be rewritten in C. Thanks. ___________________________________ Yahoo! Messenger: chiamate gratuite in tutto il mondo http://it.messenger.yahoo.com From support at intercable.ru Tue Nov 8 08:59:42 2005 From: support at intercable.ru (Technical Support of Intercable Co) Date: Tue, 08 Nov 2005 10:59:42 +0300 Subject: [Python-Dev] For Python 3k, drop default/implicit hash, and comparison Message-ID: <43705AEE.2050600@intercable.ru> Why 'identity' objects can't define: def __getKey__(self): return Key(self, id(self)) Then they would act as usually, while value object can define def __getKey__(self): return Key(self, self.i, self.j, self.a[1]) (Key is an abstraction to handle subclassing) Of course, there should be a way to handle comparison off the class ierarhy (i think) Today one can write: >>> class Boo(object): def __init__(self,s=""): self.s=s def __hash__(self): return hash(self.s) def __cmp__(self,other): if type(self)==type(other): return cmp(self.s,other.s) if type(other)==str: return cmp(self.s,other) >>> a={} >>> a['s']=1 >>> a[Boo('s')] 1 >>> a[Boo('z')]=2 >>> a['z'] 2 It is confused and hardly usefull, but possible. Excuse my english. From jcarlson at uci.edu Tue Nov 8 19:26:16 2005 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 08 Nov 2005 10:26:16 -0800 Subject: [Python-Dev] Unifying decimal numbers. In-Reply-To: <20051108091607.13272.qmail@web26010.mail.ukl.yahoo.com> References: <20051108091607.13272.qmail@web26010.mail.ukl.yahoo.com> Message-ID: <20051108101410.C050.JCARLSON@uci.edu> winlinchu wrote: > Now, Python unified ints and long ints. > For Python 3k, could be introduced a "Decimal" type > (yes, Decimal, the library module!) in place of the > actual float object. Of course, the Decimal type would > be rewritten in C. There is code which relies on standard IEEE 754 floating point math (speed, behavior, rounding, etc.) that would break with the replacement of floats with decimals. Further, even if it were to be converted to C, it would likely be hundreds of times slower than the processor-native float operations. This discussion has been had before use: site:mail.python.org decimal replace float python in google to discover such discussions. For example: http://mail.python.org/pipermail/python-dev/2005-June/054416.html - Josiah From mdehoon at c2b2.columbia.edu Tue Nov 8 20:46:48 2005 From: mdehoon at c2b2.columbia.edu (Michiel Jan Laurens de Hoon) Date: Tue, 08 Nov 2005 14:46:48 -0500 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter Message-ID: <437100A7.5050907@c2b2.columbia.edu> Dear Pythoneers, I use Python heavily for scientific computing, originally for computational physics and nowadays for computational biology. One of the extension modules I support is pygist, a module for scientific visualization. For this (and similar) packages to work, it is important to have an event loop in Python. Currently, event loops are available in Python via PyOS_InputHook, a pointer to a user-defined function that is called when Python is idle (waiting for user input). However, an event loop using PyOS_InputHook has some inherent limitations, so I am thinking about how to improve event loop support in Python. As an example, consider the current implementation of Tkinter. What's nice about it is that events as well as user-typed Python commands are handled without having to call mainloop() explicitly (except on some platforms): "import Tkinter; Tkinter.Tk()" causes a Tk window to pop up that remains responsive throughout. It works as follows (using Tkinter as an example; pygist works essentially the same): 1) Importing Tkinter causes PyOS_InputHook to be set to the EventHook function in _tkinter.c. 2) Before Python calls fgets to read the next Python command typed by the user, it checks PyOS_InputHook and calls it if it is not NULL. 3) The EventHook function in _tkinter runs the following loop: - Check if user input is present; if so, exit the loop - Handle a Tcl/Tk event, if present - Sleep for 20 milliseconds 4) Once the EventHook function returns, Python continues to read the next user command. After executing the command, return to 2). However, this implementation has the following problems: 1) Essentially, the event loop is a busy-wait loop with a 20 ms sleep in between. An event loop using select() (or equivalent on Windows) will give better performance. 2) Since this event loop runs inside Tkinter, there is no way for other extension modules to get their messages handled. Hence, we cannot have more than one extension module that needs an event loop. As an example, it would be nice to have a Tkinter GUI to steer a simulation and a (non-Tk) graphics output window to visualize the simulation. 3) Whereas PyOS_InputHook is called when Python is waiting for user input, it is not called when Python is waiting for anything else, for example one thread waiting for another. For example, IDLE uses two threads, one handling the GUI and one handling the user commands. When the second thread is waiting for the first thread (when waiting for user input to become available), PyOS_InputHook is not being called, and no Tkinter events are being handled. Hence, "import Tkinter; Tkinter.Tk()" does nothing when executed from an IDLE window. Which means that our scientific visualization software can only be run from Python started from the command line, whereas many users (especially on Windows) will want to use IDLE. Now the problem I'm facing is that because of its integration with Tcl, this cannot be fixed easily with Tkinter as the GUI toolkit for Python. If the events to be handled were purely graphical events (move a window, repaint a window, etc.), there would be no harm in handling these events when waiting for e.g. another thread. With Tkinter, however, we cannot enter EventHook while waiting for another thread: a) because EventHook only returns if user input is available (it doesn't wait for threads); b) because EventHook also runs Tcl/Tk commands, and we wouldn't want to run some Tcl commands in some thread while waiting for another thread. Therefore, as far as I can tell, there is no way to set up a true event loop in Python that works nicely with Tkinter, and there is no way to have an event loop function from IDLE. So I'd like to ask the following questions: 1) Did I miss something? Is there some way to get an event loop with Tkinter? 2) Will Tkinter always be the standard toolkit for Python, or are there plans to replace it at some point? I realize that Tkinter has been an important part of Python for some time now, and I don't expect it to be ditched just because of my event loop problems. At the same time, event loop support could use some improvement, so I'd like to call your attention to this issue. Tcl actually has event loops implemented quite nicely, and may serve as an example of how event loops may work in Python. --Michiel. -- Michiel de Hoon Center for Computational Biology and Bioinformatics Columbia University 1150 St Nicholas Avenue New York, NY 10032 From janssen at parc.com Tue Nov 8 21:32:59 2005 From: janssen at parc.com (Bill Janssen) Date: Tue, 8 Nov 2005 12:32:59 PST Subject: [Python-Dev] Unifying decimal numbers. In-Reply-To: Your message of "Tue, 08 Nov 2005 10:26:16 PST." <20051108101410.C050.JCARLSON@uci.edu> Message-ID: <05Nov8.123259pst."58633"@synergy1.parc.xerox.com> Might be more interesting to think about replacing ints and Decimal with implicit-denominator rational type. In the HTTP-NG typing proposal, we called this a "fixed-point" type. See Section 4.5.1 of http://www.w3.org/Protocols/HTTP-NG/1998/08/draft-frystyk-httpng-arch-00.txt for details. The current notion of "int" would be defined as a specific kind of fixed-point type (a denominator of 1), but other fixed-point types such as dollars (denominator of 100) or dozens (denominator of 1/12) could also be defined. The nice thing about type systems like this is that they can accurately describe non-binary values, like 1/3. Bill From martin at v.loewis.de Tue Nov 8 21:37:41 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 08 Nov 2005 21:37:41 +0100 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <437100A7.5050907@c2b2.columbia.edu> References: <437100A7.5050907@c2b2.columbia.edu> Message-ID: <43710C95.30209@v.loewis.de> Michiel Jan Laurens de Hoon wrote: > 1) Did I miss something? Is there some way to get an event loop with > Tkinter? Yes, and yes. You are missing multi-threading, which is the widely used approach to doing things simultaneously in a single process. In one thread, user interaction can occur; in another, computation. If you need non-blocking interaction between the threads, use queues, or other global variables. If you have other event sources, deal with them in separate threads. Yes, it is possible to get event loops with Tkinter. Atleast on Unix, you can install a file handler into the Tk event loop (through createfilehandler), which gives you callbacks whenever there is some activity on the files. Furthermore, it is possible to turn the event loop around, by doing dooneevent explicitly. In principle, it would also be possible to expose Tcl events and notifications in Tkinter (i.e. the Tcl_CreateEventSource/Tcl_WaitForEvent family of APIs). If you think this would help in your case, then contributions are welcome. > 2) Will Tkinter always be the standard toolkit for Python, or are there > plans to replace it at some point? Python does not evolve along a grand master plan. Instead, individual contributors propose specific modifications, e.g. through PEPs. I personally have no plan to replace Tkinter. Regards, Martin From goodger at python.org Wed Nov 9 00:54:41 2005 From: goodger at python.org (David Goodger) Date: Tue, 08 Nov 2005 18:54:41 -0500 Subject: [Python-Dev] PEP submission broken? In-Reply-To: <43695FE5.1080803__36597.7150541314$1131334275$gmane$org@nowhere.org> References: <43695FE5.1080803__36597.7150541314$1131334275$gmane$org@nowhere.org> Message-ID: <43713AC1.5060803@python.org> [Bryan Olson] > Though I tried to submit a (pre-) PEP in the proper form through the > proper channels, it has disappeared into the ether. Indeed, it has; I can't find it in my mailbox. Could you re-send the latest text? I'll review it right away. > From what I can tell, We need to address fixing the > PEP process before there is any point in working on PEP's, Email is imperfect; just send it again. And "fakeaddress at nowhere.org" doesn't help ;-) -- David Goodger -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20051108/c039f37b/signature.pgp From goodger at python.org Wed Nov 9 00:55:35 2005 From: goodger at python.org (David Goodger) Date: Tue, 08 Nov 2005 18:55:35 -0500 Subject: [Python-Dev] PEP submission broken? In-Reply-To: <436F1F49.90606@gmail.com> References: <43695FE5.1080803@nowhere.org> <436F1F49.90606@gmail.com> Message-ID: <43713AF7.1080600@python.org> [Nick Coghlan] > Would it be worth having a PEP category on the RFE tracker, rather than > submitting pre-PEP's directly to the PEP editors? Couldn't hurt. -- David Goodger -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20051108/c555115d/signature.pgp From osantana at gmail.com Wed Nov 9 03:33:47 2005 From: osantana at gmail.com (Osvaldo Santana Neto) Date: Tue, 8 Nov 2005 23:33:47 -0300 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks Message-ID: <20051109023347.GA15823@localhost.localdomain> Hi, I'm working on Python[1] port for Maemo Platform[2] and I've found a inconsistent behavior in zipimport and import hook with '.pyc' and '.pyo' files. The shell section below show this problem using a 'module_c.pyc', 'module_o.pyo' and 'modules.zip' (with module_c and module_o inside): $ ls module_c.pyc module_o.pyo modules.zip $ python >>> import module_c >>> import module_o ImportError: No module named module_o $ python -O >>> import module_c ImportError: No module named module_c >>> import module_o $ rm *.pyc *.pyo $ PYTHONPATH=modules.zip python >>> import module_c module_c >>> import module_o module_o $ PYTHONPATH=modules.zip python -O >>> import module_c module_c >>> import module_o module_o I've create a patch suggestion to remove this inconsistency[3] (*I* think zipimport behaviour is better). [1] http://pymaemo.sf.net/ [2] http://www.maemo.org/ [3] http://python.org/sf/1346572 -- Osvaldo Santana Neto (aCiDBaSe) icq, url = (11287184, "http://www.pythonbrasil.com.br") From guido at python.org Wed Nov 9 04:14:51 2005 From: guido at python.org (Guido van Rossum) Date: Tue, 8 Nov 2005 19:14:51 -0800 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: <20051109023347.GA15823@localhost.localdomain> References: <20051109023347.GA15823@localhost.localdomain> Message-ID: You didn't show us what's in the zip file. Can you show a zipinfo output? My intention with import was always that without -O, *.pyo files are entirely ignored; and with -O, *.pyc files are entirely ignored. It sounds like you're saying that you want to change this so that .pyc and .pyo are always honored (with .pyc preferred if -O is not present and .pyo preferred if -O is present). I'm not sure that I like that better. If that's how zipimport works, I think it's broken! --Guido On 11/8/05, Osvaldo Santana Neto wrote: > Hi, > > I'm working on Python[1] port for Maemo Platform[2] and I've found a > inconsistent behavior in zipimport and import hook with '.pyc' and > '.pyo' files. The shell section below show this problem using a > 'module_c.pyc', 'module_o.pyo' and 'modules.zip' (with module_c and > module_o inside): > > $ ls > module_c.pyc module_o.pyo modules.zip > > $ python > >>> import module_c > >>> import module_o > ImportError: No module named module_o > > $ python -O > >>> import module_c > ImportError: No module named module_c > >>> import module_o > > $ rm *.pyc *.pyo > $ PYTHONPATH=modules.zip python > >>> import module_c > module_c > >>> import module_o > module_o > > $ PYTHONPATH=modules.zip python -O > >>> import module_c > module_c > >>> import module_o > module_o > > I've create a patch suggestion to remove this inconsistency[3] (*I* think > zipimport behaviour is better). > > [1] http://pymaemo.sf.net/ > [2] http://www.maemo.org/ > [3] http://python.org/sf/1346572 > > -- > Osvaldo Santana Neto (aCiDBaSe) > icq, url = (11287184, "http://www.pythonbrasil.com.br") > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gjc at inescporto.pt Wed Nov 9 12:40:25 2005 From: gjc at inescporto.pt (Gustavo J. A. M. Carneiro) Date: Wed, 09 Nov 2005 11:40:25 +0000 Subject: [Python-Dev] Weak references: dereference notification Message-ID: <1131536425.9130.10.camel@localhost> Hello, I have come across a situation where I find the current weak references interface for extension types insufficient. Currently you only have a tp_weaklistoffset slot, pointing to a PyObject with weak references. However, in my case[1] I _really_ need to be notified when a weak reference is dereferenced. What happens now is that, when you call a weakref object, a simple Py_INCREF is done on the referenced object. It would be easy to implement a new slot to contain a function that should be called when a weak reference is dereferenced. Or, alternatively, a slot or class attribute that indicates an alternative type that should be used to create weak references: instead of the builtin weakref object, a subtype of it, so you can override tp_call. Does this sounds acceptable? Regards. [1] http://bugzilla.gnome.org/show_bug.cgi?id=320428 -- Gustavo J. A. M. Carneiro The universe is always one step beyond logic. From osantana at gmail.com Wed Nov 9 15:33:02 2005 From: osantana at gmail.com (Osvaldo Santana) Date: Wed, 9 Nov 2005 12:33:02 -0200 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: On 11/9/05, Guido van Rossum wrote: > You didn't show us what's in the zip file. Can you show a zipinfo output? $ zipinfo modules.zip Archive: modules.zip 426 bytes 2 files -rw-r--r-- 2.3 unx 109 bx defN 31-Oct-05 14:49 module_o.pyo -rw-r--r-- 2.3 unx 109 bx defN 31-Oct-05 14:48 module_c.pyc 2 files, 218 bytes uncompressed, 136 bytes compressed: 37.6% > My intention with import was always that without -O, *.pyo files are > entirely ignored; and with -O, *.pyc files are entirely ignored. > > It sounds like you're saying that you want to change this so that .pyc > and .pyo are always honored (with .pyc preferred if -O is not present > and .pyo preferred if -O is present). I'm not sure that I like that > better. If that's how zipimport works, I think it's broken! Yes, this is how zipimport works and I think this is good in cases where a third-party binary module/package is available only with .pyo files and others only with .pyc files (without .py source files, of course). I know we can rename the files, but this is a good solution? Well, I don't have a strong opinion about the solution adopted and I really like to see other alternatives and opinions. Thanks, Osvaldo -- Osvaldo Santana Neto (aCiDBaSe) icq, url = (11287184, "http://www.pythonbrasil.com.br") From guido at python.org Wed Nov 9 16:39:29 2005 From: guido at python.org (Guido van Rossum) Date: Wed, 9 Nov 2005 07:39:29 -0800 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: Maybe it makes more sense to deprecate .pyo altogether and instead have a post-load optimizer optimize .pyc files according to the current optimization settings? Unless others are interested in this nothing will happen. I've never heard of a third party making their code available only as .pyo, so the use case for changing things isn't very strong. In fact the only use cases I know for not making .py available are in situations where a proprietary "canned" application is distributed to end users who have no intention or need to ever add to the code. --Guido On 11/9/05, Osvaldo Santana wrote: > On 11/9/05, Guido van Rossum wrote: > > You didn't show us what's in the zip file. Can you show a zipinfo output? > > $ zipinfo modules.zip > Archive: modules.zip 426 bytes 2 files > -rw-r--r-- 2.3 unx 109 bx defN 31-Oct-05 14:49 module_o.pyo > -rw-r--r-- 2.3 unx 109 bx defN 31-Oct-05 14:48 module_c.pyc > 2 files, 218 bytes uncompressed, 136 bytes compressed: 37.6% > > > My intention with import was always that without -O, *.pyo files are > > entirely ignored; and with -O, *.pyc files are entirely ignored. > > > > It sounds like you're saying that you want to change this so that .pyc > > and .pyo are always honored (with .pyc preferred if -O is not present > > and .pyo preferred if -O is present). I'm not sure that I like that > > better. If that's how zipimport works, I think it's broken! > > Yes, this is how zipimport works and I think this is good in cases > where a third-party binary module/package is available only with .pyo > files and others only with .pyc files (without .py source files, of > course). > > I know we can rename the files, but this is a good solution? Well, I > don't have a strong opinion about the solution adopted and I really > like to see other alternatives and opinions. > > Thanks, > Osvaldo > > -- > Osvaldo Santana Neto (aCiDBaSe) > icq, url = (11287184, "http://www.pythonbrasil.com.br") > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jim at zope.com Wed Nov 9 17:50:44 2005 From: jim at zope.com (Jim Fulton) Date: Wed, 09 Nov 2005 11:50:44 -0500 Subject: [Python-Dev] Weak references: dereference notification In-Reply-To: <1131536425.9130.10.camel@localhost> References: <1131536425.9130.10.camel@localhost> Message-ID: <437228E4.4070800@zope.com> Gustavo J. A. M. Carneiro wrote: > Hello, > > I have come across a situation where I find the current weak > references interface for extension types insufficient. > > Currently you only have a tp_weaklistoffset slot, pointing to a > PyObject with weak references. However, in my case[1] I _really_ need > to be notified when a weak reference is dereferenced. What happens now > is that, when you call a weakref object, a simple Py_INCREF is done on > the referenced object. It would be easy to implement a new slot to > contain a function that should be called when a weak reference is > dereferenced. Or, alternatively, a slot or class attribute that > indicates an alternative type that should be used to create weak > references: instead of the builtin weakref object, a subtype of it, so > you can override tp_call. > > Does this sounds acceptable? Since you can now (as of 2.4) subclass the weakref.ref class, you should be able to do this yourself in Python. See for example, weakref.KeyedRef. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From gjc at inescporto.pt Wed Nov 9 18:14:59 2005 From: gjc at inescporto.pt (Gustavo J. A. M. Carneiro) Date: Wed, 09 Nov 2005 17:14:59 +0000 Subject: [Python-Dev] Weak references: dereference notification In-Reply-To: <437228E4.4070800@zope.com> References: <1131536425.9130.10.camel@localhost> <437228E4.4070800@zope.com> Message-ID: <1131556500.9130.18.camel@localhost> Qua, 2005-11-09 ?s 11:50 -0500, Jim Fulton escreveu: > Gustavo J. A. M. Carneiro wrote: > > Hello, > > > > I have come across a situation where I find the current weak > > references interface for extension types insufficient. > > > > Currently you only have a tp_weaklistoffset slot, pointing to a > > PyObject with weak references. However, in my case[1] I _really_ need > > to be notified when a weak reference is dereferenced. What happens now > > is that, when you call a weakref object, a simple Py_INCREF is done on > > the referenced object. It would be easy to implement a new slot to > > contain a function that should be called when a weak reference is > > dereferenced. Or, alternatively, a slot or class attribute that > > indicates an alternative type that should be used to create weak > > references: instead of the builtin weakref object, a subtype of it, so > > you can override tp_call. > > > > Does this sounds acceptable? > > Since you can now (as of 2.4) subclass the weakref.ref class, you should be able to > do this yourself in Python. See for example, weakref.KeyedRef. I know I can subclass it, but it doesn't change anything. If people keep writing code like weakref.ref(myobj) instead of myweakref(myobj), it still won't work. I wouldn't want to have to teach users of the library that they need to use an alternative type; that seldom doesn't work. Now, if there was a place in the type that contained information like "for creating weak references of instances of this type, use this weakref class" and weakref.ref was smart enough to lookup this type and use it, only _then_ it could work. Thanks, -- Gustavo J. A. M. Carneiro The universe is always one step beyond logic. From guido at python.org Wed Nov 9 18:23:34 2005 From: guido at python.org (Guido van Rossum) Date: Wed, 9 Nov 2005 09:23:34 -0800 Subject: [Python-Dev] Weak references: dereference notification In-Reply-To: <1131556500.9130.18.camel@localhost> References: <1131536425.9130.10.camel@localhost> <437228E4.4070800@zope.com> <1131556500.9130.18.camel@localhost> Message-ID: > > Gustavo J. A. M. Carneiro wrote: > > > I have come across a situation where I find the current weak > > > references interface for extension types insufficient. > > > > > > Currently you only have a tp_weaklistoffset slot, pointing to a > > > PyObject with weak references. However, in my case[1] I _really_ need > > > to be notified when a weak reference is dereferenced. I find reading through the bug discussion a bit difficult to understand your use case. Could you explain it here? If you can't explain it you certainly won't get your problem solved! :-) > > > What happens now > > > is that, when you call a weakref object, a simple Py_INCREF is done on > > > the referenced object. It would be easy to implement a new slot to > > > contain a function that should be called when a weak reference is > > > dereferenced. Or, alternatively, a slot or class attribute that > > > indicates an alternative type that should be used to create weak > > > references: instead of the builtin weakref object, a subtype of it, so > > > you can override tp_call. > > > > > > Does this sounds acceptable? [Jim Fulton] > > Since you can now (as of 2.4) subclass the weakref.ref class, you should be able to > > do this yourself in Python. See for example, weakref.KeyedRef. > > I know I can subclass it, but it doesn't change anything. If people > keep writing code like weakref.ref(myobj) instead of myweakref(myobj), > it still won't work. > > I wouldn't want to have to teach users of the library that they need > to use an alternative type; that seldom doesn't work. > > Now, if there was a place in the type that contained information like > > "for creating weak references of instances of this type, use this > weakref class" > > and weakref.ref was smart enough to lookup this type and use it, only > _then_ it could work. Looks what you're looking for is a customizable factory fuction. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From nico at tekNico.net Wed Nov 9 17:24:01 2005 From: nico at tekNico.net (Nicola Larosa) Date: Wed, 09 Nov 2005 17:24:01 +0100 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: > Maybe it makes more sense to deprecate .pyo altogether and instead > have a post-load optimizer optimize .pyc files according to the > current optimization settings? That would not be enough, because it would leave the docstrings in the .pyc files. > Unless others are interested in this nothing will happen. The status quo is good enough, for "normal" imports. If zipimport works differently, well, that's not nice. > I've never heard of a third party making their code available only as > .pyo, *cough* Ahem, here we are (the firm I work for). > so the use case for changing things isn't very strong. In fact > the only use cases I know for not making .py available are in > situations where a proprietary "canned" application is distributed to > end users who have no intention or need to ever add to the code. Well, exactly. :-) -- Nicola Larosa - nico at tekNico.net No inventions have really significantly eased the cognitive difficulty of writing scalable concurrent applications and it is unlikely that any will in the near term. [...] Most of all, threads do not help, in fact, they make the problem worse in many cases. -- G. Lefkowitz, August 2005 From gjc at inescporto.pt Wed Nov 9 18:52:19 2005 From: gjc at inescporto.pt (Gustavo J. A. M. Carneiro) Date: Wed, 09 Nov 2005 17:52:19 +0000 Subject: [Python-Dev] Weak references: dereference notification In-Reply-To: References: <1131536425.9130.10.camel@localhost> <437228E4.4070800@zope.com> <1131556500.9130.18.camel@localhost> Message-ID: <1131558739.9130.40.camel@localhost> Qua, 2005-11-09 ?s 09:23 -0800, Guido van Rossum escreveu: > > > Gustavo J. A. M. Carneiro wrote: > > > > I have come across a situation where I find the current weak > > > > references interface for extension types insufficient. > > > > > > > > Currently you only have a tp_weaklistoffset slot, pointing to a > > > > PyObject with weak references. However, in my case[1] I _really_ need > > > > to be notified when a weak reference is dereferenced. > > I find reading through the bug discussion a bit difficult to > understand your use case. Could you explain it here? If you can't > explain it you certainly won't get your problem solved! :-) This is a typical PyObject wrapping C object (GObject) problem. Both PyObject and GObject have independent reference counts. For each GObject there is at most one PyObject wrapper. When the refcount on the wrapper drops to zero, tp_dealloc is called. In tp_dealloc, and if the GObject refcount is > 1, I do something slightly evil: I 'resurect' the PyObject (calling PyObject_Init), create a weak reference to the GObject, and drop the "strong" reference. I call this a 'hibernation state'. Now the problem. Suppose the user had a weak ref to the PyObject: 1- At certain point in time, when the wrapper is in hibernation state, the user calls the weak ref 2- It gets a PyObject that contains a weak reference to the GObject; 3- Now suppose whatever was holding the GObject ref drops its reference, which was the last one, and the GObject dies; 4- Now the user does something with the PyObject obtained through the weakref -> invalid memory access. The cause for the problem is that between steps 2 and 3 the wrapper needs to change the weak reference to the GObject to a strong one. Unfortunately, I don't get any notification that 2 happened. BTW, I fixed this problem in the mean time with a bit more of slightly evil code. I override tp_call of the standard weakref type :-P [...] > > and weakref.ref was smart enough to lookup this type and use it, only > > _then_ it could work. > > Looks what you're looking for is a customizable factory fuction. Sure, if weakref.ref could be such a factory, and could take "advice" on what type of weakref to use for each class. Regards. -- Gustavo J. A. M. Carneiro The universe is always one step beyond logic. From osantana at gmail.com Wed Nov 9 19:15:04 2005 From: osantana at gmail.com (Osvaldo Santana) Date: Wed, 9 Nov 2005 16:15:04 -0200 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: On 11/9/05, Guido van Rossum wrote: > Maybe it makes more sense to deprecate .pyo altogether and instead > have a post-load optimizer optimize .pyc files according to the > current optimization settings? I agree with this idea, but we've to think about docstrings (like Nicola said in his e-mail). Maybe we want to create a different and optimization-independent option to remove docstrings from modules? > Unless others are interested in this nothing will happen. > > I've never heard of a third party making their code available only as > .pyo, so the use case for changing things isn't very strong. In fact > the only use cases I know for not making .py available are in > situations where a proprietary "canned" application is distributed to > end users who have no intention or need to ever add to the code. I've other use case: I'm porting Python to Maemo Platform and I want to reduce the size of modules. The .pyo (-OO) are smaller than .pyc files (mainly because docstring removing) and we start to use this optimization flag to compile our Python distribution. In this case we want to force developers to call Python Interpreter with -O flags, set PYTHONOPTIMIZE, or apply my patch :) to make this more transparent. I've noticed this inconsistency when we stop to use zipimport in our Python For Maemo distribution. We've decided to stop using zipimport because the device (Nokia 770) uses a compressed filesystem. Some friends (mainly Gustavo Barbieri) help me to create the suggested patch after some discussion in our PythonBrasil mailing list. Thanks, Osvaldo -- Osvaldo Santana Neto (aCiDBaSe) icq, url = (11287184, "http://www.pythonbrasil.com.br") From guido at python.org Wed Nov 9 20:32:54 2005 From: guido at python.org (Guido van Rossum) Date: Wed, 9 Nov 2005 11:32:54 -0800 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: On 11/9/05, Osvaldo Santana wrote: > I've noticed this inconsistency when we stop to use zipimport in our > Python For Maemo distribution. We've decided to stop using zipimport > because the device (Nokia 770) uses a compressed filesystem. I won't comment further on the brainstorm that's going on (this is becoming a topic for c.l.py) but I think you are misunderstanding the point of zipimport. It's not done (usually) for the compression but for the index. Finding a name in the zipfile index is much more efficient than doing a directory search; and the zip index can be cached. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ronaldoussoren at mac.com Wed Nov 9 20:40:02 2005 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 9 Nov 2005 20:40:02 +0100 Subject: [Python-Dev] Weak references: dereference notification In-Reply-To: <1131558739.9130.40.camel@localhost> References: <1131536425.9130.10.camel@localhost> <437228E4.4070800@zope.com> <1131556500.9130.18.camel@localhost> <1131558739.9130.40.camel@localhost> Message-ID: <9E82C8B1-8A32-457D-827A-F0135EB9F8D3@mac.com> On 9-nov-2005, at 18:52, Gustavo J. A. M. Carneiro wrote: > Qua, 2005-11-09 ?s 09:23 -0800, Guido van Rossum escreveu: >>>> Gustavo J. A. M. Carneiro wrote: >>>>> I have come across a situation where I find the current weak >>>>> references interface for extension types insufficient. >>>>> >>>>> Currently you only have a tp_weaklistoffset slot, pointing to a >>>>> PyObject with weak references. However, in my case[1] I >>>>> _really_ need >>>>> to be notified when a weak reference is dereferenced. >> >> I find reading through the bug discussion a bit difficult to >> understand your use case. Could you explain it here? If you can't >> explain it you certainly won't get your problem solved! :-) > > This is a typical PyObject wrapping C object (GObject) problem. > Both > PyObject and GObject have independent reference counts. For each > GObject there is at most one PyObject wrapper. > > When the refcount on the wrapper drops to zero, tp_dealloc is > called. > In tp_dealloc, and if the GObject refcount is > 1, I do something > slightly evil: I 'resurect' the PyObject (calling PyObject_Init), > create > a weak reference to the GObject, and drop the "strong" reference. I > call this a 'hibernation state'. Why do you do that? The only reasons I can think of are that you hope to gain some speed from this or that you want to support weak references to the GObject. For what its worth, in PyObjC we don't support weak references to the underlying Objective-C object and delete the proxy object when it is garbage collected. Objective-C also has reference counts, we increase that in the constructor for the proxy object and decrease it again in the destroctor. Ronald From pje at telecommunity.com Wed Nov 9 20:48:25 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Nov 2005 14:48:25 -0500 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: <5.1.1.6.0.20051109144523.01f4a6a8@mail.telecommunity.com> At 11:32 AM 11/9/2005 -0800, Guido van Rossum wrote: >On 11/9/05, Osvaldo Santana wrote: > > I've noticed this inconsistency when we stop to use zipimport in our > > Python For Maemo distribution. We've decided to stop using zipimport > > because the device (Nokia 770) uses a compressed filesystem. > >I won't comment further on the brainstorm that's going on (this is >becoming a topic for c.l.py) but I think you are misunderstanding the >point of zipimport. It's not done (usually) for the compression but >for the index. Finding a name in the zipfile index is much more >efficient than doing a directory search; and the zip index can be >cached. zipimport also helps distribution convenience - a large and elaborate package can be distributed in a single zipfile (such as is built by setuptools' "bdist_egg" command) and simply placed on PYTHONPATH or directly on sys.path. And tools like py2exe can also append all an application's modules to an executable file in zipped form. From barbieri at gmail.com Wed Nov 9 22:12:38 2005 From: barbieri at gmail.com (Gustavo Sverzut Barbieri) Date: Wed, 9 Nov 2005 19:12:38 -0200 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: <9ef20ef30511091312lcaa1caetbe0c4bade802738a@mail.gmail.com> On 11/9/05, Guido van Rossum wrote: > On 11/9/05, Osvaldo Santana wrote: > > I've noticed this inconsistency when we stop to use zipimport in our > > Python For Maemo distribution. We've decided to stop using zipimport > > because the device (Nokia 770) uses a compressed filesystem. > > I won't comment further on the brainstorm that's going on (this is > becoming a topic for c.l.py) but I think you are misunderstanding the > point of zipimport. It's not done (usually) for the compression but > for the index. Finding a name in the zipfile index is much more > efficient than doing a directory search; and the zip index can be > cached. Any way, not loading .pyo if no .pyc or .py is available is a drawback, specially on unices that have scripts starting with "#!/usr/bin/python" or "#!/usr/bin/env python" and the system just have .pyo files, due a bunch of reasons, in this case the small disc space. -- Gustavo Sverzut Barbieri -------------------------------------- Computer Engineer 2001 - UNICAMP Mobile: +55 (19) 9165 8010 Phone: +1 (347) 624 6296 @ sip.stanaphone.com Jabber: gsbarbieri at jabber.org ICQ#: 17249123 MSN: barbieri at gmail.com GPG: 0xB640E1A2 @ wwwkeys.pgp.net From janssen at parc.com Wed Nov 9 22:22:35 2005 From: janssen at parc.com (Bill Janssen) Date: Wed, 9 Nov 2005 13:22:35 PST Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: Your message of "Wed, 09 Nov 2005 11:48:25 PST." <5.1.1.6.0.20051109144523.01f4a6a8@mail.telecommunity.com> Message-ID: <05Nov9.132241pst."58633"@synergy1.parc.xerox.com> It's a shame that 1) there's no equivalent of "java -jar", i.e., "python -z FILE.ZIP", and 2) the use of zipfiles is so poorly documented. Bill From bob at redivi.com Wed Nov 9 22:38:33 2005 From: bob at redivi.com (Bob Ippolito) Date: Wed, 9 Nov 2005 13:38:33 -0800 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: <05Nov9.132241pst."58633"@synergy1.parc.xerox.com> References: <05Nov9.132241pst."58633"@synergy1.parc.xerox.com> Message-ID: On Nov 9, 2005, at 1:22 PM, Bill Janssen wrote: > It's a shame that > > 1) there's no equivalent of "java -jar", i.e., "python -z > FILE.ZIP", and This should work on a few platforms: env PYTHONPATH=FILE.zip python -m some_module_in_the_zip -bob From theller at python.net Wed Nov 9 22:48:07 2005 From: theller at python.net (Thomas Heller) Date: Wed, 09 Nov 2005 22:48:07 +0100 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks References: <05Nov9.132241pst."58633"@synergy1.parc.xerox.com> Message-ID: <1x1ppk6g.fsf@python.net> Bob Ippolito writes: > On Nov 9, 2005, at 1:22 PM, Bill Janssen wrote: > >> It's a shame that >> >> 1) there's no equivalent of "java -jar", i.e., "python -z >> FILE.ZIP", and > > This should work on a few platforms: > env PYTHONPATH=FILE.zip python -m some_module_in_the_zip It should, yes - but it doesn't: -m doesn't work with zipimport. Thomas From bob at redivi.com Wed Nov 9 22:55:04 2005 From: bob at redivi.com (Bob Ippolito) Date: Wed, 9 Nov 2005 13:55:04 -0800 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: <1x1ppk6g.fsf@python.net> References: <05Nov9.132241pst."58633"@synergy1.parc.xerox.com> <1x1ppk6g.fsf@python.net> Message-ID: On Nov 9, 2005, at 1:48 PM, Thomas Heller wrote: > Bob Ippolito writes: > >> On Nov 9, 2005, at 1:22 PM, Bill Janssen wrote: >> >>> It's a shame that >>> >>> 1) there's no equivalent of "java -jar", i.e., "python -z >>> FILE.ZIP", and >> >> This should work on a few platforms: >> env PYTHONPATH=FILE.zip python -m some_module_in_the_zip > > It should, yes - but it doesn't: -m doesn't work with zipimport. That's dumb, someone should fix that. Is there a bug filed? -bob From ncoghlan at gmail.com Wed Nov 9 22:58:44 2005 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 Nov 2005 07:58:44 +1000 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <05Nov9.132241pst."58633"@synergy1.parc.xerox.com> Message-ID: <43727114.4030107@gmail.com> Bob Ippolito wrote: > On Nov 9, 2005, at 1:22 PM, Bill Janssen wrote: > >> It's a shame that >> >> 1) there's no equivalent of "java -jar", i.e., "python -z >> FILE.ZIP", and > > This should work on a few platforms: > env PYTHONPATH=FILE.zip python -m some_module_in_the_zip Really? I wrote the '-m' code, and I wouldn't expect that to work anywhere because 'execfile' and the C equivalent that -m relies on expect a real file. PEP 328 goes some way towards fixing that by having a Python fallback to find and execute the module if the current C code fails. If we had execmodule as a Python function, it would make it much easier to add support for compiling and executing the target module directly, rather than indirecting through the file-system-dependent execfile. In theory this could be done in C, but execmodule is fairly long even written in Python. I'm actually fairly sure it *could* be written in C, but I think doing so would be horribly tedious (and not as useful in the long run). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com From gjc at inescporto.pt Wed Nov 9 23:44:38 2005 From: gjc at inescporto.pt (Gustavo J. A. M. Carneiro) Date: Wed, 09 Nov 2005 22:44:38 +0000 Subject: [Python-Dev] Weak references: dereference notification In-Reply-To: <9E82C8B1-8A32-457D-827A-F0135EB9F8D3@mac.com> References: <1131536425.9130.10.camel@localhost> <437228E4.4070800@zope.com> <1131556500.9130.18.camel@localhost> <1131558739.9130.40.camel@localhost> <9E82C8B1-8A32-457D-827A-F0135EB9F8D3@mac.com> Message-ID: <1131576278.8540.14.camel@localhost.localdomain> On Wed, 2005-11-09 at 20:40 +0100, Ronald Oussoren wrote: > On 9-nov-2005, at 18:52, Gustavo J. A. M. Carneiro wrote: > > > Qua, 2005-11-09 ?s 09:23 -0800, Guido van Rossum escreveu: > >>>> Gustavo J. A. M. Carneiro wrote: > >>>>> I have come across a situation where I find the current weak > >>>>> references interface for extension types insufficient. > >>>>> > >>>>> Currently you only have a tp_weaklistoffset slot, pointing to a > >>>>> PyObject with weak references. However, in my case[1] I > >>>>> _really_ need > >>>>> to be notified when a weak reference is dereferenced. > >> > >> I find reading through the bug discussion a bit difficult to > >> understand your use case. Could you explain it here? If you can't > >> explain it you certainly won't get your problem solved! :-) > > > > This is a typical PyObject wrapping C object (GObject) problem. > > Both > > PyObject and GObject have independent reference counts. For each > > GObject there is at most one PyObject wrapper. > > > > When the refcount on the wrapper drops to zero, tp_dealloc is > > called. > > In tp_dealloc, and if the GObject refcount is > 1, I do something > > slightly evil: I 'resurect' the PyObject (calling PyObject_Init), > > create > > a weak reference to the GObject, and drop the "strong" reference. I > > call this a 'hibernation state'. > > Why do you do that? The only reasons I can think of are that you hope > to gain > some speed from this or that you want to support weak references to > the GObject. We want to support weak references to GObjects. Mainly because that support has always been there and we don't want/can't break API. And it does have some uses... > > For what its worth, in PyObjC we don't support weak references to the > underlying > Objective-C object and delete the proxy object when it is garbage > collected. > Objective-C also has reference counts, we increase that in the > constructor for > the proxy object and decrease it again in the destroctor. OK, but what if it is a subclass of a builtin type, with instance variables? What if the PyObject is GC'ed but the ObjC object remains alive, and later you get a new reference to it? Do you create a new PyObject wrapper for it? What happened to the instance variables? Our goal in wrapping GObject is that, once a Python wrapper for a GObject instance is created, it never dies until the GObject dies too. At the same time, once the python wrapper loses all references, it should not stop keeping the GObject alive. What happens currently, which is what I'm trying to change, is that there is a reference loop between PyObject and GObject, so that deallocation only happens with the help of the cyclic GC. But relying on the GC for _everything_ causes annoying problems: 1- The GC runs only once in a while, not soon enough if eg. you have an image object with several megabytes; 2- It makes it hard to debug reference counting bugs, as the symptom only appears when the GC runs, far away from the code that cause the problem in the first place; 3- Generally the GC has a lot more work, since every PyGTK object needs it, and a GUI app can have lots of PyGTK objects. Regards. -- Gustavo J. A. M. Carneiro The universe is always one step beyond logic From p.f.moore at gmail.com Wed Nov 9 23:56:13 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Nov 2005 22:56:13 +0000 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <1x1ppk6g.fsf@python.net> Message-ID: <79990c6b0511091456y329f1c5ey53b7428e59c97bc7@mail.gmail.com> On 11/9/05, Bob Ippolito wrote: > > On Nov 9, 2005, at 1:48 PM, Thomas Heller wrote: > > > Bob Ippolito writes: > > > >> On Nov 9, 2005, at 1:22 PM, Bill Janssen wrote: > >> > >>> It's a shame that > >>> > >>> 1) there's no equivalent of "java -jar", i.e., "python -z > >>> FILE.ZIP", and > >> > >> This should work on a few platforms: > >> env PYTHONPATH=FILE.zip python -m some_module_in_the_zip > > > > It should, yes - but it doesn't: -m doesn't work with zipimport. > > That's dumb, someone should fix that. Is there a bug filed? I did, a while ago. http://www.python.org/sf/1250389 Paul. From bcannon at gmail.com Thu Nov 10 00:05:13 2005 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 9 Nov 2005 15:05:13 -0800 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: On 11/9/05, Guido van Rossum wrote: > Maybe it makes more sense to deprecate .pyo altogether and instead > have a post-load optimizer optimize .pyc files according to the > current optimization settings? > But I thought part of the point of .pyo files was that they left out docstrings and thus had a smaller footprint? Plus I wouldn't be surprised if we started to move away from bytecode optimization and instead tried to do more AST transformations which would remove possible post-load optimizations. I would have no issue with removing .pyo files and have .pyc files just be as optimized as they the current settings are and leave it at that. Could have some metadata listing what optimizations occurred, but do we really need to have a specific way to denote if bytecode has been optimized? Binary files compiled from C don't note what -O optimization they were compiled with. If someone distributes optimized .pyc files chances are they are going to have a specific compile step with py_compile and they will know what optimizations they are using. -Brett From foom at fuhm.net Thu Nov 10 00:15:02 2005 From: foom at fuhm.net (James Y Knight) Date: Wed, 9 Nov 2005 18:15:02 -0500 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: <4132F2DC-0981-49F0-8DF4-B20FF840290D@fuhm.net> On Nov 9, 2005, at 6:05 PM, Brett Cannon wrote: > I would have no issue with removing .pyo files and have .pyc files > just be as optimized as they the current settings are and leave it at > that. Could have some metadata listing what optimizations occurred, > but do we really need to have a specific way to denote if bytecode has > been optimized? Binary files compiled from C don't note what -O > optimization they were compiled with. If someone distributes > optimized .pyc files chances are they are going to have a specific > compile step with py_compile and they will know what optimizations > they are using. > This sounds quite sensible. The only thing I'd add is that iff there is a .py file of the same name, and the current optimization settings are different from those in the .pyc file, python should recompile the .py file. James From guido at python.org Thu Nov 10 00:25:03 2005 From: guido at python.org (Guido van Rossum) Date: Wed, 9 Nov 2005 15:25:03 -0800 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: On 11/9/05, Brett Cannon wrote: > On 11/9/05, Guido van Rossum wrote: > > Maybe it makes more sense to deprecate .pyo altogether and instead > > have a post-load optimizer optimize .pyc files according to the > > current optimization settings? > > But I thought part of the point of .pyo files was that they left out > docstrings and thus had a smaller footprint? Very few people care about the smaller footprint (although one piped up here). > Plus I wouldn't be > surprised if we started to move away from bytecode optimization and > instead tried to do more AST transformations which would remove > possible post-load optimizations. > > I would have no issue with removing .pyo files and have .pyc files > just be as optimized as they the current settings are and leave it at > that. Could have some metadata listing what optimizations occurred, > but do we really need to have a specific way to denote if bytecode has > been optimized? Binary files compiled from C don't note what -O > optimization they were compiled with. If someone distributes > optimized .pyc files chances are they are going to have a specific > compile step with py_compile and they will know what optimizations > they are using. Currently, .pyo files have some important semantic differences with .pyc files; -O doesn't remove docstrings (that's -OO) but it does remove asserts. I wouldn't want to accidentally use a .pyc file without asserts compiled in unless the .py file wasn't around. For application distribution, the following probably would work: - instead of .pyo files, we use .pyc files - the .pyc file records whether optimizations were applied, whether asserts are compiled, and whether docstrings are retained - if the compiler finds a .pyc that is inconsistent with the current command line, it ignores it and rewrites it (if it is writable) just as if the .py file were newer However, this would be a major pain for the standard library and other shared code -- there it's really nice to have a cache for each of the optimization levels since usually regular users can't write the .py[co] files there, meaning very slow always-recompilation if the standard .pyc files aren't of the right level, causing unacceptable start-up times. The only solutions I can think of that use a single file actually *increase* the file size by having unoptimized and optimized code side-by-side, or some way to quickly skip the assertions -- the -OO option is a special case that probably needs to be done differently anyway and only for final distribution. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bcannon at gmail.com Thu Nov 10 01:04:15 2005 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 9 Nov 2005 16:04:15 -0800 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: On 11/9/05, Guido van Rossum wrote: > On 11/9/05, Brett Cannon wrote: > > Plus I wouldn't be > > surprised if we started to move away from bytecode optimization and > > instead tried to do more AST transformations which would remove > > possible post-load optimizations. > > > > I would have no issue with removing .pyo files and have .pyc files > > just be as optimized as they the current settings are and leave it at > > that. Could have some metadata listing what optimizations occurred, > > but do we really need to have a specific way to denote if bytecode has > > been optimized? Binary files compiled from C don't note what -O > > optimization they were compiled with. If someone distributes > > optimized .pyc files chances are they are going to have a specific > > compile step with py_compile and they will know what optimizations > > they are using. > > Currently, .pyo files have some important semantic differences with > .pyc files; -O doesn't remove docstrings (that's -OO) but it does > remove asserts. I wouldn't want to accidentally use a .pyc file > without asserts compiled in unless the .py file wasn't around. > > For application distribution, the following probably would work: > > - instead of .pyo files, we use .pyc files > - the .pyc file records whether optimizations were applied, whether > asserts are compiled, and whether docstrings are retained > - if the compiler finds a .pyc that is inconsistent with the current > command line, it ignores it and rewrites it (if it is writable) just > as if the .py file were newer > > However, this would be a major pain for the standard library and other > shared code -- there it's really nice to have a cache for each of the > optimization levels since usually regular users can't write the > .py[co] files there, meaning very slow always-recompilation if the > standard .pyc files aren't of the right level, causing unacceptable > start-up times. > What if PEP 304 came into being? Then people would have a place to have the shared code's recompiled version stored and thus avoid the overhead from repeated use. > The only solutions I can think of that use a single file actually > *increase* the file size by having unoptimized and optimized code > side-by-side, or some way to quickly skip the assertions -- the -OO > option is a special case that probably needs to be done differently > anyway and only for final distribution. > One option would be to introduce an ASSERTION bytecode that has an argument specifying the amount of bytecode for the assertion. The eval loop can then just igonore the bytecode if assertions are being evaluated and fall through to the bytecode for the assertions (and thus be the equivalent of NOP) or use the argument to jump forward that number of bytes in the bytecode and completely skip over the assertion (and thus be just like a JUMP_FORWARD). Either way assertions becomes slightly more costly but it should be very minimal. -Brett From pje at telecommunity.com Thu Nov 10 01:16:11 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Nov 2005 19:16:11 -0500 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: <5.1.1.6.0.20051109190838.01f51838@mail.telecommunity.com> At 03:25 PM 11/9/2005 -0800, Guido van Rossum wrote: >The only solutions I can think of that use a single file actually >*increase* the file size by having unoptimized and optimized code >side-by-side, or some way to quickly skip the assertions -- the -OO >option is a special case that probably needs to be done differently >anyway and only for final distribution. We could have a "JUMP_IF_NOT_DEBUG" opcode to skip over asserts and "if __debug__" blocks. Then under -O we could either patch this to a plain jump, or compact the bytecode to remove the jumped-over part(s). By the way, while we're on this subject, can we make the optimization options be part of the compile() interface? Right now the distutils has to actually exec another Python process whenever you want to compile code with a different optimization level than what's currently in effect, whereas if it could pass the desired level to compile(), this wouldn't be necessary. From guido at python.org Thu Nov 10 01:33:00 2005 From: guido at python.org (Guido van Rossum) Date: Wed, 9 Nov 2005 16:33:00 -0800 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: <5.1.1.6.0.20051109190838.01f51838@mail.telecommunity.com> References: <20051109023347.GA15823@localhost.localdomain> <5.1.1.6.0.20051109190838.01f51838@mail.telecommunity.com> Message-ID: On 11/9/05, Phillip J. Eby wrote: > At 03:25 PM 11/9/2005 -0800, Guido van Rossum wrote: > >The only solutions I can think of that use a single file actually > >*increase* the file size by having unoptimized and optimized code > >side-by-side, or some way to quickly skip the assertions -- the -OO > >option is a special case that probably needs to be done differently > >anyway and only for final distribution. > > We could have a "JUMP_IF_NOT_DEBUG" opcode to skip over asserts and "if > __debug__" blocks. Then under -O we could either patch this to a plain > jump, or compact the bytecode to remove the jumped-over part(s). That sounds very reasonable. > By the way, while we're on this subject, can we make the optimization > options be part of the compile() interface? Right now the distutils has to > actually exec another Python process whenever you want to compile > code with > a different optimization level than what's currently in effect, whereas if > it could pass the desired level to compile(), this wouldn't be necessary. Makes sense to me; we need a patch of course. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Nov 10 01:35:14 2005 From: guido at python.org (Guido van Rossum) Date: Wed, 9 Nov 2005 16:35:14 -0800 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: [Guido] > > However, this would be a major pain for the standard library and other > > shared code -- there it's really nice to have a cache for each of the > > optimization levels since usually regular users can't write the > > .py[co] files there, meaning very slow always-recompilation if the > > standard .pyc files aren't of the right level, causing unacceptable > > start-up times. [Brett] > What if PEP 304 came into being? Then people would have a place to > have the shared code's recompiled version stored and thus avoid the > overhead from repeated use. Still sounds suboptimal for the standard library; IMO it should "just work". > > The only solutions I can think of that use a single file actually > > *increase* the file size by having unoptimized and optimized code > > side-by-side, or some way to quickly skip the assertions -- the -OO > > option is a special case that probably needs to be done differently > > anyway and only for final distribution. > > One option would be to introduce an ASSERTION bytecode that has an > argument specifying the amount of bytecode for the assertion. The > eval loop can then just igonore the bytecode if assertions are being > evaluated and fall through to the bytecode for the assertions (and > thus be the equivalent of NOP) or use the argument to jump forward > that number of bytes in the bytecode and completely skip over the > assertion (and thus be just like a JUMP_FORWARD). Either way > assertions becomes slightly more costly but it should be very minimal. I like Phillip's suggestion -- no new opcode, just a conditional jump that can be easily optimized out. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bcannon at gmail.com Thu Nov 10 01:57:07 2005 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 9 Nov 2005 16:57:07 -0800 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: On 11/9/05, Guido van Rossum wrote: > [Guido] > > > However, this would be a major pain for the standard library and other > > > shared code -- there it's really nice to have a cache for each of the > > > optimization levels since usually regular users can't write the > > > .py[co] files there, meaning very slow always-recompilation if the > > > standard .pyc files aren't of the right level, causing unacceptable > > > start-up times. > [Brett] > > What if PEP 304 came into being? Then people would have a place to > > have the shared code's recompiled version stored and thus avoid the > > overhead from repeated use. > > Still sounds suboptimal for the standard library; IMO it should "just work". > Fair enough. > > > The only solutions I can think of that use a single file actually > > > *increase* the file size by having unoptimized and optimized code > > > side-by-side, or some way to quickly skip the assertions -- the -OO > > > option is a special case that probably needs to be done differently > > > anyway and only for final distribution. > > > > One option would be to introduce an ASSERTION bytecode that has an > > argument specifying the amount of bytecode for the assertion. The > > eval loop can then just igonore the bytecode if assertions are being > > evaluated and fall through to the bytecode for the assertions (and > > thus be the equivalent of NOP) or use the argument to jump forward > > that number of bytes in the bytecode and completely skip over the > > assertion (and thus be just like a JUMP_FORWARD). Either way > > assertions becomes slightly more costly but it should be very minimal. > > I like Phillip's suggestion -- no new opcode, just a conditional jump > that can be easily optimized out. Huh? But Phillip is suggesting a new opcode that is essentially the same as my proposal but naming it differently and saying the bytecode should get changed directly instead of having the eval loop handle the semantic differences based on whether -O is being used. -Brett From greg.ewing at canterbury.ac.nz Thu Nov 10 01:57:43 2005 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 10 Nov 2005 13:57:43 +1300 Subject: [Python-Dev] Weak references: dereference notification In-Reply-To: <1131576278.8540.14.camel@localhost.localdomain> References: <1131536425.9130.10.camel@localhost> <437228E4.4070800@zope.com> <1131556500.9130.18.camel@localhost> <1131558739.9130.40.camel@localhost> <9E82C8B1-8A32-457D-827A-F0135EB9F8D3@mac.com> <1131576278.8540.14.camel@localhost.localdomain> Message-ID: <43729B07.6010907@canterbury.ac.nz> Gustavo J. A. M. Carneiro wrote: > OK, but what if it is a subclass of a builtin type, with instance > variables? What if the PyObject is GC'ed but the ObjC object remains > alive, and later you get a new reference to it? Do you create a new > PyObject wrapper for it? What happened to the instance variables? Your proposed scheme appears to involve destroying and then re-initialising the Python wrapper. Isn't that going to wipe out any instance variables it may have had? Also, it seems to me that as soon as the refcount on the wrapper drops to zero, any weak references to it will be broken. Or does your resurrection code intervene before that happens? -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing at canterbury.ac.nz +--------------------------------------+ From guido at python.org Thu Nov 10 02:01:57 2005 From: guido at python.org (Guido van Rossum) Date: Wed, 9 Nov 2005 17:01:57 -0800 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: > > I like Phillip's suggestion -- no new opcode, just a conditional jump > > that can be easily optimized out. > > Huh? But Phillip is suggesting a new opcode that is essentially the > same as my proposal but naming it differently and saying the bytecode > should get changed directly instead of having the eval loop handle the > semantic differences based on whether -O is being used. Sorry. Looking back they look pretty much the same to me. Somehow I glanced over Phillip's code and thought he was proposing to use a regular JUMP_IF opcode with the special __debug__ variable (which would be a 3rd possibility, good if we had backwards compatibility requirements for bytecode -- which we don't, fortunately :-). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mdehoon at c2b2.columbia.edu Thu Nov 10 02:04:43 2005 From: mdehoon at c2b2.columbia.edu (Michiel Jan Laurens de Hoon) Date: Wed, 09 Nov 2005 20:04:43 -0500 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <43710C95.30209@v.loewis.de> References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> Message-ID: <43729CAB.5070106@c2b2.columbia.edu> Martin v. L?wis wrote: > Michiel Jan Laurens de Hoon wrote: > >> 2) Will Tkinter always be the standard toolkit for Python, or are >> there plans to replace it at some point? > > > Python does not evolve along a grand master plan. Instead, individual > contributors propose specific modifications, e.g. through PEPs. At this point, I can't propose a specific modification yet because I don't know the reasoning that went behind the original choice of Tk as the default GUI toolkit for Python (and hence, I don't know if those reasons are still valid today). I can see one disadvantage (using Tk limits our options to run an event loop for other Python extensions), and I am trying to find out why Tk was deemed more appropriate than other GUI toolkits anyway. So let me rephrase the question: What is the advantage of Tk in comparison to other GUI toolkits? Is it Mac availability? More advanced widget set? Installation is easier? Portability? Switching to a different GUI toolkit would break too much existing code? I think that having the answer to this will stimulate further development of alternative GUI toolkits, which may give some future Python version a toolkit at least as good as Tk, and one that doesn't interfere with Python's event loop capabilities. --Michiel. -- Michiel de Hoon Center for Computational Biology and Bioinformatics Columbia University 1150 St Nicholas Avenue New York, NY 10032 From bcannon at gmail.com Thu Nov 10 02:49:39 2005 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 9 Nov 2005 17:49:39 -0800 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: On 11/9/05, Guido van Rossum wrote: > > > I like Phillip's suggestion -- no new opcode, just a conditional jump > > > that can be easily optimized out. > > > > Huh? But Phillip is suggesting a new opcode that is essentially the > > same as my proposal but naming it differently and saying the bytecode > > should get changed directly instead of having the eval loop handle the > > semantic differences based on whether -O is being used. > > Sorry. No problem. Figured you just misread mine. > Looking back they look pretty much the same to me. Somehow I > glanced over Phillip's code and thought he was proposing to use a > regular JUMP_IF opcode with the special __debug__ variable (which > would be a 3rd possibility, good if we had backwards compatibility > requirements for bytecode -- which we don't, fortunately :-). > Fortunately. =) So does this mean you like the idea? Should this all move forward somehow? -Brett From guido at python.org Thu Nov 10 02:51:19 2005 From: guido at python.org (Guido van Rossum) Date: Wed, 9 Nov 2005 17:51:19 -0800 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> Message-ID: On 11/9/05, Brett Cannon wrote: > On 11/9/05, Guido van Rossum wrote: > > > > I like Phillip's suggestion -- no new opcode, just a conditional jump > > > > that can be easily optimized out. > > > > > > Huh? But Phillip is suggesting a new opcode that is essentially the > > > same as my proposal but naming it differently and saying the bytecode > > > should get changed directly instead of having the eval loop handle the > > > semantic differences based on whether -O is being used. > > > > Sorry. > > No problem. Figured you just misread mine. > > > Looking back they look pretty much the same to me. Somehow I > > glanced over Phillip's code and thought he was proposing to use a > > regular JUMP_IF opcode with the special __debug__ variable (which > > would be a 3rd possibility, good if we had backwards compatibility > > requirements for bytecode -- which we don't, fortunately :-). > > > > Fortunately. =) > > So does this mean you like the idea? Should this all move forward somehow? I guess so. :-) It will need someone thinking really hard about all the use cases, edge cases, etc., implementation details, and writing up a PEP. Feel like volunteering? You might squeeze Phillip as a co-author. He's a really good one. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From Scott.Daniels at Acm.Org Thu Nov 10 03:41:59 2005 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Wed, 09 Nov 2005 18:41:59 -0800 Subject: [Python-Dev] int(string) (was: DRAFT: python-dev Summary for 2005-09-01 through 2005-09-16) In-Reply-To: <1f7befae0510211952x5eb2000bicdf3c1a80a3f5749@mail.gmail.com> References: <1f7befae0510211952x5eb2000bicdf3c1a80a3f5749@mail.gmail.com> Message-ID: <4372B377.6050806@Acm.Org> Tim Peters wrote: > ... > Someone want a finite project that would _really_ help their Uncle > Timmy in his slow-motion crusade to get Python on the list of "solved > it!" languages for each problem on that magnificent site? ... > Turns out it's _not_ input speed that's the problem here, and not even > mainly the speed of integer mod: the bulk of the time is spent in > int(string).... OK, I got an idea about how to do this fast. I started with Python code, and I now have C code that should beat int(string) always while getting a lot of speed making long values. The working tables can be used to do the reverse transformation (int or long to string in some base) with a little finesse, but I haven't done that yet in C. The code is pretty sprawly now (a lot left in for instrumentation and testing pieces), but can eventually get smaller. I gave myself time to do this as a birthday present to myself. It may take a while to build a patch, but perhaps you can let me know how much speedup you get using this code. if you build this module, I'd suggest using "from to_int import chomp" to get a function that works like int (producing a long when needed and so on). > If you can even track all the levels of C function calls that ends up > invoking , you find yourself in PyOS_strtoul(), which is a > nifty all-purpose routine that accepts inputs in bases 2 thru 36, can > auto-detect base, and does platform-independent overflow checking at > the cost of a division per digit. All those are features, but it > makes for sloooow conversion. OK, this code doesn't deal with unicode at all. The key observations are: A) to figure out the base, you pretty much need to get to the first digit; getting to the first non-zero digit is not that much worse. B) If you know the length of a string of digits (starting at the first non-zero digit) and the base, you know approximately how bits the result will have. You can do a single allocation if you are building a long. You can tell if you need to test for overflow in building an int; there is one length per base where you must. So the question becomes, is it worth taking two passes at the digits? Well, it sure looks like it to me, but I haven't timed one or two- character integers. I do longs in "megadigits" -- the largest set of digits that fits safely in SHIFT bits, so they have no need for overflow checks. For further excitement, you can use a similar technique to go from the number of bits to the string length. That should make for a fast convert int/long to string in any of 36 (or more, really) bases. I pass all of your mentioned test cases (including the one from a later message). I'm pretty much out of time for this project at the moment, but encouraging words would help me steal some time to finish. For anyone wanting to look at the code, or try it themselves: Installer: http://members.dsl-only.net/~daniels/dist/to_int-0.10.win32-py2.4.exe Just the 2.4 dll: http://members.dsl-only.net/~daniels/dist/to_int-0.10.win32.zip Sources: http://members.dsl-only.net/~daniels/dist/to_int-0.10.zip --Scott David Daniels Scott.Daniels at Acm.Org From greg.ewing at canterbury.ac.nz Thu Nov 10 04:02:04 2005 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 10 Nov 2005 16:02:04 +1300 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <43729CAB.5070106@c2b2.columbia.edu> References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> Message-ID: <4372B82C.9010800@canterbury.ac.nz> Michiel Jan Laurens de Hoon wrote: > At this point, I can't propose a specific modification yet because I > don't know the reasoning that went behind the original choice of Tk as > the default GUI toolkit for Python Probably because at the time it was really the only cross-platform GUI toolkit that worked about equally well (or equally badly, depending on your point of view) on all the major platforms. I'm not sure the event-loop situation would be much different with another one, anyway. From what I've seen of GUI toolkits, they all have their own form of event loop, and they all provide some way of hooking other things into it (as does Tkinter), but whichever one you're using, it likes to be in charge. Code which blocks reading from standard input doesn't fit very well into any of them. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing at canterbury.ac.nz +--------------------------------------+ From exarkun at divmod.com Thu Nov 10 04:08:52 2005 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Wed, 9 Nov 2005 22:08:52 -0500 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <4372B82C.9010800@canterbury.ac.nz> Message-ID: <20051110030852.10365.1719239053.divmod.quotient.6042@ohm> On Thu, 10 Nov 2005 16:02:04 +1300, Greg Ewing wrote: >Michiel Jan Laurens de Hoon wrote: > >> At this point, I can't propose a specific modification yet because I >> don't know the reasoning that went behind the original choice of Tk as >> the default GUI toolkit for Python > >Probably because at the time it was really the >only cross-platform GUI toolkit that worked >about equally well (or equally badly, depending >on your point of view) on all the major >platforms. > >I'm not sure the event-loop situation would be >much different with another one, anyway. From what >I've seen of GUI toolkits, they all have their own >form of event loop, and they all provide some way >of hooking other things into it (as does Tkinter), >but whichever one you're using, it likes to be in >charge. Code which blocks reading from standard >input doesn't fit very well into any of them. > Of course, the problem could be approached from the other direction: the blocking reads could be replaced with something else... Jean-Paul From janssen at parc.com Thu Nov 10 05:00:44 2005 From: janssen at parc.com (Bill Janssen) Date: Wed, 9 Nov 2005 20:00:44 PST Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: Your message of "Wed, 09 Nov 2005 13:38:33 PST." Message-ID: <05Nov9.200052pst."58633"@synergy1.parc.xerox.com> > This should work on a few platforms: > env PYTHONPATH=FILE.zip python -m some_module_in_the_zip Yeah, that's not bad, but I hate setting PYTHONPATH. I was thinking more along the line of python -z ZIPFILE where python would look at the ZIPFILE to see if there's a top-level module called "__init__", and if so, load it. That would allow existing PYTHONPATH settings to still be used if the user cares. Bill From falcon at intercable.ru Wed Nov 9 08:24:04 2005 From: falcon at intercable.ru (Sokolov Yura) Date: Wed, 09 Nov 2005 10:24:04 +0300 Subject: [Python-Dev] Unifying decimal numbers. Message-ID: <4371A414.2020400@intercable.ru> Excuse my English I think, we could just segregate tokens for decimal and real float and make them interoperable. Motivation: Most of us works with business databases - all "floats" are really decimals, algebraic operations should work without float inconsistency and those operations rare so speed is not important. But some of us use floats for speed in scientific and multimedia programs. with from __future__ import Decimal we could: a) interpret regular float constants as decimal b) interpret float constants with suffix 'f' as float (like 1.5f 345.2e-5f etc) c) result of operation with decimal operands should be decimal >>> 1.0/3.0 0.33333333333333333 d) result of operation with float operands should be float >>> 1.0f/3.0f 0.33333333333333331f e) result of operation with decimal and float should be float (decimal converts into float and operation perfomed) >>> 1.0f/3.0 0.33333333333333331f >>> 1.0/3.0f 0.33333333333333331f From bcannon at gmail.com Thu Nov 10 06:14:14 2005 From: bcannon at gmail.com (Brett Cannon) Date: Wed, 9 Nov 2005 21:14:14 -0800 Subject: [Python-Dev] dev FAQ updated with day-to-day svn questions Message-ID: I just finished fleshing out the dev FAQ (http://www.python.org/dev/devfaq.html) with questions covering what someone might need to know for regular usage. If anyone thinks I didn't cover something I should have, let me know. -Brett From stephen at xemacs.org Thu Nov 10 06:19:42 2005 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 10 Nov 2005 14:19:42 +0900 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <43729CAB.5070106@c2b2.columbia.edu> (Michiel Jan Laurens de Hoon's message of "Wed, 09 Nov 2005 20:04:43 -0500") References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> Message-ID: <87br0tw041.fsf@tleepslib.sk.tsukuba.ac.jp> >>>>> "Michiel" == Michiel Jan Laurens de Hoon writes: Michiel> What is the advantage of Tk in comparison to other GUI Michiel> toolkits? IMO, Tk's _advantage_ is that it's there already. As a standard component, it works well for typical simple GUI applications (thus satisfying "batteries included" IMO), and it's self-contained. So I would say it's at _no disadvantage_ to other toolkits. Alternatives like PyGtk and wxWidgets are easily available and provide some degree of cross-platform support for those who need something more/different. Is there some reason why you can't require users to install a toolkit more suited to your application's needs? -- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software. From mdehoon at c2b2.columbia.edu Thu Nov 10 06:27:22 2005 From: mdehoon at c2b2.columbia.edu (Michiel Jan Laurens de Hoon) Date: Thu, 10 Nov 2005 00:27:22 -0500 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <4372B82C.9010800@canterbury.ac.nz> References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> <4372B82C.9010800@canterbury.ac.nz> Message-ID: <4372DA3A.8010206@c2b2.columbia.edu> Greg Ewing wrote: >I'm not sure the event-loop situation would be >much different with another one, anyway. From what >I've seen of GUI toolkits, they all have their own >form of event loop, and they all provide some way >of hooking other things into it (as does Tkinter), >but whichever one you're using, it likes to be in >charge. > It's not because it likes to be in charge, it's because there's no other way to do it in Python. In our scientific visualization software, we also have our own event loop. I'd much rather let a Python event loop handle our messages. Not only would it save us time programming (setting up an event loop in an extension module that passes control back to Python when needed is tricky), it would also give better performance, it would work with IDLE (which an event loop in an extension module cannot as explained in my previous post), and it would let different extension modules live happily together all using the same event loop. Tkinter is a special case among GUI toolkits because it is married to Tcl. It doesn't just need to handle its GUI events, it also needs to run the Tcl interpreter in between. Which is why Tkinter needs to be in charge of the event loop. For other GUI toolkits, I don't see a reason why they'd need their own event loop. > Code which blocks reading from standard >input doesn't fit very well into any of them. > > Actually, this is not difficult to accomplish. For example, try Tcl's wish on Linux: It will pop up a (responsive) graphics window but continue to read Tcl commands from the terminal. This is done via a call to select (on Unix) or MsgWaitForMultipleObjects (on Windows). Both of these can listen for terminal input and GUI events at the same time. --Michiel. -- Michiel de Hoon Center for Computational Biology and Bioinformatics Columbia University 1150 St Nicholas Avenue New York, NY 10032 From mdehoon at c2b2.columbia.edu Thu Nov 10 06:40:47 2005 From: mdehoon at c2b2.columbia.edu (Michiel Jan Laurens de Hoon) Date: Thu, 10 Nov 2005 00:40:47 -0500 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <87br0tw041.fsf@tleepslib.sk.tsukuba.ac.jp> References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> <87br0tw041.fsf@tleepslib.sk.tsukuba.ac.jp> Message-ID: <4372DD5F.70203@c2b2.columbia.edu> Stephen J. Turnbull wrote: > Michiel> What is the advantage of Tk in comparison to other GUI > Michiel> toolkits? > >IMO, Tk's _advantage_ is that it's there already. As a standard >component, it works well for typical simple GUI applications (thus >satisfying "batteries included" IMO), and it's self-contained. So I >would say it's at _no disadvantage_ to other toolkits. > >Alternatives like PyGtk and wxWidgets are easily available and provide >some degree of cross-platform support for those who need something >more/different. > >Is there some reason why you can't require users to install a toolkit >more suited to your application's needs? > > My application doesn't need a toolkit at all. My problem is that because of Tkinter being the standard Python toolkit, we cannot have a decent event loop in Python. So this is the disadvantage I see in Tkinter. --Michiel. -- Michiel de Hoon Center for Computational Biology and Bioinformatics Columbia University 1150 St Nicholas Avenue New York, NY 10032 From ronaldoussoren at mac.com Thu Nov 10 08:06:02 2005 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 10 Nov 2005 08:06:02 +0100 Subject: [Python-Dev] Weak references: dereference notification In-Reply-To: <1131576278.8540.14.camel@localhost.localdomain> References: <1131536425.9130.10.camel@localhost> <437228E4.4070800@zope.com> <1131556500.9130.18.camel@localhost> <1131558739.9130.40.camel@localhost> <9E82C8B1-8A32-457D-827A-F0135EB9F8D3@mac.com> <1131576278.8540.14.camel@localhost.localdomain> Message-ID: <49FAFACC-3892-49EA-9154-1AC43E533179@mac.com> On 9-nov-2005, at 23:44, Gustavo J. A. M. Carneiro wrote: > On Wed, 2005-11-09 at 20:40 +0100, Ronald Oussoren wrote: >> On 9-nov-2005, at 18:52, Gustavo J. A. M. Carneiro wrote: >> >>> Qua, 2005-11-09 ?s 09:23 -0800, Guido van Rossum escreveu: >>>>>> Gustavo J. A. M. Carneiro wrote: >>>>>>> I have come across a situation where I find the current weak >>>>>>> references interface for extension types insufficient. >>>>>>> >>>>>>> Currently you only have a tp_weaklistoffset slot, pointing >>>>>>> to a >>>>>>> PyObject with weak references. However, in my case[1] I >>>>>>> _really_ need >>>>>>> to be notified when a weak reference is dereferenced. >>>> >>>> I find reading through the bug discussion a bit difficult to >>>> understand your use case. Could you explain it here? If you can't >>>> explain it you certainly won't get your problem solved! :-) >>> >>> This is a typical PyObject wrapping C object (GObject) problem. >>> Both >>> PyObject and GObject have independent reference counts. For each >>> GObject there is at most one PyObject wrapper. >>> >>> When the refcount on the wrapper drops to zero, tp_dealloc is >>> called. >>> In tp_dealloc, and if the GObject refcount is > 1, I do something >>> slightly evil: I 'resurect' the PyObject (calling PyObject_Init), >>> create >>> a weak reference to the GObject, and drop the "strong" reference. I >>> call this a 'hibernation state'. >> >> Why do you do that? The only reasons I can think of are that you hope >> to gain >> some speed from this or that you want to support weak references to >> the GObject. > > We want to support weak references to GObjects. Mainly because that > support has always been there and we don't want/can't break API. > And it > does have some uses... > >> >> For what its worth, in PyObjC we don't support weak references to the >> underlying >> Objective-C object and delete the proxy object when it is garbage >> collected. >> Objective-C also has reference counts, we increase that in the >> constructor for >> the proxy object and decrease it again in the destroctor. > > OK, but what if it is a subclass of a builtin type, with instance > variables? What if the PyObject is GC'ed but the ObjC object remains > alive, and later you get a new reference to it? Do you create a new > PyObject wrapper for it? What happened to the instance variables? Our main goal is that there is at most one wrapper for a python object alive at any one time. And likewise there is at most one Objective-C wrapper for a python object. If a PyObject is GC'ed and the ObjC object remains alive you will get a new PyObject when a reference to the ObjC object passes into python space again. That is no problem because the proxy object contains no state other than the pointer to the ObjC object. ObjC's runtime might be more flexible than that of GObject. If you create a subclass of an ObjC class the PyObjC runtime will create a real ObjC class for you and all object state, including Python instance variables, are stored on the ObjC side. > > Our goal in wrapping GObject is that, once a Python wrapper for a > GObject instance is created, it never dies until the GObject dies too. > At the same time, once the python wrapper loses all references, it > should not stop keeping the GObject alive. I tried that too, but ran into some very ugly issues and decided that weak references are not important enough for that. There's also the problem that this will keep the python proxy alive even when it is not needed anymore, which gives significant overhead of you traverse a large datastructure. What I don't quite understand is how you know that your python wrapper is the last reference to the GObject and your wrapper should not be forcefully kept alive. > > What happens currently, which is what I'm trying to change, is that > there is a reference loop between PyObject and GObject, so that > deallocation only happens with the help of the cyclic GC. But relying > on the GC for _everything_ causes annoying problems: At one time I used Python's reference counts as the reference count of the Objective-C object (ObjC's reference count management is done through method calls and can therefore be overridden in subclasses). That did work, but getting the semantics completely correct turned the code into a mess. Our current solution is much more satisfying. > > 1- The GC runs only once in a while, not soon enough if eg. you > have an > image object with several megabytes; > > 2- It makes it hard to debug reference counting bugs, as the symptom > only appears when the GC runs, far away from the code that cause the > problem in the first place; > > 3- Generally the GC has a lot more work, since every PyGTK object > needs > it, and a GUI app can have lots of PyGTK objects. > > Regards. > > -- > Gustavo J. A. M. Carneiro > > The universe is always one step beyond logic > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ > ronaldoussoren%40mac.com From martin at v.loewis.de Thu Nov 10 08:15:00 2005 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 10 Nov 2005 08:15:00 +0100 Subject: [Python-Dev] Weak references: dereference notification In-Reply-To: <1131576278.8540.14.camel@localhost.localdomain> References: <1131536425.9130.10.camel@localhost> <437228E4.4070800@zope.com> <1131556500.9130.18.camel@localhost> <1131558739.9130.40.camel@localhost> <9E82C8B1-8A32-457D-827A-F0135EB9F8D3@mac.com> <1131576278.8540.14.camel@localhost.localdomain> Message-ID: <4372F374.4060709@v.loewis.de> Gustavo J. A. M. Carneiro wrote: > OK, but what if it is a subclass of a builtin type, with instance > variables? What if the PyObject is GC'ed but the ObjC object remains > alive, and later you get a new reference to it? Do you create a new > PyObject wrapper for it? What happened to the instance variables? Normally, wrappers don't have state. But if you do have state, this is how it could work: 1. Make two Python objects, PyState and PyWrapper (actually, PyState doesn't need to be a Python object) PyState holds the instance variables, and PyWrapper just holds a pointer to a GObject. 2. When a Python reference to a GObject is created for the first time, create both a PyState and a PyWrapper. Have the GObject point to the PyState, and the PyWrapper to the GObject. Have the PyState weakly reference the PyWrapper. 3. When the refcount to the PyWrapper drops to zero, discard it. 4. When somebody asks for the data in the PyWrapper, go to the GObject, then to the PyState, and return the data from there. 5. When somebody wants a reference to a GObject which already has a PyState, check the weak reference to find out whether there is a PyWrapper already. If yes, return it; if not, create a new one (and weakly reference it). 6. When the GObject is discarded, drop the PyState as well. This has the following properties: 1. There are no cyclic references for wrapping GObjects. 2. Weakly-referencing wrappers is supported; if there are no strong Python references to the wrapper, the wrapper goes away, and, potentially, the GObject as well. 3. The PyState object lives as long as the GObject. 4. Using "is" for GObjects/PyWrappers "works": there is at most one PyWrapper per GObject at any time. 5. id() of a GObject may change over time, if the wrapper becomes unreferenced and then recreated. Regards, Martin From martin at v.loewis.de Thu Nov 10 08:26:51 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 10 Nov 2005 08:26:51 +0100 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <43729CAB.5070106@c2b2.columbia.edu> References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> Message-ID: <4372F63B.70301@v.loewis.de> Michiel Jan Laurens de Hoon wrote: > At this point, I can't propose a specific modification yet because I > don't know the reasoning that went behind the original choice of Tk as > the default GUI toolkit for Python (and hence, I don't know if those > reasons are still valid today). I don't know, either, but I guess that it was the only option as a cross-platform GUI at the time when it was created. > I can see one disadvantage (using Tk > limits our options to run an event loop for other Python extensions), > and I am trying to find out why Tk was deemed more appropriate than > other GUI toolkits anyway. I don't think this is a disadvantage: my guess is that other GUI toolkits share the very same problems. So even though it looks like a limitation of Tkinter, it really is a fundamental limitation, and Tk is not any worse than the competitors. Also, I firmly believe that whatever your event processing requirements are, that there is a solution that meets all your end-user needs. That solution would fail the requirement to be easy to implement for you (IOW, it may take some work). > So let me rephrase the question: What is the advantage of Tk in > comparison to other GUI toolkits? It comes bundled with Python. If this sounds circular: It is. Whatever the historical reasons for original inclusion where, I'm sure they are not that important anymore. Today, what matters is that an actual implementation of a GUI toolkit integration is actively being maintained, in the Python core. This is something that is not the case for any other GUI toolkit. If you think it would be easy to change: it isn't. Somebody would have to step forward and say "I will maintain it for the next 10 years". Removal of Tkinter would meet strong resistance, so it would have to be maintained in addition to Tkinter. Nobody has stepped forward making such an offer. For Tkinter, it's different: because it *already* is part of Python, various maintainers fix problems as they find them, and contributors contribute improvements. > Is it Mac availability? More advanced > widget set? Installation is easier? Portability? These are all important, yes. But other GUI toolkits likely have the same properties. > Switching to a > different GUI toolkit would break too much existing code? Most definitely, yes. Switching would not be an option at all. Another GUI toolkit would have to be an addition, not a replacement. > I think that > having the answer to this will stimulate further development of > alternative GUI toolkits, which may give some future Python version a > toolkit at least as good as Tk, and one that doesn't interfere with > Python's event loop capabilities. I personally don't think so. The task is just too huge for volunteers to commit to. Regards, Martin From Scott.Daniels at Acm.Org Thu Nov 10 08:28:11 2005 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Wed, 09 Nov 2005 23:28:11 -0800 Subject: [Python-Dev] to_int -- oops, one step missing for use. In-Reply-To: <4372B377.6050806@Acm.Org> References: <1f7befae0510211952x5eb2000bicdf3c1a80a3f5749@mail.gmail.com> <4372B377.6050806@Acm.Org> Message-ID: <4372F68B.5050106@Acm.Org> Well, wouldn't you know it. I get the code right and mess up the directions. Scott David Daniels wrote: > if you build this module, I'd suggest using > "from to_int import chomp" to get a function that works like int > (producing a long when needed and so on). Well, actually it is a bit more than that. "from to_int import chomp, _flag; _flag(1)" This sets a flag to suppress the return of the length along with the value from chomp. --Scott David Daniels Scott.Daniels at Acm.Org From martin at v.loewis.de Thu Nov 10 08:30:51 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 10 Nov 2005 08:30:51 +0100 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <4372DA3A.8010206@c2b2.columbia.edu> References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> <4372B82C.9010800@canterbury.ac.nz> <4372DA3A.8010206@c2b2.columbia.edu> Message-ID: <4372F72B.9060501@v.loewis.de> Michiel Jan Laurens de Hoon wrote: > It's not because it likes to be in charge, it's because there's no other > way to do it in Python. As I said: this is simply not true. > Tkinter is a special case among GUI toolkits because it is married to > Tcl. It doesn't just need to handle its GUI events, it also needs to run > the Tcl interpreter in between. That statement is somewhat deceiving: there isn't much interpreter to run, really. > Which is why Tkinter needs to be in > charge of the event loop. For other GUI toolkits, I don't see a reason > why they'd need their own event loop. They need to fetch events from the operating system level, and dispatch them to the widgets. Regards, Martin From fredrik at pythonware.com Thu Nov 10 09:04:24 2005 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 10 Nov 2005 09:04:24 +0100 Subject: [Python-Dev] dev FAQ updated with day-to-day svn questions References: Message-ID: Brett Cannon wrote: >I just finished fleshing out the dev FAQ > (http://www.python.org/dev/devfaq.html) with questions covering what > someone might need to know for regular usage. If anyone thinks I > didn't cover something I should have, let me know. SVK! From ncoghlan at gmail.com Thu Nov 10 10:11:50 2005 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 Nov 2005 19:11:50 +1000 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: <79990c6b0511091456y329f1c5ey53b7428e59c97bc7@mail.gmail.com> References: <1x1ppk6g.fsf@python.net> <79990c6b0511091456y329f1c5ey53b7428e59c97bc7@mail.gmail.com> Message-ID: <43730ED6.1010807@gmail.com> Paul Moore wrote: > On 11/9/05, Bob Ippolito wrote: >> On Nov 9, 2005, at 1:48 PM, Thomas Heller wrote: >> >>> Bob Ippolito writes: >>> >>>> On Nov 9, 2005, at 1:22 PM, Bill Janssen wrote: >>>> >>>>> It's a shame that >>>>> >>>>> 1) there's no equivalent of "java -jar", i.e., "python -z >>>>> FILE.ZIP", and >>>> This should work on a few platforms: >>>> env PYTHONPATH=FILE.zip python -m some_module_in_the_zip >>> It should, yes - but it doesn't: -m doesn't work with zipimport. >> That's dumb, someone should fix that. Is there a bug filed? > > I did, a while ago. http://www.python.org/sf/1250389 Please consider looking at and commenting on PEP 328 - I got zero feedback when I wrote it, and basically assumed no-one else was bothered by the -m switch's fairly significant limitations (it went in close to the first Python 2.4 alpha release, so we wanted to keep it simple). The PEP and the associated patch currently only cover lifting the limitation against executing modules inside packages, but it should be possible to extend it to cover executing modules inside zip files as well (as you say, increasing use of eggs will only make the current limitations more annoying). That discussion should probably happen on c.l.p, though. cc' me if you start one, and I can keep on eye on it through Google (I won't have time to participate actively, unfortunately :() Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com From ncoghlan at gmail.com Thu Nov 10 10:15:14 2005 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 Nov 2005 19:15:14 +1000 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: <43730ED6.1010807@gmail.com> References: <1x1ppk6g.fsf@python.net> <79990c6b0511091456y329f1c5ey53b7428e59c97bc7@mail.gmail.com> <43730ED6.1010807@gmail.com> Message-ID: <43730FA2.6070404@gmail.com> Nick Coghlan wrote: > Please consider looking at and commenting on PEP 328 - I got zero feedback > when I wrote it, and basically assumed no-one else was bothered by the -m > switch's fairly significant limitations (it went in close to the first Python > 2.4 alpha release, so we wanted to keep it simple). Oops, that should be PEP 3*3*8. PEP 328 is something completely different. That'll teach me to post without checking the PEP number ;) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com From ncoghlan at gmail.com Thu Nov 10 10:21:55 2005 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 Nov 2005 19:21:55 +1000 Subject: [Python-Dev] dev FAQ updated with day-to-day svn questions In-Reply-To: References: Message-ID: <43731133.9000904@gmail.com> Brett Cannon wrote: > I just finished fleshing out the dev FAQ > (http://www.python.org/dev/devfaq.html) with questions covering what > someone might need to know for regular usage. If anyone thinks I > didn't cover something I should have, let me know. Should the section "Developing on Windows" disappear now? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com From ncoghlan at gmail.com Thu Nov 10 10:43:43 2005 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 Nov 2005 19:43:43 +1000 Subject: [Python-Dev] dev FAQ updated with day-to-day svn questions In-Reply-To: References: Message-ID: <4373164F.8070606@gmail.com> Brett Cannon wrote: > I just finished fleshing out the dev FAQ > (http://www.python.org/dev/devfaq.html) with questions covering what > someone might need to know for regular usage. If anyone thinks I > didn't cover something I should have, let me know. For question 1.2.10, I believe you also want: [miscellany] enable-auto-props = yes so that "svn add" works properly. Question 1.4.1 should cover the use of "svn diff" instead of "cvs diff" to make the patch. On that note, we need to update the patch submission guidelines to point to SVN instead of CVS (those guidelines also still say context diffs are preferred to unified diffs, which I believe is no longer true). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com From p.f.moore at gmail.com Thu Nov 10 11:23:48 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 10 Nov 2005 10:23:48 +0000 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: <43730ED6.1010807@gmail.com> References: <1x1ppk6g.fsf@python.net> <79990c6b0511091456y329f1c5ey53b7428e59c97bc7@mail.gmail.com> <43730ED6.1010807@gmail.com> Message-ID: <79990c6b0511100223l20d66617ka681223cf2fb7c0@mail.gmail.com> On 11/10/05, Nick Coghlan wrote: > Paul Moore wrote: > > On 11/9/05, Bob Ippolito wrote: > >> On Nov 9, 2005, at 1:48 PM, Thomas Heller wrote: > >> > >>> Bob Ippolito writes: > >>> > >>>> On Nov 9, 2005, at 1:22 PM, Bill Janssen wrote: > >>>> > >>>>> It's a shame that > >>>>> > >>>>> 1) there's no equivalent of "java -jar", i.e., "python -z > >>>>> FILE.ZIP", and > >>>> This should work on a few platforms: > >>>> env PYTHONPATH=FILE.zip python -m some_module_in_the_zip > >>> It should, yes - but it doesn't: -m doesn't work with zipimport. > >> That's dumb, someone should fix that. Is there a bug filed? > > > > I did, a while ago. http://www.python.org/sf/1250389 > > Please consider looking at and commenting on PEP 328 - I got zero feedback > when I wrote it, and basically assumed no-one else was bothered by the -m > switch's fairly significant limitations (it went in close to the first Python > 2.4 alpha release, so we wanted to keep it simple). > > The PEP and the associated patch currently only cover lifting the limitation > against executing modules inside packages, but it should be possible to extend > it to cover executing modules inside zip files as well (as you say, increasing > use of eggs will only make the current limitations more annoying). > > That discussion should probably happen on c.l.p, though. cc' me if you start > one, and I can keep on eye on it through Google (I won't have time to > participate actively, unfortunately :() I didn't respond simply because it seemed obvious that this should go in, and I expected no debate. I assumed the only reason it didn't go into 2.4 was because the issue came up too close to the release. Teach me to assume, I guess... FWIW, I'm +1 on PEP 338. Paul. From gjc at inescporto.pt Thu Nov 10 13:04:23 2005 From: gjc at inescporto.pt (Gustavo J. A. M. Carneiro) Date: Thu, 10 Nov 2005 12:04:23 +0000 Subject: [Python-Dev] Weak references: dereference notification In-Reply-To: <4372F374.4060709@v.loewis.de> References: <1131536425.9130.10.camel@localhost> <437228E4.4070800@zope.com> <1131556500.9130.18.camel@localhost> <1131558739.9130.40.camel@localhost> <9E82C8B1-8A32-457D-827A-F0135EB9F8D3@mac.com> <1131576278.8540.14.camel@localhost.localdomain> <4372F374.4060709@v.loewis.de> Message-ID: <1131624263.4292.16.camel@localhost> Qui, 2005-11-10 ?s 08:15 +0100, "Martin v. L?wis" escreveu: > Gustavo J. A. M. Carneiro wrote: > > OK, but what if it is a subclass of a builtin type, with instance > > variables? What if the PyObject is GC'ed but the ObjC object remains > > alive, and later you get a new reference to it? Do you create a new > > PyObject wrapper for it? What happened to the instance variables? > > Normally, wrappers don't have state. But if you do have state, this > is how it could work: > > 1. Make two Python objects, PyState and PyWrapper (actually, > PyState doesn't need to be a Python object) > PyState holds the instance variables, and PyWrapper just > holds a pointer to a GObject. > 2. When a Python reference to a GObject is created for the > first time, create both a PyState and a PyWrapper. Have > the GObject point to the PyState, and the PyWrapper to > the GObject. Have the PyState weakly reference the > PyWrapper. > 3. When the refcount to the PyWrapper drops to zero, discard it. > 4. When somebody asks for the data in the PyWrapper, > go to the GObject, then to the PyState, and return the > data from there. > 5. When somebody wants a reference to a GObject which already > has a PyState, check the weak reference to find out > whether there is a PyWrapper already. If yes, return it; > if not, create a new one (and weakly reference it). > 6. When the GObject is discarded, drop the PyState as well. > > This has the following properties: > 1. There are no cyclic references for wrapping GObjects. > 2. Weakly-referencing wrappers is supported; if there > are no strong Python references to the wrapper, > the wrapper goes away, and, potentially, the GObject > as well. > 3. The PyState object lives as long as the GObject. > 4. Using "is" for GObjects/PyWrappers "works": there is > at most one PyWrapper per GObject at any time. > 5. id() of a GObject may change over time, if the wrapper > becomes unreferenced and then recreated. This was my first approach, actually, in patch 4.1 in [1]. Only your property 2 above drove me to try a different approach -- the weakrefs may become invalid while the GObject may still be alive. That's a bit "surprising". Of course, if I could override weakref.ref() for GObject wrapper types, even that could be worked around... ;-) Thanks, [1] http://bugzilla.gnome.org/show_bug.cgi?id=320428 -- Gustavo J. A. M. Carneiro The universe is always one step beyond logic. From gjc at inescporto.pt Thu Nov 10 13:12:58 2005 From: gjc at inescporto.pt (Gustavo J. A. M. Carneiro) Date: Thu, 10 Nov 2005 12:12:58 +0000 Subject: [Python-Dev] Weak references: dereference notification In-Reply-To: <43729B07.6010907@canterbury.ac.nz> References: <1131536425.9130.10.camel@localhost> <437228E4.4070800@zope.com> <1131556500.9130.18.camel@localhost> <1131558739.9130.40.camel@localhost> <9E82C8B1-8A32-457D-827A-F0135EB9F8D3@mac.com> <1131576278.8540.14.camel@localhost.localdomain> <43729B07.6010907@canterbury.ac.nz> Message-ID: <1131624778.4292.22.camel@localhost> Qui, 2005-11-10 ?s 13:57 +1300, Greg Ewing escreveu: > Gustavo J. A. M. Carneiro wrote: > > > OK, but what if it is a subclass of a builtin type, with instance > > variables? What if the PyObject is GC'ed but the ObjC object remains > > alive, and later you get a new reference to it? Do you create a new > > PyObject wrapper for it? What happened to the instance variables? > > Your proposed scheme appears to involve destroying and > then re-initialising the Python wrapper. Isn't that > going to wipe out any instance variables it may > have had? The object isn't really destroyed. Simply ob_refcnt drops to zero, then tp_dealloc is called, which is supposed to destroy it. But since I wrote tp_dealloc, I choose not to destroy it, and revive it by calling PyObject_Init(), which makes ob_refcnt == 1 again, among other things. > > Also, it seems to me that as soon as the refcount on > the wrapper drops to zero, any weak references to it > will be broken. Or does your resurrection code > intervene before that happens? Yes, I intervene before that happens. Regards. -- Gustavo J. A. M. Carneiro The universe is always one step beyond logic. From abo at minkirri.apana.org.au Thu Nov 10 14:47:00 2005 From: abo at minkirri.apana.org.au (Donovan Baarda) Date: Thu, 10 Nov 2005 13:47:00 +0000 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <4372DD5F.70203@c2b2.columbia.edu> References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> <87br0tw041.fsf@tleepslib.sk.tsukuba.ac.jp> <4372DD5F.70203@c2b2.columbia.edu> Message-ID: <1131630420.12077.44.camel@warna.corp.google.com> On Thu, 2005-11-10 at 00:40 -0500, Michiel Jan Laurens de Hoon wrote: > Stephen J. Turnbull wrote: > > > Michiel> What is the advantage of Tk in comparison to other GUI > > Michiel> toolkits? [...] > My application doesn't need a toolkit at all. My problem is that because > of Tkinter being the standard Python toolkit, we cannot have a decent > event loop in Python. So this is the disadvantage I see in Tkinter. [...] I'm kind of surprised no-one has mentioned Twisted in this thread. Twisted is an async-framework that I believe has support for using a variety of different event-loops, including Tkinter and wxWidgets, as well as it's own. It has been heavily re-factored many times, so if you want to see the current Python "state of the art" way of doing this, I'd be having a look at what they are doing. -- Donovan Baarda http://minkirri.apana.org.au/~abo/ From guido at python.org Thu Nov 10 17:50:40 2005 From: guido at python.org (Guido van Rossum) Date: Thu, 10 Nov 2005 08:50:40 -0800 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <4372DD5F.70203@c2b2.columbia.edu> References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> <87br0tw041.fsf@tleepslib.sk.tsukuba.ac.jp> <4372DD5F.70203@c2b2.columbia.edu> Message-ID: On 11/9/05, Michiel Jan Laurens de Hoon wrote: > My application doesn't need a toolkit at all. My problem is that because > of Tkinter being the standard Python toolkit, we cannot have a decent > event loop in Python. So this is the disadvantage I see in Tkinter. That's a non-sequitur if I ever saw one. Who gave you that idea? There is no connection. (If there's *any* reason for Python not having a standard event loop it's probably because I've never needed one.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mdehoon at c2b2.columbia.edu Thu Nov 10 18:16:36 2005 From: mdehoon at c2b2.columbia.edu (Michiel Jan Laurens de Hoon) Date: Thu, 10 Nov 2005 12:16:36 -0500 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <4372F72B.9060501@v.loewis.de> References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> <4372B82C.9010800@canterbury.ac.nz> <4372DA3A.8010206@c2b2.columbia.edu> <4372F72B.9060501@v.loewis.de> Message-ID: <43738074.2030508@c2b2.columbia.edu> Martin v. L?wis wrote: > Michiel Jan Laurens de Hoon wrote: > >> It's not because it likes to be in charge, it's because there's no >> other way to do it in Python. > > As I said: this is simply not true. You are right in the sense it is possible to get events handled using the solutions you proposed before (sorry for not responding to those earlier). But I don't believe that these are very good solutions: > You are missing multi-threading, which is the widely used > approach to doing things simultaneously in a single process. In one > thread, user interaction can occur; in another, computation. If you need > non-blocking interaction between the threads, use queues, or other > global variables. If you have other event sources, deal with them > in separate threads. The problem with threading (apart from potential portability problems) is that Python doesn't let us know when it's idle. This would cause excessive repainting (I can give you an explicit example if you're interested). But there is another solution with threads: Can we let Tkinter run in a separate thread instead? > Yes, it is possible to get event loops with Tkinter. Atleast on Unix, > you can install a file handler into the Tk event loop (through > createfilehandler), which gives you callbacks whenever there is some > activity on the files. This works, but only if Tkinter is installed, and even then it will give poor performance due to the busy-loop with 20 ms sleep in between in Tkinter. Furthermore, this will not work with IDLE, because the Python thread that handles user commands never enters the Tkinter event loop, even if we import Tkinter. AFAIK, there is no easy solution to this. > Furthermore, it is possible to turn the event loop around, by doing > dooneevent explicitly. Here, the problem is that we don't know *when* to call dooneevent, so we'd have to do a busy-loop and sleep in between. >> Tkinter is a special case among GUI toolkits because it is married to >> Tcl. It doesn't just need to handle its GUI events, it also needs to >> run the Tcl interpreter in between. > > That statement is somewhat deceiving: there isn't much interpreter to > run, really. I may be wrong here, but I'd think that it would be dangerous to run Tkinter's event loop when one thread is waiting for another (as happens in IDLE). >> Which is why Tkinter needs to be in charge of the event loop. For >> other GUI toolkits, I don't see a reason why they'd need their own >> event loop. > > They need to fetch events from the operating system level, and dispatch > them to the widgets. This is a perfect task for an event loop located in Python, instead of in an extension module. I could write a prototype event loop for Python to demonstrate how this would work. Sorry if I'm sounding negative, but we've actually considered many different things to get the event loop working for our scientific visualization software, and we were never able to come up with a satisfactory scheme within the current Python framework. Other packages have run into the same problem (e.g. matplotlib, which now recommends using the interactive ipython instead of regular python; the python extension for the Rasmol protein viewer is another). --Michiel. -- Michiel de Hoon Center for Computational Biology and Bioinformatics Columbia University 1150 St Nicholas Avenue New York, NY 10032 From pje at telecommunity.com Thu Nov 10 18:46:34 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 10 Nov 2005 12:46:34 -0500 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <5.1.1.6.0.20051109190838.01f51838@mail.telecommunity.com> <20051109023347.GA15823@localhost.localdomain> <5.1.1.6.0.20051109190838.01f51838@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20051110124246.02bac470@mail.telecommunity.com> At 04:33 PM 11/9/2005 -0800, Guido van Rossum wrote: >On 11/9/05, Phillip J. Eby wrote: > > By the way, while we're on this subject, can we make the optimization > > options be part of the compile() interface? Right now the distutils has to > > actually exec another Python process whenever you want to compile > > code with > > a different optimization level than what's currently in effect, whereas if > > it could pass the desired level to compile(), this wouldn't be necessary. > >Makes sense to me; we need a patch of course. But before we can do that, it's not clear to me if it should be part of the existing "flags" argument, or whether it should be separate. Similarly, whether it's just going to be a level or an optimization bitmask in its own right might be relevant too. For the current use case, obviously, a level argument suffices, with 'None' meaning "whatever the command-line level was" for backward compatibility. And I guess we could go with that for now easily enough, I'd just like to know whether any of the AST or optimization mavens had anything they were planning in the immediate future that might affect how the API addition should be structured. From guido at python.org Thu Nov 10 18:53:28 2005 From: guido at python.org (Guido van Rossum) Date: Thu, 10 Nov 2005 09:53:28 -0800 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: <5.1.1.6.0.20051110124246.02bac470@mail.telecommunity.com> References: <20051109023347.GA15823@localhost.localdomain> <5.1.1.6.0.20051109190838.01f51838@mail.telecommunity.com> <5.1.1.6.0.20051110124246.02bac470@mail.telecommunity.com> Message-ID: On 11/10/05, Phillip J. Eby wrote: > At 04:33 PM 11/9/2005 -0800, Guido van Rossum wrote: > >On 11/9/05, Phillip J. Eby wrote: > > > By the way, while we're on this subject, can we make the optimization > > > options be part of the compile() interface? Right now the distutils has to > > > actually exec another Python process whenever you want to compile > > > code with > > > a different optimization level than what's currently in effect, whereas if > > > it could pass the desired level to compile(), this wouldn't be necessary. > > > >Makes sense to me; we need a patch of course. > > But before we can do that, it's not clear to me if it should be part of the > existing "flags" argument, or whether it should be separate. Similarly, > whether it's just going to be a level or an optimization bitmask in its own > right might be relevant too. > > For the current use case, obviously, a level argument suffices, with 'None' > meaning "whatever the command-line level was" for backward > compatibility. And I guess we could go with that for now easily enough, > I'd just like to know whether any of the AST or optimization mavens had > anything they were planning in the immediate future that might affect how > the API addition should be structured. I'm not a big user of this API, please design as you see fit. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Thu Nov 10 18:55:19 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 10 Nov 2005 12:55:19 -0500 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <1131630420.12077.44.camel@warna.corp.google.com> References: <4372DD5F.70203@c2b2.columbia.edu> <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> <87br0tw041.fsf@tleepslib.sk.tsukuba.ac.jp> <4372DD5F.70203@c2b2.columbia.edu> Message-ID: <5.1.1.6.0.20051110125301.02b2d318@mail.telecommunity.com> At 01:47 PM 11/10/2005 +0000, Donovan Baarda wrote: >Twisted is an async-framework that I believe has support for using a >variety of different event-loops, including Tkinter and wxWidgets, as >well as it's own. Technically, it just gives Tkinter a chance to run every so often; you specifically *can't* use Tkinter's event loop. Instead, you run the Twisted event loop after telling it that you'd like Tkinter to be kept in the loop, as it were. But Twisted is definitely worth looking at for this sort of thing. It's the nearest thing to a "standard Python event loop" that exists, apart from the asyncore stuff in the stdlib (which doesn't have any GUI support AFAIK). From mdehoon at c2b2.columbia.edu Thu Nov 10 19:07:17 2005 From: mdehoon at c2b2.columbia.edu (Michiel Jan Laurens de Hoon) Date: Thu, 10 Nov 2005 13:07:17 -0500 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> <87br0tw041.fsf@tleepslib.sk.tsukuba.ac.jp> <4372DD5F.70203@c2b2.columbia.edu> Message-ID: <43738C55.60509@c2b2.columbia.edu> Guido van Rossum wrote: >On 11/9/05, Michiel Jan Laurens de Hoon wrote: > > >>My application doesn't need a toolkit at all. My problem is that because >>of Tkinter being the standard Python toolkit, we cannot have a decent >>event loop in Python. So this is the disadvantage I see in Tkinter. >> >> > >That's a non-sequitur if I ever saw one. Who gave you that idea? There is no connection. > I have come to this conclusion after several years of maintaining a scientific plotting package and trying to set up an event loop for it. Whereas there are some solutions that more or less work, none of them work very well, and the solutions that we found tend to break. Other visualization packages are struggling with the same problem. I'm trying the best I can to explain in my other posts why I feel that Tkinter is the underlying reason, and why it would be difficult to solve. >(If there's *any* reason for Python not having a standard event loop >it's probably because I've never needed one.) > It's probably because we have gotten away with piggy-backing on Tcl's event loop for so long. --Michiel. -- Michiel de Hoon Center for Computational Biology and Bioinformatics Columbia University 1150 St Nicholas Avenue New York, NY 10032 From mcherm at mcherm.com Thu Nov 10 19:12:03 2005 From: mcherm at mcherm.com (Michael Chermside) Date: Thu, 10 Nov 2005 10:12:03 -0800 Subject: [Python-Dev] (no subject) Message-ID: <20051110101203.2v6dz00ya8ogs08o@login.werra.lunarpages.com> Sokolov Yura writes: > Excuse my English No problem. You command of english probably exceeds my command of any other language. > I think, we could just segregate tokens for decimal and real float and > make them interoperable. > Most of us works with business databases - all "floats" are really > decimals, algebraic operations > should work without float inconsistency and those operations rare so > speed is not important. > But some of us use floats for speed in scientific and multimedia programs. I'm not sure why you say "most" (have you seen some surveys of Python programmers that I haven't seen?), but I think we all agree that there are Python users who rarely have a need for machine floats, and others who badly need them. I'll take your specific suggestions out of order: > with "from __future__ import Decimal" we could: > c) result of operation with decimal operands should be decimal > >>> 1.0/3.0 > 0.33333333333333333 This already works. > d) result of operation with float operands should be float > >>> 1.0f/3.0f > 0.33333333333333331f This already works. > e) result of operation with decimal and float should be float (decimal > converts into float and operation perfomed) > >>> 1.0f/3.0 > 0.33333333333333331f > >>> 1.0/3.0f > 0.33333333333333331f Mixing Decimal and float is nearly ALWAYS a user error. Doing it correctly requires significant expertise in the peculiarities of floating point representations. So Python protects the user by throwing exceptions when attempts are made to mix Decimal and floats. This is the desired behavior (and the experts already know how to work around it in the RARE occasions when they need to). > a) interpret regular float constants as decimal > b) interpret float constants with suffix 'f' as float (like 1.5f > 345.2e-5f etc) There are two different ideas here, which I will separate. The first is a proposal that there be a way to provide Decimal literals. The second proposal is that the ###.### be the literal for Decimals and that ###.###f be the literal for floats. I'm in favor of the first idea. Decimals are useful enough that it would be a good idea to provide some sort of literal for their use. This is well worth a PEP. But if we DO agree that we ought to have literals for both floats and Decimals, then we also need to decide which gets the coveted "unadorned decimal literal" (ie, ###.###). Performance argues in favor of floats (they run *MUCH* faster). Usability (particularly for beginners) argues in favor of Decimals (they sometimes still have surprising behavior, but less often than with binary floats). And backward compatibility argues in favor of floats. Myself, I'm an "expert" user (at least to this extent) and I could easily handle either choice. If others felt like me, then its likely that the backward compatibility argument and the need to fight the pervasive meme that "Python is slow" will win the day. -- Michael Chermside From tzot at mediconsa.com Thu Nov 10 20:29:45 2005 From: tzot at mediconsa.com (Christos Georgiou) Date: Thu, 10 Nov 2005 21:29:45 +0200 Subject: [Python-Dev] Building Python with Visual C++ 2005 Express Edition Message-ID: I didn't see any mention of this product in the Python-Dev list, so I thought to let you know. http://msdn.microsoft.com/vstudio/express/visualc/download/ There is also a link for a CD image (.img) file to download. I am downloading now, so I don't know yet whether Python compiles with it without any problems. So if anyone has previous experience, please reply. PS This page ( http://msdn.microsoft.com/vstudio/express/support/faq/default.aspx#pricing ) says that if you download it until Nov 7, 2006, it's a gift --the Microsoft VC++ compiler for free (perhaps a cut-down version). Bits from the FAQ: http://msdn.microsoft.com/vstudio/express/support/faq/default.aspx 4. Can be used even for commercial products without licensing restrictions 40. It includes the optimizing compiler (without stuff like Profile Guided Optimizations) 41. Builds both native and managed applications (you 99% need to download the SDK too) 42. No MFC or ATL included From bcannon at gmail.com Thu Nov 10 20:36:51 2005 From: bcannon at gmail.com (Brett Cannon) Date: Thu, 10 Nov 2005 11:36:51 -0800 Subject: [Python-Dev] dev FAQ updated with day-to-day svn questions In-Reply-To: <43731133.9000904@gmail.com> References: <43731133.9000904@gmail.com> Message-ID: On 11/10/05, Nick Coghlan wrote: > Brett Cannon wrote: > > I just finished fleshing out the dev FAQ > > (http://www.python.org/dev/devfaq.html) with questions covering what > > someone might need to know for regular usage. If anyone thinks I > > didn't cover something I should have, let me know. > > Should the section "Developing on Windows" disappear now? > Well, the whole dev doc section needs cleaning up and that includes the dev FAQ. I was planning on doing this at some point; might as well start talking about it now. In my mind, the steps in each of the major things to do (bugs and patches) needs better docs. With that fleshed out, Intro to Development can act as an overview of the process. This should, together with the dev FAQ, cover what someone needs to do dev work. The question is how to structure the bug/patch guidelines. There are two options; dev FAQ entires much like the svn section or a more classic layout of the info. Both would have a bulleted list of the steps necessary for a bug/patch. The question is whether the information is presented in paragraphs of text following the bulleted list or as a list of questions. What do people prefer? -Brett From martin at v.loewis.de Thu Nov 10 20:40:04 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 10 Nov 2005 20:40:04 +0100 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <43738074.2030508@c2b2.columbia.edu> References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> <4372B82C.9010800@canterbury.ac.nz> <4372DA3A.8010206@c2b2.columbia.edu> <4372F72B.9060501@v.loewis.de> <43738074.2030508@c2b2.columbia.edu> Message-ID: <4373A214.6060201@v.loewis.de> Michiel Jan Laurens de Hoon wrote: >>You are missing multi-threading, which is the widely used >>approach to doing things simultaneously in a single process. > > The problem with threading (apart from potential portability problems) > is that Python doesn't let us know when it's idle. This would cause > excessive repainting (I can give you an explicit example if you're > interested). I don't understand how these are connected: why do you need to know when Python is idle for multi-threaded applications, and why does not knowing that it is idle cause massive repainting? Not sure whether an explicit example would help, though; one would probably need to understand a lot of details of your application. Giving a simplified version of the example might help (which would do 'print "Repainting"' instead of actually repainting). > But there is another solution with threads: Can we let Tkinter run in a > separate thread instead? Yes, you can. Actually, Tkinter *always* runs in a separate thread (separate from all other threads). > This works, but only if Tkinter is installed, and even then it will give > poor performance due to the busy-loop with 20 ms sleep in between in > Tkinter. Furthermore, this will not work with IDLE, because the Python > thread that handles user commands never enters the Tkinter event loop, > even if we import Tkinter. AFAIK, there is no easy solution to this. Here I'm losing track. What is "this" which is no easy solution for? Why do you need a callback when Python is idle in the first place? > I may be wrong here, but I'd think that it would be dangerous to run > Tkinter's event loop when one thread is waiting for another (as happens > in IDLE). I don't understand. Threads don't wait for each other. Threads wait for events (which might be generated by some other thread, of course). However, there is no problem to run the Tkinter event loop when some unnrelated thread is blocked. > Sorry if I'm sounding negative, but we've actually considered many > different things to get the event loop working for our scientific > visualization software, and we were never able to come up with a > satisfactory scheme within the current Python framework. I really don't see what the problem is. Why does the visualization framework care that Tkinter is around? Why are the events that the visualization framework needs to process? Regards, Martin From martin at v.loewis.de Thu Nov 10 20:44:00 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 10 Nov 2005 20:44:00 +0100 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <43738C55.60509@c2b2.columbia.edu> References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> <87br0tw041.fsf@tleepslib.sk.tsukuba.ac.jp> <4372DD5F.70203@c2b2.columbia.edu> <43738C55.60509@c2b2.columbia.edu> Message-ID: <4373A300.3080501@v.loewis.de> Michiel Jan Laurens de Hoon wrote: > I have come to this conclusion after several years of maintaining a > scientific plotting package and trying to set up an event loop for > it. Whereas there are some solutions that more or less work, none of > them work very well, and the solutions that we found tend to break. > Other visualization packages are struggling with the same problem. As you can see, the problem is not familiar to anybody reading python-dev. > I'm trying the best I can to explain in my other posts why I feel > that Tkinter is the underlying reason, and why it would be difficult > to solve. Before trying to explain the reason, please try to explain the problem first. What is it *really* that you want to do which you feel you currently can't do? Regards, Martin From martin at v.loewis.de Thu Nov 10 20:47:04 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 10 Nov 2005 20:47:04 +0100 Subject: [Python-Dev] dev FAQ updated with day-to-day svn questions In-Reply-To: <43731133.9000904@gmail.com> References: <43731133.9000904@gmail.com> Message-ID: <4373A3B8.3090402@v.loewis.de> Nick Coghlan wrote: > Should the section "Developing on Windows" disappear now? I think so, yes (along with the document it refers to). Regards, Martin From martin at v.loewis.de Thu Nov 10 20:52:16 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 10 Nov 2005 20:52:16 +0100 Subject: [Python-Dev] Building Python with Visual C++ 2005 Express Edition In-Reply-To: References: Message-ID: <4373A4F0.7010202@v.loewis.de> Christos Georgiou wrote: > I didn't see any mention of this product in the Python-Dev list, so I > thought to let you know. > > http://msdn.microsoft.com/vstudio/express/visualc/download/ > > There is also a link for a CD image (.img) file to download. > > I am downloading now, so I don't know yet whether Python compiles with it > without any problems. So if anyone has previous experience, please reply. I don't have previous experience, but I think this it likely shares the issues that VS.NET 2005 has with the current code: 1. the project files are for VS.NET 2003. In theory, conversion to the new format is supported, but I don't know whether this conversion works flawlessly. 2. MS broke ISO C conformance in VS.NET 2005 in a way that affects Python's signal handling. There is a patch on SF which addresses the issue, but that hasn't been checked in yet. Regards, Martin From bcannon at gmail.com Thu Nov 10 22:38:50 2005 From: bcannon at gmail.com (Brett Cannon) Date: Thu, 10 Nov 2005 13:38:50 -0800 Subject: [Python-Dev] dev FAQ updated with day-to-day svn questions In-Reply-To: <4373164F.8070606@gmail.com> References: <4373164F.8070606@gmail.com> Message-ID: On 11/10/05, Nick Coghlan wrote: > Brett Cannon wrote: > > I just finished fleshing out the dev FAQ > > (http://www.python.org/dev/devfaq.html) with questions covering what > > someone might need to know for regular usage. If anyone thinks I > > didn't cover something I should have, let me know. > > For question 1.2.10, I believe you also want: > > [miscellany] > enable-auto-props = yes > > so that "svn add" works properly. Added. Missed that I had it in my personal config. =) > > Question 1.4.1 should cover the use of "svn diff" instead of "cvs diff" to > make the patch. > Changed. > On that note, we need to update the patch submission guidelines to point to > SVN instead of CVS (those guidelines also still say context diffs are > preferred to unified diffs, which I believe is no longer true). > Fixed and fixed. -Brett From mhammond at skippinet.com.au Thu Nov 10 22:49:20 2005 From: mhammond at skippinet.com.au (Mark Hammond) Date: Fri, 11 Nov 2005 08:49:20 +1100 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <43738C55.60509@c2b2.columbia.edu> Message-ID: Michiel wrote: > Guido van Rossum wrote: > > >On 11/9/05, Michiel Jan Laurens de Hoon > wrote: > > > > > >>My application doesn't need a toolkit at all. My problem is that because > >>of Tkinter being the standard Python toolkit, we cannot have a decent > >>event loop in Python. So this is the disadvantage I see in Tkinter. > >> > >> > > > >That's a non-sequitur if I ever saw one. Who gave you that idea? > There is no connection. > > > I have come to this conclusion after several years of maintaining > a scientific plotting package and trying to set up an event loop > for it. Whereas there are some solutions that more or less work, > none of them work very well, and the solutions that we found tend > to break. Other visualization packages are struggling with the > same problem. I'm trying the best I can to explain in my other > posts why I feel that Tkinter is the underlying reason, and why > it would be difficult to solve. I believe this problem all boils down to this paragraph from the first mail on this topic: : Currently, event loops are available in Python via PyOS_InputHook, a : pointer to a user-defined function that is called when Python is idle : (waiting for user input). However, an event loop using PyOS_InputHook : has some inherent limitations, so I am thinking about how to improve : event loop support in Python. Either we have an unusual definition of "event loop" (as many many other toolkits have implemented event loops without PyOS_InputHook), or the requirement is for an event loop that plays nicely with the "interactive loop" in Python.exe. Assuming the latter, I would suggest simply not trying to do that! Look at the "code" module for a way you can create your own interactive loop that plays nicely with your event loop (rather than trying to do it the other way around). Otherwise, I suggest you get very specific about what this event loop should do. From a previous mail in this thread (an exchange between you and Martin): > >> Which is why Tkinter needs to be in charge of the event loop. For > >> other GUI toolkits, I don't see a reason why they'd need their own > >> event loop. > > > > They need to fetch events from the operating system level, and dispatch > > them to the widgets. > This is a perfect task for an event loop located in Python, instead of > in an extension module. I believe the point Martin was trying to make is that we have 2 "unknown" quantities here - the "operating system" and the "widgets". Each OS delivers raw GUI events differently, and each GUI framework consumes and generates events differently. I can't see what a single event loop would look like. Even on Windows there is no single, standard "event loop" construct - MFC and VB apps both have custom message loops. Mozilla XUL applications (which are very close to being able to be written in Python ) have an event loop that could not possibly be expressed in Python - but they do expose a way to *call* their standard event loop (which is quite a different thing - you are asking to *implement* it.) > I could write a prototype event loop for Python > to demonstrate how this would work. I think that would be the best way forward - this may all simply be one big misunderstanding . The next step after that would be to find even one person who currently uses an event-loop based app, and for whom your event loop would work. Mark. From ulrich.berning at desys.de Fri Nov 11 14:32:21 2005 From: ulrich.berning at desys.de (Ulrich Berning) Date: Fri, 11 Nov 2005 14:32:21 +0100 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: <5.1.1.6.0.20051110124246.02bac470@mail.telecommunity.com> References: <5.1.1.6.0.20051109190838.01f51838@mail.telecommunity.com> <20051109023347.GA15823@localhost.localdomain> <5.1.1.6.0.20051109190838.01f51838@mail.telecommunity.com> <5.1.1.6.0.20051110124246.02bac470@mail.telecommunity.com> Message-ID: <43749D65.4040001@desys.de> Phillip J. Eby schrieb: >At 04:33 PM 11/9/2005 -0800, Guido van Rossum wrote: > > >>On 11/9/05, Phillip J. Eby wrote: >> >> >>>By the way, while we're on this subject, can we make the optimization >>>options be part of the compile() interface? Right now the distutils has to >>>actually exec another Python process whenever you want to compile >>>code with >>>a different optimization level than what's currently in effect, whereas if >>>it could pass the desired level to compile(), this wouldn't be necessary. >>> >>> >>Makes sense to me; we need a patch of course. >> >> > >But before we can do that, it's not clear to me if it should be part of the >existing "flags" argument, or whether it should be separate. Similarly, >whether it's just going to be a level or an optimization bitmask in its own >right might be relevant too. > >For the current use case, obviously, a level argument suffices, with 'None' >meaning "whatever the command-line level was" for backward >compatibility. And I guess we could go with that for now easily enough, >I'd just like to know whether any of the AST or optimization mavens had >anything they were planning in the immediate future that might affect how >the API addition should be structured. > > > I'm using a totally different approach for the above problem. I have implemented two functions in the sys module, that make the startup flags accessible at runtime. This also solves some other problems I had, as you will see in the examples below: The first function makes most of the flags readable (I have ommited the flags, that are documented as deprecated in the code): sys.getrunflag(name) -> integer Return one of the interpreter run flags. Possible names are 'Optimize', 'Verbose', 'Interactive', 'IgnoreEnvironment', 'Debug', 'DivisionWarning', 'NoSite', 'NoZipImport', 'UseClassExceptions', 'Unicode', 'Frozen', 'Tabcheck'. getrunflag('Optimize') for example returns the current value of Py_OptimizeFlag. The second function makes a few flags writable: sys.setrunflag(name, value) -> integer Set an interpreter run flag. The only flags that can be changed at runtime are Py_VerboseFlag ('Verbose') and Py_OptimizeFlag ('Optimize'). Returns the previous value of the flag. As you can see, I have also introduced the new flag Py_NoZipImport that can be activated with -Z at startup. This bypasses the activation of zipimport and is very handy, if you edit modules stored in the filesystem, that are normally imported from a zip archive and you want to test your modifications. With this flag, there is no need to delete, rename or update the zip archive or to modify sys.path to ensure that your changed modules are imported from the filesystem and not from the zip archive. And here are a few usable examples for the new functions: 1.) You have an application, that does a huge amount of imports and some of them are mysterious, so you want to track them in verbose mode. You could start python with -v or -vv, but then you get hundreds or thousands of lines of output. Instead, you can do the following: import sys import ... import ... oldval = sys.setrunflag('Verbose', 1) # -v, use 2 for -vv import ... import ... sys.setrunflag('Verbose', oldval) import ... import ... Now, you get only verbose messages for the imports that you want to track. 2.) You need to generate optimized byte code (without assertions and docstrings) from a source code, no matter how the interpreter was started: import sys ... source = ... oldval = sys.setrunflag('Optimize', 2) # -OO, use 1 for -O bytecode = compile(source, ...) sys.setrunflag('Optimize', oldval) ... 3.) You have to build a command line for the running application (e.g. for registration in the registry) and need to check, if you are running a script or a frozen executable (this assumes, that your freeze tool sets the Py_FrozenFlag): import sys ... if sys.getrunflag('Frozen'): commandline = sys.executable else: commandline = '%s %s' % (sys.executable, sys.argv[0]) ... NOTE: My own freeze tool sib.py, which is part of the VendorID package (www.riverbankcomputing.co.uk/vendorid) doesn't set the Py_FrozenFlag yet. I will provide an update soon. ---- And now back to the original subject: I have done nearly the same changes, that Osvaldo provided with his patch and I would highly appreciate if this patch goes into the next release. The main reason why I changed the import behavior was pythonservice.exe from the win32 extensions. pythonservice.exe imports the module that contains the service class, but because pythonservice.exe doesn't run in optimized mode, it will only import a .py or a .pyc file, not a .pyo file. Because we always generate bytecode with -OO at distribution time, we either had to change the behavior of pythonservice.exe or change the import behavior of Python. It is essential for us to remove assertions and docstrings in our commercial Python applications at distribution time, because assertions are meaningful only at development time and docstrings may contain implementation details, that our customers should never see (this makes reverse engineering a little bit harder, but not impossible). Another important reason is, that the number of files in the standard library is reduced dramatically. I can have .py and .pyc files in lib/pythonX.Y containing assertions and docstrings and put optimized code in lib/pythonXY.zip. Then, if I need docstrings, I just bypass zipimport as described above with -Z. On the customer site, we only need to install the zip archive (our customers are not developers, just end users; they may not even recognize that we use Python for application development). Guido, if it was intentional to separate slightly different generated bytecode into different files and if you have good reasons for doing this, why have I never seen a .pyoo file :-) For instance, nobody would give the output of a C compiler a different extension when different compiler flags are used. I would appreciate to see the generation of .pyo files completely removed in the next release. Just create them as .pyc files, no matter if -O or -OO is used or not. At runtime, Python should just ignore (jump over?) assert statements when started with -O or ignore assert statements and docstrings when started with -OO if they are in the .pyc file. ---- There are two reasons, why I haven't started a discussion about those issues earlier on this list: 1.) I have done those changes (and a lot of other minor and major changes) in Python 2.3.x at a time when Python 2.4.x came up and we still use Python 2.3.5. I just wanted to wait until I have the next release of our Python runtime environment (RTE) ready that will contain the most recent Python version. 2.) In the last months, my time was limited, because I first had to stabilize the current RTE (developers and customers were waiting on it). A few notes about this Python runtime environment: The RTE that I maintain contains everything (mainly, but not only Python) that is needed to run our applications on the customer site. The applications are distributed as frozen binaries either containing the whole application or only the frozen main script together with an additional zip archive holding the application specific modules and packages. Tools like py2exe or cx_Freeze follow a different approach: they always package a kind of runtime environment together with the application. There is nothing bad with this approach, but if you provide more than one application, you waste resources and it may be harder to maintain (bugfixes, updates, etc.). Our RTE currently runs on AIX, HP-UX, IRIX, Windows and Linux, SunOS/Solaris will follow in the near future. It gets installed together with the application(s) on the customer site if it is not already there in the appropriate version. It is vendor specific (more in the sense of a provider, not necessarily in the sense of a seller) and shouldn't interfere with any other software already installed on the target site. This RTE is the result of years of experience of commercial application development with Python (we started with Python-1.3.x a long long time ago). My guideline is: Application developers and/or customers should not worry about the requirements needed to get the applications running on the target site. Developers should spend all their time for application development and customers can expect a complete installation with only a very small set of system requirements/dependencies. In general, I would like to discuss those things that increase Python's usability especially in a commercial environment and things that make Python more consistent across platforms (e.g. unifying the filesystem layout on Windows and UNIX/Linux), but I don't know if this is the right mailing list. Ulli From jimjjewett at gmail.com Fri Nov 11 15:27:13 2005 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 11 Nov 2005 09:27:13 -0500 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks Message-ID: Ulrich Berning schrieb: [He already has a patch that does much of what is being discussed] > I have also introduced the new flag Py_NoZipImport that > can be activated with -Z at startup. This bypasses the > activation of zipimport I think -Z could be confusing; I would expect it to work more like the recent suggestion that it name a specific Zip file to use as the only (or at least the first) source of modules. I do see that the switch is handy; I'm just suggesting a different name, such as -nozip or -skip file.zip. -jJ From jimjjewett at gmail.com Fri Nov 11 16:32:42 2005 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 11 Nov 2005 10:32:42 -0500 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter - Summary attempt Message-ID: There has been enough misunderstanding in this thread that the summarizers are going to have trouble. So I'm posting this draft in hopes of clarification; please correct me. (1) There is some pre-discussion attached to patches 1049855 and 1252236. Martin Loewis and Michiel de Hoon agreed that the fixes were fragile, and that a larger change should be discussed on python-dev. (2) Michiel writes visualization software; he (and others, such as the writers of matplotlib) has trouble creating a good event loop, because the GUI toolkit (especially Tkinter?) wants its own event loop to be in charge. Note that this isn't the first time this sort of problem has come up; usually it is phrased in terms of a problem with Tix, or not being able to run turtle while in IDLE. Event loops by their very nature are infinite loops; once they start, everything else is out of luck unless it gets triggered by an event or is already started. (3) Donovan Baarda suggested looking at Twisted for state of the art in event loop integration. Unfortunately, as Phillip Eby notes, it works by not using the Tkinter event loop. It decides for itself when to call dooneevent. (4) Michiel doesn't actually need Tkinter (or any other GUI framework?) for his own project, but he has to play nice with it because his users expect to be able to use other tools -- particularly IDLE -- while running his software. (5) It is possible to run Tkinter's dooneevent version as part of your own event loop (as Twisted does), but you can't really listen for its events, so you end up with a busy loop polling, and stepping into lots of "I have nothing to do" functions for every client eventloop. You can use Tkinter's loop, but once it goes to sleep waiting for input, everything sort of stalls out for a while, and even non-Tkinter events get queued instead of processed. (6) Mark Hammond suggests that it might be easier to replace the interactive portions of python based on the "code" module. matplotlib suggests using ipython instead of standard python for similar reasons. If that is really the simplest answer (and telling users which IDE to use is acceptable), then ... I think Michiel has a point. (7) One option might be to always start Tk in a new thread, rather than letting it take over the main thread. There was some concern (see patch 1049855) that Tkinter doesn't -- and shouldn't -- require threading. My thoughts are that some of the biggest problems with the event loop (waiting on a mutex) won't happen in non-threaded python, and that even dummy_thread would be an improvement over the current state (by forcing the event loop to start last). I may well be missing something, but obviously I'm not sure what that is. -jJ From guido at python.org Fri Nov 11 17:15:18 2005 From: guido at python.org (Guido van Rossum) Date: Fri, 11 Nov 2005 08:15:18 -0800 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: <43749D65.4040001@desys.de> References: <20051109023347.GA15823@localhost.localdomain> <5.1.1.6.0.20051109190838.01f51838@mail.telecommunity.com> <5.1.1.6.0.20051110124246.02bac470@mail.telecommunity.com> <43749D65.4040001@desys.de> Message-ID: On 11/11/05, Ulrich Berning wrote: > Guido, if it was intentional to separate slightly different generated > bytecode into different files and if you have good reasons for doing > this, why have I never seen a .pyoo file :-) Because -OO was an afterthought and not implemented by me. > For instance, nobody would give the output of a C compiler a different > extension when different compiler flags are used. But the usage is completely different. With C you explicitly manage when compilation happens. With Python you don't. When you first run your program with -O but it crashes, and then you run it again without -O to enable assertions, you would be very unhappy if the bytecode cached in a .pyo file would be reused! > I would appreciate to see the generation of .pyo files completely > removed in the next release. You seem to forget the realities of backwards compatibility. While there are ways to cache bytecode without having multiple extensions, we probably can't do that until Python 3.0. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mdehoon at c2b2.columbia.edu Fri Nov 11 17:58:31 2005 From: mdehoon at c2b2.columbia.edu (Michiel Jan Laurens de Hoon) Date: Fri, 11 Nov 2005 11:58:31 -0500 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter - Summary attempt In-Reply-To: References: Message-ID: <4374CDB7.2020001@c2b2.columbia.edu> I think this is an excellent summary of the discussion so far. Probably clearer than my own posts. Thanks, Jim! --Michiel. Jim Jewett wrote: >There has been enough misunderstanding in this thread >that the summarizers are going to have trouble. So I'm >posting this draft in hopes of clarification; please correct >me. > >(1) There is some pre-discussion attached to patches >1049855 and 1252236. Martin Loewis and Michiel >de Hoon agreed that the fixes were fragile, and that a >larger change should be discussed on python-dev. > >(2) Michiel writes visualization software; he (and >others, such as the writers of matplotlib) has trouble >creating a good event loop, because the GUI toolkit >(especially Tkinter?) wants its own event loop to be in >charge. > >Note that this isn't the first time this sort of problem has >come up; usually it is phrased in terms of a problem with >Tix, or not being able to run turtle while in IDLE. > >Event loops by their very nature are infinite loops; >once they start, everything else is out of luck unless it >gets triggered by an event or is already started. > >(3) Donovan Baarda suggested looking at Twisted for >state of the art in event loop integration. Unfortunately, >as Phillip Eby notes, it works by not using the Tkinter >event loop. It decides for itself when to call dooneevent. > >(4) Michiel doesn't actually need Tkinter (or any other GUI >framework?) for his own project, but he has to play nice >with it because his users expect to be able to use other >tools -- particularly IDLE -- while running his software. > >(5) It is possible to run Tkinter's dooneevent version >as part of your own event loop (as Twisted does), but >you can't really listen for its events, so you end up with >a busy loop polling, and stepping into lots of "I have >nothing to do" functions for every client eventloop. > >You can use Tkinter's loop, but once it goes to sleep >waiting for input, everything sort of stalls out for a while, >and even non-Tkinter events get queued instead of >processed. > >(6) Mark Hammond suggests that it might be easier to >replace the interactive portions of python based on the >"code" module. matplotlib suggests using ipython >instead of standard python for similar reasons. > >If that is really the simplest answer (and telling users >which IDE to use is acceptable), then ... I think Michiel >has a point. > >(7) One option might be to always start Tk in a new >thread, rather than letting it take over the main thread. > >There was some concern (see patch 1049855) that >Tkinter doesn't -- and shouldn't -- require threading. > >My thoughts are that some of the biggest problems >with the event loop (waiting on a mutex) won't happen >in non-threaded python, and that even dummy_thread >would be an improvement over the current state (by >forcing the event loop to start last). I may well be >missing something, but obviously I'm not sure what that is. > >-jJ >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: http://mail.python.org/mailman/options/python-dev/mdehoon%40c2b2.columbia.edu > > -- Michiel de Hoon Center for Computational Biology and Bioinformatics Columbia University 1150 St Nicholas Avenue New York, NY 10032 From mdehoon at c2b2.columbia.edu Fri Nov 11 18:56:57 2005 From: mdehoon at c2b2.columbia.edu (Michiel Jan Laurens de Hoon) Date: Fri, 11 Nov 2005 12:56:57 -0500 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <4373A300.3080501@v.loewis.de> References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> <87br0tw041.fsf@tleepslib.sk.tsukuba.ac.jp> <4372DD5F.70203@c2b2.columbia.edu> <43738C55.60509@c2b2.columbia.edu> <4373A300.3080501@v.loewis.de> Message-ID: <4374DB69.2080804@c2b2.columbia.edu> Martin v. L?wis wrote: >Before trying to explain the reason, please try to explain the >problem first. What is it *really* that you want to do which >you feel you currently can't do? > > Probably I should have started the discussion with this; sorry if I confused everybody. But here it is: I have an extension module for scientific visualization. This extension module opens one or more windows, in which plots can be made. Something similar to the plotting capabilities of Matlab. For the graphics windows to remain responsive, I need to make sure that its events get handled. So I need an event loop. At the same time, the user can enter new Python commands, which also need to be handled. To achieve this, I make use of PyOS_InputHook, a pointer to a function which gets called just before going into fgets to read the next Python command. I use PyOS_InputHook to enter an event loop inside my extension module. This event loop handles the window events, and returns as soon as a new Python command is available at stdin, at which point we continue to fgets as usual. While this approach more or less works, there are two problems that I have run into: 1) What if the user decides to import Tkinter next? Tkinter notices that PyOS_InputHook is already set, and does not reset it to its own event loop. Hence, Tkinter's events are not handled. Similarly, if a user imports Tkinter before my extension module, I don't reset PyOS_InputHook, so Tkinter's events are handled but not mine. If I were to reset PyOS_InputHook to my extension module's event loop, then my events get handled but not Tkinter's. 2) On Windows, many users will use IDLE to run Python. IDLE uses two Python threads, one for the GUI and one for the user's Python commands. Each has its own PyOS_InputHook. If I import my extension module (or Tkinter, for that matter), the user-Python's PyOS_InputHook gets set to the corresponding event loop function. So far so good. However, PyOS_InputHook doesn't actually get called: Between Python commands, the GUI-Python is waiting for the user to type something, and the user-Python is waiting for the GUI-Python. While the user-Python is waiting for the GUI-Python thread, no call is made to PyOS_InputHook, therefore we don't enter an event loop, and no events get handled. Hence, neither my extension module nor Tkinter work when run from IDLE. --Michiel. -- Michiel de Hoon Center for Computational Biology and Bioinformatics Columbia University 1150 St Nicholas Avenue New York, NY 10032 From fredrik at pythonware.com Fri Nov 11 19:14:57 2005 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 11 Nov 2005 19:14:57 +0100 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter - Summary attempt References: Message-ID: Jim Jewett wrote: > (6) Mark Hammond suggests that it might be easier to > replace the interactive portions of python based on the > "code" module. matplotlib suggests using ipython > instead of standard python for similar reasons. > > If that is really the simplest answer (and telling users > which IDE to use is acceptable), then ... I think Michiel > has a point. really? Python comes with a module that makes it trivial to get a fully working interpreter console under any kind of UI toolkit with very little effort, and you think that proves that we need to reengineer the CPython interpreter to support arbitary event loops so you don't have to use that module? as usual, you make absolutely no sense whatsoever. (or was "..." short for "CPython's interactive mode should use the code module" ?) From jimjjewett at gmail.com Fri Nov 11 21:44:34 2005 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 11 Nov 2005 15:44:34 -0500 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter Message-ID: >> (6) Mark Hammond suggests that it might be easier to >> replace the interactive portions of python based on the >> "code" module. matplotlib suggests using ipython >> instead of standard python for similar reasons. >> If that is really the simplest answer (and telling users >> which IDE to use is acceptable), then ... I think Michiel >> has a point. Fredrik Lundh wrote: > really? Python comes with a module that makes it trivial to get > a fully working interpreter console ... Using an event loop (or an external GUI) should not require forking the entire interactive mode, no matter how trivial that fork is. The subtle differences between interactive mode and IDLE already cause occasional problems; the same would be true of code.interact() if it were more widely used. Part of Michiel's pain is that users want to make their own decisions on whether to use IDLE or emacs or vt100, and they want to mix and match toolkits. They already run into unexpected freezes because of the event loop conflicts. If every extension writer also relied on their own subclasses of the interactive mode, users would be in for far more unpleasant surprises. The right answer might be to run each event loop in a separate thread. The answer might be to have a python event loop that re-dispatches single events to the other frameworks. The answer might be a way to chain PyOS_InputHook functions like atexit does for shutdown functions. The answer might be something else entirely. But I'm pretty sure that the right answer does not involve adding an additional layer of potential incompatibility. -jJ From fredrik at pythonware.com Fri Nov 11 22:34:19 2005 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 11 Nov 2005 22:34:19 +0100 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter References: Message-ID: Jim Jewett wrote: > > really? Python comes with a module that makes it trivial to get > > a fully working interpreter console ... > > Using an event loop (or an external GUI) should not require > forking the entire interactive mode, no matter how trivial that > fork is. repeating a bogus argument doesn't make it any better. From skip at pobox.com Sat Nov 12 03:32:57 2005 From: skip at pobox.com (skip@pobox.com) Date: Fri, 11 Nov 2005 20:32:57 -0600 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <4374DB69.2080804@c2b2.columbia.edu> References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> <87br0tw041.fsf@tleepslib.sk.tsukuba.ac.jp> <4372DD5F.70203@c2b2.columbia.edu> <43738C55.60509@c2b2.columbia.edu> <4373A300.3080501@v.loewis.de> <4374DB69.2080804@c2b2.columbia.edu> Message-ID: <17269.21593.575449.78938@montanaro.dyndns.org> Michiel> 1) What if the user decides to import Tkinter next? Tkinter Michiel> notices that PyOS_InputHook is already set, and does not Michiel> reset it to its own event loop. Hence, Tkinter's events are Michiel> not handled. Similarly, if a user imports Tkinter before my Michiel> extension module, I don't reset PyOS_InputHook, so Tkinter's Michiel> events are handled but not mine. If I were to reset Michiel> PyOS_InputHook to my extension module's event loop, then my Michiel> events get handled but not Tkinter's. This sounds sort of like the situation that existed with sys.exitfunc before the creation of the atexit module. Can't we develop an API similar to that so that many different event-loop-wanting packages can play nice together? (Then again, maybe I'm just being too simpleminded.) Skip From avi at argo.co.il Thu Nov 10 10:26:05 2005 From: avi at argo.co.il (Avi Kivity) Date: Thu, 10 Nov 2005 11:26:05 +0200 Subject: [Python-Dev] indented longstrings? Message-ID: <4373122D.30507@argo.co.il> Python's longstring facility is very useful, but unhappily breaks indentation. I find myself writing code like msg = ('From: %s\r\n' + 'To: %s\r\n' + 'Subject: Host failure report for %s\r\n' + 'Date: %s\r\n' + '\r\n' + '%s\r\n') % (fr, ', '.join(to), host, time.ctime(), err) mail.sendmail(fr, to, msg) instead of msg = ('''From: %s To: %s Subject: Host failure report for %s Date: %s %s ''') % (fr, ', '.join(to), host, time.ctime(), err) mail.sendmail(fr, to, msg) while wishing for a msg = i'''From: %s To: %s\r\n' Subject: Host failure report for %s Date: %s %s ''' % (fr, ', '.join(to), host, time.ctime(), err) mail.sendmail(fr, to, msg.replace('\n', '\r\n')) isn't it so much prettier? (((an indented longstring, i''' ... ''' behaves like a regular longstring except that indentation on the lines following the beginning of the longstring is stripped up to the first character position of the longstring on the first line. non-blanks before that character position are a syntax error))) Avi From falcon at intercable.ru Thu Nov 10 21:48:14 2005 From: falcon at intercable.ru (Sokolov Yura) Date: Thu, 10 Nov 2005 23:48:14 +0300 Subject: [Python-Dev] (no subject) Message-ID: <4373B20E.3060002@intercable.ru> > > >Mixing Decimal and float is nearly ALWAYS a user error. Doing it correctly >requires significant expertise in the peculiarities of floating point >representations. > So that I think user should declare floats explicitly (###.###f) - he will fall into float space only if he wish it. >So Python protects the user by throwing exceptions when >attempts are made to mix Decimal and floats. I hate it. I want to get float when I wish to get float. In that case i would like to write #f. I want to stay with decimals by default. (and I want decimals written in C) But it just an opinion of young inexperienced/unpractised man. Excuse my English. From skip at pobox.com Sat Nov 12 03:56:32 2005 From: skip at pobox.com (skip@pobox.com) Date: Fri, 11 Nov 2005 20:56:32 -0600 Subject: [Python-Dev] indented longstrings? In-Reply-To: <4373122D.30507@argo.co.il> References: <4373122D.30507@argo.co.il> Message-ID: <17269.23008.424606.403292@montanaro.dyndns.org> Avi> Python's longstring facility is very useful, but unhappily breaks Avi> indentation. I find myself writing code like Avi> msg = ('From: %s\r\n' Avi> + 'To: %s\r\n' Avi> + 'Subject: Host failure report for %s\r\n' Avi> + 'Date: %s\r\n' Avi> + '\r\n' Avi> + '%s\r\n') % (fr, ', '.join(to), host, time.ctime(), err) Avi> mail.sendmail(fr, to, msg) This really belongs on comp.lang.python, at least until you've exhausted the existing possibilities and found them lacking. However, try: msg = ('From: %s\r\n' 'To: %s\r\n' 'Subject: Host failure report for %s\r\n' 'Date: %s\r\n' '\r\n' '%s\r\n') % (fr, ', '.join(to), host, time.ctime(), err) or msg = ('''\ From: %s To: %s Subject: Host failure report for %s Date: %s %s ') % (fr, ', '.join(to), host, time.ctime(), err) or (untested) def istring(s): return re.sub(r"(\r?\n)\s+", r"\1", s) msg = """From: %s To: %s Subject: Host failure report for %s Date: %s %s """ msg = istring(msg) % (fr, ', '.join(to), host, time.ctime(), err) Skip From greg.ewing at canterbury.ac.nz Sat Nov 12 04:12:58 2005 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 12 Nov 2005 16:12:58 +1300 Subject: [Python-Dev] Weak references: dereference notification In-Reply-To: <1131624778.4292.22.camel@localhost> References: <1131536425.9130.10.camel@localhost> <437228E4.4070800@zope.com> <1131556500.9130.18.camel@localhost> <1131558739.9130.40.camel@localhost> <9E82C8B1-8A32-457D-827A-F0135EB9F8D3@mac.com> <1131576278.8540.14.camel@localhost.localdomain> <43729B07.6010907@canterbury.ac.nz> <1131624778.4292.22.camel@localhost> Message-ID: <43755DBA.8070709@canterbury.ac.nz> Gustavo J. A. M. Carneiro wrote: > The object isn't really destroyed. Simply ob_refcnt drops to zero, > then tp_dealloc is called, which is supposed to destroy it. But since I > wrote tp_dealloc, I choose not to destroy it, Be aware that a C subclass of your wrapper that overrides tp_dealloc is going to have its tp_dealloc called before yours, and will therefore be partly destroyed before you get control. Greg From bob at redivi.com Sat Nov 12 04:31:34 2005 From: bob at redivi.com (Bob Ippolito) Date: Fri, 11 Nov 2005 19:31:34 -0800 Subject: [Python-Dev] indented longstrings? In-Reply-To: <4373122D.30507@argo.co.il> References: <4373122D.30507@argo.co.il> Message-ID: <2BF318ED-4FB9-4E7B-A3E2-1AA131E63B16@redivi.com> On Nov 10, 2005, at 1:26 AM, Avi Kivity wrote: > Python's longstring facility is very useful, but unhappily breaks > indentation. I find myself writing code like http://docs.python.org/lib/module-textwrap.html -bob From s.joaopaulo at gmail.com Sat Nov 12 05:23:42 2005 From: s.joaopaulo at gmail.com (=?ISO-8859-1?Q?Jo=E3o_Paulo_Silva?=) Date: Sat, 12 Nov 2005 04:23:42 +0000 Subject: [Python-Dev] Inconsistent behaviour in import/zipimport hooks In-Reply-To: References: <20051109023347.GA15823@localhost.localdomain> <5.1.1.6.0.20051109190838.01f51838@mail.telecommunity.com> <5.1.1.6.0.20051110124246.02bac470@mail.telecommunity.com> <43749D65.4040001@desys.de> Message-ID: <787073ca0511112023l29794930n@mail.gmail.com> Hi (first post here, note that English is not my native language), One thing we shoudn't forgot is that Osvaldo is porting Python to a plataform that has not so much disk space. He needs Python modules with just the essencial. I like ideias like __debug__ opcode, but in Osvaldo use case, there are limitations to expanding a Python module size. What the problem for the interpreter looks for both .pyc and .pyo? I believe zipimport way to lookup modules is more useful. -- At? mais.. Jo?o Paulo da Silva LinuxUser #355914 ICQ: 265770691 | Jabber: joaopinga at jabber.org PS: Guido, sorry the PVT... From greg.ewing at canterbury.ac.nz Sat Nov 12 05:33:31 2005 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 12 Nov 2005 17:33:31 +1300 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <17269.21593.575449.78938@montanaro.dyndns.org> References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> <87br0tw041.fsf@tleepslib.sk.tsukuba.ac.jp> <4372DD5F.70203@c2b2.columbia.edu> <43738C55.60509@c2b2.columbia.edu> <4373A300.3080501@v.loewis.de> <4374DB69.2080804@c2b2.columbia.edu> <17269.21593.575449.78938@montanaro.dyndns.org> Message-ID: <4375709B.4010009@canterbury.ac.nz> skip at pobox.com wrote: > This sounds sort of like the situation that existed with sys.exitfunc before > the creation of the atexit module. Can't we develop an API similar to that > so that many different event-loop-wanting packages can play nice together? I can't see how that would help. If the different hooks know nothing about each other, there's no way for one to know when to give up control to the next one in the chain. Greg From greg.ewing at canterbury.ac.nz Sat Nov 12 05:39:57 2005 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 12 Nov 2005 17:39:57 +1300 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <4374DB69.2080804@c2b2.columbia.edu> References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> <87br0tw041.fsf@tleepslib.sk.tsukuba.ac.jp> <4372DD5F.70203@c2b2.columbia.edu> <43738C55.60509@c2b2.columbia.edu> <4373A300.3080501@v.loewis.de> <4374DB69.2080804@c2b2.columbia.edu> Message-ID: <4375721D.6040907@canterbury.ac.nz> Michiel Jan Laurens de Hoon wrote: > I have an extension module for scientific visualization. This extension > module opens one or more windows, in which plots can be made. What sort of windows are these? Are you using an existing GUI toolkit, or rolling your own? > For the graphics windows to remain responsive, I need to make sure that > its events get handled. So I need an event loop. How about running your event loop in a separate thread? Greg From skip at pobox.com Sat Nov 12 06:15:03 2005 From: skip at pobox.com (skip@pobox.com) Date: Fri, 11 Nov 2005 23:15:03 -0600 Subject: [Python-Dev] Event loops, PyOS_InputHook, and Tkinter In-Reply-To: <4375709B.4010009@canterbury.ac.nz> References: <437100A7.5050907@c2b2.columbia.edu> <43710C95.30209@v.loewis.de> <43729CAB.5070106@c2b2.columbia.edu> <87br0tw041.fsf@tleepslib.sk.tsukuba.ac.jp> <4372DD5F.70203@c2b2.columbia.edu> <43738C55.60509@c2b2.columbia.edu> <4373A300.3080501@v.loewis.de> <4374DB69.2080804@c2b2.columbia.edu> <17269.21593.575449.78938@montanaro.dyndns.org> <4375709B.4010009@canterbury.ac.nz> Message-ID: <17269.31319.806622.939477@montanaro.dyndns.org> >> This sounds sort of like the situation that existed with sys.exitfunc >> before the creation of the atexit module. Can't we develop an API >> similar to that so that many different event-loop-wanting packages >> can play nice together? Greg> I can't see how that would help. If the different hooks know Greg> nothing about each other, there's no way for one to know when to Greg> give up control to the next one in the chain. If I have a Gtk app I have to feed other (socket, callback) pairs to it. It takes care of adding it to the select() call. Python could dictate that the way to play ball is for other packages (Tkinter, PyGtk, wxPython, etc) to feed Python the (socket, callback) pair. Then you have a uniform way to control event-driven applications. Today, a package like Michiel's has no idea what sort of event loop it will encounter. If Python provided the event loop API it would be the same no matter what widget set happened to be used. The sticking point is probably that a number of such packages presume they will always provide the main event loop and have to way to feed their sockets to another event loop controller. That might present some hurdles for the various package writers/Python wrappers. Skip From skip at pobox.com Sat Nov 12 14:21:59 2005 From: skip at pobox.com (skip@pobox.com) Date: Sat, 12 Nov 2005 07:21:59 -0600 Subject: [Python-Dev] Mapping cvs version numbers to svn revisions? Message-ID: <17269.60535.243900.974801@montanaro.dyndns.org> In a bug report I filed Neal Norwitz referred me to an earlier, fixed, bug report from before the cvs-to-svn switch. The file versions were thus cvs version numbers instead of svn revisions. Is it possible to map from cvs version number to svn? In this particular situation I can fairly easily infer the revision number because I know Neal made the change and roughly where in the given file(s) he was making changes, but I doubt that would always be true. I guess, did cvstosvn save that mapping somewhere? Thx, Skip From p.f.moore at gmail.com Sat Nov 12 16:20:35 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 12 Nov 2005 15:20:35 +0000 Subject: [Python-Dev] Building Python with Visual C++ 2005 Express Edition In-Reply-To: <4373A4F0.7010202@v.loewis.de> References: <4373A4F0.7010202@v.loewis.de> Message-ID: <79990c6b0511120720w2c8b318do3a41051ba4eb0c6b@mail.gmail.com> On 11/10/05, "Martin v. L?wis" wrote: > Christos Georgiou wrote: > > I didn't see any mention of this product in the Python-Dev list, so I > > thought to let you know. > > > > http://msdn.microsoft.com/vstudio/express/visualc/download/ > > > > There is also a link for a CD image (.img) file to download. > > > > I am downloading now, so I don't know yet whether Python compiles with it > > without any problems. So if anyone has previous experience, please reply. > > I don't have previous experience, but I think this it likely shares the > issues that VS.NET 2005 has with the current code: > 1. the project files are for VS.NET 2003. In theory, conversion to > the new format is supported, but I don't know whether this conversion > works flawlessly. > 2. MS broke ISO C conformance in VS.NET 2005 in a way that affects > Python's signal handling. There is a patch on SF which addresses > the issue, but that hasn't been checked in yet. FWIW, I downloaded Visual C++ 2005 Express edition, and the latest platform SDK, and had a go at building Python trunk. I just followed the build instructions from PCBuild\readme.txt as best I could - of the optional packages, I only got zlib to work. The issues with the other modules may or may not be serious - for example, bzip2 is now at version 1.0.3, and the old source isn't available. Just renaming the directory didn't work, but I didn't bother investigating further. I applied the patch you mentioned, but otherwise left everything unchanged. The project file conversions seemed to go fine, and the debug builds were OK, although the deprecation warnings for all the "insecure" CRT functions was a pain. It might be worth adding _CRT_SECURE_NO_DEPRECATE to the project defines somehow. I then ran the test suite, which mostly worked. Results: 235 tests OK. 5 tests failed: test_asynchat test_cookie test_grammar test_mmap test_profile 58 tests skipped: test__locale test_aepack test_al test_applesingle test_bsddb185 test_bsddb3 test_cd test_cl test_cmd_line test_code test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_coding test_commands test_crypt test_curses test_dbm test_dl test_fcntl test_float test_fork1 test_functional test_gdbm test_gl test_grp test_hashlib test_hashlib_speed test_imgfile test_ioctl test_largefile test_linuxaudiodev test_macfs test_macostools test_mhlib test_nis test_normalization test_openpty test_ossaudiodev test_plistlib test_poll test_posix test_pty test_pwd test_resource test_scriptpackages test_signal test_socket_ssl test_socketserver test_sunaudiodev test_threadsignals test_timeout test_timing test_urllib2net test_urllibnet test_xdrlib 7 skips unexpected on win32: test_hashlib test_cmd_line test_xdrlib test_code test_float test_coding test_functional I'm not sure what to make of the "unexpected" skips... The output for the failed tests was: test_asynchat test test_asynchat produced unexpected output: ********************************************************************** *** lines 2-3 of actual output doesn't appear in expected output after line 1: + Connected + Received: 'hello world' ********************************************************************** test_cookie test test_cookie produced unexpected output: ********************************************************************** *** mismatch between lines 3-4 of expected output and lines 3-4 of actual output: - Set-Cookie: chips=ahoy + Set-Cookie: chips=ahoy; ? + - Set-Cookie: vienna=finger + Set-Cookie: vienna=finger; ? + *** mismatch between line 6 of expected output and line 6 of actual output: - Set-Cookie: chips=ahoy + Set-Cookie: chips=ahoy; ? + *** mismatch between line 8 of expected output and line 8 of actual output: - Set-Cookie: vienna=finger + Set-Cookie: vienna=finger; ? + *** mismatch between line 10 of expected output and line 10 of actual output: - Set-Cookie: keebler="E=mc2; L=\"Loves\"; fudge=\012;" + Set-Cookie: keebler="E=mc2; L=\"Loves\"; fudge=\012;"; ? + *** mismatch between line 12 of expected output and line 12 of actual output: - Set-Cookie: keebler="E=mc2; L=\"Loves\"; fudge=\012;" + Set-Cookie: keebler="E=mc2; L=\"Loves\"; fudge=\012;"; ? + *** mismatch between line 14 of expected output and line 14 of actual output: - Set-Cookie: keebler=E=mc2 + Set-Cookie: keebler=E=mc2; ? + *** mismatch between lines 16-17 of expected output and lines 16-17 of actual output: - Set-Cookie: keebler=E=mc2 + Set-Cookie: keebler=E=mc2; ? + - Set-Cookie: Customer="WILE_E_COYOTE"; Path=/acme + Set-Cookie: Customer="WILE_E_COYOTE"; Path=/acme; ? + *** mismatch between line 19 of expected output and line 19 of actual output: -